P2 slice 7: backups and restore (D-30, D-31)

The server backs each book up itself: a per-book git archive, pushed
fast-forward only to a git remote and to a folder, when the books open,
after a quiet spell, before they close and on request; every commit,
push and drill is a record in ops/backup.jsonl. Drills clone each
target and verify the chain, the blobs, the vault and the head, weekly
and after a failure. `wanderbooks books restore` previews, then
restores beside or (with --replace) over, never moving books something
holds. The archive refuses anything but a plain bare repository and
pins git against the owner's configuration. books.backup and
books.back_up_now for the agent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T3gXMVt7XJPi6ens1or9Dj
This commit is contained in:
swaitsandClaude Opus 5.5 committed 2026-10-20 11:14:21 -06:00
1 parent 5bc0633cdd
commit 019d8d76a3
29 files changed
+6443 -136

No files matched your search

File diff suppressed because it is too large. Load diff
+127 -9
View File
@@ -19,8 +19,7 @@ use std::time::{Duration, Instant};
/// Short timings for tests (the defaults are the plan's), and agents' /// Short timings for tests (the defaults are the plan's), and agents'
/// tokens kept in files in the test root, not this computer's keyring. /// tokens kept in files in the test root, not this computer's keyring.
const KNOBS: &str = const KNOBS: &str = "close_after=2500,backoff=50,exe_poll=100,init_wait=5000,queue_wait=30000,file_keyring=1,backup_quiet=300";
"close_after=2500,backoff=50,exe_poll=100,init_wait=5000,queue_wait=30000,file_keyring=1";
/// A home with a data root, and the program the tests run (possibly a /// A home with a data root, and the program the tests run (possibly a
/// copy, for restarts). /// copy, for restarts).
@@ -921,17 +920,13 @@ fn the_lease_socket_takes_only_the_owners_intents() {
"{\"proto\":1,\"intent\":{\"consent_format_upgrade\":{\"book\":\"acme\"}}}\n", "{\"proto\":1,\"intent\":{\"consent_format_upgrade\":{\"book\":\"acme\"}}}\n",
"passphrase", "passphrase",
), ),
(
"{\"proto\":1,\"intent\":{\"restore\":{\"staged\":\"s1\",\"replace\":\"acme\"}}}\n",
"passphrase",
),
( (
"{\"proto\":1,\"intent\":{\"back_up_now\":{\"book\":\"acme\"}}}\n", "{\"proto\":1,\"intent\":{\"back_up_now\":{\"book\":\"acme\"}}}\n",
"isn't built yet", "books.back_up_now",
), ),
( (
"{\"proto\":1,\"intent\":{\"restore\":{\"staged\":\"s1\"}}}\n", "{\"proto\":1,\"intent\":{\"restore\":{\"source\":\"/x\",\"id\":\"acme\",\"replace\":true}}}\n",
"isn't built yet", "wanderbooks books restore",
), ),
( (
"{\"proto\":1,\"intent\":{\"new_books\":{\"id\":\"beta\",\"currency\":\"USD\"}}}\n", "{\"proto\":1,\"intent\":{\"new_books\":{\"id\":\"beta\",\"currency\":\"USD\"}}}\n",
@@ -2346,3 +2341,126 @@ fn the_commands_given_run_as_the_agents_own_token() {
assert!(String::from_utf8_lossy(&out.stderr).contains("isn't UTF-8")); assert!(String::from_utf8_lossy(&out.stderr).contains("isn't UTF-8"));
a.end(); a.end();
} }
#[test]
fn a_backup_is_restored_from_the_command_line() {
let home = Home::new("restore");
home.create("acme");
let disk = home.dir.path().join("disk");
std::fs::create_dir(&disk).unwrap();
let mut a = Agent::start(&home, "acme");
a.initialize();
chart(&mut a);
post_one(&mut a, 1).unwrap();
a.call(
"books.set_setting",
json!({"changes": [{"key": "backup.folder", "value": disk.display().to_string()}],
"decided_by": "Test Owner"}),
)
.unwrap();
// Backed up by the server alone, soon after (the host passed on its
// quiet).
let mut s = Value::Null;
assert!(
wait_for(30, || {
s = a.call("books.backup", json!({})).unwrap();
s["targets"][0]["through_seq"] == s["head_seq"]
}),
"never backed up: {s}"
);
assert_eq!(s["targets"][0]["last_error"], Value::Null, "{s}");
let head = s["head_seq"].as_u64().unwrap();
let backup = disk.join("acme.git");
let backup = backup.to_str().unwrap();
// Previewed, and not done without a yes.
let out = home.books(&["restore", backup, "--id", "copy"]);
let said = String::from_utf8_lossy(&out.stdout);
assert!(said.contains(&format!("through event {head}")), "{out:?}");
assert!(
String::from_utf8_lossy(&out.stderr).contains("confirm at a terminal"),
"{out:?}"
);
assert_eq!(out.status.code(), Some(1));
// Under its own name, over books that exist: refused.
let out = home.books(&["restore", backup, "--yes"]);
assert!(
String::from_utf8_lossy(&out.stderr).contains("already books called"),
"{out:?}"
);
// A repository named otherwise: restored as the books it holds, by
// default (here: over those, refused), said.
let renamed = home.dir.path().join("disk").join("fernhill-backup.git");
let out = Command::new("git")
.args(["clone", "-q", "--bare"])
.arg(backup)
.arg(&renamed)
.output()
.unwrap();
assert!(out.status.success(), "{out:?}");
let out = home.books(&["restore", renamed.to_str().unwrap(), "--yes"]);
assert!(
String::from_utf8_lossy(&out.stderr).contains("already books called \"acme\""),
"{out:?}"
);
let out = home.books(&["restore", renamed.to_str().unwrap(), "--id", "other"]);
let said = String::from_utf8_lossy(&out.stdout);
assert!(
said.contains("It is a backup of the books \"acme\""),
"{out:?}"
);
assert!(said.contains("--id acme"), "{out:?}");
// A repository whose name is no id: still restored as its books.
let odd = home.dir.path().join("disk").join("Fernhill_Backup.git");
std::fs::rename(&renamed, &odd).unwrap();
let out = home.books(&["restore", odd.to_str().unwrap(), "--yes"]);
assert!(
String::from_utf8_lossy(&out.stderr).contains("already books called \"acme\""),
"{out:?}"
);
std::fs::rename(&odd, &renamed).unwrap();
// Beside them.
let out = home.books(&["restore", backup, "--id", "copy", "--yes"]);
assert!(out.status.success(), "{out:?}");
assert!(
String::from_utf8_lossy(&out.stdout).contains("Restored as \"copy\""),
"{out:?}"
);
// Replacing them, the old kept: from the repository named otherwise,
// given no id (the books it holds are replaced).
post_one(&mut a, 2).unwrap();
let out = home.books(&["restore", renamed.to_str().unwrap(), "--replace", "--yes"]);
assert!(out.status.success(), "{out:?}");
let said = String::from_utf8_lossy(&out.stdout);
assert!(
said.contains("kept in the legacy folder as acme-"),
"{out:?}"
);
// The head it restored (post 2 may have been backed up by then).
let restored: u64 = said
.split("Restored as \"acme\": through event ")
.nth(1)
.and_then(|rest| rest.split(' ').next())
.and_then(|n| n.parse().ok())
.expect("the restored head is said");
assert!(restored >= head, "{said}");
let head = restored;
assert!(!said.contains(&home.root().display().to_string()), "{said}");
let out = home.books(&["list"]);
let listed = String::from_utf8_lossy(&out.stdout);
assert!(
listed.contains("acme open") && listed.contains("copy open"),
"{out:?}"
);
// The session of before never reaches the books replaced: refused, or
// (its bridge opening a new session) the restored books.
if let Ok(start) = a.call("books.start", json!({})) {
assert_eq!(start["head"]["seq"].as_u64(), Some(head), "{start}");
}
a.end();
// A new session sees the restored books, at the backup's head.
let mut fresh = Agent::start(&home, "acme");
fresh.initialize();
let start = fresh.call("books.start", json!({})).unwrap();
assert_eq!(start["head"]["seq"].as_u64(), Some(head), "{start}");
fresh.end();
}
+99
View File
@@ -122,6 +122,12 @@ pub enum Kind {
NameControl, NameControl,
/// An asset tag prefix: one to eight letters or digits. /// An asset tag prefix: one to eight letters or digits.
TagPrefix, TagPrefix,
/// A repository's address, as git takes it: one line, never one that
/// reads as an option.
Repository,
/// A folder named from the root of its disk: one line, an absolute
/// path.
Folder,
} }
/// One setting the books know. /// One setting the books know.
@@ -360,6 +366,20 @@ pub static KEYS: &[Key] = &[
sensitivity: Sensitivity::Public, sensitivity: Sensitivity::Public,
doc: "off: approval_rule conventions are recorded but never block a post. enforce: an entry a rule covers posts only once its approver approves it (books.approve_entry). Unset: off.", doc: "off: approval_rule conventions are recorded but never block a post. enforce: an entry a rule covers posts only once its approver approves it (books.approve_entry). Unset: off.",
}, },
Key {
name: "backup.git_remote",
toml: ("backup", "git_remote"),
kind: Kind::Repository,
sensitivity: Sensitivity::Public,
doc: "A private repository the books back themselves up to (plan D-30): its address as its host shows it for cloning, over ssh or https (such as https://host/owner/books), or a path on this computer. Unset: none.",
},
Key {
name: "backup.folder",
toml: ("backup", "folder"),
kind: Kind::Folder,
sensitivity: Sensitivity::Public,
doc: "A folder the books back themselves up to as well (D-30): an absolute path, an external disk or a synced folder; the backup is a repository in it named for the book. Unset: none.",
},
]; ];
/// The key named `name`. /// The key named `name`.
@@ -873,6 +893,37 @@ pub fn check(key: &Key, v: &Value) -> Result<Value, String> {
} }
Ok(json!(s)) Ok(json!(s))
} }
Kind::Repository => {
let s = line_text(name, want_str(name, v)?, MAX_TEXT)?;
if s.starts_with('-') {
return Err(format!(
"{name} {s:?}: a repository's address never starts with -"
));
}
// A path on this computer is named from the root of its disk:
// the server's working folder is no place to name one from.
let networked = match s.split_once("://") {
Some((scheme, _)) => scheme != "file",
None => s
.split_once(':')
.is_some_and(|(host, _)| !host.contains('/')),
};
if !networked && !s.starts_with("file://") && !std::path::Path::new(&s).is_absolute() {
return Err(format!(
"{name} {s:?}: a repository on this computer is named from the root of its disk (such as /srv/books)"
));
}
Ok(json!(s))
}
Kind::Folder => {
let s = line_text(name, want_str(name, v)?, MAX_TEXT)?;
if !std::path::Path::new(&s).is_absolute() {
return Err(format!(
"{name} {s:?}: name the folder from the root of its disk (such as /media/backup)"
));
}
Ok(json!(s))
}
} }
} }
@@ -1243,6 +1294,16 @@ impl Settings {
} }
} }
/// `backup.git_remote`, if set.
pub fn backup_git_remote(&self) -> Option<&str> {
self.str("backup.git_remote")
}
/// `backup.folder`, if set.
pub fn backup_folder(&self) -> Option<&str> {
self.str("backup.folder")
}
/// `company.timezone`, if set. /// `company.timezone`, if set.
pub fn timezone(&self) -> Option<&str> { pub fn timezone(&self) -> Option<&str> {
self.str("company.timezone") self.str("company.timezone")
@@ -1423,6 +1484,44 @@ pub fn display(key: &Key, v: &Value, full: bool) -> String {
mod tests { mod tests {
use super::*; use super::*;
#[test]
fn where_backups_go_is_checked() {
let remote = key("backup.git_remote").unwrap();
let folder = key("backup.folder").unwrap();
for ok in [
"git@host:owner/books.git",
"https://host/owner/books",
"/srv/books.git",
] {
assert_eq!(check(remote, &json!(ok)).unwrap(), json!(ok), "{ok}");
}
assert_eq!(
check(remote, &json!(" ssh://h/b ")).unwrap(),
json!("ssh://h/b")
);
for bad in [
"--upload-pack=x",
"-",
"",
"a\nb",
"acme.git",
"backups/acme.git",
"~/x.git",
"./x",
] {
assert!(check(remote, &json!(bad)).is_err(), "{bad}");
}
assert!(check(remote, &json!("file:///srv/b.git")).is_ok());
#[cfg(unix)]
assert_eq!(
check(folder, &json!("/media/backup")).unwrap(),
json!("/media/backup")
);
for bad in ["backup", "./backup", "~/backup", "-x", ""] {
assert!(check(folder, &json!(bad)).is_err(), "{bad}");
}
}
fn src() -> Source { fn src() -> Source {
Source { Source {
decided_by: "Owner".into(), decided_by: "Owner".into(),
+194 -4
View File
@@ -14,9 +14,9 @@
//! 1. A command that needs the server holds a lease while it runs, and //! 1. A command that needs the server holds a lease while it runs, and
//! lets go when it ends. //! lets go when it ends.
//! 2. Nothing printed names the data root. //! 2. Nothing printed names the data root.
//! 3. Adoption is previewed first and goes ahead only on a yes at the //! 3. Adoption and restoring are previewed first and go ahead only on a
//! terminal, or --yes; a vault beside the books must be named (or //! yes at the terminal, or --yes; a vault beside the books must be
//! --no-vault given): it is never silently left behind. //! named (or --no-vault given): it is never silently left behind.
//! 4. A token made is kept in the keyring before anything is said of it, //! 4. A token made is kept in the keyring before anything is said of it,
//! and its text is printed only by `token print`; one that can't be //! and its text is printed only by `token print`; one that can't be
//! kept is revoked again. A token revoked is forgotten by the keyring //! kept is revoked again. A token revoked is forgotten by the keyring
@@ -392,12 +392,16 @@ fn books_usage(prog: &str) {
eprintln!( eprintln!(
" {prog} adopt DATA-DIR --id ID [--toml FILE] [--config-json FILE] [--vault DIR --vault-key FILE | --no-vault] [--skip-unknown] [--yes]" " {prog} adopt DATA-DIR --id ID [--toml FILE] [--config-json FILE] [--vault DIR --vault-key FILE | --no-vault] [--skip-unknown] [--yes]"
); );
eprintln!(" {prog} restore BACKUP [--id ID] [--replace] [--yes]");
eprintln!(" list the books, and whether each is open"); eprintln!(" list the books, and whether each is open");
eprintln!(" create start new, empty books (base currency USD unless given)"); eprintln!(" create start new, empty books (base currency USD unless given)");
eprintln!(" adopt copy existing books in, verified; the original is left as it was"); eprintln!(" adopt copy existing books in, verified; the original is left as it was");
eprintln!(
" restore bring back a backup (a git repository's address, or BOOK.git in the backup folder), proven whole first: as new books, or replacing books (kept in legacy/)"
);
} }
/// `wanderbooks books list|create|adopt`. /// `wanderbooks books list|create|adopt|restore`.
pub fn books( pub fn books(
prog: &str, prog: &str,
args: impl IntoIterator<Item = String>, args: impl IntoIterator<Item = String>,
@@ -413,6 +417,7 @@ pub fn books(
let mut no_vault = false; let mut no_vault = false;
let mut skip_unknown = false; let mut skip_unknown = false;
let mut yes = false; let mut yes = false;
let mut replace = false;
let mut args = args.into_iter(); let mut args = args.into_iter();
while let Some(arg) = args.next() { while let Some(arg) = args.next() {
if common.take(&arg, &mut args)? { if common.take(&arg, &mut args)? {
@@ -428,6 +433,7 @@ pub fn books(
"--vault-key" => vault_key = Some(PathBuf::from(value("a file")?)), "--vault-key" => vault_key = Some(PathBuf::from(value("a file")?)),
"--no-vault" => no_vault = true, "--no-vault" => no_vault = true,
"--skip-unknown" => skip_unknown = true, "--skip-unknown" => skip_unknown = true,
"--replace" => replace = true,
"--yes" => yes = true, "--yes" => yes = true,
"--help" | "-h" => { "--help" | "-h" => {
books_usage(prog); books_usage(prog);
@@ -471,6 +477,7 @@ pub fn books(
yes, yes,
); );
} }
["restore", source] => return restore(&common, source, id, replace, yes),
_ => { _ => {
books_usage(prog); books_usage(prog);
return Err(exit(2)); return Err(exit(2));
@@ -514,6 +521,172 @@ pub fn books(
} }
} }
/// The id a backup is restored as when none is given: its repository's
/// name (`acme.git` → `acme`).
fn restore_id(source: &str) -> Option<String> {
let last = source
.trim_end_matches('/')
.rsplit(['/', ':'])
.next()?
.trim_end_matches(".git");
ledger_store::root::BookId::parse(last)
.ok()
.map(|id| id.to_string())
}
/// Ask at the terminal (or take --yes): whether to go ahead.
fn confirmed(question: &str, yes: bool) -> Result<bool, String> {
if yes {
return Ok(true);
}
if !at_a_terminal() {
return Err("confirm at a terminal, or run again with --yes".into());
}
print!("{question} [y/N] ");
use std::io::Write;
let _ = std::io::stdout().flush();
let mut answer = String::new();
std::io::stdin()
.read_line(&mut answer)
.map_err(|e| e.to_string())?;
Ok(matches!(answer.trim(), "y" | "Y" | "yes" | "Yes"))
}
/// `wanderbooks books restore` (INVARIANT 3).
fn restore(
common: &Common,
source: &str,
id: Option<String>,
replace: bool,
yes: bool,
) -> Result<(), Box<dyn std::error::Error>> {
let fail = |why: String| -> Result<(), Box<dyn std::error::Error>> {
eprintln!("error: {why}");
Err(exit(1))
};
// A folder is named to the server by its full path (the server's
// working folder isn't this one); anything else is an address.
let source = match Path::new(source) {
p if p.exists() => std::path::absolute(p)
.unwrap_or_else(|_| p.to_path_buf())
.display()
.to_string(),
_ => source.to_string(),
};
// One preview (under the id given, or any: it only reports), then the
// id: given, or else the id of the books the backup holds, or else the
// repository's name.
let probe = id
.clone()
.or_else(|| restore_id(&source))
.unwrap_or_else(|| "restored".to_string());
let preview = match with_server(
common,
Request::Restore {
id: probe,
source: source.clone(),
replace,
confirmed: false,
expect_head: None,
},
) {
Ok(Response::RestorePreview(p)) => p,
Ok(Response::Refused(why)) | Err(why) => return fail(format!("{source}: {why}")),
Ok(other) => return fail(format!("the books server answered oddly: {other:?}")),
};
let Some(id) = id
.or_else(|| {
preview
.origin
.clone()
.filter(|o| ledger_store::root::BookId::parse(o).is_ok())
})
.or_else(|| restore_id(&source))
else {
eprintln!("error: name the restored books with --id ID");
return Err(exit(2));
};
let exists = match with_server(common, Request::List) {
Ok(Response::Books(books)) => books.iter().any(|b| b.id == id),
Ok(Response::Refused(why)) | Err(why) => return fail(why),
Ok(other) => return fail(format!("the books server answered oddly: {other:?}")),
};
if exists && !replace {
return fail(format!(
"there are already books called \"{id}\": restore them under another id (--id), or replace them (--replace; they are kept in legacy/)"
));
}
if !exists && replace {
return fail(format!(
"there are no books called \"{id}\" to replace: nothing was restored"
));
}
let request = |expect_head: Option<(u64, String)>| Request::Restore {
id: id.clone(),
source: source.clone(),
replace,
confirmed: expect_head.is_some(),
expect_head,
};
println!(
"The backup at {source} is whole: through event {} ({}), {} stored files re-hashed, {} TIN vault records.",
preview.head.0,
short_hash(&preview.head.1),
preview.blobs,
preview.vault_files
);
if let Some(origin) = &preview.origin {
println!("It is a backup of the books \"{origin}\".");
}
if replace {
println!(
"It replaces the books \"{id}\": they are closed and kept whole in the legacy folder, not deleted."
);
} else {
println!("It comes back as new books \"{id}\", beside any others.");
}
match preview.origin.as_deref() {
Some(o) if o == id => {}
Some(o) => println!(
"Under a name other than its books' (\"{o}\"), they won't back up to the private repository those books back up to (that would stop theirs): if these are those books, restore them as \"{o}\" (--id {o}); else name a repository of their own."
),
None => println!(
"It doesn't say whose books it holds: they won't back up to the private repository their settings name until one of their own is named."
),
}
match confirmed("Restore it?", yes) {
Ok(true) => {}
Ok(false) => {
println!("Not restored.");
return Ok(());
}
Err(why) => return fail(format!("not restored: {why}")),
}
match with_server(common, request(Some(preview.head.clone()))) {
Ok(Response::Restored(r)) => {
println!(
"Restored as \"{}\": through event {} ({}), {} stored files checked, {} TIN vault records.",
r.id,
r.head.0,
short_hash(&r.head.1),
r.blobs_verified,
r.vault_files
);
if let Some(kept) = &r.replaced {
println!("The books it replaced are kept in the legacy folder as {kept}.");
}
if let Some(why) = r.problem {
return fail(format!(
"the restored books didn't open: {why}. Closing the books and opening them again tries again."
));
}
Ok(())
}
Ok(Response::Refused(why)) | Err(why) => fail(format!("{source}: {why}")),
Ok(other) => fail(format!("the books server answered oddly: {other:?}")),
}
}
/// The files an adoption brings in with the books. /// The files an adoption brings in with the books.
struct AdoptFiles { struct AdoptFiles {
toml: Option<PathBuf>, toml: Option<PathBuf>,
@@ -1743,6 +1916,23 @@ mod tests {
} }
} }
#[test]
fn a_backup_is_restored_under_its_repository_name() {
for (source, want) in [
("/media/disk/backups/acme.git", Some("acme")),
("/media/disk/backups/acme.git/", Some("acme")),
("git@host:owner/acme-books.git", Some("acme-books")),
("ssh://host/owner/acme", Some("acme")),
("https://host/owner/acme.git", Some("acme")),
("acme.git", Some("acme")),
("/media/disk/Backups/Acme.git", None),
("/", None),
("", None),
] {
assert_eq!(restore_id(source).as_deref(), want, "{source}");
}
}
#[test] #[test]
fn an_adoption_says_what_it_did_and_only_that() { fn an_adoption_says_what_it_did_and_only_that() {
let plain = adopted_lines(&adopted()); let plain = adopted_lines(&adopted());
+3 -1
View File
@@ -162,7 +162,9 @@ fn answer(ask: HelperAsk, root: Option<&Path>, knobs: &Knobs) -> Answer {
HelperAsk::OpenWith { .. } => { HelperAsk::OpenWith { .. } => {
Answer::Refused("opening documents comes with the viewer's menu".into()) Answer::Refused("opening documents comes with the viewer's menu".into())
} }
HelperAsk::StageBackup => Answer::Refused("restoring isn't built yet".into()), HelperAsk::StageBackup => Answer::Refused(
"restore from a terminal for now: `wanderbooks books restore BACKUP`".into(),
),
HelperAsk::StartUpdater => Answer::Refused("updates come with the installers".into()), HelperAsk::StartUpdater => Answer::Refused("updates come with the installers".into()),
} }
} }
+16 -2
View File
@@ -657,6 +657,15 @@ impl Host {
if self.explicit_root { if self.explicit_root {
command.arg("--root").arg(self.root.path()); command.arg("--root").arg(self.root.path());
} }
let d = Knobs::default();
for (flag, now, default) in [
("--backup-quiet", self.knobs.backup_quiet, d.backup_quiet),
("--drill-first", self.knobs.drill_first, d.drill_first),
] {
if now != default {
command.arg(flag).arg(now.as_millis().to_string());
}
}
let mut child = command let mut child = command
.stdin(std::process::Stdio::piped()) .stdin(std::process::Stdio::piped())
.stdout(std::process::Stdio::piped()) .stdout(std::process::Stdio::piped())
@@ -701,11 +710,16 @@ impl Host {
None, None,
), ),
Intent::BackUpNow { .. } => ( Intent::BackUpNow { .. } => (
HostMsg::Refused("backing up isn't built yet (it comes later in P2)".into()), HostMsg::Refused(
"the books back themselves up; to back up now, ask your agent (books.back_up_now)"
.into(),
),
None, None,
), ),
Intent::Restore { .. } => ( Intent::Restore { .. } => (
HostMsg::Refused("restoring isn't built yet (it comes later in P2)".into()), HostMsg::Refused(
"restore from a terminal for now: `wanderbooks books restore BACKUP`".into(),
),
None, None,
), ),
Intent::ConsentFormatUpgrade { .. } => ( Intent::ConsentFormatUpgrade { .. } => (
+17 -1
View File
@@ -30,6 +30,12 @@ pub struct Knobs {
/// Keep agents' tokens in files in the root's `run/`, not the OS /// Keep agents' tokens in files in the root's `run/`, not the OS
/// keyring (tests: tokens.rs INVARIANT 3). `file_keyring=1`. /// keyring (tests: tokens.rs INVARIANT 3). `file_keyring=1`.
pub file_keyring: bool, pub file_keyring: bool,
/// How long the books stay unwritten before the server backs them up
/// (plan 3.7: 30 s), passed to the server as `--backup-quiet`.
pub backup_quiet: Duration,
/// How long after books open the server first looks for a drill due
/// (60 s), passed to the server as `--drill-first`.
pub drill_first: Duration,
} }
impl Default for Knobs { impl Default for Knobs {
@@ -43,6 +49,8 @@ impl Default for Knobs {
queue_wait: Duration::from_secs(60), queue_wait: Duration::from_secs(60),
open_wait: Duration::from_secs(120), open_wait: Duration::from_secs(120),
file_keyring: false, file_keyring: false,
backup_quiet: Duration::from_secs(30),
drill_first: Duration::from_secs(60),
} }
} }
} }
@@ -78,6 +86,8 @@ impl Knobs {
"init_wait" => k.init_wait = d, "init_wait" => k.init_wait = d,
"queue_wait" => k.queue_wait = d, "queue_wait" => k.queue_wait = d,
"open_wait" => k.open_wait = d, "open_wait" => k.open_wait = d,
"backup_quiet" => k.backup_quiet = d,
"drill_first" => k.drill_first = d,
other => return Err(format!("{FLAG}: there is no knob {other:?}")), other => return Err(format!("{FLAG}: there is no knob {other:?}")),
} }
} }
@@ -97,6 +107,8 @@ impl Knobs {
("init_wait", self.init_wait, d.init_wait), ("init_wait", self.init_wait, d.init_wait),
("queue_wait", self.queue_wait, d.queue_wait), ("queue_wait", self.queue_wait, d.queue_wait),
("open_wait", self.open_wait, d.open_wait), ("open_wait", self.open_wait, d.open_wait),
("backup_quiet", self.backup_quiet, d.backup_quiet),
("drill_first", self.drill_first, d.drill_first),
] { ] {
if now != default { if now != default {
parts.push(format!("{name}={}", now.as_millis())); parts.push(format!("{name}={}", now.as_millis()));
@@ -131,6 +143,8 @@ mod tests {
assert_eq!(k.exe_poll, Duration::from_secs(5)); assert_eq!(k.exe_poll, Duration::from_secs(5));
assert_eq!(k.server_delay, Duration::ZERO); assert_eq!(k.server_delay, Duration::ZERO);
assert_eq!(k.open_wait, Duration::from_secs(120)); assert_eq!(k.open_wait, Duration::from_secs(120));
assert_eq!(k.backup_quiet, Duration::from_secs(30));
assert_eq!(k.drill_first, Duration::from_secs(60));
assert_eq!(k.to_arg(), None); assert_eq!(k.to_arg(), None);
assert!(k.args().is_empty()); assert!(k.args().is_empty());
assert_eq!(Knobs::parse("").unwrap(), k); assert_eq!(Knobs::parse("").unwrap(), k);
@@ -139,7 +153,7 @@ mod tests {
#[test] #[test]
fn knobs_pass_on_exactly() { fn knobs_pass_on_exactly() {
let k = Knobs::parse( let k = Knobs::parse(
"close_after=500,backoff=20,server_delay=3000,init_wait=100,queue_wait=900,exe_poll=50,open_wait=70", "close_after=500,backoff=20,server_delay=3000,init_wait=100,queue_wait=900,exe_poll=50,open_wait=70,backup_quiet=40,drill_first=50",
) )
.unwrap(); .unwrap();
assert_eq!(k.close_after, Duration::from_millis(500)); assert_eq!(k.close_after, Duration::from_millis(500));
@@ -149,6 +163,8 @@ mod tests {
assert_eq!(k.queue_wait, Duration::from_millis(900)); assert_eq!(k.queue_wait, Duration::from_millis(900));
assert_eq!(k.exe_poll, Duration::from_millis(50)); assert_eq!(k.exe_poll, Duration::from_millis(50));
assert_eq!(k.open_wait, Duration::from_millis(70)); assert_eq!(k.open_wait, Duration::from_millis(70));
assert_eq!(k.backup_quiet, Duration::from_millis(40));
assert_eq!(k.drill_first, Duration::from_millis(50));
let args = k.args(); let args = k.args();
assert_eq!(args[0], FLAG); assert_eq!(args[0], FLAG);
assert_eq!(Knobs::parse(&args[1]).unwrap(), k); assert_eq!(Knobs::parse(&args[1]).unwrap(), k);
+25 -33
View File
@@ -5,7 +5,8 @@
//! the server, through one socket that accepts nothing else. //! the server, through one socket that accepts nothing else.
//! SOURCE: plan/served.md 3.9 ("Who holds them", "Three local channels": //! SOURCE: plan/served.md 3.9 ("Who holds them", "Three local channels":
//! leases record their kind and PID; exactly five owner intents; //! leases record their kind and PID; exactly five owner intents;
//! consent and a replacing restore carry the owner passphrase; //! consent carries the owner passphrase (a replacing restore no
//! longer does: D-31, the books it replaces are kept in legacy/);
//! "Closing": the host says why on each lease; protocol versions N //! "Closing": the host says why on each lease; protocol versions N
//! and N-1). //! and N-1).
//! INVARIANTS: //! INVARIANTS:
@@ -13,11 +14,10 @@
//! connection (the lease lasts until it closes), anything else is //! connection (the lease lasts until it closes), anything else is
//! answered once and the connection closed. //! answered once and the connection closed.
//! 2. Exactly five intents exist: close, back_up_now, new_books, //! 2. Exactly five intents exist: close, back_up_now, new_books,
//! restore (a staged backup's id, never a path) and //! restore (a backup's address or folder, as `books restore` takes
//! consent_format_upgrade; any other request is refused. //! it) and consent_format_upgrade; any other request is refused.
//! 3. consent_format_upgrade, and a restore that replaces a book, carry //! 3. consent_format_upgrade carries the owner passphrase; without one
//! the owner passphrase; without one they are refused before //! it is refused before anything is done.
//! anything is done.
//! 4. Before the host ends a lease it sends the reason; a lease that //! 4. Before the host ends a lease it sends the reason; a lease that
//! ends with no reason means the host is gone. //! ends with no reason means the host is gone.
//! 5. Each side accepts its own protocol version and the one before. //! 5. Each side accepts its own protocol version and the one before.
@@ -120,16 +120,17 @@ pub enum Intent {
/// Their base currency. /// Their base currency.
currency: String, currency: String,
}, },
/// Restore a staged backup, as its own book or replacing one. /// Restore a backup, as its own book or replacing one (kept in
/// legacy/).
Restore { Restore {
/// The staged backup's id (never a path). /// The backup: a repository's address, or its folder (`<book>.git`
staged: String, /// in the backup folder).
/// The book it replaces, if it replaces one. source: String,
/// The books' id.
id: String,
/// Replace the books of that id.
#[serde(default)] #[serde(default)]
replace: Option<String>, replace: bool,
/// The owner passphrase: needed to replace (INVARIANT 3).
#[serde(default)]
passphrase: Option<String>,
}, },
/// Let the server upgrade a book's journal format. /// Let the server upgrade a book's journal format.
ConsentFormatUpgrade { ConsentFormatUpgrade {
@@ -151,13 +152,6 @@ impl Intent {
"consenting to a format upgrade needs the owner passphrase, typed in the app" "consenting to a format upgrade needs the owner passphrase, typed in the app"
.into(), .into(),
), ),
Intent::Restore {
replace: Some(_),
passphrase,
..
} if missing(passphrase) => Err(
"a restore that replaces books needs the owner passphrase, typed in the app".into(),
),
_ => Ok(()), _ => Ok(()),
} }
} }
@@ -288,7 +282,7 @@ mod tests {
json!("close"), json!("close"),
json!({"back_up_now": {"book": "acme"}}), json!({"back_up_now": {"book": "acme"}}),
json!({"new_books": {"id": "acme", "currency": "USD"}}), json!({"new_books": {"id": "acme", "currency": "USD"}}),
json!({"restore": {"staged": "s1"}}), json!({"restore": {"source": "/backups/acme.git", "id": "acme"}}),
json!({"consent_format_upgrade": {"book": "acme", "passphrase": "x"}}), json!({"consent_format_upgrade": {"book": "acme", "passphrase": "x"}}),
] { ] {
let h = parse(json!({"proto": 1, "intent": intent})).unwrap(); let h = parse(json!({"proto": 1, "intent": intent})).unwrap();
@@ -297,7 +291,8 @@ mod tests {
for bad in [ for bad in [
json!({"proto": 1, "intent": "delete"}), json!({"proto": 1, "intent": "delete"}),
json!({"proto": 1, "intent": {"restore": {"path": "/x"}}}), json!({"proto": 1, "intent": {"restore": {"path": "/x"}}}),
json!({"proto": 1, "intent": {"restore": {"staged": "s1", "path": "/x"}}}), json!({"proto": 1, "intent": {"restore": {"source": "/x", "id": "a", "path": "/x"}}}),
json!({"proto": 1, "intent": {"restore": {"staged": "s1"}}}),
json!({"proto": 1, "intent": {"run": {"command": "sh"}}}), json!({"proto": 1, "intent": {"run": {"command": "sh"}}}),
json!({"proto": 1, "exec": "sh"}), json!({"proto": 1, "exec": "sh"}),
json!({"proto": 1}), json!({"proto": 1}),
@@ -322,7 +317,7 @@ mod tests {
} }
#[test] #[test]
fn consent_and_a_replacing_restore_need_the_passphrase() { fn only_consent_needs_the_passphrase() {
let consent = |p: Option<&str>| Intent::ConsentFormatUpgrade { let consent = |p: Option<&str>| Intent::ConsentFormatUpgrade {
book: "acme".into(), book: "acme".into(),
passphrase: p.map(str::to_string), passphrase: p.map(str::to_string),
@@ -330,16 +325,13 @@ mod tests {
assert!(consent(None).check_passphrase().is_err()); assert!(consent(None).check_passphrase().is_err());
assert!(consent(Some("")).check_passphrase().is_err()); assert!(consent(Some("")).check_passphrase().is_err());
assert!(consent(Some("pw")).check_passphrase().is_ok()); assert!(consent(Some("pw")).check_passphrase().is_ok());
let restore = |replace: Option<&str>, p: Option<&str>| Intent::Restore {
staged: "s1".into(),
replace: replace.map(str::to_string),
passphrase: p.map(str::to_string),
};
assert!(restore(None, None).check_passphrase().is_ok());
assert!(restore(Some("acme"), None).check_passphrase().is_err());
assert!(restore(Some("acme"), Some("")).check_passphrase().is_err());
assert!(restore(Some("acme"), Some("pw")).check_passphrase().is_ok());
for other in [ for other in [
// D-31: replacing keeps the old books in legacy/; no passphrase.
Intent::Restore {
source: "/backups/acme.git".into(),
id: "acme".into(),
replace: true,
},
Intent::Close, Intent::Close,
Intent::BackUpNow { book: "a".into() }, Intent::BackUpNow { book: "a".into() },
Intent::NewBooks { Intent::NewBooks {
+81 -7
View File
@@ -9,8 +9,11 @@
//! 1. The host's environment is a fixed list: PATH (fixed), HOME (from //! 1. The host's environment is a fixed list: PATH (fixed), HOME (from
//! the account's record, never the opener's), and XDG_RUNTIME_DIR //! the account's record, never the opener's), and XDG_RUNTIME_DIR
//! and DBUS_SESSION_BUS_ADDRESS only when they point at this //! and DBUS_SESSION_BUS_ADDRESS only when they point at this
//! account's own private runtime folder. Nothing else passes (a //! account's own private runtime folder, and SSH_AUTH_SOCK only when
//! scope's own identifiers, such as INVOCATION_ID, are systemd's). //! it names a socket this account owns (the owner's ssh agent, so the
//! server's backups reach a private git repository, D-30). Nothing
//! else passes (a scope's own identifiers, such as INVOCATION_ID, are
//! systemd's).
//! 2. Its standard input and output are null, its errors go to //! 2. Its standard input and output are null, its errors go to
//! run/host.log (0600); it starts a new session with umask 077. Every //! run/host.log (0600); it starts a new session with umask 077. Every
//! other descriptor is closed (std opens them close-on-exec, and the //! other descriptor is closed (std opens them close-on-exec, and the
@@ -112,8 +115,20 @@ fn session_bus(runtime: &Path) -> Option<String> {
} }
} }
/// The host's environment (INVARIANT 1). /// The ssh agent's socket named in `named`, if it is a socket this
fn host_env(home: &Path) -> Vec<(String, std::ffi::OsString)> { /// account owns at an absolute path (INVARIANT 1).
fn own_agent(named: Option<&std::ffi::OsStr>) -> Option<std::ffi::OsString> {
use std::os::unix::fs::FileTypeExt;
let path = Path::new(named?);
(path.is_absolute()
&& std::fs::symlink_metadata(path)
.is_ok_and(|m| m.file_type().is_socket() && m.uid() == own_uid()))
.then(|| path.as_os_str().to_os_string())
}
/// The host's environment (INVARIANT 1), given the opener's
/// `SSH_AUTH_SOCK`.
fn host_env(home: &Path, ssh_agent: Option<&std::ffi::OsStr>) -> Vec<(String, std::ffi::OsString)> {
let mut env = vec![ let mut env = vec![
("PATH".to_string(), PATH.into()), ("PATH".to_string(), PATH.into()),
("HOME".to_string(), home.as_os_str().to_os_string()), ("HOME".to_string(), home.as_os_str().to_os_string()),
@@ -124,6 +139,9 @@ fn host_env(home: &Path) -> Vec<(String, std::ffi::OsString)> {
} }
env.push(("XDG_RUNTIME_DIR".into(), runtime.into_os_string())); env.push(("XDG_RUNTIME_DIR".into(), runtime.into_os_string()));
} }
if let Some(agent) = own_agent(ssh_agent) {
env.push(("SSH_AUTH_SOCK".into(), agent));
}
env env
} }
@@ -199,7 +217,10 @@ pub fn start_host(root: Option<&Path>, knobs: &Knobs) -> Result<(), String> {
}; };
c.args(&args) c.args(&args)
.env_clear() .env_clear()
.envs(host_env(&home)) .envs(host_env(
&home,
std::env::var_os("SSH_AUTH_SOCK").as_deref(),
))
.current_dir("/") .current_dir("/")
.stdin(Stdio::null()) .stdin(Stdio::null())
.stdout(Stdio::null()) .stdout(Stdio::null())
@@ -357,12 +378,17 @@ mod tests {
#[test] #[test]
fn the_host_environment_carries_nothing_else() { fn the_host_environment_carries_nothing_else() {
let env = host_env(Path::new("/home/someone")); let env = host_env(Path::new("/home/someone"), None);
let names: Vec<&str> = env.iter().map(|(n, _)| n.as_str()).collect(); let names: Vec<&str> = env.iter().map(|(n, _)| n.as_str()).collect();
assert_eq!(&names[..2], ["PATH", "HOME"]); assert_eq!(&names[..2], ["PATH", "HOME"]);
for n in &names[2..] { for n in &names[2..] {
assert!( assert!(
["XDG_RUNTIME_DIR", "DBUS_SESSION_BUS_ADDRESS"].contains(n), [
"XDG_RUNTIME_DIR",
"DBUS_SESSION_BUS_ADDRESS",
"SSH_AUTH_SOCK"
]
.contains(n),
"{n}" "{n}"
); );
} }
@@ -370,6 +396,54 @@ mod tests {
assert_eq!(env[1].1, "/home/someone"); assert_eq!(env[1].1, "/home/someone");
} }
#[test]
fn only_an_own_ssh_agent_socket_passes() {
let dir = tempfile::tempdir().unwrap();
let sock = dir.path().join("agent.sock");
let _listener = std::os::unix::net::UnixListener::bind(&sock).unwrap();
assert_eq!(
own_agent(Some(sock.as_os_str())),
Some(sock.clone().into_os_string())
);
// It reaches the host's environment, once.
let env = host_env(Path::new("/home/someone"), Some(sock.as_os_str()));
let agents: Vec<_> = env.iter().filter(|(n, _)| n == "SSH_AUTH_SOCK").collect();
assert_eq!(agents.len(), 1);
assert_eq!(agents[0].1, sock.as_os_str());
// A link to it is not it (whoever made the link chose where it
// points).
let link = dir.path().join("link.sock");
std::os::unix::fs::symlink(&sock, &link).unwrap();
assert_eq!(own_agent(Some(link.as_os_str())), None);
let env = host_env(Path::new("/home/someone"), Some(link.as_os_str()));
assert!(env.iter().all(|(n, _)| n != "SSH_AUTH_SOCK"));
// Named relative to here: not taken, even where it leads to one.
let here = std::env::current_dir().unwrap();
let up = here.components().count() - 1;
let relative: std::path::PathBuf = std::iter::repeat_n("..", up)
.collect::<std::path::PathBuf>()
.join(sock.strip_prefix("/").unwrap());
assert!(relative.exists() && relative.is_relative(), "{relative:?}");
assert_eq!(own_agent(Some(relative.as_os_str())), None);
// Another account's socket (the system's bus, owned by root, where
// there is one): not ours.
let system = Path::new("/run/dbus/system_bus_socket");
if system.exists() && own_uid() != 0 {
assert_eq!(own_agent(Some(system.as_os_str())), None);
}
let file = dir.path().join("plain");
std::fs::write(&file, "x").unwrap();
for bad in [
file.as_os_str(),
"agent.sock".as_ref(),
"/nonexistent/agent.sock".as_ref(),
"".as_ref(),
] {
assert_eq!(own_agent(Some(bad)), None, "{bad:?}");
}
assert_eq!(own_agent(None), None);
}
#[test] #[test]
fn this_executable_is_a_file_on_disk() { fn this_executable_is_a_file_on_disk() {
let exe = this_executable().unwrap(); let exe = this_executable().unwrap();
+9
View File
@@ -0,0 +1,9 @@
# Backups
- The books back themselves up: a little after each change, when they open, before they close. Each backup goes to the targets the settings name: backup.git_remote, a private repository the human owns, and backup.folder, a folder on another disk. Both is best. The backups aren't encrypted: the human's repository host and disk keep them private.
- No targets set: books.start says so. Ask the human for the repository's address and the folder, then set them with books.set_setting (who decided, and when); choosing where backups go needs books:admin, as backing up does.
- books.backup says how each target stands: when it last took the books, through which journal seq, why the last try failed, and whether its weekly restore check passed. A target that is stale, failing, or failed its check: tell the human what it says, in plain words (a disk not connected, a repository that refused).
- books.back_up_now backs up at once, for instance before the human changes computers. A backup never changes the books.
- A backup folder that took these books' backups but holds none now is a disk not connected: ask the human to connect it. Only if they say the disk is new, back up now with new_disk: true to begin the books' backup there again.
- Books restored as a copy of other books (under another name) don't back up to those books' repository (that would stop theirs): books.backup says so; ask the human for one of their own.
- Restoring is the human's to do, outside these tools: a backup comes back as new books, or replacing these (the old books are kept, never deleted), and each is proven whole before it is restored.
+101 -6
View File
@@ -1,6 +1,6 @@
//! The admin socket's messages (`run/admin.sock`, plan 3.1, 3.3, 3.9): how //! The admin socket's messages (`run/admin.sock`, plan 3.1, 3.3, 3.9): how
//! the command line asks the books server to list, create and adopt books, //! the command line asks the books server to list, create, adopt and
//! and to make, list and revoke agents' tokens. //! restore books, and to make, list and revoke agents' tokens.
//! //!
//! ```text //! ```text
//! GOAL: the command line makes and adopts books without ever touching //! GOAL: the command line makes and adopts books without ever touching
@@ -16,9 +16,10 @@
//! that says to restart. //! that says to restart.
//! 3. No answer names the data root or a path in it, nor echoes a path //! 3. No answer names the data root or a path in it, nor echoes a path
//! the caller gave (which could be the root under another name). //! the caller gave (which could be the root under another name).
//! 4. Adopting is two steps: without `confirmed` the answer is a //! 4. Adopting and restoring are two steps: without `confirmed` the
//! preview, and nothing is written; `confirmed: true` adopts, and //! answer is a preview, and nothing is written; `confirmed: true`
//! only books still at the head the preview read (`expect_head`). //! adopts (restores), and only books (a backup) still at the head the
//! preview read (`expect_head`).
//! 5. Nothing in a settings file is dropped silently: entries the books //! 5. Nothing in a settings file is dropped silently: entries the books
//! don't know are refused unless `skip_unknown`, and its notes and //! don't know are refused unless `skip_unknown`, and its notes and
//! comments are in the preview. //! comments are in the preview.
@@ -97,6 +98,24 @@ pub enum Request {
#[serde(default)] #[serde(default)]
skip_unknown: bool, skip_unknown: bool,
}, },
/// Restore a backup (plan 3.7): as new books, or replacing books
/// (which are moved into `legacy/`, kept).
Restore {
/// The books' id.
id: String,
/// The backup: a git repository's address, or its folder
/// (`<book>.git` in the backup folder).
source: String,
/// Replace the books called `id`.
#[serde(default)]
replace: bool,
/// Restore (true), or only preview it.
#[serde(default)]
confirmed: bool,
/// The head the preview read: restored only if still so.
#[serde(default)]
expect_head: Option<(u64, String)>,
},
/// Make a token for an agent (plan 3.3). /// Make a token for an agent (plan 3.3).
TokenCreate { TokenCreate {
/// The books it opens. /// The books it opens.
@@ -237,6 +256,44 @@ pub struct Adopted {
pub problem: Option<String>, pub problem: Option<String>,
} }
/// What a restore would bring in.
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct RestorePreview {
/// The backup's head, proven.
pub head: (u64, String),
/// Stored files in it, each re-hashed.
pub blobs: u64,
/// TIN vault records in it.
pub vault_files: u64,
/// Whether books of that id exist now.
pub exists: bool,
/// The id of the books it holds, as they were backed up.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub origin: Option<String>,
}
/// What a restore did.
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct Restored {
/// The books' id.
pub id: String,
/// Their head: the backup's.
pub head: (u64, String),
/// Stored files the books name, re-hashed through the store.
pub blobs_verified: u64,
/// TIN vault records restored.
pub vault_files: u64,
/// The `legacy/` folder the replaced books were moved to.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub replaced: Option<String>,
/// The id of the books it held, as they were backed up.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub origin: Option<String>,
/// Why the restored books didn't open, if they didn't (they are kept).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub problem: Option<String>,
}
/// The server's answer. /// The server's answer.
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case", deny_unknown_fields)] #[serde(rename_all = "snake_case", deny_unknown_fields)]
@@ -255,6 +312,10 @@ pub enum Response {
Preview(AdoptPreview), Preview(AdoptPreview),
/// Books adopted and open. /// Books adopted and open.
Adopted(Adopted), Adopted(Adopted),
/// What a restore would bring in.
RestorePreview(RestorePreview),
/// A backup restored, and open.
Restored(Restored),
/// A token made: its text, shown this once (INVARIANT 6). /// A token made: its text, shown this once (INVARIANT 6).
TokenCreated { TokenCreated {
/// The token. /// The token.
@@ -307,7 +368,7 @@ pub fn call_within(
// A list changes nothing: its only question is whether it answered. // A list changes nothing: its only question is whether it answered.
let look = match request { let look = match request {
Request::List | Request::TokenList => None, Request::List | Request::TokenList => None,
Request::Create { .. } | Request::Adopt { .. } => { Request::Create { .. } | Request::Adopt { .. } | Request::Restore { .. } => {
Some("`wanderbooks books list` shows the books there are") Some("`wanderbooks books list` shows the books there are")
} }
Request::TokenCreate { .. } | Request::TokenRevoke { .. } => { Request::TokenCreate { .. } | Request::TokenRevoke { .. } => {
@@ -417,6 +478,24 @@ mod tests {
} }
} }
#[test]
fn a_restore_request_is_a_preview_unless_confirmed() {
let r: Envelope = serde_json::from_value(
json!({"proto": 1, "op": "restore", "id": "acme", "source": "/backups/acme.git"}),
)
.unwrap();
assert_eq!(
r.request,
Request::Restore {
id: "acme".into(),
source: "/backups/acme.git".into(),
replace: false,
confirmed: false,
expect_head: None,
}
);
}
#[test] #[test]
fn unknown_operations_and_fields_are_refused() { fn unknown_operations_and_fields_are_refused() {
for bad in [ for bad in [
@@ -455,6 +534,22 @@ mod tests {
text: TokenText("wb_00".into()), text: TokenText("wb_00".into()),
}, },
Response::Tokens(vec![]), Response::Tokens(vec![]),
Response::RestorePreview(RestorePreview {
head: (3, "ab".into()),
blobs: 2,
vault_files: 1,
exists: true,
origin: Some("acme".into()),
}),
Response::Restored(Restored {
id: "acme".into(),
head: (3, "ab".into()),
blobs_verified: 2,
vault_files: 1,
replaced: Some("acme-20261019-120000".into()),
origin: Some("acme".into()),
problem: None,
}),
] { ] {
let text = serde_json::to_string(&r).unwrap(); let text = serde_json::to_string(&r).unwrap();
assert_eq!(serde_json::from_str::<Response>(&text).unwrap(), r); assert_eq!(serde_json::from_str::<Response>(&text).unwrap(), r);
+994
View File
@@ -0,0 +1,994 @@
//! The server's backups of a book (plan 3.7, D-30, D-31): made a while
//! after the books are written (when the writing stops), when they open,
//! before they close, and when asked; each pushed to the targets the
//! book's settings name; each recorded in the book's `ops/backup.jsonl`;
//! each target's backup drilled weekly.
//!
//! ```text
//! GOAL: every write reaches the owner's backups soon after it is made,
//! and the owner (and agent) can see that it did, or why not.
//! SOURCE: plan 3.7 (triggers, "ops/backup.jsonl", books.backup, Needs
//! attention for a target more than 24 h behind, drills), D-30 (a
//! git repository and a folder), D-31 (plain git).
//! INVARIANTS:
//! 1. A backup is committed under the store's lock (ledger_store::backup
//! INVARIANT 1; its stored files staged before, outside it) and
//! pushed outside it: the books are never held while the network is
//! slow. One run or drill at a time.
//! 2. Every commit, push and drill is a record of `ops/backup.jsonl` (a
//! chain log, not the journal: a backup never moves the books' head),
//! each target's outcome a record of its own, keyed by the target
//! itself (a new folder or address starts with no standing).
//! 3. A target is stale when it lacks the books' head and the first try
//! to back up what it lacks (the first commit or failure since its
//! last good push, or since its first try if it never had one, or
//! since the failed commits that kept it from any) is more than
//! STALE_AFTER old; its last error is its last push's, or a
//! later failed commit's.
//! 4. Nothing here names a path of the data root to an agent.
//! 5. Every target that holds a backup is drilled (ledger_store::backup::
//! drill: cloned, proven whole, its journal the live one's beginning)
//! once DRILL_EVERY has passed since its last good drill, or
//! RETRY_AFTER since a failed one (at the server's next look for
//! drills); a failed drill is said in the standing until one passes.
//! 6. A target whose push failed lately (RETRY_AFTER) is left out of a
//! backup after a write, and only that one; a folder target's
//! repository is made only while the target never took a backup of
//! these books (one gone since is a disk not connected), restored
//! books counting the backups of the books they were.
//! 7. Books restored beside the books they were (under another id) never
//! push to the repository those books back up to (it would stop
//! theirs): it is paused for them, and said, until another is named.
//! 8. A damaged backup record never keeps the books shut: it is set
//! aside and a new one begun.
//! ```
use crate::server::SharedStore;
use ledger_store::backup::{self, BackupError, Git, Target};
use ledger_store::chain::ChainLog;
use serde::Serialize;
use serde_json::{Map, Value, json};
use std::path::{Path, PathBuf};
use std::sync::{Arc, Mutex};
/// The book's backup record, in its `ops/`.
pub const LOG_FILE: &str = "backup.jsonl";
/// The record's genesis text.
const GENESIS: &str = "wanderbooks-backup-v1";
/// How long a target may lack the books' head before it is stale.
pub const STALE_AFTER: chrono::Duration = chrono::Duration::hours(24);
/// How often each target's backup is drilled (plan 3.7: weekly).
pub const DRILL_EVERY: chrono::Duration = chrono::Duration::days(7);
/// How long after a failed run (or drill) the next is tried.
pub const RETRY_AFTER: std::time::Duration = std::time::Duration::from_secs(15 * 60);
/// The most of an error kept.
const ERROR_MAX: usize = 500;
/// A book's backups.
pub struct Backups {
book: String,
dir: PathBuf,
store: Arc<SharedStore>,
log: Mutex<ChainLog>,
/// One run at a time.
running: tokio::sync::Mutex<()>,
}
/// One target's standing (INVARIANT 3).
#[derive(Debug, Clone, Serialize, PartialEq, Eq)]
pub struct TargetStatus {
/// "repository" or "folder".
pub target: String,
/// Where (the repository's address, or "the backup folder").
pub at: String,
/// When it last took a push, if ever (UTC).
pub last_ok_at: Option<String>,
/// The journal seq it holds, if any.
pub through_seq: Option<u64>,
/// Whether it is stale (INVARIANT 3).
pub stale: bool,
/// Why its last backup failed, if it did.
pub last_error: Option<String>,
/// When its backup was last drilled (UTC), if ever.
pub last_drill_at: Option<String>,
/// Why its last drill failed, if it did.
pub drill_error: Option<String>,
/// Paused for books restored as a copy of others (INVARIANT 7).
pub paused: bool,
}
/// The books' backup standing.
#[derive(Debug, Clone, Serialize, PartialEq, Eq)]
pub struct Status {
/// The books' head now.
pub head_seq: u64,
/// Each target the settings name.
pub targets: Vec<TargetStatus>,
/// None set up: what to do.
pub note: Option<String>,
}
/// A target the settings name.
#[derive(Debug, Clone)]
struct Named {
target: Target,
/// Where, as an agent is told it.
at: String,
/// The target itself, as its records name it (INVARIANT 2): a digest
/// of where it is (never a path an agent is shown).
key: String,
}
/// The targets the book's settings name (D-30).
fn targets(store: &ledger_store::Store, book: &str) -> Vec<Named> {
use sha2::{Digest, Sha256};
let s = store.ledger().settings();
let mut out = Vec::new();
if let Some(url) = s.backup_git_remote() {
out.push((Target::Remote(url.to_string()), url.to_string()));
}
if let Some(folder) = s.backup_folder() {
out.push((
Target::Folder(PathBuf::from(folder)),
"the backup folder".to_string(),
));
}
out.into_iter()
.map(|(target, at)| Named {
key: Sha256::digest(target.source(book).as_bytes())[..8]
.iter()
.map(|b| format!("{b:02x}"))
.collect(),
target,
at,
})
.collect()
}
impl Backups {
/// The backups of the book `book`, kept in `dir`, served from `store`.
pub fn open(book: &str, dir: &Path, store: Arc<SharedStore>) -> Result<Self, String> {
ledger_store::ops::prepare(dir)
.map_err(|e| format!("its backup record can't be kept: {e}"))?;
let path = ledger_store::ops::ops_dir(dir).join(LOG_FILE);
let (log, carried) = match ChainLog::open(&path, GENESIS) {
Ok((log, _)) => (log, Vec::new()),
// INVARIANT 8: set aside, begun again, what it knew of whose
// backups the targets hold carried into the new one.
Err(ledger_store::chain::ChainError::Damaged { .. }) => {
let known = known_before_damage(&path);
let aside = path.with_extension(format!(
"jsonl.damaged-{}",
chrono::Utc::now().format("%Y%m%d-%H%M%S")
));
eprintln!(
"wanderbooks server: a backup record was damaged: set aside, begun again"
);
std::fs::rename(&path, &aside)
.map_err(|e| format!("its backup record can't be set aside: {}", e.kind()))?;
let log = ChainLog::open(&path, GENESIS)
.map_err(|e| format!("its backup record can't be used: {e}"))?
.0;
(log, known)
}
Err(e) => return Err(format!("its backup record can't be used: {e}")),
};
let me = Self {
book: book.to_string(),
dir: dir.to_path_buf(),
store,
log: Mutex::new(log),
running: tokio::sync::Mutex::new(()),
};
for fields in carried {
me.record(fields);
}
// Books just restored: whose they were (INVARIANTS 6, 7).
let marker = ledger_store::ops::ops_dir(dir).join(ledger_store::root::RESTORED_FROM);
if let Ok(origin) = std::fs::read_to_string(&marker) {
let origin = Some(origin.trim()).filter(|o| !o.is_empty());
let named = me
.store
.try_with_store(|s| targets(s, book))
.ok_or("its backups can't be read yet: the books are busy")?;
me.restored(origin, &named);
std::fs::remove_file(&marker)
.map_err(|e| format!("its backup record can't be kept: {}", e.kind()))?;
}
Ok(me)
}
fn record(&self, fields: Value) {
let Value::Object(mut map) = fields else {
return;
};
// INVARIANT 4: a git or store message may quote a path in the root.
if let Some(Value::String(e)) = map.get_mut("error") {
*e = hide(e, &self.dir);
}
let mut log = self.log.lock().unwrap_or_else(|e| e.into_inner());
if let Err(e) = log.append(map) {
eprintln!("wanderbooks server: the books' backup record couldn't be written: {e}");
}
}
fn records(&self) -> Vec<Map<String, Value>> {
ledger_store::chain::read(
&ledger_store::ops::ops_dir(&self.dir).join(LOG_FILE),
GENESIS,
)
.map(|r| r.records)
.unwrap_or_default()
}
/// The targets the settings name now.
async fn named(&self) -> Vec<Named> {
let (store, book) = (self.store.clone(), self.book.clone());
tokio::task::spawn_blocking(move || store.with_store(|s| targets(s, &book)).ok())
.await
.ok()
.flatten()
.unwrap_or_default()
}
/// Note that these books were just restored from a backup of the books
/// `origin` (INVARIANTS 6, 7), `named` the targets they name: the same
/// books (their targets took their backups), or others' (the
/// repository those back up to is paused for these).
fn restored(&self, origin: Option<&str>, named: &[Named]) {
let same = origin == Some(self.book.as_str());
let keys = |remote: bool| -> Vec<String> {
named
.iter()
.filter(|n| !remote || matches!(n.target, Target::Remote(_)))
.map(|n| n.key.clone())
.collect()
};
if same {
self.record(json!({"kind": "restored", "origin": origin, "held": keys(false)}));
} else {
self.record(json!({"kind": "restored", "origin": origin, "paused": keys(true)}));
}
}
/// Wait for any run or drill to end, and start none while the guard
/// lives: books being replaced by a restore push nothing more.
pub async fn pause(&self) -> tokio::sync::MutexGuard<'_, ()> {
self.running.lock().await
}
/// The books' head now.
pub async fn head(&self) -> Option<(u64, String)> {
let store = self.store.clone();
tokio::task::spawn_blocking(move || store.with_store(|s| s.head()).ok())
.await
.ok()
.flatten()
}
/// Back the books up now (INVARIANTS 1, 2): `why` is recorded (a
/// write, open, close, asked). The status after.
pub async fn run(&self, why: &str) -> Result<Status, String> {
self.run_with(why, false).await
}
/// [`Self::run`]; with `new_disk`, a folder target that took these
/// books' backups and holds none now is begun again there (the owner
/// says it is a new disk).
pub async fn run_with(&self, why: &str, new_disk: bool) -> Result<Status, String> {
let _one = self.running.lock().await;
if self.named().await.is_empty() {
return Ok(self.status().await);
}
let (store, dir, book) = (self.store.clone(), self.dir.clone(), self.book.clone());
let made = tokio::task::spawn_blocking(move || {
let git = Git::find()?;
// The stored files first, outside the lock: they never change.
// Only a head start (a file being written can trip it): the
// commit stages them all again.
let _ = backup::stage_blobs(&git, &dir);
// The commit, under it.
store
.with_store(|s| -> Result<_, BackupError> {
let made = backup::commit(&git, &dir, &book, s.head())?;
Ok((made, targets(s, &book), s.head()))
})
.map_err(BackupError::Damaged)
.and_then(|r| r)
})
.await
.unwrap_or(Err(BackupError::Damaged("the backup stopped".into())));
let (made, named, head) = match made {
Ok(m) => m,
Err(e) => {
let e = hide(&e.to_string(), &self.dir);
self.record(json!({"kind": "commit", "why": why, "ok": false, "error": e}));
return Err(e);
}
};
if let Some(m) = &made {
self.record(
json!({"kind": "commit", "why": why, "ok": true, "commit": m.commit,
"head": {"seq": m.head.0, "hash": m.head.1}}),
);
}
// Each target, outside the lock; after a write, one that failed
// lately waits out RETRY_AFTER (the others don't wait for it).
let records = self.records();
let now = chrono::Utc::now();
for n in named {
if paused(&n.key, &records) || (why == "write" && failed_lately(&n.key, &records, now))
{
continue;
}
// Its repository is made only where it never took a backup.
let may_create = new_disk || !held_before(&n.key, &records);
let (dir, book, t) = (self.dir.clone(), self.book.clone(), n.target.clone());
let pushed = tokio::task::spawn_blocking(move || {
let git = Git::find()?;
backup::push(&git, &dir, &book, &t, may_create)
})
.await
.unwrap_or(Err(BackupError::Damaged("the push stopped".into())));
match pushed {
Ok(commit) => self.record(
json!({"kind": "push", "why": why, "target": n.target.kind(),
"at": n.at, "key": n.key, "ok": true, "commit": commit,
"head": {"seq": head.0, "hash": head.1}}),
),
Err(e) => self.record(
json!({"kind": "push", "why": why, "target": n.target.kind(),
"at": n.at, "key": n.key, "ok": false, "error": e.to_string()}),
),
}
}
Ok(self.status().await)
}
/// Whether a drill is due (INVARIANT 5).
pub async fn drill_due(&self) -> bool {
let records = self.records();
let now = chrono::Utc::now();
self.named()
.await
.iter()
.any(|n| drill_due(&n.key, &records, now))
}
/// Drill each target whose drill is due (INVARIANT 5), each recorded.
pub async fn drill(&self) {
let _one = self.running.lock().await;
let records = self.records();
let now = chrono::Utc::now();
for n in self.named().await {
if paused(&n.key, &records) || !drill_due(&n.key, &records, now) {
continue;
}
let (dir, book, t) = (self.dir.clone(), self.book.clone(), n.target.clone());
let drilled = tokio::task::spawn_blocking(move || {
let scratch = ledger_store::ops::ops_dir(&dir).join(format!("drill-{}", t.kind()));
let _ = std::fs::remove_dir_all(&scratch);
let git = Git::find()?;
backup::drill(
&git,
&t,
&book,
&dir.join(ledger_store::JOURNAL_FILE),
&scratch,
)
})
.await
.unwrap_or(Err(BackupError::Damaged("the drill stopped".into())));
match drilled {
Ok(head) => {
self.record(json!({"kind": "drill", "target": n.target.kind(),
"at": n.at, "key": n.key, "ok": true, "head": {"seq": head.0, "hash": head.1}}))
}
Err(e) => self.record(json!({"kind": "drill", "target": n.target.kind(),
"at": n.at, "key": n.key, "ok": false, "error": e.to_string()})),
}
}
}
/// The books' backup standing (INVARIANT 3).
pub async fn status(&self) -> Status {
let store = self.store.clone();
let (book, dir) = (self.book.clone(), self.dir.clone());
tokio::task::spawn_blocking(move || store.with_store(|s| standing(s, &book, &dir)).ok())
.await
.ok()
.flatten()
.unwrap_or_else(|| standing_of(&[], &[], 0, chrono::Utc::now()))
}
/// The standing, read from `store` (held by the caller: books.start).
pub fn standing(&self, store: &ledger_store::Store) -> Status {
standing(store, &self.book, &self.dir)
}
}
fn standing(store: &ledger_store::Store, book: &str, dir: &Path) -> Status {
let records =
ledger_store::chain::read(&ledger_store::ops::ops_dir(dir).join(LOG_FILE), GENESIS)
.map(|r| r.records)
.unwrap_or_default();
standing_of(
&targets(store, book),
&records,
store.head().0,
chrono::Utc::now(),
)
}
fn standing_of(
named: &[Named],
records: &[Map<String, Value>],
head_seq: u64,
now: chrono::DateTime<chrono::Utc>,
) -> Status {
Status {
head_seq,
note: named.is_empty().then(|| {
"no backups are set up: set backup.git_remote (a private repository) and backup.folder (a folder on another disk) with books.set_setting".to_string()
}),
targets: named
.iter()
.map(|n| status_of(n.target.kind(), &n.at, &n.key, records, head_seq, now))
.collect(),
}
}
impl Status {
/// Whether the human should hear of it: none set up, or one stale,
/// failing, or failing its drill.
pub fn needs_attention(&self) -> bool {
self.note.is_some()
|| self
.targets
.iter()
.any(|t| t.stale || t.last_error.is_some() || t.drill_error.is_some())
}
}
/// `text` with the book's folder `dir`, and the data root above it, named
/// in place of their paths (INVARIANT 4); at most ERROR_MAX characters.
fn hide(text: &str, dir: &Path) -> String {
let mut text = text.to_string();
let real = std::fs::canonicalize(dir).unwrap_or_else(|_| dir.to_path_buf());
for d in [dir, real.as_path()] {
text = text.replace(&d.display().to_string(), "the book's folder");
if let Some(root) = d.parent().and_then(Path::parent) {
text = text.replace(&root.display().to_string(), ledger_store::root::THE_ROOT);
}
}
text.chars().take(ERROR_MAX).collect()
}
fn is(r: &Map<String, Value>, kind: &str) -> bool {
r.get("kind").and_then(Value::as_str) == Some(kind)
}
fn ok(r: &Map<String, Value>) -> bool {
r.get("ok") == Some(&Value::Bool(true))
}
fn of(r: &Map<String, Value>, key: &str) -> bool {
r.get("key").and_then(Value::as_str) == Some(key)
}
fn ts(r: &Map<String, Value>) -> Option<chrono::DateTime<chrono::Utc>> {
r.get("ts")
.and_then(Value::as_str)
.and_then(|t| chrono::DateTime::parse_from_rfc3339(t).ok())
.map(|t| t.with_timezone(&chrono::Utc))
}
fn text(r: &Map<String, Value>, field: &str) -> Option<String> {
r.get(field).and_then(Value::as_str).map(str::to_string)
}
/// What a damaged record knew, up to its first line that can't be read at
/// all (a line read but out of the chain still counts: what it adds only
/// holds more back), of whose backups the
/// targets hold: its restored notes, and each target that took a backup
/// (as held).
fn known_before_damage(path: &Path) -> Vec<Value> {
let text = std::fs::read(path).unwrap_or_default();
let mut held: Vec<String> = Vec::new();
let mut out = Vec::new();
for line in text.split(|b| *b == b'\n') {
let Ok(Value::Object(r)) = serde_json::from_slice::<Value>(line) else {
break;
};
if is(&r, "push") && ok(&r) {
if let Some(k) = text_field(&r, "key")
&& !held.contains(&k)
{
held.push(k);
}
} else if is(&r, "restored") {
let mut r = r.clone();
for k in ["seq", "ts", "prev", "hash"] {
r.remove(k);
}
out.push(Value::Object(r));
}
}
if !held.is_empty() {
out.push(json!({"kind": "restored", "carried": true, "held": held}));
}
out
}
fn text_field(r: &Map<String, Value>, field: &str) -> Option<String> {
r.get(field).and_then(Value::as_str).map(str::to_string)
}
/// Whether the target `key` is paused for books restored beside others
/// (INVARIANT 7).
fn paused(key: &str, records: &[Map<String, Value>]) -> bool {
records.iter().any(|r| {
is(r, "restored")
&& r.get("paused")
.and_then(Value::as_array)
.is_some_and(|keys| keys.iter().any(|k| k.as_str() == Some(key)))
})
}
/// Whether the target `key` took a backup of these books (INVARIANT 6).
fn held_before(key: &str, records: &[Map<String, Value>]) -> bool {
records.iter().any(|r| {
(is(r, "push") && of(r, key) && ok(r))
|| (is(r, "restored")
&& r.get("held")
.and_then(Value::as_array)
.is_some_and(|keys| keys.iter().any(|k| k.as_str() == Some(key))))
})
}
/// What books restored beside others are told of the repository paused
/// for them.
pub const PAUSED: &str = "not backed up here: these books were restored as a copy of the books this repository keeps, and backing up here would stop theirs; name a repository of their own (backup.git_remote)";
/// Whether the target `key`'s last push failed less than RETRY_AFTER ago.
fn failed_lately(
key: &str,
records: &[Map<String, Value>],
now: chrono::DateTime<chrono::Utc>,
) -> bool {
records
.iter()
.rfind(|r| is(r, "push") && of(r, key))
.filter(|r| !ok(r))
.and_then(ts)
.is_some_and(|at| {
now.signed_duration_since(at)
< chrono::Duration::from_std(RETRY_AFTER).unwrap_or(chrono::Duration::zero())
})
}
/// Whether the target `key`'s drill is due (INVARIANT 5).
fn drill_due(
key: &str,
records: &[Map<String, Value>],
now: chrono::DateTime<chrono::Utc>,
) -> bool {
if !records.iter().any(|r| is(r, "push") && of(r, key) && ok(r)) {
return false;
}
match records.iter().rfind(|r| is(r, "drill") && of(r, key)) {
None => true,
Some(d) => ts(d).is_none_or(|at| {
let wait = if ok(d) {
DRILL_EVERY
} else {
chrono::Duration::from_std(RETRY_AFTER).unwrap_or(DRILL_EVERY)
};
now.signed_duration_since(at) >= wait
}),
}
}
/// A target's standing from the backup record (INVARIANT 3).
fn status_of(
kind: &str,
at: &str,
key: &str,
records: &[Map<String, Value>],
head_seq: u64,
now: chrono::DateTime<chrono::Utc>,
) -> TargetStatus {
let last_ok = records
.iter()
.rposition(|r| is(r, "push") && of(r, key) && ok(r));
let through_seq = last_ok
.and_then(|i| records[i].get("head"))
.and_then(|h| h.get("seq"))
.and_then(Value::as_u64);
// Since its last good push (or, never having had one, since its first
// try; or, never tried, the failed commits that kept it from one):
// this target's tries, and the commits.
let from = match last_ok {
Some(i) => i + 1,
None => records.iter().position(|r| of(r, key)).unwrap_or_else(|| {
records
.iter()
.rposition(|r| !(is(r, "commit") && !ok(r)))
.map_or(0, |i| i + 1)
}),
};
let since: Vec<&Map<String, Value>> = records[from..]
.iter()
.filter(|r| is(r, "commit") || (is(r, "push") && of(r, key)))
.collect();
let behind = through_seq.is_none_or(|s| s < head_seq);
let old = since
.first()
.and_then(|r| ts(r))
.is_some_and(|t| now.signed_duration_since(t) > STALE_AFTER);
let last_error = since
.iter()
.rev()
.find(|r| !ok(r) || is(r, "push"))
.filter(|r| !ok(r))
.and_then(|r| text(r, "error"));
let drill = records.iter().rfind(|r| is(r, "drill") && of(r, key));
let paused = paused(key, records);
TargetStatus {
target: kind.to_string(),
at: at.to_string(),
last_ok_at: last_ok.and_then(|i| text(&records[i], "ts")),
through_seq,
stale: behind && old && head_seq > 0 && !paused,
last_error: if paused {
Some(PAUSED.to_string())
} else {
last_error
},
last_drill_at: drill.and_then(|r| text(r, "ts")),
drill_error: drill.filter(|r| !ok(r)).and_then(|r| text(r, "error")),
paused,
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn no_error_names_the_root() {
let dir = Path::new("/home/u/.local/share/wanderbooks/books/acme");
assert_eq!(
hide(
"fatal: '/home/u/.local/share/wanderbooks/books/acme/archive.git' and /home/u/.local/share/wanderbooks/run",
dir
),
format!(
"fatal: 'the book's folder/archive.git' and {}/run",
ledger_store::root::THE_ROOT
)
);
assert_eq!(hide(&"x".repeat(2000), dir).len(), ERROR_MAX);
}
/// A record of `kind` for target `key`, `ago` before `now`.
fn rec(
now: chrono::DateTime<chrono::Utc>,
kind: &str,
key: &str,
ok: bool,
seq: u64,
ago: chrono::Duration,
) -> Map<String, Value> {
let mut m = Map::new();
m.insert("kind".into(), json!(kind));
if kind != "commit" {
m.insert("key".into(), json!(key));
}
m.insert("ok".into(), json!(ok));
m.insert("ts".into(), json!((now - ago).to_rfc3339()));
if ok {
m.insert("head".into(), json!({"seq": seq, "hash": "h"}));
} else {
m.insert("error".into(), json!(format!("{kind} failed")));
}
m
}
#[test]
fn a_target_is_stale_only_when_behind_since_a_day() {
let now = chrono::Utc::now();
let r = |kind, ok, seq, ago| rec(now, kind, "k", ok, seq, ago);
let st = |records: &[Map<String, Value>], head| {
status_of("folder", "f", "k", records, head, now)
};
let day = chrono::Duration::hours(24);
let hour = chrono::Duration::hours(1);
// Nothing tried yet: not stale.
assert!(!st(&[], 0).stale);
assert!(!st(&[], 4).stale);
// Never pushed, the first try a day ago: stale; an hour ago: not.
assert!(
st(
&[
r("commit", true, 3, day + hour),
r("push", false, 0, day + hour)
],
3
)
.stale
);
assert!(!st(&[r("commit", true, 3, hour), r("push", false, 0, hour)], 3).stale);
// Pushed long ago, and quiet since: a new write isn't stale at once.
let quiet = [r("commit", true, 3, day * 3), r("push", true, 3, day * 3)];
assert!(!st(&quiet, 4).stale);
// ... nor while it holds the head.
assert!(!st(&quiet, 3).stale);
// Behind since a commit a day old, its pushes failing: stale.
let failing = [
r("push", true, 3, day * 3),
r("commit", true, 4, day + hour),
r("push", false, 0, day),
r("push", false, 0, hour),
];
let t = st(&failing, 4);
assert!(t.stale);
assert_eq!(
(t.through_seq, t.last_error.as_deref()),
(Some(3), Some("push failed"))
);
// A good push ends the error.
let mended = [
&failing[..],
&[r("push", true, 4, chrono::Duration::zero())],
]
.concat();
let t = st(&mended, 4);
assert_eq!(
(t.stale, t.last_error, t.through_seq),
(false, None, Some(4))
);
// A failed commit is every target's error, until a good push.
let t = st(&[r("push", true, 3, day), r("commit", false, 0, hour)], 4);
assert_eq!(t.last_error.as_deref(), Some("commit failed"));
// A good commit after a failed push doesn't hide the failure.
let t = st(&[r("push", false, 0, hour), r("commit", true, 4, hour)], 4);
assert_eq!(t.last_error.as_deref(), Some("push failed"));
// Another target's records are not its: a new folder starts clean.
let other = [
rec(now, "push", "other", true, 3, hour),
rec(now, "push", "other", false, 0, hour),
];
let t = st(&other, 3);
assert_eq!(
(t.through_seq, t.last_error, t.last_ok_at),
(None, None, None)
);
}
#[test]
fn a_target_failing_lately_is_left_out_of_a_write_alone() {
let now = chrono::Utc::now();
let retry = chrono::Duration::from_std(RETRY_AFTER).unwrap();
let second = chrono::Duration::seconds(1);
let failed = |ago| rec(now, "push", "k", false, 0, ago);
assert!(failed_lately("k", &[failed(retry - second)], now));
assert!(!failed_lately("k", &[failed(retry)], now));
assert!(!failed_lately(
"k",
&[failed(second), rec(now, "push", "k", true, 3, second)],
now
));
assert!(!failed_lately("other", &[failed(second)], now));
assert!(!failed_lately("k", &[], now));
}
#[test]
fn commits_failing_from_the_start_are_every_targets_trouble() {
let now = chrono::Utc::now();
let day = chrono::Duration::hours(24);
let hour = chrono::Duration::hours(1);
// git missing (say) from the first run on: never a push of its own.
let failing = [
rec(now, "commit", "", false, 0, day + hour),
rec(now, "commit", "", false, 0, hour),
];
let t = status_of("folder", "f", "k", &failing, 3, now);
assert_eq!(t.last_error.as_deref(), Some("commit failed"));
assert!(t.stale, "{t:?}");
// Failing an hour only: said, not yet stale.
let t = status_of("folder", "f", "k", &failing[1..], 3, now);
assert_eq!(
(t.last_error.as_deref(), t.stale),
(Some("commit failed"), false)
);
// A commit that worked since (its push still to come): no error.
let mended = [
&failing[..],
&[rec(now, "commit", "", true, 3, chrono::Duration::zero())],
]
.concat();
assert_eq!(
status_of("folder", "f", "k", &mended, 3, now).last_error,
None
);
}
#[test]
fn restored_books_know_which_backups_were_theirs() {
let now = chrono::Utc::now();
let restored = |field: &str, keys: &[&str]| {
let mut m = Map::new();
m.insert("kind".into(), json!("restored"));
m.insert(field.into(), json!(keys));
m.insert("ts".into(), json!(now.to_rfc3339()));
m
};
// The same books: their targets took their backups.
let same = [restored("held", &["k"])];
assert!(held_before("k", &same));
assert!(!held_before("new", &same));
assert!(!paused("k", &same));
// Others' (restored beside them): their repository is paused, said.
let beside = [restored("paused", &["r"])];
assert!(paused("r", &beside));
assert!(!paused("k", &beside));
assert!(!held_before("r", &beside));
let t = status_of("repository", "x", "r", &beside, 9, now);
assert_eq!(t.last_error.as_deref(), Some(PAUSED));
assert!(!t.stale && t.paused);
// A push of its own counts as held, too.
assert!(held_before(
"k",
&[rec(now, "push", "k", true, 3, chrono::Duration::zero())]
));
assert!(!held_before(
"k",
&[rec(now, "push", "k", false, 0, chrono::Duration::zero())]
));
}
#[test]
fn a_target_named_later_starts_its_own_clock() {
let now = chrono::Utc::now();
let day = chrono::Duration::hours(24);
let hour = chrono::Duration::hours(1);
// Commits for a week, to another target; then this one's first try.
let records = [
rec(now, "commit", "", true, 2, day * 7),
rec(now, "push", "old", true, 2, day * 7),
rec(now, "commit", "", true, 3, day * 2),
rec(now, "push", "k", false, 0, hour),
];
let t = status_of("folder", "f", "k", &records, 3, now);
assert!(!t.stale, "{t:?}");
assert_eq!(t.last_error.as_deref(), Some("push failed"));
// Holding the head: never stale; one behind since long: stale.
let held = [
rec(now, "push", "k", true, 3, day * 3),
rec(now, "push", "k", false, 0, day * 2),
];
assert!(!status_of("folder", "f", "k", &held, 3, now).stale);
assert!(status_of("folder", "f", "k", &held, 4, now).stale);
}
#[test]
fn drills_are_due_weekly_after_a_pass_and_soon_after_a_failure() {
let now = chrono::Utc::now();
let r = |kind, ok, ago| rec(now, kind, "k", ok, 3, ago);
let hour = chrono::Duration::hours(1);
let week = DRILL_EVERY;
let retry = chrono::Duration::from_std(RETRY_AFTER).unwrap();
// Nothing backed up there: nothing to drill.
assert!(!drill_due("k", &[], now));
assert!(!drill_due("k", &[r("push", false, hour)], now));
// Backed up, never drilled: due.
let pushed = r("push", true, hour);
assert!(drill_due("k", std::slice::from_ref(&pushed), now));
// Passed: due a week after.
assert!(!drill_due(
"k",
&[pushed.clone(), r("drill", true, week - hour)],
now
));
assert!(drill_due(
"k",
&[pushed.clone(), r("drill", true, week)],
now
));
// Failed: tried again RETRY_AFTER after.
let t = drill_due(
"k",
&[
pushed.clone(),
r("drill", false, retry - chrono::Duration::seconds(1)),
],
now,
);
assert!(!t);
assert!(drill_due(
"k",
&[pushed.clone(), r("drill", false, retry)],
now
));
// Another target's drill isn't its, nor its push.
assert!(drill_due(
"k",
&[pushed.clone(), rec(now, "drill", "o", true, 3, hour)],
now
));
assert!(!drill_due(
"k",
&[rec(now, "push", "o", true, 3, hour)],
now
));
// A failed drill is said until one passes.
let t = status_of(
"folder",
"f",
"k",
&[pushed.clone(), r("drill", false, hour)],
3,
now,
);
assert_eq!(t.drill_error.as_deref(), Some("drill failed"));
assert!(t.last_drill_at.is_some());
let t = status_of(
"folder",
"f",
"k",
&[pushed, r("drill", false, hour), r("drill", true, hour)],
3,
now,
);
assert_eq!(t.drill_error, None);
}
#[test]
fn each_kind_of_trouble_needs_attention_alone() {
let clean = TargetStatus {
target: "folder".into(),
at: "f".into(),
last_ok_at: Some("t".into()),
through_seq: Some(3),
stale: false,
last_error: None,
last_drill_at: None,
drill_error: None,
paused: false,
};
let status = |t: TargetStatus| Status {
head_seq: 3,
targets: vec![t],
note: None,
};
assert!(!status(clean.clone()).needs_attention());
assert!(
status(TargetStatus {
stale: true,
..clean.clone()
})
.needs_attention()
);
assert!(
status(TargetStatus {
last_error: Some("e".into()),
..clean.clone()
})
.needs_attention()
);
assert!(
status(TargetStatus {
drill_error: Some("e".into()),
..clean.clone()
})
.needs_attention()
);
let none = Status {
head_seq: 3,
targets: vec![],
note: Some("n".into()),
};
assert!(none.needs_attention());
}
}
+1
View File
@@ -81,6 +81,7 @@ pub const SECTIONS: &[(&str, &str)] = &[
include_str!("../handbook/settings-and-conventions.md"), include_str!("../handbook/settings-and-conventions.md"),
), ),
("viewer", include_str!("../handbook/viewer.md")), ("viewer", include_str!("../handbook/viewer.md")),
("backups", include_str!("../handbook/backups.md")),
( (
"rules-that-change", "rules-that-change",
include_str!("../handbook/rules-that-change.md"), include_str!("../handbook/rules-that-change.md"),
+1
View File
@@ -16,6 +16,7 @@
pub mod access; pub mod access;
pub mod admin; pub mod admin;
pub mod backup;
mod handbook; mod handbook;
#[cfg(unix)] #[cfg(unix)]
pub mod local; pub mod local;
+475 -2
View File
@@ -75,7 +75,7 @@ use std::os::fd::AsRawFd;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicUsize, Ordering}; use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::{Arc, Mutex, RwLock}; use std::sync::{Arc, Mutex, RwLock};
use std::time::Duration; use std::time::{Duration, Instant};
use tokio_util::sync::CancellationToken; use tokio_util::sync::CancellationToken;
use tokio_util::task::TaskTracker; use tokio_util::task::TaskTracker;
@@ -103,6 +103,18 @@ pub const REFUSED_HEADER: &str = "x-wanderbooks-refused";
/// How often each book's lapsed uploads are swept. /// How often each book's lapsed uploads are swept.
const SWEEP_EVERY: Duration = Duration::from_secs(10); const SWEEP_EVERY: Duration = Duration::from_secs(10);
/// How long the books stay unwritten before they are backed up.
const BACKUP_QUIET: Duration = Duration::from_secs(30);
/// How often the books are looked at for a backup due.
const BACKUP_LOOK: Duration = Duration::from_secs(2);
/// How often each book is looked at for a drill due.
const DRILL_LOOK: Duration = Duration::from_secs(3600);
/// How long after books are first seen open their first drill is looked
/// for (a drill that fell due while they were closed): long enough that a
/// backup on opening comes first.
const DRILL_FIRST: Duration = Duration::from_secs(60);
/// The most the last backup at a close may take.
const BACKUP_AT_CLOSE: Duration = Duration::from_secs(60);
type Service = StreamableHttpService<BooksServer, LocalSessionManager>; type Service = StreamableHttpService<BooksServer, LocalSessionManager>;
@@ -122,6 +134,8 @@ enum Book {
uploads: ledger_store::uploads::Uploads, uploads: ledger_store::uploads::Uploads,
/// The files it made for agents (`/b/{id}/x/{export}`). /// The files it made for agents (`/b/{id}/x/{export}`).
exports: ledger_store::exports::Exports, exports: ledger_store::exports::Exports,
/// Its backups (backup.rs).
backups: Arc<crate::backup::Backups>,
}, },
/// Not opened, with why (INVARIANT 3). /// Not opened, with why (INVARIANT 3).
Failed(String), Failed(String),
@@ -160,6 +174,14 @@ struct Books {
/// Which token opened each MCP session: a session is used only with /// Which token opened each MCP session: a session is used only with
/// the token that opened it (INVARIANT 8). /// the token that opened it (INVARIANT 8).
session_tokens: Mutex<BTreeMap<String, String>>, session_tokens: Mutex<BTreeMap<String, String>>,
/// How long the books stay unwritten before a backup is made
/// (`--backup-quiet`, for tests; else BACKUP_QUIET).
backup_quiet: Duration,
/// How long after books are first seen open a drill is first looked
/// for (`--drill-first`, for tests; else DRILL_FIRST).
drill_first: Duration,
/// Held through a restore: one at a time.
restoring: Mutex<()>,
} }
impl Books { impl Books {
@@ -284,7 +306,17 @@ impl Books {
// INVARIANT 2): a viewer can show its presence from the start. // INVARIANT 2): a viewer can show its presence from the start.
ledger_store::ops::prepare(store.data_dir()).map_err(|e| self.hide(e.to_string()))?; ledger_store::ops::prepare(store.data_dir()).map_err(|e| self.hide(e.to_string()))?;
let live = Arc::new(Mutex::new(live)); let live = Arc::new(Mutex::new(live));
let book_dir = store.data_dir().to_path_buf();
let book_id = book_dir
.file_name()
.map(|n| n.to_string_lossy().into_owned())
.unwrap_or_default();
let store = SharedStore::new(store); let store = SharedStore::new(store);
let backups = Arc::new(
crate::backup::Backups::open(&book_id, &book_dir, store.clone())
.map_err(|e| self.hide(e))?,
);
let server_backups = backups.clone();
let log = access.clone(); let log = access.clone();
let shared = store.clone(); let shared = store.clone();
let config = StreamableHttpServerConfig::default() let config = StreamableHttpServerConfig::default()
@@ -301,7 +333,8 @@ impl Books {
.with_layout(layout.clone()) .with_layout(layout.clone())
.with_access(log.clone()) .with_access(log.clone())
.with_uploads(server_uploads.clone()) .with_uploads(server_uploads.clone())
.with_exports(server_exports.clone()), .with_exports(server_exports.clone())
.with_backups(server_backups.clone()),
) )
}, },
Arc::new(sessions()), Arc::new(sessions()),
@@ -313,6 +346,7 @@ impl Books {
access, access,
uploads, uploads,
exports, exports,
backups,
}) })
} }
@@ -449,6 +483,13 @@ impl Books {
Err(e) => refused(e), Err(e) => refused(e),
} }
} }
Request::Restore {
id,
source,
replace,
confirmed,
expect_head,
} => self.restore(id, source, replace, confirmed, expect_head),
Request::Adopt { Request::Adopt {
id, id,
source, source,
@@ -589,6 +630,119 @@ impl Books {
} }
} }
impl Books {
/// Restore a backup (admin INVARIANT 4; ledger_store::root INVARIANT
/// 7): a preview, or the books restored and opened. Books it replaces
/// are let go first, and opened again if the restore fails.
fn restore(
&self,
id: String,
source: String,
replace: bool,
confirmed: bool,
expect_head: Option<(u64, String)>,
) -> Response {
// admin INVARIANT 3: never the root, nor the address given.
let said = |text: String| {
let text = if source.chars().count() > 1 {
text.replace(&source, "the backup given")
} else {
text
};
Response::Refused(self.hide(text))
};
let id = match BookId::parse(&id) {
Ok(id) => id,
Err(e) => return said(e.to_string()),
};
let _one = self.restoring.lock().unwrap_or_else(|e| e.into_inner());
let git = match ledger_store::backup::Git::find() {
Ok(git) => git,
Err(e) => return said(e.to_string()),
};
// A preview only reports (whether books of that id exist among
// what it says): the restore itself is refused over books, or
// replacing none.
if !confirmed {
return match self.root.restore_preview(&git, &source, &id) {
Ok(p) => Response::RestorePreview(admin::RestorePreview {
head: p.head,
blobs: p.blobs,
vault_files: p.vault_files,
exists: p.exists,
origin: p.origin,
}),
Err(e) => said(e.to_string()),
};
}
let Some(expect_head) = expect_head else {
return said("a restore is confirmed against the preview's head: preview first".into());
};
let held = self.held().contains_key(id.as_str());
if replace && !held {
return said(format!(
"there are no books called {:?} to replace: nothing was restored",
id.as_str()
));
}
if !replace && held {
return said(format!(
"there are already books called {:?}: restore them under another id (--id), or replace them (--replace; they are kept in legacy/)",
id.as_str()
));
}
// The books replaced are let go (their writer lock released),
// their backups idle until the restore is done: none of theirs
// reaches a target after the backup is read.
let old = if replace {
self.held
.write()
.unwrap_or_else(|e| e.into_inner())
.remove(id.as_str())
} else {
None
};
let old_backups = match &old {
Some(Book::Open { backups, .. }) => Some(backups.clone()),
_ => None,
};
let rt = tokio::runtime::Handle::current();
// Its requests end first (a books.back_up_now among them waits for
// a run of the loop's), then its last run.
if let Some(Book::Open { store, .. }) = &old {
rt.block_on(store.let_go());
}
let _paused = old_backups.as_ref().map(|b| rt.block_on(b.pause()));
match self.root.restore(&git, &source, &id, &expect_head, replace) {
Ok(r) => {
// Restored: kept whether or not it opens; its backups told
// whose books these were (backup.rs INVARIANTS 6, 7).
let book = self.open(&id);
let problem = book.problem();
self.hold(&id, book);
Response::Restored(admin::Restored {
id: r.id.to_string(),
head: r.head,
blobs_verified: r.blobs_verified,
vault_files: r.vault_files,
replaced: r.replaced,
origin: r.origin,
problem,
})
}
Err(e) => {
// The books it would have replaced, let go above, open
// again (only this restore runs, so nothing else holds them).
if replace && !self.held().contains_key(id.as_str()) {
let book = self.open(&id);
self.hold(&id, book);
}
said(e.to_string())
}
}
}
}
/// The settings a settings file moves in at adoption, checked as /// The settings a settings file moves in at adoption, checked as
/// `books.import_settings` checks them: they parse, they apply, and the /// `books.import_settings` checks them: they parse, they apply, and the
/// vault key file it names (which the books don't keep) is the one given. /// vault key file it names (which the books don't keep) is the one given.
@@ -1603,10 +1757,28 @@ pub fn run(
args: impl IntoIterator<Item = String>, args: impl IntoIterator<Item = String>,
) -> Result<(), Box<dyn std::error::Error>> { ) -> Result<(), Box<dyn std::error::Error>> {
let mut root: Option<PathBuf> = None; let mut root: Option<PathBuf> = None;
let mut backup_quiet = BACKUP_QUIET;
let mut drill_first = DRILL_FIRST;
let mut args = args.into_iter(); let mut args = args.into_iter();
while let Some(arg) = args.next() { while let Some(arg) = args.next() {
match arg.as_str() { match arg.as_str() {
"--root" => root = Some(PathBuf::from(args.next().ok_or("--root needs a folder")?)), "--root" => root = Some(PathBuf::from(args.next().ok_or("--root needs a folder")?)),
// How long the books stay unwritten before a backup (tests).
"--backup-quiet" => {
let ms: u64 = args
.next()
.and_then(|v| v.parse().ok())
.ok_or("--backup-quiet needs milliseconds")?;
backup_quiet = Duration::from_millis(ms);
}
// How soon after books open a drill is first looked for (tests).
"--drill-first" => {
let ms: u64 = args
.next()
.and_then(|v| v.parse().ok())
.ok_or("--drill-first needs milliseconds")?;
drill_first = Duration::from_millis(ms);
}
"--help" | "-h" => { "--help" | "-h" => {
print_usage(prog); print_usage(prog);
return Ok(()); return Ok(());
@@ -1684,6 +1856,9 @@ pub fn run(
started: std::time::Instant::now(), started: std::time::Instant::now(),
principals: RwLock::new(Err(String::new())), principals: RwLock::new(Err(String::new())),
session_tokens: Mutex::new(BTreeMap::new()), session_tokens: Mutex::new(BTreeMap::new()),
backup_quiet,
drill_first,
restoring: Mutex::new(()),
}); });
{ {
let loaded = books.load_principals(); let loaded = books.load_principals();
@@ -1753,6 +1928,10 @@ async fn serve(
} }
let held: Vec<String> = books.held().keys().cloned().collect(); let held: Vec<String> = books.held().keys().cloned().collect();
eprintln!("wanderbooks server: ready, holding {} books", held.len()); eprintln!("wanderbooks server: ready, holding {} books", held.len());
// Every open book is backed up when it opens, once its writing has
// stopped for a while, and again after a failed run (backup.rs).
let (backing, backup_stop) = (books.clone(), stop.clone());
let backup_loop = tokio::spawn(async move { keep_backed_up(backing, backup_stop).await });
// Every open book's lapsed uploads go within SWEEP_EVERY of lapsing, // Every open book's lapsed uploads go within SWEEP_EVERY of lapsing,
// whatever else the server is doing (uploads.rs INVARIANT 3). // whatever else the server is doing (uploads.rs INVARIANT 3).
let (sweeping, sweep_stop) = (books.clone(), stop.clone()); let (sweeping, sweep_stop) = (books.clone(), stop.clone());
@@ -1823,6 +2002,22 @@ async fn serve(
for store in books.stores() { for store in books.stores() {
store.close().await; store.close().await;
} }
// A last backup of what was written since the last one, for a while
// at most: the books are closing, nothing more is written. The loop
// stops at its next step (`stop`); a run of its still under way
// finishes first (each book runs one at a time).
drop(backup_loop);
let held: Vec<Arc<crate::backup::Backups>> = books
.held()
.values()
.filter_map(|b| match b {
Book::Open { backups, .. } => Some(backups.clone()),
Book::Failed(_) => None,
})
.collect();
for b in held {
let _ = tokio::time::timeout(BACKUP_AT_CLOSE, b.run("close")).await;
}
// Answers on their way out are sent, for a while: a client sending // Answers on their way out are sent, for a while: a client sending
// a request it never finishes doesn't hold the books open. // a request it never finishes doesn't hold the books open.
let answers = async { let answers = async {
@@ -1916,10 +2111,288 @@ fn watch_for_close(stops: &Stops) -> std::io::Result<()> {
Ok(()) Ok(())
} }
/// What the backup loop knows of one open book.
struct Seen {
/// Its backups, by identity: books restored over others are new.
backups: Arc<crate::backup::Backups>,
watch: Watch,
}
/// One open book's backups, as the loop has watched them.
#[derive(Debug, Clone, Copy)]
struct Watch {
/// Its head's seq when last looked at, and when that changed.
seq: u64,
moved_at: Instant,
/// The seq last committed and offered to the targets.
backed: Option<u64>,
/// When the last run's commit failed, if it did: nothing runs again
/// for RETRY_AFTER.
failed_at: Option<Instant>,
/// When the last run left a target failing, if it did: it is tried
/// again RETRY_AFTER on, written to or not.
target_failed_at: Option<Instant>,
/// When a drill is next looked for.
drill_next: Instant,
}
impl Watch {
/// Books first seen open now, at `seq`: a drill is first looked for
/// `drill_first` from now.
fn new(seq: u64, now: Instant, drill_first: Duration) -> Self {
Watch {
seq,
moved_at: now,
backed: None,
failed_at: None,
target_failed_at: None,
drill_next: now + drill_first,
}
}
/// A run through `seq` came to `outcome` at `now`: None when its
/// commit failed, else whether every target holds it.
fn ran(&mut self, seq: u64, outcome: Option<bool>, now: Instant) {
match outcome {
None => self.failed_at = Some(now),
Some(all) => {
self.backed = Some(seq);
self.failed_at = None;
self.target_failed_at = (!all).then_some(now);
}
}
}
}
/// Why a backup is due now, if one is: the books just seen open; or
/// written and quiet for `quiet` since (not yet backed up); or a target
/// left failing RETRY_AFTER ago. Nothing within RETRY_AFTER of a failed
/// commit.
fn backup_due(
seen: Option<&Watch>,
seq: u64,
now: Instant,
quiet: Duration,
) -> Option<&'static str> {
let Some(s) = seen else {
return Some("open");
};
let after =
|t: Option<Instant>| t.is_some_and(|f| now.duration_since(f) >= crate::backup::RETRY_AFTER);
if s.failed_at.is_some() && !after(s.failed_at) {
return None;
}
let quiet_now = if s.seq == seq {
now.duration_since(s.moved_at) >= quiet
} else {
quiet.is_zero()
};
if s.backed != Some(seq) && quiet_now {
return Some("write");
}
after(s.target_failed_at).then_some("retry")
}
/// Whether a run's every target holds the books: a target paused for
/// these books (restored as a copy of others) is no failure to try again.
fn every_target_held(status: &crate::backup::Status) -> bool {
status
.targets
.iter()
.all(|t| t.paused || t.last_error.is_none())
}
/// Whether to look for a drill due now (then again DRILL_LOOK on).
fn drill_look_due(seen: &Watch, now: Instant) -> bool {
now >= seen.drill_next
}
/// Back every open book up (backup.rs) whenever [`backup_due`] says, and
/// drill its backups when a drill is due (looked for every DRILL_LOOK).
/// Until `stop`.
async fn keep_backed_up(books: Arc<Books>, stop: CancellationToken) {
let mut seen: BTreeMap<String, Seen> = BTreeMap::new();
let mut tick = tokio::time::interval(BACKUP_LOOK);
loop {
tokio::select! {
() = stop.cancelled() => return,
_ = tick.tick() => {}
}
let open: Vec<(String, Arc<crate::backup::Backups>)> = books
.held()
.iter()
.filter_map(|(id, b)| match b {
Book::Open { backups, .. } => Some((id.clone(), backups.clone())),
Book::Failed(_) => None,
})
.collect();
seen.retain(|id, _| open.iter().any(|(o, _)| o == id));
for (id, backups) in open {
if stop.is_cancelled() {
return;
}
let Some((seq, _)) = backups.head().await else {
continue;
};
let now = Instant::now();
// Books restored over others are seen anew.
if seen
.get(&id)
.is_some_and(|s| !Arc::ptr_eq(&s.backups, &backups))
{
seen.remove(&id);
}
let due = backup_due(
seen.get(&id).map(|s| &s.watch),
seq,
now,
books.backup_quiet,
);
let s = &mut seen
.entry(id)
.or_insert(Seen {
backups: backups.clone(),
watch: Watch::new(seq, now, books.drill_first),
})
.watch;
if s.seq != seq {
s.seq = seq;
s.moved_at = now;
}
if let Some(why) = due {
let outcome = backups
.run(why)
.await
.ok()
.map(|status| every_target_held(&status));
s.ran(seq, outcome, now);
}
if stop.is_cancelled() {
return;
}
// Drills (backup.rs INVARIANT 5).
if drill_look_due(s, now) {
s.drill_next = now + DRILL_LOOK;
if backups.drill_due().await {
backups.drill().await;
}
}
}
}
}
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
#[test]
fn a_backup_is_due_on_open_then_once_quiet_after_a_write() {
let t0 = Instant::now();
let quiet = Duration::from_secs(30);
let s = |seq, moved: u64, backed, failed: Option<u64>| Watch {
seq,
moved_at: t0 + Duration::from_secs(moved),
backed,
failed_at: failed.map(|f| t0 + Duration::from_secs(f)),
target_failed_at: None,
drill_next: t0,
};
let at = |secs: u64| t0 + Duration::from_secs(secs);
// First seen: at once.
assert_eq!(backup_due(None, 5, t0, quiet), Some("open"));
// Backed up through the head: nothing.
assert_eq!(
backup_due(Some(&s(5, 0, Some(5), None)), 5, at(100), quiet),
None
);
// Written: not until it has been quiet `quiet`.
assert_eq!(
backup_due(Some(&s(5, 0, Some(5), None)), 6, at(1), quiet),
None
);
assert_eq!(
backup_due(Some(&s(6, 10, Some(5), None)), 6, at(39), quiet),
None
);
assert_eq!(
backup_due(Some(&s(6, 10, Some(5), None)), 6, at(40), quiet),
Some("write")
);
// No quiet asked for: at once.
assert_eq!(
backup_due(Some(&s(5, 0, Some(5), None)), 6, at(1), Duration::ZERO),
Some("write")
);
// A failure waits RETRY_AFTER.
let retry = crate::backup::RETRY_AFTER.as_secs();
let failed = s(6, 10, Some(5), Some(50));
assert_eq!(
backup_due(Some(&failed), 6, at(50 + retry - 1), quiet),
None
);
assert_eq!(
backup_due(Some(&failed), 6, at(50 + retry), quiet),
Some("write")
);
// A drill is first looked for `drill_first` after the books are
// seen open, never at once.
let first = Duration::from_secs(60);
let w = Watch::new(1, t0, first);
assert!(!drill_look_due(&w, t0));
assert!(!drill_look_due(&w, t0 + first - Duration::from_millis(1)));
assert!(drill_look_due(&w, t0 + first));
assert_eq!((w.backed, w.failed_at, w.seq), (None, None, 1));
// A run's outcome: its commit failed (nothing again for a
// while), or made and some target failing (tried again later,
// while writes still go to the others), or all well.
let mut w = Watch::new(1, t0, first);
w.ran(4, None, at(5));
assert_eq!(
(w.backed, w.failed_at, w.target_failed_at),
(None, Some(at(5)), None)
);
w.ran(4, Some(false), at(9));
assert_eq!(
(w.backed, w.failed_at, w.target_failed_at),
(Some(4), None, Some(at(9)))
);
w.seq = 5;
w.moved_at = at(10);
assert_eq!(backup_due(Some(&w), 5, at(10 + 30), quiet), Some("write"));
w.backed = Some(5);
assert_eq!(backup_due(Some(&w), 5, at(9 + retry - 1), quiet), None);
assert_eq!(backup_due(Some(&w), 5, at(9 + retry), quiet), Some("retry"));
// A paused target is no failure; another's error is.
let target = |paused: bool, error: Option<&str>| crate::backup::TargetStatus {
target: "repository".into(),
at: "x".into(),
last_ok_at: None,
through_seq: None,
stale: false,
last_error: error.map(str::to_string),
last_drill_at: None,
drill_error: None,
paused,
};
let status = |targets| crate::backup::Status {
head_seq: 5,
targets,
note: None,
};
assert!(every_target_held(&status(vec![
target(true, Some("paused")),
target(false, None)
])));
assert!(!every_target_held(&status(vec![
target(false, Some("no")),
target(false, None)
])));
assert!(every_target_held(&status(vec![])));
w.ran(5, Some(true), at(9 + retry));
assert_eq!((w.failed_at, w.target_failed_at), (None, None));
assert_eq!(backup_due(Some(&w), 5, at(9 + retry * 3), quiet), None);
}
#[test] #[test]
fn a_socket_pair_is_this_account_at_both_ends() { fn a_socket_pair_is_this_account_at_both_ends() {
let (a, b) = std::os::unix::net::UnixStream::pair().unwrap(); let (a, b) = std::os::unix::net::UnixStream::pair().unwrap();
+112 -11
View File
@@ -23,6 +23,11 @@
//! with include_tins, export_cpa_package, get_w9) need books:admin, //! with include_tins, export_cpa_package, get_w9) need books:admin,
//! and sealing one into the vault (set_tin, set_w9, which can replace //! and sealing one into the vault (set_tin, set_w9, which can replace
//! the one there) needs books:post (review of slice 4). //! the one there) needs books:post (review of slice 4).
//! 6. Where the books' backups go is books:admin's, as backing up is
//! (a backup is the whole book, its vault and key included):
//! setting a `backup.*` key, or importing settings that hold one,
//! needs books:admin; other settings books:post as before (review of
//! slice 7).
//! ``` //! ```
use ledger_store::auth::Scope; use ledger_store::auth::Scope;
@@ -38,9 +43,17 @@ pub enum Needs {
/// books:post for an approval rule (the control that gates posting), /// books:post for an approval rule (the control that gates posting),
/// books:write for any other convention (INVARIANT 4). /// books:write for any other convention (INVARIANT 4).
PostForApprovalRules, PostForApprovalRules,
/// books:admin for a change to a `backup.*` setting, books:post for
/// any other (INVARIANT 6).
AdminForBackupSettings,
/// As PostWhenConfirmed, and books:admin when the settings imported
/// hold a `[backup]` table (INVARIANT 6).
ImportSettings,
} }
use Needs::{Always, PostForApprovalRules, PostWhenConfirmed}; use Needs::{
AdminForBackupSettings, Always, ImportSettings, PostForApprovalRules, PostWhenConfirmed,
};
const R: Needs = Always(Scope::Read); const R: Needs = Always(Scope::Read);
const W: Needs = Always(Scope::Write); const W: Needs = Always(Scope::Write);
const P: Needs = Always(Scope::Post); const P: Needs = Always(Scope::Post);
@@ -49,12 +62,12 @@ const C: Needs = PostWhenConfirmed;
const K: Needs = PostForApprovalRules; const K: Needs = PostForApprovalRules;
/// Every tool and what it needs (INVARIANT 1). /// Every tool and what it needs (INVARIANT 1).
pub const TABLE: [(&str, Needs); 109] = [ pub const TABLE: [(&str, Needs); 111] = [
("books.handbook", R), ("books.handbook", R),
("books.start", R), ("books.start", R),
// Settings are committed at once: the settings commit of the plan. // Settings are committed at once: the settings commit of the plan.
("books.set_setting", P), ("books.set_setting", AdminForBackupSettings),
("books.import_settings", C), ("books.import_settings", ImportSettings),
("books.export_settings", R), ("books.export_settings", R),
("books.conventions", R), ("books.conventions", R),
("books.set_convention", K), ("books.set_convention", K),
@@ -166,6 +179,8 @@ pub const TABLE: [(&str, Needs); 109] = [
("books.profit_and_loss_by_class", R), ("books.profit_and_loss_by_class", R),
("books.export_cpa_package", A), ("books.export_cpa_package", A),
("books.access_log", A), ("books.access_log", A),
("books.backup", R),
("books.back_up_now", A),
]; ];
/// What `tool` needs, given its arguments; None for a tool this server /// What `tool` needs, given its arguments; None for a tool this server
@@ -178,11 +193,38 @@ pub fn needs(
Some(match needs { Some(match needs {
Always(scope) => *scope, Always(scope) => *scope,
PostWhenConfirmed => { PostWhenConfirmed => {
let confirmed = args if confirmed(args) {
.and_then(|a| a.get("confirmed")) Scope::Post
.and_then(serde_json::Value::as_bool) } else {
.unwrap_or(false); Scope::Write
if confirmed { Scope::Post } else { Scope::Write } }
}
AdminForBackupSettings => {
let backup = args
.and_then(|a| a.get("changes"))
.and_then(serde_json::Value::as_array)
.is_some_and(|changes| {
changes.iter().any(|c| {
c.get("key")
.and_then(serde_json::Value::as_str)
.is_some_and(|k| k.trim().to_ascii_lowercase().starts_with("backup."))
})
});
if backup { Scope::Admin } else { Scope::Post }
}
ImportSettings => {
let backup = args
.and_then(|a| a.get("toml_text"))
.and_then(serde_json::Value::as_str)
.and_then(|t| t.parse::<toml::Table>().ok())
.is_some_and(|t| t.contains_key("backup"));
if backup {
Scope::Admin
} else if confirmed(args) {
Scope::Post
} else {
Scope::Write
}
} }
PostForApprovalRules => { PostForApprovalRules => {
let kind = args let kind = args
@@ -197,15 +239,24 @@ pub fn needs(
}) })
} }
fn confirmed(args: Option<&serde_json::Map<String, serde_json::Value>>) -> bool {
args.and_then(|a| a.get("confirmed"))
.and_then(serde_json::Value::as_bool)
.unwrap_or(false)
}
/// Whether a token with `scopes` may list `tool` (it can call it in some /// Whether a token with `scopes` may list `tool` (it can call it in some
/// form: a tool that posts when confirmed shows to a books:write token, /// form: a tool that posts when confirmed shows to a books:write token,
/// which may preview). /// which may preview).
pub fn shown(tool: &str, scopes: &std::collections::BTreeSet<Scope>) -> bool { pub fn shown(tool: &str, scopes: &std::collections::BTreeSet<Scope>) -> bool {
match TABLE.iter().find(|(name, _)| *name == tool) { match TABLE.iter().find(|(name, _)| *name == tool) {
Some((_, Always(scope))) => scopes.contains(scope), Some((_, Always(scope))) => scopes.contains(scope),
Some((_, PostWhenConfirmed | PostForApprovalRules)) => { Some((_, PostWhenConfirmed | PostForApprovalRules | ImportSettings)) => {
scopes.contains(&Scope::Write) || scopes.contains(&Scope::Post) scopes.contains(&Scope::Write) || scopes.contains(&Scope::Post)
} }
Some((_, AdminForBackupSettings)) => {
scopes.contains(&Scope::Post) || scopes.contains(&Scope::Admin)
}
None => false, None => false,
} }
} }
@@ -253,6 +304,56 @@ mod tests {
} }
} }
#[test]
fn where_backups_go_is_admins() {
let args = |v: serde_json::Value| v.as_object().cloned();
let set = |keys: &[&str]| {
let changes: Vec<_> = keys
.iter()
.map(|k| json!({"key": k, "value": "x"}))
.collect();
needs(
"books.set_setting",
args(json!({"changes": changes})).as_ref(),
)
};
assert_eq!(set(&["company.name"]), Some(Scope::Post));
assert_eq!(set(&["backup.folder"]), Some(Scope::Admin));
assert_eq!(
set(&["company.name", "backup.git_remote"]),
Some(Scope::Admin)
);
assert_eq!(set(&[" Backup.Folder"]), Some(Scope::Admin));
assert_eq!(needs("books.set_setting", None), Some(Scope::Post));
let import = |text: &str, confirmed: bool| {
needs(
"books.import_settings",
args(json!({"toml_text": text, "confirmed": confirmed})).as_ref(),
)
};
assert_eq!(
import("[company]\nname = \"A\"\n", false),
Some(Scope::Write)
);
assert_eq!(import("[company]\nname = \"A\"\n", true), Some(Scope::Post));
assert_eq!(
import("[backup]\nfolder = \"/x\"\n", false),
Some(Scope::Admin)
);
assert_eq!(
import("[backup]\nfolder = \"/x\"\n", true),
Some(Scope::Admin)
);
// Shown to whoever can set some setting.
let post: std::collections::BTreeSet<Scope> = [Scope::Post].into();
let write: std::collections::BTreeSet<Scope> = [Scope::Write].into();
let admin: std::collections::BTreeSet<Scope> = [Scope::Admin].into();
assert!(shown("books.set_setting", &post));
assert!(shown("books.set_setting", &admin));
assert!(!shown("books.set_setting", &write));
assert!(shown("books.import_settings", &write));
}
#[test] #[test]
fn a_tool_that_takes_confirmed_posts_only_when_confirmed() { fn a_tool_that_takes_confirmed_posts_only_when_confirmed() {
// Exactly the tools whose arguments have `confirmed`. // Exactly the tools whose arguments have `confirmed`.
@@ -263,7 +364,7 @@ mod tests {
.and_then(|p| p.as_object()) .and_then(|p| p.as_object())
.is_some_and(|p| p.contains_key("confirmed")); .is_some_and(|p| p.contains_key("confirmed"));
let (_, needs) = TABLE.iter().find(|(n, _)| *n == tool.name).unwrap(); let (_, needs) = TABLE.iter().find(|(n, _)| *n == tool.name).unwrap();
assert_eq!(takes, *needs == C, "{}", tool.name); assert_eq!(takes, matches!(*needs, C | ImportSettings), "{}", tool.name);
let takes_kind = tool let takes_kind = tool
.input_schema .input_schema
.get("properties") .get("properties")
+99 -28
View File
@@ -78,6 +78,9 @@ pub struct BooksServer {
/// The files the books made for agents, waiting to be fetched (books /// The files the books made for agents, waiting to be fetched (books
/// served from the data root; over stdio a file comes back inline). /// served from the data root; over stdio a file comes back inline).
exports: Option<ledger_store::exports::Exports>, exports: Option<ledger_store::exports::Exports>,
/// The books' backups (books served from the data root; none over
/// stdio).
backups: Option<Arc<crate::backup::Backups>>,
} }
/// Where a book's own files live around its data dir. /// Where a book's own files live around its data dir.
@@ -146,7 +149,8 @@ impl Layout {
/// whether or not anyone still waits for its answer. /// whether or not anyone still waits for its answer.
/// ``` /// ```
pub struct SharedStore { pub struct SharedStore {
store: tokio::sync::Mutex<Store>, /// None once let go (`let_go`): its books were replaced by a restore.
store: tokio::sync::Mutex<Option<Store>>,
stopped: std::sync::atomic::AtomicBool, stopped: std::sync::atomic::AtomicBool,
/// Closing: no new request runs (INVARIANT 3). /// Closing: no new request runs (INVARIANT 3).
closing: std::sync::atomic::AtomicBool, closing: std::sync::atomic::AtomicBool,
@@ -160,6 +164,25 @@ pub struct SharedStore {
} }
impl SharedStore { impl SharedStore {
/// Run `f` on the store under its lock, for a backup (backup.rs):
/// blocking. It reads only, and runs even while the books close (a
/// last backup), never after the server stopped.
pub fn with_store<R>(&self, f: impl FnOnce(&Store) -> R) -> Result<R, String> {
if self.stopped() {
return Err(STOPPED.to_string());
}
let store = self.store.blocking_lock();
let store = store.as_ref().ok_or(STOPPED)?;
Ok(f(store))
}
/// Run `f` on the store if no one holds it now (books just opened):
/// never waits, so never blocks a runtime's thread.
pub fn try_with_store<R>(&self, f: impl FnOnce(&Store) -> R) -> Option<R> {
let store = self.store.try_lock().ok()?;
store.as_ref().map(f)
}
/// An attachment's name, type and bytes (hash-verified), for a /// An attachment's name, type and bytes (hash-verified), for a
/// download: blocking (the store's lock, then the file). /// download: blocking (the store's lock, then the file).
/// None when there is no such attachment. /// None when there is no such attachment.
@@ -168,6 +191,7 @@ impl SharedStore {
return Err(STOPPED.to_string()); return Err(STOPPED.to_string());
} }
let store = self.store.blocking_lock(); let store = self.store.blocking_lock();
let store = store.as_ref().ok_or(STOPPED)?;
let Some(meta) = store let Some(meta) = store
.list_attachments(None) .list_attachments(None)
.map_err(|e| e.to_string())? .map_err(|e| e.to_string())?
@@ -189,7 +213,7 @@ impl SharedStore {
pub fn new(store: Store) -> Arc<Self> { pub fn new(store: Store) -> Arc<Self> {
Arc::new(Self { Arc::new(Self {
currency: store.base_currency(), currency: store.base_currency(),
store: tokio::sync::Mutex::new(store), store: tokio::sync::Mutex::new(Some(store)),
stopped: std::sync::atomic::AtomicBool::new(false), stopped: std::sync::atomic::AtomicBool::new(false),
closing: std::sync::atomic::AtomicBool::new(false), closing: std::sync::atomic::AtomicBool::new(false),
running: std::sync::atomic::AtomicUsize::new(0), running: std::sync::atomic::AtomicUsize::new(0),
@@ -220,6 +244,15 @@ impl SharedStore {
self.stopped.load(std::sync::atomic::Ordering::SeqCst) self.stopped.load(std::sync::atomic::Ordering::SeqCst)
} }
/// Let go of the books for good (a restore replaces them): close as
/// [`Self::close`], stop, and drop the store, so its writer lock is
/// released while sessions still hold this.
pub async fn let_go(&self) {
self.close().await;
self.stop();
self.store.lock().await.take();
}
/// Stop taking requests (INVARIANT 1): after an internal error, or an /// Stop taking requests (INVARIANT 1): after an internal error, or an
/// access record that couldn't be written (access.rs INVARIANT 3). /// access record that couldn't be written (access.rs INVARIANT 3).
pub(crate) fn stop(&self) { pub(crate) fn stop(&self) {
@@ -275,7 +308,8 @@ const VAULT_IN_HISTORY: &str = "the TIN vault can't store anything here yet: the
/// task is suspended. Dropped by a panic, it stops the server /// task is suspended. Dropped by a panic, it stops the server
/// (SharedStore INVARIANT 1). /// (SharedStore INVARIANT 1).
pub struct StoreGuard<'a> { pub struct StoreGuard<'a> {
guard: tokio::sync::MutexGuard<'a, Store>, /// Holds a store (checked when it was taken).
guard: tokio::sync::MutexGuard<'a, Option<Store>>,
shared: &'a SharedStore, shared: &'a SharedStore,
/// The journal's head when the lock was taken. /// The journal's head when the lock was taken.
before: (u64, String), before: (u64, String),
@@ -289,7 +323,10 @@ impl Drop for StoreGuard<'_> {
} }
// A write while it was held: the head after it, for the call's // A write while it was held: the head after it, for the call's
// access record (taken under the same lock, so it is this call's). // access record (taken under the same lock, so it is this call's).
let after = self.guard.head(); let Some(store) = self.guard.as_ref() else {
return;
};
let after = store.head();
if after != self.before { if after != self.before {
WROTE.with(|w| *w.borrow_mut() = Some(after)); WROTE.with(|w| *w.borrow_mut() = Some(after));
} }
@@ -299,13 +336,13 @@ impl Drop for StoreGuard<'_> {
impl std::ops::Deref for StoreGuard<'_> { impl std::ops::Deref for StoreGuard<'_> {
type Target = Store; type Target = Store;
fn deref(&self) -> &Store { fn deref(&self) -> &Store {
&self.guard self.guard.as_ref().expect("a guard holds a store")
} }
} }
impl std::ops::DerefMut for StoreGuard<'_> { impl std::ops::DerefMut for StoreGuard<'_> {
fn deref_mut(&mut self) -> &mut Store { fn deref_mut(&mut self) -> &mut Store {
&mut self.guard self.guard.as_mut().expect("a guard holds a store")
} }
} }
@@ -598,6 +635,7 @@ impl BooksServer {
access: None, access: None,
uploads: None, uploads: None,
exports: None, exports: None,
backups: None,
} }
} }
@@ -613,6 +651,12 @@ impl BooksServer {
self self
} }
/// Back the books up through `backups` (backup.rs).
pub fn with_backups(mut self, backups: Arc<crate::backup::Backups>) -> Self {
self.backups = Some(backups);
self
}
/// Check every call against its caller's token, and record it in /// Check every call against its caller's token, and record it in
/// `log` (access.rs). /// `log` (access.rs).
pub fn with_access(mut self, log: Arc<ledger_store::access::AccessLog>) -> Self { pub fn with_access(mut self, log: Arc<ledger_store::access::AccessLog>) -> Self {
@@ -701,7 +745,11 @@ impl BooksServer {
/// Every tool the server serves. /// Every tool the server serves.
pub(crate) fn tool_router() -> rmcp::handler::server::router::tool::ToolRouter<Self> { pub(crate) fn tool_router() -> rmcp::handler::server::router::tool::ToolRouter<Self> {
Self::core_router() + Self::kept_router() + Self::access_router() + Self::files_router() Self::core_router()
+ Self::kept_router()
+ Self::access_router()
+ Self::files_router()
+ Self::backup_router()
} }
/// The settings in force for this request (reloaded at its start, in /// The settings in force for this request (reloaded at its start, in
@@ -1052,21 +1100,25 @@ impl BooksServer {
return Err(STOPPED.to_string()); return Err(STOPPED.to_string());
} }
let guard = self.store.store.blocking_lock(); let guard = self.store.store.blocking_lock();
// A request that panicked while this one waited stopped it. // A request that panicked while this one waited stopped it; books
// let go hold no store.
if self.store.stopped() { if self.store.stopped() {
return Err(STOPPED.to_string()); return Err(STOPPED.to_string());
} }
let Some(store) = guard.as_ref() else {
return Err(STOPPED.to_string());
};
// Books that keep their own settings: take any change (crate:: // Books that keep their own settings: take any change (crate::
// settings INVARIANT 1). // settings INVARIANT 1).
let timezone = { let timezone = {
let mut live = self.live(); let mut live = self.live();
live.sync(guard.ledger().settings(), self.store.currency.code()); live.sync(store.ledger().settings(), self.store.currency.code());
live.current().timezone.clone() live.current().timezone.clone()
}; };
// "Today" from here on is the zone in force now (a change another // "Today" from here on is the zone in force now (a change another
// session made while this request waited included). // session made while this request waited included).
TIMEZONE.with(|z| *z.borrow_mut() = timezone); TIMEZONE.with(|z| *z.borrow_mut() = timezone);
let before = guard.head(); let before = store.head();
Ok(StoreGuard { Ok(StoreGuard {
guard, guard,
shared: &self.store, shared: &self.store,
@@ -2523,6 +2575,8 @@ fn lines_out(lines: &[ledger_core::StatementLine]) -> Vec<StatementLineOut> {
#[path = "tools/access.rs"] #[path = "tools/access.rs"]
mod access_tools; mod access_tools;
#[path = "tools/backup.rs"]
mod backup_tools;
#[path = "tools/files.rs"] #[path = "tools/files.rs"]
mod files; mod files;
#[path = "tools/kept.rs"] #[path = "tools/kept.rs"]
@@ -6982,7 +7036,7 @@ impl BooksServer {
#[tool( #[tool(
name = "books.purge_tin", name = "books.purge_tin",
title = "Purge a TIN from the vault", title = "Purge a TIN from the vault",
description = "Delete a party's full TIN and W-9 file from the TIN vault (after the retention period, at the payee's request, or to replace a wrong one), with the reason. Without confirmed=true it previews what the vault holds and changes nothing. Confirmed, the records are deleted for good and the books journal it: the party's TIN facts (party.tin_set: purged says when and why) and W-9 facts (party.w9_set: kept_at says so) keep the last four and drop the vault record. Works without the vault key. The next CPA package then lacks that TIN (and says so).", description = "Delete a party's full TIN and W-9 file from the TIN vault (after the retention period, at the payee's request, or to replace a wrong one), with the reason. Without confirmed=true it previews what the vault holds and changes nothing. Confirmed, the records are deleted from the vault (backups made before keep their copies) and the books journal it: the party's TIN facts (party.tin_set: purged says when and why) and W-9 facts (party.w9_set: kept_at says so) keep the last four and drop the vault record. Works without the vault key. The next CPA package then lacks that TIN (and says so).",
annotations( annotations(
read_only_hint = false, read_only_hint = false,
destructive_hint = true, destructive_hint = true,
@@ -7048,7 +7102,7 @@ impl BooksServer {
.or_else(|| party.tin.as_ref().map(|t| t.last4.clone())), .or_else(|| party.tin.as_ref().map(|t| t.last4.clone())),
journaled: false, journaled: false,
note: format!( note: format!(
"Confirming deletes {} for good{}. Make sure the CPA has what they need first.", "Confirming deletes {}{}; backups made before keep their copies. Make sure the CPA has what they need first.",
match (held_tin, held_w9) { match (held_tin, held_w9) {
(true, true) => "the TIN and the W-9 file", (true, true) => "the TIN and the W-9 file",
(true, false) => "the TIN", (true, false) => "the TIN",
@@ -8831,8 +8885,11 @@ impl ServerHandler for BooksServer {
.unwrap_or_default(); .unwrap_or_default();
tools.retain(|t| crate::scopes::shown(&t.name, &scopes)); tools.retain(|t| crate::scopes::shown(&t.name, &scopes));
} else { } else {
// Books served over stdio keep no access log: not offered. // Books served over stdio keep no access log and aren't backed
tools.retain(|t| t.name != "books.access_log"); // up here: not offered.
tools.retain(|t| {
!["books.access_log", "books.backup", "books.back_up_now"].contains(&&*t.name)
});
} }
let hints = context let hints = context
.protocol_version() .protocol_version()
@@ -8852,7 +8909,9 @@ impl ServerHandler for BooksServer {
// the books' own when they keep them (never waiting for the books // the books' own when they keep them (never waiting for the books
// here; a request holding them syncs them when it locks), else the // here; a request holding them syncs them when it locks), else the
// files'. // files'.
if let Ok(store) = self.store.store.try_lock() { if let Ok(store) = self.store.store.try_lock()
&& let Some(store) = store.as_ref()
{
self.live() self.live()
.sync(store.ledger().settings(), self.store.currency.code()); .sync(store.ledger().settings(), self.store.currency.code());
} }
@@ -9373,7 +9432,7 @@ mod tests {
const D: Hints = (Some(false), Some(true), None); const D: Hints = (Some(false), Some(true), None);
const WI: Hints = (Some(false), Some(false), Some(true)); const WI: Hints = (Some(false), Some(false), Some(true));
const DI: Hints = (Some(false), Some(true), Some(true)); const DI: Hints = (Some(false), Some(true), Some(true));
let table: [(&str, Hints); 109] = [ let table: [(&str, Hints); 111] = [
("books.handbook", R), ("books.handbook", R),
("books.start", R), ("books.start", R),
("books.set_setting", DI), ("books.set_setting", DI),
@@ -9483,6 +9542,8 @@ mod tests {
("books.profit_and_loss_by_class", R), ("books.profit_and_loss_by_class", R),
("books.export_cpa_package", W), ("books.export_cpa_package", W),
("books.access_log", R), ("books.access_log", R),
("books.backup", R),
("books.back_up_now", WI),
]; ];
let tools = BooksServer::tool_router().list_all(); let tools = BooksServer::tool_router().list_all();
assert_eq!(tools.len(), table.len()); assert_eq!(tools.len(), table.len());
@@ -9514,11 +9575,12 @@ mod tests {
for word in ["requiresuserinteraction", "requires_user_interaction"] { for word in ["requiresuserinteraction", "requires_user_interaction"] {
assert!(!json.contains(word), "{name} asks for user interaction"); assert!(!json.contains(word), "{name} asks for user interaction");
} }
// The books are a closed world: no tool reaches beyond them. // The books are a closed world: no tool reaches beyond them,
// but the one that pushes a backup to the owner's targets.
assert_eq!( assert_eq!(
a.open_world_hint, a.open_world_hint,
Some(false), Some(*name == "books.back_up_now"),
"{name} has no openWorldHint false" "{name}: openWorldHint"
); );
} }
// docs/mcp-interface.md names the 28 destructive tools. // docs/mcp-interface.md names the 28 destructive tools.
@@ -9581,15 +9643,24 @@ mod tests {
} }
assert!(n > 0, "{krate}: no error templates found"); assert!(n > 0, "{krate}: no error templates found");
} }
// The files tool's refusals (and description): its sentences. // The files and backup tools' refusals (and descriptions), and the
for line in include_str!("tools/files.rs").lines() { // backups' own words: their sentences. (What git itself says of a
let line = line.trim_start(); // failed backup reaches the human as git said it: not ours.)
if line.starts_with("//") { for (name, src) in [
continue; ("tools/files.rs", include_str!("tools/files.rs")),
} ("tools/backup.rs", include_str!("tools/backup.rs")),
for (i, text) in line.split('"').enumerate() { ("backup.rs", include_str!("backup.rs")),
if i % 2 == 1 && text.len() > 30 && text.contains(' ') { ] {
out.push(("tools/files.rs text".into(), text.to_string())); let src = src.split("#[cfg(test)]").next().unwrap_or(src);
for line in src.lines() {
let line = line.trim_start();
if line.starts_with("//") {
continue;
}
for (i, text) in line.split('"').enumerate() {
if i % 2 == 1 && text.len() > 30 && text.contains(' ') {
out.push((format!("{name} text"), text.to_string()));
}
} }
} }
} }
+127
View File
@@ -0,0 +1,127 @@
//! `books.backup` and `books.back_up_now` (plan 3.7, D-30): the books'
//! backups as they stand, and one made now.
//!
//! ```text
//! GOAL: the agent can tell the human whether the books are backed up,
//! and back them up now when asked.
//! SOURCE: plan 3.7 ("books.backup {status|run} reports it": two tools,
//! one read-only), D-30.
//! INVARIANTS:
//! 1. Only books served from the data root are backed up: over stdio
//! both tools are hidden, and refuse.
//! 2. A status reads the backup record only; a run makes and pushes a
//! backup (backup.rs) and answers the status after it.
//! ```
use super::*;
use schemars::JsonSchema;
use serde::{Deserialize, Serialize};
#[derive(Debug, Default, Deserialize, JsonSchema)]
pub struct BackupParams {}
#[derive(Debug, Default, Deserialize, JsonSchema)]
pub struct BackUpNowParams {
/// The backup folder's disk is new (or another): begin these books'
/// backup there again. Only when the human says so: a folder that
/// should hold one is otherwise a disk not connected.
#[serde(default)]
pub new_disk: bool,
}
#[derive(Debug, Serialize, JsonSchema)]
pub struct TargetOut {
/// "repository" (the private repository) or "folder".
pub target: String,
/// Where it is.
pub at: String,
/// When it last took a backup (UTC), if ever.
pub last_ok_at: Option<String>,
/// The journal seq it holds, if any.
pub through_seq: Option<u64>,
/// More than a day without the books' latest: tell the human.
pub stale: bool,
/// Why its last backup failed, if it did.
pub last_error: Option<String>,
/// When its backup was last proven restorable (UTC), if ever: each
/// is cloned and checked weekly.
pub last_drill_at: Option<String>,
/// Why that check failed, if it did: tell the human.
pub drill_error: Option<String>,
/// Not backed up there: these books were restored as a copy of others.
pub paused: bool,
}
#[derive(Debug, Serialize, JsonSchema)]
pub struct BackupOut {
/// The books' journal seq now.
pub head_seq: u64,
/// Each backup the settings name.
pub targets: Vec<TargetOut>,
/// What to do, when none is set up.
pub note: Option<String>,
}
impl From<crate::backup::Status> for BackupOut {
fn from(s: crate::backup::Status) -> Self {
BackupOut {
head_seq: s.head_seq,
note: s.note,
targets: s
.targets
.into_iter()
.map(|t| TargetOut {
target: t.target,
at: t.at,
last_ok_at: t.last_ok_at,
through_seq: t.through_seq,
stale: t.stale,
last_error: t.last_error,
last_drill_at: t.last_drill_at,
drill_error: t.drill_error,
paused: t.paused,
})
.collect(),
}
}
}
impl BooksServer {
fn backups(&self) -> Result<&std::sync::Arc<crate::backup::Backups>, String> {
self.backups.as_ref().ok_or_else(|| {
"these books aren't backed up by this server: only books kept in the books' own private folder are".to_string()
})
}
}
#[tool_router(router = backup_router, vis = "pub(super)")]
impl BooksServer {
#[tool(
name = "books.backup",
title = "Backups",
description = "Whether the books are backed up: each backup target the settings name (backup.git_remote, a private repository; backup.folder, a folder on another disk), when it last took the books, through which journal seq, and why it failed if it did. stale: it lacks the latest, and the first try to back that up was more than a day ago: tell the human. The books back themselves up a little after each change, when they open and before they close; books.back_up_now makes one now.",
annotations(read_only_hint = true, open_world_hint = false)
)]
async fn backup(&self, _params: Parameters<BackupParams>) -> Result<Json<BackupOut>, String> {
Ok(Json(self.backups()?.status().await.into()))
}
#[tool(
name = "books.back_up_now",
title = "Back up now",
description = "Back the books up now to every target the settings name, and answer how each went (as books.backup). new_disk: true only when the human says the backup folder's disk is new: the books' backup is then begun there again. Needs books:admin.",
annotations(
read_only_hint = false,
destructive_hint = false,
idempotent_hint = true,
open_world_hint = true
)
)]
async fn back_up_now(
&self,
params: Parameters<BackUpNowParams>,
) -> Result<Json<BackupOut>, String> {
let b = self.backups()?;
Ok(Json(b.run_with("asked", params.0.new_disk).await?.into()))
}
}
+18
View File
@@ -803,6 +803,24 @@ impl BooksServer {
"An open item is due within 30 days, or overdue: raise it with the human.", "An open item is due within 30 days, or overdue: raise it with the human.",
)); ));
} }
// The books' backups (backup.rs): none set up, or one behind or
// failing.
if let Some(b) = &self.backups {
let st = b.standing(store);
if st.note.is_some() {
next_steps.push(step(
"books.backup",
None,
"No backups are set up: ask the human for a private repository and a folder on another disk, then set backup.git_remote and backup.folder (books.set_setting).",
));
} else if st.needs_attention() {
next_steps.push(step(
"books.backup",
None,
"A backup is behind or failing: books.backup says which and why; tell the human.",
));
}
}
match focus { match focus {
"close" => next_steps.push(step( "close" => next_steps.push(step(
"books.conventions", "books.conventions",
+1 -1
View File
@@ -3383,7 +3383,7 @@ pub struct ReconciliationsOut {
/// Parameters for `books.handbook`. /// Parameters for `books.handbook`.
#[derive(Debug, Default, Deserialize, JsonSchema)] #[derive(Debug, Default, Deserialize, JsonSchema)]
pub struct HandbookParams { pub struct HandbookParams {
/// The section to read: contract, entries, drafts-and-reversals, evidence-and-receipts, attachments, files-in-out, reconciliation, fixed-assets, prepaid-and-recurring, revenue-and-payouts, parties-and-tins, 1099, cpa-handoff, settings-and-conventions, viewer, rules-that-change, control-roles, tracing-figures, or "all" for the whole handbook in order. Omit for the index. /// The section to read: contract, entries, drafts-and-reversals, evidence-and-receipts, attachments, files-in-out, reconciliation, fixed-assets, prepaid-and-recurring, revenue-and-payouts, parties-and-tins, 1099, cpa-handoff, settings-and-conventions, viewer, backups, rules-that-change, control-roles, tracing-figures, or "all" for the whole handbook in order. Omit for the index.
#[serde(default)] #[serde(default)]
pub section: Option<String>, pub section: Option<String>,
} }
File diff suppressed because it is too large. Load diff
+1
View File
@@ -20,6 +20,7 @@
pub mod access; pub mod access;
pub mod auth; pub mod auth;
pub mod backup;
pub mod blob; pub mod blob;
pub mod chain; pub mod chain;
pub mod error; pub mod error;
+279
View File
@@ -32,6 +32,10 @@
//! disk, and its head equal to the source's. //! disk, and its head equal to the source's.
//! 6. No error names the root or a path inside it: nothing an agent is //! 6. No error names the root or a path inside it: nothing an agent is
//! shown names the books' folder. //! shown names the books' folder.
//! 7. Restoring a backup (plan 3.7) is proven before use, as an adoption
//! is (INVARIANT 5), and its head must be the one the preview read;
//! it never overwrites: books it replaces are moved whole into
//! `legacy/`, only when asked, and only while nothing holds them.
//! ``` //! ```
use crate::error::StoreError; use crate::error::StoreError;
@@ -70,6 +74,12 @@ pub const LEGACY_KEY_FILE: &str = "legacy.key";
const BOOKS_LOCK: &str = "books.lock"; const BOOKS_LOCK: &str = "books.lock";
/// Staging folders under `books/` start so; none is a valid book id. /// Staging folders under `books/` start so; none is a valid book id.
const STAGING_PREFIX: &str = ".adopt-"; const STAGING_PREFIX: &str = ".adopt-";
/// A backup being restored, in `books/` (never a book: ids don't start
/// with a dot).
const RESTORE_PREFIX: &str = ".restore-";
/// Left in a restored book's `ops/`: the id of the books its backup held
/// (empty when it named none), for the server to note when it opens them.
pub const RESTORED_FROM: &str = "restored-from";
/// How the root is named in messages (INVARIANT 6). /// How the root is named in messages (INVARIANT 6).
pub const THE_ROOT: &str = "the books' private folder"; pub const THE_ROOT: &str = "the books' private folder";
@@ -802,7 +812,276 @@ impl Root {
} }
} }
/// What restoring a backup would bring in ([`Root::restore_preview`]).
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RestorePreview {
/// The backup's head, `(seq, hash)`, proven.
pub head: (u64, String),
/// Stored files in it, each re-hashed.
pub blobs: u64,
/// Vault records in it.
pub vault_files: u64,
/// Whether books of the id asked for exist now.
pub exists: bool,
/// The id of the books it holds, as they were backed up.
pub origin: Option<String>,
}
/// What [`Root::restore`] did.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RestoreReport {
/// The restored book's id.
pub id: BookId,
/// Its head, `(seq, hash)`: the backup's.
pub head: (u64, String),
/// Stored files the books name, re-hashed through the store.
pub blobs_verified: u64,
/// Vault records restored.
pub vault_files: u64,
/// The `legacy/` folder the books it replaced were moved to.
pub replaced: Option<String>,
/// The id of the books it held, as they were backed up.
pub origin: Option<String>,
}
impl Root { impl Root {
/// Clone the backup at `source` (a git repository's address, or its
/// folder: `<book>.git` in a backup folder) into a scratch folder in `books/` and prove it
/// whole (backup.rs INVARIANT 3); the caller holds the books lock and
/// removes the folder.
fn fetch_backup(
&self,
git: &crate::backup::Git,
source: &str,
books: &Path,
id: &BookId,
) -> Result<(PathBuf, crate::backup::Verified), RootError> {
// Anything left from one that never finished goes first: none is
// in progress (each holds the books lock).
for entry in std::fs::read_dir(books)? {
let entry = entry?;
if entry
.file_name()
.to_str()
.is_some_and(|n| n.starts_with(RESTORE_PREFIX))
{
std::fs::remove_dir_all(entry.path())?;
}
}
let scratch = books.join(format!("{RESTORE_PREFIX}{id}-{}", std::process::id()));
match crate::backup::verify(git, source, &scratch) {
Ok(v) => Ok((scratch, v)),
Err(e) => {
let _ = std::fs::remove_dir_all(&scratch);
Err(RootError::Adopt(format!(
"that backup can't be restored: {e}"
)))
}
}
}
/// What restoring the backup at `source` as books `id` would bring in,
/// proven as [`Self::restore`] proves it, adding nothing to the root.
pub fn restore_preview(
&self,
git: &crate::backup::Git,
source: &str,
id: &BookId,
) -> Result<RestorePreview, RootError> {
(|| {
let _lock = self.books_lock()?;
let (books, _) = self.private_subdir(BOOKS_DIR, false)?;
let (scratch, v) = self.fetch_backup(git, source, &books, id)?;
let vault = scratch.join(VAULT_DIR);
let vault_files = if vault.is_dir() {
std::fs::read_dir(&vault)
.map(|d| d.count() as u64)
.unwrap_or(0)
} else {
0
};
let _ = std::fs::remove_dir_all(&scratch);
Ok(RestorePreview {
head: v.head,
blobs: v.blobs,
vault_files,
exists: std::fs::symlink_metadata(books.join(id.as_str())).is_ok(),
origin: v.book,
})
})()
.map_err(|e| self.hide(e))
}
/// Restore the backup at `source` as books `id` (INVARIANT 7): proven
/// whole, its head `expect_head` (as the preview read it), built as an
/// adoption builds (every stored file re-hashed, opened, verified,
/// synced) with its backup history (so its next backups carry on from
/// it), and only then renamed into place. Books `id` that exist are
/// refused, unless `replace`: then they are moved into `legacy/`
/// first, kept whole; refused while anything holds them open.
pub fn restore(
&self,
git: &crate::backup::Git,
source: &str,
id: &BookId,
expect_head: &(u64, String),
replace: bool,
) -> Result<RestoreReport, RootError> {
self.restore_inner(git, source, id, expect_head, replace)
.map_err(|e| match e {
// The copy's checks are the adoption's: said as a restore.
RootError::Adopt(text) => {
RootError::Adopt(text.replace("nothing was adopted", "nothing was restored"))
}
other => other,
})
.map_err(|e| self.hide(e))
}
fn restore_inner(
&self,
git: &crate::backup::Git,
source: &str,
id: &BookId,
expect_head: &(u64, String),
replace: bool,
) -> Result<RestoreReport, RootError> {
let _lock = self.books_lock()?;
let (books, _) = self.private_subdir(BOOKS_DIR, false)?;
let dest = books.join(id.as_str());
let exists = std::fs::symlink_metadata(&dest).is_ok();
if exists && !replace {
return Err(RootError::Exists(id.to_string()));
}
if !exists && replace {
return Err(RootError::Adopt(format!(
"there are no books called {:?} to replace: nothing was restored",
id.as_str()
)));
}
// Books being replaced are moved only while nothing holds them.
let _held = if exists {
Some(lock_source(&dest).map_err(|_| {
RootError::Adopt(format!(
"the books called {:?} are open: close them first; nothing was restored",
id.as_str()
))
})?)
} else {
None
};
let (scratch, v) = self.fetch_backup(git, source, &books, id)?;
let staging = books.join(format!("{RESTORE_PREFIX}{id}-{}-book", std::process::id()));
let built = (|| -> Result<RestoreReport, RootError> {
if v.head != *expect_head {
return Err(RootError::Adopt(format!(
"that backup isn't the one previewed (it is now at event {}): nothing was restored; preview it again",
v.head.0
)));
}
let (raw, lines) = read_source_journal(&scratch)?;
if head_of(&lines) != v.head {
return Err(RootError::Adopt(
"the backup's journal changed while it was read: nothing was restored".into(),
));
}
make_private_dir(&staging)?;
write_private_file(&staging.join(JOURNAL_FILE), &raw)?;
let blob_dir = crate::store::BLOBS_DIR;
let copied = copy_blobs(&scratch.join(blob_dir), &staging.join(blob_dir))?;
let vault_files = if scratch.join(VAULT_DIR).is_dir() {
copy_flat_private(&scratch.join(VAULT_DIR), &staging.join(VAULT_DIR))?
} else {
0
};
if scratch.join(KEYS_DIR).is_dir() {
copy_flat_private(&scratch.join(KEYS_DIR), &staging.join(KEYS_DIR))?;
}
let access = scratch.join(crate::access::ACCESS_FILE);
if access.is_file() {
write_private_file(
&staging.join(crate::access::ACCESS_FILE),
&std::fs::read(&access)?,
)?;
}
// Its backup history: the next backup carries on from it.
std::fs::rename(
scratch.join(".git"),
staging.join(crate::backup::ARCHIVE_DIR),
)?;
// Whose books they were, placed with them.
crate::ops::prepare(&staging)?;
write_private_file(
&crate::ops::ops_dir(&staging).join(RESTORED_FROM),
v.book.as_deref().unwrap_or("").as_bytes(),
)?;
let mut store = Store::open(&staging)?;
if store.head() != v.head {
return Err(RootError::Adopt(format!(
"the copy's head {:?} isn't the backup's {:?}: nothing was restored",
store.head(),
v.head
)));
}
let named: BTreeSet<String> = store.stored_blob_shas()?.into_iter().collect();
if !named.is_subset(&copied.shas) {
return Err(RootError::Adopt(
"the backup lacks stored files its books name: nothing was restored".into(),
));
}
store.verify_and_heal()?;
let blobs_verified = store.verify_blobs()?;
let head = store.head();
drop(store);
sync_dir(&staging)?;
Ok(RestoreReport {
id: id.clone(),
head,
blobs_verified,
vault_files,
replaced: None,
origin: v.book.clone(),
})
})();
let _ = std::fs::remove_dir_all(&scratch);
let placed = built.and_then(|mut report| {
if exists {
let (legacy, _) = self.private_subdir(LEGACY_DIR, false)?;
let stamp = chrono::Utc::now().format("%Y%m%d-%H%M%S");
let mut name = format!("{id}-{stamp}");
let mut n = 1;
while std::fs::symlink_metadata(legacy.join(&name)).is_ok() {
n += 1;
name = format!("{id}-{stamp}-{n}");
}
std::fs::rename(&dest, legacy.join(&name))?;
if let Err(e) = std::fs::rename(&staging, &dest) {
// The books it would replace go back where they were.
if std::fs::rename(legacy.join(&name), &dest).is_err() {
return Err(RootError::Adopt(format!(
"the backup couldn't be put in place ({}), and the books it would \
replace are in the legacy folder as {name}: nothing was restored",
e.kind()
)));
}
return Err(e.into());
}
// In place: a folder that won't sync now is said no more
// than any later write's would be.
let _ = sync_dir(&legacy);
report.replaced = Some(name);
} else {
std::fs::rename(&staging, &dest)?;
}
let _ = sync_dir(&books);
Ok(report)
});
if placed.is_err() {
let _ = std::fs::remove_dir_all(&staging);
}
placed
}
/// The source of an adoption, made absolute: books outside the root /// The source of an adoption, made absolute: books outside the root
/// (and not holding it) with a journal. /// (and not holding it) with a journal.
fn adoptable(&self, source: &Path) -> Result<PathBuf, RootError> { fn adoptable(&self, source: &Path) -> Result<PathBuf, RootError> {
+64
View File
@@ -1284,3 +1284,67 @@ fn books_that_changed_since_the_preview_are_not_adopted() {
.unwrap(); .unwrap();
assert_eq!(report.source_head, now); assert_eq!(report.source_head, now);
} }
/// A backup of `store`'s books (in `dir`) in a folder target under
/// `home`: its address.
fn backed_up(home: &Path, dir: &Path, store: &Store) -> Option<String> {
use ledger_store::backup::{self, Git, Target};
let git = Git::find().ok()?;
backup::commit(&git, dir, "acme", store.head()).unwrap();
let folder = home.join("disk");
std::fs::create_dir_all(&folder).unwrap();
let target = Target::Folder(folder);
backup::push(&git, dir, "acme", &target, true).unwrap();
Some(target.source("acme"))
}
#[test]
fn a_restore_never_moves_books_something_holds() {
use ledger_store::backup::Git;
let home = tmp("restore-held");
let (root, _) = Root::setup(home.path(), None, None).unwrap();
let acme = id("acme");
let held = root.create_book(&acme, Currency::USD).unwrap();
let dir = root.book_dir(&acme);
let source = backed_up(home.path(), &dir, &held).expect("git is installed");
let git = Git::find().unwrap();
let preview = root.restore_preview(&git, &source, &acme).unwrap();
assert!(preview.exists);
// Open (its writer lock held): refused, nothing moved.
let e = root
.restore(&git, &source, &acme, &preview.head, true)
.unwrap_err();
assert!(e.to_string().contains("are open"), "{e}");
assert!(dir.join(JOURNAL_FILE).is_file());
assert!(
!root.path().join("legacy").exists()
|| std::fs::read_dir(root.path().join("legacy"))
.unwrap()
.count()
== 0
);
// Not over books without --replace; nothing to replace that isn't.
let e = root
.restore(&git, &source, &acme, &preview.head, false)
.unwrap_err();
assert!(matches!(e, RootError::Exists(_)), "{e}");
let e = root
.restore(&git, &source, &id("nobody"), &preview.head, true)
.unwrap_err();
assert!(e.to_string().contains("no books called"), "{e}");
// Let go: replaced, the old kept whole.
drop(held);
let r = root
.restore(&git, &source, &acme, &preview.head, true)
.unwrap();
let kept = root.path().join("legacy").join(r.replaced.unwrap());
assert!(kept.join(JOURNAL_FILE).is_file());
assert_eq!(r.head, preview.head);
// Nothing of the restore is left behind in books/.
let left: Vec<_> = std::fs::read_dir(root.path().join("books"))
.unwrap()
.map(|e| e.unwrap().file_name().to_string_lossy().into_owned())
.filter(|n| n.starts_with('.'))
.collect();
assert!(left.is_empty(), "{left:?}");
}
+75 -1
View File
@@ -352,7 +352,8 @@ seals a new vault record, every time, so none carries it;
`idempotent_tools_change_nothing_when_repeated` calls each candidate twice `idempotent_tools_change_nothing_when_repeated` calls each candidate twice
to check). Every tool's to check). Every tool's
`openWorldHint` is false: the books are a closed world, and no tool `openWorldHint` is false: the books are a closed world, and no tool
reaches beyond them. `books.export_cpa_package`, `books.vendor_1099` reaches beyond them, but `books.back_up_now`, which pushes the books to
the owner's backups (true). `books.export_cpa_package`, `books.vendor_1099`
and `books.get_w9` keep a file for the agent (an export) and record the and `books.get_w9` keep a file for the agent (an export) and record the
ones with TINs, so they are not read-only. No tool carries ones with TINs, so they are not read-only. No tool carries
`requiresUserInteraction` (owner decision `requiresUserInteraction` (owner decision
@@ -584,6 +585,79 @@ through the policy.
every stored file. The server runs the same check, and heals the same every stored file. The server runs the same check, and heals the same
way, every time it starts. way, every time it starts.
## Tools — backups
Books served from the data root back themselves up (plan 3.7, D-30,
D-31): each keeps `archive.git`, a bare repository whose commits are its
journal, stored files, vault, keys and access log as they were (one
commit per backup, made under the store's lock, its message
`seq N head H` and, below, `book <id>`), pushed fast-forward only to the targets the settings
name: `backup.git_remote` (the owner's private repository) and
`backup.folder` (a repository `<book>.git` in a folder on another disk).
Not encrypted: the destinations keep it private. Backups run a little
after the books are last written (`--backup-quiet`, 30 s), when they
open, before they close, and on request; a failed one is retried after
15 minutes. Each target is drilled weekly: cloned, its chain replayed,
every stored file re-hashed, every TIN record the books name there with
its key, its head the one its commit records, its journal a byte prefix
of the live one (a failed drill is tried again at the server's next
hourly look, once 15 minutes have passed). A backup's tree is checked
before any of it is written out: plain files under the backed-up paths
only, never a link or a submodule. git runs with no hooks (in a
repository on this computer it pushes to as well), no ignore or
attributes rules of anyone's (a `.gitattributes` in a book's folder
refuses the backup) and no command-running transports; a repository on
this computer is used only if plain (a bare repository at that very
path, no settings of its own but where a copy came from, nothing sending
git elsewhere, no link where it or its parts are), and a backup folder that took backups and is gone is
reported as a disk not connected, never made again (restored books count
the backups of the books they were). Books restored beside the books they
were (under another id) don't push to those books' repository, which
would stop theirs: `books.backup` says it is paused for them until
another is named. A damaged backup record is set aside and begun again;
it never keeps the books shut. One book's drill holds the others'
backups until it ends (a backup at close waits at most a minute for
it). A target
whose push failed is left out of the backups after writes for 15
minutes, then tried again, written to or not; the other targets don't
wait for it. Every
commit, push and drill is a record in the book's `ops/backup.jsonl` (a
chain log; a backup never moves the books' head), each target's records
keyed by the target itself. Setting `backup.*` (books.set_setting, or
books.import_settings with a `[backup]` table) needs books:admin, as
backing up does: a backup is the whole book, its vault and key included.
Over stdio neither tool is offered.
- `books.backup {}` (books:read) → `{ head_seq, targets: [{ target:
"repository" | "folder", at, last_ok_at, through_seq, stale,
last_error, last_drill_at, drill_error, paused }], note }`. `paused`:
not backed up there, these books being restored as a copy of others
(its `last_error` says so). `stale`: the target
lacks the head, and the first try to back up what it lacks (since its
last good backup, or ever) is more than a day old. `last_error`: its
last push's, or a later failed commit's. `note`
says what to set when no target is. `books.start` adds a next step when
none is set up, or one is stale, failing, paused or failed its drill.
- `books.back_up_now { new_disk? }` (books:admin) → the same, after
backing up to every target now. `new_disk`: the backup folder's disk is
new, so a folder that took these books' backups and holds none now is
begun again (otherwise it is a disk not connected). The one tool whose
`openWorldHint` is true. A target paused for books restored as a copy
of others shows `paused` (and is no failure to retry).
Restoring is the owner's, from a terminal: `wanderbooks books restore
BACKUP [--id ID] [--replace] [--yes]` (the admin socket's `restore`,
previewed then confirmed against the previewed head; without `--id`, the
books keep the id of the books the backup holds, which the preview names,
else the repository's name; under another id they are a copy, and don't
push to those books' repository). The backup is
cloned into the root, proven whole as a drill proves it (but for the
live-journal check: there may be no live books), and built as an
adoption builds; it arrives as new books, or replaces books (`--replace`,
which are let go and moved whole into `legacy/<id>-<time>/`). The
restored books keep the backup's history, so their next backups carry
on from it.
## Tools — payouts ## Tools — payouts
- `books.record_payout { date, description, deposit_account, deposit, - `books.record_payout { date, description, deposit_account, deposit,
+2 -1
View File
@@ -273,7 +273,8 @@ damn CPA package."
takes a path. The MCP server takes a path. The MCP server
never returns a TIN in any other tool, log, or error. Tests check that never returns a TIN in any other tool, log, or error. Tests check that
no other output contains a nine-digit run. no other output contains a nine-digit run.
- **Books without a vault** (another machine, a restored backup) still - **Books without a vault** (another machine, books adopted without
theirs; a restored backup brings its vault and key, D-31) still
open and verify; the 1099 worksheet and the CPA package then say which open and verify; the 1099 worksheet and the CPA package then say which
TINs are missing (`tin_gaps`), and the package is still built. TINs are missing (`tin_gaps`), and the package is still built.
- **Tools** (built; `docs/mcp-interface.md` "Tools — the TIN vault"): - **Tools** (built; `docs/mcp-interface.md` "Tools — the TIN vault"):
+10 -6
View File
@@ -27,10 +27,13 @@ easily backupable, journable":
bytes to `journal.jsonl.torn.<time>`, truncates to the last newline, and bytes to `journal.jsonl.torn.<time>`, truncates to the last newline, and
journals a `journal.tail_repaired` line; `books.digest` reports it. journals a `journal.tail_repaired` line; `books.digest` reports it.
Damage before the last newline is corruption and refuses to open: that Damage before the last newline is corruption and refuses to open: that
is for a human, never repaired automatically. Better: the journal is is for a human, never repaired automatically. The server does better
git-friendly, so an optional `wanderbooks backup` that commits it to a (plan 3.7): it commits the journal, the blobs, the vault, its key and
private git repo gives versioned, distributed, deduplicated backup with the access log to each book's own git repository (the blobs staged
full history. ahead, the commit under the store's lock) and pushes it to the
owner's private repository and backup folder, versioned and
deduplicated, with full history (docs/mcp-interface.md, "Tools —
backups").
- **Journable.** It literally is a journal: `{seq, ts, type, payload, prev}`. - **Journable.** It literally is a journal: `{seq, ts, type, payload, prev}`.
Greppable, diffable, replayable. The audit trail isn't a feature bolted Greppable, diffable, replayable. The audit trail isn't a feature bolted
on — it's the storage format. on — it's the storage format.
@@ -76,8 +79,9 @@ is correct: the journal records what happened, including abandoned work.
`wanderbooks verify` (a `ledger-store` API, later a CLI subcommand): `wanderbooks verify` (a `ledger-store` API, later a CLI subcommand):
re-read the journal, recompute the hash chain, replay every line into a fresh re-read the journal, recompute the hash chain, replay every line into a fresh
in-memory `Ledger`, and compare its reports against the SQLite projection's. in-memory `Ledger`, and compare its reports against the SQLite projection's.
Any mismatch is a loud failure. Run it on startup (cheap at this scale) and Any mismatch is a loud failure. Run it on startup (cheap at this scale);
before backups. a backup is then proven where it lands, by a drill (docs/mcp-interface.md,
"Tools — backups").
## Blob store ## Blob store
+2 -2
View File
@@ -72,8 +72,8 @@ that would fail if it broke:
| Text filed into the books is refused when it holds a TIN the vault keeps (nine digits, 3-2-4 by hyphen, space or dot, 2-7 by hyphen or space, anywhere) or a stand-alone printed SSN or EIN (123-45-6789, 12-3456789), and only then: a bank export (routing numbers, long references, a date next to an amount) and a longer number holding an SSN shape go in; a locked vault leaves the printed shapes (rules.md B-26) | `tin_files.rs` (tests) `text_with_a_vault_tin_or_a_printed_one_is_refused_and_a_bank_export_is_not`; unit tests `text_holding_a_vault_tin_is_refused_in_every_rendering`, `a_printed_ssn_or_ein_is_refused`, `a_longer_number_holding_a_tin_shape_passes`, `an_ordinary_bank_export_passes`, `binary_bytes_are_not_scanned` | | Text filed into the books is refused when it holds a TIN the vault keeps (nine digits, 3-2-4 by hyphen, space or dot, 2-7 by hyphen or space, anywhere) or a stand-alone printed SSN or EIN (123-45-6789, 12-3456789), and only then: a bank export (routing numbers, long references, a date next to an amount) and a longer number holding an SSN shape go in; a locked vault leaves the printed shapes (rules.md B-26) | `tin_files.rs` (tests) `text_with_a_vault_tin_or_a_printed_one_is_refused_and_a_bank_export_is_not`; unit tests `text_holding_a_vault_tin_is_refused_in_every_rendering`, `a_printed_ssn_or_ein_is_refused`, `a_longer_number_holding_a_tin_shape_passes`, `an_ordinary_bank_export_passes`, `binary_bytes_are_not_scanned` |
| Filing is no TIN oracle: text with more than two TIN-shaped numbers ending in a known TIN's last four (the books' TIN and W-9 facts, then the vault's own) is refused before the vault is opened, byte for byte alike whether one is the vault's TIN or none is, the vault open or locked, by every filing tool; one or two such numbers are compared against the vault, one number written many ways counts once, and other last fours never count (rules.md B-26) | `tin_files.rs` (tests) `text_probing_for_a_tin_is_refused_alike_by_every_filing_tool`; unit tests `text_probing_for_a_tin_is_refused_alike_whether_or_not_it_holds_it` (the vault never opened for a probe), `one_or_two_candidates_are_compared_against_the_vault` | | Filing is no TIN oracle: text with more than two TIN-shaped numbers ending in a known TIN's last four (the books' TIN and W-9 facts, then the vault's own) is refused before the vault is opened, byte for byte alike whether one is the vault's TIN or none is, the vault open or locked, by every filing tool; one or two such numbers are compared against the vault, one number written many ways counts once, and other last fours never count (rules.md B-26) | `tin_files.rs` (tests) `text_probing_for_a_tin_is_refused_alike_by_every_filing_tool`; unit tests `text_probing_for_a_tin_is_refused_alike_whether_or_not_it_holds_it` (the vault never opened for a probe), `one_or_two_candidates_are_compared_against_the_vault` |
| The connect text fits its budgets, its lead stands alone, it states the receipt rule enforced (exact threshold in the base currency, required and exempt accounts, reversals) and the parties rule in force, and the company name can't break it (control and format characters) | `handbook.rs` `instructions_budget`, `a_typical_company_gets_the_whole_connect_text`, `the_company_name_is_one_plain_line`; `receipt_policy.rs` `the_rule_told_is_the_rule_enforced`, `a_long_or_odd_account_list_stays_short_and_plain`; a new session is told the settings in force: `a_new_session_is_told_the_settings_in_force` | | The connect text fits its budgets, its lead stands alone, it states the receipt rule enforced (exact threshold in the base currency, required and exempt accounts, reversals) and the parties rule in force, and the company name can't break it (control and format characters) | `handbook.rs` `instructions_budget`, `a_typical_company_gets_the_whole_connect_text`, `the_company_name_is_one_plain_line`; `receipt_policy.rs` `the_rule_told_is_the_rule_enforced`, `a_long_or_odd_account_list_stays_short_and_plain`; a new session is told the settings in force: `a_new_session_is_told_the_settings_in_force` |
| No text the agent is given (whole tool JSON, resources, connect text, handbook, every error template and refusal) points at files, git, GitHub, repo docs, flags or helper commands; descriptions fit 700 characters (listed exceptions, P5) | `tool_text_lint`, `tool_descriptions_fit_700_characters`; `ledger-core` `an_unknown_year_threshold_is_never_guessed` | | No text the agent is given (whole tool JSON, resources, connect text, handbook, every error template and refusal) points at files, git, GitHub, repo docs, flags or helper commands; descriptions fit 700 characters (listed exceptions, P5; and what git itself said of a failed backup, which `books.backup` passes on to the human as git said it, its paths hidden) | `tool_text_lint`, `tool_descriptions_fit_700_characters`; `ledger-core` `an_unknown_year_threshold_is_never_guessed` |
| Every tool is annotated (openWorldHint false on all), and the read-only hint is true in behaviour (journal, every data-dir file's contents but `ops/`, the viewer's side channel for `books.show`, which is not the books; the database's rows, the vault and the settings unchanged) | `annotations_complete`; `spec_mcp.rs` `read_only_tools_change_nothing` (checked on every call in `Mcp::tool`) | | Every tool is annotated (openWorldHint false on all but `books.back_up_now`), and the read-only hint is true in behaviour (journal, every data-dir file's contents but `ops/`, the viewer's side channel for `books.show`, which is not the books; the database's rows, the vault and the settings unchanged) | `annotations_complete`; `spec_mcp.rs` `read_only_tools_change_nothing` (checked on every call in `Mcp::tool`) |
| A bare (or empty-`since`) digest covers this month so far, on any day of it, as the server counts it (the stdio test reads the server's own month, and starts over if it turns) | `a_bare_digest_covers_this_month_so_far`; `mcp_stdio.rs` `digest_resource_bare_reads` | | A bare (or empty-`since`) digest covers this month so far, on any day of it, as the server counts it (the stdio test reads the server's own month, and starts over if it turns) | `a_bare_digest_covers_this_month_so_far`; `mcp_stdio.rs` `digest_resource_bare_reads` |
| The handbook: every section, "all" in order, the schema names every id | `handbook_sections_render_the_rule_in_force`, `the_whole_handbook_is_every_section_in_order`, `mcp_stdio.rs` `handbook_tool_and_resources_over_stdio` | | The handbook: every section, "all" in order, the schema names every id | `handbook_sections_render_the_rule_in_force`, `the_whole_handbook_is_every_section_in_order`, `mcp_stdio.rs` `handbook_tool_and_resources_over_stdio` |
+30 -21
View File
@@ -66,8 +66,8 @@ Paths are relative to `/home/swaits/Code/wanderbooks/crates/` unless they are ab
| I-6 | **A confirmed commit is bound to the server's most recent preview of that exact draft** (stale or altered drafts are refused). It does not prove that a human saw the preview, and locally the key is readable by any process running as the human (advisory). | `preview_token` = HMAC-SHA256 over (entry id, canonical draft state, settings seq, evidence-link set, policy result), keyed with a server secret held in `keys/`. A post with a stale token is refused with guard `stale_preview`. | | I-6 | **A confirmed commit is bound to the server's most recent preview of that exact draft** (stale or altered drafts are refused). It does not prove that a human saw the preview, and locally the key is readable by any process running as the human (advisory). | `preview_token` = HMAC-SHA256 over (entry id, canonical draft state, settings seq, evidence-link set, policy result), keyed with a server secret held in `keys/`. A post with a stale token is refused with guard `stale_preview`. |
| I-7 | **The agent never holds a full TIN or vault key material.** | Vault records are sealed with HPKE to a public key whose sha256 is journaled. The private key exists only wrapped in keyslots held by the customer. Unwrap code is reachable only from human-page handlers and the GUI unlock path (module-privacy test). A pattern-based TIN scanner fails closed on every tool result. A canary TIN/key fuzz test covers responses, errors, logs and the access log. | | I-7 | **The agent never holds a full TIN or vault key material.** | Vault records are sealed with HPKE to a public key whose sha256 is journaled. The private key exists only wrapped in keyslots held by the customer. Unwrap code is reachable only from human-page handlers and the GUI unlock path (module-privacy test). A pattern-based TIN scanner fails closed on every tool result. A canary TIN/key fuzz test covers responses, errors, logs and the access log. |
| I-8 | **The GUI never mutates and never uses the MCP.** Locally it reads the data root directly, read-only, as today. It finds the root from the platform's well-known location or from `wanderbooks gui <book>`, never from a project directory or `wanderbooks.toml`. `books.show` reaches it through `ops/show.jsonl` in the book's own data root, never through the MCP. Remote books (self-host) are viewed through a read-only replica stream (P9). | `ledger-gui/tests/read_only.rs`. Tests `gui_works_under_read_only_landlock` and `gui_finds_root_without_project_dir`. Test `replica_snapshot_equals_store_snapshot` (P9). The GUI writes nothing under the data root and leaves no files under `$HOME` except eframe preferences. | | I-8 | **The GUI never mutates and never uses the MCP.** Locally it reads the data root directly, read-only, as today. It finds the root from the platform's well-known location or from `wanderbooks gui <book>`, never from a project directory or `wanderbooks.toml`. `books.show` reaches it through `ops/show.jsonl` in the book's own data root, never through the MCP. Remote books (self-host) are viewed through a read-only replica stream (P9). | `ledger-gui/tests/read_only.rs`. Tests `gui_works_under_read_only_landlock` and `gui_finds_root_without_project_dir`. Test `replica_snapshot_equals_store_snapshot` (P9). The GUI writes nothing under the data root and leaves no files under `$HOME` except eframe preferences. |
| I-9 | **Every access through the server is logged, and logging fails closed.** | A hash-chained `access.jsonl` (own genesis). Property: exactly one record per tool call, upload, download, human action, remote replica connection and backup, refusals included. The local viewer's direct reads are not logged, because the GUI never writes; the docs say so. | | I-9 | **Every access through the server is logged, and logging fails closed.** | A hash-chained `access.jsonl` (own genesis). Property: exactly one record per tool call, upload, download, human action, remote replica connection and backup, refusals included (as built, slice 7: a backup's own runs are records of `ops/backup.jsonl`; `books.back_up_now`, a tool call, is in the access log). The local viewer's direct reads are not logged, because the GUI never writes; the docs say so. |
| I-10 | **Tamper evidence.** Every mutating result carries `head {seq, hash}`. The head is handed off when a session goes quiet. The customer can clone the archive and run `wanderbooks verify-archive --expect-head`. Backup and access records stay out of the journal, so they never advance the head they report. | Test `backup_does_not_advance_head`. Test `clone_verify_expect_head`. | | I-10 | **Tamper evidence.** Every mutating result carries `head {seq, hash}`. The head is handed off when a session goes quiet. The customer can clone the archive and run `wanderbooks verify-archive --expect-head`. Backup and access records stay out of the journal, so they never advance the head they report. (As built, slice 7, D-31: each backup commit records its head; a restore's preview proves a clone and names its head, and weekly drills prove each target; no separate `verify-archive`.) | Test `backup_does_not_advance_head`. Test `clone_verify_expect_head` (as built: `a_restore_is_of_the_backup_previewed_only`, the drills). |
| I-11 | **No loss of ease of use.** Local setup is at most 3 commands on a clean machine, with no sudo, no system account and no container, beyond installing the app itself (D-24: on Linux the package manager asks for `sudo` as it does for every package; the dmg, the Windows installer and the AppImage ask for nothing). There is nothing to do per session, it works from any directory, the daily flow adds no client permission prompts (no `requiresUserInteraction`), and the viewer updates within 1 s of a post. | Acceptance criteria in P2 and P5. Tests `setup_needs_no_root` and `annotations_complete`. The agent-eval harness. | | I-11 | **No loss of ease of use.** Local setup is at most 3 commands on a clean machine, with no sudo, no system account and no container, beyond installing the app itself (D-24: on Linux the package manager asks for `sudo` as it does for every package; the dmg, the Windows installer and the AppImage ask for nothing). There is nothing to do per session, it works from any directory, the daily flow adds no client permission prompts (no `requiresUserInteraction`), and the viewer updates within 1 s of a post. | Acceptance criteria in P2 and P5. Tests `setup_needs_no_root` and `annotations_complete`. The agent-eval harness. |
| I-12 | **Guidance survives client truncation, and never names a path.** Connect text is at most 1,400 characters, and its first 512 stand alone and name `books.start`. Tool descriptions are at most 700 characters. No tool text mentions `just`, `wanderbooks.toml`, `~/`, git, GitHub PRs or any filesystem path, the data root included, and no result names one. | Tests `instructions_budget`, `tool_text_lint` and `no_result_names_a_path`. | | I-12 | **Guidance survives client truncation, and never names a path.** Connect text is at most 1,400 characters, and its first 512 stand alone and name `books.start`. Tool descriptions are at most 700 characters. No tool text mentions `just`, `wanderbooks.toml`, `~/`, git, GitHub PRs or any filesystem path, the data root included, and no result names one. | Tests `instructions_budget`, `tool_text_lint` and `no_result_names_a_path`. |
| I-13 | **Formal-methods bar.** Kani harnesses cover kernel arithmetic. Every new ledger command is a Lean `Step` with a preservation proof. A failing proof is a finding. All tools are pinned in `mise.toml`. Clippy reports zero warnings. Every mutation survivor is killed or documented as equivalent. | `just check` and `just prove` pass in every phase. | | I-13 | **Formal-methods bar.** Kani harnesses cover kernel arithmetic. Every new ledger command is a Lean `Step` with a preservation proof. A failing proof is a finding. All tools are pinned in `mise.toml`. Clippy reports zero warnings. Every mutation survivor is killed or documented as equivalent. | `just check` and `just prove` pass in every phase. |
@@ -92,7 +92,7 @@ Paths are relative to `/home/swaits/Code/wanderbooks/crates/` unless they are ab
| Replica keyring route | A: `/replica/keyring` serves wrapped keyslots to `books:view`. | **Removed.** No route ever serves wrapped keyslots. Local GUI TIN click: the GUI reads the wrapped keyslot from the data root (read-only), unwraps in its own process after the human types the passphrase, copies the value to the clipboard only, and zeroizes. A remote replica (P9) carries no keyslot material; remote TINs are seen only on `/h/` pages. | | Replica keyring route | A: `/replica/keyring` serves wrapped keyslots to `books:view`. | **Removed.** No route ever serves wrapped keyslots. Local GUI TIN click: the GUI reads the wrapped keyslot from the data root (read-only), unwraps in its own process after the human types the passphrase, copies the value to the clipboard only, and zeroizes. A remote replica (P9) carries no keyslot material; remote TINs are seen only on `/h/` pages. |
| TIN output scanner | A: match decrypted vault records. | **Pattern-based** (extend `has_full_tin`/`redact_full_tins` behind S:870 to every result, with the vault's last-four set as a hint). Fails closed. Documented as a pattern check. | | TIN output scanner | A: match decrypted vault records. | **Pattern-based** (extend `has_full_tin`/`redact_full_tins` behind S:870 to every result, with the vault's last-four set as a hint). Fails closed. Documented as a pattern check. |
| Extraction | In-process (A/B). | **Sandboxed subprocess** `ledger-extract` (C): rlimits, seccomp, no network, Landlock read-only on the input blob. | | Extraction | In-process (A/B). | **Sandboxed subprocess** `ledger-extract` (C): rlimits, seccomp, no network, Landlock read-only on the input blob. |
| Archive vs Landlock | A: server shells out to git, which contradicts its own Landlock rules. | **A separate archive process, `ledger-mcp archive`,** that the host starts beside the server, after the server holds every book's lock, and stops with it (3.9); each confines itself on its own. The archive process has read-only Landlock on `books/<id>/{journal.jsonl,blobs}`. Locally, Landlock ABI ≥ 4 limits its TCP connect to the target's port (443 or 22); it cannot filter by host, and an unprivileged process has no host-level egress filter, so that exists only under a system unit on self-host (`IPAddressAllow`). Locally the server listens on Unix sockets (3.9); if the owner turns TCP on, it binds 127.0.0.1 first. Either way Landlock ABI ≥ 4 then denies all further TCP bind and connect, so the local server has no egress. On self-host, a system unit adds `IPAddressDeny=any` (loopback allowed); if the built-in authorization server fetches CIMD documents (D-7 a), connect is allowed to port 443 only and `IPAddressDeny` narrows to private and link-local ranges (loopback still allowed), matching the CIMD SSRF guard. This confines the server only; it does not restrict the agent. | | Archive vs Landlock | A: server shells out to git, which contradicts its own Landlock rules. | **A separate archive process, `ledger-mcp archive`,** that the host starts beside the server, after the server holds every book's lock, and stops with it (3.9); each confines itself on its own. The archive process has read-only Landlock on `books/<id>/{journal.jsonl,blobs}`. Locally, Landlock ABI ≥ 4 limits its TCP connect to the target's port (443 or 22); it cannot filter by host, and an unprivileged process has no host-level egress filter, so that exists only under a system unit on self-host (`IPAddressAllow`). Locally the server listens on Unix sockets (3.9); if the owner turns TCP on, it binds 127.0.0.1 first. Either way Landlock ABI ≥ 4 then denies all further TCP bind and connect, so the local server has no egress. As built (slice 7, D-31): no archive process; the server runs git itself (3.7 as built), so slice 8's Landlock must let it run git and reach the backup folder and repository. On self-host, a system unit adds `IPAddressDeny=any` (loopback allowed); if the built-in authorization server fetches CIMD documents (D-7 a), connect is allowed to port 443 only and `IPAddressDeny` narrows to private and link-local ranges (loopback still allowed), matching the CIMD SSRF guard. This confines the server only; it does not restrict the agent. |
| Attention engine location | A: `ledger-store/attention.rs`. B: wasm-clean `ledger-view`. | **New crate `ledger-view`** (depends only on `ledger-core`), holding `attention::find`, `next_steps`, the evidence gate view, and a `BooksView` trait implemented by Store and Snapshot. A pure-Rust evidence projection (dropping SQLite from the viewer) is deferred to the hosted service's web viewer (section 11). | | Attention engine location | A: `ledger-store/attention.rs`. B: wasm-clean `ledger-view`. | **New crate `ledger-view`** (depends only on `ledger-core`), holding `attention::find`, `next_steps`, the evidence gate view, and a `BooksView` trait implemented by Store and Snapshot. A pure-Rust evidence projection (dropping SQLite from the viewer) is deferred to the hosted service's web viewer (section 11). |
| Inline bytes | A: 256 KiB. B: 1 MiB without sha. C: 256 KiB plus sha. | **256 KiB with a mandatory matching sha256.** A mismatch is refused with guard `base64_corrupt`, whose remedy is `books.begin_upload`. `/mcp` body cap is 1 MiB. | | Inline bytes | A: 256 KiB. B: 1 MiB without sha. C: 256 KiB plus sha. | **256 KiB with a mandatory matching sha256.** A mismatch is refused with guard `base64_corrupt`, whose remedy is `books.begin_upload`. `/mcp` body cap is 1 MiB. |
| Same-user mode | C: an announced tier. B: refuse to start without an acknowledgement flag. | **The local default, with no gate flag** (owner decision 2026-10-16). `books.start`, Needs attention, `wanderbooks doctor` and the docs report `isolation: "same_user (advisory)"`. | | Same-user mode | C: an announced tier. B: refuse to start without an acknowledgement flag. | **The local default, with no gate flag** (owner decision 2026-10-16). `books.start`, Needs attention, `wanderbooks doctor` and the docs report `isolation: "same_user (advisory)"`. |
@@ -135,7 +135,7 @@ The data root is `$XDG_DATA_HOME/wanderbooks` on Linux (default `~/.local/share/
|---|---|---|---| |---|---|---|---|
| Books server (local) | the server mode of `wanderbooks` (the `ledger-mcp` crate), listening on `run/mcp.sock` (and on 127.0.0.1:7411 only if the owner turns TCP on), started by the first opener (the app, an agent's bridge, or `wanderbooks serve`) through the host mode, and running while any opener holds it (D-21, D-23, 3.9; the host reads no books: it starts the server and the archive process, each confining itself, and stops both). TCP is off by default (3.9) | the human | yes, sole writer, holds the flock (`ledger-store/src/store.rs:74-89`) | | Books server (local) | the server mode of `wanderbooks` (the `ledger-mcp` crate), listening on `run/mcp.sock` (and on 127.0.0.1:7411 only if the owner turns TCP on), started by the first opener (the app, an agent's bridge, or `wanderbooks serve`) through the host mode, and running while any opener holds it (D-21, D-23, 3.9; the host reads no books: it starts the server and the archive process, each confining itself, and stops both). TCP is off by default (3.9) | the human | yes, sole writer, holds the flock (`ledger-store/src/store.rs:74-89`) |
| Books server (self-host, P9) | the same binary under a system unit on another machine | a dedicated account on that machine | yes, sole writer | | Books server (self-host, P9) | the same binary under a system unit on another machine | a dedicated account on that machine | yes, sole writer |
| Archive pusher | `ledger-mcp archive --book <id>` | the server's account, a separate process started beside the server | read-only journal and blobs | | Archive pusher | `ledger-mcp archive --book <id>` (as built, slice 7, D-31: none; the server pushes) | the server's account, a separate process started beside the server | read-only journal and blobs |
| Extractor | `ledger-extract` (subprocess) | inherits, sandboxed | the single input blob, via fd | | Extractor | `ledger-extract` (subprocess) | inherits, sandboxed | the single input blob, via fd |
| Viewer | `wanderbooks gui <book>`, the viewer mode (the `ledger-gui` crate; local, direct, the default; what opening the app shows); `wanderbooks gui --books <url>` (remote replica, P9) | the human | local: yes, read-only, never writes. Remote: none | | Viewer | `wanderbooks gui <book>`, the viewer mode (the `ledger-gui` crate; local, direct, the default; what opening the app shows); `wanderbooks gui --books <url>` (remote replica, P9) | the human | local: yes, read-only, never writes. Remote: none |
| CLI | `wanderbooks` (ledger-cli) | the human | **by design never**. Talks to `run/admin.sock` (peer-credential check: same user ID as the server). The admin socket cannot read books data; it can create or adopt books and mint agent tokens, admin-scope ones included. The peer-credential check keeps other local accounts out; any process running as the human, agents included, can use it (advisory, like the rest of the desktop deployment). | | CLI | `wanderbooks` (ledger-cli) | the human | **by design never**. Talks to `run/admin.sock` (peer-credential check: same user ID as the server). The admin socket cannot read books data; it can create or adopt books and mint agent tokens, admin-scope ones included. The peer-credential check keeps other local accounts out; any process running as the human, agents included, can use it (advisory, like the rest of the desktop deployment). |
@@ -195,7 +195,7 @@ No scope unwraps the vault. Unwrap happens only inside an `/h/` request that has
- **Tokens** are `wb_` and 64 hex digits (32 random bytes), one book and one name each (`agent` unless named), kept only as a SHA-256 in `auth/principals.jsonl`: a chain of `token.created` (with the token it replaces) and `token.revoked` records, each with its own hash besides the chain's (so a change to the last record is found too), replayed at start; one that can't be read refuses every agent with why (the list of books still answers). Checked in constant time against every live token. Made, listed and revoked only through the admin socket: `wanderbooks token create --book ID [--name N] [--scopes read,write,post,admin] [--replace] | list | revoke | print`. A token made is kept in the keyring before anything is said of it, or revoked again; `print` is the only way its text is shown. `connect` (slice 8) will mint through the same path. - **Tokens** are `wb_` and 64 hex digits (32 random bytes), one book and one name each (`agent` unless named), kept only as a SHA-256 in `auth/principals.jsonl`: a chain of `token.created` (with the token it replaces) and `token.revoked` records, each with its own hash besides the chain's (so a change to the last record is found too), replayed at start; one that can't be read refuses every agent with why (the list of books still answers). Checked in constant time against every live token. Made, listed and revoked only through the admin socket: `wanderbooks token create --book ID [--name N] [--scopes read,write,post,admin] [--replace] | list | revoke | print`. A token made is kept in the keyring before anything is said of it, or revoked again; `print` is the only way its text is shown. `connect` (slice 8) will mint through the same path.
- **Kept in the OS keyring** (Secret Service on Linux, through keyring-core; macOS and Windows stores come with P11), under service `wanderbooks` and an account naming the root by a hash of its path, the book and the token's name. The bridge (`wanderbooks mcp --book ID [--token NAME]`) reads it before each new session and each request until it has one, and again when the door refuses it, so a token made or replaced mid-session is used from the next call (with a new session: a session belongs to the token that opened it); without one, the server's refusal says how to make one. Tests keep tokens in files under the root's `run/` (the `file_keyring` knob). - **Kept in the OS keyring** (Secret Service on Linux, through keyring-core; macOS and Windows stores come with P11), under service `wanderbooks` and an account naming the root by a hash of its path, the book and the token's name. The bridge (`wanderbooks mcp --book ID [--token NAME]`) reads it before each new session and each request until it has one, and again when the door refuses it, so a token made or replaced mid-session is used from the next call (with a new session: a session belongs to the token that opened it); without one, the server's refusal says how to make one. Tests keep tokens in files under the root's `run/` (the `file_keyring` knob).
- **The door** (`/b/{id}/mcp`): no token or an unknown one, 401 with `WWW-Authenticate: Bearer` (the scheme in any case); another book's, 403; a session is bound to its token when opened and used only with it (403), forgotten when it ends, is found gone (an idle one, when next used), or its token is revoked or replaced; an id the server never opened binds nothing; past 100,000 bindings an initialize is refused, never a binding dropped. Each refusal at the door is recorded in that book's access log (action `connect`). - **The door** (`/b/{id}/mcp`): no token or an unknown one, 401 with `WWW-Authenticate: Bearer` (the scheme in any case); another book's, 403; a session is bound to its token when opened and used only with it (403), forgotten when it ends, is found gone (an idle one, when next used), or its token is revoked or replaced; an id the server never opened binds nothing; past 100,000 bindings an initialize is refused, never a binding dropped. Each refusal at the door is recorded in that book's access log (action `connect`).
- **Scopes** (`ledger-mcp/src/scopes.rs`): a table of every tool. Read-only tools need books:read; a tool with `confirmed` needs books:write to preview and books:post to act; tools that post at once, settings commits, voids, renumbering, corrections and role changes need books:post, and so do the controls on posting (setting an approval rule, retiring a convention, approving an entry) and sealing a TIN (`set_tin`, `set_w9`, which can replace one); `books.access_log` and the tools that can still put a full TIN out (`vendor_1099`, `export_cpa_package`, `get_w9`) need books:admin. A token is offered only the tools it can call (the list is then cached per caller, `cacheScope: private`); a call beyond its scopes is refused, unrun, naming the scope. Reading a resource needs books:read. A local token gets all four unless told otherwise. - **Scopes** (`ledger-mcp/src/scopes.rs`): a table of every tool. Read-only tools need books:read; a tool with `confirmed` needs books:write to preview and books:post to act; tools that post at once, settings commits, voids, renumbering, corrections and role changes need books:post, and so do the controls on posting (setting an approval rule, retiring a convention, approving an entry) and sealing a TIN (`set_tin`, `set_w9`, which can replace one); `books.access_log` and the tools that can still put a full TIN out (`vendor_1099`, `export_cpa_package`, `get_w9`) need books:admin, and so do backing up now and choosing where backups go (a `backup.*` setting, by `set_setting` or an imported `[backup]` table; slice 7): a backup is the whole book. A token is offered only the tools it can call (the list is then cached per caller, `cacheScope: private`); a call beyond its scopes is refused, unrun, naming the scope. Reading a resource needs books:read. A local token gets all four unless told otherwise.
- **The access log** (`books/<id>/access.jsonl`, own genesis) records every tool call and resource read, refusals included: who (token id, name), the client as it named itself (cut to 64 characters, any full TIN masked), the request id, the tool or resource, a SHA-256 of the arguments' canonical JSON with what may be a TIN masked first (a value under a key naming one, any key or text holding one in a shape the TIN scanner knows, any number of eight digits or more: an unsalted digest of a TIN whose last four the books show is broken by 10^5 guesses; a file's base64 is hashed as it is, never spelling a TIN in the clear), what came of it (`ok`, `error`, or `refused` with a code), and the journal's head after the call's own last write, if it wrote, whatever it answered (taken under the store's lock as the call lets go). A call's record is written on its own blocking thread while it still counts as running (one that panics part way included, with what it wrote), so a close waits for it too, and before the answer goes; one that can't be written stops the book (no more requests until a restart) and the answer says the call ran (a call refused as not run, by a close or a stopped server, keeps its "not run" code). A tool no table names is refused unrun. Known gap: a crash of the process between a call's journal write and its record leaves the journal ahead of the log for that call. A damaged log, or one holding a record that isn't an access, keeps its book closed, with why. `books.access_log` (admin) reads it newest first, filtered by token, action or outcome, paged with `before`; over stdio (no log) the tool isn't offered. - **The access log** (`books/<id>/access.jsonl`, own genesis) records every tool call and resource read, refusals included: who (token id, name), the client as it named itself (cut to 64 characters, any full TIN masked), the request id, the tool or resource, a SHA-256 of the arguments' canonical JSON with what may be a TIN masked first (a value under a key naming one, any key or text holding one in a shape the TIN scanner knows, any number of eight digits or more: an unsalted digest of a TIN whose last four the books show is broken by 10^5 guesses; a file's base64 is hashed as it is, never spelling a TIN in the clear), what came of it (`ok`, `error`, or `refused` with a code), and the journal's head after the call's own last write, if it wrote, whatever it answered (taken under the store's lock as the call lets go). A call's record is written on its own blocking thread while it still counts as running (one that panics part way included, with what it wrote), so a close waits for it too, and before the answer goes; one that can't be written stops the book (no more requests until a restart) and the answer says the call ran (a call refused as not run, by a close or a stopped server, keeps its "not run" code). A tool no table names is refused unrun. Known gap: a crash of the process between a call's journal write and its record leaves the journal ahead of the log for that call. A damaged log, or one holding a record that isn't an access, keeps its book closed, with why. `books.access_log` (admin) reads it newest first, filtered by token, action or outcome, paged with `before`; over stdio (no log) the tool isn't offered.
- **The owner passphrase** is an Argon2id hash (m=64 MiB, t=3, p=1, its own salt) in `auth/owner.json`, replaced whole; 12 to 1024 characters, no other rule (NIST SP 800-63B); a check refuses a hash weaker than that (changed by hand). `setup` at a terminal asks for it (twice); elsewhere it says to run `wanderbooks passphrase`, which works only at a terminal, unechoed, and asks for the one in force before a change. (Advisory, like everything local: a process running as the human can drive a pseudo-terminal and set the first one.) A stored hash of another Argon2 version, with a short salt or output, or asking a check for more than 1 GiB, 16 passes or 8 lanes is refused as changed by hand. The command line wipes token texts and passphrases from memory once used (the server and the bridge, which hold tokens for their lifetime, don't). `doctor` says whether it is set and lists the tokens. Its uses (`/h/` pages, consents) come with the slices that build them. - **The owner passphrase** is an Argon2id hash (m=64 MiB, t=3, p=1, its own salt) in `auth/owner.json`, replaced whole; 12 to 1024 characters, no other rule (NIST SP 800-63B); a check refuses a hash weaker than that (changed by hand). `setup` at a terminal asks for it (twice); elsewhere it says to run `wanderbooks passphrase`, which works only at a terminal, unechoed, and asks for the one in force before a change. (Advisory, like everything local: a process running as the human can drive a pseudo-terminal and set the first one.) A stored hash of another Argon2 version, with a short salt or output, or asking a check for more than 1 GiB, 16 passes or 8 lanes is refused as changed by hand. The command line wipes token texts and passphrases from memory once used (the server and the bridge, which hold tokens for their lifetime, don't). `doctor` says whether it is set and lists the tokens. Its uses (`/h/` pages, consents) come with the slices that build them.
- **Known, product-wide (for P7):** the TIN scanner (`has_full_tin`) takes hyphens and spaces as separators, not commas or parentheses, so "123,45,6789" passes it in party fields and the access-log digest alike. - **Known, product-wide (for P7):** the TIN scanner (`has_full_tin`) takes hyphens and spaces as separators, not commas or parentheses, so "123,45,6789" passes it in party fields and the access-log digest alike.
@@ -221,9 +221,9 @@ No scope unwraps the vault. Unwrap happens only inside an `/h/` request that has
ops/backup.jsonl ops/heads.jsonl not journaled ops/backup.jsonl ops/heads.jsonl not journaled
ops/show.jsonl books.show requests: the server appends (bounded, rotated), the GUI watches; not journaled ops/show.jsonl books.show requests: the server appends (bounded, rotated), the GUI watches; not journaled
ops/viewer.lock created by the server; the GUI holds a shared flock on a read-only descriptor (presence) ops/viewer.lock created by the server; the GUI holds a shared flock on a read-only descriptor (presence)
uploads/<id>.part staging, swept after 15 min (a staged backup: kept until its restore ends, at most 24 h) uploads/<id>.part staging, swept after 15 min (a backup being restored is cloned into books/.restore-* instead: 3.7 as built)
derived/<sha>/page-N.png disposable page-raster cache derived/<sha>/page-N.png disposable page-raster cache
archive.git/ bare; journal.jsonl + blobs/ + vault ciphertext; age-encrypted on push; destination Q-1 archive.git/ bare; journal.jsonl + blobs/ + vault/ + keys/ + access.jsonl, as they are (D-31, not encrypted); destinations D-30
``` ```
The server, the CLI and the GUI find the data root with one shared function (`ledger_store::root::Root::find`, built in P2 slice 2): `--root` if given, else the root `wanderbooks setup` recorded in `wanderbooks.location` beside the default, else the platform default under the account's home, taken from the password database on Unix, never `$HOME`. Finding it never reads the environment, the working directory, a project directory or a toml file; `$XDG_DATA_HOME` is honoured once, by `setup`, which records the result. The root and every folder made in it are created with an explicit 0700, not the umask, and a root that is a symlink, belongs to another user or that others can enter is refused (`setup` tightens one of ours). `setup` leaves a `.wanderbooks-root` marker, and a folder without one is never opened as a root nor, if it holds anything, made into one; every folder above the root must be one no other user can swap it in (owned by this account or the system, not writable by others unless sticky). Run again, `setup` keeps the root it recorded, and it never makes a second root or moves books while the current one holds any. Neither `setup` nor any refusal prints the root's location: nothing an agent is shown names it. Books are created and adopted one at a time, under `run/books.lock`. Windows (P11): `%LOCALAPPDATA%` from the known-folder API and an owner-only DACL; until then the Windows paths are read from the profile and carry no ACL. Extracted text is stored as a content-addressed blob, recorded in the journal by `attachment.text_extracted`, and archived with the books. Page rasters are disposable. Deleted assumptions: `books_dir = data_dir.parent()` (S:1155, `ledger-gui/src/books.rs:1474`, `ledger-mcp/src/main.rs:102`), the `.gitignore` check (S:531-543), and the `$XDG_RUNTIME_DIR` viewer file (`ledger-store/src/viewer.rs:98-130`), which `ops/show.jsonl` replaces. The server, the CLI and the GUI find the data root with one shared function (`ledger_store::root::Root::find`, built in P2 slice 2): `--root` if given, else the root `wanderbooks setup` recorded in `wanderbooks.location` beside the default, else the platform default under the account's home, taken from the password database on Unix, never `$HOME`. Finding it never reads the environment, the working directory, a project directory or a toml file; `$XDG_DATA_HOME` is honoured once, by `setup`, which records the result. The root and every folder made in it are created with an explicit 0700, not the umask, and a root that is a symlink, belongs to another user or that others can enter is refused (`setup` tightens one of ours). `setup` leaves a `.wanderbooks-root` marker, and a folder without one is never opened as a root nor, if it holds anything, made into one; every folder above the root must be one no other user can swap it in (owned by this account or the system, not writable by others unless sticky). Run again, `setup` keeps the root it recorded, and it never makes a second root or moves books while the current one holds any. Neither `setup` nor any refusal prints the root's location: nothing an agent is shown names it. Books are created and adopted one at a time, under `run/books.lock`. Windows (P11): `%LOCALAPPDATA%` from the known-folder API and an owner-only DACL; until then the Windows paths are read from the profile and carry no ACL. Extracted text is stored as a content-addressed blob, recorded in the journal by `attachment.text_extracted`, and archived with the books. Page rasters are disposable. Deleted assumptions: `books_dir = data_dir.parent()` (S:1155, `ledger-gui/src/books.rs:1474`, `ledger-mcp/src/main.rs:102`), the `.gitignore` check (S:531-543), and the `$XDG_RUNTIME_DIR` viewer file (`ledger-store/src/viewer.rs:98-130`), which `ops/show.jsonl` replaces.
@@ -236,7 +236,7 @@ The server, the CLI and the GUI find the data root with one shared function (`le
| Vault private key | wrapped per keyslot; plaintext only inside one `/h/` request (server) or one GUI unlock (in the GUI process, local), then zeroized | the human holding the passphrase, passkey PRF, recovery key, or an optional CPA keyslot | | Vault private key | wrapped per keyslot; plaintext only inside one `/h/` request (server) or one GUI unlock (in the GUI process, local), then zeroized | the human holding the passphrase, passkey PRF, recovery key, or an optional CPA keyslot |
| `preview.key` | `keys/` | server | | `preview.key` | `keys/` | server |
| Agent tokens | sha256 in `auth/`; plaintext in the user's keyring, which any process running as that user can read (advisory locally) | the agent's user | | Agent tokens | sha256 in `auth/`; plaintext in the user's keyring, which any process running as that user can read (advisory locally) | the agent's user |
| Archive credentials | OS keyring entry read by the archive process (local); systemd-creds for the archive unit (self-host) | archive pusher | | Archive credentials | OS keyring entry read by the archive process (local); systemd-creds for the archive unit (self-host). As built (slice 7): the owner's own ssh agent and git credential helpers, through the server | archive pusher |
| At-rest encryption | full-disk encryption (desktop); encrypted volume (self-host recipe) | protects against stolen disks | | At-rest encryption | full-disk encryption (desktop); encrypted volume (self-host recipe) | protects against stolen disks |
### 3.6 GUI ### 3.6 GUI
@@ -279,7 +279,7 @@ The archive process (`ledger-mcp archive`: locally started by the host after the
It writes `ops/backup.jsonl`. `books.backup {status|run}` reports it, and Needs attention flags a target more than 24 h behind. Head handoff: It writes `ops/backup.jsonl`. `books.backup {status|run}` reports it, and Needs attention flags a target more than 24 h behind. Head handoff:
- every mutating result carries the head; - every mutating result carries the head;
- `ops/heads.jsonl` records each 10-minute quiet period; - `ops/heads.jsonl` records each 10-minute quiet period;
- the customer clones the archive and runs `verify-archive --expect-head`. - the customer clones the archive and checks it (as built: a restore's preview, or the drills; no `verify-archive`).
Email or webhook head delivery and a witness log belong to the hosted service (section 11). Email or webhook head delivery and a witness log belong to the hosted service (section 11).
@@ -289,12 +289,20 @@ Email or webhook head delivery and a witness log belong to the hosted service (s
3. **The vault comes back only with its key.** The backup carries the vault sealed; the passphrase or the printed recovery key (P7) unlocks it on any machine. Until P7 the v1 key is a separate file: a backup counts as complete only when that key is saved too, and Needs attention says so when it isn't. 3. **The vault comes back only with its key.** The backup carries the vault sealed; the passphrase or the printed recovery key (P7) unlocks it on any machine. Until P7 the v1 key is a separate file: a backup counts as complete only when that key is saved too, and Needs attention says so when it isn't.
4. **Restore drills.** On a schedule (default weekly, or at the next start if the books were closed when it fell due) and after each close, the archive process restores the latest backup into a scratch folder, verifies it as in step 1, compares its reports with the live book's as of the backup head, and throws it away. A failed drill is a Needs-attention item; `books.backup status` reports the last drill. 4. **Restore drills.** On a schedule (default weekly, or at the next start if the books were closed when it fell due) and after each close, the archive process restores the latest backup into a scratch folder, verifies it as in step 1, compares its reports with the live book's as of the backup head, and throws it away. A failed drill is a Needs-attention item; `books.backup status` reports the last drill.
**As built (P2 slice 7), and where it departs (D-30, D-31):**
- **No archive process, no encryption.** D-31 made a backup the book as it is, in git, so there is nothing to encrypt and nothing to segment: the server itself backs each book up (`ledger_store::backup`, `ledger-mcp backup.rs`). Each book keeps a bare `archive.git` beside its journal; a backup is one commit of its journal, `blobs/`, `vault/`, `keys/` and `access.jsonl` exactly as they are, made under the store's lock (so whole lines and a vault that matches), its message `seq N head H` with `book <id>` below, then pushed outside the lock, fast-forward only (a target that moved on is refused and said, never forced), to `backup.git_remote` and to `backup.folder/<book>.git` (made bare on first use; a folder that isn't there, a disk not connected, is said plainly). git runs with no system config, no hooks (in a repository on this computer it pushes to too, through `--receive-pack`), no prompts, signing, ignore or attributes files or file watchers, its own identity, only the file, ssh and https protocols (never `ext::` or `fd::`, whatever the owner's own configuration says), and a limit of five minutes without a word (pushes and clones report progress; a git stopped is stopped with all it started); the owner's own `~/.gitconfig` still applies otherwise, for credential helpers. The stored files are staged outside the store's lock (they never change; a failure there is only a lost head start) and the commit, under it, is checked to hold every file there is, by name; bytes are kept as they are because no attributes rule of the owner's, the system's or the archive's is read, and a `.gitattributes` in a book's folder refuses the backup. A repository on this computer (a folder target, or an address that is a path) is pushed to or read only when plain: a bare repository at that very path (never one git would find beside or inside it), nothing in it sending git to another repository (a `.git`, `commondir` or `gitdir` file, objects borrowed from elsewhere, a link where it or its own parts are, a `<path>.git` or `<path>.bundle` beside it), its configuration only what `git init --bare` writes (and, for a copy made with `git clone --bare`, where it came from), since a setting of its own could run a command on a push; repositories are made with the files ref format and SHA-1 whatever the owner's own defaults; one is read as a far one is (`--no-local`, so its copying reports progress). A folder target's repository is made only while the target never took a backup: one gone since is a disk not connected (a mount point with nothing on it), said, never made again where the disk should be. A lock a stopped git left in the archive is cleared once older than any git may run quiet. The host now passes the owner's `SSH_AUTH_SOCK` (only a socket the account owns, not a link to one) so a private repository over ssh is reachable. `backup.folder` must be absolute and `backup.git_remote` can't read as an option; setting either (or importing a `[backup]` table) needs books:admin, as backing up does (a backup is the whole book, vault and key included).
- **When.** When the books open; a while after they were last written (`--backup-quiet`, 30 s, the `backup_quiet` knob for tests); before they close (at most 60 s); and on `books.back_up_now`. A failed commit holds every run off for 15 minutes; a target whose push failed is left out of the backups after writes for 15 minutes and then tried again, written to or not, while the others carry on. `period.closed` needs no trigger of its own: a close is a write. No head handoff files (`ops/heads.jsonl`) yet: the backup's commits are the head record.
- **The record and the tools.** Every commit, push and drill is a record of `ops/backup.jsonl` (a chain log; a backup never moves the journal's head), each target's keyed by a digest of where it is (a new folder or address starts with no standing). Stale means behind, with the first try to back up what it lacks more than a day old (not a day since the last good push: quiet books aren't stale the moment they're written); a failed commit is every target's error until a push succeeds. The chain isn't verified at each run: the server verifies it when it opens the books, and drills prove what reached each target. Two tools, not `{status|run}`: `books.backup` (books:read) for the standing of each target (last good backup, through which seq, stale after a day behind, the last error, the last drill and why it failed) and `books.back_up_now` (books:admin, the one open-world tool). `books.start` adds a step when none is set up, or one is stale, failing or failed its drill. Over stdio neither is offered. No error names a path in the data root.
- **Drills** run weekly per target that holds a backup, and at the next look after a failed one once 15 minutes have passed (looked for a minute after the books are first seen open, so also at the first start after one fell due, then hourly; `--drill-first`, the `drill_first` knob, for tests): clone, check the tree is plain files under the backed-up paths before any is written out (a link or submodule could otherwise point the checkout, or the store opened on it, at the live books), replay the chain, re-hash every stored file, check every TIN record the books name is there with its key, check the head against the commit's, and check the journal is a byte prefix of the live one (the reports then follow from the same journal, so no report comparison). Not after each close: weekly is the floor and a close changes nothing a drill would see.
- **Restore** is `wanderbooks books restore BACKUP [--id ID] [--replace] [--yes]` (without `--id`, the id of the books the backup holds, as its preview names them, else the repository's name), the admin socket's `restore`, previewed then confirmed against the previewed head. The server clones the backup into `books/.restore-*` (the address or folder crosses the owner-only admin socket as an adoption's path does; no staging through `uploads/`), proves it as a drill does (but for the live-journal check), one restore at a time, builds the book as an adoption builds (every stored file re-hashed, opened, verified, synced; the vault and its key come with it, D-31), gives it the backup's history as its own `archive.git` (so its next backups fast-forward), and renames it into place. Over existing books it is refused unless `--replace`: then the server lets go of the open books (their writer lock released; a session still holding them is refused, its client's next session sees the restored books), waits for their last backup run and starts no other until the restore is done (one in flight could otherwise push past the backup being restored), and moves them whole into `legacy/<id>-<time>/`; a restore that fails puts them back and opens them again. `books.purge_tin` now says backups made before keep their copies (D-29, D-31). Each backup commit names its books (`book <id>` in its message), so a restore knows whose books it brings back (and the command line, given no `--id`, restores them under that id, deciding after one preview, which only reports; a restore leaves `ops/restored-from` in the books, noted by the server when it opens them, so the note lands with the books): the same books count their targets' backups as their own (a backup folder gone since is a disk not connected, never made again); books restored beside the books they were, under another id, never push to those books' repository (it would stop their backups): it is paused for them, said by `books.backup`, until another is named. A damaged `ops/backup.jsonl` is set aside and begun again, never keeping the books shut, what it knew of whose backups the targets hold carried over; a backup folder that took the books' backups and holds none now says so (a disk not connected, or a new one: `books.back_up_now {new_disk: true}` begins it again, only when the human says so). One book's drill holds the others' backups until it ends; a backup at close waits at most a minute for it. No passphrase for a replacing restore (D-31: the old books are kept). Viewer and helper: "Restore…" and "Back up now" come with the menu intents (slice 8); the lease socket's `restore` intent now names the backup and the id, and both intents point to the terminal and the agent until then.
- **Tests:** `crates/ledger-cli/tests/backups.rs` (both targets after a write; a write not backed up before the quiet; close, and catching up at the next open; admin only and the head unmoved; a target that moved on; restore beside, on a clean install, of the previewed backup only, with its vault and key, refusing five kinds of damage and writing nothing, never overwriting and keeping the replaced books, a failed replace leaving the books open; a drill on open; a failed drill flagged), `open_while_held.rs` (`books restore` from the command line), `ledger-store` (backup unit tests: links and strangers refused before checkout, target hooks never run, ignore rules never leave a file out, stale locks, drills; `root.rs`: never moving books something holds), the knobs, the host's environment, the scopes, the settings' checks, and the loop's and the standing's unit tests. The suite passed nine of nine runs three at a time (the replace race above showed only under load).
### 3.8 Local vs self-host ### 3.8 Local vs self-host
| | Local (T1, the default) | Self-host (T2, P9) | | | Local (T1, the default) | Self-host (T2, P9) |
|---|---|---| |---|---|---|
| Where | the human's computer | another machine the owner controls: home server, NAS or VPS | | Where | the human's computer | another machine the owner controls: home server, NAS or VPS |
| Processes | one server and its archive process, open while the app, an agent session or a terminal holds them (D-23) | the same binaries under system units, plus TLS (Caddy) and, optionally, `wanderbooks tunnel` | | Processes | one server (as built, slice 7: which backs the books up itself), open while the app, an agent session or a terminal holds them (D-23) | the same binaries under system units, plus TLS (Caddy) and, optionally, `wanderbooks tunnel` |
| Runs as | the human | a dedicated account; agents have no account on that machine | | Runs as | the human | a dedicated account; agents have no account on that machine |
| Isolation | `same_user (advisory)`: hidden, not locked | `service_user (enforced)`: machine and user-ID boundary | | Isolation | `same_user (advisory)`: hidden, not locked | `service_user (enforced)`: machine and user-ID boundary |
| Auth | static bearer in the keyring; `/h/` uses the owner passphrase (or a passkey) | static bearer over TLS for header-capable clients; OAuth 2.1 for claude.ai and ChatGPT (D-7); `/h/` uses a passkey or the passphrase | | Auth | static bearer in the keyring; `/h/` uses the owner passphrase (or a passkey) | static bearer over TLS for header-capable clients; OAuth 2.1 for claude.ai and ChatGPT (D-7); `/h/` uses a passkey or the passphrase |
@@ -313,7 +321,7 @@ The books are open while something holds them; there is no background service. T
**Who holds them.** The app (the viewer), an agent session's bridge (`wanderbooks mcp`), a terminal running `wanderbooks serve`, and any CLI command that needs the server. Each holds a **lease**. **Who holds them.** The app (the viewer), an agent session's bridge (`wanderbooks mcp`), a terminal running `wanderbooks serve`, and any CLI command that needs the server. Each holds a **lease**.
**Three local channels, all in `run/` under the data root** (0700 on Linux and macOS; on Windows `%LOCALAPPDATA%\wanderbooks`, never the roaming profile, with an owner-only DACL; named pipes with an owner-only DACL, `FILE_FLAG_FIRST_PIPE_INSTANCE`, and a check of the client's SID): **Three local channels, all in `run/` under the data root** (0700 on Linux and macOS; on Windows `%LOCALAPPDATA%\wanderbooks`, never the roaming profile, with an owner-only DACL; named pipes with an owner-only DACL, `FILE_FLAG_FIRST_PIPE_INSTANCE`, and a check of the client's SID):
- `run/lease.sock`: served by the **host**, leases only. Each lease records its kind (app, bridge, serve, cli) and PID; the app and `wanderbooks doctor` list them, so a stray holder keeping the books open can be seen and closed. Besides holding, the socket accepts exactly five **owner intents**, which is how the viewer hands work to the server: `close`, `back_up_now`, `new_books`, `restore` (a backup, as its own book beside the others), and `consent_format_upgrade` (a book). `consent_format_upgrade`, and a restore that replaces a book, carry the owner passphrase typed in the viewer, checked by the server (Argon2id, as on `/h/`); no MCP tool can give either, so an agent never consents. Nothing else is accepted. - `run/lease.sock`: served by the **host**, leases only. Each lease records its kind (app, bridge, serve, cli) and PID; the app and `wanderbooks doctor` list them, so a stray holder keeping the books open can be seen and closed. Besides holding, the socket accepts exactly five **owner intents**, which is how the viewer hands work to the server: `close`, `back_up_now`, `new_books`, `restore` (a backup, as its own book beside the others), and `consent_format_upgrade` (a book). `consent_format_upgrade` carries the owner passphrase typed in the viewer (a restore that replaces a book no longer does, D-31: the books it replaces are kept), checked by the server (Argon2id, as on `/h/`); no MCP tool can give either, so an agent never consents. Nothing else is accepted.
- `run/mcp.sock`: served by the **server**, MCP for the bridge. Local agents never use TCP, so no other account on the machine can squat a port and catch the token or the traffic. Both ends check the peer's user ID (`SO_PEERCRED`/`getpeereid`, the pipe's SID). TCP is off by default. The owner can turn it on (`server.toml`, or `wanderbooks serve --http`) for a client that needs HTTP; it then binds 127.0.0.1:7411, a failed bind is reported by `doctor` and is not fatal, and such clients hold no lease, so the books must already be open. - `run/mcp.sock`: served by the **server**, MCP for the bridge. Local agents never use TCP, so no other account on the machine can squat a port and catch the token or the traffic. Both ends check the peer's user ID (`SO_PEERCRED`/`getpeereid`, the pipe's SID). TCP is off by default. The owner can turn it on (`server.toml`, or `wanderbooks serve --http`) for a client that needs HTTP; it then binds 127.0.0.1:7411, a failed bind is reported by `doctor` and is not fatal, and such clients hold no lease, so the books must already be open.
- `run/admin.sock`: served by the server, admin only (create, adopt, tokens), as in 3.1. Neither the viewer nor the bridge opens it. - `run/admin.sock`: served by the server, admin only (create, adopt, tokens), as in 3.1. Neither the viewer nor the bridge opens it.
@@ -321,11 +329,11 @@ The books are open while something holds them; there is no background service. T
**The host's process (P2 slice 3).** `wanderbooks host` closes every descriptor above 2 it was handed, refuses to run confined, takes `run/host.lock` (waiting up to 30 s for a host that is closing; one that finds a live host exits at once), binds `run/lease.sock`, and starts the server from `/proc/self/exe`, naming the root on its command line only when the host itself was given one (tests): nothing in a process list names it. Ending leases, it tells each holder why and closes the server only once they have let go (at most 2 s), so holders stop sending before it closes; a call already on its way is refused at the door, unrun, and sent again once the books are back. A closing server is never killed, however long its last call runs; a new server waits for an old one still finishing (its host killed) rather than fail. A restart begins when the new version appears, not at a quiet moment: the server first finishes every call in progress (none is cut off), and calls arriving meanwhile wait for the new server (up to 60 s); waiting for idle comes with jobs (`books.job`), which are what can run long. On a restart the old host hands over: it lets go of `host.lock` and keeps `update.lock` until the new host answers, so waiting holders find it. A holder starts a host only when none holds the lock; one that died at once is noticed within 2 s; a server that fails soon after opening counts as failing (only 60 s open clears the count). Lease records carry the holder's process as the kernel reports it. The lease reasons are `closed_by_owner`, `restarting`, `closing` (the host is closing because nothing held it: hold again, a new host will answer; the owner's close, by the menu, `wanderbooks close` or a signal to the host, is always `closed_by_owner`, and a holder that had held stays closed) and `server_failed`. The bridge never sends a call the agent cancelled before it was sent (MCP owes no answer to a cancelled request), answers a ping itself while the books open, refuses JSON-RPC batches request by request, passes no client offer it can't carry back (sampling, elicitation, roots), reports a call answered with nothing as possibly run, and carries the per-request protocol headers of the 2026-07-28 era. Sessions on the server time out only after 24 h without a message (rmcp's idle timer ignores a call in progress, so a shorter one could cut a long call off); the bridge ends its own when its stdin closes or it is signalled. `wanderbooks doctor` (a first form, completed in slice 8) says whether the books are open, what holds them, and the host's and server's last log lines, naming no folder. Tests shorten the timings with a hidden `--test-knobs` flag, passed down from holder to helper to host. **The host's process (P2 slice 3).** `wanderbooks host` closes every descriptor above 2 it was handed, refuses to run confined, takes `run/host.lock` (waiting up to 30 s for a host that is closing; one that finds a live host exits at once), binds `run/lease.sock`, and starts the server from `/proc/self/exe`, naming the root on its command line only when the host itself was given one (tests): nothing in a process list names it. Ending leases, it tells each holder why and closes the server only once they have let go (at most 2 s), so holders stop sending before it closes; a call already on its way is refused at the door, unrun, and sent again once the books are back. A closing server is never killed, however long its last call runs; a new server waits for an old one still finishing (its host killed) rather than fail. A restart begins when the new version appears, not at a quiet moment: the server first finishes every call in progress (none is cut off), and calls arriving meanwhile wait for the new server (up to 60 s); waiting for idle comes with jobs (`books.job`), which are what can run long. On a restart the old host hands over: it lets go of `host.lock` and keeps `update.lock` until the new host answers, so waiting holders find it. A holder starts a host only when none holds the lock; one that died at once is noticed within 2 s; a server that fails soon after opening counts as failing (only 60 s open clears the count). Lease records carry the holder's process as the kernel reports it. The lease reasons are `closed_by_owner`, `restarting`, `closing` (the host is closing because nothing held it: hold again, a new host will answer; the owner's close, by the menu, `wanderbooks close` or a signal to the host, is always `closed_by_owner`, and a holder that had held stays closed) and `server_failed`. The bridge never sends a call the agent cancelled before it was sent (MCP owes no answer to a cancelled request), answers a ping itself while the books open, refuses JSON-RPC batches request by request, passes no client offer it can't carry back (sampling, elicitation, roots), reports a call answered with nothing as possibly run, and carries the per-request protocol headers of the 2026-07-28 era. Sessions on the server time out only after 24 h without a message (rmcp's idle timer ignores a call in progress, so a shorter one could cut a long call off); the bridge ends its own when its stdin closes or it is signalled. `wanderbooks doctor` (a first form, completed in slice 8) says whether the books are open, what holds them, and the host's and server's last log lines, naming no folder. Tests shorten the timings with a hidden `--test-knobs` flag, passed down from holder to helper to host.
**Starting the host cleanly.** Every mode that confines itself (the viewer, the bridge) first starts a small unconfined **helper** (`wanderbooks helper`, spawned as its own process over an inherited socket pair before any thread, window or Cocoa start; never a fork of a running GUI), then confines itself. The helper accepts a fixed message set and nothing else: `start_host`; `open_with` an attachment id (the helper itself copies that blob, read-only, into a 0700 per-user directory, `$XDG_RUNTIME_DIR/wanderbooks-open/` or its per-system equivalent, swept when the helper exits, and hands it to the system's default app; it never accepts a path or a command line); `stage_backup`, whose file the helper asks the owner for in its own file dialog (never a path from the viewer), copying it into the data root's `uploads/` and returning a staged id for the `restore` intent; and `start_updater`, with no arguments. A confined mode starts the host only through its helper; unconfined holders (the CLI commands, `serve`) use the same clean start directly, and a restarting host starts its successor itself, already clean, in its own session and scope (P2 slice 3). The clean start is a re-exec: environment scrubbed to a fixed allow-list (keeping `XDG_RUNTIME_DIR` and `DBUS_SESSION_BUS_ADDRESS`, which `systemd-run --user` needs, only after checking they point at the host's own user's runtime directory), stdin, stdout and stderr to null (a log file under `run/`), a new session (`setsid`), a fixed umask. On Linux under systemd (uwsm on Omarchy puts every app in its own scope, which would take the host down with the app) it re-execs through a transient `systemd-run --user --scope`, which installs nothing, persists nothing, and disappears with the host; elsewhere `setsid` alone. On Windows, `CREATE_BREAKAWAY_FROM_JOB` where the job allows it, and `DETACHED_PROCESS`. The host resolves the data root itself from the user's account (`getpwuid` home, or the known folder on Windows), never from the opener's environment, unless `wanderbooks setup` recorded another root. It refuses to run if it finds itself confined (seccomp in `/proc/self/status`, a Landlock write probe). **Starting the host cleanly.** Every mode that confines itself (the viewer, the bridge) first starts a small unconfined **helper** (`wanderbooks helper`, spawned as its own process over an inherited socket pair before any thread, window or Cocoa start; never a fork of a running GUI), then confines itself. The helper accepts a fixed message set and nothing else: `start_host`; `open_with` an attachment id (the helper itself copies that blob, read-only, into a 0700 per-user directory, `$XDG_RUNTIME_DIR/wanderbooks-open/` or its per-system equivalent, swept when the helper exits, and hands it to the system's default app; it never accepts a path or a command line); `stage_backup`, whose file the helper asks the owner for in its own file dialog (never a path from the viewer), copying it into the data root's `uploads/` and returning a staged id for the `restore` intent (as built, slice 7: a backup is a repository, and the `restore` intent names its folder or address, which the server clones; 3.7); and `start_updater`, with no arguments. A confined mode starts the host only through its helper; unconfined holders (the CLI commands, `serve`) use the same clean start directly, and a restarting host starts its successor itself, already clean, in its own session and scope (P2 slice 3). The clean start is a re-exec: environment scrubbed to a fixed allow-list (keeping `XDG_RUNTIME_DIR` and `DBUS_SESSION_BUS_ADDRESS`, which `systemd-run --user` needs, only after checking they point at the host's own user's runtime directory), stdin, stdout and stderr to null (a log file under `run/`), a new session (`setsid`), a fixed umask. On Linux under systemd (uwsm on Omarchy puts every app in its own scope, which would take the host down with the app) it re-execs through a transient `systemd-run --user --scope`, which installs nothing, persists nothing, and disappears with the host; elsewhere `setsid` alone. On Windows, `CREATE_BREAKAWAY_FROM_JOB` where the job allows it, and `DETACHED_PROCESS`. The host resolves the data root itself from the user's account (`getpwuid` home, or the known folder on Windows), never from the opener's environment, unless `wanderbooks setup` recorded another root. It refuses to run if it finds itself confined (seccomp in `/proc/self/status`, a Landlock write probe).
**One host, even in a race.** The host takes `run/host.lock` (an exclusive flock) before anything else; a second host exits and its opener attaches. The host starts the server, which takes every book's writer lock, and only then the archive process, so two archive pushers never meet on one `archive.git`. An opener that finds no live host and an unlocked `host.lock` starts one; one that finds a host shutting down (the lease socket refuses during shutdown) waits on `host.lock`, then starts a new one. **One host, even in a race.** The host takes `run/host.lock` (an exclusive flock) before anything else; a second host exits and its opener attaches. The host starts the server, which takes every book's writer lock (as built, slice 7: and backs the books up itself, one run at a time per book, so two pushers never meet on one `archive.git`). An opener that finds no live host and an unlocked `host.lock` starts one; one that finds a host shutting down (the lease socket refuses during shutdown) waits on `host.lock`, then starts a new one.
**Closing.** When the last lease ends, the host waits 10 s; a lease arriving in that time cancels the close. Then: no new requests, the one in progress finishes, every writer lock is released, the archive process finishes its push, and the host exits. "Close the books" (the app's menu, `wanderbooks close`) ends every lease at once; a bridge then tells its agent the owner closed the books and reopens them only in a new session. Before it ends leases the host says why on each one (`closed_by_owner`, `restarting`, `closing`, `server_failed`; P2 slice 3). A holder that sees the connection end with no reason (a crash, a kill) starts a new host and re-takes its lease, with backoff (1, 2, 4 … 60 s, at most five tries); a host that keeps failing to start (a damaged journal, a panic) becomes a Needs-attention and `doctor` item, and a bridge answers its agent with a clear error instead of retrying forever. `host_respawns` and `respawn_backs_off` test it. **Closing.** When the last lease ends, the host waits 10 s; a lease arriving in that time cancels the close. Then: no new requests, the one in progress finishes, the server makes a last backup of each book (as built, slice 7, at most 60 s), every writer lock is released, and the host exits. "Close the books" (the app's menu, `wanderbooks close`) ends every lease at once; a bridge then tells its agent the owner closed the books and reopens them only in a new session. Before it ends leases the host says why on each one (`closed_by_owner`, `restarting`, `closing`, `server_failed`; P2 slice 3). A holder that sees the connection end with no reason (a crash, a kill) starts a new host and re-takes its lease, with backoff (1, 2, 4 … 60 s, at most five tries); a host that keeps failing to start (a damaged journal, a panic) becomes a Needs-attention and `doctor` item, and a bridge answers its agent with a clear error instead of retrying forever. `host_respawns` and `respawn_backs_off` test it.
**Restarting** (an update, or the executable replaced by a package manager) is not a close. What lives between calls survives it: upload grants, download links, `/h/` sessions and finished job results are kept in `run/state/` (0600, the session key with them), and a running job delays the restart until it finishes (jobs, `books.job`, come later; until then a restart begins at once and the server finishes every call in progress first, P2 slice 3). The host takes `run/update.lock`, closes the books as above but keeps the lease socket's holders informed, swaps (or, for a package-managed install, finds the new executable already in place), and restarts from the new executable. Openers that arrive meanwhile wait on `update.lock`. Bridges reconnect and re-initialize upstream without the agent noticing. A package-managed upgrade is noticed by the host (its executable's inode changed) and restarts it as above. **Restarting** (an update, or the executable replaced by a package manager) is not a close. What lives between calls survives it: upload grants, download links, `/h/` sessions and finished job results are kept in `run/state/` (0600, the session key with them), and a running job delays the restart until it finishes (jobs, `books.job`, come later; until then a restart begins at once and the server finishes every call in progress first, P2 slice 3). The host takes `run/update.lock`, closes the books as above but keeps the lease socket's holders informed, swaps (or, for a package-managed install, finds the new executable already in place), and restarts from the new executable. Openers that arrive meanwhile wait on `update.lock`. Bridges reconnect and re-initialize upstream without the agent noticing. A package-managed upgrade is noticed by the host (its executable's inode changed) and restarts it as above.
@@ -479,7 +487,7 @@ Report results also carry `as_of_status {drafts_open_on_or_before, unreconciled,
| chart | `books.rename_account` | | chart | `books.rename_account` |
| evidence | `books.begin_upload`, `books.inbox`, `books.discard_inbox_item`, `books.attach_email`, `books.find_attachment {sha256\|md5\|receipt_number\|text}`, `books.annotate_attachment`, `books.consolidate_duplicates`, `books.add_instrument`, `books.instruments`, `books.set_entry_instrument`, `books.download_link` | | evidence | `books.begin_upload`, `books.inbox`, `books.discard_inbox_item`, `books.attach_email`, `books.find_attachment {sha256\|md5\|receipt_number\|text}`, `books.annotate_attachment`, `books.consolidate_duplicates`, `books.add_instrument`, `books.instruments`, `books.set_entry_instrument`, `books.download_link` |
| assurance | `books.audit {through, scope}` (job, deterministic workpaper), `books.workpaper`, `books.attest_receipt`, `books.review_entry`, `books.attest_period`, `books.attestations` | | assurance | `books.audit {through, scope}` (job, deterministic workpaper), `books.workpaper`, `books.attest_receipt`, `books.review_entry`, `books.attest_period`, `books.attestations` |
| ops | `books.backup {status\|run}`, `books.access_log` (admin), `books.invite {kind: cpa\|member\|viewer, email}` (admin; remote people need the self-host server, P9) | | ops | `books.backup`, `books.back_up_now` (admin), `books.access_log` (admin), `books.invite {kind: cpa\|member\|viewer, email}` (admin; remote people need the self-host server, P9) |
| migration | `books.compare_reference {reference_attachment, as_of, mapping_convention}`, `books.record_inventory {source, items\|upload_id}`, `books.check_inventory {listing}` | | migration | `books.compare_reference {reference_attachment, as_of, mapping_convention}`, `books.record_inventory {source, items\|upload_id}`, `books.check_inventory {listing}` |
| reports | `export` kinds `members_equity`, `general_ledger`, `general_journal` (JSON with `entry_id`) | | reports | `export` kinds `members_equity`, `general_ledger`, `general_journal` (JSON with `entry_id`) |
| vault (human links) | `books.request_tin_entry {party, kind: ssn\|ein\|itin\|w9}`, `books.request_human_download {what: cpa_package_with_tins\|vendor_1099_with_tins\|w9, party?, year?}`, `books.set_w9_facts {party, name, classification, address, signed_on, exempt_payee?}` | | vault (human links) | `books.request_tin_entry {party, kind: ssn\|ein\|itin\|w9}`, `books.request_human_download {what: cpa_package_with_tins\|vendor_1099_with_tins\|w9, party?, year?}`, `books.set_w9_facts {party, name, classification, address, signed_on, exempt_payee?}` |
@@ -828,8 +836,8 @@ The first company keeps working between phases.
- **Every server-path parameter deleted:** `import_local_file`, `save_to`, `save_csv_to`, `w9_file_path`, get_w9 `save_to`. W-9 upload moves to `/h/w9/{party}` (sealed with the v1 key until P7). TIN-bearing outputs come only through `request_human_download` → `/h/`, behind the owner passphrase. This closes defect 19. - **Every server-path parameter deleted:** `import_local_file`, `save_to`, `save_csv_to`, `w9_file_path`, get_w9 `save_to`. W-9 upload moves to `/h/w9/{party}` (sealed with the v1 key until P7). TIN-bearing outputs come only through `request_human_download` → `/h/`, behind the owner passphrase. This closes defect 19.
- `get_receipt view=link|bytes(≤256 KiB+sha)` (defect 3, transport part). - `get_receipt view=link|bytes(≤256 KiB+sha)` (defect 3, transport part).
- **Viewer, direct and read-only (3.6):** data-root discovery through `ledger-store::paths::data_root()` or `--root` from `wanderbooks gui <book>`, never a project directory or `wanderbooks.toml`; `books.show` through `ops/show.jsonl` (delete `ledger-store/src/viewer.rs`); presence through the `ops/viewer.lock` shared flock; Save-a-copy and Open-with (defect 3, viewer part). Locally there is no replica; the viewer's only channels are the lease socket's owner intents and its helper (3.9). - **Viewer, direct and read-only (3.6):** data-root discovery through `ledger-store::paths::data_root()` or `--root` from `wanderbooks gui <book>`, never a project directory or `wanderbooks.toml`; `books.show` through `ops/show.jsonl` (delete `ledger-store/src/viewer.rs`); presence through the `ops/viewer.lock` shared flock; Save-a-copy and Open-with (defect 3, viewer part). Locally there is no replica; the viewer's only channels are the lease socket's owner intents and its helper (3.9).
- Archive process and module; `books.backup`; `ops/heads.jsonl`; `verify-archive`; `fetch`. - Archive process and module; `books.backup`; `ops/heads.jsonl`; `verify-archive`; `fetch`. (As built, slice 7, D-31: the server's backup module, `books.backup` and `books.back_up_now`; no archive process, `ops/heads.jsonl`, `verify-archive` or `fetch`: a backup is a repository, cloned and proven by a restore's preview and the drills.)
- **Restore (3.7, D-19):** `books restore`, proven before use, never destructive, onto a new or an existing install; restore drills; the vault key's backup reported until P7. - **Restore (3.7, D-19):** `books restore`, proven before use, never destructive, onto a new or an existing install; restore drills; the vault and its key travel in the backup (D-31).
- **Viewer menu (D-19):** "Export for CPA" (done by the viewer, into Downloads); "Back up now", "Restore…", "New books…", "Close the books" and format-upgrade consent handed over as the lease socket's owner intents (3.9); "Connect Claude Code" and "Connect Codex". Nothing in the viewer writes a book. - **Viewer menu (D-19):** "Export for CPA" (done by the viewer, into Downloads); "Back up now", "Restore…", "New books…", "Close the books" and format-upgrade consent handed over as the lease socket's owner intents (3.9); "Connect Claude Code" and "Connect Codex". Nothing in the viewer writes a book.
- `wanderbooks mcp`, the agent bridge (D-23, 3.9): stdio to the agent, MCP over `run/mcp.sock` to the local server (or HTTPS with `--url`, P9), the token from the keyring. It opens the books if they are closed and holds a lease for the session. Across a server restart (an update) it reconnects and re-initializes upstream without the agent noticing, sending `notifications/tools/list_changed` only if the tool list changed. - `wanderbooks mcp`, the agent bridge (D-23, 3.9): stdio to the agent, MCP over `run/mcp.sock` to the local server (or HTTPS with `--url`, P9), the token from the keyring. It opens the books if they are closed and holds a lease for the session. Across a server restart (an update) it reconnects and re-initializes upstream without the agent noticing, sending `notifications/tools/list_changed` only if the tool list changed.
- CLI subcommands and modes: `setup`, `serve`, `close`, `mcp`, `host`, `helper`, `rollback`, `books create|adopt|list|export-legacy`, `connect`, `token`, `gui`, `doctor`, `verify-archive`, `fetch`, `update`. `books export-legacy <id>` (rollback only) writes a P1-format data dir outside the data root, verifies its head equals the server's, and stops the server's writes to that book. - CLI subcommands and modes: `setup`, `serve`, `close`, `mcp`, `host`, `helper`, `rollback`, `books create|adopt|list|export-legacy`, `connect`, `token`, `gui`, `doctor`, `verify-archive`, `fetch`, `update`. `books export-legacy <id>` (rollback only) writes a P1-format data dir outside the data root, verifies its head equals the server's, and stops the server's writes to that book.
@@ -843,7 +851,7 @@ The first company keeps working between phases.
**Tests:** **Tests:**
- `agent_uid_cannot_read_tenant` (the two-user CI deployment; not the desktop default). - `agent_uid_cannot_read_tenant` (the two-user CI deployment; not the desktop default).
- `setup_needs_no_root` (setup, serve, adopt, connect and gui as an unprivileged user in a clean container; nothing is left installed to run in the background). - `setup_needs_no_root` (setup, serve, adopt, connect and gui as an unprivileged user in a clean container; nothing is left installed to run in the background).
- `books_close_cleanly` (the last lease ending, "close the books", SIGTERM and SIGHUP of the host, all during a burst of posts: every request either completes or is refused, the head verifies at the next open, the lock is released); `openers_share_one_server` (the app, `serve`, a CLI command and two bridges started together: one host, one writer); `books_close_with_last_holder` (each kind of holder ending, `kill -9` included: closed 10 s after the last, not before); `bridge_opens_closed_books` and `bridge_survives_server_restart` (the agent's session carries on, no reconnect); `close_now_ends_every_lease`; `host_starts_clean` (started from a confined viewer or bridge with a hostile environment, `XDG_DATA_HOME` and `LD_PRELOAD` included, the host is unconfined, finds the real root, and holds no inherited descriptor); `host_respawns` (kill the host: a holder starts a new one); `host_survives_app_scope` (under a systemd app scope, as uwsm sets up, closing the app that started the host leaves it running for the other holders); `no_local_agent_tcp` (the bridge never connects to TCP; TCP is off by default; with it on, a squatter on 7411 makes `doctor` report a failed bind and sees no bridge traffic); `lease_socket_accepts_only_owner_intents` (exactly the five; consent and replace without the passphrase refused); `restart_is_not_close` (bridges carry on across a restart; openers wait on `update.lock`); `children_match_host_version` (replace the executable mid-run: no child of the new version joins the old host); `format_upgrade_waits_for_consent` (read-only until the owner's consent, a backup with the keys made first, no MCP tool can consent, declining leaves the book openable after `wanderbooks rollback`); `no_rollback_after_consent` ("Go back" refuses for an upgraded book; the pre-upgrade restore arrives beside it, states the entries it lacks, and needs the passphrase); `restore_reads_only_staged_backups` (a `restore` intent naming a path is refused); `bridge_initializes_during_cold_start`; `archive_catches_up_at_start` (a push that failed before a close is made at the next open). - `books_close_cleanly` (the last lease ending, "close the books", SIGTERM and SIGHUP of the host, all during a burst of posts: every request either completes or is refused, the head verifies at the next open, the lock is released); `openers_share_one_server` (the app, `serve`, a CLI command and two bridges started together: one host, one writer); `books_close_with_last_holder` (each kind of holder ending, `kill -9` included: closed 10 s after the last, not before); `bridge_opens_closed_books` and `bridge_survives_server_restart` (the agent's session carries on, no reconnect); `close_now_ends_every_lease`; `host_starts_clean` (started from a confined viewer or bridge with a hostile environment, `XDG_DATA_HOME` and `LD_PRELOAD` included, the host is unconfined, finds the real root, and holds no inherited descriptor); `host_respawns` (kill the host: a holder starts a new one); `host_survives_app_scope` (under a systemd app scope, as uwsm sets up, closing the app that started the host leaves it running for the other holders); `no_local_agent_tcp` (the bridge never connects to TCP; TCP is off by default; with it on, a squatter on 7411 makes `doctor` report a failed bind and sees no bridge traffic); `lease_socket_accepts_only_owner_intents` (exactly the five; consent without the passphrase refused); `restart_is_not_close` (bridges carry on across a restart; openers wait on `update.lock`); `children_match_host_version` (replace the executable mid-run: no child of the new version joins the old host); `format_upgrade_waits_for_consent` (read-only until the owner's consent, a backup with the keys made first, no MCP tool can consent, declining leaves the book openable after `wanderbooks rollback`); `no_rollback_after_consent` ("Go back" refuses for an upgraded book; the pre-upgrade restore arrives beside it, states the entries it lacks, and needs the passphrase); `restore_reads_only_staged_backups` (superseded by D-31: the `restore` intent names the backup's folder or address); `bridge_initializes_during_cold_start`; `archive_catches_up_at_start` (a push that failed before a close is made at the next open).
- `one_executable` (every mode from one binary; the viewer mode under read-only Landlock opens and follows a book); `server_build_has_no_gui` (the self-host build links no window libraries). - `one_executable` (every mode from one binary; the viewer mode under read-only Landlock opens and follows a book); `server_build_has_no_gui` (the self-host build links no window libraries).
- `viewer_shows_the_engines_numbers` (the viewer's two musts, owner 2026-10-18: it never writes the books, and every figure it shows is correct): on random books, every report page, account register, entry and Needs-attention count the viewer renders equals the engine's report for the same books and dates, extending `tests/view_model.rs` from the overview to every page. No mutation runs on the viewer. - `viewer_shows_the_engines_numbers` (the viewer's two musts, owner 2026-10-18: it never writes the books, and every figure it shows is correct): on random books, every report page, account register, entry and Needs-attention count the viewer renders equals the engine's report for the same books and dates, extending `tests/view_model.rs` from the overview to every page. No mutation runs on the viewer.
- `viewer_without_server` (with the server stopped or failing to start, the viewer still opens and follows the books from disk; "Restore…" and "New books…" say why they can't run). - `viewer_without_server` (with the server stopped or failing to start, the viewer still opens and follows the books from disk; "Restore…" and "New books…" say why they can't run).
@@ -1187,7 +1195,7 @@ Run from clients that hold only a URL and a credential: Claude Code started in `
7. **The books are open only while something holds them** (D-21, D-23: the app, an agent session, a `serve` terminal; no background service). 7. **The books are open only while something holds them** (D-21, D-23: the app, an agent session, a `serve` terminal; no background service).
- An agent session opens them itself through its bridge, so it needs no reconnect. A client connected over HTTP instead of the bridge holds no lease and needs them already open. - An agent session opens them itself through its bridge, so it needs no reconnect. A client connected over HTTP instead of the bridge holds no lease and needs them already open.
- The viewer reads the data root directly, so it works even if the server can't start; "Restore…" and "New books…" say why they can't run then. - The viewer reads the data root directly, so it works even if the server can't start; "Restore…" and "New books…" say why they can't run then.
- Backups run only while the books are open. The books change only then, so nothing goes unbacked; the archive process pushes anything left over, and a drill that fell due, at the next open. - Backups run only while the books are open. The books change only then, so nothing goes unbacked; the server pushes anything left over, and a drill that fell due, at the next open.
- The host outlives the opener that started it, by design, until the last lease ends; it is never left running with no holder for more than 10 s. A crash or power loss is what the append-only journal already survives. - The host outlives the opener that started it, by design, until the last lease ends; it is never left running with no holder for more than 10 s. A crash or power loss is what the append-only journal already survives.
8. **The OAuth authorization server is security-critical code written in-house** (rmcp has no server-side auth). It is needed only on the self-host remote path. 8. **The OAuth authorization server is security-critical code written in-house** (rmcp has no server-side auth). It is needed only on the self-host remote path.
- Small surface: passkeys only, no password grant. - Small surface: passkeys only, no password grant.
@@ -1238,8 +1246,9 @@ The owner decisions of 2026-10-16 (at the top of this document) replace the form
| D-27 | The agent dock and books homes (owner, 2026-10-19: bundle the agents "then it'd be a one stop shop … a chat window maybe full span across the bottom"; "Yeah I like the idea of the embedded alacritty launching claude/codex … maybe MVP defaults to claude"; "you'd want like a 'books home' dir somewhere that makes sense so it has a place to work"; "add it to the plan!") | The app docks a terminal (alacritty_terminal; ConPTY on Windows) along the bottom of its window and runs Claude Code (the default) or Codex in it, found or installed through the vendors' installers, never bundled; started by the viewer's helper, in the book's **books home**: a visible folder per book holding only the agent's pointer and configuration for that book, never the data root. Trust in that folder is the person's to give. **Never required** (owner, 2026-10-19: "we wouldn't want to prevent people from using their OWN AI agent. They should never HAVE to use our built in tty+claude session! Ideally they could even use the claude.ai browser version to connect to it. And of course, eventually we'd host the entire MCP (and gui!) remotely ... use any agent they prefer to remote connect into that.. and they'd view the GUI in a browser."): the books stay fully usable from any agent the person runs themselves, locally or (P9) remotely, claude.ai in a browser included; the dock's one extra is pasting the app's suggested instructions straight into its session. A hosted MCP and a browser viewer are the later service (section 11). P11 (the agent dock and books homes). | | D-27 | The agent dock and books homes (owner, 2026-10-19: bundle the agents "then it'd be a one stop shop … a chat window maybe full span across the bottom"; "Yeah I like the idea of the embedded alacritty launching claude/codex … maybe MVP defaults to claude"; "you'd want like a 'books home' dir somewhere that makes sense so it has a place to work"; "add it to the plan!") | The app docks a terminal (alacritty_terminal; ConPTY on Windows) along the bottom of its window and runs Claude Code (the default) or Codex in it, found or installed through the vendors' installers, never bundled; started by the viewer's helper, in the book's **books home**: a visible folder per book holding only the agent's pointer and configuration for that book, never the data root. Trust in that folder is the person's to give. **Never required** (owner, 2026-10-19: "we wouldn't want to prevent people from using their OWN AI agent. They should never HAVE to use our built in tty+claude session! Ideally they could even use the claude.ai browser version to connect to it. And of course, eventually we'd host the entire MCP (and gui!) remotely ... use any agent they prefer to remote connect into that.. and they'd view the GUI in a browser."): the books stay fully usable from any agent the person runs themselves, locally or (P9) remotely, claude.ai in a browser included; the dock's one extra is pasting the app's suggested instructions straight into its session. A hosted MCP and a browser viewer are the later service (section 11). P11 (the agent dock and books homes). |
| D-28 | Who does what, locally (owner, 2026-10-19: "Just have the agent ask for them and the user replies. The interface is through claude/codex and that's that. Don't fucking stray from it. TIN, W-9 is NOT critical or super secret PII." and "for downloading the CPA package or other assets, I don't mind if that can happen through the GUI app ... the GUI is for READING the data. The claude/codex is for MUTATAING (and reading) the data!!!") | **Two interfaces, no third.** Every change to the books goes through the agent (Claude Code or Codex) over the MCP; the agent asks the human for what it needs, TINs and W-9s included, and seals them in the vault (`books.set_tin`, `books.set_w9`): no human web pages, terminal commands or open ports for that locally. Reading and downloading (the CPA package and other exports, receipts, documents) happen in the viewer, which still never writes the books. This supersedes, for local books, the `/h/` pages of 3.3, 4.7 and P7's TIN entry: I-7's "the agent never holds a full TIN" becomes "a TIN passes through the agent only on its way into the vault; no result shows one". `/h/` pages remain for self-hosted books reached remotely (P9). | | D-28 | Who does what, locally (owner, 2026-10-19: "Just have the agent ask for them and the user replies. The interface is through claude/codex and that's that. Don't fucking stray from it. TIN, W-9 is NOT critical or super secret PII." and "for downloading the CPA package or other assets, I don't mind if that can happen through the GUI app ... the GUI is for READING the data. The claude/codex is for MUTATAING (and reading) the data!!!") | **Two interfaces, no third.** Every change to the books goes through the agent (Claude Code or Codex) over the MCP; the agent asks the human for what it needs, TINs and W-9s included, and seals them in the vault (`books.set_tin`, `books.set_w9`): no human web pages, terminal commands or open ports for that locally. Reading and downloading (the CPA package and other exports, receipts, documents) happen in the viewer, which still never writes the books. This supersedes, for local books, the `/h/` pages of 3.3, 4.7 and P7's TIN entry: I-7's "the agent never holds a full TIN" becomes "a TIN passes through the agent only on its way into the vault; no result shows one". `/h/` pages remain for self-hosted books reached remotely (P9). |
| D-29 | The full-TIN outputs, from both doors (owner, 2026-10-19, asked whether the agent's CPA export should go away once the viewer has one: "Both WITH TINs. Stop acting like TINS MATTER SO MUCH THEY DONT") | **The agent and the viewer both make them, TINs included.** `books.export_cpa_package` (TINs by default), `books.vendor_1099` with `include_tins` and `books.get_w9` keep working for the agent; their files are made by the server and handed over by a command, `wanderbooks download`, like a receipt too large for a call (no path parameter, nothing in the agent's context). The viewer's menu makes the same files into Downloads. Over stdio (the development transport) the file comes back inline. TINs get no handling beyond the vault: no passphrase, no extra prompts, no special cleanup. | | D-29 | The full-TIN outputs, from both doors (owner, 2026-10-19, asked whether the agent's CPA export should go away once the viewer has one: "Both WITH TINs. Stop acting like TINS MATTER SO MUCH THEY DONT") | **The agent and the viewer both make them, TINs included.** `books.export_cpa_package` (TINs by default), `books.vendor_1099` with `include_tins` and `books.get_w9` keep working for the agent; their files are made by the server and handed over by a command, `wanderbooks download`, like a receipt too large for a call (no path parameter, nothing in the agent's context). The viewer's menu makes the same files into Downloads. Over stdio (the development transport) the file comes back inline. TINs get no handling beyond the vault: no passphrase, no extra prompts, no special cleanup. |
| D-30 | Q-1 answered: where the backups go (owner, 2026-10-19, asked "Where should the books' automatic backups go?" with a private git repo, a folder or other disk, or both: chose "Both") | **Two targets.** The archive pushes to a private git repository the owner names and also copies to a folder the owner names (an external disk or a synced folder); both hold only the encrypted archive. The repository's address and the folder are asked for when backups are first set up (slice 7, or at the cutover for the first company), and kept as the book's backup settings. | | D-30 | Q-1 answered: where the backups go (owner, 2026-10-19, asked "Where should the books' automatic backups go?" with a private git repo, a folder or other disk, or both: chose "Both") | **Two targets.** The archive pushes to a private git repository the owner names and also copies to a folder the owner names (an external disk or a synced folder); both hold the archive (not encrypted since D-31). The repository's address and the folder are asked for when backups are first set up (slice 7, or at the cutover for the first company), and kept as the book's backup settings. |
| D-31 | Backups are not encrypted (owner, 2026-10-19, asked where to keep the backup's recovery key: "Why encrypt them? How about we let the backup system handle the security???? You're over complicating everything.") | **A backup is the book as it is, in git.** No age encryption, no archive key, no recovery key: the archive is a plain git repository of the book's journal, receipts, vault (as the vault keeps it) and keys, one commit per backup, pushed to the owner's private git repository and to a git repository in the owner's folder (D-30). The destinations are trusted to keep it private. This replaces 3.7's and 3.4's age encryption on push and 3.5's archive key. | | D-31 | Backups are not encrypted (owner, 2026-10-19, asked where to keep the backup's recovery key: "Why encrypt them? How about we let the backup system handle the security???? You're over complicating everything.") | **A backup is the book as it is, in git.** No age encryption, no archive key, no recovery key: the archive is a plain git repository of the book's journal, receipts, vault (as the vault keeps it) and keys, one commit per backup, pushed to the owner's private git repository and to a git repository in the owner's folder (D-30). The destinations are trusted to keep it private. This replaces 3.7's and 3.4's age encryption on push and 3.5's archive key. |
| D-32 | No self-confinement (owner, 2026-10-20, asked how much of the planned Landlock/seccomp sandboxing of the server and the viewer, with the viewer's helper process, to build now that the server runs git for its backups: chose "None") | **No Landlock, no seccomp, no helper process for confinement.** The viewer stays read-only by design and by test (it never writes the books; its menu hands work to the server through the lease socket); agents still reach the books only through the MCP. Isolation is reported everywhere as `same_user (advisory)`. This replaces the self-confinement of P2's scope, 3.9's confined modes and their helper, and the archive-vs-Landlock decision row. |
### 10.2 Deferred ### 10.2 Deferred
@@ -1270,7 +1279,7 @@ These are the only open owner questions. Ask them just before the move, one at a
- the agent's own notes that name the old paths (its memory), rewritten to the new arrangement. - the agent's own notes that name the old paths (its memory), rewritten to the new arrangement.
The cutover's check adds: the project folder holds none of the above, and `wanderbooks doctor` is clean. The cutover's check adds: the project folder holds none of the above, and `wanderbooks doctor` is clean.
*For the engineer:* Q-1 was D-3, Q-2 was D-4 and Q-3 was D-2. Q-1's technical side is settled: `archive.git` is seeded from the existing git history, filtered to the journal and blob paths, includes the vault ciphertext, and is age-encrypted as a whole on push (3.4, 5.5). *For the engineer:* Q-1 was D-3, Q-2 was D-4 and Q-3 was D-2. Q-1's technical side is settled: `archive.git` is seeded from the existing git history, filtered to the journal and blob paths, includes the vault ciphertext, and is age-encrypted as a whole on push (3.4, 5.5). (Since D-31 not encrypted, and slice 7 begins each book's archive at its first backup: whether to seed it from the old history is the cutover's question.)
--- ---