From f043db35ca29600c84428c3ae590a519b22ecf12 Mon Sep 17 00:00:00 2001 From: Stephen Waits Date: Thu, 29 Oct 2026 14:03:39 -0600 Subject: [PATCH 1/6] docs(plan): wave 1 complete (P4, U0e landed); wave 2 building Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01HS8WZziDnyxunC23MUxrLp --- plan/STATUS.md | 6 ++++-- plan/served.md | 14 +++++++------- 2 files changed, 11 insertions(+), 9 deletions(-) diff --git a/plan/STATUS.md b/plan/STATUS.md index 416174a..d83c4ee 100644 --- a/plan/STATUS.md +++ b/plan/STATUS.md @@ -6,9 +6,11 @@ session is the EM and launches Sonnet subagents in parallel worktrees, one per unit, briefed from `plan/HANDOFF.md`; it reviews each against the unit's Done-when, merges with a compile and the unit's own tests -(D-60), and runs one exhaustive gate per wave. Wave 1 so far: U0d +(D-60), and runs one exhaustive gate per wave. Wave 1, all landed: U0d (`39c76d3f`), P11a step 0 (`5e220cca`; see below), S4.5 (`ac0c949b`: -P4b S4 is complete), P13 steps 4-5 (`d839e709`); P4 is still building. +P4b S4 is complete), P13 steps 4-5 (`d839e709`), U0e (`75882c5e`), P4 +(`b97234ac`, schema 19). Wave 2 is building: P5, P8, P11a's trial, +update/rollback and Linux packages, P11c. **Two decisions the wave forced.** D-62: native Windows is a port, not cfg fixes (the host, the MCP transport, the store's locking and diff --git a/plan/served.md b/plan/served.md index 5124f44..9aa8303 100644 --- a/plan/served.md +++ b/plan/served.md @@ -675,7 +675,7 @@ The `WANDERBOOKS_TOML` template (`ledger-cli/src/main.rs:237-347`) becomes the e | P2 cutover (about 10 min) | 1. Ask the owner the two open cutover questions (section 10.3: Q-1 backups, Q-3 old copies; Q-2 is answered), one at a time, in plain English.
2. Quit the agent sessions and the GUI.
3. `wanderbooks setup`, with no sudo: it creates the data root (0700) and asks for the owner passphrase. Nothing is installed to run in the background (D-21).
4. Nothing to start: each command below opens the books while it runs, and afterwards the app or Claude opens them (D-23).
5. `wanderbooks books adopt /home/swaits/Documents/the first company-Books/books-data --id fernhill` (archive target from Q-1; old copies moved per Q-3).
6. `wanderbooks connect claude-code --book fernhill` (removes the old stdio registration made by `just mcp-add`, adds the bridge's stdio registration, `wanderbooks mcp --book fernhill`).
7. `wanderbooks connect codex --book fernhill`.
8. `wanderbooks gui fernhill`.
9. Remove `justfile`, `.claude/settings.json` hook, `.codex/hooks.json`, `scripts/` (archived under the data root's `legacy/`); `AGENTS.md`/`CLAUDE.md` become a 3-line pointer ("connect the wanderbooks MCP; call books.start").
10. The `company/` records are already filed (Q-2, D-20); the local copies stay until P6. | `books.verify` over MCP equals H. Goldens byte-equal. `books.start` works from Claude Code started in `~` and in `/tmp/empty`, and reports `isolation: "same_user (advisory)"`. No connect text, tool text, handbook section or result in a recorded cutover session names the data root. No step asked for sudo. If Q-3 is "move": the project folder holds no journal, blobs, vault or git history containing them. Viewer updates within 1 s. With every holder closed the books close cleanly (`books.verify` ok with the same head at the next open), and a Claude Code session started with the books closed opens them by itself, with no reconnect. **Rollback (nothing posted on the server):** close the books (`wanderbooks close`), move `legacy/fernhill-/` back to the project folder (or use the untouched source if Q-3 was "no"), and re-register the P1 binary's stdio server as before cutover. **Rollback (entries posted):** `wanderbooks books export-legacy fernhill` writes a data dir with the same head check, then the same re-registration | | P3 | Defect fixes change reports (cash flow, 1099 sections, asset fields, digest). Each difference is hand-adjudicated in the P3 change log. 1099 classification and 704(c) fields are shown to the CPA. | Change log signed off | | P4b (one upgrade, D-52) | After S7 and P4 land: `wanderbooks books upgrade` (a verified backup first); then the owner sets `invoicing.*` (prefix, first number, default terms, payment methods), each party's email and terms, and the goldens are compared (the figures must not move). | `books.verify` ok; `books.control_check` agrees; the goldens unchanged to the cent | -| P4 | `list_drafts`. Leftover drafts are discarded or set to `awaiting_evidence`. | Zero unexplained drafts | +| P4 | `list_drafts`. Leftover drafts are discarded or set to `awaiting_evidence`. | landed 2026-10-29 (`b97234ac`; schema 19; `Store::post_entry` itself is not gated, the server's `post_entry_checked` is: see the As-built block) | P6 | • `consolidate_duplicates` resolves the 11 redundant records (preview first).
• Extraction is backfilled for all 66 attachments.
• Receipt numbers 1000–1029 are annotated from the "Receipt ID" in each description. Anomalies become annotations: 1009 unissued; 1011/1020 byte-identical pairs (merged); 43/45/47 have no ID. The next number is derived as 1030.
• Instruments: Quillmere, joint Amex, member cards, Chase Ink 2050.
~~• `verify-sheet.py`'s tables become a `mapping` convention, then `compare_reference` runs against attachment 37 as of 2026-10-11.
• `drive-manifest.json` text goes to `record_inventory`; the Drive connector listing goes to `check_inventory`.~~ → not built (8.4): the final `verify-sheet` run is a filed workpaper (P2 pre-cutover) and P10 cites it. | Every attachment's `get_receipt` default answer under 8 KiB; the 11 duplicates consolidated; receipt numbers annotated | | ~~P7~~ | ~~`wanderbooks vault init` … `keys/legacy.key` shredded~~ → D-48: nothing; the v1 key stays in the books and in every backup (D-31) | — | | P10 | Re-audit (8.2 P10; `docs/runbooks/re-audit.md`). | the exit criteria there | @@ -820,23 +820,23 @@ One row per unit of work. A unit is what one engineer takes into one git worktre | U0b | The attention engine over `PackageBooks` | landed 2026-10-27 (`26d3a994`) | none | U0a, U0c, S4.1, S4.2 | none | 4-6 | | U0c | `docs/testing.md` sections for S1-B, S2, S3; `docs/mcp-interface.md` cleanup of stale P4b notes | landed 2026-10-26 (`198aa2a1`) | none | everything | none | 0.5-1 | | U0d | The reader crate: `extract.rs` moves out of `ledger-import` so `ledger-store` sheds the typesetter (Q-9 → D-61) | landed 2026-10-29 (`39c76d3f`) | P6 (landed) | everything | none | 0.5 | -| U0e | Tokens in a file under the data root; the OS keyring stores removed (D-63) | not started; **next**, beside P4 | none | everything but P11c | none | 1-2 | +| U0e | Tokens in a file under the data root; the OS keyring stores removed (D-63) | landed 2026-10-29 (`75882c5e`) | none | everything but P11c | none | 1-2 | | P4b S4 | `documents_v2`: numbers, terms, line accounts, snapshots, preview and confirm, the issued PDF filed, party email and terms, invoicing settings | in progress: S4.1-S4.3 landed 2026-10-27 (`d19fb464`); S4.4 landed 2026-10-29 (`d329bd6f`, gate 2,883 tests green); S4.5 landed 2026-10-29 (`ac0c949b`): P4b S4 is complete | S4.1 and S4.2 need nothing; S4.3 needs S4.1, S4.2; S4.4 needs U0a, S4.3; S4.5 needs S4.4 | S5 (after S4.2), S7 (core after S1), P6, P9a, P11a, P13 | `documents_v2` (one upgrade after S7, D-52) | 12-16 | | P4b S5 | Sent tracking | landed 2026-10-29 (`3a3f03d8`; schema 17; law 35) | S1; S4.2 for the defaults (works without it, see the unit) | S4.3 onward, S6 | `documents_v2` | 3-4 | | P4b S6 | Paying on terms: multi-document payments, discounts, the processor fee, bills due, aging exports | landed 2026-10-29 (`be580598`; no schema change; law 36); `confirmed` deferred to P4 | S4.1 (`Terms`), U0a | S5, S7, P6 | `documents_v2` | 8-10 | | P4b S7 | Bad debts (write-offs) | landed 2026-10-29 (`7ebeeed2`; law 37; `just prove` green) | S1; S4.3 for the same-call evidence attach; U0a | S4.4 onward, S5, S6 | `documents_v2` | 4-6 | | P4 | Draft states, the evidence gate in one place, the preview digest, the digest tool | not started | S4-S7 merged (one sweep over every confirm tool), U0b | P6, P9a, P11a, P13 | `drafts_v2` (same upgrade as `documents_v2`, D-52) | 8-10 | -| P5 | Agent surface complete: the result envelope, prompts, `books.start` v2, journal history, agent evals | not started | P4, U0b; the envelope step is single-owner and merges when no other MCP branch is open | P6, P9a, P11a, P13 | `source_revisions` | 10-12 | +| P5 | Agent surface complete: the result envelope, prompts, `books.start` v2, journal history, agent evals | in progress (wave 2, branch `P5`); merges after P8 | P4, U0b; the envelope step is single-owner and merges when no other MCP branch is open | P6, P9a, P11a, P13 | `source_revisions` | 10-12 | | P6 | Evidence pipeline: extraction, dedup, annotations, company record categories | landed 2026-10-29 (`9351f89d`, schema 18, with the integration fixes `b66fc893`, `0cbb49b6`, `fb83af59`, `6c26160e`, `bfd453b6`); the `ledger-store` → `ledger-import` dependency is Q-9 | U0a, U0b | S4-S7, P4, P5 (not its envelope step), P9a, P11a, P13 | `attachment_text` | 8-10 | | P7 | The TIN scanner's separators; vault v2 closed (D-48) | landed 2026-10-26 (`3e65c83f`) | none | everything | none | 1 | -| P8 | Audit and attestations | not started | P6 (extraction), P4 (drafts), U0b | P9a, P11a, P13 | `audits` | 10-12 | +| P8 | Audit and attestations | in progress (wave 2, branch `P8`) | P6 (extraction), P4 (drafts), U0b | P9a, P11a, P13 | `audits` | 10-12 | | P9a | Self-host recipe, `books adopt` move with the head check, static bearer over a tunnel, the two-user CI job | landed 2026-10-27 (`c8541ea9`); Done-when 1 verified 2026-10-29 (the two-user job green on `fb83af59`, after `79fb7ff2` gave the server account the fixture copies); Done-when 2 (the recipe on a clean VM) is the owner's | none (code); a second machine to try it on | everything | none | 3-4 | | P9b | claude.ai through an external identity provider (D-7 b, Q-6) | not planned until Q-6 is answered | P9a | | | | | P9c | Remote viewer replica, invitations | not planned (D-51) | | | | | | P10 | The re-audit runbook and its CI mirror; the owner runs the re-audit | not started | P5, P6, P8 | P11a, P13 | none | 1 (+ owner days) | | P11a | Packaging and updates: step 0 the platform groundwork, the trial, `wanderbooks update`/`rollback`, Linux packages, signing, macOS and Windows bundles | in progress: step 0 landed 2026-10-29 (`5e220cca`) for the macOS keyring store and the portable CI job; **the keyring stores go away under D-63 (unit U0e)** and native Windows is its own unit P11c (D-62); steps 1-7 open, macOS and Linux first | none (code); certificates and keys from the owner for the signing steps; Q-4 before the manifest step | everything | none | 20-25 | | P11b | The agent dock and books homes | not started | P11a (an installed app to dock into) | P12 | none | 8-10 | -| P11c | The native Windows port: named pipes with the SID check, owner-only DACLs, file locking, the `%LOCALAPPDATA%` data root, the Windows CI job made gating (D-62) | not started; after U0e, beside P11b | U0e; P11a step 0 (the portable job) | P11b, P12 | none | 5-8 | +| P11c | The native Windows port: named pipes with the SID check, owner-only DACLs, file locking, the `%LOCALAPPDATA%` data root, the Windows CI job made gating (D-62) | in progress (wave 2, branch `P11c`) | U0e; P11a step 0 (the portable job) | P11b, P12 | none | 5-8 | | P12 | Max testing, again | not started | everything else merged | | none | 2-3 (+ machine time) | | P13 | bookspec: checker crate, `Engine` trait, rule ids, the spec text, the runner | in progress: steps 1-3 landed 2026-10-26 (`579d0eed`); steps 4-5 landed 2026-10-29 (`d839e709`; ten stated rules have no example yet because the schema has no event or query for them, listed in the As-built block); step 6 (the spec's own public repository) is the owner's call | none for steps 1-4; the engine stable for step 5 | everything | none | 15-20 | @@ -1306,7 +1306,7 @@ Each unit follows the 8.0.4 template. "A-n" is an owner answer numbered in 8.3.1 #### U0e: tokens in a file, the OS keyring removed (D-63) -1. **Status:** not started; next, beside P4 (it touches `ledger-host`, the CLI tests, the two-user job and the deploy docs; P4 touches none of them). +1. **Status:** landed 2026-10-29 (`75882c5e`, branch `U0e` at `d8816f9e`). 2. **Depends on:** none. No owner go-ahead (tokens are not in the journal). 3. **Feature and schema:** none. 4. **Build:** @@ -1466,7 +1466,7 @@ Each unit follows the 8.0.4 template. "A-n" is an owner answer numbered in 8.3.1 #### P4: draft states, the evidence gate in one place, the preview digest, the digest tool -1. **Status:** not started. Rewritten 2026-10-26 against P4b S1-S3 and D-47 (reversals stay posted; no reversal drafts). +1. **Status:** landed 2026-10-29 (`b97234ac`, branch `P4` at `6a66a285`; schema 19). Rewritten 2026-10-26 against P4b S1-S3 and D-47 (reversals stay posted; no reversal drafts). 2. **Depends on:** S4-S7 merged (step 4 is one sweep over every confirm tool, including S4's and S6's); U0b (attention items). Go-ahead: `drafts_v2`, declared in the same owner upgrade as `documents_v2` (D-52). 3. **Feature and schema:** feature `drafts_v2`; new event types `entry.draft_state_set { id, state, note }`, `entry.draft_discarded { id, reason }` (R-B); provisional schema 19: `entries.draft_state`, `entries.posted_seq`. 4. **Build:** From 23f9e9395b9e651f6b077fdbdd88cb85c772dc79 Mon Sep 17 00:00:00 2001 From: Stephen Waits Date: Thu, 29 Oct 2026 14:39:13 -0600 Subject: [PATCH 2/6] feat(cli): wanderbooks update and rollback (P11a step 2) `wanderbooks update` reads a release manifest (default: GitHub Releases' fixed address, D-26), refuses anything not higher than this version, downloads the platform's file and accepts it only if its sha256 matches the manifest AND its minisign signature verifies against the release key compiled into the program (WANDERBOOKS_RELEASE_PUBKEY at build time; a build without one updates from nothing but a file:// feed whose key the owner names with --trusted-key, so a local release can be tried). The verified file is staged under /updates//, the running version is kept under updates/previous/ with the journal format it understood, and the program file is replaced by a rename of a complete copy made beside it (a reader sees the old file or the new, never a part). The new program must answer --help; if not, the old one is put back. Nothing tells the books to stop: the host already notices a replaced executable and restarts onto it, finishing calls in progress (host.rs INVARIANT 7). `wanderbooks rollback` swaps the kept version back, and refuses while any book declares a journal format or feature the kept version does not know (plan 3.9: after the owner consents to a newer format the only way back is the pre-upgrade restore). A package-managed install (a package ships share/wanderbooks/install-method beside bin/) refuses both and says to use the package manager. An AppImage replaces the file named by $APPIMAGE. Tests (crates/ledger-cli/tests/update.rs): update_is_atomic, failed_update_rolls_back, unsigned_update_refused (another key, other bytes, a sha that differs, no signature, not a signature), bridge_survives_update, update_waits_for_a_running_call, package_managed_install_refuses_update, the kept-version and format-gate tests; unit tests in ledger-host/src/update.rs. Dependencies: minisign-verify 0.3 in ledger-host (what cargo-packager's updater uses to check release files; no dependencies of its own) and minisign 0.10 as a ledger-cli dev-dependency (the tests sign with a key they make). sha2 and the two minisign crates are built optimized in debug and test builds: the tests hash a debug build's program. The release profile strips symbols, for the packages (P11a step 3). Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01HS8WZziDnyxunC23MUxrLp --- Cargo.lock | 101 +++- Cargo.toml | 16 + crates/ledger-cli/Cargo.toml | 14 + crates/ledger-cli/src/main.rs | 9 +- crates/ledger-cli/tests/update.rs | 765 +++++++++++++++++++++++++++++ crates/ledger-host/Cargo.toml | 1 + crates/ledger-host/src/lib.rs | 2 + crates/ledger-host/src/update.rs | 774 ++++++++++++++++++++++++++++++ 8 files changed, 1672 insertions(+), 10 deletions(-) create mode 100644 crates/ledger-cli/tests/update.rs create mode 100644 crates/ledger-host/src/update.rs diff --git a/Cargo.lock b/Cargo.lock index 8ea4bb4..bce5dbd 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -130,7 +130,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1973cfbc1a2daf9cf550e74e1f088c28e7f7d8c1e1418fb6c9dc5184b7e84c99" dependencies = [ "crypto-common 0.2.2", - "inout", + "inout 0.2.2", ] [[package]] @@ -139,7 +139,7 @@ version = "0.9.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "35f0f96ce78e38c3dc6d8948aa8163d06385be74000f3c7a95bf1eef35d3ea32" dependencies = [ - "cipher", + "cipher 0.5.2", "cpubits", "cpufeatures 0.3.1", ] @@ -985,7 +985,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "65c35e4b699c7e15ccbe7ee35c005e4fc0a278d22238a2857e6ce2dadeda1b06" dependencies = [ "cfg-if", - "cipher", + "cipher 0.5.2", "cpufeatures 0.3.1", "rand_core 0.10.1", ] @@ -998,7 +998,7 @@ checksum = "9b89e1c441e926b9c82a8d023f6e1b7ae0adcfaa7d621814e4d60789bac751cb" dependencies = [ "aead", "chacha20", - "cipher", + "cipher 0.5.2", "poly1305", ] @@ -1061,6 +1061,16 @@ dependencies = [ "half", ] +[[package]] +name = "cipher" +version = "0.4.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "773f3b9af64447d2ce9850330c473515014aa235e6a783b02db81ff39e4a3dad" +dependencies = [ + "crypto-common 0.1.7", + "inout 0.1.4", +] + [[package]] name = "cipher" version = "0.5.2" @@ -1069,7 +1079,7 @@ checksum = "e8cf2a2c93cd704877c0858356ed03480ff301ee950b43f1cbe4573b088bfa6c" dependencies = [ "block-buffer 0.12.1", "crypto-common 0.2.2", - "inout", + "inout 0.2.2", ] [[package]] @@ -1450,6 +1460,12 @@ dependencies = [ "memchr", ] +[[package]] +name = "ct-codecs" +version = "1.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fd9f3db6f2cfef61c10613071955154ffdc9e515daebff26de4b54e35038fdd" + [[package]] name = "ctutils" version = "0.4.3" @@ -1592,6 +1608,7 @@ checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" dependencies = [ "block-buffer 0.10.4", "crypto-common 0.1.7", + "subtle", ] [[package]] @@ -3011,6 +3028,15 @@ version = "0.4.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70" +[[package]] +name = "hmac" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6c49c37c09c17a53d937dfbb742eb3a961d65a994e6bcdcf37e7399d0cc8ab5e" +dependencies = [ + "digest 0.10.7", +] + [[package]] name = "hmac" version = "0.13.0" @@ -3464,6 +3490,15 @@ dependencies = [ "libc", ] +[[package]] +name = "inout" +version = "0.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "879f10e63c20629ecabbb64a8010319738c66a5cd0c29b02d63d272b03751d01" +dependencies = [ + "generic-array", +] + [[package]] name = "inout" version = "0.2.2" @@ -3848,6 +3883,7 @@ dependencies = [ "ledger-store", "ledger-vault", "libc", + "minisign", "proptest", "reqwest", "rmcp", @@ -3932,6 +3968,7 @@ dependencies = [ "ledger-mcp", "ledger-store", "libc", + "minisign-verify", "reqwest", "rpassword", "serde", @@ -4260,6 +4297,24 @@ dependencies = [ "unicase", ] +[[package]] +name = "minisign" +version = "0.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3062dbbf4d8f57899e2252804f7512a4fd89c8211c58fa6c786d38794a0c809d" +dependencies = [ + "ct-codecs", + "getrandom 0.4.3", + "rpassword", + "scrypt", +] + +[[package]] +name = "minisign-verify" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "871285dc19d8d0ebe0eef3d0e99a205f2a71363b122632cbbfa4a6c370a960ce" + [[package]] name = "miniz_oxide" version = "0.8.9" @@ -5016,6 +5071,16 @@ version = "0.2.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2ee67f1008b1ba2321834326597b8e186293b049a023cdef258527550b9935b4" +[[package]] +name = "pbkdf2" +version = "0.12.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8ed6a7761f76e3b9f92dfb0a60a6a6477c61024b775147ff0973a02653abaf2" +dependencies = [ + "digest 0.10.7", + "hmac 0.12.1", +] + [[package]] name = "pbkdf2" version = "0.13.0" @@ -5023,7 +5088,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "112d82ceb8c5bf524d9af484d4e4970c9fd5a0cc15ba14ad93dccd28873b0629" dependencies = [ "digest 0.11.3", - "hmac", + "hmac 0.13.0", ] [[package]] @@ -6255,6 +6320,15 @@ version = "1.0.23" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f" +[[package]] +name = "salsa20" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "97a22f5af31f73a954c10289c93e8a50cc23d971e80ee446f1f6f7137a088213" +dependencies = [ + "cipher 0.4.4", +] + [[package]] name = "same-file" version = "1.0.6" @@ -6311,6 +6385,17 @@ version = "1.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" +[[package]] +name = "scrypt" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0516a385866c09368f0b5bcd1caff3366aace790fcd46e2bb032697bb172fd1f" +dependencies = [ + "pbkdf2 0.12.2", + "salsa20", + "sha2 0.10.9", +] + [[package]] name = "security-framework" version = "3.7.0" @@ -9314,12 +9399,12 @@ dependencies = [ "deflate64", "flate2", "getrandom 0.4.3", - "hmac", + "hmac 0.13.0", "indexmap", "lzma-rust2", "memchr", "nix", - "pbkdf2", + "pbkdf2 0.13.0", "ppmd-rust", "sha1", "time", diff --git a/Cargo.toml b/Cargo.toml index dfc8621..2d380a2 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -19,3 +19,19 @@ opt-level = 3 [profile.dev.package.blake2] opt-level = 3 + +# `wanderbooks update` hashes and verifies a whole program (P11a); its +# tests do it over a debug build's hundreds of megabytes. +[profile.dev.package.sha2] +opt-level = 3 + +[profile.dev.package.minisign] +opt-level = 3 + +[profile.dev.package.minisign-verify] +opt-level = 3 + +# The program that ships in the packages (P11a): symbols stripped, so the +# AppImage, .deb and .rpm are not carrying debug tables. +[profile.release] +strip = "symbols" diff --git a/crates/ledger-cli/Cargo.toml b/crates/ledger-cli/Cargo.toml index bd2739a..30242f2 100644 --- a/crates/ledger-cli/Cargo.toml +++ b/crates/ledger-cli/Cargo.toml @@ -45,6 +45,20 @@ serde_json = "1" sha2 = "0.10" tempfile = "3" zip = "9.0.1" +minisign = "0.10.0" [lints] workspace = true + +# The .rpm (P11a step 3), built by cargo-generate-rpm from the release +# program: `just package-linux`. Paths are relative to the workspace root. +[package.metadata.generate-rpm] +name = "wanderbooks" +summary = "Double-entry bookkeeping your agent keeps and you can read" +license = "MIT OR Apache-2.0" +url = "https://wanderbooks.ai" +assets = [ + { source = "target/release/wanderbooks", dest = "/usr/bin/wanderbooks", mode = "0755" }, + { source = "deploy/packaging/install-method-rpm", dest = "/usr/share/wanderbooks/install-method", mode = "0644" }, + { source = "deploy/packaging/wanderbooks-static.desktop", dest = "/usr/share/applications/wanderbooks.desktop", mode = "0644" }, +] diff --git a/crates/ledger-cli/src/main.rs b/crates/ledger-cli/src/main.rs index 9ea981f..e308782 100644 --- a/crates/ledger-cli/src/main.rs +++ b/crates/ledger-cli/src/main.rs @@ -28,6 +28,7 @@ fn print_usage() { eprintln!(" wanderbooks upload --book ID [--token NAME] UPLOAD [--] FILE"); eprintln!(" wanderbooks download --book ID [--token NAME] ATTACHMENT --to FOLDER"); eprintln!(" wanderbooks serve | close | doctor"); + eprintln!(" wanderbooks update [--check] | rollback"); eprintln!(); eprintln!(" setup make the books' private folder (only your account can open it)"); eprintln!(" gui open the live, read-only view of the books (a book of the data root)"); @@ -43,6 +44,7 @@ fn print_usage() { eprintln!(" serve hold the books open until Ctrl-C"); eprintln!(" close close the books now: everything holding them lets go"); eprintln!(" doctor whether the books are open, what holds them, what went wrong"); + eprintln!(" update bring this program to the newest release (`rollback` goes back)"); } /// The viewer's arguments for `wanderbooks gui ARGS` (plan 3.6): a lone @@ -329,6 +331,8 @@ fn run_local(mode: &str, args: Vec) -> i32 { "token" => ledger_host::commands::token(&prog, args), "connect" => ledger_host::commands::connect(&prog, args), "passphrase" => ledger_host::commands::passphrase(&prog, args), + "update" => ledger_host::update::update(&prog, env!("CARGO_PKG_VERSION"), args), + "rollback" => ledger_host::update::rollback(&prog, args), _ => ledger_host::commands::books(&prog, args), }; match result { @@ -409,7 +413,7 @@ fn main() { Some("server") => std::process::exit(run_server(all.collect())), Some( mode @ ("host" | "mcp" | "serve" | "close" | "books" | "doctor" | "token" | "connect" - | "passphrase"), + | "passphrase" | "update" | "rollback"), ) => { let mode = mode.to_string(); std::process::exit(run_local(&mode, all.collect())) @@ -435,7 +439,8 @@ fn main() { { Some( mode @ ("gui" | "setup" | "server" | "host" | "mcp" | "serve" | "connect" | "close" - | "books" | "doctor" | "token" | "passphrase" | "upload" | "download"), + | "books" | "doctor" | "token" | "passphrase" | "update" | "rollback" + | "upload" | "download"), ) => { eprintln!("`{mode}` must come first: wanderbooks {mode} [its own flags]"); std::process::exit(2); diff --git a/crates/ledger-cli/tests/update.rs b/crates/ledger-cli/tests/update.rs new file mode 100644 index 0000000..4f121b2 --- /dev/null +++ b/crates/ledger-cli/tests/update.rs @@ -0,0 +1,765 @@ +//! `wanderbooks update` and `wanderbooks rollback` (plan P11a step 2, 3.9, +//! D-24, D-26), against a local release folder (a `file://` feed) and a +//! key the test makes. Oracles: the INVARIANTS of ledger-host's update.rs +//! and host.rs; the P11a tests update_is_atomic, failed_update_rolls_back, +//! unsigned_update_refused, bridge_survives_update, +//! update_waits_for_a_running_call, package_managed_install_refuses_update, +//! and the rollback laws (kept version, the format gate). +#![cfg(target_os = "linux")] + +mod common; + +use serde_json::{Value, json}; +use std::io::{BufRead, BufReader, Write}; +use std::path::{Path, PathBuf}; +use std::process::{Command, Stdio}; +use std::sync::mpsc; +use std::time::{Duration, Instant}; + +const KNOBS: &str = + "close_after=2500,backoff=50,exe_poll=100,init_wait=5000,queue_wait=30000,backup_quiet=300"; + +/// The platform key a manifest names, as the program computes it. +fn platform() -> String { + format!("{}-{}", std::env::consts::OS, std::env::consts::ARCH) +} + +fn sha256(bytes: &[u8]) -> String { + ledger_store::journal::sha256_hex(bytes) +} + +fn wait_for(secs: u64, mut done: impl FnMut() -> bool) -> bool { + let deadline = Instant::now() + Duration::from_secs(secs); + while Instant::now() < deadline { + if done() { + return true; + } + std::thread::sleep(Duration::from_millis(20)); + } + done() +} + +/// The program the tests run, stripped once: a debug build is hundreds of +/// megabytes, and every home holds a copy that is hashed and signed. +fn base_program() -> &'static Path { + static BASE: std::sync::OnceLock = std::sync::OnceLock::new(); + BASE.get_or_init(|| { + // Named for the build, so a run after another reuses it (the + // `sweep-test-tmp` recipe removes the `wb-` folders after a day). + let source = Path::new(env!("CARGO_BIN_EXE_wanderbooks")); + let meta = std::fs::metadata(source).unwrap(); + let stamp = meta + .modified() + .unwrap() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_secs(); + let dir = std::env::temp_dir().join(format!("wb-update-base-{}-{stamp}", meta.len())); + let base = dir.join("wanderbooks"); + if !base.is_file() { + let work = std::env::temp_dir().join(format!("wb-update-work-{}", std::process::id())); + std::fs::create_dir_all(&work).unwrap(); + let part = work.join("wanderbooks"); + std::fs::copy(source, &part).unwrap(); + // No strip tool: the program as it is. + let _ = Command::new("strip").arg("--strip-all").arg(&part).status(); + let _ = std::fs::create_dir_all(&dir); + std::fs::rename(&part, &base).unwrap(); + let _ = std::fs::remove_dir_all(&work); + } + base + }) +} + +/// A home with a data root, and a copy of the program in `bin/` that the +/// tests update. +struct Home { + dir: tempfile::TempDir, + exe: PathBuf, +} + +impl Home { + fn new(tag: &str) -> Self { + let dir = tempfile::Builder::new() + .prefix(&format!("wb-update-{tag}-")) + .tempdir() + .unwrap(); + let out = Command::new(env!("CARGO_BIN_EXE_wanderbooks")) + .args(["setup", "--home"]) + .arg(dir.path()) + .env_remove("XDG_DATA_HOME") + .output() + .unwrap(); + assert!(out.status.success(), "{out:?}"); + let bin = dir.path().join("bin"); + std::fs::create_dir(&bin).unwrap(); + let exe = bin.join("wanderbooks"); + std::fs::copy(base_program(), &exe).unwrap(); + let home = Self { dir, exe }; + home.runnable(); + home + } + + /// Wait until the copy can be run (a child another test thread forked + /// while it was written may hold it open for a moment). + fn runnable(&self) { + let deadline = Instant::now() + Duration::from_secs(10); + loop { + match Command::new(&self.exe) + .arg("--help") + .stdin(Stdio::null()) + .stdout(Stdio::null()) + .stderr(Stdio::null()) + .status() + { + Err(e) if e.kind() == std::io::ErrorKind::ExecutableFileBusy => { + assert!(Instant::now() < deadline, "the copy stayed busy"); + std::thread::sleep(Duration::from_millis(20)); + } + Err(e) => panic!("the copy can't be run: {e}"), + Ok(_) => return, + } + } + } + + fn root(&self) -> PathBuf { + self.dir.path().join(".local/share/wanderbooks") + } + + fn updates(&self) -> PathBuf { + self.root().join("updates") + } + + fn cmd(&self, mode: &str) -> Command { + let mut c = Command::new(&self.exe); + c.arg(mode) + .arg("--root") + .arg(self.root()) + .args(["--test-knobs", KNOBS]) + .env_clear() + .env("PATH", "/usr/bin:/bin") + .env("HOME", self.dir.path()); + c + } + + fn program(&self) -> Vec { + std::fs::read(&self.exe).unwrap() + } + + fn create(&self, id: &str) { + let out = self + .cmd("books") + .args(["create", id]) + .stdin(Stdio::null()) + .output() + .unwrap(); + assert!(out.status.success(), "{out:?}"); + let out = self + .cmd("token") + .args(["create", "--book", id]) + .stdin(Stdio::null()) + .output() + .unwrap(); + assert!(out.status.success(), "{out:?}"); + } + + /// `wanderbooks update` (or `rollback`) against `release`. + fn run(&self, mode: &str, release: Option<&Release>) -> std::process::Output { + let mut c = Command::new(&self.exe); + c.arg(mode).arg("--root").arg(self.root()); + if let Some(r) = release { + c.arg("--feed") + .arg(format!( + "file://{}", + r.dir.path().join("manifest.json").display() + )) + .arg("--trusted-key") + .arg(r.dir.path().join("release.pub")); + } + c.env_clear() + .env("PATH", "/usr/bin:/bin") + .env("HOME", self.dir.path()) + .stdin(Stdio::null()); + c.output().unwrap() + } + + fn pids(&self, mode: &str) -> Vec { + let root = self.root().display().to_string(); + let mut out = Vec::new(); + for entry in std::fs::read_dir("/proc").unwrap().flatten() { + let Ok(pid) = entry.file_name().to_string_lossy().parse::() else { + continue; + }; + let Ok(raw) = std::fs::read(entry.path().join("cmdline")) else { + continue; + }; + let args: Vec = raw + .split(|b| *b == 0) + .map(|a| String::from_utf8_lossy(a).into_owned()) + .collect(); + if args.get(1).map(String::as_str) == Some(mode) && args.contains(&root) { + out.push(pid); + } + } + out + } +} + +impl Drop for Home { + fn drop(&mut self) { + let _ = self + .cmd("close") + .stdout(Stdio::null()) + .stderr(Stdio::null()) + .status(); + let deadline = Instant::now() + Duration::from_secs(10); + loop { + let mut found = false; + for mode in ["serve", "mcp", "host", "server"] { + for pid in self.pids(mode) { + found = true; + // SAFETY: a process of this test's own root. + unsafe { libc::kill(pid as i32, libc::SIGKILL) }; + } + } + if !found || Instant::now() > deadline { + break; + } + std::thread::sleep(Duration::from_millis(100)); + } + } +} + +/// A release folder: a manifest, the program file, and the key it was +/// signed with (its public half in `release.pub`). +struct Release { + dir: tempfile::TempDir, + keys: minisign::KeyPair, +} + +impl Release { + fn new() -> Self { + let dir = tempfile::Builder::new() + .prefix("wb-release-") + .tempdir() + .unwrap(); + let keys = minisign::KeyPair::generate_unencrypted_keypair().unwrap(); + std::fs::write( + dir.path().join("release.pub"), + keys.pk.to_box().unwrap().to_string(), + ) + .unwrap(); + Self { dir, keys } + } + + /// The signature of `bytes` by `keys`, as the `.sig` file's text. + fn sign_with(keys: &minisign::KeyPair, bytes: &[u8]) -> String { + minisign::sign( + Some(&keys.pk), + &keys.sk, + std::io::Cursor::new(bytes), + Some("test release"), + Some("signature from a test key"), + ) + .unwrap() + .to_string() + } + + fn b64(text: &str) -> String { + use base64::Engine; + base64::engine::general_purpose::STANDARD.encode(text) + } + + /// Publish `version` with `program` as this platform's file, signed + /// by our key; `tweak` changes the manifest's entry before it's + /// written. + fn publish(&self, version: &str, program: &[u8], tweak: impl FnOnce(&mut Value)) { + let name = format!("wanderbooks-{version}"); + std::fs::write(self.dir.path().join(&name), program).unwrap(); + let mut entry = json!({ + "url": name, + "sha256": sha256(program), + "signature": Self::b64(&Self::sign_with(&self.keys, program)), + }); + tweak(&mut entry); + let manifest = json!({ + "version": version, + "notes": "a test release", + "platforms": { platform(): entry }, + }); + std::fs::write(self.dir.path().join("manifest.json"), manifest.to_string()).unwrap(); + } +} + +/// The program as a newer release: the same program with a tail (an ELF +/// file runs the same with bytes after its end), so its hash differs. +fn newer(home: &Home, tag: &str) -> Vec { + let mut bytes = home.program(); + bytes.extend_from_slice(format!("\nrelease {tag}\n").as_bytes()); + bytes +} + +fn text(out: &std::process::Output) -> String { + format!( + "{}{}", + String::from_utf8_lossy(&out.stdout), + String::from_utf8_lossy(&out.stderr) + ) +} + +/// What is in `dir`, for "nothing was left behind". +fn names(dir: &Path) -> Vec { + let mut v: Vec = std::fs::read_dir(dir) + .map(|d| { + d.flatten() + .map(|e| e.file_name().to_string_lossy().into_owned()) + .collect() + }) + .unwrap_or_default(); + v.sort(); + v +} + +#[test] +fn an_update_replaces_the_program_keeps_the_old_one_and_rollback_brings_it_back() { + let home = Home::new("basic"); + let old = home.program(); + let release = Release::new(); + let next = newer(&home, "9.0.0"); + release.publish("9.0.0", &next, |_| {}); + + // The newest release is this one; an older offer is refused. + release.publish("0.1.0", &old, |_| {}); + let same = home.run("update", Some(&release)); + assert!( + same.status.success() && text(&same).contains("newest"), + "{}", + text(&same) + ); + release.publish("0.0.1", &old, |_| {}); + let older = home.run("update", Some(&release)); + assert!(!older.status.success(), "{}", text(&older)); + assert_eq!(home.program(), old); + assert!(!home.updates().join("previous").exists()); + release.publish("9.0.0", &next, |_| {}); + + let look = home.run("update", None); + assert!(!look.status.success(), "no key, no feed: {}", text(&look)); + assert!(text(&look).contains("no release key"), "{}", text(&look)); + + let out = home.run("update", Some(&release)); + assert!(out.status.success(), "{}", text(&out)); + assert!(text(&out).contains("from 0.1.0 to 9.0.0"), "{}", text(&out)); + assert_eq!(home.program(), next, "the new program is in place"); + let kept = home.updates().join("previous"); + assert_eq!(std::fs::read(kept.join("wanderbooks")).unwrap(), old); + let meta: Value = + serde_json::from_slice(&std::fs::read(kept.join("meta.json")).unwrap()).unwrap(); + assert_eq!(meta["version"], "0.1.0"); + assert!(meta["format"].as_u64().unwrap() >= 2); + // Nothing staged is left, and the program's folder holds the program + // alone. + assert_eq!(names(&home.updates()), ["previous", "update.lock"]); + assert_eq!(names(home.exe.parent().unwrap()), ["wanderbooks"]); + + // Rollback: the old program is back, and nothing is kept to go back to. + let out = home.run("rollback", None); + assert!(out.status.success(), "{}", text(&out)); + assert_eq!(home.program(), old); + assert!(!kept.exists()); + let nothing = home.run("rollback", None); + assert!(!nothing.status.success()); + assert!( + text(¬hing).contains("no earlier version"), + "{}", + text(¬hing) + ); +} + +#[test] +fn update_is_atomic() { + let home = Home::new("atomic"); + let old = home.program(); + let release = Release::new(); + let next = newer(&home, "atomic"); + release.publish("9.0.0", &next, |_| {}); + // A reader of the program file sees the old one or the new, whole, + // all through the swap. + let (old_hash, new_hash) = (sha256(&old), sha256(&next)); + let path = home.exe.clone(); + let stop = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false)); + let seen = { + let stop = stop.clone(); + std::thread::spawn(move || { + let mut reads = 0u32; + let mut bad = Vec::new(); + while !stop.load(std::sync::atomic::Ordering::Relaxed) { + match std::fs::read(&path) { + Ok(bytes) => { + let h = sha256(&bytes); + if h != old_hash && h != new_hash { + bad.push(format!("{} bytes", bytes.len())); + } + reads += 1; + } + Err(e) => bad.push(e.to_string()), + } + } + (reads, bad) + }) + }; + let out = home.run("update", Some(&release)); + stop.store(true, std::sync::atomic::Ordering::Relaxed); + let (reads, bad) = seen.join().unwrap(); + assert!(out.status.success(), "{}", text(&out)); + assert!(reads > 0); + assert!(bad.is_empty(), "a reader saw a part of a program: {bad:?}"); + assert_eq!(home.program(), next); +} + +#[test] +fn failed_update_rolls_back() { + let home = Home::new("failed"); + let old = home.program(); + let release = Release::new(); + // Validly signed, but not a program that starts. + release.publish("9.0.0", b"#!/bin/sh\nexit 3\n", |_| {}); + let out = home.run("update", Some(&release)); + assert!(!out.status.success(), "{}", text(&out)); + assert!(text(&out).contains("didn't start"), "{}", text(&out)); + assert_eq!(home.program(), old, "the old program is back"); + assert_eq!(names(home.exe.parent().unwrap()), ["wanderbooks"]); + // And it still works. + home.runnable(); + // The failed one is not left staged to be tried again. + assert!(!home.updates().join("9.0.0").exists()); +} + +#[test] +fn unsigned_update_refused() { + let home = Home::new("unsigned"); + let old = home.program(); + let release = Release::new(); + let next = newer(&home, "unsigned"); + let other = minisign::KeyPair::generate_unencrypted_keypair().unwrap(); + type Tweak = Box; + let mut cases: Vec<(&str, Tweak)> = Vec::new(); + // A bad signature: a good one by another key. + let by_other = Release::b64(&Release::sign_with(&other, &next)); + cases.push(( + "another key", + Box::new(move |e| e["signature"] = json!(by_other)), + )); + // A signature of other bytes. + let of_other_bytes = Release::b64(&Release::sign_with(&release.keys, b"something else")); + cases.push(( + "other bytes", + Box::new(move |e| e["signature"] = json!(of_other_bytes)), + )); + // A sha that differs from the manifest's. + cases.push(("sha256", Box::new(|e| e["sha256"] = json!("0".repeat(64))))); + cases.push(("no signature", Box::new(|e| e["signature"] = json!("")))); + cases.push(( + "not a signature", + Box::new(|e| e["signature"] = json!("AAAA")), + )); + for (name, tweak) in cases { + release.publish("9.0.0", &next, tweak); + let out = home.run("update", Some(&release)); + assert!(!out.status.success(), "{name}: {}", text(&out)); + assert!(text(&out).contains("refused"), "{name}: {}", text(&out)); + assert_eq!(home.program(), old, "{name}: the program changed"); + assert!(!home.updates().join("previous").exists(), "{name}: kept"); + assert!(!home.updates().join("9.0.0").exists(), "{name}: staged"); + assert_eq!(names(home.exe.parent().unwrap()), ["wanderbooks"], "{name}"); + } + // The key must be named only with a local feed. + let out = Command::new(&home.exe) + .args(["update", "--root"]) + .arg(home.root()) + .args([ + "--feed", + "https://example.invalid/manifest.json", + "--trusted-key", + ]) + .arg(release.dir.path().join("release.pub")) + .env_clear() + .output() + .unwrap(); + assert!(!out.status.success()); + assert!(text(&out).contains("file:// feed"), "{}", text(&out)); +} + +#[test] +fn package_managed_install_refuses_update() { + let home = Home::new("package"); + let old = home.program(); + let share = home.dir.path().join("share/wanderbooks"); + std::fs::create_dir_all(&share).unwrap(); + std::fs::write(share.join("install-method"), "deb\n").unwrap(); + let release = Release::new(); + release.publish("9.0.0", &newer(&home, "pkg"), |_| {}); + for mode in ["update", "rollback"] { + let out = home.run(mode, Some(&release)); + let out = if mode == "rollback" { + home.run(mode, None) + } else { + out + }; + assert!(!out.status.success(), "{mode}: {}", text(&out)); + assert!( + text(&out).contains("managed by deb"), + "{mode}: {}", + text(&out) + ); + assert!( + text(&out).contains("package manager"), + "{mode}: {}", + text(&out) + ); + } + assert_eq!(home.program(), old); + assert!(!home.updates().exists(), "nothing was made"); +} + +#[test] +fn rollback_refuses_after_a_book_consented_to_a_newer_format() { + let home = Home::new("consent"); + home.create("acme"); + let release = Release::new(); + let next = newer(&home, "consent"); + release.publish("9.0.0", &next, |_| {}); + let out = home.run("update", Some(&release)); + assert!(out.status.success(), "{}", text(&out)); + // The version kept understood less than this one writes into new + // books (a feature it lacks). + let meta_path = home.updates().join("previous/meta.json"); + let mut meta: Value = serde_json::from_slice(&std::fs::read(&meta_path).unwrap()).unwrap(); + let features = meta["features"].as_array_mut().unwrap(); + assert!(!features.is_empty()); + features.pop(); + std::fs::write(&meta_path, meta.to_string()).unwrap(); + let out = home.run("rollback", None); + assert!(!out.status.success(), "{}", text(&out)); + assert!( + text(&out).contains("newer journal format"), + "{}", + text(&out) + ); + assert_eq!(home.program(), next, "the program stayed"); + assert!(home.updates().join("previous/wanderbooks").is_file()); + // Declared what the kept version knows: it goes back. + let mut meta: Value = serde_json::from_slice(&std::fs::read(&meta_path).unwrap()).unwrap(); + meta["features"] = json!(ledger_store::format::current().features); + std::fs::write(&meta_path, meta.to_string()).unwrap(); + let out = home.run("rollback", None); + assert!(out.status.success(), "{}", text(&out)); +} + +/// The bridge over stdio, spoken to as an MCP client. +struct Agent { + child: std::process::Child, + lines: mpsc::Receiver, + next: u64, +} + +impl Agent { + fn start(home: &Home, book: &str) -> Self { + let mut c = home.cmd("mcp"); + c.args(["--book", book]); + let mut child = c + .stdin(Stdio::piped()) + .stdout(Stdio::piped()) + .stderr(Stdio::null()) + .spawn() + .unwrap(); + let (tx, lines) = mpsc::channel(); + let out = child.stdout.take().unwrap(); + std::thread::spawn(move || { + for line in BufReader::new(out).lines() { + let Ok(line) = line else { break }; + if tx.send(line).is_err() { + break; + } + } + }); + let mut a = Self { + child, + lines, + next: 1, + }; + a.request( + "initialize", + json!({"protocolVersion": "2025-06-18", "capabilities": {}, + "clientInfo": {"name": "test", "version": "0"}}), + ); + a.send(json!({"jsonrpc": "2.0", "method": "notifications/initialized"})); + a + } + + fn send(&mut self, message: Value) { + let stdin = self.child.stdin.as_mut().unwrap(); + writeln!(stdin, "{message}").unwrap(); + stdin.flush().unwrap(); + } + + fn request(&mut self, method: &str, params: Value) -> Value { + let id = self.next; + self.next += 1; + self.send(json!({"jsonrpc": "2.0", "id": id, "method": method, "params": params})); + let deadline = Instant::now() + Duration::from_secs(60); + loop { + let left = deadline.saturating_duration_since(Instant::now()); + let line = self + .lines + .recv_timeout(left) + .unwrap_or_else(|_| panic!("no answer to {method}")); + let v: Value = serde_json::from_str(&line).unwrap(); + if v.get("id") == Some(&json!(id)) { + return v; + } + } + } + + fn call(&mut self, tool: &str, args: Value) -> Result { + let r = common::tools_call(|m, p| self.request(m, p), tool, args); + if let Some(e) = r.get("error") { + return Err(e["message"].as_str().unwrap_or_default().to_string()); + } + if r["result"]["isError"] == true { + return Err(r["result"]["content"][0]["text"] + .as_str() + .unwrap_or_default() + .into()); + } + Ok(r["result"]["structuredContent"].clone()) + } + + fn add_account(&mut self, code: &str) -> Result { + self.call( + "books.add_account", + json!({"code": code, "name": format!("A{code}"), "account_type": "expense"}), + ) + } +} + +impl Drop for Agent { + fn drop(&mut self) { + let _ = self.child.kill(); + let _ = self.child.wait(); + } +} + +/// Update while an agent's session writes without pause: every write is +/// answered and made once, the session carries on with no reconnect, the +/// host and server run the new file afterwards, and the head is the one +/// the writes made. +fn update_under_writes(tag: &str) -> (Home, Agent, Vec) { + let home = Home::new(tag); + home.create("acme"); + let release = Release::new(); + let next = newer(&home, tag); + release.publish("9.0.0", &next, |_| {}); + let mut agent = Agent::start(&home, "acme"); + agent.add_account("4000").unwrap(); + let before = agent.request("tools/list", json!({})); + let host_before = wait_host(&home); + + let updating = { + let exe = home.exe.clone(); + let (root, dir) = (home.root(), home.dir.path().to_path_buf()); + let feed = format!( + "file://{}", + release.dir.path().join("manifest.json").display() + ); + let key = release.dir.path().join("release.pub"); + std::thread::spawn(move || { + let _keep = &release; + Command::new(exe) + .args(["update", "--root"]) + .arg(root) + .args(["--feed", &feed, "--trusted-key"]) + .arg(key) + .env_clear() + .env("PATH", "/usr/bin:/bin") + .env("HOME", dir) + .output() + .unwrap() + }) + }; + let mut written = vec!["4000".to_string()]; + let mut k = 0u32; + // Writes all through the update and the restart it causes. + let deadline = Instant::now() + Duration::from_secs(90); + let mut restarted = false; + while !restarted { + assert!( + Instant::now() < deadline, + "the host never restarted onto the update" + ); + let code = format!("{}", 5000 + k); + k += 1; + let r = agent.add_account(&code); + assert!(r.is_ok(), "a write across the update: {r:?}"); + written.push(code); + if updating.is_finished() { + restarted = wait_for(1, || { + home.pids("host").len() == 1 && home.pids("host")[0] != host_before + }); + } + } + let out = updating.join().unwrap(); + assert!(out.status.success(), "{}", text(&out)); + assert_eq!(home.program(), next); + // The host and the server run the program now on disk. + let host = wait_host(&home); + assert_ne!(host, host_before); + let exe = |pid: u32| std::fs::read_link(format!("/proc/{pid}/exe")).unwrap(); + assert!(wait_for(20, || !home.pids("server").is_empty())); + assert_eq!(exe(home.pids("server")[0]), exe(host)); + assert!(!exe(host).to_string_lossy().contains("(deleted)")); + // The session carries on, its tools unchanged. + assert!(agent.add_account("4999").is_ok()); + written.push("4999".into()); + let after = agent.request("tools/list", json!({})); + assert_eq!(before["result"], after["result"]); + (home, agent, written) +} + +fn wait_host(home: &Home) -> u32 { + assert!( + wait_for(20, || home.pids("host").len() == 1), + "no single host" + ); + home.pids("host")[0] +} + +#[test] +fn bridge_survives_update() { + let (_home, mut agent, written) = update_under_writes("bridge"); + let chart = agent.call("books.chart", json!({})).unwrap(); + let in_books = chart["accounts"].as_array().unwrap().len(); + assert!(in_books >= written.len(), "{in_books} < {}", written.len()); +} + +#[test] +fn update_waits_for_a_running_call() { + // No call is cut off or made twice: each account written across the + // update is in the books exactly once. + let (_home, mut agent, written) = update_under_writes("running"); + let chart = agent.call("books.chart", json!({})).unwrap(); + let mut in_books: Vec = chart["accounts"] + .as_array() + .unwrap() + .iter() + .filter_map(|a| a["code"].as_str().map(str::to_string)) + .filter(|c| c.starts_with('4') || c.starts_with('5')) + .collect(); + let mut written = written; + in_books.sort(); + written.sort(); + assert_eq!( + in_books, written, + "a write lost, or made twice, across the update" + ); +} diff --git a/crates/ledger-host/Cargo.toml b/crates/ledger-host/Cargo.toml index 8c677cd..b435f3a 100644 --- a/crates/ledger-host/Cargo.toml +++ b/crates/ledger-host/Cargo.toml @@ -22,6 +22,7 @@ tokio-util = { version = "0.7", features = ["rt"] } rpassword = "7" # Token texts and passphrases are wiped from memory when dropped. zeroize = "1" +minisign-verify = "0.3.0" [target.'cfg(unix)'.dependencies] libc = "0.2" diff --git a/crates/ledger-host/src/lib.rs b/crates/ledger-host/src/lib.rs index cb4f568..6c17834 100644 --- a/crates/ledger-host/src/lib.rs +++ b/crates/ledger-host/src/lib.rs @@ -24,6 +24,8 @@ pub mod opener; #[cfg(unix)] mod start; pub mod tokens; +#[cfg(unix)] +pub mod update; /// How a mode stops when it can't go on: the reason is already printed, /// and the number is the process's exit code. diff --git a/crates/ledger-host/src/update.rs b/crates/ledger-host/src/update.rs new file mode 100644 index 0000000..b1f2266 --- /dev/null +++ b/crates/ledger-host/src/update.rs @@ -0,0 +1,774 @@ +//! `wanderbooks update` and `wanderbooks rollback` (plan P11a step 2, +//! 3.9, D-24, D-26). +//! +//! The owner's way to a newer version of the program, for an install that +//! updates itself (an AppImage, or the executable put somewhere by hand). +//! A package-managed install refuses and says to use the package manager. +//! +//! ```text +//! GOAL: a newer version replaces the running program only if it is the +//! one we released, whole, and the books never see a half-written +//! or unverified program. +//! SOURCE: plan/served.md 3.9 (Restarting; Format changes are the +//! server's gate, not the updater's), D-24 (every self-update is +//! verified against our own release keys), D-26 (the feed is a +//! manifest at a fixed address). +//! INVARIANTS: +//! 1. The manifest names a version, and for this platform a file's +//! address, its sha256 and its minisign signature. An update goes +//! ahead only if the version is higher than this one's, the bytes +//! downloaded have that sha256 AND verify against the release key +//! this program carries; anything else is refused before a byte is +//! written (nothing staged, nothing swapped). +//! 2. The program file is replaced by a rename of a complete file made +//! in its own folder: a reader sees the old file or the new, never a +//! part. +//! 3. The version replaced is kept under `/updates/previous/` with +//! the journal format it understood, until a rollback uses it or the +//! next update replaces it. +//! 4. The new program must answer `--help` once swapped in; if not, the +//! old one is put back and the update is an error. +//! 5. The running books are never told to stop: the host notices its +//! file was replaced and restarts as it does for a package upgrade +//! (host.rs INVARIANT 7); calls in progress finish first. +//! 6. A rollback refuses while any book declares a journal format or +//! feature the kept version doesn't know (the owner consented to a +//! newer one: the only way back is a restore, plan 3.9). +//! 7. A package-managed install is never touched. +//! 8. A program with no release key compiled in updates from nothing +//! but a local (`file://`) feed whose key the owner names. +//! SCENARIOS: +//! 1. IF the feed offers a higher version, signed by our key: SHOULD +//! replace the program and keep the old one. +//! 2. IF the file's sha256 differs from the manifest's, or its signature +//! is by another key or missing: SHOULD refuse, changing nothing. +//! 3. IF the new program can't start: SHOULD put the old one back. +//! 4. IF a book consented to a newer format: SHOULD refuse to roll back. +//! ``` + +use serde::Deserialize; +use sha2::{Digest, Sha256}; +use std::collections::BTreeMap; +use std::io::Write; +use std::os::unix::fs::{DirBuilderExt, OpenOptionsExt, PermissionsExt}; +use std::path::{Path, PathBuf}; +use std::process::{Command, Stdio}; +use std::time::{Duration, Instant}; + +type Failure = Box; + +/// Where the release manifest is (D-26): the fixed address GitHub +/// Releases serves the newest release's files at. +pub const DEFAULT_FEED: &str = + "https://github.com/swaits/wanderbooks/releases/latest/download/manifest.json"; + +/// The release public key (minisign, base64) this program trusts, put in +/// at build time by the release workflow (P11a step 5). A build without +/// one trusts nothing. +const RELEASE_KEY: Option<&str> = option_env!("WANDERBOOKS_RELEASE_PUBKEY"); + +/// The updates folder in the data root, and what it holds. +const UPDATES: &str = "updates"; +const PREVIOUS: &str = "previous"; +const PROGRAM: &str = "wanderbooks"; +const META: &str = "meta.json"; + +const MAX_MANIFEST: usize = 1 << 20; +const MAX_PROGRAM: u64 = 1 << 30; +/// How long the new program has to answer `--help`. +const HEALTH_WAIT: Duration = Duration::from_secs(30); + +/// The release manifest (the shape `cargo-packager-updater` reads: a +/// version and a file per platform, its signature the base64 of the +/// `.sig` file), with the file's sha256 added. +#[derive(Debug, Deserialize)] +struct Manifest { + version: String, + #[serde(default)] + notes: String, + platforms: BTreeMap, +} + +#[derive(Debug, Deserialize)] +struct Entry { + /// The file's address: absolute, or relative to the manifest's. + url: String, + sha256: String, + /// The minisign signature file's text, or its base64. + signature: String, +} + +/// How this program was installed. +#[derive(Debug, PartialEq, Eq)] +pub enum Install { + /// A file this program may replace: the AppImage, or the executable. + SelfUpdating(PathBuf), + /// A package manager owns the file (`deb`, `rpm`, `aur`, ...). + PackageManaged(String), +} + +/// What kind of install the executable at `exe` is. A package puts +/// `/share/wanderbooks/install-method` beside `/bin`, with +/// its manager's name in it; an AppImage says where it is in `APPIMAGE`. +pub fn install_kind(exe: &Path, appimage: Option<&Path>) -> Install { + if let Some(image) = appimage.filter(|p| p.is_file()) { + return Install::SelfUpdating(image.to_path_buf()); + } + if let Some(prefix) = exe.parent().and_then(Path::parent) + && let Ok(text) = std::fs::read_to_string(prefix.join("share/wanderbooks/install-method")) + { + let manager = text.split_whitespace().next().unwrap_or("package"); + return Install::PackageManaged(manager.to_string()); + } + Install::SelfUpdating(exe.to_path_buf()) +} + +fn this_install() -> Result { + let exe = crate::start::this_executable()?; + let image = std::env::var_os("APPIMAGE").map(PathBuf::from); + Ok(install_kind(&exe, image.as_deref())) +} + +fn refuse_package_managed(manager: &str, what: &str) -> Failure { + format!( + "this install is managed by {manager}, which also {what} it: use the package manager (our apt and dnf repositories keep every release, and the AUR keeps old PKGBUILDs)" + ) + .into() +} + +/// `1.2.3` as numbers; anything else (a pre-release, a `v`) is no version +/// of ours. +pub fn parse_version(text: &str) -> Option<(u64, u64, u64)> { + let mut parts = text.split('.'); + let mut next = || -> Option { + let p = parts.next()?; + (!p.is_empty() && p.bytes().all(|b| b.is_ascii_digit())) + .then(|| p.parse().ok()) + .flatten() + }; + let v = (next()?, next()?, next()?); + parts.next().is_none().then_some(v) +} + +fn hex(bytes: &[u8]) -> String { + bytes.iter().map(|b| format!("{b:02x}")).collect() +} + +fn sha256_hex(bytes: &[u8]) -> String { + hex(&Sha256::digest(bytes)) +} + +/// The flags of `update`. +#[derive(Default)] +struct Flags { + root: Option, + feed: Option, + key_file: Option, + check: bool, +} + +fn parse_flags( + prog: &str, + args: impl IntoIterator, + update: bool, +) -> Result, Failure> { + let mut flags = Flags::default(); + let mut args = args.into_iter(); + while let Some(arg) = args.next() { + match arg.as_str() { + "--root" => { + flags.root = Some(PathBuf::from(args.next().ok_or("--root needs a folder")?)) + } + "--feed" if update => flags.feed = Some(args.next().ok_or("--feed needs an address")?), + "--trusted-key" if update => { + flags.key_file = Some(PathBuf::from( + args.next().ok_or("--trusted-key needs a file")?, + )) + } + "--check" if update => flags.check = true, + "--help" | "-h" => { + if update { + eprintln!("usage: {prog} [--check] [--feed ADDRESS [--trusted-key FILE]]"); + eprintln!( + " Brings this program to the newest release: downloads it, checks it" + ); + eprintln!( + " against our release key, keeps the version it replaces, and swaps" + ); + eprintln!(" it in; the books restart onto it when idle. --check only looks."); + eprintln!( + " --feed reads a manifest from elsewhere (a file:// folder, to update" + ); + eprintln!( + " from a local release); only then may --trusted-key name the minisign" + ); + eprintln!(" public key to check against."); + } else { + eprintln!("usage: {prog}"); + eprintln!(" Goes back to the version the last update replaced, unless a book"); + eprintln!(" has consented to a newer journal format that version can't open."); + } + return Ok(None); + } + other => { + eprintln!("unknown argument: {other}"); + eprintln!( + "usage: {prog} {}", + if update { + "[--check] [--feed ADDRESS [--trusted-key FILE]]" + } else { + "" + } + ); + return Err(Box::new(crate::Exit(2))); + } + } + } + Ok(Some(flags)) +} + +/// `wanderbooks update`: bring this program to the newest release. +/// `current` is this program's version. +pub fn update( + prog: &str, + current: &str, + args: impl IntoIterator, +) -> Result<(), Failure> { + let Some(flags) = parse_flags(prog, args, true)? else { + return Ok(()); + }; + let target = match this_install()? { + Install::PackageManaged(manager) => { + return Err(refuse_package_managed(&manager, "updates")); + } + Install::SelfUpdating(target) => target, + }; + let current_v = parse_version(current).ok_or("this program's version isn't one of ours")?; + let feed = flags.feed.as_deref().unwrap_or(DEFAULT_FEED); + let key = trusted_key(feed, flags.key_file.as_deref())?; + let manifest_bytes = fetch(feed, MAX_MANIFEST as u64)?; + let manifest: Manifest = serde_json::from_slice(&manifest_bytes) + .map_err(|e| format!("the release manifest isn't one we can read: {e}"))?; + let offered = parse_version(&manifest.version).ok_or_else(|| { + format!( + "the manifest's version {:?} isn't one of ours", + manifest.version + ) + })?; + if offered == current_v { + println!("wanderbooks {current} is the newest release."); + return Ok(()); + } + if offered < current_v { + return Err(format!( + "the feed offers {}, older than this {current}: not installing it (to go back, `wanderbooks rollback`)", + manifest.version + ) + .into()); + } + let platform = format!("{}-{}", std::env::consts::OS, std::env::consts::ARCH); + let entry = manifest + .platforms + .get(&platform) + .ok_or_else(|| format!("release {} has no build for {platform}", manifest.version))?; + if flags.check { + println!( + "wanderbooks {} is available (this is {current}). Run `{}` to install it.", + manifest.version, + prog.trim_end_matches(" --check") + ); + if !manifest.notes.trim().is_empty() { + println!("{}", manifest.notes.trim()); + } + return Ok(()); + } + let root = ledger_store::root::Root::find(flags.root.as_deref()).map_err(|e| e.to_string())?; + let updates = updates_dir(&root)?; + let _lock = take_lock(&updates)?; + + let bytes = fetch(&resolve(feed, &entry.url)?, MAX_PROGRAM)?; + verify(&bytes, entry, &key)?; + + // Verified: from here the files are ours to write. Staged, then the + // old one kept, then swapped in. + let staged_dir = updates.join(&manifest.version); + private_dir(&staged_dir)?; + let staged = staged_dir.join(PROGRAM); + write_program(&staged, &bytes)?; + keep_previous(&updates, &target, current)?; + swap_in(&target, &staged)?; + if let Err(why) = healthy(&target) { + // INVARIANT 4: put the old one back. + let old = updates.join(PREVIOUS).join(PROGRAM); + let back = swap_in(&target, &old); + let _ = std::fs::remove_dir_all(&staged_dir); + return Err(match back { + Ok(()) => format!( + "the new version didn't start ({why}); the old one is back and nothing changed" + ), + Err(e) => format!( + "the new version didn't start ({why}) and the old one couldn't be put back ({e}); it is kept in the updates folder" + ), + } + .into()); + } + let _ = std::fs::remove_dir_all(&staged_dir); + println!( + "Updated wanderbooks from {current} to {}.", + manifest.version + ); + println!( + "The books that are open restart onto it by themselves when they are idle; nothing in progress is cut off. `wanderbooks rollback` goes back." + ); + Ok(()) +} + +/// `wanderbooks rollback`: go back to the version the last update +/// replaced. +pub fn rollback(prog: &str, args: impl IntoIterator) -> Result<(), Failure> { + let Some(flags) = parse_flags(prog, args, false)? else { + return Ok(()); + }; + let target = match this_install()? { + Install::PackageManaged(manager) => { + return Err(refuse_package_managed(&manager, "downgrades")); + } + Install::SelfUpdating(target) => target, + }; + let root = ledger_store::root::Root::find(flags.root.as_deref()).map_err(|e| e.to_string())?; + let updates = updates_dir(&root)?; + let _lock = take_lock(&updates)?; + let previous = updates.join(PREVIOUS); + let meta: serde_json::Value = std::fs::read(previous.join(META)) + .ok() + .and_then(|b| serde_json::from_slice(&b).ok()) + .ok_or("there is no earlier version kept to go back to")?; + if !previous.join(PROGRAM).is_file() { + return Err("there is no earlier version kept to go back to".into()); + } + let version = meta["version"].as_str().unwrap_or("the earlier version"); + let known = ledger_store::format::JournalFormat { + format: meta["format"].as_u64().unwrap_or(0) as u32, + features: meta["features"] + .as_array() + .map(|a| { + a.iter() + .filter_map(|f| f.as_str().map(str::to_string)) + .collect() + }) + .unwrap_or_default(), + }; + refuse_if_consented(&root, version, &known)?; + // The newer program is set aside while the older one is tried. + let aside = updates.join("rolling-back"); + let newer = std::fs::read(&target) + .map_err(|e| format!("can't read this program to set it aside: {}", e.kind()))?; + write_program(&aside, &newer)?; + drop(newer); + swap_in(&target, &previous.join(PROGRAM))?; + if let Err(why) = healthy(&target) { + let back = swap_in(&target, &aside); + let _ = std::fs::remove_file(&aside); + return Err(match back { + Ok(()) => format!("version {version} didn't start ({why}); the newer one is back and nothing changed"), + Err(e) => format!("version {version} didn't start ({why}) and the newer one couldn't be put back ({e})"), + } + .into()); + } + let _ = std::fs::remove_file(&aside); + let _ = std::fs::remove_dir_all(&previous); + println!("Went back to wanderbooks {version}."); + println!("The books that are open restart onto it by themselves when they are idle."); + Ok(()) +} + +/// INVARIANT 6: refuse while a book declares more than `known`. +fn refuse_if_consented( + root: &ledger_store::root::Root, + version: &str, + known: &ledger_store::format::JournalFormat, +) -> Result<(), Failure> { + for id in root.list_books().map_err(|e| e.to_string())? { + let journal = root.book_dir(&id).join(ledger_store::JOURNAL_FILE); + let declared = match ledger_store::journal::read_journal_and_format(&journal) { + Ok((_, declared)) => declared, + Err(e) => { + return Err(format!( + "can't tell what format the books {id} are in ({e}), so not going back" + ) + .into()); + } + }; + let newer_format = declared.format > known.format; + let newer: Vec<&String> = declared + .features + .iter() + .filter(|f| !known.features.contains(f)) + .collect(); + if newer_format || !newer.is_empty() { + return Err(format!( + "the books {id} were moved to a newer journal format, which version {version} can't open: going back would leave them unreadable. The only way back is restoring the backup made before the upgrade (it says how many entries would be lost)" + ) + .into()); + } + } + Ok(()) +} + +/// The key to check against: the one compiled in, or, for a `file://` +/// feed only, the one the owner names (INVARIANT 8). +fn trusted_key(feed: &str, key_file: Option<&Path>) -> Result { + let text = match (key_file, RELEASE_KEY) { + (Some(file), _) => { + if !feed.starts_with("file://") { + return Err("--trusted-key goes with a file:// feed only: the key a release is checked against is the one built into this program".into()); + } + std::fs::read_to_string(file).map_err(|e| format!("can't read the key: {e}"))? + } + (None, Some(key)) => key.to_string(), + (None, None) => { + return Err("this build carries no release key, so it can't check an update: install a release build, or update from a local release with --feed file://… --trusted-key FILE".into()); + } + }; + parse_public_key(&text) + .map_err(|e| format!("the release key isn't a minisign public key: {e}").into()) +} + +/// A minisign public key as `minisign -G` writes it (two lines), as +/// `cargo packager signer generate` writes it (those lines in base64), or +/// as the bare key line. +pub fn parse_public_key(text: &str) -> Result { + use base64::Engine; + let text = text.trim(); + if text.starts_with("untrusted comment:") { + return minisign_verify::PublicKey::decode(text); + } + if let Ok(raw) = base64::engine::general_purpose::STANDARD.decode(text) + && let Ok(inner) = String::from_utf8(raw) + && inner.starts_with("untrusted comment:") + { + return minisign_verify::PublicKey::decode(&inner); + } + minisign_verify::PublicKey::from_base64(text) +} + +/// INVARIANT 1: the sha256 AND the signature. +fn verify(bytes: &[u8], entry: &Entry, key: &minisign_verify::PublicKey) -> Result<(), Failure> { + let have = sha256_hex(bytes); + if !have.eq_ignore_ascii_case(entry.sha256.trim()) { + return Err( + "the download doesn't match the checksum in the manifest: refused, nothing changed" + .into(), + ); + } + let text = entry.signature.trim(); + let text = if text.starts_with("untrusted comment:") { + text.to_string() + } else { + use base64::Engine; + let raw = base64::engine::general_purpose::STANDARD + .decode(text) + .map_err( + |_| "the manifest's signature isn't a minisign signature: refused, nothing changed", + )?; + String::from_utf8(raw).map_err(|_| "the manifest's signature isn't text")? + }; + let signature = minisign_verify::Signature::decode(&text).map_err(|e| { + format!( + "the manifest's signature isn't a minisign signature ({e}): refused, nothing changed" + ) + })?; + key.verify(bytes, &signature, false).map_err(|e| { + Box::::from(format!( + "the download isn't signed by our release key ({e}): refused, nothing changed" + )) + }) +} + +/// The address of `url` as the manifest at `feed` means it. +fn resolve(feed: &str, url: &str) -> Result { + if url.contains("://") { + return Ok(url.to_string()); + } + if let Some(path) = feed.strip_prefix("file://") { + let dir = Path::new(path).parent().unwrap_or(Path::new("/")); + return Ok(format!("file://{}", dir.join(url).display())); + } + let base = reqwest::Url::parse(feed).map_err(|e| format!("the feed address is bad: {e}"))?; + Ok(base + .join(url) + .map_err(|e| format!("the manifest's file address is bad: {e}"))? + .to_string()) +} + +/// The bytes at `address` (`file://` or `https://`), at most `max`. +fn fetch(address: &str, max: u64) -> Result, Failure> { + if let Some(path) = address.strip_prefix("file://") { + let file = std::fs::File::open(path).map_err(|e| format!("can't read {address}: {e}"))?; + let mut bytes = Vec::new(); + std::io::Read::read_to_end(&mut std::io::Read::take(file, max + 1), &mut bytes) + .map_err(|e| format!("can't read {address}: {e}"))?; + if bytes.len() as u64 > max { + return Err(format!("{address} is larger than a release can be").into()); + } + return Ok(bytes); + } + if !address.starts_with("https://") { + return Err(format!("{address}: only https:// and file:// feeds are read").into()); + } + let rt = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .map_err(|e| e.to_string())?; + rt.block_on(async { + let mut response = reqwest::Client::new() + .get(address) + .timeout(Duration::from_secs(600)) + .send() + .await + .and_then(|r| r.error_for_status()) + .map_err(|e| format!("can't fetch {address}: {}", e.without_url()))?; + let mut bytes = Vec::new(); + while let Some(chunk) = response + .chunk() + .await + .map_err(|e| format!("can't fetch {address}: {}", e.without_url()))? + { + bytes.extend_from_slice(&chunk); + if bytes.len() as u64 > max { + return Err(format!("{address} is larger than a release can be")); + } + } + Ok(bytes) + }) + .map_err(Into::into) +} + +/// `/updates`, made 0700 if missing. +fn updates_dir(root: &ledger_store::root::Root) -> Result { + let dir = root.path().join(UPDATES); + private_dir(&dir)?; + Ok(dir) +} + +fn private_dir(dir: &Path) -> Result<(), Failure> { + std::fs::DirBuilder::new() + .recursive(true) + .mode(0o700) + .create(dir) + .map_err(|e| format!("can't make the updates folder: {}", e.kind()))?; + Ok(()) +} + +/// One update or rollback at a time. +fn take_lock(updates: &Path) -> Result { + let file = std::fs::OpenOptions::new() + .create(true) + .append(true) + .mode(0o600) + .open(updates.join("update.lock")) + .map_err(|e| format!("can't open the update lock: {}", e.kind()))?; + match file.try_lock() { + Ok(()) => Ok(file), + Err(std::fs::TryLockError::WouldBlock) => { + Err("another update or rollback is running".into()) + } + Err(std::fs::TryLockError::Error(e)) => { + Err(format!("can't take the update lock: {}", e.kind()).into()) + } + } +} + +/// Write a program file whole and executable, readable by its owner only +/// until the swap. +fn write_program(path: &Path, bytes: &[u8]) -> Result<(), Failure> { + let partial = path.with_extension("partial"); + let _ = std::fs::remove_file(&partial); + let mut file = std::fs::OpenOptions::new() + .write(true) + .create_new(true) + .mode(0o700) + .open(&partial) + .map_err(|e| format!("can't stage the update: {}", e.kind()))?; + file.write_all(bytes) + .and_then(|()| file.sync_all()) + .map_err(|e| format!("can't stage the update: {}", e.kind()))?; + drop(file); + std::fs::rename(&partial, path).map_err(|e| format!("can't stage the update: {}", e.kind()))?; + Ok(()) +} + +/// INVARIANT 3: the program about to be replaced, with what it understood +/// of the journal. +fn keep_previous(updates: &Path, target: &Path, version: &str) -> Result<(), Failure> { + let previous = updates.join(PREVIOUS); + let _ = std::fs::remove_dir_all(&previous); + private_dir(&previous)?; + let bytes = std::fs::read(target) + .map_err(|e| format!("can't read this program to keep it: {}", e.kind()))?; + write_program(&previous.join(PROGRAM), &bytes)?; + let known = ledger_store::format::current(); + let meta = serde_json::json!({ + "version": version, + "format": known.format, + "features": known.features, + }); + std::fs::write(previous.join(META), meta.to_string()) + .map_err(|e| format!("can't keep this program: {}", e.kind()))?; + Ok(()) +} + +/// INVARIANT 2: replace `target` with a copy of `from`, by renaming a +/// complete file made beside it. +fn swap_in(target: &Path, from: &Path) -> Result<(), Failure> { + let dir = target.parent().ok_or("this program has no folder")?; + let name = target + .file_name() + .map(|n| n.to_string_lossy().into_owned()) + .unwrap_or_default(); + let temp = dir.join(format!(".{name}.update-{}", std::process::id())); + let result = (|| -> std::io::Result<()> { + let _ = std::fs::remove_file(&temp); + let mut source = std::fs::File::open(from)?; + let mut out = std::fs::OpenOptions::new() + .write(true) + .create_new(true) + .mode(0o755) + .open(&temp)?; + std::io::copy(&mut source, &mut out)?; + out.sync_all()?; + drop(out); + std::fs::set_permissions(&temp, std::fs::Permissions::from_mode(0o755))?; + std::fs::rename(&temp, target) + })(); + if let Err(e) = result { + let _ = std::fs::remove_file(&temp); + return Err(format!( + "can't replace the program ({}): is its folder yours to write? Nothing was changed", + e.kind() + ) + .into()); + } + Ok(()) +} + +/// INVARIANT 4: the program at `path` starts and answers `--help`. +fn healthy(path: &Path) -> Result<(), String> { + let deadline = Instant::now() + HEALTH_WAIT; + let mut child = loop { + match Command::new(path) + .arg("--help") + .env("APPIMAGE_EXTRACT_AND_RUN", "1") + .stdin(Stdio::null()) + .stdout(Stdio::null()) + .stderr(Stdio::null()) + .spawn() + { + Ok(child) => break child, + // The file was just written; a process forked meanwhile may + // still hold it open for writing. + Err(e) + if e.kind() == std::io::ErrorKind::ExecutableFileBusy + && Instant::now() < deadline => + { + std::thread::sleep(Duration::from_millis(50)); + } + Err(e) => return Err(format!("can't run it: {}", e.kind())), + } + }; + loop { + match child.try_wait() { + Ok(Some(status)) if status.success() => return Ok(()), + Ok(Some(status)) => return Err(format!("it exited with {status}")), + Ok(None) if Instant::now() < deadline => std::thread::sleep(Duration::from_millis(20)), + Ok(None) => { + let _ = child.kill(); + let _ = child.wait(); + return Err("it didn't answer in time".into()); + } + Err(e) => return Err(e.to_string()), + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn versions_are_three_numbers_and_order_by_number() { + assert_eq!(parse_version("1.2.3"), Some((1, 2, 3))); + assert!(parse_version("0.10.0") > parse_version("0.9.9")); + for bad in ["", "1.2", "1.2.3.4", "v1.2.3", "1.2.3-rc1", "1..3", "1.2.x"] { + assert_eq!(parse_version(bad), None, "{bad}"); + } + } + + #[test] + fn a_package_marker_beside_the_program_makes_it_package_managed() { + let dir = tempfile::tempdir().unwrap(); + let bin = dir.path().join("bin"); + std::fs::create_dir_all(&bin).unwrap(); + let exe = bin.join("wanderbooks"); + std::fs::write(&exe, b"x").unwrap(); + assert_eq!(install_kind(&exe, None), Install::SelfUpdating(exe.clone())); + let share = dir.path().join("share/wanderbooks"); + std::fs::create_dir_all(&share).unwrap(); + std::fs::write(share.join("install-method"), "deb\n").unwrap(); + assert_eq!( + install_kind(&exe, None), + Install::PackageManaged("deb".into()) + ); + // An AppImage is its own file, whatever is beside the mount. + let image = dir.path().join("Wanderbooks.AppImage"); + std::fs::write(&image, b"x").unwrap(); + assert_eq!( + install_kind(&exe, Some(&image)), + Install::SelfUpdating(image) + ); + } + + #[test] + fn a_public_key_is_read_in_every_form_minisign_and_the_packager_write_it() { + let bare = "RWQdrs7W5/jf6drdD0G/ymxFmRWYAIOyggYlubz4NQzdQm5e/Fj0nrb0"; + let two = format!("untrusted comment: minisign public key: E9DFF8E7D6CEAE1D\n{bare}\n"); + use base64::Engine; + let wrapped = base64::engine::general_purpose::STANDARD.encode(&two); + for form in [bare.to_string(), two, wrapped] { + parse_public_key(&form).unwrap_or_else(|e| panic!("{form}: {e}")); + } + assert!(parse_public_key("not a key").is_err()); + } + + #[test] + fn a_manifest_address_is_resolved_beside_the_manifest() { + assert_eq!( + resolve("file:///rel/manifest.json", "wb").unwrap(), + "file:///rel/wb" + ); + assert_eq!( + resolve("https://h.example/a/b/manifest.json", "wb").unwrap(), + "https://h.example/a/b/wb" + ); + assert_eq!( + resolve("file:///x/m.json", "https://o/wb").unwrap(), + "https://o/wb" + ); + } + + #[test] + fn a_swap_leaves_the_whole_new_file_and_no_temporary() { + let dir = tempfile::tempdir().unwrap(); + let target = dir.path().join("wanderbooks"); + let from = dir.path().join("new"); + std::fs::write(&target, b"old").unwrap(); + std::fs::write(&from, b"new").unwrap(); + swap_in(&target, &from).unwrap(); + assert_eq!(std::fs::read(&target).unwrap(), b"new"); + let names: Vec<_> = std::fs::read_dir(dir.path()) + .unwrap() + .map(|e| e.unwrap().file_name()) + .collect(); + assert_eq!(names.len(), 2, "{names:?}"); + } +} From 4b5f44689d2cdfb8c493ffc56348e29331653141 Mon Sep 17 00:00:00 2001 From: Stephen Waits Date: Thu, 29 Oct 2026 15:11:35 -0600 Subject: [PATCH 3/6] feat(cli): a launch with nothing to say and no terminal is the viewer (P11a step 3) An AppImage's launcher runs the desktop file's Exec line and appends the arguments it was given, so with `Exec=wanderbooks gui` the AppImage's `update` ran `wanderbooks gui update`: the program's own commands were unreachable through it. The desktop files now say plain `Exec=wanderbooks` and a bare launch with stdin not a terminal (a menu entry, a double-click) is the viewer; at a terminal it is the usage, as before. The AppImage is thereby also the command line (`Wanderbooks.AppImage update`). Test: bare_launch_tests::only_nothing_at_all_with_no_terminal_is_the_viewer (the launch itself cannot run under test: the viewer finds the account's real data root from the password database, not from the environment). Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01HS8WZziDnyxunC23MUxrLp --- crates/ledger-cli/src/main.rs | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/crates/ledger-cli/src/main.rs b/crates/ledger-cli/src/main.rs index e308782..cdb39b6 100644 --- a/crates/ledger-cli/src/main.rs +++ b/crates/ledger-cli/src/main.rs @@ -389,6 +389,15 @@ fn run_server(_args: Vec) -> i32 { 2 } +/// Whether the program was launched with nothing to say and no terminal +/// to say it to: a menu entry, a double-click, an AppImage (whose launcher +/// runs the desktop file's command and appends what it was given, so the +/// desktop file says plain `wanderbooks`). At a terminal it is the usage, +/// as ever. +fn is_bare_launch(words: &[String], stdin_is_terminal: bool) -> bool { + words.is_empty() && !stdin_is_terminal +} + fn main() { // A file's path may be any name the system takes: the file commands // get the words as given. Everything else is text. @@ -405,6 +414,10 @@ fn main() { eprintln!("error: an argument isn't UTF-8 text"); std::process::exit(2); }; + // A bare launch is the viewer. + if is_bare_launch(&words, std::io::IsTerminal::is_terminal(&std::io::stdin())) { + std::process::exit(run_gui(Vec::new())); + } // The modes run in this process and take their own flags. let mut all = words.clone().into_iter(); match all.next().as_deref() { @@ -479,3 +492,18 @@ mod gui_args_tests { } } } + +#[cfg(test)] +mod bare_launch_tests { + use super::is_bare_launch; + + #[test] + fn only_nothing_at_all_with_no_terminal_is_the_viewer() { + assert!(is_bare_launch(&[], false)); + // At a terminal it is the usage, as ever. + assert!(!is_bare_launch(&[], true)); + // Anything said is a mode or a mistake, never the viewer. + assert!(!is_bare_launch(&["--help".to_string()], false)); + assert!(!is_bare_launch(&["update".to_string()], false)); + } +} From 3058b5c229d267fb7298df0ee5ddce3582e50639 Mon Sep 17 00:00:00 2001 From: Stephen Waits Date: Thu, 29 Oct 2026 15:11:47 -0600 Subject: [PATCH 4/6] feat(packaging): the trial's Linux column and the Linux packages (P11a steps 1 and 3) The Linux half of the packaging trial, run on this machine and in Debian, Fedora and Arch containers, and the recipes it produced (docs/packaging.md holds the table and what it found): - mise.toml pins cargo-packager 0.11.8 and cargo-generate-rpm 0.21.1, now that `just package-linux` calls them. cargo-packager-updater is a library, not a tool, so it is not pinned (update.rs does the checking with minisign-verify, which is what the updater uses). - deploy/packaging/: packager.toml (.deb, AppImage, the pacman tarball and its PKGBUILD), the rpm table in crates/ledger-cli/Cargo.toml, the AUR's own PKGBUILD (repacks the release tarball), build-linux.sh, repo-indexes.sh (the apt and dnf indexes, unsigned until the signing step), local-release.sh (a throwaway-keyed feed to try `update`), test-linux-packages.sh (linux_packages_install_and_upgrade: install 0.1.0, open books as an ordinary user, upgrade to 0.1.1, check the host restarted onto the new file, the books' last journal line unchanged and `update` refuses; passes on Debian trixie, Fedora and Arch). A package marks itself package-managed with share/wanderbooks/install-method. - The icon is a placeholder square (the AppImage cannot be built without one). - .github/workflows/release.yml: on a version tag, build on ubuntu-24.04 (the oldest glibc the packages claim: a program linked on this machine needs GLIBC_2.43 and does not start on Debian 13), attach the packages to the release, and add them to the apt and dnf repositories on gh-pages. No signing and no macOS or Windows job yet. - The release profile strips symbols. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01HS8WZziDnyxunC23MUxrLp --- .github/workflows/release.yml | 77 ++++++++++ README.md | 9 ++ deploy/README.md | 6 +- deploy/packaging/aur/PKGBUILD | 25 ++++ deploy/packaging/build-linux.sh | 51 +++++++ deploy/packaging/icons/512x512.png | Bin 0 -> 3463 bytes deploy/packaging/install-method-aur | 1 + deploy/packaging/install-method-deb | 1 + deploy/packaging/install-method-rpm | 1 + deploy/packaging/local-release.sh | 32 +++++ deploy/packaging/packager.toml | 36 +++++ deploy/packaging/repo-indexes.sh | 37 +++++ deploy/packaging/test-linux-packages.sh | 92 ++++++++++++ deploy/packaging/wanderbooks-static.desktop | 7 + deploy/packaging/wanderbooks.desktop | 10 ++ docs/architecture.md | 42 +++++- docs/packaging.md | 148 ++++++++++++++++++-- docs/testing.md | 76 ++++++++++ justfile | 32 +++++ mise.toml | 4 + 20 files changed, 669 insertions(+), 18 deletions(-) create mode 100644 .github/workflows/release.yml create mode 100644 deploy/packaging/aur/PKGBUILD create mode 100755 deploy/packaging/build-linux.sh create mode 100644 deploy/packaging/icons/512x512.png create mode 100644 deploy/packaging/install-method-aur create mode 100644 deploy/packaging/install-method-deb create mode 100644 deploy/packaging/install-method-rpm create mode 100755 deploy/packaging/local-release.sh create mode 100644 deploy/packaging/packager.toml create mode 100755 deploy/packaging/repo-indexes.sh create mode 100755 deploy/packaging/test-linux-packages.sh create mode 100644 deploy/packaging/wanderbooks-static.desktop create mode 100644 deploy/packaging/wanderbooks.desktop diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml new file mode 100644 index 0000000..231444f --- /dev/null +++ b/.github/workflows/release.yml @@ -0,0 +1,77 @@ +# Releases (plan P11a, D-24, D-26): on a version tag, build the Linux +# packages on the oldest system they claim (ubuntu-24.04, glibc 2.39; a +# program linked on a newer one does not start on an older one), attach +# them to the GitHub release, and add them to the apt and dnf repositories +# on the gh-pages branch (GitHub Pages serves them). Unsigned: step 5 of +# the unit (the release key, Q-4) signs the manifest and the indexes, and +# step 6 adds the macOS and Windows jobs. The release holds no key today. +name: release + +on: + push: + tags: ["v*"] + workflow_dispatch: + +permissions: + contents: write + +jobs: + linux: + runs-on: ubuntu-24.04 + # Same disk discipline as check.yml: no incremental caches. + env: + CARGO_INCREMENTAL: "0" + steps: + - uses: actions/checkout@v4 + - name: Room for the build (the image ships toolchains we never use) + run: | + sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc \ + /opt/hostedtoolcache/CodeQL /usr/local/.ghcup + df -h / | tail -1 + - uses: jdx/mise-action@v2 + with: + # Only what packaging needs, not the proof and fuzz tools. + install_args: "rust just cargo:cargo-packager cargo:cargo-generate-rpm" + - name: GUI build dependencies + run: sudo apt-get update && sudo apt-get install -y libxkbcommon-dev libgtk-3-dev apt-utils createrepo-c + - name: The tag is the program's version + id: version + run: | + own=$(sed -n 's/^version = "\(.*\)"/\1/p' crates/ledger-cli/Cargo.toml | head -n1) + tag="${GITHUB_REF_NAME#v}" + if [ "${GITHUB_REF_TYPE}" = tag ] && [ "$tag" != "$own" ]; then + echo "tag $GITHUB_REF_NAME is not version $own" >&2 + exit 1 + fi + echo "version=$own" >> "$GITHUB_OUTPUT" + - name: Build the packages + run: just package-linux ${{ steps.version.outputs.version }} + - name: Attach them to the release + if: github.ref_type == 'tag' + env: + GH_TOKEN: ${{ github.token }} + run: | + v=${{ steps.version.outputs.version }} + gh release view "$GITHUB_REF_NAME" >/dev/null 2>&1 || gh release create "$GITHUB_REF_NAME" --title "$GITHUB_REF_NAME" --notes "wanderbooks $v" + gh release upload "$GITHUB_REF_NAME" --clobber target/packages/$v/*.AppImage target/packages/$v/*.deb target/packages/$v/*.rpm target/packages/$v/*.tar.gz + - name: Add them to the apt and dnf repositories (gh-pages) + if: github.ref_type == 'tag' + run: | + v=${{ steps.version.outputs.version }} + git fetch origin gh-pages || true + if git rev-parse --verify origin/gh-pages >/dev/null 2>&1; then + git worktree add repo origin/gh-pages + else + git worktree add --detach repo && git -C repo checkout --orphan gh-pages && git -C repo rm -rfq . || true + fi + mkdir -p repo/apt/pool repo/dnf + cp target/packages/$v/*.deb repo/apt/pool/ + cp target/packages/$v/*.rpm repo/dnf/ + deploy/packaging/repo-indexes.sh apt repo/apt + deploy/packaging/repo-indexes.sh dnf repo/dnf + cd repo + git config user.name "wanderbooks release" + git config user.email "release@wanderbooks.ai" + git add -A + git commit -m "wanderbooks $v" + git push origin HEAD:gh-pages diff --git a/README.md b/README.md index 748343f..55c3315 100644 --- a/README.md +++ b/README.md @@ -95,6 +95,15 @@ create acme`, `wanderbooks connect claude --book acme` (or `connect codex`). Then talk to the agent; `wanderbooks gui acme` shows the books, and its Books menu backs them up, restores, starts new books and closes them. +Updating: an AppImage, or a `wanderbooks` you put somewhere yourself, brings +itself to the newest release with `wanderbooks update` (it checks the +download against our release key, keeps the version it replaces, and the +open books restart onto the new one when idle); `wanderbooks rollback` goes +back, unless a book has since moved to a newer journal format. A `.deb`, +`.rpm` or AUR install is updated by its package manager and `update` says +so. The Linux packages are built here with `just package-linux` +(docs/packaging.md); none is published yet. + ## Developing Every tool the recipes use (Rust, just, Kani, Lean, cargo-mutants, diff --git a/deploy/README.md b/deploy/README.md index 3a72587..fe454b3 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -32,8 +32,10 @@ sudo install -D -m 0644 deploy/wanderbooks-server.service \ ``` The executable is the server-only build: no viewer, no window libraries. -On a Linux package install (P11) this is the same binary and the package -manager writes the file; until then, copy it as above. +The Linux packages (`.deb`, `.rpm`, AUR; `just package-linux`, docs/packaging.md) +carry the viewer build, which also serves: on a package install the package +manager writes the file and `wanderbooks update` refuses. A server with no +window libraries is still this server-only build, copied as above. ## 2. Make the data root, once diff --git a/deploy/packaging/aur/PKGBUILD b/deploy/packaging/aur/PKGBUILD new file mode 100644 index 0000000..5a4ae40 --- /dev/null +++ b/deploy/packaging/aur/PKGBUILD @@ -0,0 +1,25 @@ +# Maintainer: Stephen Waits +# The AUR package (P11a step 3, D-24, D-26): it repacks the tarball of the +# release's files that GitHub Releases serves; nothing is built from source. +# `updpkgsums` fills the checksum when a release is published. +pkgname=wanderbooks-bin +pkgver=0.1.0 +pkgrel=1 +pkgdesc="Double-entry bookkeeping your agent keeps and you can read" +arch=('x86_64') +url="https://wanderbooks.ai" +license=('MIT' 'Apache-2.0') +depends=('gcc-libs' 'glibc') +provides=('wanderbooks') +conflicts=('wanderbooks') +options=('!lto' '!strip') +# The tarball holds usr/bin/wanderbooks, usr/share/applications/ and +# usr/share/wanderbooks/install-method (which makes `wanderbooks update` +# refuse and send the owner to pacman). +source=("wanderbooks_${pkgver}_x86_64.tar.gz::https://github.com/swaits/wanderbooks/releases/download/v${pkgver}/wanderbooks_${pkgver}_x86_64.tar.gz") +sha256sums=('SKIP') +noextract=() + +package() { + cp -r "${srcdir}/usr" "${pkgdir}/usr" +} diff --git a/deploy/packaging/build-linux.sh b/deploy/packaging/build-linux.sh new file mode 100755 index 0000000..26cce1b --- /dev/null +++ b/deploy/packaging/build-linux.sh @@ -0,0 +1,51 @@ +#!/usr/bin/env bash +# The Linux packages (P11a step 3), from the release program already built +# at target/release/wanderbooks (`cargo build --release -p ledger-cli`). +# +# build-linux.sh [VERSION] [FORMATS] +# +# Puts into target/packages/VERSION/: the .deb, the AppImage and the +# pacman tarball with its PKGBUILD (cargo-packager), and the .rpm +# (cargo-generate-rpm). VERSION defaults to the program's own; a different +# one repackages the same program under that version, which is how the +# install-and-upgrade test gets its second release. FORMATS is +# cargo-packager's list (default deb,appimage,pacman); the .rpm is always +# built. Run from the repository root. +# +# The program is target/release/wanderbooks, or the file WB_PROGRAM names: +# a package must run on the oldest system it claims, and a program linked +# on a newer one (this machine's glibc) fails there with "GLIBC_2.43 not +# found". CI builds on ubuntu-24.04; `just test-linux-packages` builds in a +# Debian trixie container. +set -euo pipefail +own=$(sed -n 's/^version = "\(.*\)"/\1/p' crates/ledger-cli/Cargo.toml | head -n1) +version=${1:-$own} +formats=${2:-deb,appimage,pacman} +out="target/packages/$version" +program=${WB_PROGRAM:-target/release/wanderbooks} +test -x "$program" || { echo "build the release program first: cargo build --release -p ledger-cli" >&2; exit 1; } +rm -rf "$out"; mkdir -p "$out" +stage="target/packages/.stage-$version" +rm -rf "$stage"; mkdir -p "$stage" +cp "$program" "$stage/wanderbooks" +config=$(mktemp --suffix=.toml) +trap 'rm -f "$config"' EXIT +# cargo-packager reads paths relative to the config file, which is in /tmp +# here, so they are made absolute. +sed "s/^version = \".*\"/version = \"$version\"/ + s#^out-dir = .*#out-dir = \"$PWD/$out\"# + s#\"target/release\"#\"$PWD/$stage\"# + s#\"deploy/packaging/#\"$PWD/deploy/packaging/# + s#\"LICENSE-MIT\"#\"$PWD/LICENSE-MIT\"#" \ + deploy/packaging/packager.toml >"$config" +# The AppImage tools are themselves AppImages; run them without FUSE. +export APPIMAGE_EXTRACT_AND_RUN=1 +cargo packager --config "$config" --formats "$formats" --release +assets='assets = [ + { source = "'$PWD/$stage'/wanderbooks", dest = "/usr/bin/wanderbooks", mode = "0755" }, + { source = "deploy/packaging/install-method-rpm", dest = "/usr/share/wanderbooks/install-method", mode = "0644" }, + { source = "deploy/packaging/wanderbooks-static.desktop", dest = "/usr/share/applications/wanderbooks.desktop", mode = "0644" }, +]' +cargo generate-rpm -p crates/ledger-cli --set-metadata "version=\"$version\"" --set-metadata "$assets" -o "$out/" +rm -rf "$stage" +ls -l "$out" diff --git a/deploy/packaging/icons/512x512.png b/deploy/packaging/icons/512x512.png new file mode 100644 index 0000000000000000000000000000000000000000..3505023a07843977234c232b950677b4be57fe8c GIT binary patch literal 3463 zcmbVPU2GIp6#nkrX?JW3odRi#pgT1fqY_xfhsK|FC|jU`RHB%asObs{JZTdQ7GulJ z7NMq18WV#&Kxj1ZqS1vH6n_|gHMW7&_y9;{1ERzqgi7c_x5abrOuMvM+O5eXbMLw5 zeD^!wIrr}Ddn;;}Ov$@955Sa~rIF_VNka)UmtNN!f4W8X+M3A1m5pcrKDV#vNZ7){ zA3Ck7l4UQ@5j^v#EKGeIHc_-LDeF8zd5h~_YaX!uuH|%APm3;)FtU2h7*>v z^I$@UP&UciXs}_mW1HYYQKTX)FM>Bodot+jB4KM#aW|K7qWz^&h!eZpfX`zrH}}tt zy<7W2CVv!&wXC@E%7nMhjFCKEz&ifpiHh5gw>k&OX#h-3^HqXO6fQ=NG_ z^j3NIft(BNH=a5F&EU_Q&Isw~XhO6#7;RuNHiw(M6{9gd!Qc{xw-<5MTPwJ+X*_19 z;tz%c;+;F+*X{(ze){F3Y8(z=&cb{B{wNIgw6XK^;W)iFc_exw9>U~!F*GT3M|$y2 zgUM5>D(6AyF^eZX4YdL+sZNb)>G+J(M3Kjz*^xiMvlv z1l59#p|yFqy>H-hg^7YviGQX`SIrC?_~ju(i3QOsEpP|6ARNu@ZnDt0+d^))`m%7u zv40^FO;soj2Yw@jub(v$k}@QZY*RidT@zNxubDU0^=F>hO-~ESuB;J9`c=qgN^z{b z)>OX~8Wv|wrM&>_{;KGnfuS}1J(PcGS{N_cz*_fQP^kWUMAR+VF<|3rMHvcqhz+4} z6s$0e3SDDx5-%MXGr`l#sBnbkBNFch6v)`%lXqim<-=m{zP7afpxUn+=+sFcpc z$xes1acnjlEq0 znc5}efxB8}(?e|}xEcJqr}=qQoh&aPzXjBNlfHob+seQCeJcDivCb=A7I&fhJsM72 zIl5dVDmR9D>B12s!fc)F{hU@XvB;r^+DLMJ{JLwYJbqbU0-VgS^HY<)fV_4YZdIR( zcdAZ$#Y>~>T6$C3k%|Z6P2~Aohe!~J{Ci};N^7n%p%Q&1/dev/null + cp "$out/key.pub" "$out/release.pub" +fi +case "$(uname -m)" in arm64) arch=aarch64 ;; *) arch=$(uname -m) ;; esac +platform="$(uname -s | tr '[:upper:]' '[:lower:]')-$arch" +name="wanderbooks-$version-$platform" +cp "$file" "$out/$name" +cargo packager signer sign "$out/$name" --private-key "$out/key" --password "" >/dev/null +sha=$(sha256sum "$out/$name" | cut -d' ' -f1) +sig=$(tr -d '\n' <"$out/$name.sig") +printf '{"version":"%s","notes":"a local release","platforms":{"%s":{"url":"%s","sha256":"%s","signature":"%s"}}}\n' \ + "$version" "$platform" "$name" "$sha" "$sig" >"$out/manifest.json" +echo "feed: file://$(cd "$out" && pwd)/manifest.json key: $out/release.pub" diff --git a/deploy/packaging/packager.toml b/deploy/packaging/packager.toml new file mode 100644 index 0000000..be918ba --- /dev/null +++ b/deploy/packaging/packager.toml @@ -0,0 +1,36 @@ +# cargo-packager's description of the Linux bundles (P11a step 3; D-25). +# `just package-linux` builds the release program and runs +# cargo packager --config deploy/packaging/packager.toml +# from the repository root. The .rpm is cargo-generate-rpm's, from the +# [package.metadata.generate-rpm] table of crates/ledger-cli/Cargo.toml. +name = "wanderbooks" +product-name = "Wanderbooks" +version = "0.1.0" +identifier = "ai.wanderbooks.app" +description = "Double-entry bookkeeping your agent keeps and you can read." +long-description = "One program: the viewer, the books server, and the bridge that lets Claude Code or Codex keep the books." +homepage = "https://wanderbooks.ai" +authors = ["Stephen Waits"] +license-file = "LICENSE-MIT" +category = "Finance" +binaries-dir = "target/release" +out-dir = "target/packages" +icons = ["deploy/packaging/icons/512x512.png"] +formats = ["deb", "appimage", "pacman"] +binaries = [{ path = "wanderbooks", main = true }] + +[deb] +section = "utils" +# A deb marks itself package-managed: `wanderbooks update` refuses on it +# and says to use apt (plan 3.9). +desktop-template = "deploy/packaging/wanderbooks.desktop" +depends = ["libc6", "libgcc-s1"] + +[deb.files] +"deploy/packaging/install-method-deb" = "usr/share/wanderbooks/install-method" + +[pacman] +depends = ["gcc-libs", "glibc"] + +[pacman.files] +"deploy/packaging/install-method-aur" = "usr/share/wanderbooks/install-method" diff --git a/deploy/packaging/repo-indexes.sh b/deploy/packaging/repo-indexes.sh new file mode 100755 index 0000000..671d60d --- /dev/null +++ b/deploy/packaging/repo-indexes.sh @@ -0,0 +1,37 @@ +#!/usr/bin/env bash +# The apt and dnf repository indexes (P11a step 3), served from GitHub +# Pages (D-26). Unsigned until step 5 gives them the release key. +# +# repo-indexes.sh apt DIR DIR/pool/*.deb -> DIR/dists/stable/... +# repo-indexes.sh dnf DIR DIR/*.rpm -> DIR/repodata/ +# +# Every release stays in the pool: a downgrade is the package manager's +# (plan 3.9). apt needs apt-utils (`apt-ftparchive`), dnf needs +# createrepo_c. +set -euo pipefail +kind=${1:?usage: repo-indexes.sh apt|dnf DIR} +dir=${2:?usage: repo-indexes.sh apt|dnf DIR} +case "$kind" in +apt) + cd "$dir" + arch=binary-amd64 + mkdir -p "dists/stable/main/$arch" + apt-ftparchive packages pool >"dists/stable/main/$arch/Packages" + gzip -9 -kf "dists/stable/main/$arch/Packages" + apt-ftparchive \ + -o APT::FTPArchive::Release::Origin=wanderbooks \ + -o APT::FTPArchive::Release::Suite=stable \ + -o APT::FTPArchive::Release::Codename=stable \ + -o APT::FTPArchive::Release::Architectures=amd64 \ + -o APT::FTPArchive::Release::Components=main \ + release dists/stable >dists/stable/Release.tmp + mv dists/stable/Release.tmp dists/stable/Release + ;; +dnf) + createrepo_c --update "$dir" + ;; +*) + echo "usage: repo-indexes.sh apt|dnf DIR" >&2 + exit 2 + ;; +esac diff --git a/deploy/packaging/test-linux-packages.sh b/deploy/packaging/test-linux-packages.sh new file mode 100755 index 0000000..431b24d --- /dev/null +++ b/deploy/packaging/test-linux-packages.sh @@ -0,0 +1,92 @@ +#!/usr/bin/env bash +# linux_packages_install_and_upgrade (P11a step 3): in a clean Debian, +# Fedora and Arch container, install release 0.1.0 from its package, +# open the books as an ordinary user, upgrade to 0.1.1 as root, and check +# that the books' head is unchanged, that the host the books were open +# under restarted onto the new file, and that `wanderbooks update` refuses +# (a package manager owns the program). The containers run with +# seccomp=unconfined: the host refuses to run under a filter (host.rs +# INVARIANT 2), and Docker's default profile is one. Run from the repository root after +# `just test-linux-packages` built both releases (target/packages/). +set -euo pipefail +export DEBIAN_FRONTEND=noninteractive +old=0.1.0 new=0.1.1 +pk=$PWD/target/packages +mount=(-v "$pk:/packages:ro") + +# The steps every distro shares, run inside the container: `install` and +# `upgrade` are shell snippets for that distro. +common() { cat <<'INNER' +set -euo pipefail +export DEBIAN_FRONTEND=noninteractive +useradd -m tester +as_tester() { su tester -c "$*"; } +install_old +test -x /usr/bin/wanderbooks +grep -q . /usr/share/wanderbooks/install-method +as_tester 'wanderbooks --help' >/dev/null 2>&1 || true +as_tester 'wanderbooks setup' >/dev/null +as_tester 'wanderbooks books create acme' >/dev/null +head_of() { as_tester 'tail -n1 ~/.local/share/wanderbooks/books/acme/journal.jsonl | sha256sum'; } +before=$(as_tester 'wanderbooks books list') +head_before=$(head_of) +echo "before: $before head $head_before" +# The books open: a holder keeps the host and server running. +as_tester 'nohup wanderbooks serve >/tmp/serve.log 2>&1 &' +for _ in $(seq 50); do pgrep -u tester -x wanderbooks -a | grep -q ' host' && break; sleep 0.2; done +host_before=$(pgrep -u tester -f 'wanderbooks host' | head -n1) +test -n "$host_before" +upgrade_new +# The host notices the replaced file and restarts onto it. +for _ in $(seq 150); do + host_after=$(pgrep -u tester -f 'wanderbooks host' | head -n1 || true) + [ -n "$host_after" ] && [ "$host_after" != "$host_before" ] && break + sleep 0.2 +done +test -n "$host_after" && test "$host_after" != "$host_before" +! readlink "/proc/$host_after/exe" | grep -q deleted +after=$(as_tester 'wanderbooks books list') +echo "after: $after head $(head_of)" +test "$before" = "$after" +test "$head_before" = "$(head_of)" +if as_tester 'wanderbooks update' >/tmp/update.out 2>&1; then echo "update did not refuse" >&2; exit 1; fi +grep -q 'package manager' /tmp/update.out +as_tester 'wanderbooks close' >/dev/null 2>&1 || true +echo "OK" +INNER +} + +run() { # name image install_old upgrade_new + local name=$1 image=$2 + echo "== $name ($image)" + docker run --rm --security-opt seccomp=unconfined -e DEBIAN_FRONTEND=noninteractive "${mount[@]}" "$image" bash -c " +$3 +$4 +$(common)" +} + +# Debian and Fedora install from a repository built by repo-indexes.sh +# (the one GitHub Pages serves), the new release added to it afterwards. +mount+=(-v "$PWD/deploy/packaging:/tools:ro") +run debian debian:trixie-slim \ + "apt-get update -qq && apt-get install -y -qq apt-utils procps >/dev/null +mkdir -p /repo/apt/pool && cp /packages/$old/*.deb /repo/apt/pool/ && /tools/repo-indexes.sh apt /repo/apt +echo 'deb [trusted=yes] file:///repo/apt stable main' >/etc/apt/sources.list.d/wanderbooks.list +install_old() { apt-get update -qq && apt-get install -y -qq wanderbooks >/dev/null; }" \ + "upgrade_new() { cp /packages/$new/*.deb /repo/apt/pool/ && /tools/repo-indexes.sh apt /repo/apt && apt-get update -qq && apt-get install -y -qq --only-upgrade wanderbooks >/dev/null; }" + +run fedora fedora:latest \ + "dnf install -y -q procps-ng util-linux createrepo_c >/dev/null +mkdir -p /repo/dnf && cp /packages/$old/*.rpm /repo/dnf/ && /tools/repo-indexes.sh dnf /repo/dnf +printf '[wanderbooks]\nname=wanderbooks\nbaseurl=file:///repo/dnf\ngpgcheck=0\n' >/etc/yum.repos.d/wanderbooks.repo +install_old() { dnf install -y -q wanderbooks >/dev/null; }" \ + "upgrade_new() { cp /packages/$new/*.rpm /repo/dnf/ && /tools/repo-indexes.sh dnf /repo/dnf && dnf upgrade -y -q --refresh wanderbooks >/dev/null; }" + +# Arch: the AUR's own PKGBUILD (deploy/packaging/aur), its source pointed at +# the tarball just built instead of the release URL, makes the package as +# an ordinary user. +arch_build='build() { rm -rf /build/$1 && mkdir -p /build/$1 && cp /packages/$1/*.tar.gz /build/$1/ && cp /tools/aur/PKGBUILD /build/$1/PKGBUILD && sed -i "s/^pkgver=.*/pkgver=$1/; s|^source=.*|source=(wanderbooks_$1_x86_64.tar.gz)|" /build/$1/PKGBUILD && chown -R builder /build/$1 && su builder -c "cd /build/$1 && makepkg -f --nodeps >/dev/null 2>&1"; }' +run arch archlinux:latest \ + "useradd -m builder; pacman -Sy --noconfirm --needed base-devel procps-ng >/dev/null 2>&1; $arch_build; install_old() { build $old; pacman -U --noconfirm /build/$old/*.pkg.tar.zst >/dev/null 2>&1; }" \ + "upgrade_new() { build $new; pacman -U --noconfirm /build/$new/*.pkg.tar.zst >/dev/null 2>&1; }" +echo "all packages install and upgrade" diff --git a/deploy/packaging/wanderbooks-static.desktop b/deploy/packaging/wanderbooks-static.desktop new file mode 100644 index 0000000..ffa4bbe --- /dev/null +++ b/deploy/packaging/wanderbooks-static.desktop @@ -0,0 +1,7 @@ +[Desktop Entry] +Categories=Office;Finance; +Comment=Double-entry bookkeeping your agent keeps and you can read. +Exec=wanderbooks +Name=Wanderbooks +Terminal=false +Type=Application diff --git a/deploy/packaging/wanderbooks.desktop b/deploy/packaging/wanderbooks.desktop new file mode 100644 index 0000000..574cd59 --- /dev/null +++ b/deploy/packaging/wanderbooks.desktop @@ -0,0 +1,10 @@ +[Desktop Entry] +Categories={{categories}} +{{#if comment}} +Comment={{comment}} +{{/if}} +Exec={{exec}} +Icon={{icon}} +Name={{name}} +Terminal=false +Type=Application diff --git a/docs/architecture.md b/docs/architecture.md index a00682d..771f10e 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -36,7 +36,7 @@ the specification in `bookspec/` (which never depends on the engine). | `ledger-import` | formats: QuickBooks CSV in; IIF, XLSX, Typst PDFs (statements, the invoice, labels) out | `ledger-core`, `ledger-reader` | | `ledger-package` | the shared read-only view over `Store` or `Snapshot` (`PackageBooks`): the CPA package, the 1099 worksheet, the invoice document, Needs attention | `ledger-core`, `ledger-store`, `ledger-config`, `ledger-import`, `ledger-vault` | | `ledger-mcp` | the books server: the MCP tools (122), resources, the handbook, scopes, the door and the access log, backups' scheduling | all of the above | -| `ledger-host` | the host and its leases, the agent bridge (`wanderbooks mcp`), tokens in 0600 files under the data root (D-63), the owner's commands, uploads and downloads | `ledger-store`, `ledger-mcp`, `ledger-config` | +| `ledger-host` | the host and its leases, the agent bridge (`wanderbooks mcp`), tokens in 0600 files under the data root (D-63), the owner's commands, uploads and downloads, `update` and `rollback` (P11a) | `ledger-store`, `ledger-mcp`, `ledger-config` | | `ledger-gui` | the viewer: read-only, live, egui; the Books menu hands work to the server | `ledger-core`, `ledger-store`, `ledger-config`, `ledger-import`, `ledger-package`, `ledger-vault`, `ledger-host` | | `ledger-cli` | `wanderbooks`, the one executable: every mode (`serve`, `server`, `host`, `mcp`, `gui`, `setup`, `books`, `token`, `connect`, `doctor`, …) and the over-the-wire test suites | everything | | `bookspec/checker`, `bookspec/runner` | the example schema and parser, the rule catalogue, the `Engine` trait; the per-rule runner | nothing of ours (the adapter in `ledger-core`'s tests depends on them) | @@ -313,3 +313,43 @@ still run on the server). The recipe, the head check for the move and the two-account CI job are in `deploy/README.md`, `deploy/wanderbooks-server.service` and `.github/workflows/two-user.yml`; the checks are `crates/ledger-cli/tests/two_user.rs`. + +## Packaging and updates (`ledger-host`'s `update.rs`, `deploy/packaging/`, P11a) + +One program, installed one of two ways, updated one of two ways +(plan D-24). The install says which: a package ships +`share/wanderbooks/install-method` beside `bin/`, and a file there makes +the program package-managed; `wanderbooks update` and `rollback` refuse it +and name the manager (apt, dnf and the AUR keep every release, so +downgrading is theirs). Anything else (an AppImage, whose file is named by +`$APPIMAGE`; an executable put somewhere by hand) updates itself: + +- the feed is a `manifest.json` (a version, and per platform a file's + address, sha256 and minisign signature; the shape `cargo-packager`'s + updater reads, with the sha256 added) at GitHub Releases' fixed address + (D-26); `--feed` reads another, and only with a `file://` feed may + `--trusted-key` name the key; +- the key a release is checked against is compiled in + (`WANDERBOOKS_RELEASE_PUBKEY`, set by the release workflow once the + owner's key exists; step 5): a build without one updates from nothing; +- a download is accepted only if its sha256 matches AND its signature + verifies; then it is staged, the running version kept under + `/updates/previous/` with the journal format it understood, and the + file replaced by a rename of a complete copy made beside it; +- nothing tells the books to stop. The host notices its executable's file + was replaced and restarts as it does for a package upgrade (host.rs + INVARIANT 7), finishing calls in progress; bridges reconnect upstream + without the agent noticing. The format gate stays the server's (3.9): + `rollback` refuses while a book declares a format or feature the kept + version does not know; +- an AppImage is the exception to the restart: the host runs from the old + image's mount, whose path does not change, so it keeps running the old + version until the books are next closed and opened (docs/packaging.md). + +`deploy/packaging/` holds the Linux packaging: `packager.toml` +(cargo-packager: `.deb`, AppImage, the pacman tarball with its PKGBUILD), +the rpm table in `crates/ledger-cli/Cargo.toml`, `aur/PKGBUILD` (the AUR +package, which repacks the release's tarball), `build-linux.sh`, +`repo-indexes.sh` (the apt and dnf indexes), `local-release.sh` (a +throwaway-keyed feed on this machine) and `test-linux-packages.sh`. +`.github/workflows/release.yml` builds them on a version tag. diff --git a/docs/packaging.md b/docs/packaging.md index dba150e..3ec5b02 100644 --- a/docs/packaging.md +++ b/docs/packaging.md @@ -1,23 +1,141 @@ # Packaging and updates: the trial (P11a step 1) -Status 2026-10-29: **the trial has not been run.** It needs clean Linux, -macOS and Windows machines for the six formats (`.deb`, `.rpm`, AUR, -AppImage, `.dmg`, NSIS `.exe`), and the one machine that built step 0 is a -Linux desktop with neither a macOS nor a Windows target, no container -runtime and none of the packager tools installed. The tools are pinned in -`mise.toml` now (`cargo-packager` 0.11.8, `cargo-packager-updater` 0.2.3, -`cargo-generate-rpm` 0.21.1), so the trial is `mise install` and the VMs. -Nothing below is a result; it is what is known before the trial, and the -table is filled by the person who runs it. +Status 2026-10-29: **the Linux half of the trial is run** (on the Arch +desktop that builds the project, with Debian, Fedora and Arch containers +for the installs); **the macOS and Windows halves are not**: they need +those machines (and for the `.dmg` and the NSIS `.exe`, the signing +identities of step 5), and this machine has neither a macOS nor a Windows +target. `cargo-packager` 0.11.8 and `cargo-generate-rpm` 0.21.1 are pinned in +`mise.toml`; `cargo-packager-updater` is **not** pinned: it is a library +(the program checks releases itself, `crates/ledger-host/src/update.rs`), +not a tool, so there is nothing for a recipe to call. | format | built by | installs as | updated by | signed by | |---|---|---|---|---| -| `.deb` | `cargo packager` | (trial) | apt repository (step 3) | (trial) | -| `.rpm` | `cargo-generate-rpm` | (trial) | dnf repository (step 3) | (trial) | -| AUR `PKGBUILD` | hand-written | (trial) | the package manager | n/a | -| AppImage | `cargo packager` | (trial) | `cargo-packager-updater` | (trial) | -| `.dmg` | `cargo packager` | (trial) | `cargo-packager-updater` | Developer ID (step 5) | -| NSIS `.exe` | `cargo packager` | (trial) | `cargo-packager-updater` | Windows identity (step 5) | +| `.deb` | `cargo packager` (`deb`) | `/usr/bin/wanderbooks`, a menu entry, `/usr/share/wanderbooks/install-method` = `deb`; 47 MB | apt repository (`repo-indexes.sh apt`, GitHub Pages) | unsigned until step 5 (the repository's index is `trusted=yes` in the test) | +| `.rpm` | `cargo-generate-rpm` (table in `crates/ledger-cli/Cargo.toml`) | same paths, marker `rpm`; 36 MB | dnf repository (`repo-indexes.sh dnf`) | unsigned until step 5 (`--signing-key` takes the PGP key then) | +| AUR `PKGBUILD` | `cargo packager` makes the tarball (`pacman` format) and a PKGBUILD for it; ours is `deploy/packaging/aur/PKGBUILD`, which repacks the release's tarball (package `wanderbooks-bin`) | same paths, marker `aur`; built with `makepkg` | pacman (the AUR keeps old PKGBUILDs) | n/a (pacman trusts the AUR's checksums) | +| AppImage | `cargo packager` (`appimage`); 44 MB | one file the owner puts anywhere and runs; needs no admin password | `wanderbooks update` (`$APPIMAGE` is the file replaced) against the manifest; minisign | minisign by the release key (step 5) | +| `.dmg` | `cargo packager` (`dmg`) | (trial) needs a Mac | `wanderbooks update` | Developer ID and notarization (step 5) | +| NSIS `.exe` | `cargo packager` (`nsis`) | (trial) needs Windows, and native Windows is unit P11c (D-62) | `wanderbooks update` | the Windows identity (step 5) | + +`just package-linux` builds the first four (`deploy/packaging/build-linux.sh`, +about a minute after the release build, itself 8 to 12 minutes on a +loaded 16-core machine); `just test-linux-packages` is the install-and-upgrade test. + +## Linux results (2026-10-29) + +What the trial found, each of which is built around now: + +1. **A program linked here does not start on Debian.** This machine's + glibc is 2.44; the binary asks for `GLIBC_2.43` and Debian 13 (trixie) + has 2.41. A package must be built on the oldest system it claims. + `.github/workflows/release.yml` builds on `ubuntu-24.04` (glibc 2.39); + `just release-trixie` builds in `rust:1.99-trixie` for the local test. The + claim is therefore "Debian 13, Ubuntu 24.04 and newer, current Fedora and + Arch"; Debian 12 and Ubuntu 22.04 are not claimed (they would need a + build container of their own). +2. **The books refuse to run under a seccomp filter** (`host.rs` + INVARIANT 2), and Docker's default profile is one: in a container the + host fails to start ("refusing to run under a seccomp filter"). The + tests run containers with `--security-opt seccomp=unconfined`. Anyone + running the books in Docker meets this; it is not a bug of the packages. +3. **An AppImage's launcher runs the desktop file's `Exec` line and appends + the arguments it was given.** With `Exec=wanderbooks gui`, + `Wanderbooks.AppImage update` ran `wanderbooks gui update`. The desktop + files say plain `Exec=wanderbooks`, and a launch with nothing to say and + no terminal is the viewer (`main.rs::is_bare_launch`); at a terminal + it is still the usage. So the AppImage is also the command line: + `Wanderbooks.AppImage update`, `... mcp --book acme`. +4. **cargo-packager's configuration** (0.11.8): paths in a config are + relative to the *config file*, not the working directory (the build script + writes an absolute-path copy); `files` maps *source to destination*; + `category` is a fixed list (`Finance`); the AppImage needs a square icon + (`deploy/packaging/icons/512x512.png`, **a placeholder the owner should + replace**); the `pacman` format yields the tarball and PKGBUILD but not a + package (that is `makepkg`'s); its `.deb` has `Depends` from the config. +5. **The AppImage tools are downloaded when it is built** (`AppRun`, + `linuxdeploy` and its AppImage plugin from tauri-apps' and linuxdeploy's + GitHub releases, unverified by cargo-packager): a supply-chain edge of + the release build, not of the installed app. They run with + `APPIMAGE_EXTRACT_AND_RUN=1`, so the build needs no FUSE. +6. **`cargo packager signer` signs in a format `update` verifies.** Its + key and its `.sig` are the base64 of minisign's two-line text; the + manifest's `signature` is that `.sig` as written, and `update` reads the + public key in either form. Checked end to end by hand (below), and by + `a_public_key_is_read_in_every_form_minisign_and_the_packager_write_it`. +7. **The AppImage updates itself but the books it holds do not restart + onto the new image.** `update` replaces `$APPIMAGE`, and an AppImage run + from a shell (`Wanderbooks.AppImage update`, or the one that was running) + proves the swap and the rollback: after `update`, the AppImage on disk + is the new file, the old one is kept, and `rollback` restores it. But the + host that is open runs from the old image's mount + (`/tmp/.mount_*/usr/bin/wanderbooks`), a path that does not change, so + `host.rs` never sees a replaced file and the open books keep the old + version until they are closed and opened again (`wanderbooks close`). + Fixing it means the host learning the image's path (its environment is + scrubbed on purpose; `APPIMAGE` would join the allow-list of `start.rs`, + or the path would come from the mount table) and starting its successor + from it: a decision about what the host may trust, left to the EM (the + unit's report names it). Package installs and a hand-placed executable + restart as designed (`bridge_survives_update`). +8. **Onboarding with no terminal does not exist yet.** The unit's Done-when + (1) says an AppImage "opens, onboards and posts an entry with no + terminal". The AppImage opens (a bare launch is the viewer), but the viewer + opens only an existing book (`ledger_gui::book_in_root`), and a data root + and a first book come from `wanderbooks setup` and `books create`, which + need a terminal. A first-run path in the viewer ("New books…" is in its + menu, but the menu is not reachable until a book is open) is not in P11a's + steps. It is the gap between this wave's AppImage and Done-when (1). + +By hand, on this machine, against a throwaway release folder made by `just +local-release` (a key from `cargo packager signer generate`; the feed +`file://…/manifest.json`; a scratch home made by `wanderbooks setup --home`, +nothing of the owner's): + +```text +Wanderbooks.AppImage --help -> the usage (FUSE mount, and with APPIMAGE_EXTRACT_AND_RUN=1) +Wanderbooks.AppImage setup --home H -> the private folder; books create acme -> books list: "acme open" +Wanderbooks.AppImage update --check --feed F --trusted-key K + -> "wanderbooks 0.2.0 is available (this is 0.1.0)" +Wanderbooks.AppImage update --feed F --trusted-key K + -> "Updated wanderbooks from 0.1.0 to 0.2.0."; the file's sha256 + is the release's; updates/previous/ holds the 0.1.0 image +Wanderbooks.AppImage rollback -> "Went back to wanderbooks 0.1.0."; the file is the original again +``` + +`linux_packages_install_and_upgrade` (`deploy/packaging/test-linux-packages.sh`), +2026-10-29, 0.1.0 then 0.1.1, all three passing: in `debian:trixie-slim` +and `fedora:latest` the package installs from a repository made by +`repo-indexes.sh` (the release workflow runs the same script), in +`archlinux:latest` from `makepkg` over `deploy/packaging/aur/PKGBUILD`; then +books are made as an ordinary user and held open, 0.1.1 is added and +upgraded to by the package manager (`apt-get install --only-upgrade`, `dnf +upgrade`, `pacman -U`), the host restarts onto the new file (a new pid whose +`/proc/PID/exe` is not deleted), the journal's last line (the head) is +byte-for-byte unchanged, and `wanderbooks update` exits non-zero naming the +package manager. Not tested: Ubuntu, openSUSE, a package-manager +*downgrade* (the repositories keep every release by construction), and the +`.rpm` and `.deb` under their real signatures (step 5). + +## What `update` and `rollback` need from a release + +`manifest.json` at `releases/latest/download/` (D-26): + +```json +{ "version": "0.2.0", "notes": "…", + "platforms": { "linux-x86_64": { "url": "wanderbooks-0.2.0-linux-x86_64", + "sha256": "…hex…", "signature": "…base64 of the .sig file…" } } } +``` + +`url` is absolute or relative to the manifest; the platform key is +`std::env::consts::OS-ARCH`; the file is the whole program (the AppImage +on Linux). The program checks the version is higher, the sha256, and the +minisign signature against `WANDERBOOKS_RELEASE_PUBKEY` (compiled in by the +release workflow once the owner has a key: **step 5, Q-4**; today no build +carries one, so a build updates only from a `file://` feed with +`--trusted-key`). The two-key release manifest of D-25 would wrap this +(rotation, expiry, a monotone version) and is step 5's. ## What step 0 found about the platforms (read before the macOS and Windows rows) diff --git a/docs/testing.md b/docs/testing.md index bd944d8..d75e105 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -3669,3 +3669,79 @@ it); the digest's `since_seq` comparison (`>`), its `as_of` bound, the `zero_net_accounts` branch and the `never_reconciled` split; `report_status`'s three open conditions; the open-drafts refusal's `<= as_of` and its decision; the attention `awaiting_evidence` branch. + +## P11a steps 2 and 3: update, rollback, the Linux packages (2026-10-29) + +`wanderbooks update` and `rollback` (`ledger-host/src/update.rs`) are +tested over a local release folder (a `file://` feed) and a minisign key +the test makes, with real programs: each test makes a home, a stripped +copy of the program in its `bin/`, and a manifest naming a newer copy of +it (the same program with a tail, so its hash differs). All in +`crates/ledger-cli/tests/update.rs` (Linux only): + +- `an_update_replaces_the_program_keeps_the_old_one_and_rollback_brings_it_back`: + no key and no feed is refused ("no release key"); the same version is "the + newest"; an older offer is refused and nothing is kept; an update puts + the new bytes in place, keeps the old under `updates/previous/` with the + format it understood, leaves no temporary or staged file; rollback puts + the old bytes back and consumes what was kept; a second rollback says + there is nothing to go back to. +- `update_is_atomic`: a reader polling the program file all through the + update sees, on every read, the old file whole or the new whole. +- `failed_update_rolls_back`: a validly signed file that does not start + (`exit 3`) is swapped in, fails `--help`, and the old program is put + back; nothing is staged or left beside the program. +- `unsigned_update_refused`: five bad manifests (a good signature by + another key; a signature of other bytes; a sha256 that differs; an empty + signature; a signature that is not one), each refused with the program + unchanged and nothing kept or staged; `--trusted-key` with an `https` + feed is refused. +- `package_managed_install_refuses_update`: a + `share/wanderbooks/install-method` marker beside `bin/` makes both + commands refuse, naming the manager, with nothing made in the data root. +- `rollback_refuses_after_a_book_consented_to_a_newer_format`: with a + book that declares this version's format and a kept version that knows + one feature less, rollback refuses and the program stays; with the kept + version knowing everything the book declares, it goes back. +- `bridge_survives_update` and `update_waits_for_a_running_call`: an + agent's bridge session writes without pause while `update` runs and the + host restarts onto the new file; every write is answered and made once + (the account is in the chart exactly once), the session needs no + reconnect, its tool list is unchanged, and the host and server run the + file now on disk (not a deleted one). The host's own restart laws stay + `open_while_held.rs`'s (`a_restart_is_not_a_close`). + +Unit tests in `update.rs`: `versions_are_three_numbers_and_order_by_number`, +`a_package_marker_beside_the_program_makes_it_package_managed`, +`a_manifest_address_is_resolved_beside_the_manifest`, +`a_swap_leaves_the_whole_new_file_and_no_temporary`, +`a_public_key_is_read_in_every_form_minisign_and_the_packager_write_it`. +`main.rs`: `only_nothing_at_all_with_no_terminal_is_the_viewer` (the bare +launch an AppImage's launcher makes; it cannot be run as a test, because the +viewer finds the account's real data root from the password database). + +Not tests, run by hand on this machine and recorded in `docs/packaging.md`: +the AppImage updating and rolling back itself against a feed signed by +`cargo packager signer` (`just local-release`), which is the check that the +packager's signature format is the one `update` verifies. + +**`linux_packages_install_and_upgrade`** is `deploy/packaging/test-linux-packages.sh` +(`just test-linux-packages`, needs docker; not part of `just check`): in a +clean Debian, Fedora and Arch container it installs release 0.1.0 (Debian +and Fedora from an apt and a dnf repository made by `repo-indexes.sh`, the +same script the release workflow runs; Arch by `makepkg` from the +packager's PKGBUILD), makes books as an ordinary user, holds them open, +adds 0.1.1 to the repository and upgrades; then checks the host restarted +onto the new file, `wanderbooks books list` is unchanged, and +`wanderbooks update` refuses and names the package manager. The programs +are built in a `rust:1.99-trixie` container (`just release-trixie`): +see `docs/packaging.md` for why the build machine's own binary is not used. + +**For P12 (mutation targets, none run, D-33):** `update.rs`'s `verify` +(a mutant that drops the sha check or the signature check must die on +`unsigned_update_refused`'s sha and key cases), `refuse_if_consented`'s +two comparisons (`rollback_refuses_after_a_book_consented_to_a_newer_format` +covers the feature one; a case with a higher format number is a gap to +add), `swap_in`'s temporary-then-rename (`update_is_atomic`), `healthy` +(`failed_update_rolls_back`), `install_kind` (the marker test), and +`parse_version`'s rejections. diff --git a/justfile b/justfile index 1fa699d..a6d248e 100644 --- a/justfile +++ b/justfile @@ -263,3 +263,35 @@ deploy: site # Deploy a preview of the website and print its URL deploy-preview: site {{netlify}} deploy --dir site/dist --no-build + +# The Linux packages (P11a step 3), in target/packages/VERSION. Builds the +# release program first; needs the tools `mise install` pins. A VERSION other +# than the program's repackages the same program under it (the upgrade +# test's second release). +# Build the AppImage, .deb, pacman tarball and .rpm +package-linux version="" formats="deb,appimage,pacman": + cargo build --release -p ledger-cli + deploy/packaging/build-linux.sh {{version}} {{formats}} + +# A release folder on this machine signed with a throwaway key; then +# `wanderbooks update --feed file://DIR/manifest.json --trusted-key DIR/release.pub` +# Make a local release feed to try `wanderbooks update` against +local-release file version dir: + deploy/packaging/local-release.sh {{file}} {{version}} {{dir}} + +# A program linked here does not start on Debian (docs/packaging.md); this +# builds it in rust:1.99-trixie, into target/container. Needs docker. +# Build the release program in a Debian trixie container +release-trixie: + mkdir -p target/container + docker run --rm --user "$(id -u):$(id -g)" -e HOME=/tmp -e CARGO_HOME=/tmp/cargo-home -e CARGO_TARGET_DIR=/target -v "$PWD:/src:ro" -v "$PWD/target/container:/target" -w /src rust:1.99-trixie cargo build --release --locked -p ledger-cli + +# Install each Linux package in a clean Arch, Debian and Fedora container, +# open the books, upgrade to the next release, and check the books' head is +# unchanged and the open books restarted onto the new program. Builds the +# program in a trixie container, then releases 0.1.0 and 0.1.1 of it. +# Install and upgrade the Linux packages in Arch, Debian and Fedora containers +test-linux-packages: release-trixie + WB_PROGRAM=target/container/release/wanderbooks deploy/packaging/build-linux.sh 0.1.0 deb,pacman + WB_PROGRAM=target/container/release/wanderbooks deploy/packaging/build-linux.sh 0.1.1 deb,pacman + deploy/packaging/test-linux-packages.sh diff --git a/mise.toml b/mise.toml index 0cd80e8..88b925e 100644 --- a/mise.toml +++ b/mise.toml @@ -23,3 +23,7 @@ node = "26" # A compile cache shared by every jj workspace's target dir: point # cargo's build.rustc-wrapper at it (~/.cargo/config.toml). sccache = "0.18.0" +# `just package-linux` (docs/packaging.md): the AppImage and .deb by +# cargo-packager, the .rpm by cargo-generate-rpm (D-25). +"cargo:cargo-packager" = "0.11.8" +"cargo:cargo-generate-rpm" = "0.21.1" From d6c90a115ffff6f7e0bc785d04985fbcb231932f Mon Sep 17 00:00:00 2001 From: Stephen Waits Date: Thu, 29 Oct 2026 15:12:18 -0600 Subject: [PATCH 5/6] docs(plan): P11a as built for steps 1 (Linux), 2 and 3 Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01HS8WZziDnyxunC23MUxrLp --- plan/served.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plan/served.md b/plan/served.md index 9aa8303..0e556fa 100644 --- a/plan/served.md +++ b/plan/served.md @@ -1638,7 +1638,7 @@ Each unit follows the 8.0.4 template. "A-n" is an owner answer numbered in 8.3.1 5. **Files:** owns `crates/ledger-host/src/update.rs`, `deploy/packaging/*`, `.github/workflows/release.yml`, `docs/packaging.md`; touches `commands.rs`, `crates/ledger-gui/src/owner.rs`, `mise.toml`. 6. **Tests and proofs:** named above. 7. **Done when:** (1) on a clean Linux VM the AppImage installs, opens, onboards and posts an entry with no terminal and no admin password, and updates itself from a test feed; (2) on Arch, Debian and Fedora the packages install and upgrade with the head unchanged; (3) an update leaves every book's head and goldens unchanged and a bridge session carries on across it; (4) a backup made on one system restores on another; (5) macOS and Windows: the same as (1), once the signing prerequisites exist (record which are done in the As-built block); (6) `just check` green. -8. **As built:** (steps 0 and 1 only, 2026-10-29; steps 2 to 7 are later waves.) **Step 0, partly.** Built: the macOS login Keychain store (`apple-native-keyring-store` 1.0.2, feature `keychain`) and the Windows Credential Manager store (`windows-native-keyring-store` 1.1.0) behind `tokens.rs::Keeper`, on the same `Items` trait and the same laws as the Secret Service (`os` modules per OS; shared `refused`, `make_in`, `read_item`, `remove_item`); `NOT_YET` now only for other OSes; the ignored `os_keyring_keeps_and_forgets` compiles on all three; the `portable` job in `.github/workflows/check.yml` (`macos-latest` gating, `windows-latest` allowed to fail) builds `ledger-cli --no-default-features`, `cargo build --workspace --all-targets`, `cargo test --workspace` and the ignored keyring test. Checked here: Linux only (fmt, clippy, the tokens tests); no macOS/Windows target is installed, so both stores and the job are proven only by the first CI run. **Not built, and not a `cfg` gap: the Windows host.** Windows data root from the known-folder API with an owner-only DACL, named pipes with a SID check, `flock` and the Unix-only `host`, `start`, `bridge`, `opener` and `local`/`admin` sockets are a port of several crates (inventory in `docs/packaging.md`); macOS needs the CI run to show which Linux-only items (`seccomp` in `refuse.rs`, `renameat2`, `close_range`) need fences. So the step 0 Done-when ("setup ... gui work on each OS") is open for both. **Step 1 (the trial): not run**: it needs clean macOS, Windows and Linux machines; `mise.toml` now pins `cargo-packager` 0.11.8, `cargo-packager-updater` 0.2.3, `cargo-generate-rpm` 0.21.1 and `docs/packaging.md` holds the empty table and what is known. +8. **As built:** (steps 0 to 3 as far as stated, 2026-10-29; steps 4 to 7 are later waves.) **Step 0, partly.** Built: the macOS login Keychain store (`apple-native-keyring-store` 1.0.2, feature `keychain`) and the Windows Credential Manager store (`windows-native-keyring-store` 1.1.0) behind `tokens.rs::Keeper`, on the same `Items` trait and the same laws as the Secret Service (`os` modules per OS; shared `refused`, `make_in`, `read_item`, `remove_item`); `NOT_YET` now only for other OSes; the ignored `os_keyring_keeps_and_forgets` compiles on all three; the `portable` job in `.github/workflows/check.yml` (`macos-latest` gating, `windows-latest` allowed to fail) builds `ledger-cli --no-default-features`, `cargo build --workspace --all-targets`, `cargo test --workspace` and the ignored keyring test. Checked here: Linux only (fmt, clippy, the tokens tests); no macOS/Windows target is installed, so both stores and the job are proven only by the first CI run. **Not built, and not a `cfg` gap: the Windows host.** Windows data root from the known-folder API with an owner-only DACL, named pipes with a SID check, `flock` and the Unix-only `host`, `start`, `bridge`, `opener` and `local`/`admin` sockets are a port of several crates (inventory in `docs/packaging.md`); macOS needs the CI run to show which Linux-only items (`seccomp` in `refuse.rs`, `renameat2`, `close_range`) need fences. So the step 0 Done-when ("setup ... gui work on each OS") is open for both. **Step 1 (the trial): not run in that wave**: it needs clean macOS, Windows and Linux machines; `mise.toml` now pins `cargo-packager` 0.11.8, `cargo-packager-updater` 0.2.3, `cargo-generate-rpm` 0.21.1 and `docs/packaging.md` holds the empty table and what is known. **Steps 1 (Linux half), 2 and 3 (2026-10-29, branch `P11a`).** *Pins:* the EM removed step 0's early pins; `mise.toml` now pins `cargo-packager` 0.11.8 and `cargo-generate-rpm` 0.21.1, which `just package-linux` calls; **`cargo-packager-updater` is a library, not a tool, so it is not pinned** (the plan's three-tool list was wrong on that); `update.rs` verifies with `minisign-verify` 0.3 (the crate the updater uses) and the tests sign with the `minisign` 0.10 dev-dependency. *Step 1:* `docs/packaging.md` has the Linux rows filled from real builds (47 MB .deb, 36 MB .rpm, 44 MB AppImage, the pacman tarball and PKGBUILD) and eight findings; the `.dmg` and NSIS rows stay "(trial)" (need a Mac and Windows; Windows is P11c). The findings that shaped the build: a binary linked on this machine (glibc 2.44) does not start on Debian 13, so packages are built on `ubuntu-24.04` in CI and in `rust:1.99-trixie` locally (`just release-trixie`); the books refuse to run under Docker's default seccomp filter (tests use `seccomp=unconfined`); an AppImage's launcher appends arguments to the desktop file's `Exec`, so the desktop files say plain `wanderbooks` and a bare launch with no terminal is the viewer (`main.rs::is_bare_launch`); the AppImage needs a square icon (**a placeholder PNG is committed; the owner should replace it**). *Step 2:* `crates/ledger-host/src/update.rs` (`update`, `rollback`) and the two CLI modes: a manifest (version; per platform `url`, `sha256`, `signature` = base64 of the minisign `.sig`, the shape cargo-packager signs in), accepted only if higher, sha256 equal AND signature valid against `WANDERBOOKS_RELEASE_PUBKEY` (compiled in; **no build carries one until step 5**, so today a build updates only from a `file://` feed with `--trusted-key`); stage under `/updates//`, keep the running version under `updates/previous/` with the journal format it knew, swap by renaming a complete copy made beside the target, `--help` health check with automatic put-back; `rollback` refuses when a book declares a format or feature the kept version lacks; a package marks itself with `share/wanderbooks/install-method` and then both refuse. The restart is the host's existing one (nothing new in `host.rs`). Tests: `crates/ledger-cli/tests/update.rs` (`update_is_atomic`, `failed_update_rolls_back`, `unsigned_update_refused`, `bridge_survives_update`, `update_waits_for_a_running_call`, `package_managed_install_refuses_update`, `rollback_refuses_after_a_book_consented_to_a_newer_format`, `an_update_replaces_the_program_keeps_the_old_one_and_rollback_brings_it_back`) all pass (8 in 12 s); five unit tests in `update.rs`. *Step 3:* `deploy/packaging/` (`packager.toml`, the rpm table in `crates/ledger-cli/Cargo.toml`, `aur/PKGBUILD`, `build-linux.sh`, `repo-indexes.sh`, `local-release.sh`, `test-linux-packages.sh`), the `just` recipes `package-linux`, `local-release`, `release-trixie`, `test-linux-packages`, and `.github/workflows/release.yml` (Linux packages on a `v*` tag, attached to the release, apt and dnf indexes pushed to `gh-pages`; untested, GitHub only). `linux_packages_install_and_upgrade` is `test-linux-packages.sh`: install 0.1.0 from an apt, a dnf repository and `makepkg` over the AUR PKGBUILD in Debian trixie, Fedora and Arch containers, hold books open as an ordinary user, upgrade to 0.1.1, host restarted onto the new file, journal head unchanged, `update` refuses: passes on all three. By hand: the AppImage updated itself and rolled back against a feed signed by `cargo packager signer` (`just local-release`). *Done-when so far:* (1) **partly**: the AppImage opens (bare launch) and updates itself from a test feed, but "onboards and posts an entry with no terminal" is false today (the viewer opens only an existing book and `setup`/`books create` need a terminal; not in these steps), and a held-open books' host does not restart onto an updated AppImage (it runs from the old image's mount; fix needs the host to learn the image's path, a trust decision left to the EM); (2) true on Debian, Fedora and Arch containers; (3) true for a hand-placed program (`bridge_survives_update`) and for packages (the container test), not yet for the AppImage's host; (4) not touched; (5) and (6) not done (macOS/Windows need machines and signing; `just check` is the EM's). *Not done, as instructed:* step 4 (daily check, staged update applied at restart), step 5 (signing, Q-4), step 6 (macOS and Windows bundles), step 7 (architecture note is written; the website and `docs/parity.md` are not touched, since nothing is published). #### P11b: the agent dock and books homes (D-27) From 169b914e1f16b18bb03c534ac8e486c4a0c9a7ae Mon Sep 17 00:00:00 2001 From: Stephen Waits Date: Thu, 29 Oct 2026 15:25:37 -0600 Subject: [PATCH 6/6] fix: what wave 1's gate found: B-43 speaks CHOICE, a Linux-only socket test is Linux-only GitHub's check on f043db35: bookspec's `rules_md_carries_every_id` refuses "DECISION" in docs/spec/rules.md (bookspec's word is CHOICE, P13 step 3); P4's B-43 wrote "DECISION D-47" twice. The portable macOS job: `a_socket_with_no_other_end_names_no_process_and_isnt_us` uses `SOCK_CLOEXEC` and Linux's answer for a peerless socket; it is gated to Linux, which is what it tests. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01HS8WZziDnyxunC23MUxrLp --- crates/ledger-mcp/src/local.rs | 4 +++- docs/spec/rules.md | 4 ++-- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/crates/ledger-mcp/src/local.rs b/crates/ledger-mcp/src/local.rs index 4511b2e..256619f 100644 --- a/crates/ledger-mcp/src/local.rs +++ b/crates/ledger-mcp/src/local.rs @@ -2640,7 +2640,9 @@ mod tests { /// A socket with no other end: the system answers for it (Linux: no /// process, no user), and that answer names neither this process nor - /// this account; the client's check refuses it. + /// this account; the client's check refuses it. Linux only: the + /// answer and `SOCK_CLOEXEC` are Linux's (the portable job found it). + #[cfg(target_os = "linux")] #[test] fn a_socket_with_no_other_end_names_no_process_and_isnt_us() { use std::os::fd::FromRawFd; diff --git a/docs/spec/rules.md b/docs/spec/rules.md index ff76422..585f1bd 100644 --- a/docs/spec/rules.md +++ b/docs/spec/rules.md @@ -3436,7 +3436,7 @@ The Lean step is `Documents.Step.document` with every control line tagged **bookspec:** `BS-entry-5` **Status:** AUTHORITY (the evidence a draft waits for, Pub. 583 and Rev. -Proc. 98-25); DECISION D-47 (owner-adopted recommendation, 2026-10-26: +Proc. 98-25); CHOICE D-47 (owner-adopted recommendation, 2026-10-26: reversals stay posted and need no evidence of their own); CONVENTION (the two states, and discarding a draft rather than posting and reversing it). @@ -3448,7 +3448,7 @@ GOAL: no entry the receipt policy covers posts without the document that SOURCE: IRS Pub. 583 (docs/spec/sources/irs-records.md: "you must keep supporting documents ... they support the entries in your books"); Rev. Proc. 98-25 §5.01 (the records must let the books be - demonstrated); DECISION D-47 (a reversal's evidence is its + demonstrated); CHOICE D-47 (a reversal's evidence is its original's, D-12); CONVENTION (a draft's state is a note: it moves no figure). INVARIANTS: