wanderbooks/docs/receipt-store.md

Receipt & document store + persistence spec

Three pieces: the journal (the source of truth), the blob store (the bytes), and SQLite (the queryable projection). All immutable-first, matching the ledger's philosophy: the books never forget, and neither does the filing cabinet.

The journal: the source of truth

SQLite is the query engine, not the source of truth. The source of truth is an append-only journal of domain events: one JSON object per line (JSONL), hash-chained. This is the deliberate answer to "insanely durable, easily backupable, journable":

• Durable. Every state change is an immutable fact appended to journal.jsonl, fsync'd per append. Each line carries the SHA-256 of the previous line's canonical bytes (prev), forming a tamper-evident chain: bit rot, torn writes, and casual editing are detected on read/verify, not discovered at tax time. Honest scope: this detects corruption and casual tampering. It does not stop a determined attacker with full write access from rewriting the chain — nothing local does.
• Backupable. The backup is cp -r $WANDERBOOKS_DATA /backup: the journal file plus the blob directory. Plain files, no WAL-file timing windows, no special backup API. A copy taken mid-write, like a crash or power loss mid-append, can end in a cut-off line (bytes after the last newline). It was never acknowledged, so opening the books saves those bytes to journal.jsonl.torn.<time>, truncates to the last newline, and journals a journal.tail_repaired line; books.digest reports it. Damage before the last newline is corruption and refuses to open: that is for a human, never repaired automatically. The server does better (plan 3.7): it commits the journal, the blobs, the vault, its key and the access log to each book's own git repository (the blobs staged ahead, the commit under the store's lock) and pushes it to the owner's private repository and backup folder, versioned and deduplicated, with full history (docs/mcp-interface.md, "Tools — backups").
• Journable. It literally is a journal: {seq, ts, type, payload, prev}. Greppable, diffable, replayable. The audit trail isn't a feature bolted on — it's the storage format.

Write path

There is exactly one write path: append to the journal, fsync, then apply to SQLite. No dual writes, no background applier thread (the volume doesn't justify it): the MCP server appends, fsyncs, applies the new lines to the projection synchronously, and returns. A crash between append and apply is safe — on startup the applier replays every journal line with seq > last_applied_seq.

Line format

{"seq": 42, "ts": "2026-10-10T18:20:00Z", "type": "entry.posted",
 "payload": { "...": "..." }, "prev": "abc123..."}

• seq: 1-based, gapless, assigned by the writer.
• ts: UTC, RFC 3339. Recorded, never trusted for ordering (seq orders).
• type: account.added, entry.posted, entry.reversed, invoice.created, bill.created, invoice.sent (a recorded send: a document's record, never a ledger line or a figure; P4b S5), invoice.written_off (P4b S7), receipt.attached, receipt.linked, receipt.unlinked, document.filed, attachment.withdrawn, attachment.text_extracted, attachment.annotated, entry.settles (P6), entry.draft_state_set (a draft building or awaiting its evidence: a note on the draft, no figure) and entry.draft_discarded (a draft thrown away, with why; the draft's earlier lines stay in the journal) (P4, feature drafts_v2), and convention.source_revised {id, source, reason}, decision.source_revised {id, source: {decided_by: [..], decided_on, citation?}, reason} and open_item.source_revised {id, source, reason} (P5, feature source_revisions: who decided a convention, a decision or an open item, when and on what, corrected without touching its words; replay sets the source and nothing else, and the projection updates the row's existing source columns, so no schema changed) audit.run, receipt.attested, entry.reviewed and period.attested (P8, feature audits: claims about the books, no figure) (plus the A/R, A/P, asset, period, payroll, and class events listed on JournalLine).
• payload: the full domain facts (account fields, entry with lines, blob sha, ...). Everything the applier needs; nothing it must look up elsewhere. An invoice.created or bill.created written after the documents_v2 upgrade (plan P4b S4.3) carries only the new fields it names, so a document naming none is byte for byte the payload every earlier version wrote: the invoice's number, terms, bill_to and seller snapshots (as they were at issue), customer_ref and memo; the bill's vendor_ref, received_on, terms, memo and the duplicate_reason that allowed a duplicate of a standing vendor invoice; and each line's account, quantity and unit_price. The issued copy of an invoice is not a field on the event (its attachment id is allocated after the entry exists): it is a receipt.attached on the invoice's own entry (INV-0007.pdf, application/pdf, "Issued invoice"), its SHA-256 the blob_sha, linked to the invoice with attachment.linked and found through that link (plan 8.3.11). An invoice.written_off (plan 8.3.6, P4b S7) carries {invoice_id, entry_id, date, amount, reason}; replay tags the entry's receivables line WriteOff, and the projection keeps no written-off figure either: it is the tagged lines' sum (rules.md B-38, B-41). Its evidence of worthlessness rides beside it, as a receipt.attached on the write-off's entry and an attachment.linked to the invoice, filed in the same call.
• prev: hex SHA-256 of the previous line's canonical JSON (keys sorted, no whitespace). The genesis line uses "prev": "000...0" (64 zeros).

Drafts are journaled (durable)

Draft entries are durable: entry.created and entry.line_added journal lines record the draft as the agent builds it, so drafts (and receipts attached to them) survive server restarts. Drafts never touch the SQLite projection until posted — reports only see posted entries. A draft that never posts leaves its created/line_added lines in the journal, which is correct: the journal records what happened, including abandoned work.

Verification

wanderbooks verify (a ledger-store API, later a CLI subcommand): re-read the journal, recompute the hash chain, replay every line into a fresh in-memory Ledger, and compare its reports against the SQLite projection's. Any mismatch is a loud failure. Run it on startup (cheap at this scale); a backup is then proven where it lands, by a drill (docs/mcp-interface.md, "Tools — backups").

Blob store

• Content-addressed. The key is SHA-256 of the exact bytes. Path: $DATA/blobs/<first2>/<next2>/<full-hex>. Same bytes, same path, forever.
• Immutable. Blobs are never modified or deleted by the application. A retention/deletion policy is an explicit future human decision, not an accident of implementation.
• Deduplicated. The same bytes are one blob, and since P6 one live attachment record: filing them again returns that record (with deduplicated: true) and links it to the new subject instead of adding a second row. Records an older version filed twice stay as written until books.consolidate_duplicates links the oldest to every subject the others had and withdraws the rest (duplicate_of).
• Limits. 25 MiB per file, as an upload (books.begin_upload): the only way a file comes in (D-46: nothing is given inline). The name and MIME type are the client's, checked when the file is filed: a name of 1 to 255 bytes, no / or \, not . or .., no control or invisible characters; a type/subtype of visible ASCII, at most 127 bytes (docs/mcp-interface.md, "Tools — receipts & documents"). The type is stored as given; server-side sniffing is future work (only a W-9 is judged by its first bytes).
• Metadata lives in SQLite (blobs table), not in the filesystem: sha256 PK, size_bytes, mime_type, original_name, created_at.

Attachments & documents

• attachments: id PK, blob_sha FK, entry_id FK NULL, category, description, created_at.
  • entry_id is nullable: a receipt can be filed before its journal entry exists, then linked with books.link_receipt.
  • category mirrors today's Drive layout: formation, legal, tax, accounting (today's 01 Formation / 02 Legal / 03 Tax / 04 Accounting), plus receipts, and every other company record (D-20): statement, contract, photo, warranty, import_source, notices, correspondence, plans, history, other.
  • duplicate_of names the attachment a withdrawn duplicate stands for (attachment.withdrawn's optional duplicate_of, schema v16).
• Documents filed via books.file_document and receipts via books.attach_receipt share the blob store; they differ only in metadata.
• Receipts change by events, never by edits. receipt.unlinked (attachment_id, entry_id, reason) detaches a receipt from its entry; attachment.withdrawn (attachment_id, reason, duplicate_of?) retires an unlinked receipt or a document (withdrawn_at, withdrawn_reason, duplicate_of columns, schema v5 and v16). Linking requires an unlinked, non-withdrawn attachment, so evidence only leaves an entry through a recorded unlink. Blobs are untouched: a withdrawn file is still hash-verified and readable, and the journal shows when it was filed, linked, unlinked, and withdrawn, and why.
• Extraction is derived, and journaled (P6, rules.md B-44): attachment.text_extracted {attachment_id, text_sha256, extractor, version, pages, status} records what reading a file found (ok, none, failed); the text itself is a content-addressed blob like any other, and books.verify checks its presence and hash, never re-derives it. One record per attachment, extractor and version: an extractor upgrade adds a record, never replaces one.
• Annotations and settlements are records, not edits (P6): attachment.annotated {attachment_id, fields, note?} is what a human read off a file (the closed field list; a later annotation replaces the earlier record), and entry.settles {entry_id, settles[]} says which entries a payment pays back (the only source of "reimbursed").
• Audits and attestations are records, not figures (P8, rules.md B-45, feature audits): audit.run {id, through, through_seq, scope, program_version, tb_hash, findings_sha256, workpaper_attachment?, workpaper_pdf?} says a programme of checks was run through a day at the books' figure head through_seq (the last line that bears on a figure; audits, attestations and filings of evidence do not move it), and which two filed documents (category workpaper) hold its findings; receipt.attested {attachment_id, entry_id, by, at, method, amount_checked, date_checked, disagrees, note?}, entry.reviewed {entry_id, by, at, method, note?} and period.attested {period_id, by, at, method, audit_run_id} are a named client's claim that it checked. method ({model?, reasoning_effort?, adversarial?, verifier_agents?}) is that client's own word. Whether a claim counts is never journaled: it is derived against policy.attestation_bar when asked.
• The store is policy-free: it records attachments faithfully. Whether a posting requires a receipt is decided by the MCP layer's receipt policy (docs/mcp-interface.md), which reads the attachments table.

ledger-store: the SQLite projection

SQLite is the read-optimized projection of the journal. Queries, reports, the MCP server's reads, and the future GUI all hit SQLite — and SQLite is disposable by design: rebuild drops it and replays the journal. Suspected corruption is fixed by rebuild, never by forensics.

• Location. $WANDERBOOKS_DATA/wanderbooks.v{N}.db, falling back to ~/.local/share/wanderbooks/wanderbooks.v{N}.db, N the schema: each schema keeps a file of its own, and a version never migrates, writes or deletes another schema's (ledger_store::db_file_for; the legacy wanderbooks.db is used only while it holds this version's schema; rules.md B-27). The journal lives next to it at $WANDERBOOKS_DATA/journal.jsonl; blobs under $WANDERBOOKS_DATA/blobs/. WAL mode on, synchronous=FULL, integrity_check on startup.
• Schema. meta(key, value) — schema_version, base_currency, last_applied_seq; accounts, entries, lines, blobs, attachments as above; extractions(attachment_id, extractor, version, text_sha256, pages, status, seq) and annotations(attachment_id, receipt_number, payee, amount, date, instrument, category, source, reimbursable_to, note, seq) (P6; amounts TEXT, never REAL); audits(id, through, through_seq, scope, program_version, tb_hash, findings_sha256, workpaper_attachment, workpaper_pdf, seq) and attestations(seq, kind, attachment_id, entry_id, period_id, audit_run_id, by_who, at_ts, method, amount_checked, date_checked, disagrees, note) (P8, schema 20); events(seq, type, payload, created_at) — the applier's output, a queryable mirror of the journal stream and the feed the future GUI tails. (Poll WHERE seq > ?; later SSE over the HTTP transport. This is the correct answer to "watch its files to update live" — watch the event stream, not the files.)
• Document fields. Since schema v16 (plan P4b S4.3) the documents' new fields are columns: invoices(number, terms, bill_to, seller, customer_ref, memo) and bills(vendor_ref, received_on, terms, memo, duplicate_reason), with terms and snapshots as canonical JSON TEXT and a document written before them NULL there. A line's account, quantity and unit_price ride its document's lines_json, as its project does. The party's email and default terms are journaled and replayed into the ledger; the projection keeps no party table to hold them.
• Amounts as TEXT, never REAL. Unchanged and still load-bearing: a REAL column would silently reintroduce binary floats into the money path.
• Append-only, three layers. (1) The engine's API never mutates posted state. (2) The journal is append-only by file discipline plus the hash chain. (3) The applier only ever INSERTs into the projection. The SQLite triggers from the earlier draft are deliberately dropped: they protected the projection, which is disposable — the journal is the trust anchor, and piling mechanisms on the disposable layer adds complexity without adding safety. Simplicity is a durability feature too.
• Migrations. Numbered SQL files, applied in order, bumping meta.schema_version. Every migration is tested by rebuilding a projection from the journal and comparing reports before/after. A newer schema version than the binary understands is a hard error, not a guess.

Migration from Google Drive (done)

The company's records were filed into the books at the P1 cutover (2026-10-17, D-20: formation papers, notices, plans and history, through books.file_document with their categories) and the books cut over on 2026-10-20. The Drive manifest and comparison tools once planned for P6 were struck (plan/served.md 8.4): the Drive folder is frozen history. P6 (on its branch) adds the categories notices, correspondence, plans, history and text extraction for every filed document.