merge: P6, the evidence pipeline (branch P6, checkpoint ce366408)

The last of the paused branches, from its "not for merge" checkpoint:
extraction in ledger-import (text, pages, page images; capped, caught),
journaled as attachment.text_extracted; dedup by hash with
books.find_attachment and books.consolidate_duplicates; annotations and
books.set_entry_settles; the company record categories; the two fuzz
targets; feature attachment_text. Nineteen files conflicted with the
four P4b merges, resolved as plan 8.0.2 says: P6 retakes schema 18 (it
had taken 16, which S4.3 holds; S5 holds 17); the tool table is 129 and
the destructive count 31; both sides' appends kept with their seams
restored; one B-27 paragraph naming every feature; in the old-books
fixture P6's hand-written reimbursement is entry 7 (its ids collided
with S4.3's invoice); the replay fuzzer's list gains S7's
invoice.written_off; S4.3's issued-PDF test expects the extraction
record P6 journals right after a filing.

One integration defect between S7 and P6, found by the arap property
test and fixed here: under P6's dedup, a write-off whose evidence bytes
were filed already (the same collection letter for a second write-off of
the same invoice) was refused only after its entry had been journaled,
at the final link to the invoice ("already linked"). The write-off now
resolves the dedup before anything is journaled and keeps a link the
record already has to the invoice instead of refusing it; an explicit
existing_attachment_id already on an entry is still refused (S7's
one-receipt-once rule). P6's gate never ran before the pause; it runs on
main after this merge, and its ledger-store -> ledger-import dependency
is the open review point (plan 8.0.1).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HS8WZziDnyxunC23MUxrLp
This commit is contained in:
swaitsandClaude Fable 5.1 committed 2026-10-28 23:08:59 -06:00
commit 2ad192bbcd
70 files changed
+4618 -384

No files matched your search

+20
View File
@@ -115,6 +115,24 @@ the agent's cannot drift apart. The crate depends inward (`ledger-core`,
`ledger-store`, `ledger-config`, `ledger-import`, `ledger-vault`), never on
`ledger-mcp` or `ledger-gui`.
## The evidence pipeline (`ledger-import`, `ledger-core`, `ledger-store`, P6)
One extractor lives in `ledger-import` (`src/extract.rs`, D-53: in-process,
no new crate): a PDF's text and pages, an XLSX's cells as exact decimal
strings, and plain text; the caps are 25 MiB in, 30 pages and 2 MiB of
text, and a reader panic is caught there. `ledger-store` depends on
`ledger-import` so the writer can read a file as it is filed and journal
`attachment.text_extracted` (content-addressed text blob, extractor,
version, pages, `ok`/`none`/`failed`), one record per attachment, extractor
and version; `Store::verify` checks the text blob's hash and never
re-derives it. The server's filing scan (`tin_files`) and the viewer's PDF
rasterizer read through the same module, so a PDF's text is read one way.
Annotations and settlements are ledger state (`ledger_core::evidence`,
`Ledger::differs_from`), projected into schema 16's `annotations` table;
extractions have a table of their own. The viewer's Documents page groups
by category and filters by subject; Needs attention reads both through
`PackageBooks`.
## The journal format gate (`ledger-store`, P0)
Every journal line passes a gate after its parse, seq and chain checks
@@ -187,6 +205,8 @@ groups are:
- `write_offs` — `books.write_off_invoice`, the bad debt (P4b S7; rules.md
B-41), kept apart from `documents` so S4.4's document tools and this one
land in different files;
- `evidence` — finding a file by its hash, number or text, annotating it,
recording what a payment settles, and consolidating duplicates (P6);
- `settings` — the handbook, `books.settings` and the recorded FX rates
(the module is `settings_tools`, since `crate::settings` is the live
settings);
+80 -15
View File
@@ -365,7 +365,8 @@ that changes anything also has a `destructiveHint`, true for these 30:
`books.set_tin` (both overwrite a vault record), `books.purge_tin`,
`books.create_invoice`, `books.create_bill`, `books.receive_payment`,
`books.pay_bill`, `books.void_document`, `books.write_off_invoice`,
`books.run_payroll`, `books.remit_payroll_taxes`; and an
`books.run_payroll`, `books.remit_payroll_taxes`,
`books.consolidate_duplicates`; and an
`idempotentHint` on the ten that a repeat with the same arguments leaves
the journal and the vault as they were: `books.verify`,
`books.run_depreciation`, `books.run_schedules`, `books.run_recurring`,
@@ -1598,22 +1599,82 @@ and no control or invisible (Unicode format) characters; a MIME type
subject it is linked to, its filing entry first). Withdrawn attachments
are omitted unless `include_withdrawn` is true. `entry_id` lists the
files filed on the entry and linked to it.
- `books.get_receipt { attachment_id, view?: "meta" | "bytes" }` → `{
attachment_id, sha256, entry_id?, category, file_name, mime_type,
size_bytes, description?, links, file? }`. `meta` (the default) returns
no file. `bytes` adds `file`, the file handed over (a `HandedFile`, read
hash-verified, whatever its size: D-46), whose `download` saves a copy
as a new file in a folder the human names. The stored file is also
fetched by `wanderbooks download --book B <id> --to <a folder>` (`GET
- `books.get_receipt { attachment_id, view?: "meta" | "text" |
"page_image" | "bytes", page?, max_px? }` → `{ attachment_id, sha256,
entry_id?, category, file_name, mime_type, size_bytes, description?,
links, text?, text_truncated, text_status, pages, receipt_number?,
annotation?, file? }`. `meta` (the default) returns no file: the first
8 KiB of the file's extracted text (`text`, `text_truncated` when cut),
its page count (`pages`), and `text_status` (`ok`, `none`, `failed`, or
`not_extracted` for books filed before this version: `wanderbooks books
extract`). `text` gives the whole extracted text (past 64 KiB it comes
as a handed-over `.txt` instead). `page_image` with `page` (1 is the
first) and `max_px` (default 1600, longest side) gives one page as a
PNG, handed over (D-46; D-15: no OCR, so a picture-only receipt is read
from its pages). Read the text first; a page picture only when the text
is empty or looks wrong; never `bytes` into context. `bytes` adds
`file`, the file handed over (a `HandedFile`, read hash-verified,
whatever its size: D-46), whose `download` saves a copy as a new file
in a folder the human names. The stored file is also fetched by
`wanderbooks download --book B <id> --to <a folder>` (`GET
/b/{book}/a/{attachment}`, books:read, "Transports").
- `books.file_document { category: "formation" | "legal" | "tax" |
"accounting" | "statement" | "contract" | "photo" | "warranty" |
"import_source" | "other", file_name?, mime_type?, upload_id,
description?, link_to? }` → `{ attachment_id,
sha256, deduplicated, links }`. The bytes, name and type as in
`books.attach_receipt`. The Drive-folder replacement (today's 01
Formation / 02 Legal / 03 Tax / 04 Accounting). Never a W-9: it shows
the full TIN (`books.set_w9 w9_upload_id`).
"import_source" | "notices" | "correspondence" | "plans" | "history" |
"other", file_name?, mime_type?, upload_id, description?, link_to? }` →
`{ attachment_id, sha256, deduplicated, links }`. The bytes, name and
type as in `books.attach_receipt`. Every company record has a home
(D-20): the Drive folders' replacement, and formation papers,
registrations, plans, notices, correspondence and history besides what
the CPA asks for. Never a W-9: it shows the full TIN (`books.set_w9
w9_upload_id`).
- **One file, one record** (plan P6 step 4): filing bytes already stored
(and not withdrawn) returns that attachment's `attachment_id` with
`deduplicated: true` and links it to the new subject, instead of filing
a second record. Records an older version filed twice stay as written
until `books.consolidate_duplicates`.
## Tools — evidence
What the books know about a filed file beyond its bytes (plan P6): its
extracted text, the facts a human read off it, and which entries a payment
settles. Extraction is journaled as it is made (`attachment.text_extracted`,
content-addressed, one record per attachment, extractor and version): an
extractor upgrade adds a record, never replaces one, and `books.verify`
checks the text blob's presence and hash.
- `books.find_attachment { sha256 | receipt_number | text }` → `{
attachments: [...] }`. Exactly one of the three: the file's exact
SHA-256 (64 hex digits), the number recorded with
`books.annotate_attachment`, or words its extracted text says
(case-insensitively). Withdrawn files are listed too, marked, so a
duplicate is found before it is consolidated. Each attachment is as in
`books.list_receipts`, with its links.
- `books.annotate_attachment { attachment_id, fields { receipt_number?,
payee?, amount?, date?, instrument?, category?, source?,
reimbursable_to? }, note? }` → `{ attachment_id, fields,
next_receipt_number }`. The closed list of what a human reads off a
file: `amount` an exact decimal string, `date` YYYY-MM-DD, `instrument`
a payment method's id or label. It changes no amount, no file and no
posting; a later annotation replaces it, journaled as
`attachment.annotated`. `next_receipt_number` is one past the highest
number annotated, so numbers run without reuse. At least one field or
the note is required; an unknown or withdrawn attachment is refused.
- `books.set_entry_settles { entry_id, settles }` → `{ ok, entry_id,
settles }`. Records which entries an entry pays back (a reimbursement
settling the expenses a member paid personally), journaled as
`entry.settles`; an empty list clears it. Every named entry must exist,
none may be the paying entry itself or be named twice. It changes no
amount. "The member was paid back" is read from this record only, never
guessed (IZ-18); Needs attention asks until it is recorded.
- `books.consolidate_duplicates { confirmed? }` → `{ status: "preview" |
"consolidated", groups: [{ sha256, keep, withdraw, links }], withdrawn
}`. For every set of attachments holding the same bytes it keeps the
oldest (`keep`), links it to every subject the others had, unlinks the
others and withdraws them as duplicates (`attachment.withdrawn` with
`duplicate_of`; nothing is deleted). A posted entry that would be left
with no receipt the policy requires refuses the whole call, journaling
nothing. Without `confirmed`, a preview that changes nothing.
## Receipt enforcement
@@ -1868,8 +1929,12 @@ Hand-rolled on `ServerHandler` (rmcp 3.x has no resource macros):
this month so far
- `books://handbook` (the index) and `books://handbook/{section}` (one per
section, Markdown)
- `books://receipts/{id}` — receipt metadata (as `books.get_receipt`
- `books://receipts/{id}` (an alias of `books://attachments/{id}`) —
receipt metadata and the first 8 KiB of its text (as `books.get_receipt`
view `meta`; no bytes)
- `books://attachments/{id}/page/{n}` — page `n` (from 1) as a PNG blob,
the first binary resource (plan P6; D-46: the bytes are the resource's
contents, never a path)
Resources exist so read-only clients (dashboards, the viewer,
auditors) never need tool credentials. Tested end to end through a real
+17
View File
@@ -173,3 +173,20 @@ stay derived, and its evidence of worthlessness filed in the same call
-3(a)). A same-year recovery is the reversal D-49 records, not a new
movement; bills are never written off. No row above changed status (the
A/R and A/P rows already say `done`), so the floor stands (10 done).
## P6: the evidence pipeline (2026-10-26–27)
No semantic feature of the catalog above moved, so no status changed and
the floor stands (10 done): P6 changes how evidence is kept, not how a
figure is computed. It adds one in-process extractor (a PDF's text and
pages, an XLSX's cells as exact decimal strings, plain text; 25 MiB, 30
pages, 2 MiB of text), journals each extraction as its own record (one per
attachment, extractor and version, its text a content-addressed blob whose
hash `books.verify` checks), holds identical bytes as one live attachment
(linked to every subject, legacy duplicates consolidated with
`duplicate_of`), records what a human read off a file (the closed
annotation field list, engine receipt numbers running from the max) and
what a payment settles (`entry.settles`, the only source of "reimbursed"),
and gives company records four more homes (`notices`, `correspondence`,
`plans`, `history`, D-20). Rules.md B-44 states it; `docs/testing.md`'s P6
section lists its tests and the P12 mutation targets.
+36 -15
View File
@@ -57,13 +57,11 @@ apply is safe — on startup the applier replays every journal line with
- `seq`: 1-based, gapless, assigned by the writer.
- `ts`: UTC, RFC 3339. Recorded, never trusted for ordering (`seq` orders).
- `type`: `account.added`, `entry.posted`, `entry.reversed`, `invoice.created`,
`bill.created`, `receipt.attached`, `receipt.linked`, `receipt.unlinked`,
`document.filed`, `attachment.withdrawn`, `invoice.sent` (a recorded
send: a document's record, never a ledger line or a figure; P4b S5)
(plus the A/R, A/P, asset, period, payroll, and class events listed on
`JournalLine`).
`invoice.written_off`, `bill.created`, `receipt.attached`, `receipt.linked`, `receipt.unlinked`,
`document.filed`, `attachment.withdrawn` (plus the A/R, A/P, asset,
`bill.created`, `invoice.sent` (a recorded send: a document's record,
never a ledger line or a figure; P4b S5), `invoice.written_off` (P4b S7),
`receipt.attached`, `receipt.linked`, `receipt.unlinked`,
`document.filed`, `attachment.withdrawn`, `attachment.text_extracted`,
`attachment.annotated`, `entry.settles` (P6) (plus the A/R, A/P, asset,
period, payroll, and class events listed on `JournalLine`).
- `payload`: the full domain facts (account fields, entry with lines, blob
sha, ...). Everything the applier needs; nothing it must look up elsewhere.
@@ -114,9 +112,12 @@ a backup is then proven where it lands, by a drill (docs/mcp-interface.md,
- **Immutable.** Blobs are never modified or deleted by the application. A
retention/deletion policy is an explicit future human decision, not an
accident of implementation.
- **Deduplicated.** Attaching the same receipt twice stores one blob and two
attachment rows. `attach_receipt` / `file_document` return
`deduplicated: true` when the blob already existed.
- **Deduplicated.** The same bytes are one blob, and since P6 one live
attachment record: filing them again returns that record (with
`deduplicated: true`) and links it to the new subject instead of adding
a second row. Records an older version filed twice stay as written until
`books.consolidate_duplicates` links the oldest to every subject the
others had and withdraws the rest (`duplicate_of`).
- **Limits.** 25 MiB per file, as an upload (`books.begin_upload`): the
only way a file comes in (D-46: nothing is given inline). The name and MIME type are the client's, checked
when the file is filed: a name of 1 to 255 bytes, no `/` or `\`, not
@@ -135,17 +136,33 @@ a backup is then proven where it lands, by a drill (docs/mcp-interface.md,
exists, then linked with `books.link_receipt`.
- `category` mirrors today's Drive layout: `formation`, `legal`, `tax`,
`accounting` (today's 01 Formation / 02 Legal / 03 Tax / 04 Accounting),
plus `receipts`.
plus `receipts`, and every other company record (D-20): `statement`,
`contract`, `photo`, `warranty`, `import_source`, `notices`,
`correspondence`, `plans`, `history`, `other`.
- `duplicate_of` names the attachment a withdrawn duplicate stands for
(`attachment.withdrawn`'s optional `duplicate_of`, schema v16).
- Documents filed via `books.file_document` and receipts via
`books.attach_receipt` share the blob store; they differ only in metadata.
- **Receipts change by events, never by edits.** `receipt.unlinked`
(`attachment_id`, `entry_id`, `reason`) detaches a receipt from its entry;
`attachment.withdrawn` (`attachment_id`, `reason`) retires an unlinked
receipt or a document (`withdrawn_at`, `withdrawn_reason` columns, schema
v5). Linking requires an unlinked, non-withdrawn attachment, so evidence
`attachment.withdrawn` (`attachment_id`, `reason`, `duplicate_of?`)
retires an unlinked receipt or a document (`withdrawn_at`,
`withdrawn_reason`, `duplicate_of` columns, schema v5 and v16). Linking requires an unlinked, non-withdrawn attachment, so evidence
only leaves an entry through a recorded unlink. Blobs are untouched: a
withdrawn file is still hash-verified and readable, and the journal shows
when it was filed, linked, unlinked, and withdrawn, and why.
- **Extraction is derived, and journaled** (P6, rules.md B-44):
`attachment.text_extracted {attachment_id, text_sha256, extractor,
version, pages, status}` records what reading a file found (`ok`,
`none`, `failed`); the text itself is a content-addressed blob like any
other, and `books.verify` checks its presence and hash, never re-derives
it. One record per attachment, extractor and version: an extractor
upgrade adds a record, never replaces one.
- **Annotations and settlements are records, not edits** (P6):
`attachment.annotated {attachment_id, fields, note?}` is what a human
read off a file (the closed field list; a later annotation replaces the
earlier record), and `entry.settles {entry_id, settles[]}` says which
entries a payment pays back (the only source of "reimbursed").
- The store is policy-free: it records attachments faithfully. Whether a
posting *requires* a receipt is decided by the MCP layer's receipt policy
(`docs/mcp-interface.md`), which reads the `attachments` table.
@@ -168,7 +185,11 @@ Suspected corruption is fixed by rebuild, never by forensics.
`integrity_check` on startup.
- **Schema.** `meta(key, value)` — `schema_version`, `base_currency`,
`last_applied_seq`; `accounts`, `entries`, `lines`, `blobs`, `attachments`
as above; `events(seq, type, payload, created_at)` — the applier's output,
as above; `extractions(attachment_id, extractor, version, text_sha256,
pages, status, seq)` and `annotations(attachment_id, receipt_number,
payee, amount, date, instrument, category, source, reimbursable_to,
note, seq)` (P6; amounts TEXT, never REAL); `events(seq, type, payload,
created_at)` — the applier's output,
a queryable mirror of the journal stream and the feed the future GUI
tails. (Poll `WHERE seq > ?`; later SSE over the HTTP transport. This is
the correct answer to "watch its files to update live" — watch the event
+81 -1
View File
@@ -1753,7 +1753,11 @@ that moves no figure); S6's payments with `line`, `discount`,
`invoice.written_off`. An older version would
restore the document without them silently: every invoice line would post
to the revenue role and every bill line to general expenses, and every new
event type would be refused. A command asks for
event type would be refused. P6's is `attachment_text` (an
`attachment.text_extracted`, `attachment.annotated` or `entry.settles`, a
new event type, and an `attachment.withdrawn` carrying `duplicate_of`; an
older version would refuse the types, or withdraw a duplicate without
saying what it duplicates, B-44). A command asks for
the feature it needs (`Store::ensure_feature`) before its first journaled
line, and only when the line it is about to write uses it, so a write
refused for waiting on the upgrade journals nothing, never an entry
@@ -3316,3 +3320,79 @@ check_discount}`, `arap::{Application, Fee, SettleParams, SettlementRow}`,
`ledger-cli` `tests/documents.rs`; `ledger-package`'s aging tie-out test.
The Lean step is `Documents.Step.document` with every control line tagged
(`Documents.settle_unattached`); no new `Step` constructor.
## B-44 Evidence is read from the bytes, never trusted over them; identical files are kept once
```text
GOAL: a receipt's text and the facts read off it are derived records the
books can re-make; the file's bytes stay the authority; the same
bytes are one file wherever they are evidence; and a payment says
what it settles instead of the books guessing.
SOURCE: Rev. Proc. 98-25 §5.01 (machine-sensible records with an audit
trail: the relationship between the records and the books must be
demonstrable); IRS Pub. 583 (supporting documents are kept with
the entries they support); plan/served.md P6 (defects 4 and 10);
IZ-18 (no FIFO guess);
CONVENTION (the extractor's version is part of its record; a
failed read is recorded, not hidden; a duplicate is withdrawn, not
deleted).
INVARIANTS:
1. An attachment's text is a record (`attachment.text_extracted`) with
the text's content-addressed blob, the extractor and its version, the
page count and `ok`, `none` or `failed`. One record per attachment,
extractor and version: an upgrade adds a record and NEVER replaces
one, and the bytes are never re-derived by `verify` (its hash is
checked).
2. Extraction is derived and journaled, never authoritative: a report,
a posting or an audit reads the file's bytes when the text matters,
and a failed extraction is a record that a human must read the
pages (D-15: no OCR).
3. Identical bytes are one file: filing bytes already stored (and not
withdrawn) returns the existing attachment, linked to the new
subject. Consolidating legacy duplicates keeps the oldest, links it
to every subject the others had, and withdraws the others with
`duplicate_of`; nothing is deleted and no posted entry the receipt
policy covers is left without evidence.
4. An annotation (`attachment.annotated`) records what a human read off
a file and changes no amount, no file and no posting; its fields are
a closed list. At least one field or the note is present.
5. `entry.settles` names the entries a payment pays back, every named
entry existing, none itself, none twice. "Reimbursed" is derived from
it alone: where no settlement is recorded, Needs attention asks; the
books never guess (IZ-18).
6. Every refusal leaves the journal unchanged; NEVER panics.
SCENARIOS:
1. A text PDF filed: one extraction record, its text a blob, `ok`.
2. The same PDF filed again: one attachment, linked to the second
subject, `deduplicated: true`.
3. Two legacy records with one blob, the oldest withdrawn, the youngest
an entry's only receipt: consolidation is refused whole.
4. The same, both live: the oldest is linked to the other's subjects,
the other is unlinked and withdrawn with `duplicate_of`.
5. A receipt annotated 7 on two different files: the next engine number
is 8 (numbers run from the max; a human's duplicate is not renumbered).
6. An amount 20.00 paid by a member, reimbursed: the reimbursement's
`entry.settles` names the expense; no Needs-attention item remains.
7. The same with no settlement: Needs attention asks, once.
```
**Enforced:** `ledger_import::extract` (`pdf_text`, `pdf_lines`,
`page_png`, `xlsx_text`, `text_of`, `extract`; caps 25 MiB, 30 pages,
2 MiB of text; a reader panic is caught), `ledger_core::evidence::{Annotation,
AnnotationFields}`, `Ledger::{annotate_attachment, next_receipt_number,
entry_settles, settles_entry, set_entry_settles}`,
`Store::{extract_attachment, annotate_attachment, set_entry_settles,
withdraw_attachment_duplicate}` and `Store::attach`'s dedup; feature
`attachment_text`.
**Checked:** `ledger-import` `tests/extract.rs`
(`extraction_deterministic`, `xlsx_cells_are_decimal_strings`,
`a_pdf_that_crashes_the_reader_is_failed_not_fatal`); `ledger-store`
`tests/attachments.rs` (`extraction_is_journaled_once_per_version`,
`extract_waits_for_attachment_text`, `same_bytes_one_record_many_links`,
`legacy_duplicates_replay_as_written`,
`annotations_replay_and_project`, `receipt_numbers_run_from_the_max`,
`reimbursed_is_derived_from_settles_only`); `ledger-cli`
`tests/evidence.rs` (`get_receipt_default_has_text_and_no_bytes`,
`page_image_is_a_png_of_the_page`,
`consolidation_never_removes_an_entrys_only_evidence`); fuzz targets
`fuzz_extract_pdf` and `fuzz_extract_xlsx`.
+108
View File
@@ -3268,3 +3268,111 @@ the blob-before-journal order, the `File`/`Existing` link steps) and the
feature gate; the MCP handler's amount default, the exactly-one evidence
check, and the preview's clone; attention's year comparison and its
`stands_on` filter; the bookspec loader's `may_refuse` arm.
## P6: the evidence pipeline (2026-10-27)
Extraction, dedup, annotations and company record categories (plan P6;
rules.md B-44). One in-process extractor in `ledger-import`
(`src/extract.rs`): `pdf_text` and `pdf_lines` (promoted from the PDF
reader test), `page_count`, `page_png` (promoted from the viewer's private
rasterizer), `xlsx_text` (cells as the exact decimal strings the sheet XML
holds), `text_of` (moved from `tin_files`, now reading a PDF's text too)
and `extract`, which reports `ok`, `none` or `failed` under the caps (25
MiB in, 30 pages, 2 MiB of text). A reader panic is caught there, never
reaching the server (D-32: no sandbox, and the plan says so). The viewer
and the server's filing scan both read through it: `ledger-gui/src/receipts.rs`
keeps no `hayro` use of its own.
- **ledger-import:** `tests/extract.rs` `extraction_deterministic` (same
bytes, same text and PNG bytes; plain text and an image's `none`),
`xlsx_cells_are_decimal_strings` (proptest: the cell text is the decimal
string written, never a float), `a_pdf_that_crashes_the_reader_is_failed_not_fatal`
(garbage and truncated PDFs are `Err`, the too-large cap is refused
before the reader, page 31 and a zero-size page are refused); unit tests
in `src/extract.rs` for the panic catch and the input cap. `tests/spec_import.rs`
keeps its script/RTL PDF test; its `pdf_lines` helper is now the
extractor's.
- **ledger-core:** `src/evidence.rs` (`Annotation`, `AnnotationFields`;
`checked`), `Ledger::{annotate_attachment, next_receipt_number,
entry_settles, settles_entry, set_entry_settles}`, and `differs_from`
gains both. No kernel or Lean change: no money moves.
- **ledger-store:** feature `attachment_text` (format gate;
`attachment_text_is_its_own_feature`), the three event types
(`format.rs`, `meta.rs` readers, replay and projection arms), schema v16
(`extractions`, `annotations`, `attachments.duplicate_of`). `attach`
returns the live record for bytes already stored (linked to the new
subject), `extract_attachment`, `annotate_attachment`,
`set_entry_settles`, `withdraw_attachment_duplicate`,
`attachments_holding_text` and `attachments_with_receipt_number`;
`verify` checks each extraction's text blob hash and never re-derives it;
`stored_blob_shas` includes it. `tests/attachments.rs`:
`extraction_is_journaled_once_per_version`,
`extract_waits_for_attachment_text` (R-A: refused before the upgrade,
journaling nothing; accepted after), `same_bytes_one_record_many_links`
(proptest, eight cases: one record, one extraction, a link per filing),
`legacy_duplicates_replay_as_written`, `annotations_replay_and_project`,
`receipt_numbers_run_from_the_max`,
`reimbursed_is_derived_from_settles_only`. The `p6_evidence` fixture
(`tests/fixtures/old-books/`) was written by the version before the unit
and opens to the same figures.
- **ledger-mcp:** `tools/evidence.rs` (`books.find_attachment`,
`books.annotate_attachment`, `books.set_entry_settles`,
`books.consolidate_duplicates`), `books.get_receipt`'s `text` and
`page_image` views (meta carries the first 8 KiB and the page count),
`books.file_document`'s four new categories, the new store and snapshot
methods, and the resource reads `books://attachments/{id}` and
`books://attachments/{id}/page/{n}` (a PNG blob).
- **ledger-package / ledger-gui:** Needs attention gains "extraction
failed" and "member-paid entry with no settlement recorded" (R-F) over
`PackageBooks::extractions`; the viewer's Documents page groups by
category and filters by subject; `Books` carries the extractions.
- **ledger-cli:** `tests/evidence.rs` `get_receipt_default_has_text_and_no_bytes`
(the default is under 8 KiB, has the text and no file; the resource
matches), `page_image_is_a_png_of_the_page` (IHDR-checked, page 2
refused, `bytes` unchanged, the page resource a PNG blob),
`annotations_and_searches` (the closed fields, the engine number, the
three searches, a settlement, exactly-one-selector),
`consolidation_never_removes_an_entrys_only_evidence` (a hand-written
legacy pair whose kept record is withdrawn: `receipt_required`, journal
unchanged) and `consolidation_links_the_kept_file_to_every_subject`
(preview journals nothing; the kept file takes the subject; the other is
withdrawn with `duplicate_of`; verify ok),
`every_p2_served_attachment_has_a_small_default_answer` (all 64 of the
fixture's attachments under 8 KiB), and
`the_backfill_reads_an_old_fixtures_files_once` (the `p6_evidence`
fixture upgraded, read through the admin command and then through
`wanderbooks books extract`, a second run journaling nothing). The
cutover test `books_p2_served_cut_over_whole` gained the annotation
call: refused before consent, accepted after.
- **fuzz:** `fuzz_extract_pdf` and `fuzz_extract_xlsx` in
`fuzz/fuzz_targets/`, in the scheduled workflow's matrix.
**Done when, checked:** (1) the two fixture tests above; (2) no `hayro`
use in `ledger-gui` or `ledger-mcp` (the extract crate is the only one);
(3) the dedup property and the only-evidence refusal pass; (4) the
backfill command test; (5) the PR carries the 8.0.6 checklist; (6)
`just check` green and both fuzz targets run ten minutes clean.
**For P12 (mutation targets, none run, D-33):** `ledger-import::extract`:
the caps' comparisons (`MAX_BYTES`, `MAX_PAGES`, `MAX_TEXT_BYTES`, the
`cap_text` boundary walk), `extract`'s order (PDF by type or magic, XLSX
by type or extension, then plain text) and `found`'s blank-text arm;
`pdf_lines`' page `take`, the baseline rounding and the left-to-right
sort; `page_png`'s scale floor and its page bounds; `xlsx_text`'s shared
string index, the `t="s"`/`inlineStr`/`b` arms and the sheet-name filter.
`ledger-core::evidence`: `checked`'s trims, the zero number, the negative
amount and the empty-annotation refusal; `next_receipt_number`'s
`saturating_add`. `ledger-store`: `attach`'s live-record lookup and link
arms, `extract_and_journal`'s text/blob branch and status mapping,
`extraction_for`'s triple (attachment, extractor, version),
`withdraw_attachment_as`'s `duplicate_of` arm and feature check, the
`verify` text-blob loop, and the projection arms' conflict keys.
`ledger-mcp::evidence`: `find_attachment`'s exactly-one selector, the
hex check and the text length floor; `consolidate_duplicates`' grouping
(`< 2`), the keep/dups split, the evidence `left == 0` gate, the keep
`withdrawn` test, the unlink dedup and the `withdrawn` count; the
annotation amount/date parse and the next number; `get_receipt`'s view
arms, the 8 KiB cut, the 64 KiB hand-over threshold and `page - 1`.
`ledger-package::attention`: the extraction `seq` comparison and the
`Failed` filter, the owed-account name test, the credit side and the
`settles_entry` skip.
+14
View File
@@ -876,3 +876,17 @@ each harness's whole run, its compile of about 50 s included):
| `bounds_kept_is_the_definition_on_books_in_bounds` | 34 | prove | (as above) |
| `settle_is_exact` | 36 | prove | 1 |
| `kani_symbolic_copy_canary` | (the checker: "Trusting the checker") | prove | 3 |
## P6: the evidence pipeline
No kernel arithmetic, `Ledger` command in the formal sense, or Lean step
changed (P6 moves no money): `just prove` is unchanged and stays green.
The new `Ledger` methods (`annotate_attachment`, `set_entry_settles`) are
replay-side records with validation only, covered by
`annotations_replay_and_project` and `reimbursed_is_derived_from_settles_only`;
the arithmetic-shaped work (the extraction caps and the page scale) is in
`ledger-import`, outside the kernel, and is tested and fuzzed rather than
proved. The formal bar for the next unit that touches these paths is
unchanged; the P12 paragraph in `docs/testing.md` lists the mutation
targets.