merge: P6, the evidence pipeline (branch P6, checkpoint ce366408)
The last of the paused branches, from its "not for merge" checkpoint:
extraction in ledger-import (text, pages, page images; capped, caught),
journaled as attachment.text_extracted; dedup by hash with
books.find_attachment and books.consolidate_duplicates; annotations and
books.set_entry_settles; the company record categories; the two fuzz
targets; feature attachment_text. Nineteen files conflicted with the
four P4b merges, resolved as plan 8.0.2 says: P6 retakes schema 18 (it
had taken 16, which S4.3 holds; S5 holds 17); the tool table is 129 and
the destructive count 31; both sides' appends kept with their seams
restored; one B-27 paragraph naming every feature; in the old-books
fixture P6's hand-written reimbursement is entry 7 (its ids collided
with S4.3's invoice); the replay fuzzer's list gains S7's
invoice.written_off; S4.3's issued-PDF test expects the extraction
record P6 journals right after a filing.
One integration defect between S7 and P6, found by the arap property
test and fixed here: under P6's dedup, a write-off whose evidence bytes
were filed already (the same collection letter for a second write-off of
the same invoice) was refused only after its entry had been journaled,
at the final link to the invoice ("already linked"). The write-off now
resolves the dedup before anything is journaled and keeps a link the
record already has to the invoice instead of refusing it; an explicit
existing_attachment_id already on an entry is still refused (S7's
one-receipt-once rule). P6's gate never ran before the pause; it runs on
main after this merge, and its ledger-store -> ledger-import dependency
is the open review point (plan 8.0.1).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HS8WZziDnyxunC23MUxrLp
This commit is contained in:
commit
2ad192bbcd
70 files changed
+4618
-384
No files matched your search
@@ -115,6 +115,24 @@ the agent's cannot drift apart. The crate depends inward (`ledger-core`,
|
||||
`ledger-store`, `ledger-config`, `ledger-import`, `ledger-vault`), never on
|
||||
`ledger-mcp` or `ledger-gui`.
|
||||
|
||||
## The evidence pipeline (`ledger-import`, `ledger-core`, `ledger-store`, P6)
|
||||
|
||||
One extractor lives in `ledger-import` (`src/extract.rs`, D-53: in-process,
|
||||
no new crate): a PDF's text and pages, an XLSX's cells as exact decimal
|
||||
strings, and plain text; the caps are 25 MiB in, 30 pages and 2 MiB of
|
||||
text, and a reader panic is caught there. `ledger-store` depends on
|
||||
`ledger-import` so the writer can read a file as it is filed and journal
|
||||
`attachment.text_extracted` (content-addressed text blob, extractor,
|
||||
version, pages, `ok`/`none`/`failed`), one record per attachment, extractor
|
||||
and version; `Store::verify` checks the text blob's hash and never
|
||||
re-derives it. The server's filing scan (`tin_files`) and the viewer's PDF
|
||||
rasterizer read through the same module, so a PDF's text is read one way.
|
||||
Annotations and settlements are ledger state (`ledger_core::evidence`,
|
||||
`Ledger::differs_from`), projected into schema 16's `annotations` table;
|
||||
extractions have a table of their own. The viewer's Documents page groups
|
||||
by category and filters by subject; Needs attention reads both through
|
||||
`PackageBooks`.
|
||||
|
||||
## The journal format gate (`ledger-store`, P0)
|
||||
|
||||
Every journal line passes a gate after its parse, seq and chain checks
|
||||
@@ -187,6 +205,8 @@ groups are:
|
||||
- `write_offs` — `books.write_off_invoice`, the bad debt (P4b S7; rules.md
|
||||
B-41), kept apart from `documents` so S4.4's document tools and this one
|
||||
land in different files;
|
||||
- `evidence` — finding a file by its hash, number or text, annotating it,
|
||||
recording what a payment settles, and consolidating duplicates (P6);
|
||||
- `settings` — the handbook, `books.settings` and the recorded FX rates
|
||||
(the module is `settings_tools`, since `crate::settings` is the live
|
||||
settings);
|
||||
|
||||
+80
-15
@@ -365,7 +365,8 @@ that changes anything also has a `destructiveHint`, true for these 30:
|
||||
`books.set_tin` (both overwrite a vault record), `books.purge_tin`,
|
||||
`books.create_invoice`, `books.create_bill`, `books.receive_payment`,
|
||||
`books.pay_bill`, `books.void_document`, `books.write_off_invoice`,
|
||||
`books.run_payroll`, `books.remit_payroll_taxes`; and an
|
||||
`books.run_payroll`, `books.remit_payroll_taxes`,
|
||||
`books.consolidate_duplicates`; and an
|
||||
`idempotentHint` on the ten that a repeat with the same arguments leaves
|
||||
the journal and the vault as they were: `books.verify`,
|
||||
`books.run_depreciation`, `books.run_schedules`, `books.run_recurring`,
|
||||
@@ -1598,22 +1599,82 @@ and no control or invisible (Unicode format) characters; a MIME type
|
||||
subject it is linked to, its filing entry first). Withdrawn attachments
|
||||
are omitted unless `include_withdrawn` is true. `entry_id` lists the
|
||||
files filed on the entry and linked to it.
|
||||
- `books.get_receipt { attachment_id, view?: "meta" | "bytes" }` → `{
|
||||
attachment_id, sha256, entry_id?, category, file_name, mime_type,
|
||||
size_bytes, description?, links, file? }`. `meta` (the default) returns
|
||||
no file. `bytes` adds `file`, the file handed over (a `HandedFile`, read
|
||||
hash-verified, whatever its size: D-46), whose `download` saves a copy
|
||||
as a new file in a folder the human names. The stored file is also
|
||||
fetched by `wanderbooks download --book B <id> --to <a folder>` (`GET
|
||||
- `books.get_receipt { attachment_id, view?: "meta" | "text" |
|
||||
"page_image" | "bytes", page?, max_px? }` → `{ attachment_id, sha256,
|
||||
entry_id?, category, file_name, mime_type, size_bytes, description?,
|
||||
links, text?, text_truncated, text_status, pages, receipt_number?,
|
||||
annotation?, file? }`. `meta` (the default) returns no file: the first
|
||||
8 KiB of the file's extracted text (`text`, `text_truncated` when cut),
|
||||
its page count (`pages`), and `text_status` (`ok`, `none`, `failed`, or
|
||||
`not_extracted` for books filed before this version: `wanderbooks books
|
||||
extract`). `text` gives the whole extracted text (past 64 KiB it comes
|
||||
as a handed-over `.txt` instead). `page_image` with `page` (1 is the
|
||||
first) and `max_px` (default 1600, longest side) gives one page as a
|
||||
PNG, handed over (D-46; D-15: no OCR, so a picture-only receipt is read
|
||||
from its pages). Read the text first; a page picture only when the text
|
||||
is empty or looks wrong; never `bytes` into context. `bytes` adds
|
||||
`file`, the file handed over (a `HandedFile`, read hash-verified,
|
||||
whatever its size: D-46), whose `download` saves a copy as a new file
|
||||
in a folder the human names. The stored file is also fetched by
|
||||
`wanderbooks download --book B <id> --to <a folder>` (`GET
|
||||
/b/{book}/a/{attachment}`, books:read, "Transports").
|
||||
- `books.file_document { category: "formation" | "legal" | "tax" |
|
||||
"accounting" | "statement" | "contract" | "photo" | "warranty" |
|
||||
"import_source" | "other", file_name?, mime_type?, upload_id,
|
||||
description?, link_to? }` → `{ attachment_id,
|
||||
sha256, deduplicated, links }`. The bytes, name and type as in
|
||||
`books.attach_receipt`. The Drive-folder replacement (today's 01
|
||||
Formation / 02 Legal / 03 Tax / 04 Accounting). Never a W-9: it shows
|
||||
the full TIN (`books.set_w9 w9_upload_id`).
|
||||
"import_source" | "notices" | "correspondence" | "plans" | "history" |
|
||||
"other", file_name?, mime_type?, upload_id, description?, link_to? }` →
|
||||
`{ attachment_id, sha256, deduplicated, links }`. The bytes, name and
|
||||
type as in `books.attach_receipt`. Every company record has a home
|
||||
(D-20): the Drive folders' replacement, and formation papers,
|
||||
registrations, plans, notices, correspondence and history besides what
|
||||
the CPA asks for. Never a W-9: it shows the full TIN (`books.set_w9
|
||||
w9_upload_id`).
|
||||
- **One file, one record** (plan P6 step 4): filing bytes already stored
|
||||
(and not withdrawn) returns that attachment's `attachment_id` with
|
||||
`deduplicated: true` and links it to the new subject, instead of filing
|
||||
a second record. Records an older version filed twice stay as written
|
||||
until `books.consolidate_duplicates`.
|
||||
|
||||
## Tools — evidence
|
||||
|
||||
What the books know about a filed file beyond its bytes (plan P6): its
|
||||
extracted text, the facts a human read off it, and which entries a payment
|
||||
settles. Extraction is journaled as it is made (`attachment.text_extracted`,
|
||||
content-addressed, one record per attachment, extractor and version): an
|
||||
extractor upgrade adds a record, never replaces one, and `books.verify`
|
||||
checks the text blob's presence and hash.
|
||||
|
||||
- `books.find_attachment { sha256 | receipt_number | text }` → `{
|
||||
attachments: [...] }`. Exactly one of the three: the file's exact
|
||||
SHA-256 (64 hex digits), the number recorded with
|
||||
`books.annotate_attachment`, or words its extracted text says
|
||||
(case-insensitively). Withdrawn files are listed too, marked, so a
|
||||
duplicate is found before it is consolidated. Each attachment is as in
|
||||
`books.list_receipts`, with its links.
|
||||
- `books.annotate_attachment { attachment_id, fields { receipt_number?,
|
||||
payee?, amount?, date?, instrument?, category?, source?,
|
||||
reimbursable_to? }, note? }` → `{ attachment_id, fields,
|
||||
next_receipt_number }`. The closed list of what a human reads off a
|
||||
file: `amount` an exact decimal string, `date` YYYY-MM-DD, `instrument`
|
||||
a payment method's id or label. It changes no amount, no file and no
|
||||
posting; a later annotation replaces it, journaled as
|
||||
`attachment.annotated`. `next_receipt_number` is one past the highest
|
||||
number annotated, so numbers run without reuse. At least one field or
|
||||
the note is required; an unknown or withdrawn attachment is refused.
|
||||
- `books.set_entry_settles { entry_id, settles }` → `{ ok, entry_id,
|
||||
settles }`. Records which entries an entry pays back (a reimbursement
|
||||
settling the expenses a member paid personally), journaled as
|
||||
`entry.settles`; an empty list clears it. Every named entry must exist,
|
||||
none may be the paying entry itself or be named twice. It changes no
|
||||
amount. "The member was paid back" is read from this record only, never
|
||||
guessed (IZ-18); Needs attention asks until it is recorded.
|
||||
- `books.consolidate_duplicates { confirmed? }` → `{ status: "preview" |
|
||||
"consolidated", groups: [{ sha256, keep, withdraw, links }], withdrawn
|
||||
}`. For every set of attachments holding the same bytes it keeps the
|
||||
oldest (`keep`), links it to every subject the others had, unlinks the
|
||||
others and withdraws them as duplicates (`attachment.withdrawn` with
|
||||
`duplicate_of`; nothing is deleted). A posted entry that would be left
|
||||
with no receipt the policy requires refuses the whole call, journaling
|
||||
nothing. Without `confirmed`, a preview that changes nothing.
|
||||
|
||||
## Receipt enforcement
|
||||
|
||||
@@ -1868,8 +1929,12 @@ Hand-rolled on `ServerHandler` (rmcp 3.x has no resource macros):
|
||||
this month so far
|
||||
- `books://handbook` (the index) and `books://handbook/{section}` (one per
|
||||
section, Markdown)
|
||||
- `books://receipts/{id}` — receipt metadata (as `books.get_receipt`
|
||||
- `books://receipts/{id}` (an alias of `books://attachments/{id}`) —
|
||||
receipt metadata and the first 8 KiB of its text (as `books.get_receipt`
|
||||
view `meta`; no bytes)
|
||||
- `books://attachments/{id}/page/{n}` — page `n` (from 1) as a PNG blob,
|
||||
the first binary resource (plan P6; D-46: the bytes are the resource's
|
||||
contents, never a path)
|
||||
|
||||
Resources exist so read-only clients (dashboards, the viewer,
|
||||
auditors) never need tool credentials. Tested end to end through a real
|
||||
|
||||
@@ -173,3 +173,20 @@ stay derived, and its evidence of worthlessness filed in the same call
|
||||
-3(a)). A same-year recovery is the reversal D-49 records, not a new
|
||||
movement; bills are never written off. No row above changed status (the
|
||||
A/R and A/P rows already say `done`), so the floor stands (10 done).
|
||||
|
||||
## P6: the evidence pipeline (2026-10-26–27)
|
||||
|
||||
No semantic feature of the catalog above moved, so no status changed and
|
||||
the floor stands (10 done): P6 changes how evidence is kept, not how a
|
||||
figure is computed. It adds one in-process extractor (a PDF's text and
|
||||
pages, an XLSX's cells as exact decimal strings, plain text; 25 MiB, 30
|
||||
pages, 2 MiB of text), journals each extraction as its own record (one per
|
||||
attachment, extractor and version, its text a content-addressed blob whose
|
||||
hash `books.verify` checks), holds identical bytes as one live attachment
|
||||
(linked to every subject, legacy duplicates consolidated with
|
||||
`duplicate_of`), records what a human read off a file (the closed
|
||||
annotation field list, engine receipt numbers running from the max) and
|
||||
what a payment settles (`entry.settles`, the only source of "reimbursed"),
|
||||
and gives company records four more homes (`notices`, `correspondence`,
|
||||
`plans`, `history`, D-20). Rules.md B-44 states it; `docs/testing.md`'s P6
|
||||
section lists its tests and the P12 mutation targets.
|
||||
+36
-15
@@ -57,13 +57,11 @@ apply is safe — on startup the applier replays every journal line with
|
||||
- `seq`: 1-based, gapless, assigned by the writer.
|
||||
- `ts`: UTC, RFC 3339. Recorded, never trusted for ordering (`seq` orders).
|
||||
- `type`: `account.added`, `entry.posted`, `entry.reversed`, `invoice.created`,
|
||||
`bill.created`, `receipt.attached`, `receipt.linked`, `receipt.unlinked`,
|
||||
`document.filed`, `attachment.withdrawn`, `invoice.sent` (a recorded
|
||||
send: a document's record, never a ledger line or a figure; P4b S5)
|
||||
(plus the A/R, A/P, asset, period, payroll, and class events listed on
|
||||
`JournalLine`).
|
||||
`invoice.written_off`, `bill.created`, `receipt.attached`, `receipt.linked`, `receipt.unlinked`,
|
||||
`document.filed`, `attachment.withdrawn` (plus the A/R, A/P, asset,
|
||||
`bill.created`, `invoice.sent` (a recorded send: a document's record,
|
||||
never a ledger line or a figure; P4b S5), `invoice.written_off` (P4b S7),
|
||||
`receipt.attached`, `receipt.linked`, `receipt.unlinked`,
|
||||
`document.filed`, `attachment.withdrawn`, `attachment.text_extracted`,
|
||||
`attachment.annotated`, `entry.settles` (P6) (plus the A/R, A/P, asset,
|
||||
period, payroll, and class events listed on `JournalLine`).
|
||||
- `payload`: the full domain facts (account fields, entry with lines, blob
|
||||
sha, ...). Everything the applier needs; nothing it must look up elsewhere.
|
||||
@@ -114,9 +112,12 @@ a backup is then proven where it lands, by a drill (docs/mcp-interface.md,
|
||||
- **Immutable.** Blobs are never modified or deleted by the application. A
|
||||
retention/deletion policy is an explicit future human decision, not an
|
||||
accident of implementation.
|
||||
- **Deduplicated.** Attaching the same receipt twice stores one blob and two
|
||||
attachment rows. `attach_receipt` / `file_document` return
|
||||
`deduplicated: true` when the blob already existed.
|
||||
- **Deduplicated.** The same bytes are one blob, and since P6 one live
|
||||
attachment record: filing them again returns that record (with
|
||||
`deduplicated: true`) and links it to the new subject instead of adding
|
||||
a second row. Records an older version filed twice stay as written until
|
||||
`books.consolidate_duplicates` links the oldest to every subject the
|
||||
others had and withdraws the rest (`duplicate_of`).
|
||||
- **Limits.** 25 MiB per file, as an upload (`books.begin_upload`): the
|
||||
only way a file comes in (D-46: nothing is given inline). The name and MIME type are the client's, checked
|
||||
when the file is filed: a name of 1 to 255 bytes, no `/` or `\`, not
|
||||
@@ -135,17 +136,33 @@ a backup is then proven where it lands, by a drill (docs/mcp-interface.md,
|
||||
exists, then linked with `books.link_receipt`.
|
||||
- `category` mirrors today's Drive layout: `formation`, `legal`, `tax`,
|
||||
`accounting` (today's 01 Formation / 02 Legal / 03 Tax / 04 Accounting),
|
||||
plus `receipts`.
|
||||
plus `receipts`, and every other company record (D-20): `statement`,
|
||||
`contract`, `photo`, `warranty`, `import_source`, `notices`,
|
||||
`correspondence`, `plans`, `history`, `other`.
|
||||
- `duplicate_of` names the attachment a withdrawn duplicate stands for
|
||||
(`attachment.withdrawn`'s optional `duplicate_of`, schema v16).
|
||||
- Documents filed via `books.file_document` and receipts via
|
||||
`books.attach_receipt` share the blob store; they differ only in metadata.
|
||||
- **Receipts change by events, never by edits.** `receipt.unlinked`
|
||||
(`attachment_id`, `entry_id`, `reason`) detaches a receipt from its entry;
|
||||
`attachment.withdrawn` (`attachment_id`, `reason`) retires an unlinked
|
||||
receipt or a document (`withdrawn_at`, `withdrawn_reason` columns, schema
|
||||
v5). Linking requires an unlinked, non-withdrawn attachment, so evidence
|
||||
`attachment.withdrawn` (`attachment_id`, `reason`, `duplicate_of?`)
|
||||
retires an unlinked receipt or a document (`withdrawn_at`,
|
||||
`withdrawn_reason`, `duplicate_of` columns, schema v5 and v16). Linking requires an unlinked, non-withdrawn attachment, so evidence
|
||||
only leaves an entry through a recorded unlink. Blobs are untouched: a
|
||||
withdrawn file is still hash-verified and readable, and the journal shows
|
||||
when it was filed, linked, unlinked, and withdrawn, and why.
|
||||
- **Extraction is derived, and journaled** (P6, rules.md B-44):
|
||||
`attachment.text_extracted {attachment_id, text_sha256, extractor,
|
||||
version, pages, status}` records what reading a file found (`ok`,
|
||||
`none`, `failed`); the text itself is a content-addressed blob like any
|
||||
other, and `books.verify` checks its presence and hash, never re-derives
|
||||
it. One record per attachment, extractor and version: an extractor
|
||||
upgrade adds a record, never replaces one.
|
||||
- **Annotations and settlements are records, not edits** (P6):
|
||||
`attachment.annotated {attachment_id, fields, note?}` is what a human
|
||||
read off a file (the closed field list; a later annotation replaces the
|
||||
earlier record), and `entry.settles {entry_id, settles[]}` says which
|
||||
entries a payment pays back (the only source of "reimbursed").
|
||||
- The store is policy-free: it records attachments faithfully. Whether a
|
||||
posting *requires* a receipt is decided by the MCP layer's receipt policy
|
||||
(`docs/mcp-interface.md`), which reads the `attachments` table.
|
||||
@@ -168,7 +185,11 @@ Suspected corruption is fixed by rebuild, never by forensics.
|
||||
`integrity_check` on startup.
|
||||
- **Schema.** `meta(key, value)` — `schema_version`, `base_currency`,
|
||||
`last_applied_seq`; `accounts`, `entries`, `lines`, `blobs`, `attachments`
|
||||
as above; `events(seq, type, payload, created_at)` — the applier's output,
|
||||
as above; `extractions(attachment_id, extractor, version, text_sha256,
|
||||
pages, status, seq)` and `annotations(attachment_id, receipt_number,
|
||||
payee, amount, date, instrument, category, source, reimbursable_to,
|
||||
note, seq)` (P6; amounts TEXT, never REAL); `events(seq, type, payload,
|
||||
created_at)` — the applier's output,
|
||||
a queryable mirror of the journal stream and the feed the future GUI
|
||||
tails. (Poll `WHERE seq > ?`; later SSE over the HTTP transport. This is
|
||||
the correct answer to "watch its files to update live" — watch the event
|
||||
|
||||
+81
-1
@@ -1753,7 +1753,11 @@ that moves no figure); S6's payments with `line`, `discount`,
|
||||
`invoice.written_off`. An older version would
|
||||
restore the document without them silently: every invoice line would post
|
||||
to the revenue role and every bill line to general expenses, and every new
|
||||
event type would be refused. A command asks for
|
||||
event type would be refused. P6's is `attachment_text` (an
|
||||
`attachment.text_extracted`, `attachment.annotated` or `entry.settles`, a
|
||||
new event type, and an `attachment.withdrawn` carrying `duplicate_of`; an
|
||||
older version would refuse the types, or withdraw a duplicate without
|
||||
saying what it duplicates, B-44). A command asks for
|
||||
the feature it needs (`Store::ensure_feature`) before its first journaled
|
||||
line, and only when the line it is about to write uses it, so a write
|
||||
refused for waiting on the upgrade journals nothing, never an entry
|
||||
@@ -3316,3 +3320,79 @@ check_discount}`, `arap::{Application, Fee, SettleParams, SettlementRow}`,
|
||||
`ledger-cli` `tests/documents.rs`; `ledger-package`'s aging tie-out test.
|
||||
The Lean step is `Documents.Step.document` with every control line tagged
|
||||
(`Documents.settle_unattached`); no new `Step` constructor.
|
||||
|
||||
## B-44 Evidence is read from the bytes, never trusted over them; identical files are kept once
|
||||
|
||||
```text
|
||||
GOAL: a receipt's text and the facts read off it are derived records the
|
||||
books can re-make; the file's bytes stay the authority; the same
|
||||
bytes are one file wherever they are evidence; and a payment says
|
||||
what it settles instead of the books guessing.
|
||||
SOURCE: Rev. Proc. 98-25 §5.01 (machine-sensible records with an audit
|
||||
trail: the relationship between the records and the books must be
|
||||
demonstrable); IRS Pub. 583 (supporting documents are kept with
|
||||
the entries they support); plan/served.md P6 (defects 4 and 10);
|
||||
IZ-18 (no FIFO guess);
|
||||
CONVENTION (the extractor's version is part of its record; a
|
||||
failed read is recorded, not hidden; a duplicate is withdrawn, not
|
||||
deleted).
|
||||
INVARIANTS:
|
||||
1. An attachment's text is a record (`attachment.text_extracted`) with
|
||||
the text's content-addressed blob, the extractor and its version, the
|
||||
page count and `ok`, `none` or `failed`. One record per attachment,
|
||||
extractor and version: an upgrade adds a record and NEVER replaces
|
||||
one, and the bytes are never re-derived by `verify` (its hash is
|
||||
checked).
|
||||
2. Extraction is derived and journaled, never authoritative: a report,
|
||||
a posting or an audit reads the file's bytes when the text matters,
|
||||
and a failed extraction is a record that a human must read the
|
||||
pages (D-15: no OCR).
|
||||
3. Identical bytes are one file: filing bytes already stored (and not
|
||||
withdrawn) returns the existing attachment, linked to the new
|
||||
subject. Consolidating legacy duplicates keeps the oldest, links it
|
||||
to every subject the others had, and withdraws the others with
|
||||
`duplicate_of`; nothing is deleted and no posted entry the receipt
|
||||
policy covers is left without evidence.
|
||||
4. An annotation (`attachment.annotated`) records what a human read off
|
||||
a file and changes no amount, no file and no posting; its fields are
|
||||
a closed list. At least one field or the note is present.
|
||||
5. `entry.settles` names the entries a payment pays back, every named
|
||||
entry existing, none itself, none twice. "Reimbursed" is derived from
|
||||
it alone: where no settlement is recorded, Needs attention asks; the
|
||||
books never guess (IZ-18).
|
||||
6. Every refusal leaves the journal unchanged; NEVER panics.
|
||||
SCENARIOS:
|
||||
1. A text PDF filed: one extraction record, its text a blob, `ok`.
|
||||
2. The same PDF filed again: one attachment, linked to the second
|
||||
subject, `deduplicated: true`.
|
||||
3. Two legacy records with one blob, the oldest withdrawn, the youngest
|
||||
an entry's only receipt: consolidation is refused whole.
|
||||
4. The same, both live: the oldest is linked to the other's subjects,
|
||||
the other is unlinked and withdrawn with `duplicate_of`.
|
||||
5. A receipt annotated 7 on two different files: the next engine number
|
||||
is 8 (numbers run from the max; a human's duplicate is not renumbered).
|
||||
6. An amount 20.00 paid by a member, reimbursed: the reimbursement's
|
||||
`entry.settles` names the expense; no Needs-attention item remains.
|
||||
7. The same with no settlement: Needs attention asks, once.
|
||||
```
|
||||
|
||||
**Enforced:** `ledger_import::extract` (`pdf_text`, `pdf_lines`,
|
||||
`page_png`, `xlsx_text`, `text_of`, `extract`; caps 25 MiB, 30 pages,
|
||||
2 MiB of text; a reader panic is caught), `ledger_core::evidence::{Annotation,
|
||||
AnnotationFields}`, `Ledger::{annotate_attachment, next_receipt_number,
|
||||
entry_settles, settles_entry, set_entry_settles}`,
|
||||
`Store::{extract_attachment, annotate_attachment, set_entry_settles,
|
||||
withdraw_attachment_duplicate}` and `Store::attach`'s dedup; feature
|
||||
`attachment_text`.
|
||||
**Checked:** `ledger-import` `tests/extract.rs`
|
||||
(`extraction_deterministic`, `xlsx_cells_are_decimal_strings`,
|
||||
`a_pdf_that_crashes_the_reader_is_failed_not_fatal`); `ledger-store`
|
||||
`tests/attachments.rs` (`extraction_is_journaled_once_per_version`,
|
||||
`extract_waits_for_attachment_text`, `same_bytes_one_record_many_links`,
|
||||
`legacy_duplicates_replay_as_written`,
|
||||
`annotations_replay_and_project`, `receipt_numbers_run_from_the_max`,
|
||||
`reimbursed_is_derived_from_settles_only`); `ledger-cli`
|
||||
`tests/evidence.rs` (`get_receipt_default_has_text_and_no_bytes`,
|
||||
`page_image_is_a_png_of_the_page`,
|
||||
`consolidation_never_removes_an_entrys_only_evidence`); fuzz targets
|
||||
`fuzz_extract_pdf` and `fuzz_extract_xlsx`.
|
||||
+108
@@ -3268,3 +3268,111 @@ the blob-before-journal order, the `File`/`Existing` link steps) and the
|
||||
feature gate; the MCP handler's amount default, the exactly-one evidence
|
||||
check, and the preview's clone; attention's year comparison and its
|
||||
`stands_on` filter; the bookspec loader's `may_refuse` arm.
|
||||
|
||||
## P6: the evidence pipeline (2026-10-27)
|
||||
|
||||
Extraction, dedup, annotations and company record categories (plan P6;
|
||||
rules.md B-44). One in-process extractor in `ledger-import`
|
||||
(`src/extract.rs`): `pdf_text` and `pdf_lines` (promoted from the PDF
|
||||
reader test), `page_count`, `page_png` (promoted from the viewer's private
|
||||
rasterizer), `xlsx_text` (cells as the exact decimal strings the sheet XML
|
||||
holds), `text_of` (moved from `tin_files`, now reading a PDF's text too)
|
||||
and `extract`, which reports `ok`, `none` or `failed` under the caps (25
|
||||
MiB in, 30 pages, 2 MiB of text). A reader panic is caught there, never
|
||||
reaching the server (D-32: no sandbox, and the plan says so). The viewer
|
||||
and the server's filing scan both read through it: `ledger-gui/src/receipts.rs`
|
||||
keeps no `hayro` use of its own.
|
||||
|
||||
- **ledger-import:** `tests/extract.rs` `extraction_deterministic` (same
|
||||
bytes, same text and PNG bytes; plain text and an image's `none`),
|
||||
`xlsx_cells_are_decimal_strings` (proptest: the cell text is the decimal
|
||||
string written, never a float), `a_pdf_that_crashes_the_reader_is_failed_not_fatal`
|
||||
(garbage and truncated PDFs are `Err`, the too-large cap is refused
|
||||
before the reader, page 31 and a zero-size page are refused); unit tests
|
||||
in `src/extract.rs` for the panic catch and the input cap. `tests/spec_import.rs`
|
||||
keeps its script/RTL PDF test; its `pdf_lines` helper is now the
|
||||
extractor's.
|
||||
- **ledger-core:** `src/evidence.rs` (`Annotation`, `AnnotationFields`;
|
||||
`checked`), `Ledger::{annotate_attachment, next_receipt_number,
|
||||
entry_settles, settles_entry, set_entry_settles}`, and `differs_from`
|
||||
gains both. No kernel or Lean change: no money moves.
|
||||
- **ledger-store:** feature `attachment_text` (format gate;
|
||||
`attachment_text_is_its_own_feature`), the three event types
|
||||
(`format.rs`, `meta.rs` readers, replay and projection arms), schema v16
|
||||
(`extractions`, `annotations`, `attachments.duplicate_of`). `attach`
|
||||
returns the live record for bytes already stored (linked to the new
|
||||
subject), `extract_attachment`, `annotate_attachment`,
|
||||
`set_entry_settles`, `withdraw_attachment_duplicate`,
|
||||
`attachments_holding_text` and `attachments_with_receipt_number`;
|
||||
`verify` checks each extraction's text blob hash and never re-derives it;
|
||||
`stored_blob_shas` includes it. `tests/attachments.rs`:
|
||||
`extraction_is_journaled_once_per_version`,
|
||||
`extract_waits_for_attachment_text` (R-A: refused before the upgrade,
|
||||
journaling nothing; accepted after), `same_bytes_one_record_many_links`
|
||||
(proptest, eight cases: one record, one extraction, a link per filing),
|
||||
`legacy_duplicates_replay_as_written`, `annotations_replay_and_project`,
|
||||
`receipt_numbers_run_from_the_max`,
|
||||
`reimbursed_is_derived_from_settles_only`. The `p6_evidence` fixture
|
||||
(`tests/fixtures/old-books/`) was written by the version before the unit
|
||||
and opens to the same figures.
|
||||
- **ledger-mcp:** `tools/evidence.rs` (`books.find_attachment`,
|
||||
`books.annotate_attachment`, `books.set_entry_settles`,
|
||||
`books.consolidate_duplicates`), `books.get_receipt`'s `text` and
|
||||
`page_image` views (meta carries the first 8 KiB and the page count),
|
||||
`books.file_document`'s four new categories, the new store and snapshot
|
||||
methods, and the resource reads `books://attachments/{id}` and
|
||||
`books://attachments/{id}/page/{n}` (a PNG blob).
|
||||
- **ledger-package / ledger-gui:** Needs attention gains "extraction
|
||||
failed" and "member-paid entry with no settlement recorded" (R-F) over
|
||||
`PackageBooks::extractions`; the viewer's Documents page groups by
|
||||
category and filters by subject; `Books` carries the extractions.
|
||||
- **ledger-cli:** `tests/evidence.rs` `get_receipt_default_has_text_and_no_bytes`
|
||||
(the default is under 8 KiB, has the text and no file; the resource
|
||||
matches), `page_image_is_a_png_of_the_page` (IHDR-checked, page 2
|
||||
refused, `bytes` unchanged, the page resource a PNG blob),
|
||||
`annotations_and_searches` (the closed fields, the engine number, the
|
||||
three searches, a settlement, exactly-one-selector),
|
||||
`consolidation_never_removes_an_entrys_only_evidence` (a hand-written
|
||||
legacy pair whose kept record is withdrawn: `receipt_required`, journal
|
||||
unchanged) and `consolidation_links_the_kept_file_to_every_subject`
|
||||
(preview journals nothing; the kept file takes the subject; the other is
|
||||
withdrawn with `duplicate_of`; verify ok),
|
||||
`every_p2_served_attachment_has_a_small_default_answer` (all 64 of the
|
||||
fixture's attachments under 8 KiB), and
|
||||
`the_backfill_reads_an_old_fixtures_files_once` (the `p6_evidence`
|
||||
fixture upgraded, read through the admin command and then through
|
||||
`wanderbooks books extract`, a second run journaling nothing). The
|
||||
cutover test `books_p2_served_cut_over_whole` gained the annotation
|
||||
call: refused before consent, accepted after.
|
||||
- **fuzz:** `fuzz_extract_pdf` and `fuzz_extract_xlsx` in
|
||||
`fuzz/fuzz_targets/`, in the scheduled workflow's matrix.
|
||||
|
||||
**Done when, checked:** (1) the two fixture tests above; (2) no `hayro`
|
||||
use in `ledger-gui` or `ledger-mcp` (the extract crate is the only one);
|
||||
(3) the dedup property and the only-evidence refusal pass; (4) the
|
||||
backfill command test; (5) the PR carries the 8.0.6 checklist; (6)
|
||||
`just check` green and both fuzz targets run ten minutes clean.
|
||||
|
||||
**For P12 (mutation targets, none run, D-33):** `ledger-import::extract`:
|
||||
the caps' comparisons (`MAX_BYTES`, `MAX_PAGES`, `MAX_TEXT_BYTES`, the
|
||||
`cap_text` boundary walk), `extract`'s order (PDF by type or magic, XLSX
|
||||
by type or extension, then plain text) and `found`'s blank-text arm;
|
||||
`pdf_lines`' page `take`, the baseline rounding and the left-to-right
|
||||
sort; `page_png`'s scale floor and its page bounds; `xlsx_text`'s shared
|
||||
string index, the `t="s"`/`inlineStr`/`b` arms and the sheet-name filter.
|
||||
`ledger-core::evidence`: `checked`'s trims, the zero number, the negative
|
||||
amount and the empty-annotation refusal; `next_receipt_number`'s
|
||||
`saturating_add`. `ledger-store`: `attach`'s live-record lookup and link
|
||||
arms, `extract_and_journal`'s text/blob branch and status mapping,
|
||||
`extraction_for`'s triple (attachment, extractor, version),
|
||||
`withdraw_attachment_as`'s `duplicate_of` arm and feature check, the
|
||||
`verify` text-blob loop, and the projection arms' conflict keys.
|
||||
`ledger-mcp::evidence`: `find_attachment`'s exactly-one selector, the
|
||||
hex check and the text length floor; `consolidate_duplicates`' grouping
|
||||
(`< 2`), the keep/dups split, the evidence `left == 0` gate, the keep
|
||||
`withdrawn` test, the unlink dedup and the `withdrawn` count; the
|
||||
annotation amount/date parse and the next number; `get_receipt`'s view
|
||||
arms, the 8 KiB cut, the 64 KiB hand-over threshold and `page - 1`.
|
||||
`ledger-package::attention`: the extraction `seq` comparison and the
|
||||
`Failed` filter, the owed-account name test, the credit side and the
|
||||
`settles_entry` skip.
|
||||
@@ -876,3 +876,17 @@ each harness's whole run, its compile of about 50 s included):
|
||||
| `bounds_kept_is_the_definition_on_books_in_bounds` | 34 | prove | (as above) |
|
||||
| `settle_is_exact` | 36 | prove | 1 |
|
||||
| `kani_symbolic_copy_canary` | (the checker: "Trusting the checker") | prove | 3 |
|
||||
|
||||
|
||||
## P6: the evidence pipeline
|
||||
|
||||
No kernel arithmetic, `Ledger` command in the formal sense, or Lean step
|
||||
changed (P6 moves no money): `just prove` is unchanged and stays green.
|
||||
The new `Ledger` methods (`annotate_attachment`, `set_entry_settles`) are
|
||||
replay-side records with validation only, covered by
|
||||
`annotations_replay_and_project` and `reimbursed_is_derived_from_settles_only`;
|
||||
the arithmetic-shaped work (the extraction caps and the page scale) is in
|
||||
`ledger-import`, outside the kernel, and is tested and fuzzed rather than
|
||||
proved. The formal bar for the next unit that touches these paths is
|
||||
unchanged; the P12 paragraph in `docs/testing.md` lists the mutation
|
||||
targets.
|
||||
Reference in new issue
Block a user