swaitsandClaude Opus 5.5
beb1fcf233
Max-testing pass, steps 2-4. Spec-blind tests (rank 1-2): four agents, each given only the specs and the public interface (doc comments and signatures, every body removed, never src/ or the existing tests), wrote tests/spec_reports.rs (62), spec_ledger.rs (72), spec_arap.rs (42), spec_assets_payroll.rs (50): hand-computed examples per scenario and a proptest law per invariant (conservation, partition, invariance, scaling, reversal cancels, "a refusal leaves the books unchanged" over random invalid commands). They found: 1. A refused payroll run changed the books: an all-zero run passed validation, then failed at post after creating seven payroll accounts and using up an entry id. validate_payroll_lines now refuses a run with no gross pay. 2. A refused add_account used up an account id (empty code, duplicate, unknown parent). The id is now taken only when the account is added. 3. Stale prose: close_period_to and members_equity still described the old last-member rounding. 4. Open (left #[ignore]d pending Stephen): depreciation_schedule's doc says "through the month containing" the date; the code prorates that month by days. GnuCash oracle (rank 3, an oracle never the spec): `just oracle` builds GnuCash 5.10 in a container pinned by base digest and package versions (oracle/gnucash/) and runs tests/oracle_gnucash.rs: 200 seeded random books built through wanderbooks (entries, foreign lines, reversals, monthly closes), every account's balance at month ends and random dates and each month's P&L, compared exactly: 15,800 comparisons. First run: 4,724 differences, all from foreign lines; without them, zero. Adjudicated from principles (rules.md B-3): wanderbooks booked foreign * rate unrounded ($108.34567); a ledger in one money holds no fractions of its smallest unit, and its cash must match a bank. Stephen decided: book in the base currency's minor unit, half away from zero. add_line_fx now rounds; minor units per ISO 4217 list one (2026-09-17, SIX), generated into Currency::minor_units and recorded in docs/spec/sources/iso-4217.md. Replay now takes the journaled base amount (replay_add_line_fx), so lines booked before replay exactly as booked (contract: books_written_under_older_rules_still_open). Now: zero differences, foreign lines included. Tests: 449, from 223. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GWVo7x3UHw5kwCUaWxiejX
6.6 KiB
The semantic oracle: how we prove the books are right
storytold's clones are verified with a pixel oracle: re-render a PSD and diff it against Photoshop's own embedded merged composite, passing at ≤ 2/255 per channel. We steal the shape of that idea and change the domain: our oracle is semantic, not visual.
The core loop
- Fixture books. A canonical set of journal entries exercising a feature
(the January 2026 fixture company in
tests/invariants.rsis the first). - More than one engine. The same fixtures go through our engine and through independent implementations: GnuCash (open source, open file formats), the old Google Sheet, and later QBO/Xero exports. These are cross-checks, not authorities (see "Who is right" below).
- Exact diff.
reports::snapshot()renders trial balance, P&L, and balance sheet as canonical JSON. The outputs must be identical — decimal strings, no floats, no locale, no epsilon. Where storytold allows 2/255 of perceptual slop, we allow zero: accounting has no "close enough".
Who is right
No other program is the spec. GnuCash, the Sheet, QuickBooks: each is an oracle, and each can be wrong. The order of authority:
- Accounting principles, stated as proofs. The laws in
docs/verification.md, proved in Lean and model-checked by Kani. They do not depend on any other implementation's behavior. - Golden files recomputed by hand from those principles (Layer 1).
- External oracles (GnuCash, the Sheet, QBO/Xero exports). A
disagreement with one is a finding: work out from principles which
side is wrong and write the reasoning down. Never change our output just
to match theirs. If theirs is wrong, add a test that pins our correct
behavior, and note their bug in
docs/parity.md.
Nothing in this repo may rest on an external oracle's behavior alone.
The oracle layers (today → future)
Layer 1 — Golden files (live now). tests/golden/*.json hold
hand-verified expected reports for canonical fixtures. The golden test fails
on any behavior change, intentional or not. Intentional changes require
recomputing the goldens by hand from accounting principles and documenting
the recomputation in the commit. This is the seed of everything below.
Layer 2 — Property tests (live now, growing).
crates/ledger-core/tests/properties.rs uses proptest (with shrinking)
to assert the accounting laws over hundreds of random inputs: trial balance
always balances, the accounting equation holds, reversals restore zero,
posted entries are immutable and JSON-round-trip clean, and depreciation
schedules are exact to the penny. This layer caught a real bug on its second
case: partial first/final months in different-length calendar months don't
complement, so the lifetime total drifted by cents (fixed with the
final-month true-up in depreciation_schedule).
Layer 3 — Mutation testing (live now). cargo mutants -p ledger-core
(228 mutants as of 2026-10-10: 159 caught, 28 unviable, 2 timeouts from
induced infinite loops, 39 missed). Every missed mutant is triaged:
real gaps get new tests (this pass added parent-account validation,
wrong-typed asset links, zero-proceeds disposal, two-asset ids, 1-month
life, Money arithmetic, is_balance_sheet, movements()), equivalent
mutants are documented as such (e.g. days_held > 0 vs >=: the zero
case is unreachable), and untestable-by-construction mutants are noted
(e.g. TrialBalance::balanced -> true: the engine cannot produce an
unbalanced ledger to falsify it).
Layer 4 — Fuzzing (live now). cargo fuzz targets in fuzz/
(nightly, libfuzzer): fuzz_journal_line (journal parsing never panics;
canonical form is a fixed point), fuzz_entry_validate (a validated entry
always balances exactly), fuzz_op_sequence (arbitrary op sequences via
arbitrary; the trial balance always balances). 11.5M execs, zero crashes
as of 2026-10-10. Corpora persist in fuzz/corpus/ for regression.
Layer 4½ — Formal proofs (live now). The accounting arithmetic lives
in one generic kernel (kernel.rs). Kani model-checks it exhaustively
within bounds, Lean 4 proves the laws for books of any size (trial
balance, accounting equation, reversals, posted-is-forever, frozen closed
periods, closing to net income), and a property test bridges the two.
just prove; see docs/verification.md. Its first pass found three
engine bugs that Layers 1–4 had all missed.
Layer 5 — Cross-implementation diff harness (live: GnuCash).
just oracle builds GnuCash 5.10 in a pinned container
(oracle/gnucash/) and runs tests/oracle_gnucash.rs: 200 random books
built through wanderbooks (entries, foreign lines, reversals, closes),
every account's balance at month ends and random dates, and each month's
P&L, compared exactly (15,800 comparisons). Its first run found one real
difference, adjudicated from principles: wanderbooks kept sub-cent
amounts on foreign lines; it now books in cents (rules.md B-3). The
original plan follows. A
ledger-oracle crate will run fixture books through our engine and through
other implementations (GnuCash headless first), then diff snapshot()
output automatically: cargo xtask oracle. This is the direct analog of
storytold's cargo xtask corpus / oracle_diff, with one difference: a
diff does not mean we are wrong. Each one is adjudicated from principles
(see "Who is right"). Note also: GnuCash is a behavioral cross-check, not
a file-format target
(Stephen: "for file format import/export I could not give a single shit
about GnuCash XML — QBO is all that matters"). The harness feeds fixtures to
GnuCash via whatever it imports reliably (QIF/OFX) or its Python bindings;
the exact mechanism is decided at implementation time. The proofs and
Layers 1–4 are the foundation and are not optional. The harness adds
evidence. It never overrides them.
What the oracle does NOT cover (yet)
- Workflows, only semantics: that an invoice posts the right lines is oracle-covered; that the invoice screen is usable is a UI concern for the egui phase, where storytold's screenshot-in-the-loop comes back.
- Tax law. The engine computes what you tell it; it does not know the tax code. Tax-adjacent logic gets its own fixtures and its own goldens when it arrives, sourced from IRS publications — never from memory.
Clean-room boundary
GnuCash is GPL. We read its documentation, its file formats, and its behavior. We never copy its code, and no GPL code enters this repo (MIT OR Apache-2.0). If an agent is ever unsure whether something is "learning from behavior" vs "copying implementation", it stops and asks.