wanderbooks/docs/testing.md

Testing strategy

How wanderbooks knows its books are right. Strongest oracle first: a test's worth is where its idea of "correct" comes from, not what it runs. Formal proofs are in docs/verification.md; the authorities every rule comes from are in docs/spec/. Every crate is covered: ledger-core, the store, import, config, and MCP in full; the GUI where it computes and writes numbers (its view model and formatting) and in its read-only promise.

Layers

Layer	Oracle (rank)	Where	Command	Runtime	Gates
Spec-derived examples	written specs with sources (1)	tests/spec_*.rs, invariants.rs, statements.rs, assets.rs, ...; the store's tests/spec_store.rs (written blind, from its specs only)	just check	seconds	every change
Regressions	the bug, named (1)	beside the code they cover, *_regression/named for the defect	just check	seconds	every change
Compatibility contracts	journals written under older rules must open (1)	ledger-store/tests/store.rs (books_written_under_older_rules_still_open)	just check	<1s	every change
Properties	laws over all inputs (2)	properties.rs, spec_*.rs, kernel_bridge.rs	just check	seconds	every change
Metamorphic	scaling, reversal, ordering invariance (2)	spec_reports.rs	just check	seconds	every change
Refusal law (store)	a refused command never reaches the journal, an accepted one always does; the live ledger equals its journal's replay in every part, and the books reopen to it (2, 3)	store.rs refused_commands_never_reach_the_journal	just check (STORE_CASES=5000 deeper)	~20s	every change
Fault injection (store)	a full disk or a cut-off append leaves whole, openable books (1)	ledger-store/tests/faults.rs, store.rs (power_loss_at_any_byte_of_the_last_append_is_recovered)	just check	~5s	every change
Concurrency (store)	a viewer reading while the books are written sees only whole books, never a false "differs" (2)	ledger-store/tests/concurrency.rs	just check	<1s	every change
Model-based stateful	the Lean state machine, transcribed (3)	tests/model.rs	just check	~3s	every change
Differential: Decimal vs i64	the kernel at two types (3)	kernel_bridge.rs	just check	<1s	every change
Differential: GnuCash	another implementation, adjudicated (3)	tests/oracle_gnucash.rs, oracle/gnucash/	just oracle	~2 min	weekly
Formal proofs	Kani (bounded) and Lean (unbounded)	src/proofs.rs, verify/lean/	just prove	~3 min	weekly
Fuzzing	crash, hang, broken books (4); for fuzz_store_replay (validly chained journals with fuzz-chosen events): books that open must verify, and the viewer must accept exactly what the writer does	fuzz/fuzz_targets/, corpora in fuzz/corpus/	just fuzz <target> <secs>	10 min each	weekly
Miri	undefined behavior in our use of dependencies (4)	the core's unit, bridge, invariant tests	just miri	~2 min	on demand
Golden	tests/golden/fixture_company.json (5)	invariants.rs	just check	<1s	every change
Benchmarks	a baseline (6)	ledger-core/benches/books.rs, ledger-store/benches/store.rs	just bench	~2 min	none (no budget, by decision)
Mutation	can the tests fail?	ledger-core (2026-10-13: 1,011 caught + 13 timeouts of 1,024 viable, 0 missed)	just mutants	~3 h	weekly
Mutation (store)	can the tests fail?	ledger-store (2026-10-13: 357 caught + 1 timeout of 365 viable, from 82%; three of its seven survivors then killed; four equivalent left: sort_value's two arms, OpenFlags `	vs^`, the long form of a sweepless close)	just mutants-store	~12 min
Mutation (import)	can the tests fail?	ledger-import (2026-10-13: from 91 of 171 (53%) to 208 of 239 viable on the full run; the 31 survivors (XLSX layout, borders, alignment, exact-text grouping, IIF transliteration, tax-summary optional sections, day-equals-month dates) then killed by targeted tests, re-run clean but one: a blank amount cell written unbordered on an unruled row, invisible, so equivalent)	just mutants-import	~1 h	weekly
Mutation (roles, reload)	can the tests fail?	control roles and settings reload (2026-10-15, targeted: the role paths in ledger-core 30 caught of 30 viable; ledger-store's journaling and replay of role.assigned 20 of 20; ledger-mcp settings, reload, roles, digest 28 of 28)	cargo mutants -F on those functions (see the commit)	~15 min	with the crate's weekly run
Mutation (config)	can the tests fail?	ledger-config (2026-10-14: 55 caught of 55 viable)	just mutants-config	minutes	weekly
Mutation (MCP)	can the tests fail?	ledger-mcp (2026-10-14: from 197 caught + 5 timeouts of 328 viable (62%) to 314 caught + 2 timeouts of 331; of its 15 survivors, five then killed by targeted tests (the settings file beside the books, year earnings with an open period and with last year's partial close, year 1, a failed flush), one dead guard removed (profit_and_loss_by_class skipping placeholders, which never take postings), nine equivalent: money rescaling an amount already at two places, the lock wait's deadline instant, year_earnings starting the day the year starts either way, the bank import's rehearsal sides (a line's side can't make a draft refuse) and its zero-amount side (zero rows are dropped by the parser), a disposal's zero gain adding zero, and the CPA package index summing debits instead of credits (entries balance))	just mutants-mcp	~1 h	weekly
Mutation (GUI)	can the tests fail?	ledger-gui view model and formatting (2026-10-14: 140 caught of 176 viable on the full run, then 92 of 99 on a targeted rerun of the money paths after new tests; of its 7 survivors, one then killed by a test (burn over exactly one month, confirmed by a rerun of compute_kpis's 48 mutants), two dead arms removed (parties' bill arms: the ledger tags bill entries with their vendor, which wins), four equivalent: an entry's debits vs its credits (they balance), the three-month burn boundary (both paths give 3), burn over zero covered months (only when the first entry is after the window, which then holds no loss), and compact's sign test at zero (zero prints "$0" unsigned either way))	just mutants-gui	~2 h	weekly

CI: .github/workflows/check.yml runs just check on every push; scheduled.yml runs proofs, the oracle, fuzzing, and mutation weekly.

Seeds and failure databases: proptest's *.proptest-regressions files are checked in and replay on every run. The GnuCash oracle's books come from fixed seeds (SEED in oracle_gnucash.rs). Fuzz corpora live in fuzz/corpus/<target>.

Golden review: fixture_company.json is hand-verified (docs/oracle.md, Layer 1). The van depreciation golden was replaced on 2026-10-13 by a hand computation (van_schedule_is_the_hand_computed_one) after every row was checked against it.

Benchmarks (baseline 2026-10-13)

No budget, by decision (Stephen, 2026-10-13: the books do little work and are fast enough). The numbers are a baseline to spot a regression.

40-account chart, two-line entries across a year, on the development laptop:

	1k entries	10k entries
clone the ledger (store rehearsal, per command)	56 µs	1.0 ms
post one more entry	1.0 µs	5.8 µs
trial balance	44 µs	1.07 ms
balance sheet	47 µs	1.04 ms
one month's P&L	25 µs	248 µs

The store, 10k two-line entries on the laptop's disk (every journal append is synced; benches/store.rs in ledger-store):

	10k entries
open the books (verify the chain, replay, catch the database up)	249 ms
the viewer loading them	616 ms
a full verify (replay, projection, every table)	809 ms
post one entry (create, two lines, post: four synced appends)	62 ms

Formerly assumed: checked against the standards (2026-10-13)

Stephen isn't an accountant and won't adjudicate these; each was checked against published sources instead (rank: FASB ASC via the Deloitte DART reproduction, SEC, IRS; then textbooks; software practice is evidence of convention only). Verdicts: CONFIRMED (a source supports it), CONVENTION (no standard governs; matches mainstream practice), DESIGN (not an accounting question), CHANGE (a source says otherwise; work pending).

#	Behavior	Verdict	Source / note
1	A P&L whose start is after its end is refused	DESIGN	No standard governs a reversed range; refusing beats silently swapping a typo
2	A balance sheet omits zero-balance accounts	CONFIRMED	Reg. S-X 210.4-03(a): "No caption should be shown ... as to which the items and conditions are not present." A comparative statement must still show a zero beside a non-zero prior column
3	The lifetime profit_and_loss counts closing entries	CONFIRMED	Used as earnings not yet closed on the balance sheet; closed income is already in capital (OpenStax Principles of Financial Accounting 5.1). Renamed unclosed_earnings (2026-10-14)
4	Closing a period with nothing to close still locks it	CONVENTION	A close is a lock (B-12); a dormant month must still lock and count for close-in-order
5	Draws and distributions are closed into capital at period end	CONFIRMED	OpenStax 5.1: dividends "are closed to the permanent account"; drawing accounts close to owner's capital. Monthly closes mean the year's K-1 Item L withdrawals sum across months
6	Reconciliation balances	CONFIRMED	Signed by the account's normal side (a card reads as owed), as statements show it; reconciling_a_card_reads_its_normal_side_...
7	A member's ending on one equity statement may differ by a cent from the next statement's beginning	CHANGED 2026-10-13	A capital account is one running balance (Form 1065 K-1 instructions, Item L). Undistributed earnings now split cumulatively as of each date, so each member's capital chains exactly (each_members_capital_chains_across_statements; it fails on the old code); rules.md B-13
8	Cash flow nets one line per account; investing comes only from registered assets	CHANGED 2026-10-13	ASC 230-10-45-7: gross receipts and payments "more relevant" than net; 45-8 limits netting to quick-turnover items. Financing and investing are now gross; loans made and investments bought are investing (230-10-20, 45-13) via [books] investing_accounts. Owner-advanced expenses: PwC 6.9.3.1 constructive treatment via [books] financing_accounts (rules.md B-14)
9	An asset's life ends at purchase + life months, that day not held; day-count proration	CONVENTION	Exactly N months of service; day proration and the mid-month convention are both accepted book policies (ASC 360-10-35-4, read 2026-10-14: any systematic and rational allocation)
10	A payroll entry has one line per account total	CONFIRMED	The per-employee detail the IRS requires ("Amounts and dates of all wage ... payments", Employment Tax Recordkeeping) is kept in the run; a summary entry is standard
11	A period with exactly the dates of an existing one is allowed	DESIGN	Nothing governs duplicate periods; harmless (the second close locks only). Refusing exact duplicates is an option
12	Each close into members' capital rounds on its own (ties to the earlier member)	CHANGED 2026-10-13	Twelve monthly 50/50 closes could leave one member several cents ahead. A run of members closes now splits cumulatively (rules.md B-13; Lean runPart_telescopes; property each_member_stays_within_a_cent_of_their_share fails on the per-close rule); journaled "cumulative", older closes replay as booked

Questions are settled from the sources, not by any one company's CPA (wanderbooks is general software). Where the standards leave a real choice, it is a per-company setting with a sourced default, for that company's accountant to review: the cash flow account roles (rules.md B-14) are the first.

Gap list: what isn't covered, and what would bring it in

Technique	Status	Trigger
Fault injection	SKIP for the core (no I/O, clock, or network); DONE for the store's journal (full disk, cut-off append)	the first I/O or clock in ledger-core
Integration	SKIP for the core (no collaborators)	the core gains one; covered at the store and MCP layers
End-to-end	SKIP for the core (a library)	MCP stdio tests cover the journeys
Security properties	SKIP for the core (no roles, no secrets)	the first permission concept, or a secret-bearing field
Concurrency stress	SKIP for the core (no shared state); DONE for the store's writer + viewer (found the false "differs")	the first Arc, atomic, or lock in the core; the MCP server's HTTP transport shares one store across sessions (next)
Lean: apportion's sum law for any n	DONE (2026-10-14)	verify/lean/Wanderbooks/Apportion.lean: ranks are a permutation of 0..n-1 (strict total order + pigeonhole, core Lean only), so exactly u parts get a unit and one the fraction; apportion_sum and apportion_sum_of_shortfall, standard axioms only
ASC paragraphs not yet read	DONE (2026-10-14)	read in the ASC basic view (docs/spec/sources/fasb-asc.md): all confirm the rules; one citation fixed (230-10-45-14(b), not (2))
GnuCash's own close	OPEN	the oracle books closes as ordinary entries; comparing GnuCash's Close Book needs its Scheme API
Import: spec-blind suite, footing properties, mutation	DONE	sources vendored (Intuit bank CSV and IIF, ECMA-376/Excel numbers, IRS form lines); specs on every module; tests/spec_import.rs (spec-blind, footing and round-trip properties, XLSX read back from its XML, layout and styles); fuzz_import_csv; templates checked against the engine's control accounts; mutation one equivalent left
MCP, config, GUI crates	DONE (2026-10-14)	the same playbook: sourced module specs (config: Pub 538, Treas. Reg. 301.7701-3; MCP: the MCP tool spec and JSON-RPC 2.0); spec-blind suites spec_config.rs, spec_mcp.rs, spec_gui.rs; the MCP oracle oracle_mcp.rs (every export against ledger-core); startup, HTTP concurrency, and the viewer's read-only promise (read_only.rs) tested; fuzz_config_parse; mutation above