swaitsandClaude Opus 5.5
0dcf865ff8
check / check (push) Canceled after 0s
CI's run of the store refusal property found a payroll run that posted $65.27794: the employer tax was passed in unrounded and nothing refused it, though rules.md B-3 already says a ledger in one money holds no fractions of its smallest unit. Sums of it then differed in scale between journal replay and the SQLite read-back. - Ledger::post_entry refuses any line finer than its currency's ISO 4217 minor unit (FinerThanMinorUnit), whatever made the line: payroll, closes, disposals. add_line refuses it up front. Older journals replay as booked through replay_post_entry. Spec tests for the kernel's ten-place limit now run on the replay path. - The property's body is now run_ops, so failing sequences become named regression tests (regression_payroll_with_a_fraction_of_a_cent). The generator rounds payroll tax to the cent, as callers must. - Ledger derives PartialEq and gains differs_from; Store::verify compares the live ledger with the journal replay in full (drafts, documents, schedules, periods, settings), not just balances, and names the parts that differ. The property checks the ledger its commands built against that replay and against the reopened books. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GWVo7x3UHw5kwCUaWxiejX
99 lines
6.4 KiB
Markdown
99 lines
6.4 KiB
Markdown
# Testing strategy
|
|
|
|
How wanderbooks knows its books are right. Strongest oracle first: a
|
|
test's worth is where its idea of "correct" comes from, not what it runs.
|
|
Formal proofs are in `docs/verification.md`; the authorities every rule
|
|
comes from are in `docs/spec/`. Scope so far: `ledger-core` in full, and
|
|
the store's journal safety; the other crates come next (gap list).
|
|
|
|
## Layers
|
|
|
|
| Layer | Oracle (rank) | Where | Command | Runtime | Gates |
|
|
|---|---|---|---|---|---|
|
|
| Spec-derived examples | written specs with sources (1) | `tests/spec_*.rs`, `invariants.rs`, `statements.rs`, `assets.rs`, ... | `just check` | seconds | every change |
|
|
| Regressions | the bug, named (1) | beside the code they cover, `*_regression`/named for the defect | `just check` | seconds | every change |
|
|
| Compatibility contracts | journals written under older rules must open (1) | `ledger-store/tests/store.rs` (`books_written_under_older_rules_still_open`) | `just check` | <1s | every change |
|
|
| Properties | laws over all inputs (2) | `properties.rs`, `spec_*.rs`, `kernel_bridge.rs` | `just check` | seconds | every change |
|
|
| Metamorphic | scaling, reversal, ordering invariance (2) | `spec_reports.rs` | `just check` | seconds | every change |
|
|
| Refusal law (store) | a refused command never reaches the journal; the books reopen to the same reports (2, 3) | `store.rs` `refused_commands_never_reach_the_journal` | `just check` (`STORE_CASES=5000` deeper) | ~20s | every change |
|
|
| Fault injection (store) | a full disk or a cut-off append leaves whole, openable books (1) | `ledger-store/tests/faults.rs`, `store.rs` (`power_loss_at_any_byte_of_the_last_append_is_recovered`) | `just check` | ~5s | every change |
|
|
| Model-based stateful | the Lean state machine, transcribed (3) | `tests/model.rs` | `just check` | ~3s | every change |
|
|
| Differential: Decimal vs i64 | the kernel at two types (3) | `kernel_bridge.rs` | `just check` | <1s | every change |
|
|
| Differential: GnuCash | another implementation, adjudicated (3) | `tests/oracle_gnucash.rs`, `oracle/gnucash/` | `just oracle` | ~2 min | weekly |
|
|
| Formal proofs | Kani (bounded) and Lean (unbounded) | `src/proofs.rs`, `verify/lean/` | `just prove` | ~3 min | weekly |
|
|
| Fuzzing | crash, hang, broken books (4) | `fuzz/fuzz_targets/`, corpora in `fuzz/corpus/` | `just fuzz <target> <secs>` | 10 min each | weekly |
|
|
| Miri | undefined behavior in our use of dependencies (4) | the core's unit, bridge, invariant tests | `just miri` | ~2 min | on demand |
|
|
| Golden | `tests/golden/fixture_company.json` (5) | `invariants.rs` | `just check` | <1s | every change |
|
|
| Benchmarks | a baseline (6) | `benches/books.rs` | `just bench` | ~1 min | none (no budget, by decision) |
|
|
| Mutation | can the tests fail? | `ledger-core` | `just mutants` | ~12 min | weekly |
|
|
|
|
CI: `.github/workflows/check.yml` runs `just check` on every push;
|
|
`scheduled.yml` runs proofs, the oracle, fuzzing, and mutation weekly.
|
|
|
|
Seeds and failure databases: proptest's `*.proptest-regressions` files
|
|
are checked in and replay on every run. The GnuCash oracle's books come
|
|
from fixed seeds (`SEED` in `oracle_gnucash.rs`). Fuzz corpora live in
|
|
`fuzz/corpus/<target>`.
|
|
|
|
Golden review: `fixture_company.json` is hand-verified (docs/oracle.md,
|
|
Layer 1). The van depreciation golden was replaced on 2026-10-13 by a
|
|
hand computation (`van_schedule_is_the_hand_computed_one`) after every
|
|
row was checked against it.
|
|
|
|
## Benchmarks (baseline 2026-10-13)
|
|
|
|
No budget, by decision (Stephen, 2026-10-13: the books do little work
|
|
and are fast enough). The numbers are a baseline to spot a regression.
|
|
|
|
40-account chart, two-line entries across a year, on the development
|
|
laptop:
|
|
|
|
| | 1k entries | 10k entries |
|
|
|---|---|---|
|
|
| clone the ledger (store rehearsal, per command) | 56 µs | 1.0 ms |
|
|
| post one more entry | 1.0 µs | 5.8 µs |
|
|
| trial balance | 44 µs | 1.07 ms |
|
|
| balance sheet | 47 µs | 1.04 ms |
|
|
| one month's P&L | 25 µs | 248 µs |
|
|
|
|
## Assumed, not confirmed (reject any)
|
|
|
|
Each was inferred, not decided by a human; the tests that rely on them
|
|
say so in their doc comments.
|
|
|
|
- ASSUMED: a P&L for a period whose start is after its end is refused.
|
|
- ASSUMED: a balance sheet omits accounts with a zero balance.
|
|
- ASSUMED: the lifetime `profit_and_loss` counts closing entries
|
|
(only period statements exclude them).
|
|
- ASSUMED: closing a period with nothing to close still marks it closed.
|
|
- ASSUMED: a sweep account (draws, distributions) is zeroed as of the
|
|
period end, like income.
|
|
- ASSUMED: reconciliation balances are debits minus credits (tested on
|
|
cash; liability accounts unspecified).
|
|
- ASSUMED: a members' equity statement allocates prior unclosed earnings
|
|
and the period's income separately; one statement's ending for a member
|
|
may differ by a cent from the next statement's beginning (only totals
|
|
are guaranteed to chain).
|
|
- ASSUMED: cash flow lists one working-capital or financing line per
|
|
account that changed; with no registered assets, investing is zero.
|
|
- ASSUMED: an asset's life ends on purchase date + life months, that day
|
|
not held.
|
|
- ASSUMED: a payroll entry has one line per account total, not per
|
|
employee.
|
|
- ASSUMED: a period with exactly the dates of an existing one nests in it
|
|
(is allowed).
|
|
|
|
## Gap list: what isn't covered, and what would bring it in
|
|
|
|
| Technique | Status | Trigger |
|
|
|---|---|---|
|
|
| Fault injection | SKIP for the core (no I/O, clock, or network); DONE for the store's journal (full disk, cut-off append) | the first I/O or clock in `ledger-core` |
|
|
| Integration | SKIP for the core (no collaborators) | the core gains one; covered at the store and MCP layers |
|
|
| End-to-end | SKIP for the core (a library) | MCP stdio tests cover the journeys |
|
|
| Security properties | SKIP for the core (no roles, no secrets) | the first permission concept, or a secret-bearing field |
|
|
| Concurrency stress | SKIP for the core (no shared state) | the first `Arc`, atomic, or lock in the core; the GUI and MCP server have threads (next) |
|
|
| Lean: apportion's sum law for any n | OPEN | Kani covers n <= 3, properties n <= 5; needs a permutation lemma |
|
|
| ASC paragraphs not yet read | OPEN | ASC 360-10-35-4, 340-10, 230-10-45-10, 606-10-55-37B, 210: needs a FASB basic-view login |
|
|
| GnuCash's own close | OPEN | the oracle books closes as ordinary entries; comparing GnuCash's Close Book needs its Scheme API |
|
|
| Store: differential replay vs. projection, concurrency (writer + snapshot reader), benchmarks at 10k, mutation | NEXT | then import, MCP, config, GUI, the same playbook |
|