ci: mutation testing is local only (D-59); the weekly run keeps proofs, oracle and fuzz
The owner, 2026-10-29: mutation testing is for local runs, not CI. The weekly workflow's mutation shards (core, store, import, config, MCP) are removed; they failed every week on the survivors and timeouts D-33 deferred to P12, and their red hid the run's real findings (a fuzz_store_replay crash and a fuzz_settings_replay build break, both being fixed). just mutants-* stays for the desktop; P12 runs the pass whole. Recorded as D-59; the docs that said the shards keep running are corrected. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HS8WZziDnyxunC23MUxrLp
This commit is contained in:
1 parent
79fb7ff2fd
commit
fc7ba81f12
5 files changed
+11
-86
No files matched your search
@@ -1,8 +1,7 @@
|
||||
# Weekly, and on demand: the slow layers (docs/testing.md).
|
||||
#
|
||||
# Mutation testing is split into shards (`just mutants-* k/n`), each its
|
||||
# own job, so every job ends well inside GitHub's six hours. The viewer has
|
||||
# no mutation job (owner, 2026-10-18).
|
||||
# Weekly, and on demand: the slow layers (docs/testing.md): the proofs,
|
||||
# the GnuCash oracle and the fuzz targets. Mutation testing is not run in
|
||||
# CI (owner, 2026-10-29, D-59): it is a local run (`just mutants-*`), and
|
||||
# P12 runs it whole on the owner's desktop.
|
||||
name: scheduled
|
||||
on:
|
||||
schedule:
|
||||
@@ -39,79 +38,3 @@ jobs:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: jdx/mise-action@v2
|
||||
- run: just fuzz ${{ matrix.target }} 600
|
||||
mutants:
|
||||
runs-on: ubuntu-24.04
|
||||
timeout-minutes: 350
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
shard: [0, 1, 2, 3, 4, 5, 6, 7]
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: jdx/mise-action@v2
|
||||
- run: just mutants ${{ matrix.shard }}/8
|
||||
- if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: mutants-core-${{ matrix.shard }}
|
||||
path: mutants.out
|
||||
mutants-store:
|
||||
runs-on: ubuntu-24.04
|
||||
timeout-minutes: 350
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
shard: [0, 1, 2, 3]
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: jdx/mise-action@v2
|
||||
- run: just mutants-store ${{ matrix.shard }}/4
|
||||
- if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: mutants-store-${{ matrix.shard }}
|
||||
path: mutants-store/mutants.out
|
||||
mutants-import:
|
||||
runs-on: ubuntu-24.04
|
||||
timeout-minutes: 350
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
shard: [0, 1]
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: jdx/mise-action@v2
|
||||
- run: just mutants-import ${{ matrix.shard }}/2
|
||||
- if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: mutants-import-${{ matrix.shard }}
|
||||
path: mutants-import/mutants.out
|
||||
mutants-config:
|
||||
runs-on: ubuntu-24.04
|
||||
timeout-minutes: 350
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: jdx/mise-action@v2
|
||||
- run: just mutants-config
|
||||
- if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: mutants-config
|
||||
path: mutants-config/mutants.out
|
||||
mutants-mcp:
|
||||
runs-on: ubuntu-24.04
|
||||
timeout-minutes: 350
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
shard: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: jdx/mise-action@v2
|
||||
- run: just mutants-mcp ${{ matrix.shard }}/16
|
||||
- if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: mutants-mcp-${{ matrix.shard }}
|
||||
path: mutants-mcp/mutants.out
|
||||
@@ -51,9 +51,9 @@ oracles, never ship what they haven't demonstrated.
|
||||
pass found three engine bugs that every test, fuzzer, and mutant had
|
||||
missed: negative amounts accepted, and two crashes on oversized
|
||||
amounts. It also caught a soundness bug in the model checker itself.
|
||||
- Mutation testing via cargo-mutants (paused for development until the
|
||||
P12 max-testing pass, D-33; CI's weekly shards still run); fuzzing via
|
||||
cargo-fuzz (11.5M+ executions, zero crashes).
|
||||
- Mutation testing via cargo-mutants, a local run (paused for development
|
||||
until the P12 max-testing pass, D-33; not a CI job, D-59); fuzzing via
|
||||
cargo-fuzz (11.5M+ executions, zero crashes), weekly in CI.
|
||||
- GnuCash is *a* behavioral cross-check, never the authority and never
|
||||
a code source (clean room, MIT OR Apache-2.0). Accounting principles,
|
||||
proved, are the spec. Where any other program disagrees, we work out
|
||||
|
||||
+1
-1
@@ -53,7 +53,7 @@ case: partial first/final months in different-length calendar months don't
|
||||
complement, so the lifetime total drifted by cents (fixed with the
|
||||
final-month true-up in `depreciation_schedule`).
|
||||
|
||||
**Layer 3 — Mutation testing (paused for development until P12, D-33, 2026-10-20; CI's weekly shards still run).** `cargo mutants -p ledger-core`
|
||||
**Layer 3 — Mutation testing (a local run, never a CI job, D-59; paused for development until P12, D-33).** `cargo mutants -p ledger-core`
|
||||
(228 mutants as of 2026-10-10: 159 caught, 28 unviable, 2 timeouts from
|
||||
induced infinite loops, 39 missed). Every *missed* mutant is triaged:
|
||||
real gaps get new tests (this pass added parent-account validation,
|
||||
|
||||
+2
-1
@@ -37,7 +37,8 @@ promise.
|
||||
| Mutation (GUI) | can the tests fail? | `ledger-gui` view model and formatting (2026-10-14: 140 caught of 176 viable on the full run, then 92 of 99 on a targeted rerun of the money paths after new tests; of its 7 survivors, one then killed by a test (burn over exactly one month, confirmed by a rerun of `compute_kpis`'s 48 mutants), two dead arms removed (`parties`' bill arms: the ledger tags bill entries with their vendor, which wins), four equivalent: an entry's debits vs its credits (they balance), the three-month burn boundary (both paths give 3), burn over zero covered months (only when the first entry is after the window, which then holds no loss), and `compact`'s sign test at zero (zero prints "$0" unsigned either way)) | retired 2026-10-18 (owner: it crushed the machine; for the viewer, test its two musts directly: it never writes the books, `tests/read_only.rs`, and every figure it shows is the engine's, `tests/view_model.rs`) | | never |
|
||||
|
||||
CI: `.github/workflows/check.yml` runs `just check` on every push;
|
||||
`scheduled.yml` runs proofs, the oracle, fuzzing, and mutation weekly. Mutation is split into shards, one job each (`just mutants-mcp 3/16` runs the fourth of sixteen), because GitHub stops a job at six hours and a whole crate takes longer on its runners (2026-10-19, the first scheduled run: the core and MCP runs were stopped at the limit). Test timeouts are three times each run's own baseline, never under the recipe's floor, so a slower runner doesn't report false timeouts. The viewer has no mutation job. Development mutation runs are paused until the P12 max-testing pass (D-33, 2026-10-20): each unit records its targets in a **For P12** paragraph instead; the weekly shards keep running in CI.
|
||||
`two-user.yml` the self-host deployment's two-account check (P9a);
|
||||
`scheduled.yml` runs the proofs, the oracle and the fuzz targets weekly. Mutation ran there in shards until 2026-10-29 (`just mutants-mcp 3/16` runs the fourth of sixteen; GitHub stops a job at six hours), and is now local only (D-59). Test timeouts are three times each run's own baseline, never under the recipe's floor, so a slower runner doesn't report false timeouts. The viewer has no mutation job. Development mutation runs are paused until the P12 max-testing pass (D-33, 2026-10-20): each unit records its targets in a **For P12** paragraph instead. Mutation testing is a local run, not a CI job (the owner, 2026-10-29, D-59): the weekly shards were removed from `scheduled.yml`, and the "weekly" in the mutation rows above is history (the 2026-10-26 run: real survivors and timeouts in every crate, the debt P12 clears).
|
||||
|
||||
Seeds and failure databases: proptest's `*.proptest-regressions` files
|
||||
are checked in and replay on every run. The GnuCash oracle's books come
|
||||
|
||||
@@ -2055,6 +2055,7 @@ The owner decisions of 2026-10-16 (at the top of this document) replace the form
|
||||
| D-55 | Party contact details (owner, 2026-10-27: asked which fields, chose "Full contact block") | A party carries, all optional and journaled on `party.updated` behind `documents_v2`: `phone`, `contact` (a named person), `billing_address`, `shipping_address`, `billing_email`. The existing `email` stays the general address and the existing `address` the general/business one; when `billing_email` or `billing_address` is set, an invoice's bill-to uses it, else the general one. `PartySnapshot` carries the same at issue; the issued PDF prints the contact person, phone, billing address and billing email, and prints a Ship-to line when a shipping address is set. `books.update_party` and `books.create_invoice`'s `bill_to` override take all six; `""` clears a text field. Both emails go through `parties::check_email` (one `@`, no spaces, the full-TIN scan). S4.4 builds the core, store, journal and MCP parts; S4.5 the PDF. Folded into `documents_v2` because the owner's upgrade has not happened (D-52), so no second go-ahead. No projection columns: there is no parties table in the SQLite projection, and replay plus `differs_from` carry the facts. Terms per side are already D-52/S4.2's and are unchanged. |
|
||||
| D-56 | The destination, and when P11a starts (owner, 2026-10-28: the work is "ultimately taking it to a single bundled server+GUI installation for Mac/Windows/Linux and then beyond") | **The installed app is this plan's destination** (D-19, D-22 and D-24 to D-27 unchanged), and the hosted service of section 11 is what comes after it. So P11a is no longer "any time": it starts as soon as an engineer is free, beside the P4b finish, with its new step 0 first (the macOS and Windows keyring stores, the Windows data root and pipes, a macOS and Windows CI build), because today the executable's agent path works on Linux alone. The P10 chain (P4, P5, P8) keeps its order and runs beside it. |
|
||||
| D-57 | Q-5 answered (owner, 2026-10-28, the question tool: "Once, after everything") | **One format upgrade of the live books, after S7 and P4 have landed**, declaring `documents_v2` and `drafts_v2` together (D-52 confirmed). Until then the new writes are refused on the owner's books with the upgrade remedy; a few days without numbered invoices is the accepted cost. A-14 records it. |
|
||||
| D-59 | Mutation testing in CI (owner, 2026-10-29, asked how the always-red weekly mutation shards should run until P12: "Don't think we need to run the long ass shit like mutation testing in CI. It's more for local test runs.") | **Mutation testing is a local run, never a CI job.** The mutation shards are removed from `scheduled.yml`, which keeps the proofs, the GnuCash oracle and the fuzz targets weekly; `just mutants-*` stays for the desktop, and P12 runs the whole pass there (D-33). The 2026-10-26 weekly run's survivors (real ones in every crate, and timeouts) are P12's list. |
|
||||
| D-58 | Q-8 answered (owner, 2026-10-28, the question tool: "Yes, all stand as built") | **D-47 to D-54 are the owner's decisions**, no longer the review's recommendations: reversals stay posted (D-47), the vault stays v1 (D-48), the P4b deferrals (D-49), the preview digest with no key (D-50), the P9 split (D-51), one P4b upgrade (D-52), no jobs and no new crates (D-53), the P11 split (D-54). The "the owner may revisit" clauses on those rows are closed. |
|
||||
|
||||
### 10.2 Deferred
|
||||
|
||||
Reference in new issue
Block a user