test(agent): scripted evals follow only next_steps; the surface over the wire (P5)
Agent reason: the envelope is only worth having if an agent can do its work by following it. crates/ledger-cli/tests/agent_evals/ holds six scripted transcripts, no model: onboarding to a first post, a receipt-backed post, a draft that waits for its receipt and posts when it arrives, a reversal, a reconciliation, a month-end close. agent_evals_follow_only_next_steps plays each against a real books server: after books.start a step may call only a tool (or fetch only a prompt) the previous result's next_steps named, with the arguments the step gave and the ones the agent decides; setup calls (the chart and the books an agent finds) are not the agent's moves. agent_surface.rs holds the over-the-wire tests of the rest: the journal in words (order, subject and type filters, paging, a bad subject refused), source corrections through the three tools, the prompts rendered against their files and refused when malformed, the resource templates advertised and read, books.start v2 whole and under 8 KiB with the self-report in the access log only, a new book told to onboard, and a mutating call answering with the head the journal has and a refusal naming its remedy. Docs: testing.md's P5 section (every test, and the For-P12 paragraph naming what a mutant would flip), parity.md (no feature moved; one row), and architecture.md (the envelope and the guards); mcp-interface.md's overview sections now describe the surface as built. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HS8WZziDnyxunC23MUxrLp
This commit is contained in:
1 parent
a319a64c3a
commit
7bac25496e
12 files changed
+1358
-5
No files matched your search
+18
-1
@@ -31,7 +31,7 @@ the specification in `bookspec/` (which never depends on the engine).
|
||||
| `ledger-core` | the semantics engine: money, chart, journal, `Ledger` commands, reports, periods, assets, schedules, parties, A/R and A/P, payroll, projects, settings and conventions, the kernel and its Kani harnesses | nothing |
|
||||
| `ledger-vault` | the TIN vault: XChaCha20-Poly1305 records under the book's key | nothing |
|
||||
| `ledger-config` | the `wanderbooks.toml` codec (adoption in, `books.export_settings` out) and the receipt policy until P4 moves it into the core | `ledger-core` |
|
||||
| `ledger-store` | the source of truth: the hash-chained JSONL journal, its format gate, the SQLite projection (schema 18; a file per schema), the blob store, backups, the access log, `Snapshot` for readers | `ledger-core`, `ledger-reader` (P6, D-61) |
|
||||
| `ledger-store` | the source of truth: the hash-chained JSONL journal, its format gate, the SQLite projection (schema 18; a file per schema), the blob store, backups, the access log, `describe.rs` (the journal in words, for `books.journal_history`), `Snapshot` for readers | `ledger-core`, `ledger-reader` (P6, D-61) |
|
||||
| `ledger-reader` | the one reader: a PDF's text and pages, an XLSX's cells as exact decimal strings, plain text (P6; a plain library, in-process, D-61); `ledger-import` re-exports it as `extract` | none (`hayro`, `zip`, `quick-xml`) |
|
||||
| `ledger-import` | formats: QuickBooks CSV in; IIF, XLSX, Typst PDFs (statements, the invoice, labels) out | `ledger-core`, `ledger-reader` |
|
||||
| `ledger-package` | the shared read-only view over `Store` or `Snapshot` (`PackageBooks`): the CPA package, the 1099 worksheet, the invoice document, Needs attention | `ledger-core`, `ledger-store`, `ledger-config`, `ledger-import`, `ledger-vault` |
|
||||
@@ -353,3 +353,20 @@ package, which repacks the release's tarball), `build-linux.sh`,
|
||||
`repo-indexes.sh` (the apt and dnf indexes), `local-release.sh` (a
|
||||
throwaway-keyed feed on this machine) and `test-linux-packages.sh`.
|
||||
`.github/workflows/release.yml` builds them on a version tag.
|
||||
|
||||
## The result envelope and the guards (`ledger-mcp`, P5)
|
||||
|
||||
Every tool result is given the same fields in one place: `call_tool` runs the
|
||||
tool, then `next_steps::apply` adds `next_steps`, `guards`, `warnings` and
|
||||
(on a tool that is not read-only) `head`, in the structured content and in
|
||||
the text block alike, never overwriting a field the tool wrote. Next steps
|
||||
come from the tool's own result (`next_steps::tool_steps`: the confirm after
|
||||
a preview, what a post, a draft, a statement or a close leads to) and, after
|
||||
a change to an entry or an invoice, from the viewer's own Needs-attention
|
||||
items about it (`ledger_package::find_with`). `guards.rs` is the registry of
|
||||
named refusals, each with the tool that remedies it; a refusal whose text
|
||||
carries a guard gets it as `structuredContent.guards`. The prompts
|
||||
(`tools/prompts.rs`, text in `prompts/*.md`) and the resource templates are
|
||||
served by the same `ServerHandler`; `books.start` is loaded up front by
|
||||
clients that defer tools (`_meta`). `ledger-store`'s `describe.rs` is the
|
||||
one place that turns a journal line into a sentence for an agent.
|
||||
@@ -12,12 +12,16 @@ locked to one assistant.
|
||||
|
||||
| Crate | Role | Spec |
|
||||
|---|---|---|
|
||||
| `ledger-core` | the engine: money, entries, the laws; every rule the tools enforce | `docs/spec/rules.md`, `docs/verification.md` |
|
||||
| `ledger-package` | the read-only views shared with the viewer: reports, aging, the CPA package, Needs attention (the one list `books.start` and the viewer both draw) | `docs/architecture.md` |
|
||||
| `ledger-store` | the hash-chained journal (the source of truth), the SQLite projection, the blob store | `docs/receipt-store.md` |
|
||||
| `ledger-mcp` | the books server: the tools, resources and handbook over store + core | this document |
|
||||
| `ledger-host` | the agent bridge (`wanderbooks mcp`), the host and its leases, tokens | `plan/served.md` 3.3, 3.9 |
|
||||
|
||||
(The overview sections of this document date from the first MCP phase; P5
|
||||
rewrites them. The per-tool sections are held to the code by a test.)
|
||||
(P5 rewrote the overview sections for the surface as it is: the result
|
||||
envelope, `books.start` v2, the prompts, the resource templates and the
|
||||
journal in words, below. The per-tool sections are held to the code by a
|
||||
test: `every_tool_is_documented_and_every_documented_tool_exists`.)
|
||||
|
||||
Rust, `rmcp` SDK (the Rust MCP SDK). The MCP crate holds no accounting
|
||||
logic: it translates tool calls into `Ledger` commands and formats results.
|
||||
@@ -389,8 +393,8 @@ books' files; no tool title or description, connect text or handbook
|
||||
section names `wanderbooks.toml` (`tool_text_lint`). A rule set by law that the books
|
||||
don't know for a year names its official source (the statute or
|
||||
publication, with its address) and says to tell the human.
|
||||
Tool descriptions are at most 700 characters; eight longer ones are listed
|
||||
in `tool_descriptions_fit_700_characters`, to be shortened in P5.
|
||||
Tool descriptions are at most 700 characters (`tool_descriptions_fit_700_characters`
|
||||
allows none longer; the long text lives in the handbook).
|
||||
|
||||
**Annotations.** Every tool has a `title` and a `readOnlyHint`; every tool
|
||||
that changes anything also has a `destructiveHint`, true for these 31:
|
||||
|
||||
@@ -223,3 +223,18 @@ backdated entries included: defect 6), `zero_net_accounts`,
|
||||
`drafts_by_state` and the reconciliation split; reports gained
|
||||
`as_of_status` and `strict`; the CPA package is refused while a draft
|
||||
dated in it is open (4.8).
|
||||
|
||||
### P5: the agent surface (2026-10-29)
|
||||
|
||||
No semantic feature moved, so no status changed and the floor stands. P5
|
||||
changes how an agent is told what to do, not how a figure is computed: the
|
||||
result envelope on every tool result (`next_steps`, `guards`, `warnings`,
|
||||
`head`), `books.start` v2, the prompts, the resource templates,
|
||||
`books.journal_history` (the journal in words) and source corrections
|
||||
(`source_revisions`: `convention.source_revised`, `decision.source_revised`,
|
||||
`open_item.source_revised`). One new row, `partial` until a hand-verified
|
||||
golden covers it:
|
||||
|
||||
| Area | Counterpart (a cross-check, not the spec) | Status | Notes |
|
||||
|---|---|---|---|
|
||||
| Audit trail in words (`books.journal_history`, the `describe` sentences) | QBO "Audit log", Xero "History and notes" | partial | tests `journal_history_traces_what_the_books_did`, `every_event_type_has_a_sentence`; the viewer keeps its own sentences (the shared describer is the server's) |
|
||||
@@ -3745,3 +3745,89 @@ covers the feature one; a case with a higher format number is a gap to
|
||||
add), `swap_in`'s temporary-then-rename (`update_is_atomic`), `healthy`
|
||||
(`failed_update_rolls_back`), `install_kind` (the marker test), and
|
||||
`parse_version`'s rejections.
|
||||
|
||||
## P5: the agent surface (2026-10-29)
|
||||
|
||||
Every tool result carries the envelope (`next_steps`, `guards`, `warnings`,
|
||||
`head`), `books.start` is v2, the prompts and resource templates are
|
||||
served, `books.journal_history` reads the journal in words, and a
|
||||
convention, decision or open item can have its source corrected. Journal
|
||||
feature `source_revisions`; events `convention.source_revised`,
|
||||
`decision.source_revised`, `open_item.source_revised`; no schema change
|
||||
(the projection updates the rows' existing source columns).
|
||||
|
||||
- **`ledger-mcp` `src/server_tests.rs`:**
|
||||
`every_mutating_result_has_head_and_next_steps` (every tool of the
|
||||
router is given the envelope; the non-read-only ones have `head` equal to
|
||||
the journal's and a step each with `why` and `required`; the text block
|
||||
says what the structure says),
|
||||
`the_envelope_keeps_what_a_tool_said_and_carries_a_refusal_guard`,
|
||||
`books_start_is_always_loaded`, `tool_descriptions_fit_700_characters`
|
||||
(the allowance list is gone: every description is at most 700
|
||||
characters). `guards.rs`: `every_guard_names_an_existing_tool`,
|
||||
`a_refusal_becomes_its_guard`. `next_steps.rs`:
|
||||
`a_preview_gets_its_confirm_step`, `every_step_a_tool_names_exists`.
|
||||
`tools/prompts.rs`: `the_prompts_name_only_tools_that_exist`,
|
||||
`an_argument_is_one_token_of_the_right_shape`.
|
||||
- **`ledger-store`:** `tests/source_revisions.rs`:
|
||||
`a_source_only_revision_is_journaled` (the three events, words and
|
||||
revision unchanged, live equals replay equals a rebuilt database),
|
||||
`a_source_revision_must_change_only_the_source` (other words, the same
|
||||
source, no reason, unknown record: refused, journal unchanged),
|
||||
`source_revisions_wait_for_the_upgrade` (refused before the go-ahead with
|
||||
`format_upgrade_pending`, journaling nothing). `format.rs`:
|
||||
`source_revisions_is_its_own_feature`. `describe.rs`:
|
||||
`every_event_type_has_a_sentence`, `settings_show_their_keys_never_their_values`,
|
||||
`a_full_tin_in_a_description_is_masked`,
|
||||
`history_filters_in_order_and_says_when_more_remain`.
|
||||
- **8.0.6:** `books_written_under_older_rules_still_open` hand-writes the
|
||||
three events; the fixture `old-books/p5_before` (writer
|
||||
`old-books-writer/src/bin/p5_before.rs`, run against b97234ac, the
|
||||
revision before P5: a convention revised once, two decisions, two open
|
||||
items, a draft) opens to that version's figures, so no figure moved;
|
||||
`books_p2_served_cut_over_whole` refuses `books.raise_item` with `revises`
|
||||
before consent and takes it after.
|
||||
- **`ledger-cli` `tests/agent_surface.rs`:**
|
||||
`journal_history_traces_what_the_books_did`,
|
||||
`a_source_only_revision_over_the_wire`, `the_prompts_list_and_render`
|
||||
(each prompt rendered equals its file with the blanks filled; an
|
||||
argument that is not one token of the right shape, a missing or extra
|
||||
one, and a struck prompt are refused),
|
||||
`the_resource_templates_are_advertised_and_read`,
|
||||
`books_start_v2_is_whole_and_fits` (at most 8 KiB on the P2 fixture; the
|
||||
agent's self-report in the access log only),
|
||||
`a_new_book_is_told_to_onboard`,
|
||||
`a_mutating_call_answers_with_the_head_and_a_refusal_names_its_remedy`.
|
||||
- **`ledger-cli` `tests/agent_evals.rs`** (transcripts in `tests/agent_evals/`):
|
||||
`agent_evals_follow_only_next_steps` plays six scripted transcripts (no
|
||||
model): onboarding to a first post, a receipt-backed post, a draft that
|
||||
waits for its receipt and posts when it arrives, a reversal, a
|
||||
reconciliation, a month-end close. After `books.start` a transcript may
|
||||
call only a tool (or fetch only a prompt) the previous result's
|
||||
`next_steps` named, with the arguments the step gave and the ones the
|
||||
agent decides. `setup` calls (the chart, the books an agent finds) are
|
||||
not the agent's moves.
|
||||
- **The sweep:** the `party_id`, `document_id` and `cash_account` aliases of
|
||||
`books.create_invoice`, `books.create_bill`, `books.receive_payment` and
|
||||
`books.pay_bill` are gone (S4, S6 said "until P5"); the over-the-wire
|
||||
tests that used them name `customer`/`vendor`, `invoice`/`bill` and
|
||||
`deposit_to`/`paid_from`.
|
||||
|
||||
**For P12 (mutation targets):** `next_steps::apply`'s three branches (an
|
||||
error, a result without an object, the normal path) and the `mutating`
|
||||
test that gives `head`; `tool_steps`'s confirm branch (`!confirmed` and a
|
||||
`preview`), the `receipt_missing` pair before it, and each tool's arm;
|
||||
`attention_steps`'s subject match (entry against invoice) and its
|
||||
`Problem` severity for `required`; `subject_of`'s precedence;
|
||||
`guards::from_refusal`'s structured and worded branches and
|
||||
`Guard::remedy_tool`'s table; `prompts::checked`'s token test and each
|
||||
`Check` shape (month length and calendar, year length, the word list);
|
||||
`prompt_get`'s extra-argument and missing-argument refusals;
|
||||
`describe::subjects`'s per-prefix id choice, `describe`'s `settings.set`
|
||||
key list and TIN mask, `history`'s `>` on `after_seq`, its `limit` cut and
|
||||
`more`; `check_convention_source`'s three equalities (title/body/structure,
|
||||
source, kind), `check_decision_source` and `check_open_item_source`'s
|
||||
equal-source refusals and the empty-reason refusals; the three
|
||||
`apply_*_source` arms (a mutant that also bumps `revision` or `seq` shows
|
||||
in `a_source_only_revision_is_journaled`); `books.start`'s fit order and
|
||||
`reconciliation` split (`reconciled_through.is_some()`).
|
||||
Reference in new issue
Block a user