test(agent): scripted evals follow only next_steps; the surface over the wire (P5)
Agent reason: the envelope is only worth having if an agent can do its work by following it. crates/ledger-cli/tests/agent_evals/ holds six scripted transcripts, no model: onboarding to a first post, a receipt-backed post, a draft that waits for its receipt and posts when it arrives, a reversal, a reconciliation, a month-end close. agent_evals_follow_only_next_steps plays each against a real books server: after books.start a step may call only a tool (or fetch only a prompt) the previous result's next_steps named, with the arguments the step gave and the ones the agent decides; setup calls (the chart and the books an agent finds) are not the agent's moves. agent_surface.rs holds the over-the-wire tests of the rest: the journal in words (order, subject and type filters, paging, a bad subject refused), source corrections through the three tools, the prompts rendered against their files and refused when malformed, the resource templates advertised and read, books.start v2 whole and under 8 KiB with the self-report in the access log only, a new book told to onboard, and a mutating call answering with the head the journal has and a refusal naming its remedy. Docs: testing.md's P5 section (every test, and the For-P12 paragraph naming what a mutant would flip), parity.md (no feature moved; one row), and architecture.md (the envelope and the guards); mcp-interface.md's overview sections now describe the surface as built. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HS8WZziDnyxunC23MUxrLp
This commit is contained in:
1 parent
a319a64c3a
commit
7bac25496e
12 files changed
+1358
-5
No files matched your search
@@ -0,0 +1,102 @@
|
||||
{
|
||||
"description": "A draft that waits for its receipt, and posts when it arrives.",
|
||||
"settings": "[company]\nname = \"Eval Co\"\nfiscal_year_start = \"01-01\"\n[books]\ncash_accounts = [\"1000\"]\n",
|
||||
"setup": [
|
||||
{
|
||||
"call": "books.add_account",
|
||||
"args": {
|
||||
"code": "1000",
|
||||
"name": "Cash",
|
||||
"account_type": "asset"
|
||||
}
|
||||
},
|
||||
{
|
||||
"call": "books.add_account",
|
||||
"args": {
|
||||
"code": "3000",
|
||||
"name": "Capital",
|
||||
"account_type": "equity"
|
||||
}
|
||||
},
|
||||
{
|
||||
"call": "books.add_account",
|
||||
"args": {
|
||||
"code": "6000",
|
||||
"name": "Rent",
|
||||
"account_type": "expense"
|
||||
}
|
||||
}
|
||||
],
|
||||
"steps": [
|
||||
{
|
||||
"call": "books.start",
|
||||
"args": {}
|
||||
},
|
||||
{
|
||||
"follow": "books.draft_entry",
|
||||
"with": {
|
||||
"date": "2026-01-10",
|
||||
"description": "Landlord, January",
|
||||
"no_party_reason": "test"
|
||||
}
|
||||
},
|
||||
{
|
||||
"follow": "books.add_line",
|
||||
"with": {
|
||||
"account": "6000",
|
||||
"side": "debit",
|
||||
"amount": "40.00"
|
||||
}
|
||||
},
|
||||
{
|
||||
"follow": "books.add_line",
|
||||
"with": {
|
||||
"account": "1000",
|
||||
"side": "credit",
|
||||
"amount": "40.00"
|
||||
}
|
||||
},
|
||||
{
|
||||
"follow": "books.post_entry",
|
||||
"with": {}
|
||||
},
|
||||
{
|
||||
"follow": "books.set_draft_state",
|
||||
"with": {
|
||||
"state": "awaiting_evidence",
|
||||
"note": "the landlord's receipt"
|
||||
}
|
||||
},
|
||||
{
|
||||
"expect": {
|
||||
"/state": "awaiting_evidence"
|
||||
}
|
||||
},
|
||||
{
|
||||
"follow": "books.attach_receipt",
|
||||
"with": {
|
||||
"upload_id": {
|
||||
"$upload": {
|
||||
"name": "receipt.pdf",
|
||||
"mime": "application/pdf",
|
||||
"text": "%PDF-1.4 a receipt for rent",
|
||||
"purpose": "receipt"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"follow": "books.post_entry",
|
||||
"with": {}
|
||||
},
|
||||
{
|
||||
"follow": "books.post_entry",
|
||||
"with": {}
|
||||
},
|
||||
{
|
||||
"expect": {
|
||||
"/status": "posted"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,83 @@
|
||||
{
|
||||
"description": "A period that ended is closed: the open period is found, previewed, closed, and the books verified.",
|
||||
"settings": "[company]\nname = \"Eval Co\"\nfiscal_year_start = \"01-01\"\n[books]\ncash_accounts = [\"1000\"]\n",
|
||||
"setup": [
|
||||
{
|
||||
"call": "books.add_account",
|
||||
"args": {
|
||||
"code": "1000",
|
||||
"name": "Cash",
|
||||
"account_type": "asset"
|
||||
}
|
||||
},
|
||||
{
|
||||
"call": "books.add_account",
|
||||
"args": {
|
||||
"code": "3000",
|
||||
"name": "Capital",
|
||||
"account_type": "equity"
|
||||
}
|
||||
},
|
||||
{
|
||||
"call": "books.add_account",
|
||||
"args": {
|
||||
"code": "6000",
|
||||
"name": "Rent",
|
||||
"account_type": "expense"
|
||||
}
|
||||
},
|
||||
{
|
||||
"call": "books.create_period",
|
||||
"args": {
|
||||
"name": "2025-12",
|
||||
"start": "2025-12-01",
|
||||
"end": "2025-12-31"
|
||||
}
|
||||
}
|
||||
],
|
||||
"steps": [
|
||||
{
|
||||
"call": "books.start",
|
||||
"args": {
|
||||
"focus": "close"
|
||||
}
|
||||
},
|
||||
{
|
||||
"expect_next_steps_contains": "books.list_periods"
|
||||
},
|
||||
{
|
||||
"follow": "books.list_periods",
|
||||
"with": {}
|
||||
},
|
||||
{
|
||||
"follow": "books.close_period",
|
||||
"with": {}
|
||||
},
|
||||
{
|
||||
"expect": {
|
||||
"/status": "preview"
|
||||
}
|
||||
},
|
||||
{
|
||||
"follow": "books.close_period",
|
||||
"with": {}
|
||||
},
|
||||
{
|
||||
"expect": {
|
||||
"/status": "closed"
|
||||
}
|
||||
},
|
||||
{
|
||||
"expect_next_steps_contains": "books.verify"
|
||||
},
|
||||
{
|
||||
"follow": "books.verify",
|
||||
"with": {}
|
||||
},
|
||||
{
|
||||
"expect": {
|
||||
"/status": "ok"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,102 @@
|
||||
{
|
||||
"description": "A new book: onboarding to the first posted entry, following only next_steps.",
|
||||
"steps": [
|
||||
{
|
||||
"call": "books.start",
|
||||
"args": {
|
||||
"agent": {
|
||||
"model": "scripted",
|
||||
"reasoning_effort": "none",
|
||||
"harness": "agent_evals"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"expect": {
|
||||
"/state": "needs_onboarding"
|
||||
}
|
||||
},
|
||||
{
|
||||
"expect_next_steps_contains": "books.set_setting"
|
||||
},
|
||||
{
|
||||
"follow_prompt": "onboard"
|
||||
},
|
||||
{
|
||||
"follow": "books.set_setting",
|
||||
"with": {
|
||||
"changes": [
|
||||
{
|
||||
"key": "company.name",
|
||||
"value": "Eval Co"
|
||||
}
|
||||
],
|
||||
"decided_by": "Avery Quill"
|
||||
}
|
||||
},
|
||||
{
|
||||
"expect_next_steps_contains": "books.add_account"
|
||||
},
|
||||
{
|
||||
"follow": "books.add_account",
|
||||
"with": {
|
||||
"code": "1000",
|
||||
"name": "Cash",
|
||||
"account_type": "asset"
|
||||
}
|
||||
},
|
||||
{
|
||||
"follow": "books.add_account",
|
||||
"with": {
|
||||
"code": "3000",
|
||||
"name": "Capital",
|
||||
"account_type": "equity"
|
||||
}
|
||||
},
|
||||
{
|
||||
"follow": "books.draft_entry",
|
||||
"with": {
|
||||
"date": "2026-01-05",
|
||||
"description": "Opening capital",
|
||||
"no_party_reason": "opening balance"
|
||||
}
|
||||
},
|
||||
{
|
||||
"follow": "books.add_line",
|
||||
"with": {
|
||||
"account": "1000",
|
||||
"side": "debit",
|
||||
"amount": "10.00"
|
||||
}
|
||||
},
|
||||
{
|
||||
"follow": "books.add_line",
|
||||
"with": {
|
||||
"account": "3000",
|
||||
"side": "credit",
|
||||
"amount": "10.00"
|
||||
}
|
||||
},
|
||||
{
|
||||
"follow": "books.post_entry",
|
||||
"with": {}
|
||||
},
|
||||
{
|
||||
"expect": {
|
||||
"/status": "preview"
|
||||
}
|
||||
},
|
||||
{
|
||||
"follow": "books.post_entry",
|
||||
"with": {}
|
||||
},
|
||||
{
|
||||
"expect": {
|
||||
"/status": "posted"
|
||||
}
|
||||
},
|
||||
{
|
||||
"expect_next_steps_contains": "books.show"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,96 @@
|
||||
{
|
||||
"description": "A post the receipt policy covers: the preview asks for the receipt, the receipt is attached, the entry posts.",
|
||||
"settings": "[company]\nname = \"Eval Co\"\nfiscal_year_start = \"01-01\"\n[books]\ncash_accounts = [\"1000\"]\n",
|
||||
"setup": [
|
||||
{
|
||||
"call": "books.add_account",
|
||||
"args": {
|
||||
"code": "1000",
|
||||
"name": "Cash",
|
||||
"account_type": "asset"
|
||||
}
|
||||
},
|
||||
{
|
||||
"call": "books.add_account",
|
||||
"args": {
|
||||
"code": "3000",
|
||||
"name": "Capital",
|
||||
"account_type": "equity"
|
||||
}
|
||||
},
|
||||
{
|
||||
"call": "books.add_account",
|
||||
"args": {
|
||||
"code": "6000",
|
||||
"name": "Rent",
|
||||
"account_type": "expense"
|
||||
}
|
||||
}
|
||||
],
|
||||
"steps": [
|
||||
{
|
||||
"call": "books.start",
|
||||
"args": {}
|
||||
},
|
||||
{
|
||||
"expect_next_steps_contains": "books.draft_entry"
|
||||
},
|
||||
{
|
||||
"follow": "books.draft_entry",
|
||||
"with": {
|
||||
"date": "2026-01-10",
|
||||
"description": "January rent",
|
||||
"no_party_reason": "test"
|
||||
}
|
||||
},
|
||||
{
|
||||
"follow": "books.add_line",
|
||||
"with": {
|
||||
"account": "6000",
|
||||
"side": "debit",
|
||||
"amount": "40.00"
|
||||
}
|
||||
},
|
||||
{
|
||||
"follow": "books.add_line",
|
||||
"with": {
|
||||
"account": "1000",
|
||||
"side": "credit",
|
||||
"amount": "40.00"
|
||||
}
|
||||
},
|
||||
{
|
||||
"follow": "books.post_entry",
|
||||
"with": {}
|
||||
},
|
||||
{
|
||||
"expect_next_steps_contains": "books.attach_receipt"
|
||||
},
|
||||
{
|
||||
"follow": "books.attach_receipt",
|
||||
"with": {
|
||||
"upload_id": {
|
||||
"$upload": {
|
||||
"name": "receipt.pdf",
|
||||
"mime": "application/pdf",
|
||||
"text": "%PDF-1.4 a receipt for rent",
|
||||
"purpose": "receipt"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"follow": "books.post_entry",
|
||||
"with": {}
|
||||
},
|
||||
{
|
||||
"follow": "books.post_entry",
|
||||
"with": {}
|
||||
},
|
||||
{
|
||||
"expect": {
|
||||
"/status": "posted"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,148 @@
|
||||
{
|
||||
"description": "A cash account reconciled to its statement: record it, mark the lines, reconcile.",
|
||||
"settings": "[company]\nname = \"Eval Co\"\nfiscal_year_start = \"01-01\"\n[books]\ncash_accounts = [\"1000\"]\n",
|
||||
"setup": [
|
||||
{
|
||||
"call": "books.add_account",
|
||||
"args": {
|
||||
"code": "1000",
|
||||
"name": "Cash",
|
||||
"account_type": "asset"
|
||||
}
|
||||
},
|
||||
{
|
||||
"call": "books.add_account",
|
||||
"args": {
|
||||
"code": "3000",
|
||||
"name": "Capital",
|
||||
"account_type": "equity"
|
||||
}
|
||||
},
|
||||
{
|
||||
"call": "books.add_account",
|
||||
"args": {
|
||||
"code": "6000",
|
||||
"name": "Rent",
|
||||
"account_type": "expense"
|
||||
}
|
||||
},
|
||||
{
|
||||
"call": "books.draft_entry",
|
||||
"args": {
|
||||
"date": "2026-01-05",
|
||||
"description": "Opening capital",
|
||||
"no_party_reason": "opening balance"
|
||||
},
|
||||
"save": {
|
||||
"entry": "/entry_id"
|
||||
}
|
||||
},
|
||||
{
|
||||
"call": "books.add_line",
|
||||
"args": {
|
||||
"entry_id": {
|
||||
"$var": "entry"
|
||||
},
|
||||
"account": "1000",
|
||||
"side": "debit",
|
||||
"amount": "10.00"
|
||||
}
|
||||
},
|
||||
{
|
||||
"call": "books.add_line",
|
||||
"args": {
|
||||
"entry_id": {
|
||||
"$var": "entry"
|
||||
},
|
||||
"account": "3000",
|
||||
"side": "credit",
|
||||
"amount": "10.00"
|
||||
}
|
||||
},
|
||||
{
|
||||
"call": "books.post_entry",
|
||||
"args": {
|
||||
"entry_id": {
|
||||
"$var": "entry"
|
||||
},
|
||||
"confirmed": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"call": "books.file_document",
|
||||
"args": {
|
||||
"category": "statement",
|
||||
"file_name": "jan.pdf",
|
||||
"mime_type": "application/pdf",
|
||||
"upload_id": {
|
||||
"$upload": {
|
||||
"name": "jan.pdf",
|
||||
"mime": "application/pdf",
|
||||
"text": "%PDF-1.4 January statement",
|
||||
"purpose": "document"
|
||||
}
|
||||
}
|
||||
},
|
||||
"save": {
|
||||
"statement": "/attachment_id"
|
||||
}
|
||||
}
|
||||
],
|
||||
"steps": [
|
||||
{
|
||||
"call": "books.start",
|
||||
"args": {}
|
||||
},
|
||||
{
|
||||
"expect_next_steps_contains": "books.record_statement"
|
||||
},
|
||||
{
|
||||
"follow": "books.record_statement",
|
||||
"with": {
|
||||
"attachment_id": {
|
||||
"$var": "statement"
|
||||
},
|
||||
"account": "1000",
|
||||
"period_start": "2026-01-01",
|
||||
"period_end": "2026-01-31",
|
||||
"opening_balance": "0.00",
|
||||
"closing_balance": "10.00"
|
||||
}
|
||||
},
|
||||
{
|
||||
"follow": "books.mark_cleared",
|
||||
"with": {
|
||||
"entry_id": {
|
||||
"$var": "entry"
|
||||
},
|
||||
"line_index": 0,
|
||||
"cleared": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"follow": "books.reconcile",
|
||||
"with": {
|
||||
"account": "1000",
|
||||
"statement_date": "2026-01-31",
|
||||
"statement_balance": "10.00",
|
||||
"statement_attachment": {
|
||||
"$var": "statement"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"expect": {
|
||||
"/status": "preview"
|
||||
}
|
||||
},
|
||||
{
|
||||
"follow": "books.reconcile",
|
||||
"with": {}
|
||||
},
|
||||
{
|
||||
"expect": {
|
||||
"/status": "reconciled"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,94 @@
|
||||
{
|
||||
"description": "A posted entry reversed: previewed, confirmed, never edited.",
|
||||
"settings": "[company]\nname = \"Eval Co\"\nfiscal_year_start = \"01-01\"\n[books]\ncash_accounts = [\"1000\"]\n",
|
||||
"setup": [
|
||||
{
|
||||
"call": "books.add_account",
|
||||
"args": {
|
||||
"code": "1000",
|
||||
"name": "Cash",
|
||||
"account_type": "asset"
|
||||
}
|
||||
},
|
||||
{
|
||||
"call": "books.add_account",
|
||||
"args": {
|
||||
"code": "3000",
|
||||
"name": "Capital",
|
||||
"account_type": "equity"
|
||||
}
|
||||
},
|
||||
{
|
||||
"call": "books.add_account",
|
||||
"args": {
|
||||
"code": "6000",
|
||||
"name": "Rent",
|
||||
"account_type": "expense"
|
||||
}
|
||||
}
|
||||
],
|
||||
"steps": [
|
||||
{
|
||||
"call": "books.start",
|
||||
"args": {}
|
||||
},
|
||||
{
|
||||
"follow": "books.draft_entry",
|
||||
"with": {
|
||||
"date": "2026-01-10",
|
||||
"description": "Coffee beans",
|
||||
"no_party_reason": "test"
|
||||
}
|
||||
},
|
||||
{
|
||||
"follow": "books.add_line",
|
||||
"with": {
|
||||
"account": "6000",
|
||||
"side": "debit",
|
||||
"amount": "10.00"
|
||||
}
|
||||
},
|
||||
{
|
||||
"follow": "books.add_line",
|
||||
"with": {
|
||||
"account": "1000",
|
||||
"side": "credit",
|
||||
"amount": "10.00"
|
||||
}
|
||||
},
|
||||
{
|
||||
"follow": "books.post_entry",
|
||||
"with": {}
|
||||
},
|
||||
{
|
||||
"follow": "books.post_entry",
|
||||
"with": {}
|
||||
},
|
||||
{
|
||||
"expect": {
|
||||
"/status": "posted"
|
||||
}
|
||||
},
|
||||
{
|
||||
"follow": "books.reverse_entry",
|
||||
"with": {
|
||||
"date": "2026-01-11",
|
||||
"description": "Reverse: entered in error"
|
||||
}
|
||||
},
|
||||
{
|
||||
"expect": {
|
||||
"/status": "preview"
|
||||
}
|
||||
},
|
||||
{
|
||||
"follow": "books.reverse_entry",
|
||||
"with": {}
|
||||
},
|
||||
{
|
||||
"expect": {
|
||||
"/status": "posted"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
Reference in new issue
Block a user