feat(bookspec): the checker crate, the Engine trait and the runner (P13)
bookspec is a specification first: every rule normative and sourced,
every rule demonstrated by concrete examples, and any accounting engine
checked against them through one interface (D-34). This moves the
examples' schema out of the test that happened to parse them and makes
the interface explicit.
- `bookspec/checker` (its own workspace, inside this repository until
the harness is stable): `Example`, `Setup`, `Event`, `Expect`, `Query`,
`Refusal` and the TOML parser, every structure `deny_unknown_fields`,
with `Example::from_toml` checking the cross-field shape (closed lists,
exact decimal amounts, dates, one event kind, `reverses` counting from
1, the shares summing to 1, commands accepted or refused and not both).
- `Engine` (`load`, `apply`, `query`, `claims`) and `Refusal { rule,
reason }`; the refusal's rule is the bookspec identifier, so a refusal
names the rule it belongs to. `Claims` also carries the option an
engine picks for a CHOICE rule.
- `bookspec/runner` drives any `Engine` through the examples and reports
per rule: pass, fail, not claimed. It compares the answer itself; an
engine is never asked whether it agrees with itself.
- `crates/ledger-core/tests/bookspec.rs` is the wanderbooks adapter over
`Store`: the loader parses through the checker, its own checks are
unchanged, and the new `the_runner_reports_every_claimed_rule_green`
runs all 56 examples through the trait. The IIF appendix is answered
through `ledger_package::export_iif`, the code its own test checks.
- `just check` runs the bookspec workspace's fmt, clippy, rustdoc and
tests too, so an example cannot rot (D-34, Z-17).
The plan's trait has four methods and no path for an example's
`[[changes]]` (a rename, the policy, an asset correction), which post no
entry: `change` is the fifth, required so an engine cannot skip those
examples silently. This is reported to the EM as the one departure.
One money rule holds here too: an answer's figures are exact decimal
strings, never JSON numbers, so no engine can round a cent through a
float.
This commit is contained in:
swaits committed
1 parent
b2d45905f8
commit
7350adb07a
13 files changed
+4538
No files matched your search
@@ -0,0 +1,14 @@
|
||||
[package]
|
||||
name = "bookspec-runner"
|
||||
version = "0.1.0"
|
||||
edition.workspace = true
|
||||
license.workspace = true
|
||||
repository.workspace = true
|
||||
description = "Runs bookspec examples against any engine that implements bookspec's Engine trait."
|
||||
|
||||
[dependencies]
|
||||
bookspec-checker = { path = "../checker" }
|
||||
serde_json = "1"
|
||||
|
||||
[lints]
|
||||
workspace = true
|
||||
@@ -0,0 +1,302 @@
|
||||
//! bookspec's conformance runner: it drives any [`Engine`] through the
|
||||
//! examples and reports, per rule, pass, fail or not claimed
|
||||
//! (`plan/served.md` 8.2 P13 step 2).
|
||||
//!
|
||||
//! The runner is the only judge. An engine answers; the runner compares the
|
||||
//! answer with the example's expected values, and with the refusal the
|
||||
//! example names when the engine refuses. The engine is never asked
|
||||
//! whether its own answer is right.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
|
||||
use bookspec_checker::{Claims, Engine, Example, Expect, Query, Rule, RuleId, Values, rules};
|
||||
|
||||
/// What the runner says about one rule.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum Outcome {
|
||||
/// The engine claims the rule and every example that demonstrates it
|
||||
/// passed.
|
||||
Pass,
|
||||
/// The engine claims the rule and an example failed, or a claim has no
|
||||
/// example behind it.
|
||||
Fail,
|
||||
/// The engine does not claim the rule.
|
||||
NotClaimed,
|
||||
}
|
||||
|
||||
impl Outcome {
|
||||
/// The outcome as the words a report prints.
|
||||
pub fn key(self) -> &'static str {
|
||||
match self {
|
||||
Outcome::Pass => "pass",
|
||||
Outcome::Fail => "fail",
|
||||
Outcome::NotClaimed => "not claimed",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// One example's outcome.
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct ExampleReport {
|
||||
/// The example's id.
|
||||
pub id: String,
|
||||
/// Why it failed, if it did.
|
||||
pub failure: Option<String>,
|
||||
}
|
||||
|
||||
/// The runner's report: what it found of each rule, and each example.
|
||||
#[derive(Clone, Debug, Default, PartialEq, Eq)]
|
||||
pub struct Report {
|
||||
/// Every rule of the catalogue the runner was given, and its outcome.
|
||||
pub rules: BTreeMap<RuleId, Outcome>,
|
||||
/// Every example, in the order run.
|
||||
pub examples: Vec<ExampleReport>,
|
||||
/// Failures that belong to no rule: an example no rule lists, a rule
|
||||
/// claimed but unknown to the catalogue.
|
||||
pub unclaimed: Vec<String>,
|
||||
}
|
||||
|
||||
impl Report {
|
||||
/// True when every claimed rule passed and nothing else failed.
|
||||
pub fn passed(&self) -> bool {
|
||||
self.unclaimed.is_empty()
|
||||
&& self.rules.values().all(|o| *o != Outcome::Fail)
|
||||
&& self.examples.iter().all(|e| e.failure.is_none())
|
||||
}
|
||||
|
||||
/// Every failure, with its rule's outcome where it has one.
|
||||
pub fn failures(&self) -> Vec<String> {
|
||||
let mut out = self.unclaimed.clone();
|
||||
for (rule, outcome) in &self.rules {
|
||||
if *outcome == Outcome::Fail {
|
||||
let ids: Vec<&str> = rules()
|
||||
.iter()
|
||||
.find(|r| &r.id == rule)
|
||||
.map(|r| r.examples.to_vec())
|
||||
.unwrap_or_default();
|
||||
out.push(format!("{rule} failed (examples: {})", ids.join(", ")));
|
||||
}
|
||||
}
|
||||
for e in &self.examples {
|
||||
if let Some(f) = &e.failure {
|
||||
out.push(f.clone());
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
}
|
||||
|
||||
/// Run `examples` against engines made by `make`, one fresh engine per
|
||||
/// example, and report per rule against `catalog` (normally
|
||||
/// [`bookspec_checker::rules()`]).
|
||||
pub fn run<'a, E>(
|
||||
examples: &'a [Example],
|
||||
catalog: &[Rule],
|
||||
mut make: impl FnMut(&'a Example) -> E,
|
||||
) -> Report
|
||||
where
|
||||
E: Engine + 'a,
|
||||
{
|
||||
let mut out = Report::default();
|
||||
let mut claims = Claims::default();
|
||||
for ex in examples {
|
||||
let mut engine = make(ex);
|
||||
let declared = engine.claims();
|
||||
claims.rules.extend(declared.rules);
|
||||
for (rule, choice) in declared.choices {
|
||||
claims.choices.entry(rule).or_insert(choice);
|
||||
}
|
||||
let failure = run_example(ex, &mut engine).err();
|
||||
out.examples.push(ExampleReport {
|
||||
id: ex.id.clone(),
|
||||
failure,
|
||||
});
|
||||
}
|
||||
for rule in catalog {
|
||||
let outcome = if !claims.claims(&rule.id) {
|
||||
Outcome::NotClaimed
|
||||
} else if !rule.has_examples() {
|
||||
out.unclaimed
|
||||
.push(format!("{} is claimed but has no example", rule.id));
|
||||
Outcome::Fail
|
||||
} else {
|
||||
let failed = rule.examples.iter().any(|id| {
|
||||
out.examples
|
||||
.iter()
|
||||
.find(|e| &e.id == id)
|
||||
.is_none_or(|e| e.failure.is_some())
|
||||
});
|
||||
if failed { Outcome::Fail } else { Outcome::Pass }
|
||||
};
|
||||
out.rules.insert(rule.id.clone(), outcome);
|
||||
}
|
||||
for rule in &claims.rules {
|
||||
if !catalog.iter().any(|r| &r.id == rule) {
|
||||
out.unclaimed
|
||||
.push(format!("{rule} is claimed but is not in the catalogue"));
|
||||
}
|
||||
}
|
||||
for ex in examples {
|
||||
if !catalog.iter().any(|r| r.examples.contains(&ex.id.as_str())) {
|
||||
out.unclaimed
|
||||
.push(format!("example {} is listed under no rule", ex.id));
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// Run one example against one engine: load the setup, apply every event
|
||||
/// (accepted or refused as the example says), check the expectations named
|
||||
/// after each event, make the changes named after it, then check the rest.
|
||||
pub fn run_example(ex: &Example, engine: &mut dyn Engine) -> Result<(), String> {
|
||||
engine
|
||||
.load(&ex.setup)
|
||||
.map_err(|r| format!("load refused: {} ({})", r.reason, r.rule))?;
|
||||
for (i, event) in ex.events.iter().enumerate() {
|
||||
let n = i + 1;
|
||||
match (engine.apply(event), &event.refused) {
|
||||
(Ok(()), None) => {}
|
||||
(Err(r), Some(want)) if &r.reason == want => {}
|
||||
(Err(r), Some(want)) => {
|
||||
return Err(format!(
|
||||
"event {n}: refused {:?}, expected {want:?} ({})",
|
||||
r.reason, r.rule
|
||||
));
|
||||
}
|
||||
(Err(r), None) => {
|
||||
return Err(format!(
|
||||
"event {n}: refused {:?}, expected accepted ({})",
|
||||
r.reason, r.rule
|
||||
));
|
||||
}
|
||||
(Ok(()), Some(want)) => {
|
||||
return Err(format!("event {n}: accepted, expected refused: {want}"));
|
||||
}
|
||||
}
|
||||
for x in ex.expect.iter().filter(|x| x.after == Some(n)) {
|
||||
check(engine, x).map_err(|m| format!("after event {n}: {m}"))?;
|
||||
}
|
||||
for c in ex.changes.iter().filter(|c| c.after == n) {
|
||||
engine
|
||||
.change(c)
|
||||
.map_err(|r| format!("change after {n}: {} ({})", r.reason, r.rule))?;
|
||||
}
|
||||
}
|
||||
for x in ex.expect.iter().filter(|x| x.after.is_none()) {
|
||||
check(engine, x)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Check one expectation against the engine's answer.
|
||||
fn check(engine: &dyn Engine, x: &Expect) -> Result<(), String> {
|
||||
let q = x.query();
|
||||
let answer = engine.query(&q);
|
||||
match (answer, &x.refused, x.accepted) {
|
||||
(Ok(v), None, Some(true)) => {
|
||||
if v == serde_json::Value::Bool(true) {
|
||||
Ok(())
|
||||
} else {
|
||||
Err(format!(
|
||||
"{}: accepted, but answered {v} (a command shows no values)",
|
||||
q.name
|
||||
))
|
||||
}
|
||||
}
|
||||
(Ok(v), None, None) => compare(x, &v),
|
||||
(Ok(v), Some(want), _) => Err(format!(
|
||||
"{}: answered {v}, expected refused: {want}",
|
||||
q.name
|
||||
)),
|
||||
(Err(r), Some(want), _) if &r.reason == want => Ok(()),
|
||||
(Err(r), Some(want), _) => Err(format!(
|
||||
"{}: refused {:?}, expected {want:?} ({})",
|
||||
q.name, r.reason, r.rule
|
||||
)),
|
||||
(Err(r), None, _) => Err(format!("{}: refused {:?} ({})", q.name, r.reason, r.rule)),
|
||||
(Ok(v), None, Some(false)) => Err(format!(
|
||||
"{}: accepted = false is not the examples' vocabulary; answered {v}",
|
||||
q.name
|
||||
)),
|
||||
}
|
||||
}
|
||||
|
||||
/// Compare an answer with what the example expects.
|
||||
fn compare(x: &Expect, got: &serde_json::Value) -> Result<(), String> {
|
||||
let q: Query = x.query();
|
||||
let Some(want) = &x.values else {
|
||||
return Err(format!("{}: expects values, none given", q.name));
|
||||
};
|
||||
match want {
|
||||
Values::Figures(figures) => {
|
||||
let serde_json::Value::Object(map) = got else {
|
||||
return Err(format!("{}: answered {got}, expected a table", q.name));
|
||||
};
|
||||
for (key, want) in figures {
|
||||
match map.get(key) {
|
||||
Some(serde_json::Value::String(got)) if got == want => {}
|
||||
Some(got) => {
|
||||
return Err(format!("{}: {key}: got {got}, expected {want:?}", q.name));
|
||||
}
|
||||
None => {
|
||||
return Err(format!("{}: {key} is missing (expected {want:?})", q.name));
|
||||
}
|
||||
}
|
||||
}
|
||||
if q.is_complete() {
|
||||
let extra: Vec<&String> =
|
||||
map.keys().filter(|k| !figures.contains_key(*k)).collect();
|
||||
let missing: Vec<&String> =
|
||||
figures.keys().filter(|k| !map.contains_key(*k)).collect();
|
||||
if !extra.is_empty() || !missing.is_empty() {
|
||||
return Err(format!(
|
||||
"{}: lists exactly what it covers; missing {missing:?}, extra {extra:?}",
|
||||
q.name
|
||||
));
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
Values::Lines(lines) => {
|
||||
let serde_json::Value::Array(got) = got else {
|
||||
return Err(format!("{}: answered {got}, expected a list", q.name));
|
||||
};
|
||||
let got: Vec<&str> = got
|
||||
.iter()
|
||||
.map(|v| v.as_str().unwrap_or("<not text>"))
|
||||
.collect();
|
||||
let want: Vec<&str> = lines.iter().map(String::as_str).collect();
|
||||
if got == want {
|
||||
Ok(())
|
||||
} else {
|
||||
Err(format!("{}: got {got:?}, expected {want:?}", q.name))
|
||||
}
|
||||
}
|
||||
Values::Rows(rows) => {
|
||||
let serde_json::Value::Array(got) = got else {
|
||||
return Err(format!("{}: answered {got}, expected rows", q.name));
|
||||
};
|
||||
let got: Vec<Vec<&str>> = got
|
||||
.iter()
|
||||
.map(|row| {
|
||||
row.as_array()
|
||||
.map(|r| {
|
||||
r.iter()
|
||||
.map(|v| v.as_str().unwrap_or("<not text>"))
|
||||
.collect()
|
||||
})
|
||||
.unwrap_or_default()
|
||||
})
|
||||
.collect();
|
||||
let want: Vec<Vec<&str>> = rows
|
||||
.iter()
|
||||
.map(|r| r.iter().map(String::as_str).collect())
|
||||
.collect();
|
||||
if got == want {
|
||||
Ok(())
|
||||
} else {
|
||||
Err(format!("{}: got {got:?}, expected {want:?}", q.name))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,240 @@
|
||||
//! The runner's own tests, with an engine that only pretends: the runner
|
||||
//! must judge the answer, never the engine's belief about it.
|
||||
|
||||
use bookspec_checker::{
|
||||
Area, ChangeSpec, Claims, Engine, Event, Example, Query, Refusal, Rule, RuleId, Setup, Status,
|
||||
};
|
||||
use bookspec_runner::Outcome;
|
||||
|
||||
const EXAMPLE: &str = r#"
|
||||
id = "fake.example"
|
||||
title = "A fake engine answers"
|
||||
rule = "The chart's names MUST be the account_names the example expects."
|
||||
status = "AUTHORITY"
|
||||
sources = ["bookspec/README.md"]
|
||||
tags = ["runner"]
|
||||
|
||||
events = []
|
||||
|
||||
[setup]
|
||||
currency = "USD"
|
||||
fiscal_year_start = "01-01"
|
||||
basis = "accrual"
|
||||
rounding = "half-even"
|
||||
|
||||
[[setup.accounts]]
|
||||
code = "1000"
|
||||
name = "Cash"
|
||||
type = "asset"
|
||||
|
||||
[[setup.accounts]]
|
||||
code = "3000"
|
||||
name = "Capital"
|
||||
type = "equity"
|
||||
|
||||
[[expect]]
|
||||
query = "account_names"
|
||||
|
||||
[expect.values]
|
||||
"1000" = "Cash"
|
||||
"3000" = "Capital"
|
||||
"#;
|
||||
|
||||
fn example() -> Example {
|
||||
Example::from_toml(EXAMPLE).expect("the fake example is well formed")
|
||||
}
|
||||
|
||||
fn catalog() -> Vec<Rule> {
|
||||
vec![Rule {
|
||||
b: 1,
|
||||
id: RuleId::new(Area::Entry, 1),
|
||||
area: Area::Entry,
|
||||
title: "Every entry is a double entry: debits equal credits",
|
||||
status: Some(Status::Authority),
|
||||
examples: &["fake.example"],
|
||||
}]
|
||||
}
|
||||
|
||||
/// The rules an engine that claims the catalogue's example states.
|
||||
fn claim() -> Claims {
|
||||
Claims::new([RuleId::new(Area::Entry, 1)])
|
||||
}
|
||||
|
||||
struct Fake {
|
||||
claims: Claims,
|
||||
names: serde_json::Value,
|
||||
accept: bool,
|
||||
refuse_query: Option<Refusal>,
|
||||
change_ok: bool,
|
||||
}
|
||||
|
||||
impl Default for Fake {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
claims: claim(),
|
||||
names: serde_json::json!({"1000": "Cash", "3000": "Capital"}),
|
||||
accept: true,
|
||||
refuse_query: None,
|
||||
change_ok: true,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Engine for Fake {
|
||||
fn load(&mut self, _setup: &Setup) -> Result<(), Refusal> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn apply(&mut self, event: &Event) -> Result<(), Refusal> {
|
||||
if let Some(reason) = &event.refused {
|
||||
return Err(Refusal::of_reason(reason));
|
||||
}
|
||||
if self.accept {
|
||||
Ok(())
|
||||
} else {
|
||||
Err(Refusal::of_reason("entry_irreversible"))
|
||||
}
|
||||
}
|
||||
|
||||
fn query(&self, q: &Query) -> Result<serde_json::Value, Refusal> {
|
||||
if let Some(r) = &self.refuse_query {
|
||||
return Err(r.clone());
|
||||
}
|
||||
match q.name.as_str() {
|
||||
"account_names" => Ok(self.names.clone()),
|
||||
other => Err(Refusal::new(
|
||||
RuleId::new(Area::Entry, 1),
|
||||
format!("the fake engine answers no {other}"),
|
||||
)),
|
||||
}
|
||||
}
|
||||
|
||||
fn claims(&self) -> Claims {
|
||||
self.claims.clone()
|
||||
}
|
||||
|
||||
fn change(&mut self, _change: &ChangeSpec) -> Result<(), Refusal> {
|
||||
if self.change_ok {
|
||||
Ok(())
|
||||
} else {
|
||||
Err(Refusal::of_reason("unchanged"))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// An engine that answers what the example expects passes, and its rule is
|
||||
/// claimed and passed.
|
||||
#[test]
|
||||
fn a_passing_engine_passes() {
|
||||
let examples = [example()];
|
||||
let report = bookspec_runner::run(&examples, &catalog(), |_| Fake::default());
|
||||
assert!(report.passed(), "{:?}", report.failures());
|
||||
assert_eq!(
|
||||
report.rules.get(&RuleId::new(Area::Entry, 1)),
|
||||
Some(&Outcome::Pass)
|
||||
);
|
||||
}
|
||||
|
||||
/// The runner judges the answer: a wrong figure fails the rule, whatever
|
||||
/// the engine says about itself.
|
||||
#[test]
|
||||
fn a_wrong_answer_fails() {
|
||||
let examples = [example()];
|
||||
let report = bookspec_runner::run(&examples, &catalog(), |_| Fake {
|
||||
names: serde_json::json!({"1000": "Coin", "3000": "Capital"}),
|
||||
..Fake::default()
|
||||
});
|
||||
assert!(!report.passed());
|
||||
assert_eq!(
|
||||
report.rules.get(&RuleId::new(Area::Entry, 1)),
|
||||
Some(&Outcome::Fail)
|
||||
);
|
||||
assert!(
|
||||
report.failures().iter().any(|f| f.contains("1000")),
|
||||
"{:?}",
|
||||
report.failures()
|
||||
);
|
||||
}
|
||||
|
||||
/// An engine that claims nothing is not claimed, not failed; and the
|
||||
/// example it does not claim is still run (its answer is compared).
|
||||
#[test]
|
||||
fn an_engine_that_claims_nothing_is_not_claimed() {
|
||||
let examples = [example()];
|
||||
let report = bookspec_runner::run(&examples, &catalog(), |_| Fake {
|
||||
claims: Claims::default(),
|
||||
..Fake::default()
|
||||
});
|
||||
assert!(report.passed(), "{:?}", report.failures());
|
||||
assert_eq!(
|
||||
report.rules.get(&RuleId::new(Area::Entry, 1)),
|
||||
Some(&Outcome::NotClaimed)
|
||||
);
|
||||
}
|
||||
|
||||
/// A claim with no example behind it is a failure: a claim needs evidence.
|
||||
#[test]
|
||||
fn a_claim_without_an_example_fails() {
|
||||
let mut catalog = catalog();
|
||||
catalog[0].examples = &[];
|
||||
let examples = [example()];
|
||||
let report = bookspec_runner::run(&examples, &catalog, |_| Fake::default());
|
||||
assert!(!report.passed());
|
||||
assert_eq!(
|
||||
report.rules.get(&RuleId::new(Area::Entry, 1)),
|
||||
Some(&Outcome::Fail)
|
||||
);
|
||||
}
|
||||
|
||||
/// A query the engine refuses fails, and says which rule it refused under.
|
||||
#[test]
|
||||
fn a_refused_query_fails() {
|
||||
let examples = [example()];
|
||||
let report = bookspec_runner::run(&examples, &catalog(), |_| Fake {
|
||||
refuse_query: Some(Refusal::new(
|
||||
RuleId::new(Area::Ledger, 1),
|
||||
"not implemented",
|
||||
)),
|
||||
..Fake::default()
|
||||
});
|
||||
assert!(!report.passed());
|
||||
let failures = report.failures().join("\n");
|
||||
assert!(failures.contains("not implemented"), "{failures}");
|
||||
assert!(failures.contains("BS-ledger-1"), "{failures}");
|
||||
}
|
||||
|
||||
/// An example no rule lists is a failure: it proves nothing.
|
||||
#[test]
|
||||
fn an_example_listed_under_no_rule_fails() {
|
||||
let examples = [example()];
|
||||
let report = bookspec_runner::run(&examples, &[], |_| Fake::default());
|
||||
assert!(!report.passed());
|
||||
assert!(
|
||||
report
|
||||
.failures()
|
||||
.iter()
|
||||
.any(|f| f.contains("listed under no rule")),
|
||||
"{:?}",
|
||||
report.failures()
|
||||
);
|
||||
}
|
||||
|
||||
/// A change the engine refuses fails the example.
|
||||
#[test]
|
||||
fn a_refused_change_fails() {
|
||||
let text = EXAMPLE.replace("events = []\n", "").replace(
|
||||
"[[expect]]",
|
||||
"[[events]]\ndate = \"2026-01-02\"\ndescription = \"A\"\nlines = [\n { account = \"1000\", side = \"debit\", amount = \"1.00\" },\n { account = \"3000\", side = \"credit\", amount = \"1.00\" },\n]\n\n[[changes]]\nafter = 1\nrequire_tax_category = \"refuse\"\n\n[[expect]]",
|
||||
);
|
||||
let examples = [Example::from_toml(&text).unwrap()];
|
||||
let report = bookspec_runner::run(&examples, &catalog(), |_| Fake {
|
||||
change_ok: false,
|
||||
..Fake::default()
|
||||
});
|
||||
assert!(!report.passed());
|
||||
assert!(
|
||||
report.failures().iter().any(|f| f.contains("unchanged")),
|
||||
"{:?}",
|
||||
report.failures()
|
||||
);
|
||||
}
|
||||
Reference in new issue
Block a user