# 003. Latency analysis: hsmc hot-path cost surface This is an audit of where wall-clock and code-size cost lives in `hsmc`'s event-dispatch path. It is intended to inform mitigation work for embedded and embassy-rs use cases where every microsecond of dispatch overhead couples directly into hardware-control loop tightness and jitter. The audit is **analytical**, with three categories of evidence: 1. **Source reading** of `hsmc/src/lib.rs`, `hsmc/src/journal.rs`, and `hsmc-macros/src/codegen.rs`. 2. **`cargo expand --release`** of two test programs (one with no observation features, one with `tokio + journal`) to confirm what the macro really emits. 3. **`cargo rustc --release --emit=asm`** spot-checks of the compiled output to confirm LLVM-level claims (jump-table lowering, dead-code elimination of unused name tables, async-closure code size). The audit deliberately stops short of recommending mitigations. A ranked menu of candidates is included at the end so a follow-up planning round can pick from it; nothing here is a commitment. --- ## 1. Cost surface, ranked by per-event impact | # | Source | Class | Per-event cost (estimate) | Location | |----|-----------------------------------------------|-----------------------------|------------------------------------------------------------------------------------------------------------|---------------------------------------| | 1 | `dispatch().await` loop | Discretionary, async-only | 2 awaits cold-start / 1 await steady-state per dispatched event. Each is ~free if action body never `Pending`-yields. | `codegen.rs:2330-2345` | | 2 | LCA computation for lateral transitions | Discretionary | ~~O(H²) `heapless::Vec::contains` linear scan, H ≤ 32 hard limit~~. **Now O(H) via depth-walk over the `__DEPTH` table. Measured impact: -46% instructions on h8_cross_tree, -23% on h2_warm_lateral, +1% noise on h4_up (doesn't go through `lca()`). See `hsmc/benches/baseline.txt` and §6 in mitigation table.** | `codegen.rs:2601-2628` | | 3 | Entry-path `heapless::Vec` build | Discretionary | One stack `heapless::Vec` filled per transition; H pushes + H reads. | `codegen.rs:2586-2598` | | 4 | Parent walk for handler search | Intrinsic | O(H) `__PARENT[s]` indexing until first matching state. | `codegen.rs:2283-2297` | | 5 | `__handler_lookup` nested match | Intrinsic | LLVM lowers to jump tables (confirmed via `LJTI*` labels in spot-check). ~1 indirect jump per call. | `codegen.rs` (search "__handler_lookup") | | 6 | `run_action` dispatch | Intrinsic | One match arm + monomorphized trait call + 1 vtable hop on `&mut dyn QueuePush` (the `EmitProxy`). | `codegen.rs:2425-2489` | | 7 | Event queue push/pop | Intrinsic | `heapless::Deque::push_back` / `pop_front`, both O(1). | `codegen.rs:2316-2319` | | 8 | `__chart_observe!` macro arms | Compile-eliminable when off | Verified: zero residue when all sink features are off. Each call site collapses to `{};`. | `lib.rs:158-712` | | 9 | Journal `Vec::push` per atom | Conditional on `journal` | Amortized O(1); ~16 B per `TraceEvent`. Expanded chart shows 104 push call sites; ~11 fire per transition. | `journal.rs:26-32`, `lib.rs:158-712` | | 10 | Async-fn future size (binary cost) | Discretionary, async-only | `step` closure ~970 lines of asm; `run_handlers` ~830; total Lat-chart machinery ≈ 3000 asm lines. | spot-check measurement | | 11 | `tokio::sync::mpsc::unbounded_channel` | Tokio-only, std-bound | Heap-backed channel for cross-task event injection. Embassy uses bounded `embassy_sync::channel` instead. | runtime adapters | The top three discretionary items (1, 2, 3) account for almost all the optimization headroom. Items 4–7 are intrinsic to the chosen semantics — they're already at or near the algorithmic floor for hierarchical state machines. Item 8 is verified zero-cost. Item 9 is opt-in. Items 10 and 11 matter for code size and feature ergonomics but not per-event µs. --- ## 2. Per-event cost model (analytical) For a chart with hierarchy depth `H=3`, one entry action + one exit action, a simple lateral transition, no observation features, `cargo build --release`, on a Cortex-M4-class core (~168 MHz): | Stage | Cycles (rough) | Notes | |----------------------------------------|--------------------|------------------------------------------------| | `queue.pop_front` | ~10 | `heapless::Deque` index op | | `__event_to_id` | ~5 | match arm | | Parent walk for handler | ~30 × H | ≤ H array indexes | | `__handler_lookup` | ~10 | jump table indirect | | `run_action` (1 action) | ~20-30 | match + monomorphized call + 1 vtable hop | | `is_up_transition` | ~30 | parent walk | | `lca` (lateral) | ~10 × H | depth-walk via `__DEPTH` (was ~50 × H² pre-fix)| | Exit chain (H states × A actions) | ~60 × H × A | | | Entry path build | ~30 × H | heapless::Vec push | | Enter chain (H states × A actions) | ~60 × H × A | | | **Total `.step()` steady-state** | ~1.0 – 2.0 µs | for H=3, A=1; user action body excluded | | Async overhead (`.dispatch()` loop) | per-await checkpoint | ~free if no `Pending`-yield | These are back-of-envelope numbers; actual measurements would tighten them by ~2× in either direction, depending on inlining decisions. For empirical numbers (deterministic, instruction-counted), see `hsmc/benches/baseline.txt`. Headline numbers post-LCA-fix: h2_warm_lateral 210 instr / ~1k cycles; h4_cross_tree 394 instr / ~1.4k cycles; h8_cross_tree 508 instr / ~1.6k cycles; chain_self_emit 1036 instr across 4 events (≈260 instr/event). Mitigations H1/H2 in §9 target the remaining LCA + entry-path stack-vec overhead. --- ## 3. Spot-check evidence ### 3.1 `__chart_observe!` is zero-cost when off (cargo expand) Test: `tests/state_chart_const.rs`, with no observation features enabled. In the expanded source (`/tmp/hsmc_expand_no_features.rs`, 911 lines), every observation site collapses to an empty block: ```rust // in `step()`: if self.current.is_none() { {}; // would be observe_started self.enter_path(0); return self.timers.min_remaining(); } // ... if let Some((state, trig)) = self.timers.pop_expired(&__DEPTH) { let __t_state = state; let __t_trigger = trig; {}; // would be observe_timer_fired // ... } ``` A grep for any of `TraceEvent::`, `::log::`, `::tracing::`, `::defmt::`, `JournalSink::push` in the expanded code returns **zero matches**. The macro arms entirely vanish. `JournalSink` itself becomes a unit struct when the feature is off: ```rust #[cfg(not(feature = "journal"))] pub struct JournalSink; // ZST — 0 bytes in the chart struct ``` so the `__journal` field on the chart costs zero bytes at runtime. ### 3.2 Name lookups are dead-code-eliminated (cargo rustc --emit=asm) Test: `tests/det_lateral.rs`, built with `--features tokio,journal`. Even with `journal` enabled, a grep across the 22,477 lines of generated asm for `state_name`, `action_name`, `event_name`, or `timer_name` returns **zero matches**. The journal stores `u16` indices, not strings, and no trace-* feature is on, so the name-lookup helper functions are uncalled and DCE'd from the final binary. This means even if a user enables `journal` for replay/verification, they pay zero binary cost for the human-readable name tables until they also enable a `trace-*` sink. ### 3.3 LLVM lowers handler matches to jump tables Same asm file. Grep for `LJTI`: ```asm leaq .LJTI2_0(%rip), %rcx jmpq *%rax .LJTI2_0: .long .LBB2_1-.LJTI2_0 .long .LBB2_28-.LJTI2_0 .long .LBB2_29-.LJTI2_0 .long .LBB2_23-.LJTI2_0 ``` Multiple jump tables are present, each sized roughly to the chart's state/event count. This corroborates the assumption that `__handler_lookup`'s nested `match (state_id) { match (kind, trigger_id) }` lowers to a single indirect jump on platforms LLVM can target with one, not a chain of compares. ### 3.4 Async machinery code size Symbol-by-symbol asm sizes from `det_lateral`: | Symbol (async closure / future) | Asm lines | Notes | |---------------------------------|-----------|----------------------------------------------| | `Lat::exit_state` | ~275 | one `.await` per exit action | | `Lat::run_action` | ~349 | match arm + N monomorphized async branches | | `Lat::run_handlers` | ~826 | loop over actions, each `.await` | | `Lat::enter_state_no_descent` | ~217 | | | `Lat::step` | ~970 | inlines `lca`, `is_up_transition`, etc. | | `Lat::dispatch` | ~137 | thin loop wrapper | | **Lat-chart total** | **~2,800**| async machinery for one 5-state chart | For comparison, `Lat::new` is 141 lines. The async coloring roughly doubles to triples the binary footprint of a chart relative to a pure sync codegen. On Cortex-M0/M0+ where flash is small, this matters. ### 3.5 Journal expansion adds N pushes, nothing else (cargo expand) Same test, expanded with `--features tokio,journal` (`/tmp/hsmc_expand_with_journal.rs`, 2,426 lines vs. 911 lines without). Counts: - 104 total `TraceEvent::*` constructions. - 20 distinct variant names (the full vocabulary). - 8 `Vec::push` call sites inside `step()` — but only one fires per invocation (timer-fired path, event-received path, etc., are mutually exclusive). The diff against the no-feature expansion is exactly: every `{};` placeholder becomes a `(&mut self.__journal).push(TraceEvent::...)`. No other infrastructure is added. No locking, no allocation per push beyond the amortized `Vec` growth. --- ## 4. Observability cost matrix Distilled from `lib.rs:158-712` and the spot-checks above: | Feature(s) | Per-call CPU | Per-call alloc | Off → fully elided? | |-------------------|-------------------------------------------|--------------------------|-------------------------------| | `(none)` | 0 | 0 | Yes (verified) | | `journal` | `Vec::push` ~10 ns amortized | ~16-24 B amortized | n/a | | `trace-defmt` | format-hash u32 + field write | 0 | Yes | | `trace-log` | `log_enabled!()` check + format if passes | 0 unless format | Yes (with small residual when on but level filtered) | | `trace-tracing` | unconditional field collection + span | 0 | Yes (with ~30-cycle residual when on but no subscriber) | The codegen emits **~11 `__chart_observe!` calls per typical transition** in deterministic order: `EventReceived` → `EventDelivered` → `TransitionFired` → `ExitBegan` → `ActionInvoked{Exit}` → `Exited` → `EnterBegan` → `ActionInvoked{Entry}` → `Entered` → `TransitionComplete`, plus action-handler markers. That's the multiplier on every per-call number above. --- ## 5. What is *not* a problem (audited and dismissed) - **Handler-lookup growth with chart size** — confirmed jump-table lowering (§3.3). O(1) per event regardless of state or event count (modulo cache effects from larger jump tables). - **Trait-object cost on action dispatch** — only the `&mut dyn QueuePush` proxy is `dyn`. User actions are monomorphized trait methods (one match arm per action ID, direct calls). One vtable hop per emit, *not* per action. - **Observation overhead with features off** — confirmed zero. Verified three ways: macro arm structure (every body `#[cfg]`-gated), expanded source (every site is `{};`), and asm (no name-lookup symbols, no trace symbols). - **Heap allocation in default build** — `extern crate alloc` is confined to `journal.rs:26`. With `journal` off, the runtime is fully stack-only. Verified: no `Vec`, `Box`, or `String` in expanded sync code. - **`Pin>` futures on the hot path** — none. Chart futures are inline state machines, not boxed. - **Tokio `unbounded_channel` for embedded** — irrelevant: tokio is `std` anyway. Embassy uses bounded `embassy_sync::channel::Channel` (no alloc). --- ## 6. Event entry points Events reach the chart's internal `heapless::Deque` via four distinct paths. Each has its own latency, wakeup, ordering, and backpressure profile, summarized below. | Path | API | Backing | Wakeup | Ordering | Backpressure | |------|-------------------------------------------|--------------------------------------------|-------------------------|----------------------------------------|------------------------------------| | 1 | `chart.send(event)` (sync, single-task) | `heapless::Deque::push_back` | none — caller drives | FIFO within queue | `Err(QueueFull)` immediate | | 2a | `chart.sender().send(ev)` (tokio) | `tokio::sync::mpsc::unbounded_channel` | tokio task wake | FIFO; biased select deprioritizes vs. durings | **unbounded memory** | | 2b | `chart.sender().send(ev)` (embassy) | `embassy_sync::channel::Channel` (bounded) | embassy task wake | FIFO; same biased-select | `Err(Full)` (try_send) or backpressure-await | | 3 | `actx.emit(ev)` from inside an action | same internal `Deque` as path 1 | none — drained inline | drained synchronously after current handler completes; *before* `step()` returns | sticky overflow flag | | 4 | return from a `during:` future | `select!` arm in run loop | driver-specific | biased priority over channel events | silent drop on `Deque` full | **Path 1 (sync send)** — `codegen.rs:2316-2319`. Bare `Deque::push_back`; owning task only, no synchronization. ~5–10 cycles. Caller is responsible for invoking `step()` (or `dispatch()` under an async feature) to actually process the event. **Path 2 (cross-task `sender()`)** — `codegen.rs:1492` (tokio sender), `1638` (embassy sender), `1308` / `1405` (run-loop receive sites). The canonical cross-thread / ISR path per the spec. Tokio uses an unbounded MPSC (heap-backed; no backpressure signal); embassy uses a bounded `Channel` with `try_send` for ISR-safe non-blocking emits and an `async send` that backpressures the producer. The receiver task wakes on send; one iteration of the run-loop's `select!` consumes the event and pushes it into the internal `Deque`. **Path 3 (self-emit from action)** — `actx.emit(...)` goes through `EmitProxy::push` (`hsmc/src/lib.rs`) which writes into the same internal `Deque` as Path 1. Self-emitted events are drained in the *same* `step()` call that emitted them — `dispatch()` runs `while !queue.is_empty() { step().await }` until quiescence. Nothing crosses a task boundary. **Path 4 (`during:` return)** — `codegen.rs:1306–1313` (tokio with durings), `1392+` (embassy). Durings are racing futures spawned in the run-loop's `select!`. When one resolves to an event, the `select!` unblocks and the event is pushed into the internal `Deque`. The `select!` is `biased;` so durings have priority over the channel — relevant if a during is a poll-based driver that's almost always ready (see §8). **One ordering caveat**: there is no global FIFO across paths 1–4. Within each path, events are FIFO; across paths, the order depends on when each path's wakeup arrives at the run loop. For most charts this is a non-issue, but a chart that mixes high-frequency cross-task sends with rapid self-emits could observe self-emits processed before external events that arrived "first" in wall-clock terms. --- ## 7. Cooperative-driving audit The library targets embassy-rs's single-priority cooperative scheduler, where any tight loop without an `.await` can starve other tasks. An audit of every loop and await pattern in the runtime and codegen: | Loop / await | Location | Bounded? | Awaits per iter? | Verdict | |---------------------------------------|---------------------------|-------------------------|--------------------------|---------------------| | Main `run()` outer | codegen.rs:1290+, 1379+ | by `terminated` flag | yes — always | cooperative | | `dispatch()` drain loop | codegen.rs:2339 | by queue size | only via `step().await` | suspect — fairness | | Transition exit chain | codegen.rs:2532, 2577 | by hierarchy depth | yes (async variant) | OK | | Termination drain | codegen.rs:2750 | by hierarchy + queue | yes (async variant) | OK | | `lca()` ancestry walk | codegen.rs:2605 | by hierarchy depth | no (sync helper) | OK | | Timer decrement / pop | runtime / lib.rs | by timer-table size | sync, off the event-loop | OK | | Send/recv backends | codegen.rs:1308, 1405 | n/a | always `.await` | cooperative | No `try_recv` patterns anywhere. No fixed-period polling. The library is fundamentally event-driven and embassy-friendly. **One latent fairness characteristic**: the `dispatch()` drain loop — `while !self.queue.is_empty() { step(Duration::ZERO).await }` — never yields to the executor between iterations. If user actions are all synchronous and they emit a chain of follow-on events (action A emits B, B emits C, …), each `step().await` completes inline, and the loop becomes effectively synchronous for the duration of the chain. Embassy high-priority tasks could be delayed by the chain length. This is **not a bug** — `dispatch()` is documented to drain to quiescence — but it's worth knowing about. A future option (not part of this round of work) would be a feature-gated `embassy_futures::yield_now().await` every Nth iteration, or a chart-level config to bound chain length. Not done by default because it would serialize the common case for no benefit. --- ## 8. Correctness invariants surfaced during the audit Three contracts that aren't bugs, but warrant explicit documentation because they are easy to miss when reading the code: **Sticky overflow flag in `EmitProxy`** (`hsmc/src/lib.rs`, `EmitProxy::push`). When a self-emit fails because the internal `Deque` is full, an `__overflow` flag is set on the chart. *Every* subsequent `actx.emit(...)` in the same dispatch chain returns `Err(QueueFull)`, even after the queue has drained. The flag is cleared at the top of each `dispatch()` call by `take_overflow()`. This is intentional — it prevents a chart from observing a partially-applied sequence of self-emissions where some made it into the queue and others did not. Authors of actions that emit multiple events should treat `Err(QueueFull)` as terminal for the current chain. **Biased select in run loop** (tokio: `codegen.rs:1306–1313`; embassy: `1392+`). The `select!` is `biased;`, so durings are always polled before the cross-task channel. For most charts this is the right priority — a sensor driver should pre-empt a UI event. But for a `during:` that wraps a poll-based driver (one that returns `Ready` nearly every poll), the channel can be starved indefinitely. Cure: make the during yield (`embassy_futures::yield_now`) when there's nothing real to report. **Cancel-safety contract on durings** (`hsmc/src/lib.rs:53–60`). Every `.await` in a `during:` future must be a clean resume point, because the future will be dropped if the state transitions before it completes. This is the standard async-cancel-safety contract, but it is easy to violate by holding a mutable borrow across an await — the borrow is "valid" when resumed but the surrounding state may have moved on. --- ## 9. Candidate mitigations (menu, NOT a commitment) Sized for sequencing in a follow-up plan-mode session. "Effort" is gut feel: L = afternoon, M = couple of days, H = a week or more. | ID | Mitigation | Targets | Impact | Effort | Cost | |-----|-----------------------------------------------------------------------------------------------------------------------------------|-----------|-----------------------------------------|--------|---------------------------------------| | H0 | ~~Replace `lca()` with an O(H) depth-walk using the existing `__DEPTH` table; remove the `heapless::Vec` set entirely.~~ **Landed.** Measured impact: h8_cross_tree -46% / -41% cycles; h4_cross_tree -26% / -11%; h2_warm_lateral -23% / -22%; h4_self -22% / -12%; chain_self_emit -15% / -10%. See `hsmc/benches/baseline.txt`. | #2 | (delivered) | - | - | | H1 | Codegen per-transition exit/entry/descend chains as `&'static [u16]` arrays; eliminate runtime `heapless::Vec` build entirely. | #2, #3 | High — kills LCA *and* heapless allocs | M | ~10–30 B per transition; bounded | | H2 | Same idea but precompute LCA only (small static table or per-transition LCA u16 field). | #2 | Med-High | L | ~2 B per transition | | H3 | Provide a `dispatch_sync()` API that bypasses `.await` when the chart's actions are fully synchronous. | #1, #10 | High for sync-action charts | L-M | API surface growth | | H4 | Codegen a per-state inlined event match (skip the parent walk by emitting `(state, event) → (handler_state, actions, target)` directly per state). | #4 | Low-Med | M | binary grows ~`states × events` | | M1 | Eliminate the trailing `step().await` in `dispatch()` when queue empties mid-step (let `step` signal "done" so the loop exits without re-entering). | #1 | Low | L | minor refactor | | M2 | `#[inline]` on `__handler_lookup`, `__state_name`, etc. for tighter cross-call inlining. | #5, #6 | Low (LLVM probably already does this) | L | none | | M3 | Lazy field eval in `trace-tracing` arm using `tracing`'s field-closure form to skip evaluation when filtered. | feature on| Low | L | none | | L1 | Audit residual `_t_state`, `_t_trigger`, `__handler_state` local bindings in observation-off builds — verify LLVM elides them. | feature off| Negligible | L | none | H1 is the clear top candidate: it collapses the two largest discretionary costs (#2, #3) into one static-table lookup, removes one runtime stack allocation per transition, and is implementable purely as a codegen change in `hsmc-macros/src/codegen.rs` without touching the runtime contract. --- ## 10. Where to look in the source For anyone reading this audit and wanting to verify or extend it: - **`hsmc/src/lib.rs`** - `:94` — the `#![cfg_attr(... no_std)]` boundary - `:158-712` — `__chart_observe!` macro, all observation arms - `:737-772` — `JournalSink` ZST/Vec duality - **`hsmc/src/journal.rs`** - `:26-32` — `extern crate alloc` and `Vec` import (only here) - `:65-150+` — `TraceEvent` enum (the journal vocabulary) - **`hsmc-macros/src/codegen.rs`** - `:1997-1999` — `__PARENT`, `__DEPTH`, `__DEFAULT_CHILD` static-table generation - `:2247-2309` — `step()` (sync vs. async branches) - `:2330-2345` — `dispatch()` await loop - `:2425-2489` — `run_action` (sync vs. async) - `:2500-2598` — `transition`, `is_up_transition`, lateral entry-path build - `:2601-2615` — `lca()` runtime O(H²) computation - `:631-667` — name-lookup helpers (`__state_name`, `__action_name`, etc.) - **Test reference shape**: `hsmc/tests/state_chart_const.rs` is the smallest sync-only chart in the tree; `hsmc/tests/det_lateral.rs` is the reference async + journal shape used for the asm spot-check. --- ## 11. Reproducing the spot-checks ```sh # Zero-cost observation verification (expand → no observation residue) cargo expand --release --test state_chart_const \ > /tmp/expand_no_features.rs --manifest-path hsmc/Cargo.toml grep -E "TraceEvent::|::log::|::tracing::|::defmt::" /tmp/expand_no_features.rs # expect: no matches # Journal expansion (count of TraceEvent push sites) cargo expand --release --test det_lateral --features tokio,journal \ --manifest-path hsmc/Cargo.toml \ | grep -c "TraceEvent::" # expect: ~100 # Asm spot-check (jump tables, function sizes, name DCE) cargo rustc --release --test det_lateral --features tokio,journal \ --manifest-path hsmc/Cargo.toml -- --emit=asm ASM=$(find target -name 'det_lateral*.s' | head -1) grep -E "LJTI|jmpq?[[:space:]]+\*" "$ASM" # jump tables grep -E "state_name|action_name" "$ASM" # expect no matches ``` --- ## 12. Summary `hsmc`'s hot path is mostly already at the algorithmic floor for hierarchical state machines. The intrinsic per-event cost (parent walk, handler lookup, action dispatch, queue ops) is small and well-monomorphized. Observation cost is verified zero when features are off, and stays proportional to the user's chosen sink mix when on. **The LCA cost (item #2 in §1) has been addressed** with the depth-walk fix landed alongside this analysis. Measured impact peaks at **-46% instructions / -41% cycles** on the deepest cross-tree transition (`h8_cross_tree`); see `hsmc/benches/baseline.txt`. The former O(H²) scan and the `heapless::Vec` ancestry-set allocation are gone; the 32-state hierarchy ceiling is also gone. The two remaining optimization targets are: 1. **Entry-path `heapless::Vec` build** (item #3 in §1) — still allocates one stack `heapless::Vec` per lateral transition. Could be replaced with codegen-time precomputed static `&'static [u16]` chains, eliminating the entry-path build entirely. (See mitigation H1.) 2. **Async coloring code-size cost** (item #10) — chart machinery is ~3,000 asm lines for a 5-state chart, dominated by async-fn state machines. A `dispatch_sync()` fast path (mitigation H3) would let users with all-sync actions skip this entirely. Both are addressable purely in `hsmc-macros/src/codegen.rs` without breaking runtime semantics or feature gates. The bench harness in `hsmc/benches/dispatch.rs` is in place to measure each mitigation as it lands.