71 Commits
Author SHA1 Message Date
swaits 95891c2403 chore: repoint remaining links and host-specific tooling at git.swaits.com
CI / Lint () (push) Canceled after 0s
CI / Lint (--all-features) (push) Canceled after 0s
CI / Lint (--no-default-features) (push) Canceled after 0s
CI / MSRV Verification () (push) Canceled after 0s
CI / MSRV Verification (--all-features) (push) Canceled after 0s
CI / MSRV Verification (--no-default-features) (push) Canceled after 0s
CI / Test (macos-latest, nightly, ) (push) Canceled after 0s
CI / Test (macos-latest, stable, ) (push) Canceled after 0s
CI / Test (ubuntu-latest, nightly, ) (push) Canceled after 0s
CI / Test (ubuntu-latest, stable, ) (push) Canceled after 0s
CI / Test (windows-latest, nightly, ) (push) Canceled after 0s
CI / Test (windows-latest, stable, ) (push) Canceled after 0s
CI / Test (macos-latest, nightly, --all-features) (push) Canceled after 0s
CI / Test (macos-latest, stable, --all-features) (push) Canceled after 0s
CI / Test (ubuntu-latest, nightly, --all-features) (push) Canceled after 0s
CI / Test (ubuntu-latest, stable, --all-features) (push) Canceled after 0s
CI / Test (windows-latest, nightly, --all-features) (push) Canceled after 0s
CI / Test (windows-latest, stable, --all-features) (push) Canceled after 0s
CI / Test (macos-latest, nightly, --no-default-features) (push) Canceled after 0s
CI / Test (macos-latest, stable, --no-default-features) (push) Canceled after 0s
CI / Test (ubuntu-latest, nightly, --no-default-features) (push) Canceled after 0s
CI / Test (ubuntu-latest, stable, --no-default-features) (push) Canceled after 0s
CI / Test (windows-latest, nightly, --no-default-features) (push) Canceled after 0s
CI / Test (windows-latest, stable, --no-default-features) (push) Canceled after 0s
CI / Cargo Check () (push) Canceled after 0s
CI / Cargo Check (--all-features) (push) Canceled after 0s
CI / Cargo Check (--no-default-features) (push) Canceled after 0s
CI / Examples (push) Canceled after 0s
CI / Code Coverage (push) Canceled after 0s
CI / Security Audit (push) Canceled after 0s
CI / Documentation () (push) Canceled after 0s
CI / Documentation (--all-features) (push) Canceled after 0s
CI / Documentation (--no-default-features) (push) Canceled after 0s
Covers files missed by the first pass (.svelte/.astro components, bare org and
profile links, plugin doc headers), plus content that needed rewriting rather
than a URL swap: the chezmoi bootstrap script now provisions SSH for
git.swaits.com (User git, port 2222) and initialises from Gitea.

Link labels were updated to match their new targets.
2026-08-31 18:35:21 -06:00
swaits 2a9561a32a chore: repoint URLs at git.swaits.com and bump for release
CI / Lint () (push) Canceled after 0s
CI / Lint (--all-features) (push) Canceled after 0s
CI / Lint (--no-default-features) (push) Canceled after 0s
CI / MSRV Verification () (push) Canceled after 0s
CI / MSRV Verification (--all-features) (push) Canceled after 0s
CI / MSRV Verification (--no-default-features) (push) Canceled after 0s
CI / Test (macos-latest, nightly, ) (push) Canceled after 0s
CI / Test (macos-latest, stable, ) (push) Canceled after 0s
CI / Test (ubuntu-latest, nightly, ) (push) Canceled after 0s
CI / Test (ubuntu-latest, stable, ) (push) Canceled after 0s
CI / Test (windows-latest, nightly, ) (push) Canceled after 0s
CI / Test (windows-latest, stable, ) (push) Canceled after 0s
CI / Test (macos-latest, nightly, --all-features) (push) Canceled after 0s
CI / Test (macos-latest, stable, --all-features) (push) Canceled after 0s
CI / Test (ubuntu-latest, nightly, --all-features) (push) Canceled after 0s
CI / Test (ubuntu-latest, stable, --all-features) (push) Canceled after 0s
CI / Test (windows-latest, nightly, --all-features) (push) Canceled after 0s
CI / Test (windows-latest, stable, --all-features) (push) Canceled after 0s
CI / Test (macos-latest, nightly, --no-default-features) (push) Canceled after 0s
CI / Test (macos-latest, stable, --no-default-features) (push) Canceled after 0s
CI / Test (ubuntu-latest, nightly, --no-default-features) (push) Canceled after 0s
CI / Test (ubuntu-latest, stable, --no-default-features) (push) Canceled after 0s
CI / Test (windows-latest, nightly, --no-default-features) (push) Canceled after 0s
CI / Test (windows-latest, stable, --no-default-features) (push) Canceled after 0s
CI / Cargo Check () (push) Canceled after 0s
CI / Cargo Check (--all-features) (push) Canceled after 0s
CI / Cargo Check (--no-default-features) (push) Canceled after 0s
CI / Examples (push) Canceled after 0s
CI / Code Coverage (push) Canceled after 0s
CI / Security Audit (push) Canceled after 0s
CI / Documentation () (push) Canceled after 0s
CI / Documentation (--all-features) (push) Canceled after 0s
CI / Documentation (--no-default-features) (push) Canceled after 0s
Rewrites remaining github.com / git.sr.ht references to git.swaits.com,
translating GitHub URL shapes to Gitea's (/blob/<ref>/ -> /src/branch/<ref>/,
/tree/<tag> -> /src/tag/<tag>, /actions/workflows/X -> /actions?workflow=X,
/edit/ -> /_edit/).

Left pointing at GitHub on purpose: pull-request links (PRs were not migrated)
and GitHub-only surfaces Gitea lacks (security advisories, discussions).
2026-08-31 17:58:30 -06:00
swaits f984ec0a29 chore: point repository at git.swaits.com
CI / Lint () (push) Canceled after 0s
CI / Lint (--all-features) (push) Canceled after 0s
CI / Lint (--no-default-features) (push) Canceled after 0s
CI / MSRV Verification () (push) Canceled after 0s
CI / MSRV Verification (--all-features) (push) Canceled after 0s
CI / MSRV Verification (--no-default-features) (push) Canceled after 0s
CI / Test (macos-latest, nightly, ) (push) Canceled after 0s
CI / Test (macos-latest, stable, ) (push) Canceled after 0s
CI / Test (ubuntu-latest, nightly, ) (push) Canceled after 0s
CI / Test (ubuntu-latest, stable, ) (push) Canceled after 0s
CI / Test (windows-latest, nightly, ) (push) Canceled after 0s
CI / Test (windows-latest, stable, ) (push) Canceled after 0s
CI / Test (macos-latest, nightly, --all-features) (push) Canceled after 0s
CI / Test (macos-latest, stable, --all-features) (push) Canceled after 0s
CI / Test (ubuntu-latest, nightly, --all-features) (push) Canceled after 0s
CI / Test (ubuntu-latest, stable, --all-features) (push) Canceled after 0s
CI / Test (windows-latest, nightly, --all-features) (push) Canceled after 0s
CI / Test (windows-latest, stable, --all-features) (push) Canceled after 0s
CI / Test (macos-latest, nightly, --no-default-features) (push) Canceled after 0s
CI / Test (macos-latest, stable, --no-default-features) (push) Canceled after 0s
CI / Test (ubuntu-latest, nightly, --no-default-features) (push) Canceled after 0s
CI / Test (ubuntu-latest, stable, --no-default-features) (push) Canceled after 0s
CI / Test (windows-latest, nightly, --no-default-features) (push) Canceled after 0s
CI / Test (windows-latest, stable, --no-default-features) (push) Canceled after 0s
CI / Cargo Check () (push) Canceled after 0s
CI / Cargo Check (--all-features) (push) Canceled after 0s
CI / Cargo Check (--no-default-features) (push) Canceled after 0s
CI / Examples (push) Canceled after 0s
CI / Code Coverage (push) Canceled after 0s
CI / Security Audit (push) Canceled after 0s
CI / Documentation () (push) Canceled after 0s
CI / Documentation (--all-features) (push) Canceled after 0s
CI / Documentation (--no-default-features) (push) Canceled after 0s
2026-08-31 17:39:43 -06:00
TechnoPorg dfb794361e Merge pull request #14 from TechnoPorg/push-mzlqmxmonmyt
CI / Lint () (push) Canceled after 0s
CI / Lint (--all-features) (push) Canceled after 0s
CI / Lint (--no-default-features) (push) Canceled after 0s
CI / MSRV Verification () (push) Canceled after 0s
CI / MSRV Verification (--all-features) (push) Canceled after 0s
CI / MSRV Verification (--no-default-features) (push) Canceled after 0s
CI / Test (macos-latest, nightly, ) (push) Canceled after 0s
CI / Test (macos-latest, stable, ) (push) Canceled after 0s
CI / Test (ubuntu-latest, nightly, ) (push) Canceled after 0s
CI / Test (ubuntu-latest, stable, ) (push) Canceled after 0s
CI / Test (windows-latest, nightly, ) (push) Canceled after 0s
CI / Test (windows-latest, stable, ) (push) Canceled after 0s
CI / Test (macos-latest, nightly, --all-features) (push) Canceled after 0s
CI / Test (macos-latest, stable, --all-features) (push) Canceled after 0s
CI / Test (ubuntu-latest, nightly, --all-features) (push) Canceled after 0s
CI / Test (ubuntu-latest, stable, --all-features) (push) Canceled after 0s
CI / Test (windows-latest, nightly, --all-features) (push) Canceled after 0s
CI / Test (windows-latest, stable, --all-features) (push) Canceled after 0s
CI / Test (macos-latest, nightly, --no-default-features) (push) Canceled after 0s
CI / Test (macos-latest, stable, --no-default-features) (push) Canceled after 0s
CI / Test (ubuntu-latest, nightly, --no-default-features) (push) Canceled after 0s
CI / Test (ubuntu-latest, stable, --no-default-features) (push) Canceled after 0s
CI / Test (windows-latest, nightly, --no-default-features) (push) Canceled after 0s
CI / Test (windows-latest, stable, --no-default-features) (push) Canceled after 0s
CI / Cargo Check () (push) Canceled after 0s
CI / Cargo Check (--all-features) (push) Canceled after 0s
CI / Cargo Check (--no-default-features) (push) Canceled after 0s
CI / Examples (push) Canceled after 0s
CI / Code Coverage (push) Canceled after 0s
CI / Security Audit (push) Canceled after 0s
CI / Documentation () (push) Canceled after 0s
CI / Documentation (--all-features) (push) Canceled after 0s
CI / Documentation (--no-default-features) (push) Canceled after 0s
Remove Clone/Copy from TaskHandle
2026-02-20 19:12:49 +01:00
TechnoPorg 9fbe4c2fd8 fix: change where TaskHandle is borrowed to prevent repeat output fetches 2026-02-20 19:00:18 +01:00
TechnoPorg c8efb00175 Merge pull request #13 from TechnoPorg/push-qlytvyqokrzl
Cleanup and documentation updates before release
2026-02-19 21:24:20 +01:00
TechnoPorg 766555bef4 doc: update for next release 2026-02-19 21:14:03 +01:00
TechnoPorg a13f148778 deps: update lockfile 2026-02-19 20:12:51 +01:00
TechnoPorg 827041c1b6 chore: clean up in preparation for release 2026-02-18 22:34:31 +01:00
TechnoPorg c002161216 Merge pull request #12 from TechnoPorg/push-wummzrynoysn
Make DagRunner construction single-threaded
2026-02-17 21:33:41 +01:00
TechnoPorg 72a83e29dc test: update tests/examples/benches 2026-02-17 21:31:18 +01:00
TechnoPorg e4fc0b0d44 refactor: make DagRunner interface single-threaded 2026-02-17 21:29:14 +01:00
TechnoPorg d83a815ed0 Fix CI 2026-02-13 00:28:23 +01:00
TechnoPorg 3e02d6b679 Merge pull request #11 from TechnoPorg/push-zvntumwxvoru
test: remove/consolidate redundant tests, examples, benchmarks, get tarpaulin coverage back to 80
2026-02-13 00:23:15 +01:00
TechnoPorg a6b8ce4a8f test: bring tarpaulin coverage back up 2026-02-13 00:20:59 +01:00
TechnoPorg fe8de2784b refactor: move task_fn into its own crate 2026-02-12 23:31:24 +01:00
TechnoPorg b8204c4ed9 refactor: consolidate some tests/examples/benchmarks 2026-02-12 23:31:19 +01:00
TechnoPorg 49be497964 perf: don't double-box task outputs 2026-02-12 20:49:30 +01:00
TechnoPorg b48b3a5a2e Merge pull request #10 from TechnoPorg/push-lytsnzwpmxmy
Speed up CI by installing dependencies as binaries
2026-02-11 23:33:33 +01:00
TechnoPorg 711cebdd6e Speed up CI by installing dependencies as binaries 2026-02-11 23:28:13 +01:00
TechnoPorg 5bfb284c82 Merge pull request #9 from TechnoPorg/push-kplnqxpsroms
perf: use passthrough hashers instead of standard library default
2026-02-11 23:15:27 +01:00
TechnoPorg c02740798b perf: use passthrough hasher and preallocate capacity for outputs 2026-02-11 23:07:23 +01:00
TechnoPorg b6ff1f81e6 chore: fix formatting for CI 2026-02-11 23:07:23 +01:00
TechnoPorg 7999996bca Merge pull request #8 from TechnoPorg/push-ptxmpolotoqo
Depend directly futures-util instead of futures
2026-02-11 22:50:16 +01:00
TechnoPorg a75defa5a6 Depend on futures-util instead of futures 2026-02-11 22:44:01 +01:00
TechnoPorg f342e33f16 Merge pull request #7 from TechnoPorg/push-nmmvpvqkpsmr
Remove clone bound from task data
2026-02-11 22:35:23 +01:00
TechnoPorg 4ef7e69cdd Don't clone task inputs 2026-02-11 22:29:54 +01:00
TechnoPorg 6f7a58e4c5 Fix CI 2026-02-11 22:29:54 +01:00
TechnoPorg 86787b3a0d Update tests, examples, benches 2026-02-11 22:01:50 +01:00
TechnoPorg 0137296fb4 Remove clone bound from task data 2026-02-11 21:59:41 +01:00
TechnoPorg 064c2a15be Merge pull request #3 from TechnoPorg/push-mqpxvoyoutro
Address soundness hole in cycle prevention
2026-02-06 20:38:12 +01:00
TechnoPorg 37260f6b3a Temporarily disable code coverage CI checks
This will be re-enabled once I'm done the last few major changes I have in mind. Until then, I've already caused this check to fail on main, so this will reduce GitHub email spam for a known issue.
2026-02-06 20:36:37 +01:00
TechnoPorg 37e4d46bac Temporarily disable code coverage CI checks
This will be re-enabled once I'm done the last few major changes I have in mind.
Until then, I've already caused this check to fail on main, so this will reduce GitHub email spam for a known issue.
2026-02-06 20:29:19 +01:00
TechnoPorg f82205baf4 Address soundness hole in cycle prevention 2026-01-29 06:56:17 +01:00
TechnoPorg 4ad88bcd68 feat: Don't require an explicit return type in the task macro 2026-01-29 06:50:12 +01:00
TechnoPorg 1c9034de71 Don't require an explicit return type in the task macro 2026-01-28 08:52:52 +01:00
swaits 8f6472ea22 Merge pull request #1 from TechnoPorg/push-mmkvnrvwpuzm
Replace channel mechanism of dependency passing with a shared registry
2026-01-27 23:23:53 -07:00
TechnoPorg 098b2711d8 Don't use channels to send dependencies 2026-01-26 20:58:06 +01:00
TechnoPorg a8e39804d2 Test single-threaded runtime 2026-01-22 00:34:52 +01:00
TechnoPorg ee3c9f4417 Update non-default feature code 2026-01-22 00:30:48 +01:00
TechnoPorg 85ccd7e507 Update tests, examples, benches 2026-01-22 00:30:48 +01:00
TechnoPorg bacac903b3 Reimplement spawner function 2026-01-22 00:30:48 +01:00
TechnoPorg d7806043e1 Use single-threaded hashmap instead of dashmap
Address performance regressions
2026-01-22 00:30:48 +01:00
TechnoPorg f4160d2387 Update docs 2026-01-22 00:30:48 +01:00
TechnoPorg df39dd80b9 Don't use channels to send dependencies 2026-01-19 00:24:32 +01:00
swaits 8e507d76f3 chore: bump crate to 0.3.1, update CHANGELOG 2025-10-10 17:39:23 -06:00
swaits 9ad3bd174f chore: bump crate to 0.3.1, update CHANGELOG 2025-10-10 17:31:28 -06:00
swaits 979232bd67 docs: document custom type support throughout project
Update all documentation to reflect that custom types work automatically
with the #[task] macro, requiring only Clone + Send + Sync + 'static.

Documentation updates:
- README: Add custom types to features, new Custom Types section
- CHANGELOG: Document custom type improvement in [Unreleased]
- src/lib.rs: Add custom types example to main API docs
- src/task.rs: Update Task trait docs to mention macro-generated extraction
- src/extract.rs: Clarify ExtractInput is now internal-only (for task_fn)
- dagx-macros: Update macro docs to emphasize automatic type handling

New example:
- examples/custom_types.rs: Simple 120-line example showing User type
  flowing through DAG with fan-out pattern (one output → two consumers)
  demonstrating Arc wrapping is automatic and transparent

This makes the custom type capability discoverable and prominent for users,
turning a previous limitation into a headline feature.
2025-10-10 17:22:59 -06:00
swaits 8ee4092f9a test: increase coverage for error paths and conversions
Add targeted coverage tests to maintain >80% code coverage minimum:

- ExtractInput error paths (14 tests): Type mismatch errors for Result,
  Option, Vec, HashMap, Arc, and primitive types. Directly exercises
  error formatting code that was previously uncovered.

- DagRunner::get() type mismatch (6 tests): Covers the type mismatch
  error path when attempting to retrieve task output with wrong type.
  Tests various type combinations to ensure robust error handling.

- TaskBuilder conversions (9 tests): Covers the From<TaskBuilder> to
  TaskHandle conversion for unit-type tasks, ensuring ID preservation
  and correct handling of mixed task types.

Also fixes timing test threshold from 10ms to 50ms to account for
tarpaulin instrumentation overhead while still validating parallel
execution.

Coverage increased from ~78% to 82.04% (above 80% minimum requirement).
2025-10-10 17:06:59 -06:00
swaits fad97dbf16 fix: support arbitrary types in task dependencies
Enable tasks to accept ANY type implementing Clone + Send + Sync as
dependencies, without requiring users to implement ExtractInput trait.

The #[task] macro now generates task-specific extraction logic inline
via a new extract_and_run() method. This leverages the macro's perfect
compile-time knowledge of parameter types to handle extraction for:
- Zero parameters (unit type, source tasks)
- Single parameter (any Clone + Send + Sync type)
- Multiple parameters (tuples, awaited concurrently)

Previously, only hardcoded primitive types and specific stdlib types
worked. Custom user types required manual ExtractInput implementations.
Now custom types work automatically with zero API changes.

Changes:
- Add extract_and_run() method to Task trait (internal/hidden)
- Generate inline extraction in #[task] macro based on parameter count
- Remove ExtractInput bound from DagRunner::add_task()
- Retain ExtractInput for internal task_fn() test helper
- Add comprehensive custom type tests

No breaking changes - purely internal implementation improvement.
2025-10-10 17:06:38 -06:00
swaits 82cbf15b85 docs: clean up, split up, restructure README.md 2025-10-08 16:01:27 -06:00
swaits 748e3f65c4 perf: optimization hints (inline, cold, etc) 2025-10-08 15:50:02 -06:00
swaits e3f51308a0 chore: bump crate to 0.3.0, update CHANGELOG 2025-10-08 14:52:45 -06:00
swaits bec2f9e4c9 fix: lint, disable wonky timing test on macos 2025-10-08 14:52:32 -06:00
swaits 802e584646 refactor: remove cycle checking becuase API prevents cycles 2025-10-08 14:28:54 -06:00
swaits 683579a18b test: increase coverage, improve test matrices for CI 2025-10-08 14:23:19 -06:00
swaits 02eac133bd fix: force threads on arc test (fixes MacOS CI failure) 2025-10-08 12:20:04 -07:00
swaits 47620b6702 fix: clippy lint due to tracing-only code 2025-10-08 12:19:27 -07:00
swaits 9290eccd86 docs: add TOC to ridiculously long README 2025-10-07 21:37:10 -07:00
swaits 4e891ebd5a chore: bump crate to 0.2.3, update CHANGELOG 2025-10-07 21:30:05 -07:00
swaits 5524b2121b docs: clean up examples/_parallel_computation.rs 2025-10-07 21:23:27 -07:00
swaits ec94ade329 chore: add logging under tracing feature 2025-10-07 21:16:04 -07:00
swaits ac848c6608 chore: bump crate to 0.2.2, update CHANGELOG 2025-10-07 20:30:19 -07:00
swaits 0a91889022 chore: make release checker verify docs match crate version 2025-10-07 20:24:00 -07:00
swaits ba32767b94 chore: bump crate to 0.2.1, update CHANGELOG 2025-10-07 19:58:19 -07:00
swaits 504347a46a fix: force tokio:test("multi_thread") so tests pass on macOS 2025-10-07 19:53:05 -07:00
swaits aa604e617f chore: bump crate to 0.2.0, update CHANGELOG 2025-10-07 19:39:24 -07:00
swaits 608251618a chore: bump MSRV to 1.81.0 (due to dependencies) 2025-10-07 17:32:44 -07:00
swaits 400c01d73c test: increase coverage to 93% 2025-10-07 17:29:08 -07:00
swaits 98b8bcfd6e chore: remove inappropriate tests from tarpaulin runs 2025-10-07 16:56:35 -07:00
swaits 6b2ca6ba7e chore: fix cargo-audit GH action 2025-10-07 16:51:13 -07:00
128 changed files with 4365 additions and 15566 deletions

No files matched your search

-80
View File
@@ -1,80 +0,0 @@
{
"current_phase": "complete",
"active_task": {
"id": "OWNERSHIP-REFACTOR",
"status": "complete",
"description": "Successfully implemented ownership-based model to eliminate all Mutex/OnceLock from task execution"
},
"completed_tasks": [
{
"id": "TASK-001",
"desc": "Updated Task trait to take ownership (self instead of &mut self)",
"status": "complete"
},
{
"id": "TASK-002",
"desc": "Updated #[task] macro to generate new Task trait signature",
"status": "complete"
},
{
"id": "TASK-003",
"desc": "Removed Mutex from TypedNode - store task directly",
"status": "complete"
},
{
"id": "TASK-004",
"desc": "Updated ExecutableNode to consume self and return output",
"status": "complete"
},
{
"id": "TASK-005",
"desc": "Modified DagRunner to store Option<Box<dyn ExecutableNode>>",
"status": "complete"
},
{
"id": "TASK-006",
"desc": "Updated run() to collect and store outputs from executed nodes",
"status": "complete"
},
{
"id": "TASK-007",
"desc": "Updated get() to retrieve from centralized output storage",
"status": "complete"
},
{
"id": "TASK-008",
"desc": "Comprehensive testing - all 307 tests pass",
"status": "complete"
}
],
"backlog": [],
"next_agent": "code-reviewer",
"context": "Ownership-based refactor complete. Eliminated ALL Mutex/OnceLock from task execution path. All 307 tests pass. Ready for review.",
"technical_decisions": {
"ownership_model": "Task::run(self) takes ownership - IMPLEMENTED",
"execution_model": "ExecutableNode consumes self, returns Box<dyn Any + Send> - IMPLEMENTED",
"storage_model": "DagRunner stores ALL outputs in centralized HashMap - IMPLEMENTED",
"synchronization": "ZERO Mutex/OnceLock in task execution (only in DAG builder pattern) - ACHIEVED",
"api_preservation": "Public API unchanged - macro handles delegation transparently - VERIFIED"
},
"implementation_summary": {
"files_changed": [
"src/task.rs",
"dagx-macros/src/lib.rs",
"src/node.rs",
"src/runner.rs",
"src/task/tests.rs",
"src/node/tests.rs",
"src/runner/tests.rs"
],
"lines_changed": "~350 lines modified",
"test_results": {
"total": 307,
"passed": 307,
"failed": 0,
"ignored": 3
},
"performance_impact": "Eliminated Mutex contention during task execution - should see improved throughput",
"breaking_changes": "None - public API preserved"
}
}
+67 -23
View File
@@ -12,7 +12,10 @@ env:
jobs: jobs:
lint: lint:
name: Lint name: Lint (${{ matrix.features }})
strategy:
matrix:
features: ["--no-default-features", "--all-features", ""]
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
@@ -21,39 +24,56 @@ jobs:
with: with:
components: rustfmt, clippy components: rustfmt, clippy
- name: Check formatting - name: Check formatting (once)
if: matrix.features == ''
run: cargo fmt -- --check run: cargo fmt -- --check
- name: Clippy - name: Clippy
run: cargo clippy --all-targets --all-features -- -D warnings run: cargo clippy --all-targets ${{ matrix.features }} -- -D warnings
msrv: msrv:
name: MSRV Verification name: MSRV Verification (${{ matrix.features }})
strategy:
matrix:
features: ["--no-default-features", "--all-features", ""]
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
- name: Extract MSRV from Cargo.toml
id: msrv
run: |
MSRV=$(yq '.package.rust-version' Cargo.toml --unwrapScalar)
if [ -z "$MSRV" ]; then
echo "No MSRV specified in Cargo.toml, using stable"
MSRV="stable"
else
echo "msrv=$MSRV" >> $GITHUB_OUTPUT
echo "Using MSRV: $MSRV"
fi
- name: Install Rust stable - name: Install Rust stable
uses: dtolnay/rust-toolchain@stable uses: dtolnay/rust-toolchain@master
with:
toolchain: ${{ steps.msrv.outputs.msrv }}
- name: Install cargo-msrv - name: Install cargo-msrv
run: cargo install cargo-msrv uses: taiki-e/install-action@v2
with:
tool: cargo-msrv
- name: Verify MSRV - name: Verify MSRV
run: cargo msrv verify run: cargo msrv verify ${{ matrix.features }}
test: test:
name: Test name: Test (${{ matrix.os }}, ${{ matrix.rust }}, ${{ matrix.features }})
strategy: strategy:
matrix: matrix:
os: [ubuntu-latest, macos-latest, windows-latest] os: [ubuntu-latest, macos-latest, windows-latest]
rust: [stable, nightly] rust: [stable, nightly]
include: features: ["--no-default-features", "--all-features", ""]
- os: ubuntu-latest fail-fast: false
rust: "1.78.0" # MSRV
exclude:
- rust: nightly
runs-on: ${{ matrix.os }} runs-on: ${{ matrix.os }}
continue-on-error: ${{ matrix.rust == 'nightly' }} continue-on-error: ${{ matrix.rust == 'nightly' }}
@@ -66,9 +86,28 @@ jobs:
toolchain: ${{ matrix.rust }} toolchain: ${{ matrix.rust }}
- uses: Swatinem/rust-cache@v2 - uses: Swatinem/rust-cache@v2
with:
key: ${{ matrix.os }}-${{ matrix.rust }}-${{ hashFiles('matrix.features') }}
- name: Run tests - name: Run tests
run: cargo test --all-targets --all-features run: cargo test --all-targets ${{ matrix.features }}
check:
name: Cargo Check (${{ matrix.features }})
strategy:
matrix:
features: ["--no-default-features", "--all-features", ""]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install Rust
uses: dtolnay/rust-toolchain@stable
- uses: Swatinem/rust-cache@v2
- name: Cargo check
run: cargo check ${{ matrix.features }}
examples: examples:
name: Examples name: Examples
@@ -97,10 +136,11 @@ jobs:
uses: dtolnay/rust-toolchain@stable uses: dtolnay/rust-toolchain@stable
- name: Install cargo-tarpaulin - name: Install cargo-tarpaulin
run: cargo install cargo-tarpaulin uses: taiki-e/install-action@cargo-tarpaulin
- name: Generate code coverage - name: Generate code coverage
run: cargo tarpaulin --out Xml --out Lcov --output-dir coverage --fail-under 90 run: |
cargo tarpaulin --out Xml --out Lcov --output-dir coverage --fail-under 80
security: security:
name: Security Audit name: Security Audit
@@ -111,13 +151,17 @@ jobs:
- name: Install Rust - name: Install Rust
uses: dtolnay/rust-toolchain@stable uses: dtolnay/rust-toolchain@stable
- name: Cargo Audit - name: Install cargo-audit
uses: actions-rs/audit-check@v1 uses: taiki-e/install-action@cargo-audit
with:
token: ${{ secrets.GITHUB_TOKEN }} - name: Run cargo audit
run: cargo audit
doc: doc:
name: Documentation name: Documentation (${{ matrix.features }})
strategy:
matrix:
features: ["--no-default-features", "--all-features", ""]
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
@@ -126,4 +170,4 @@ jobs:
uses: dtolnay/rust-toolchain@stable uses: dtolnay/rust-toolchain@stable
- name: Build documentation - name: Build documentation
run: cargo doc --no-deps --document-private-items run: cargo doc --no-deps ${{ matrix.features }}
+219 -1
View File
@@ -5,6 +5,218 @@ All notable changes to dagx will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
### Fixed
- Now able to return bare tuples as task outputs
- No longer able to create a DAG with cyclic dependencies ([#3](https://github.com/swaits/dagx/pull/3))
- The `#[task]` macro no longer requires an explicit return type ([#2](https://github.com/swaits/dagx/pull/2))
### Changed
- Task inputs no longer need to be `Clone` ([#7](https://github.com/swaits/dagx/pull/7))
- Building a DAG is now a single-threaded operation ([#12](https://github.com/swaits/dagx/pull/12))
- Running a DAG now consumes the `DagRunner` and returns a `DagOutput` from which to get results ([#12](https://github.com/swaits/dagx/pull/12))
- `#[task]` macro support is now gated behind the `derive` default feature
- Tasks now receive input via the `TaskInput` type, which guarantees via the public API that dependencies can be extracted ([#7](https://github.com/swaits/dagx/pull/7))
- Task outputs are now extracted as owned values, not clones of an Arc value ([#7](https://github.com/swaits/dagx/pull/7))
### Internal
- The `task_fn` macro has been extracted to the dagx-test crate
- Task inputs are passed to tasks directly from the runner, not through channels ([#1](https://github.com/swaits/dagx/pull/1))
- Redundant tests/examples/benchmarks removed to lower maintenance and refactor burden ([#11](https://github.com/swaits/dagx/pull/11))
- parking_lot dependency removed ([#12](https://github.com/swaits/dagx/pull/12))
- futures dependency replaced with futures-util ([#8](https://github.com/swaits/dagx/pull/8))
## [0.3.1] - 2025-10-10
[View changes](https://git.swaits.com/swaits/dagx/compare/v0.3.0...v0.3.1)
### Fixed
- **Custom types now work automatically!** The `#[task]` macro now generates inline `extract_and_run()`
methods with type-specific extraction logic. This means **ANY type** implementing `Clone + Send + Sync + 'static`
works automatically without implementing any special traits.
- Previously, only types with `ExtractInput` implementations worked (primitives, String, Vec, HashMap, etc.)
- Now, custom user types work seamlessly - just derive `Clone` and you're good to go!
- No `ExtractInput` trait implementation needed for custom types
- Nested structs, collections of custom types, and complex type hierarchies all work out of the box
- New example: `examples/custom_types.rs` demonstrating custom types flowing through a DAG
### Changed
- **Documentation improvements**
- Updated README.md to highlight custom type support as a key feature
- Added "Custom Types" section to Core Concepts with examples
- Added `custom_types.rs` to the examples list
- Updated feature list to emphasize "Works with ANY type"
### Internal
- `ExtractInput` trait is now only used by the internal `task_fn` test helper
- The `#[task]` macro generates custom extraction logic per task, bypassing `ExtractInput` entirely
- This architectural change enables ANY type to work without requiring trait implementations
## [0.3.0] - 2025-10-08
[View changes](https://git.swaits.com/swaits/dagx/compare/v0.2.3...v0.3.0)
### Added
- **Compile-time cycle prevention proof** via new `src/cycle_prevention.rs` module
- Comprehensive documentation with `compile_fail` tests proving cycles are impossible
- New integration test suite `tests/cycle_prevention.rs` demonstrating type-state pattern
- Documents how the type system prevents cycles at compile-time (zero runtime cost)
- Added prominent documentation section in README.md explaining the feature
- **New `DagError::ConcurrentExecution` variant**
- Replaces misuse of `CycleDetected` for concurrent run protection
- More semantically accurate error reporting
### Removed - **BREAKING CHANGES**
- **Removed `DagError::CycleDetected` error variant**
- Cycles are provably impossible via the public API due to type-state pattern
- Removed 24 lines of unreachable cycle detection code from `runner.rs`
- **Migration**: If you were handling `DagError::CycleDetected`, remove that match arm
- This error could never occur in practice, so no runtime behavior changes
### Changed
- **Documentation improvements**
- Added comprehensive "Compile-Time Cycle Prevention" section to README.md
- Updated lib.rs documentation to highlight cycle prevention as first feature
- Fixed rustdoc links in `src/cycle_prevention.rs`
- Updated all error documentation references (removed cycle detection)
- Added code examples demonstrating type-state pattern
- Updated tagline to emphasize compile-time cycle prevention
- **Test coverage improvements**
- Added `#[cfg(not(tarpaulin_include))]` to untestable type alias
- Coverage at ~83% after removing dead code (target set to 80%)
- Remaining uncovered lines are primarily tracing macros and unreachable error paths
- **Enhanced CI configuration** (`.github/workflows/ci.yml`)
- MSRV verification now tests all 3 feature combinations (no-default, all, default)
- Test matrix expanded to 18 runs (3 platforms × 2 Rust versions × 3 feature combos)
- New cargo check job for all feature combinations
- Lint (clippy) now runs on all feature combinations
- Doc generation for all feature combinations
- Coverage threshold set to 80% (accounts for tracing macros and unreachable error paths)
- **Release check script improvements** (`scripts/release_check.sh`)
- Now reads MSRV from `Cargo.toml` instead of hardcoded version
- Uses `cargo msrv verify` for all feature combinations
- Tests all feature combinations (no-default, all, default)
- Runs clippy, check, and tests on all feature combos
- Includes coverage verification with 80% threshold
### Fixed
- **Clippy lints in test code**
- Removed unnecessary `.clone()` call on `Copy` type in `tests/cycle_prevention.rs`
- Removed unnecessary borrow in test dependency wiring
- **Flaky timing test on macOS CI**
- Increased timeout threshold from 140ms to 200ms in `test_arc_parallel_execution`
- macOS CI runners are significantly slower than Linux/Windows
- 200ms threshold still proves parallelism while accounting for CI variability
## [0.2.3] - 2025-10-08
[View changes](https://git.swaits.com/swaits/dagx/compare/v0.2.2...v0.2.3)
### Added
- **Optional tracing support** via `tracing` feature flag
- Zero-cost when disabled (literally 0ns overhead - code removed at compile time)
- Structured logging at INFO, DEBUG, TRACE, and ERROR levels
- Instrumentation for DAG construction, execution, and error paths
- New example: `examples/tracing_example.rs` demonstrating usage
- Comprehensive test coverage in `tests/tracing/` for both with/without feature
- Follows same pattern as tokio/hyper for performance-critical libraries
- Updated documentation in README.md and lib.rs
### Changed
- Improved `examples/04_parallel_computation.rs` to actually prove parallelism
- Now compares parallel vs sequential execution with timing measurements
- Calculates and displays speedup ratio (e.g., "4.0x faster")
- Clarifies difference from basic fan-in pattern in 03_fan_in.rs
- Added warnings for debug builds with low speedup
### Fixed
- Fixed broken ASCII diagram in `examples/04_parallel_computation.rs`
## [0.2.2] - 2025-10-07
[View changes](https://git.swaits.com/swaits/dagx/compare/v0.2.1...v0.2.2)
### Fixed
- Fixed README.md to reference correct crate version (`0.2` instead of `0.1`)
- Enhanced release check script to validate version consistency in documentation
- Automatically extracts major.minor version and verifies documentation matches
## [0.2.1] - 2025-10-07
[View changes](https://git.swaits.com/swaits/dagx/compare/v0.2.0...v0.2.1)
### Fixed
- Fixed test execution on macOS by forcing `tokio::test` to use multi-threaded runtime
- Ensures tests pass consistently across all platforms
## [0.2.0] - 2025-10-08
[View changes](https://git.swaits.com/swaits/dagx/compare/v0.1.0...v0.2.0)
### Changed
- **BREAKING**: Updated MSRV (Minimum Supported Rust Version) from 1.78.0 to 1.81.0
- Required by updated dependencies: `criterion@0.7.0`, `half@2.6.0`, `rayon@1.11.0`
- Updated in both `dagx` and `dagx-macros` crates
- Updated all documentation (README.md, CONTRIBUTING.md)
### Added
- Comprehensive test coverage improvements (82.61% → 92.94%)
- Added 9 new unit tests for error paths in `ExtractInput` implementations
- Added 3 new type conversion tests in `src/types/tests.rs`
- Added 1 new trait implementation test in `src/task/tests.rs`
- Added 1 new dependency tuple test in `src/deps/tests.rs`
- Added 6 new execution path tests in `tests/execution/basic.rs`
- Added error handling tests for HashMap, Result, Option, Vec, and Arc types
- Added tests for tuple dependency count validation
- Improved concurrent run protection test with better synchronization
- Added `tarpaulin_include` to lint configuration for coverage tooling compatibility
### Fixed
- Fixed GitHub Actions `cargo-audit` workflow
- Replaced deprecated `actions-rs/audit-check@v1` with direct `cargo audit` command
- Resolved "Resource not accessible by integration" error
- Fixed clippy warnings in test code
- Removed unnecessary borrows in `.depends_on()` calls
- Added `#[allow(clippy::clone_on_copy)]` for explicit clone test
- Marked timing-sensitive and resource-intensive tests with `#[cfg_attr(tarpaulin, ignore)]`
- `test_arc_parallel_execution` (timing unreliable under instrumentation)
- `test_100000_nodes_stress` (resource intensive)
- `test_10000_level_chain_stress` (resource intensive)
### Internal
- Improved test organization following existing patterns in `tests/` directory
- Enhanced error path coverage for all `ExtractInput` trait implementations
- Better separation between unit tests and integration tests
- Refactored `scripts/release_check.sh` to use configurable VERSION variable
- Replaced hardcoded version references throughout script
- Updated instructions to use `jj` commands instead of `git`
- Simplified version bumps for future releases
## [0.1.0] - 2025-10-05 ## [0.1.0] - 2025-10-05
### Added ### Added
@@ -97,4 +309,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- macOS (compatible) - macOS (compatible)
- Windows (compatible) - Windows (compatible)
[0.1.0]: https://github.com/swaits/dagx/releases/tag/v0.1.0 [0.3.1]: https://git.swaits.com/swaits/dagx/src/tag/v0.3.1
[0.3.0]: https://git.swaits.com/swaits/dagx/src/tag/v0.3.0
[0.2.3]: https://git.swaits.com/swaits/dagx/src/tag/v0.2.3
[0.2.2]: https://git.swaits.com/swaits/dagx/src/tag/v0.2.2
[0.2.1]: https://git.swaits.com/swaits/dagx/src/tag/v0.2.1
[0.2.0]: https://git.swaits.com/swaits/dagx/src/tag/v0.2.0
[0.1.0]: https://git.swaits.com/swaits/dagx/src/tag/v0.1.0
+16 -9
View File
@@ -6,7 +6,7 @@ Thanks for your interest in contributing! Please read our [Code of Conduct](CODE
```bash ```bash
# Clone and build # Clone and build
git clone https://github.com/swaits/dagx git clone https://git.swaits.com/swaits/dagx
cd dagx cd dagx
cargo build cargo build
@@ -21,14 +21,16 @@ cargo clippy -- -D warnings
## Development Setup ## Development Setup
**Prerequisites:** **Prerequisites:**
- Rust 1.78+ (edition 2021)
- Rust 1.81+ (edition 2021)
- Git - Git
**Optional tools:** **Optional tools:**
```bash ```bash
rustup component add rustfmt clippy rustup component add rustfmt clippy
cargo install cargo-llvm-cov # For coverage cargo binstall cargo-tarpaulin # For coverage
cargo install cargo-audit # For security checks cargo binstall cargo-audit # For security checks
``` ```
## Making Changes ## Making Changes
@@ -67,7 +69,8 @@ cargo build --examples # Examples compile
- All doc examples must compile - All doc examples must compile
Example: Example:
```rust
````rust
/// Adds a task to the DAG. /// Adds a task to the DAG.
/// ///
/// # Examples /// # Examples
@@ -75,30 +78,31 @@ Example:
/// ``` /// ```
/// use dagx::DagRunner; /// use dagx::DagRunner;
/// ///
/// let dag = DagRunner::new(); /// let mut dag = DagRunner::new();
/// let task = dag.add_task(MyTask::new()); /// let task = dag.add_task(MyTask::new());
/// ``` /// ```
pub fn add_task<T: Task>(&self, task: T) -> TaskBuilder<T> { pub fn add_task<T: Task>(&self, task: T) -> TaskBuilder<T> {
// ... // ...
} }
``` ````
## Testing ## Testing
**Test organization:** **Test organization:**
- Unit tests: `#[cfg(test)]` modules in source files - Unit tests: `#[cfg(test)]` modules in source files
- Integration tests: `tests/` directory - Integration tests: `tests/` directory
- Examples: `examples/` directory (must compile) - Examples: `examples/` directory (must compile)
- Doctests: In `///` comments (must pass) - Doctests: In `///` comments (must pass)
**Runtime testing:** **Runtime testing:**
```bash ```bash
cargo test # Default (tokio) cargo test # Default (tokio)
cargo test --test async_std_runtime # async-std
cargo test --test smol_runtime # smol
``` ```
**Benchmarks:** **Benchmarks:**
```bash ```bash
cargo bench cargo bench
# View results: open target/criterion/report/index.html # View results: open target/criterion/report/index.html
@@ -130,6 +134,7 @@ Closes #123
**Types**: `feat`, `fix`, `docs`, `test`, `refactor`, `perf`, `chore` **Types**: `feat`, `fix`, `docs`, `test`, `refactor`, `perf`, `chore`
**Examples:** **Examples:**
- `feat: add timeout support for tasks` - `feat: add timeout support for tasks`
- `fix: correct cycle detection in diamond DAG` - `fix: correct cycle detection in diamond DAG`
- `docs: clarify TaskHandle usage in README` - `docs: clarify TaskHandle usage in README`
@@ -137,12 +142,14 @@ Closes #123
## Pull Requests ## Pull Requests
**PR description should include:** **PR description should include:**
- What changed and why - What changed and why
- How it was tested - How it was tested
- Any breaking changes - Any breaking changes
- Link to related issue (if applicable) - Link to related issue (if applicable)
**Review process:** **Review process:**
- Maintainer reviews within 1 week - Maintainer reviews within 1 week
- Address feedback - Address feedback
- Once approved, maintainer merges - Once approved, maintainer merges
Generated
+261 -457
View File
File diff suppressed because it is too large. Load diff
+33 -11
View File
@@ -1,30 +1,49 @@
[workspace] [workspace]
members = ["dagx-macros"] members = ["dagx-macros", "dagx-test"]
[package] [package]
name = "dagx" name = "dagx"
version = "0.1.0" version = "0.3.2"
edition = "2021" edition = "2021"
rust-version = "1.78.0" rust-version = "1.81"
authors = ["Stephen Waits <steve@waits.net>"] authors = ["Stephen Waits <steve@waits.net>"]
license = "MIT" license = "MIT"
description = "A minimal, type-safe, runtime-agnostic async DAG executor with compile-time dependency validation" description = "A minimal, type-safe, runtime-agnostic async DAG (Directed Acyclic Graph) executor with compile-time cycle prevention and true parallel execution"
repository = "https://github.com/swaits/dagx" repository = "https://git.swaits.com/swaits/dagx"
homepage = "https://github.com/swaits/dagx" homepage = "https://git.swaits.com/swaits/dagx"
documentation = "https://docs.rs/dagx" documentation = "https://docs.rs/dagx"
readme = "README.md" readme = "README.md"
keywords = ["dag", "async", "task", "parallel", "executor"] keywords = ["dag", "async", "task", "parallel", "executor"]
categories = ["asynchronous", "concurrency", "data-structures"] categories = ["asynchronous", "concurrency", "data-structures"]
exclude = [".claude/", ".jj/", "dagx-spec.md"] exclude = [".claude/", ".jj/", "dagx-spec.md"]
[features]
default = ["derive"]
derive = ["dep:dagx-macros"]
tracing = ["dep:tracing"]
[workspace.lints.rust]
unexpected_cfgs = { level = "warn", check-cfg = [
'cfg(tarpaulin)',
'cfg(tarpaulin_include)',
] }
[lints]
workspace = true
[dependencies] [dependencies]
futures = "0.3" futures-util = { version = "0.3", default-features = false, features = ["std"] }
parking_lot = "0.12" dagx-macros = { version = "0.3.2", path = "./dagx-macros", optional = true }
dagx-macros = { version = "0.1.0", path = "./dagx-macros" } tracing = { version = "0.1", optional = true }
[dev-dependencies] [dev-dependencies]
tokio = { version = "1", features = ["macros", "rt-multi-thread", "time", "sync"] } dagx-test = { path = "./dagx-test" }
async-std = { version = "1", features = ["attributes"] } tokio = { version = "1", features = [
"macros",
"rt-multi-thread",
"time",
"sync",
] }
smol = "2" smol = "2"
criterion = "0.7" criterion = "0.7"
# Quirky async runtimes for edge-case testing # Quirky async runtimes for edge-case testing
@@ -34,6 +53,9 @@ futures-executor = { version = "0.3", features = ["thread-pool"] }
# Comparison benchmarks # Comparison benchmarks
dagrs = { version = "0.5", default-features = false } dagrs = { version = "0.5", default-features = false }
async-trait = "0.1" async-trait = "0.1"
# Tracing support for tests and examples
tracing-subscriber = { version = "0.3", features = ["env-filter", "fmt"] }
test-case = "3.3.1"
[[bench]] [[bench]]
name = "dagx_bench" name = "dagx_bench"
+96 -761
View File
@@ -2,138 +2,32 @@
[![Crates.io](https://img.shields.io/crates/v/dagx.svg)](https://crates.io/crates/dagx) [![Crates.io](https://img.shields.io/crates/v/dagx.svg)](https://crates.io/crates/dagx)
[![Documentation](https://docs.rs/dagx/badge.svg)](https://docs.rs/dagx) [![Documentation](https://docs.rs/dagx/badge.svg)](https://docs.rs/dagx)
[![Build Status](https://github.com/swaits/dagx/workflows/CI/badge.svg)](https://github.com/swaits/dagx/actions) [![Build Status](https://git.swaits.com/swaits/dagx/actions/workflows/ci.yml/badge.svg)](https://git.swaits.com/swaits/dagx/actions)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE) [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
[![Rust Version](https://img.shields.io/badge/rust-1.78.0+-blue.svg)](https://www.rust-lang.org) [![Rust Version](https://img.shields.io/badge/rust-1.81+-blue.svg)](https://www.rust-lang.org)
A minimal, type-safe, runtime-agnostic async DAG (Directed Acyclic Graph) executor with compile-time dependency validation and true parallel execution. A minimal, type-safe, runtime-agnostic async DAG (Directed Acyclic Graph) executor with compile-time cycle prevention and true parallel execution.
**Why dagx?** No custom scheduler—just battle-tested primitives (channels, async/await). Compile-time type safety catches wiring errors before runtime. Works with ANY async runtime (Tokio, async-std, smol). Simple API: `#[task]`, `add_task()`, `depends_on()`—that's it. **4-148x faster than dagrs** across all workloads with sub-microsecond overhead per task. ## Why dagx?
## Features ### Blazing Fast: 1-100x faster than dagrs
- **Compile-time type safety**: Dependencies are validated at compile time through the type system. No `dyn Any`, no downcasting, no runtime type errors. | Workload | Tasks | dagx | dagrs | Speedup |
- **Runtime-agnostic**: Works with any async runtime (Tokio, async-std, smol, and more) | ----------------- | ------ | --------- | --------- | ------------------ |
- **True parallelism**: Tasks spawn to multiple threads for genuine parallel execution | Sequential chain | 5 | 1.02 µs | 770.42 µs | **755x faster** 🚀 |
- **Type-state pattern**: The API prevents incorrect dependency wiring through compile-time errors | Diamond pattern | 4 | 5.16 µs | 770.87 µs | **149x faster** |
- **Zero-cost abstractions**: Leverages generics and monomorphization for minimal overhead | Sequential chain | 100 | 25.09 µs | 1.19 ms | **47.4x faster** |
- **Flexible task patterns**: Supports stateless, read-only, and mutable state tasks | Fan-out (1→100) | 101 | 100.75 µs | 1.02 ms | **10.1x faster** |
- **Simple API**: Just `#[task]`, `DagRunner`, `TaskHandle`, and `TaskBuilder` | Independent tasks | 10,000 | 8.61 ms | 15.37 ms | **1.79x faster** |
- **Comprehensive error handling**: Result-based errors with actionable messages
## Design Philosophy: Primitives as Scheduler ### Simple API
dagx takes an unconventional approach to task orchestration: **there is no traditional scheduler**. Instead of complex scheduling logic managing when tasks run, the entire system is built on communication and synchronization primitives that do the scheduling themselves.
### How It Works
Traditional DAG executors contain substantial scheduling code—algorithms that track task states, manage dependencies, coordinate execution order, and handle synchronization. This code is complex, error-prone, and difficult to verify.
dagx eliminates this entirely:
1. **Wire up tasks with primitives**: During DAG construction, tasks are connected using channels and organized into topological layers based on their dependencies
2. **Start everything simultaneously**: When you call `run()`, all tasks spawn at once—there's no scheduler deciding when each task should start
3. **Let the primitives handle coordination**: Channels naturally enforce execution order—tasks wait on their input channels until upstream tasks send data. No custom orchestration needed.
4. **Runtime joins on completion**: The runtime simply spawns all tasks and waits for the completion channel to close. That's it.
### The Implementation
Under the hood, dagx uses:
- **Oneshot channels**: Each edge in the DAG gets a `futures::channel::oneshot` - producer sends once, consumer receives once
- **Ownership model**: Tasks take ownership (`self`) and are consumed during execution - no Mutex needed for task state
- **Direct data flow**: Outputs flow from producer to consumer via channels, never stored in shared memory during execution
- **Fast-path optimization**: Single-task layers execute inline without spawning overhead
The core execution logic in `run()` is remarkably simple:
```rust ```rust
// Create oneshot channel for each edge let sum = dag.add_task(Add).depends_on((x, y));
for (producer, consumer) in edges { dag.run(|fut| async move { tokio::spawn(fut).await.unwrap() }).await?;
let (tx, rx) = oneshot::channel();
// Wire tx to producer, rx to consumer
}
// Execute layer by layer
for layer in layers {
if layer.len() == 1 {
// Fast path: Execute inline, no spawning overhead
// IMPORTANT: Panic handling is required here!
let result = task.execute().catch_unwind().await;
// Convert panic to error to match spawned task behavior
} else {
// Parallel path: Spawn all tasks in layer
for task in layer {
spawner(async {
let input = await_on_input_channels(); // Blocks until deps complete
let output = task.run(input); // Consumes task
send_to_output_channels(output); // Sends to dependents
});
}
// Wait for layer completion
}
}
// Tasks coordinate themselves via channels - no scheduler needed
``` ```
No state machines. No task queues. No wake-up logic. No Mutex locks. Just channels doing what channels do, with an inline fast-path for sequential execution. That's it. No trait boilerplate, no manual channels, no node IDs.
### Inline Execution Fast-Path
**Performance optimization for sequential workloads**: When a layer contains only a single task (common in deep chains and linear pipelines), dagx executes it inline rather than spawning it. This eliminates spawning overhead, context switching, and channel creation, resulting in 10-100x performance improvements for sequential patterns.
**Panic handling guarantee**: To maintain behavioral consistency between inline and spawned execution, panics in inline tasks are caught using `FutureExt::catch_unwind()` and converted to errors. This matches the behavior of all major async runtimes (Tokio, async-std, smol, embassy-rs), which catch panics in spawned tasks and convert them to `JoinError` or equivalent.
**Why this matters**:
- **Spawned tasks** (layer.len() > 1): Runtime catches panics automatically → becomes error
- **Inline tasks** (layer.len() == 1): We catch panics manually → becomes error
- **Result**: Identical behavior regardless of execution path
This ensures your code behaves the same whether a task runs inline or spawned, making dagx's optimizations transparent and predictable.
### Benefits
**Simplicity**: The runtime is straightforward: create channels, spawn tasks, let them coordinate via awaiting. No complex scheduler code to maintain, debug, or optimize.
**Reliability**: Built on battle-tested primitives (oneshot channels, async/await) from Rust's standard library and the futures crate. These have been used in production by thousands of projects and are orders of magnitude more reliable than custom scheduling logic.
**Bug resistance**: Fewer moving parts means fewer places for bugs to hide. The type system enforces correct wiring at compile time. Channels handle synchronization. The ownership model prevents data races. What's left to break?
**Performance**: Near zero-overhead. No Mutex locks during execution. Arc reference counting for efficient fanout (atomic operations, not locks). Oneshot channels are often lock-free. Tasks start as soon as their dependencies complete - maximum parallelism.
**Auditability**: Want to verify correctness? Check the channel wiring, verify tasks await their inputs, done. No need to trace through complex state machine transitions or wake-up cascades.
### The Insight
The key insight is that **dependencies ARE the schedule**. If task B depends on task A's output, a channel naturally enforces that B waits for A. The dependency graph already encodes all the scheduling information—we just need to wire up channels to match it.
This is dagx's core philosophy: leverage the type system for correctness, use primitives for coordination, and let the compiler optimize everything else away.
### Measured Overhead
How much overhead does this approach actually add? Benchmarks on an AMD 7840U (Zen 4 laptop CPU) show:
**DAG Construction**:
- Empty DAG creation: **~20 nanoseconds**
- Adding tasks: **~96 nanoseconds per task**
- Building a 10,000-task DAG: **~1.04 milliseconds** (104 ns/task)
**Execution Overhead** (framework coordination, excluding actual task work):
- Sequential workloads: **~0.78 microseconds per task** (inline execution fast-path)
- Parallel workloads: **~1.2 microseconds per task**
- 100-task deep chain: **~78 microseconds total**
- 100 independent tasks: **~122 microseconds total**
- 10,000 independent tasks: **~11.2 milliseconds total**
**Scaling**: Sub-microsecond per-task overhead across all workload patterns. Linear scaling verified to 10k+ tasks.
**Comparison**: dagx is **4-148x faster** than dagrs (v0.5) across all benchmark patterns (see detailed benchmarks section).
The primitives-as-scheduler approach with inline fast-path optimization delivers exceptional performance: coordination overhead is sub-microsecond per task, and for real-world workloads where tasks do meaningful work (I/O, computation, etc.), framework overhead is negligible—typically well under 1% of total execution time.
## Quick Start ## Quick Start
@@ -141,7 +35,7 @@ Add to your `Cargo.toml`:
```toml ```toml
[dependencies] [dependencies]
dagx = "0.1" dagx = "0.3"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] } tokio = { version = "1", features = ["macros", "rt-multi-thread"] }
``` ```
@@ -172,315 +66,48 @@ impl Add {
#[tokio::main] #[tokio::main]
async fn main() { async fn main() {
let dag = DagRunner::new(); let mut dag = DagRunner::new();
// Add source tasks with no dependencies // Add source tasks with no dependencies
let x = dag.add_task(Value(2)); let x = dag.add_task(Value(2));
let y = dag.add_task(Value(3)); let y = dag.add_task(Value(3));
// Add task that depends on both x and y // Add task that depends on both x and y.
let sum = dag.add_task(Add).depends_on((&x, &y)); let sum = dag.add_task(Add).depends_on((x, y));
// Execute with true parallelism // Execute with true parallelism
dag.run(|fut| { tokio::spawn(fut); }).await.unwrap(); let mut output = dag.run(|fut| async move { tokio::spawn(fut).await.unwrap() }).await.unwrap();
// Retrieve results // Retrieve results
assert_eq!(dag.get(sum).unwrap(), 5); assert_eq!(output.get(sum), 5);
} }
``` ```
## Comparison with Similar Projects ## Features
The Rust ecosystem offers several DAG execution libraries, each optimized for different use cases. This comparison helps you choose the right tool for your needs. ### Compile-Time Safety
### Developer Experience: API Comparison - **Cycles are impossible** — the type system prevents them at compile time, zero runtime overhead
- **No runtime type errors** — dependencies validated at compile time
- **Compiler-verified correctness** — no surprise failures in production
The same task (compute 2 + 3) implemented across different libraries shows how API complexity varies: See [how it works](docs/CYCLE_PREVENTION.md).
**dagx** (Simple: macro + builder): ### Runtime Agnostic
dagx works with any async runtime. Provide a spawner function to `run()`:
```rust ```rust
#[task] // With Tokio
impl Value { async fn run(&self) -> i32 { self.0 } } // The join handle result can be unwrapped because dagx catches panics internally
dag.run(|fut| async move { tokio::spawn(fut).await.unwrap() }).await.unwrap();
#[task] // With smol
impl Add { async fn run(a: &i32, b: &i32) -> i32 { a + b } } dag.run(|fut| smol::spawn(fut)).await.unwrap();
let dag = DagRunner::new(); // Single-threaded concurrency on the invoking runtime
let x = dag.add_task(Value(2)); // Can be faster in situations where waiting time dominates
let y = dag.add_task(Value(3)); dag.run(|fut| fut).await.unwrap()
let sum = dag.add_task(Add).depends_on((&x, &y));
dag.run(|fut| tokio::spawn(fut)).await?;
```
**dagrs** (Complex: traits + channels + IDs + manual wiring):
```rust
#[async_trait]
impl Action for Value {
async fn run(&self, _: &mut InChannels, out: &mut OutChannels, _: Arc<EnvVar>) -> Output {
out.broadcast(Content::new(self.0)).await;
Output::Out(Some(Content::new(self.0)))
}
}
let mut table = NodeTable::new();
let node1 = DefaultNode::with_action("x".into(), Value(2), &mut table);
let id1 = node1.id();
let node2 = DefaultNode::with_action("y".into(), Value(3), &mut table);
let id2 = node2.id();
let node3 = DefaultNode::with_action("add".into(), Add, &mut table);
let id3 = node3.id();
let mut graph = Graph::new();
graph.add_node(node1); graph.add_node(node2); graph.add_node(node3);
graph.add_edge(id1, vec![id3]); graph.add_edge(id2, vec![id3]);
graph.set_env(EnvVar::new(table)); graph.start().unwrap();
```
**async_dag** (Medium: slots + indices):
```rust
let mut graph = Graph::new();
let x = graph.add_task(|| async { 2 });
let y = graph.add_task(|| async { 3 });
let sum = graph.add_child_task(x, |a: i32| async move { a }, 0)?;
graph.update_dependency(y, sum, 1)?; // Must specify slot index
let sum = graph.add_child_task(sum, |a: i32, b: i32| async move { a + b }, 0)?;
graph.update_dependency(y, sum, 1)?;
```
**Key differences**:
- **dagx**: Type-safe dependencies, automatic wiring, no manual ID tracking, minimal boilerplate
- **dagrs**: Manual channel management, node ID tracking, Content wrapping, Action trait boilerplate
- **async_dag**: Slot indices must be tracked manually, dependencies updated separately
### Quick Comparison
| Project | License | Runtime | Type Safety | API Complexity | Performance vs dagx | Key Features |
| ------------------------------------------------------------- | -------------- | ------------------- | --------------------- | -------------- | --------------------------------- | ------------------------------------------------------------------------------------ |
| **dagx** | MIT | Any async runtime | Compile-time | Simple | Baseline (see benchmarks) | Primitives-as-scheduler, inline fast-path, automatic Arc wrapping, up to 8 deps/task |
| [**dagrs**](https://github.com/dagrs-dev/dagrs) (v0.5) | MIT/Apache-2.0 | Creates own runtime | Runtime (async_trait) | Complex | 4-148x slower across all patterns | Flow-based Programming, cyclic graphs, loops, conditional nodes, YAML config |
| [**async_dag**](https://github.com/chubei-oppen/async_dag) | MIT | Any async runtime | Compile-time | Medium | No benchmarks | Slot-based dependencies, Graph/TryGraph modes, maximum parallelism |
| [**dag-flow**](https://github.com/makisevon/dag-flow) | MIT/Apache-2.0 | Any async runtime | Runtime (HashMap) | Complex | No benchmarks | Experimental, all tasks run simultaneously, weak dependencies |
| [**RenovZ/dag-runner**](https://github.com/RenovZ/dag-runner) | MIT | Tokio only | Unclear | Simple | No benchmarks | Edge-based API, cycle detection, stops on first error |
| [**tasksitter**](https://github.com/lionkor/tasksitter) | Unspecified | Unclear | Unclear | Medium | No benchmarks | Cyclic graphs, dynamic runtime modification, pause/resume |
_GitHub stars (as of 2025): dagrs (449), async_dag (25), dag-flow (2), others (0-2)_
### Performance Benchmarks vs dagrs
Direct comparison benchmarks (lower is better):
| Workload Pattern | dagx | dagrs (v0.5) | dagx Performance |
| --------------------------------------------------- | -------- | ------------ | ------------------ |
| **Linear pipeline** (5 sequential tasks) | 3.0 µs | 446.1 µs | **148x faster** 🚀 |
| **ETL pipeline** (realistic extract-transform-load) | 26.2 µs | 460.4 µs | **17.6x faster** ✓ |
| **Deep chain** (100 purely sequential tasks) | 78.2 µs | 801.5 µs | **10.2x faster** ✓ |
| **Wide fanout** (1→100 broadcast) | 161.0 µs | 665.5 µs | **4.1x faster** ✓ |
| **Large scale** (10,000 independent tasks) | 11.2 ms | 13.6 ms | **1.2x faster** ✓ |
**Summary**: dagx is **4-148x faster** than dagrs across all benchmark patterns. The inline fast-path optimization eliminates spawning overhead for sequential workloads while maintaining excellent parallel performance.
**How does dagx achieve this?** The inline fast-path detects single-task layers (common in sequential chains) and executes them directly without spawning overhead. For parallel workloads, tasks still spawn to maximize concurrency. This adaptive execution strategy combines the best of both worlds: sub-microsecond overhead for sequential work, true parallelism for concurrent work.
**Key insight**: Most real-world DAGs mix sequential and parallel patterns. dagx automatically optimizes for both, delivering 4-148x better performance than dagrs regardless of workload shape.
_Benchmarks run on AMD Ryzen 7 7840U (Zen 4) @ 3.3GHz. Run `cargo bench` to test on your hardware._
### Detailed Comparison
#### dagrs (Most Mature)
**Best for**: Complex workflows requiring advanced flow control, Tokio-based applications, machine learning pipelines.
**Strengths**:
- Most mature (449 GitHub stars, active community)
- Rich feature set: Flow-based Programming, cyclic graphs, loops, conditional nodes
- YAML configuration support for declarative workflows
- Designed for complex orchestration patterns
**Trade-offs**:
- Creates own Tokio runtime internally (not runtime-agnostic, cannot be nested)
- More complex API: `Action` trait, `InChannels`/`OutChannels`, `NodeTable`, `Content` wrappers, manual node ID tracking
- Uses `async_trait` for type erasure (runtime overhead)
- **4-148x slower than dagx** across all benchmark patterns (see comparison benchmarks)
**API Style** (v0.5):
```rust
// Define action with channels for communication
#[async_trait]
impl Action for MyTask {
async fn run(&self, input: &mut InChannels,
output: &mut OutChannels, env: Arc<EnvVar>) -> Output {
let inputs: Vec<T> = input.map(|c| ...).await; // Manually extract
output.broadcast(Content::new(result)).await; // Manually wrap
Output::Out(Some(Content::new(result)))
}
}
let mut table = NodeTable::new();
let node = DefaultNode::with_action("name".into(), MyTask, &mut table);
let node_id = node.id(); // Must capture ID before moving
graph.add_node(node);
graph.add_edge(source_id, vec![target_id]);
graph.set_env(EnvVar::new(table)); graph.start().unwrap();
```
#### async_dag (Clean Type Safety)
**Best for**: Runtime flexibility with compile-time type safety, fail-fast workflows.
**Strengths**:
- Runtime-agnostic (works with any async runtime)
- Compile-time type checking on task connections
- Both standard (`Graph`) and fail-fast (`TryGraph`) modes
- Designed for maximum parallelism
**Trade-offs**:
- Medium API complexity: slot-based dependency management
- Must manually specify slot indices (0, 1, etc.) when connecting tasks
- Less mature (25 stars)
- No performance benchmarks
**API Style**:
```rust
let mut graph = Graph::new();
let _1 = graph.add_task(|| async { 1 });
let _2 = graph.add_task(|| async { 2 });
// add_child_task with slot index
let _3 = graph.add_child_task(_1, sum, 0).unwrap();
graph.update_dependency(_2, _3, 1).unwrap(); // Specify slot 1
graph.run().await;
```
#### dag-flow (Experimental)
**Best for**: Experimental projects, flexible dependency awaiting patterns.
**Strengths**:
- Runtime-agnostic
- All tasks run simultaneously (not in dependency layers)
- Weak dependencies support
- Flexible input awaiting at any point in task execution
**Trade-offs**:
- Explicitly experimental
- Runtime type safety via `HashMap<String, Input>`
- Complex API: implement `Task` trait with `id()`, `dependencies()`, `run()`
- Named dependencies (string-based lookup)
- Very early stage (2 stars)
**API Style**:
```rust
impl Task<String, Bytes> for MyTask {
fn id(&self) -> String { "task_name".into() }
fn dependencies(&self) -> Option<Vec<String>> { Some(vec!["dep1".into()]) }
async fn run(&self, inputs: HashMap<String, Input<'_, Bytes>>) -> Option<Bytes> {
let dep_value = inputs.get("dep1").unwrap().await;
// Process
}
}
```
#### RenovZ/dag-runner (Simple Edge-Based)
**Best for**: Simple DAGs with Tokio, straightforward edge-based dependencies.
**Strengths**:
- Simple API: `add_vertex()`, `add_edge()`
- Cycle detection
- Stops on first error
**Trade-offs**:
- Requires Tokio runtime
- Manual channel setup for task communication
- Type safety model unclear
- Very early stage (0 stars)
**API Style**:
```rust
let mut dag = Dag::default();
dag.add_vertex("one", || async move { /* task */ });
dag.add_edge("one", "two");
dag.run().await?;
```
#### tasksitter (Dynamic Workflows)
**Best for**: Dynamic workflow modification, cyclic graphs, runtime introspection.
**Strengths**:
- Supports cyclic graphs (not just DAGs)
- Dynamic graph modification at runtime
- Pause/resume capabilities
- Graph introspection
**Trade-offs**:
- Limited documentation
- Runtime and type safety model unclear
- Very early stage (0 stars)
### When to Choose dagx
Choose dagx when you value:
- **Performance**: 4-148x faster than dagrs across all workload patterns (see benchmarks)
- **Runtime flexibility**: Works with Tokio, async-std, smol, or any async runtime
- **Compile-time safety**: Full type safety with no runtime type errors in the public API
- **Minimal overhead**: ~0.78µs per task (sequential), ~1.2µs per task (parallel)
- **Simple, ergonomic API**: `#[task]` macro, `add_task()`, `depends_on()` - that's it
- **Automatic optimizations**: Arc wrapping, inline execution, adaptive spawning - all transparent
- **Predictable performance**: Linear scaling, no hidden complexity, consistent sub-µs overhead
dagx is **not** the right choice if you need:
- Cyclic graphs or dynamic flow control (loops, conditions) → Consider **dagrs** or **tasksitter**
- More than 8 dependencies per task → Consider **dagrs** or **async_dag**
- YAML-based configuration → Consider **dagrs**
### When to Consider Alternatives
- **Choose dagrs** if you need advanced flow control (loops, conditionals, cyclic graphs), YAML configuration, or are already committed to Tokio and want a mature, feature-rich solution
- **Choose async_dag** if you want compile-time type safety with runtime flexibility and the slot-based API appeals to you
- **Choose dag-flow** if you're building experimental projects and the all-tasks-run-simultaneously model fits your use case
- **Choose RenovZ/dag-runner** if you need the simplest possible edge-based API and are already using Tokio
- **Choose tasksitter** if you need dynamic graph modification at runtime or cyclic workflow support
## Core Concepts
### Task
A `Task` is a unit of async work with typed inputs and outputs. Use the `#[task]` macro to define tasks:
```rust
use dagx::{task, Task};
struct Scale(i32);
#[task]
impl Scale {
async fn run(&self, input: &i32) -> i32 {
input * self.0
}
}
``` ```
### Task Patterns ### Task Patterns
@@ -523,271 +150,52 @@ impl Counter {
} }
``` ```
### DagRunner ### Tracing
The `DagRunner` orchestrates task execution: dagx provides optional observability using the `tracing` crate, controlled by the `tracing` feature flag.
```rust **Enabling Tracing**
let dag = DagRunner::new();
let handle = dag.add_task(MyTask::new()); ```toml
[dependencies]
dagx = { version = "0.3", features = ["tracing"] }
tracing-subscriber = "0.3"
``` ```
### TaskHandle **Log Levels**
A `TaskHandle<T>` is a typed reference to a task's output. Use it to wire dependencies and retrieve results: - **INFO**: DAG execution start/completion
- **DEBUG**: Task additions, dependency wiring, layer computation
- **TRACE**: Individual task execution (inline vs spawned), detailed execution flow
- **ERROR**: Task panics, concurrent execution attempts
```rust ### Other
// Single dependency
let task = dag.add_task(my_task).depends_on(&upstream);
// Multiple dependencies (order matters!) - **True parallelism**: Chosen runtime executes tasks concurrently and/or in parallel
let task = dag.add_task(my_task).depends_on((&upstream1, &upstream2)); - **No boilerplate**: The `derive` feature and the `#[task]` macro are enabled by default to simplify task implementation.
```
## Examples ## Performance
### Fan-out Pattern (1 → n) dagx provides true parallel execution with sub-microsecond overhead per task.
One task produces a value consumed by multiple downstream tasks: **How is dagx so fast?**
```rust - **Inline fast-path**: Sequential chains execute inline without spawning
use dagx::{task, DagRunner, Task}; - **Adaptive execution**: Inline for sequential work, executor-agnostic parallelism for concurrent work
- **Zero-cost abstractions**: Compile-time graph validation eliminates overhead
struct Value(i32); See [design philosophy](docs/DESIGN_PHILOSOPHY.md) for details.
#[task] ## Tutorials & Examples
impl Value {
async fn run(&self) -> i32 { self.0 }
}
struct Increment(i32); ### Tutorials (Start Here)
#[task] Step-by-step introduction to dagx:
impl Increment {
async fn run(&self, input: &i32) -> i32 { input + self.0 }
}
struct Scale(i32); - [`01_basic.rs`](examples/01_basic.rs) - Your first DAG
- [`02_fan_out.rs`](examples/02_fan_out.rs) - One task feeds many (1→N)
#[task] - [`03_fan_in.rs`](examples/03_fan_in.rs) - Many tasks feed one (N→1)
impl Scale { - [`04_parallel_computation.rs`](examples/04_parallel_computation.rs) - Map-reduce with true parallelism
async fn run(&self, input: &i32) -> i32 { input * self.0 }
}
#[tokio::main]
async fn main() {
let dag = DagRunner::new();
let base = dag.add_task(Value(10));
let plus1 = dag.add_task(Increment(1)).depends_on(&base);
let times2 = dag.add_task(Scale(2)).depends_on(&base);
dag.run(|fut| { tokio::spawn(fut); }).await.unwrap();
assert_eq!(dag.get(plus1).unwrap(), 11);
assert_eq!(dag.get(times2).unwrap(), 20);
}
```
### Fan-in Pattern (m → 1)
Multiple tasks produce values consumed by a single downstream task:
```rust
use dagx::{task, DagRunner, Task};
struct Name(String);
#[task]
impl Name {
async fn run(&self) -> String { self.0.clone() }
}
struct Age(i32);
#[task]
impl Age {
async fn run(&self) -> i32 { self.0 }
}
struct Active(bool);
#[task]
impl Active {
async fn run(&self) -> bool { self.0 }
}
struct FormatUser;
#[task]
impl FormatUser {
async fn run(n: &String, a: &i32, f: &bool) -> String {
format!("User: {n}, Age: {a}, Active: {f}")
}
}
#[tokio::main]
async fn main() {
let dag = DagRunner::new();
let name = dag.add_task(Name("Alice".to_string()));
let age = dag.add_task(Age(30));
let active = dag.add_task(Active(true));
let result = dag.add_task(FormatUser).depends_on((&name, &age, &active));
dag.run(|fut| { tokio::spawn(fut); }).await.unwrap();
assert_eq!(dag.get(result).unwrap(), "User: Alice, Age: 30, Active: true");
}
```
### Many-to-Many Pattern (m ↔ n)
Complex DAGs with multiple layers:
```rust
use dagx::{task, DagRunner, Task};
struct Value(i32);
#[task]
impl Value {
async fn run(&self) -> i32 { self.0 }
}
struct Add;
#[task]
impl Add {
async fn run(a: &i32, b: &i32) -> i32 { a + b }
}
struct Multiply;
#[task]
impl Multiply {
async fn run(a: &i32, b: &i32) -> i32 { a * b }
}
#[tokio::main]
async fn main() {
let dag = DagRunner::new();
// Layer 1: Sources
let x = dag.add_task(Value(2));
let y = dag.add_task(Value(3));
let z = dag.add_task(Value(5));
// Layer 2: Intermediate computations
let sum_xy = dag.add_task(Add).depends_on((&x, &y)); // 2 + 3 = 5
let prod_yz = dag.add_task(Multiply).depends_on((&y, &z)); // 3 * 5 = 15
// Layer 3: Final result
let total = dag.add_task(Add).depends_on((&sum_xy, &prod_yz)); // 5 + 15 = 20
dag.run(|fut| { tokio::spawn(fut); }).await.unwrap();
assert_eq!(dag.get(total).unwrap(), 20);
}
```
## Type-State Pattern
The API uses the type-state pattern to ensure correctness at compile time:
```rust
use dagx::{task, Task};
struct Increment(i32);
#[task]
impl Increment {
async fn run(&self, input: &i32) -> i32 { input + self.0 }
}
struct Uppercase;
#[task]
impl Uppercase {
async fn run(input: &String) -> String { input.to_uppercase() }
}
// ✅ This compiles
let task = dag.add_task(Increment(5)).depends_on(&upstream); // upstream produces i32
// ❌ This doesn't compile - type mismatch!
let task = dag.add_task(Uppercase).depends_on(&upstream); // Error: expected String, got i32
```
## Runtime Agnostic
dagx works with any async runtime. Provide a spawner function to `run()`:
```rust
// With Tokio
dag.run(|fut| { tokio::spawn(fut); }).await.unwrap();
// With async-std
dag.run(|fut| { async_std::task::spawn(fut); }).await.unwrap();
// With smol
dag.run(|fut| { smol::spawn(fut).detach(); }).await.unwrap();
```
## Important Limitations
### Tasks Cannot Return Bare Tuples
**Tasks cannot return bare tuples as output types.** This is a technical limitation of the current implementation. If you need to return multiple values from a task, use one of these workarounds:
**Option 1: Use a struct (recommended)**
```rust
struct UserData {
name: String,
age: i32,
}
struct FetchUser;
#[task]
impl FetchUser {
async fn run(id: &i32) -> UserData {
UserData {
name: "Alice".to_string(),
age: 30,
}
}
}
```
**Option 2: Wrap in Result**
```rust
struct FetchData;
#[task]
impl FetchData {
async fn run(id: &i32) -> Result<(String, i32), String> {
Ok(("Alice".to_string(), 30))
}
}
```
Structs are preferred because they're self-documenting and easier to refactor.
## Examples
The `examples/` directory contains both **tutorial examples** (numbered, beginner-friendly) and **reference examples** (practical patterns).
### Tutorial Examples
Start here if you're new to dagx. These examples build progressively:
- **`01_basic.rs`** - Getting started: your first DAG
- **`02_fan_out.rs`** - Fan-out pattern: one task feeds many (1 → N)
- **`03_fan_in.rs`** - Fan-in pattern: many tasks feed one (N → 1)
- **`04_parallel_computation.rs`** - Map-reduce pattern with true parallelism
Run tutorial examples: Run tutorial examples:
@@ -798,25 +206,25 @@ cargo run --example 03_fan_in
cargo run --example 04_parallel_computation cargo run --example 04_parallel_computation
``` ```
### Reference Examples ### Advanced Examples
Practical patterns for real-world use cases: Real-world patterns:
- **`complex_dag.rs`** - Multi-layer workflows with complex dependencies - [`circuit_breaker.rs`](examples/circuit_breaker.rs) - Circuit breaker pattern for resilient systems
- **`conditional_workflow.rs`** - Conditional execution with Result types - [`data_pipeline.rs`](examples/data_pipeline.rs) - ETL data processing pipeline
- **`data_pipeline.rs`** - Real-world ETL data processing pipeline - [`error_handling.rs`](examples/error_handling.rs) - Error propagation and recovery
- **`error_handling.rs`** - Error propagation and recovery strategies
- **`timeout.rs`** - Managing task timeouts
- **`large_dag.rs`** - Performance and scalability (10,000+ nodes)
- **`parallelism_proof.rs`** - Proof of true parallel execution (10,000 tasks × 1s = 1s total)
Run reference examples: Run any example: `cargo run --example circuit_breaker`
```bash ## Documentation
cargo run --example data_pipeline
cargo run --example error_handling Full API documentation is available at [docs.rs/dagx](https://docs.rs/dagx).
cargo run --example parallelism_proof
``` Detailed documentation on dagx's internals and advanced features:
- [**Compile-Time Cycle Prevention**](docs/CYCLE_PREVENTION.md) - How the type system prevents cycles
- [**Design Philosophy**](docs/DESIGN_PHILOSOPHY.md) - Primitives as scheduler, inline fast-path optimization
- [**Library Comparisons**](docs/COMPARISONS.md) - Detailed comparison with dagrs, async_dag, and others
## When to Use dagx ## When to Use dagx
@@ -828,88 +236,15 @@ dagx is ideal for:
- **Workflow engines** with typed data flow between stages - **Workflow engines** with typed data flow between stages
- **ETL processes** with validation and transformation steps - **ETL processes** with validation and transformation steps
## Performance ## Benchmarks
dagx provides true parallel execution with sub-microsecond overhead: Run the full benchmark suite:
- **Adaptive execution**: Inline fast-path for sequential work, true parallelism for concurrent work
- **Tasks spawn to multiple threads** via your async runtime's spawner when beneficial
- **Linear scaling** verified up to 10,000+ tasks
- **~0.78µs overhead per task** for sequential workloads (inline execution)
- **~1.2µs overhead per task** for parallel workloads
- **Efficient memory usage** at ~200 bytes per task
- **Zero-cost abstractions** through generics and compile-time monomorphization
- **4-148x faster than dagrs** across all benchmark patterns (see comparison benchmarks above)
### Performance Characteristics
- ✅ **Exceptional for sequential chains**: 10-148x faster than dagrs via inline execution
- ✅ **Excellent for parallel workloads**: 4-10x faster than dagrs on fan-out, ETL, large-scale
- ✅ **Best-in-class for mixed parallelism**: Automatically optimizes execution strategy per layer
- ✅ **Sub-microsecond per-task overhead**: Fast enough that framework cost is negligible
**Result**: dagx dominates dagrs on every benchmark pattern. No trade-offs, no compromises.
### Automatic Arc Wrapping (No Manual Arc Needed!)
**Task outputs are automatically wrapped in `Arc<T>` internally** for efficient fan-out patterns. You just output `T` - the framework handles the Arc wrapping:
```rust
use dagx::{task, DagRunner, Task};
// ✅ CORRECT: Just output Vec<String>, framework handles Arc internally
struct FetchData;
#[task]
impl FetchData {
async fn run() -> Vec<String> {
vec!["data".to_string(); 10_000]
}
}
// Downstream tasks receive &Vec<String>
// Internally, Arc<Vec<String>> is cloned (cheap), then inner Vec is extracted
struct ProcessData;
#[task]
impl ProcessData {
async fn run(data: &Vec<String>) -> usize {
data.len()
}
}
```
**How it works:**
- Your task outputs `T`
- Framework wraps it in `Arc<T>` internally
- For fan-out (1→N), Arc is cloned N times (just pointer copies - O(1))
- Each downstream task receives `&T` after extracting from Arc
**Performance characteristics:**
- **Heap types** (Vec, String, HashMap): Arc overhead is negligible, fanout is essentially free
- **Copy types** (i32, usize): Small Arc overhead (~few ns) due to atomic refcounting
- See `cargo bench` for actual measurements
**Advanced - Zero-copy optimization:**
If you want true zero-copy sharing (no extraction), output `Arc<T>` explicitly:
```rust
// Your task outputs Arc<T>
async fn run() -> Arc<Vec<String>> { Arc::new(vec![]) }
// Downstream receives &Arc<T> - just clones the Arc pointer
async fn process(data: &Arc<Vec<String>>) -> usize { data.len() }
```
This becomes `Arc<Arc<T>>` internally, but ExtractInput unwraps one layer automatically.
Run benchmarks:
```bash ```bash
cargo bench cargo bench
``` ```
View the detailed HTML reports: View detailed HTML reports:
```bash ```bash
# macOS # macOS
@@ -920,22 +255,22 @@ xdg-open target/criterion/report/index.html
# Windows # Windows
start target/criterion/report/index.html start target/criterion/report/index.html
# Or manually open target/criterion/report/index.html in your browser
``` ```
## Documentation _Benchmarks run on Intel i9-13950HX @ 5.5GHz._
Full API documentation is available at [docs.rs/dagx](https://docs.rs/dagx). ## Code of Conduct
## License This project follows the [Builder's Code of Conduct](https://builderscode.org).
Licensed under the MIT License. See [LICENSE](LICENSE) for details.
Copyright (c) 2025 Stephen Waits <steve@waits.net>
## Contributing ## Contributing
Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines. Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
For security issues, see [SECURITY.md](SECURITY.md). For security issues, see [SECURITY.md](SECURITY.md).
## License
Licensed under the MIT License. See [LICENSE](LICENSE) for details.
Copyright (c) 2025 Stephen Waits <steve@waits.net>
+9 -13
View File
@@ -8,7 +8,7 @@ dagx is a task execution library designed for safety and predictability. This do
| Version | Supported | | Version | Supported |
| ------- | ------------------ | | ------- | ------------------ |
| 0.1.x | :white_check_mark: | | 0.3.x | :white_check_mark: |
## Security Model ## Security Model
@@ -17,11 +17,10 @@ dagx is a task execution library designed for safety and predictability. This do
- **Compile-time type safety**: Dependencies are validated at compile time, preventing type errors - **Compile-time type safety**: Dependencies are validated at compile time, preventing type errors
- **Cycle detection**: Runtime detection of circular dependencies - **Cycle detection**: Runtime detection of circular dependencies
- **Memory safety**: Rust's ownership system prevents memory corruption - **Memory safety**: Rust's ownership system prevents memory corruption
- **Thread safety**: Interior mutability uses `parking_lot::Mutex` for safe concurrent access - **Panic isolation**: Panics in any task will be caught and not propagated through the executor
### What dagx Does NOT Provide ### What dagx Does NOT Provide
- **Panic isolation**: Panics in one task will propagate and may affect other tasks (see Panic Handling below)
- **Resource limits**: No built-in timeout, memory limits, or CPU throttling - **Resource limits**: No built-in timeout, memory limits, or CPU throttling
- **Sandboxing**: Tasks run with full process privileges - **Sandboxing**: Tasks run with full process privileges
- **Authentication/Authorization**: No access control between tasks - **Authentication/Authorization**: No access control between tasks
@@ -31,15 +30,14 @@ dagx is a task execution library designed for safety and predictability. This do
### 1. Panic Handling ### 1. Panic Handling
**Current Behavior**: dagx does NOT isolate panics between tasks. If a task panics: **Current Behavior**: dagx isolates panics between tasks. If a task panics:
- The panic will propagate through the executor - The panic will be caught and converted to a `TaskPanicked` error
- Other tasks in the same execution layer may or may not complete (timing-dependent) - Other tasks in the same execution layer will complete
- The entire DAG execution will terminate - The entire DAG execution will terminate after layer execution has finished and return **only the first panic**
**Recommendation**: **Recommendation**:
- Use `std::panic::catch_unwind` in tasks that may panic
- Return `Result<T, E>` from tasks and handle errors explicitly - Return `Result<T, E>` from tasks and handle errors explicitly
- Test panic scenarios thoroughly - Test panic scenarios thoroughly
@@ -250,10 +248,9 @@ impl MonitoredTask {
### Known Limitations ### Known Limitations
1. **No panic recovery**: Task panics will crash the DAG execution 1. **No resource limits**: Tasks can consume unlimited CPU, memory, or time
2. **No resource limits**: Tasks can consume unlimited CPU, memory, or time 2. **No isolation**: All tasks run in the same process with same privileges
3. **No isolation**: All tasks run in the same process with same privileges 3. **Shared executor**: Tasks compete for executor resources
4. **Shared executor**: Tasks compete for executor resources
### Not Security Features ### Not Security Features
@@ -303,7 +300,6 @@ Security updates will be:
- [Rust Security Guidelines](https://anssi-fr.github.io/rust-guide/) - [Rust Security Guidelines](https://anssi-fr.github.io/rust-guide/)
- [OWASP Secure Coding Practices](https://owasp.org/www-project-secure-coding-practices-quick-reference-guide/) - [OWASP Secure Coding Practices](https://owasp.org/www-project-secure-coding-practices-quick-reference-guide/)
- [async-std Security](https://async.rs/security)
- [Tokio Security](https://tokio.rs/tokio/topics/security) - [Tokio Security](https://tokio.rs/tokio/topics/security)
## License ## License
+2 -2
View File
@@ -1,7 +1,7 @@
//! DAG creation benchmarks //! DAG creation benchmarks
use criterion::Criterion; use criterion::Criterion;
use dagx::{DagRunner, Task}; use dagx::DagRunner;
use std::hint::black_box; use std::hint::black_box;
// Simple task for benchmarking // Simple task for benchmarking
@@ -21,7 +21,7 @@ pub fn bench_dag_creation(c: &mut Criterion) {
c.bench_function("add_100_tasks", |b| { c.bench_function("add_100_tasks", |b| {
b.iter(|| { b.iter(|| {
let dag = DagRunner::new(); let mut dag = DagRunner::new();
for i in 0..100 { for i in 0..100 {
black_box(dag.add_task(Value(i))); black_box(dag.add_task(Value(i)));
} }
-122
View File
@@ -1,122 +0,0 @@
//! Basic DAG execution benchmarks
use criterion::Criterion;
use dagx::{task_fn, DagRunner, Task};
pub fn bench_dag_execution(c: &mut Criterion) {
let rt = tokio::runtime::Runtime::new().unwrap();
c.bench_function("execute_10_independent_tasks", |b| {
b.iter(|| {
rt.block_on(async {
let dag = DagRunner::new();
for i in 0..10 {
dag.add_task(task_fn(move |_: ()| async move { i }));
}
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
})
});
});
c.bench_function("execute_100_independent_tasks", |b| {
b.iter(|| {
rt.block_on(async {
let dag = DagRunner::new();
for i in 0..100 {
dag.add_task(task_fn(move |_: ()| async move { i }));
}
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
})
});
});
c.bench_function("execute_deep_chain_20", |b| {
b.iter(|| {
rt.block_on(async {
let dag = DagRunner::new();
// Build a manual chain (API doesn't support loop-based chaining due to type system)
struct Value(i32);
#[dagx::task]
impl Value {
async fn run(&self) -> i32 {
self.0
}
}
let t0 = dag.add_task(Value(0));
let t1 = dag
.add_task(task_fn(|x: i32| async move { x + 1 }))
.depends_on(&t0);
let t2 = dag
.add_task(task_fn(|x: i32| async move { x + 1 }))
.depends_on(t1);
let t3 = dag
.add_task(task_fn(|x: i32| async move { x + 1 }))
.depends_on(t2);
let t4 = dag
.add_task(task_fn(|x: i32| async move { x + 1 }))
.depends_on(t3);
let t5 = dag
.add_task(task_fn(|x: i32| async move { x + 1 }))
.depends_on(t4);
let t6 = dag
.add_task(task_fn(|x: i32| async move { x + 1 }))
.depends_on(t5);
let t7 = dag
.add_task(task_fn(|x: i32| async move { x + 1 }))
.depends_on(t6);
let t8 = dag
.add_task(task_fn(|x: i32| async move { x + 1 }))
.depends_on(t7);
let t9 = dag
.add_task(task_fn(|x: i32| async move { x + 1 }))
.depends_on(t8);
let t10 = dag
.add_task(task_fn(|x: i32| async move { x + 1 }))
.depends_on(t9);
let t11 = dag
.add_task(task_fn(|x: i32| async move { x + 1 }))
.depends_on(t10);
let t12 = dag
.add_task(task_fn(|x: i32| async move { x + 1 }))
.depends_on(t11);
let t13 = dag
.add_task(task_fn(|x: i32| async move { x + 1 }))
.depends_on(t12);
let t14 = dag
.add_task(task_fn(|x: i32| async move { x + 1 }))
.depends_on(t13);
let t15 = dag
.add_task(task_fn(|x: i32| async move { x + 1 }))
.depends_on(t14);
let t16 = dag
.add_task(task_fn(|x: i32| async move { x + 1 }))
.depends_on(t15);
let t17 = dag
.add_task(task_fn(|x: i32| async move { x + 1 }))
.depends_on(t16);
let t18 = dag
.add_task(task_fn(|x: i32| async move { x + 1 }))
.depends_on(t17);
let _t19 = dag
.add_task(task_fn(|x: i32| async move { x + 1 }))
.depends_on(t18);
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
})
});
});
}
-1
View File
@@ -1,5 +1,4 @@
//! Basic DAG benchmarks - creation, execution, scaling //! Basic DAG benchmarks - creation, execution, scaling
pub mod creation; pub mod creation;
pub mod execution;
pub mod scaling; pub mod scaling;
+6 -6
View File
@@ -1,7 +1,8 @@
//! DAG scaling benchmarks - how performance scales with task count //! DAG scaling benchmarks - how performance scales with task count
use criterion::{BenchmarkId, Criterion}; use criterion::{BenchmarkId, Criterion};
use dagx::{task_fn, DagRunner}; use dagx::DagRunner;
use dagx_test::task_fn;
pub fn bench_dag_scaling(c: &mut Criterion) { pub fn bench_dag_scaling(c: &mut Criterion) {
let rt = tokio::runtime::Runtime::new().unwrap(); let rt = tokio::runtime::Runtime::new().unwrap();
@@ -14,13 +15,12 @@ pub fn bench_dag_scaling(c: &mut Criterion) {
group.bench_with_input(BenchmarkId::from_parameter(size), size, |b, &size| { group.bench_with_input(BenchmarkId::from_parameter(size), size, |b, &size| {
b.iter(|| { b.iter(|| {
rt.block_on(async { rt.block_on(async {
let dag = DagRunner::new(); let mut dag = DagRunner::new();
for i in 0..size { for i in 0..size {
dag.add_task(task_fn(move |_: ()| async move { i })); dag.add_task(task_fn::<(), _, _>(move |_: ()| i));
} }
dag.run(|fut| { let _output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await .await
.unwrap(); .unwrap();
}) })
+11 -11
View File
@@ -6,33 +6,33 @@ use criterion::Criterion;
pub fn bench_deep_chain(c: &mut Criterion) { pub fn bench_deep_chain(c: &mut Criterion) {
let rt = tokio::runtime::Runtime::new().unwrap(); let rt = tokio::runtime::Runtime::new().unwrap();
let mut group = c.benchmark_group("deep_chain_100_tasks"); let mut group = c.benchmark_group("comparisons/deep_chain_100_tasks");
// dagx implementation // dagx implementation
group.bench_function("dagx", |b| { group.bench_function("dagx", |b| {
b.iter(|| { b.iter(|| {
rt.block_on(async { rt.block_on(async {
use dagx::{task_fn, DagRunner}; use dagx::DagRunner;
use dagx_test::task_fn;
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let first = dag.add_task(task_fn(|_: ()| async { 0 })); let first = dag.add_task(task_fn::<(), _, _>(|_: ()| 0));
let mut prev = dag let mut prev = dag
.add_task(task_fn(|x: i32| async move { x + 1 })) .add_task(task_fn::<i32, _, _>(|&x: &i32| x + 1))
.depends_on(&first); .depends_on(&first);
for i in 2..100 { for i in 2..100 {
prev = dag prev = dag
.add_task(task_fn(move |x: i32| async move { x + i })) .add_task(task_fn::<i32, _, _>(move |&x: &i32| x + i))
.depends_on(prev); .depends_on(&prev);
} }
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await .await
.unwrap(); .unwrap();
dag.get(prev).unwrap() output.get(prev)
}) })
}); });
}); });
+11 -11
View File
@@ -6,33 +6,33 @@ use criterion::Criterion;
pub fn bench_diamond_pattern(c: &mut Criterion) { pub fn bench_diamond_pattern(c: &mut Criterion) {
let rt = tokio::runtime::Runtime::new().unwrap(); let rt = tokio::runtime::Runtime::new().unwrap();
let mut group = c.benchmark_group("diamond_pattern"); let mut group = c.benchmark_group("comparisons/diamond_pattern");
// dagx implementation // dagx implementation
group.bench_function("dagx", |b| { group.bench_function("dagx", |b| {
b.iter(|| { b.iter(|| {
rt.block_on(async { rt.block_on(async {
use dagx::{task_fn, DagRunner}; use dagx::DagRunner;
use dagx_test::task_fn;
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let a = dag.add_task(task_fn(|_: ()| async { 10 })); let a = dag.add_task(task_fn::<(), _, _>(|_: ()| 10));
let b = dag let b = dag
.add_task(task_fn(|x: i32| async move { x * 2 })) .add_task(task_fn::<i32, _, _>(|&x: &i32| x * 2))
.depends_on(&a); .depends_on(&a);
let c = dag let c = dag
.add_task(task_fn(|x: i32| async move { x + 5 })) .add_task(task_fn::<i32, _, _>(|&x: &i32| x + 5))
.depends_on(&a); .depends_on(&a);
let d = dag let d = dag
.add_task(task_fn(|(x, y): (i32, i32)| async move { x + y })) .add_task(task_fn::<(i32, i32), _, _>(|(x, y): (&i32, &i32)| x + y))
.depends_on((&b, &c)); .depends_on((&b, &c));
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await .await
.unwrap(); .unwrap();
dag.get(d).unwrap() output.get(d)
}) })
}); });
}); });
+28 -25
View File
@@ -6,45 +6,46 @@ use criterion::Criterion;
pub fn bench_etl_pipeline(c: &mut Criterion) { pub fn bench_etl_pipeline(c: &mut Criterion) {
let rt = tokio::runtime::Runtime::new().unwrap(); let rt = tokio::runtime::Runtime::new().unwrap();
let mut group = c.benchmark_group("etl_pipeline"); let mut group = c.benchmark_group("comparisons/etl_pipeline");
// dagx implementation // dagx implementation
group.bench_function("dagx", |b| { group.bench_function("dagx", |b| {
b.iter(|| { b.iter(|| {
rt.block_on(async { rt.block_on(async {
use dagx::{task_fn, DagRunner}; use dagx::DagRunner;
use dagx_test::task_fn;
let dag = DagRunner::new(); let mut dag = DagRunner::new();
// Extract from 3 sources // Extract from 3 sources
let source1 = dag.add_task(task_fn(|_: ()| async { vec![1, 2, 3, 4, 5] })); let source1 = dag.add_task(task_fn::<(), _, _>(|_: ()| vec![1, 2, 3, 4, 5]));
let source2 = dag.add_task(task_fn(|_: ()| async { vec![6, 7, 8, 9, 10] })); let source2 = dag.add_task(task_fn::<(), _, _>(|_: ()| vec![6, 7, 8, 9, 10]));
let source3 = dag.add_task(task_fn(|_: ()| async { vec![11, 12, 13, 14, 15] })); let source3 = dag.add_task(task_fn::<(), _, _>(|_: ()| vec![11, 12, 13, 14, 15]));
// Transform each source // Transform each source
let transform1 = dag let transform1 = dag
.add_task(task_fn(|data: Vec<i32>| async move { .add_task(task_fn::<Vec<_>, _, _>(|data: &Vec<i32>| {
data.into_iter().map(|x| x * 2).collect::<Vec<_>>() data.iter().map(|x| x * 2).collect::<Vec<_>>()
})) }))
.depends_on(&source1); .depends_on(&source1);
let transform2 = dag let transform2 = dag
.add_task(task_fn(|data: Vec<i32>| async move { .add_task(task_fn::<Vec<_>, _, _>(|data: &Vec<i32>| {
data.into_iter().map(|x| x * 2).collect::<Vec<_>>() data.iter().map(|x| x * 2).collect::<Vec<_>>()
})) }))
.depends_on(&source2); .depends_on(&source2);
let transform3 = dag let transform3 = dag
.add_task(task_fn(|data: Vec<i32>| async move { .add_task(task_fn::<Vec<_>, _, _>(|data: &Vec<i32>| {
data.into_iter().map(|x| x * 2).collect::<Vec<_>>() data.iter().map(|x| x * 2).collect::<Vec<_>>()
})) }))
.depends_on(&source3); .depends_on(&source3);
// Merge all transformed data // Merge all transformed data
let merge = dag let merge = dag
.add_task(task_fn( .add_task(task_fn::<(Vec<_>, Vec<_>, Vec<_>), _, _>(
|(d1, d2, d3): (Vec<i32>, Vec<i32>, Vec<i32>)| async move { |(d1, d2, d3): (&Vec<i32>, &Vec<i32>, &Vec<i32>)| {
let mut result = d1; let mut result = d1.clone();
result.extend(d2); result.extend(d2);
result.extend(d3); result.extend(d3);
result result
@@ -54,24 +55,26 @@ pub fn bench_etl_pipeline(c: &mut Criterion) {
// Validate // Validate
let validate = dag let validate = dag
.add_task(task_fn(|data: Vec<i32>| async move { .add_task(task_fn::<Vec<_>, _, _>(|data: &Vec<i32>| {
data.into_iter().filter(|&x| x > 0).collect::<Vec<_>>() data.iter()
.copied()
.filter(|&x| x > 0)
.collect::<Vec<i32>>()
})) }))
.depends_on(merge); .depends_on(&merge);
// Load (compute sum) // Load (compute sum)
let load = dag let load = dag
.add_task(task_fn(|data: Vec<i32>| async move { .add_task(task_fn::<Vec<_>, _, _>(|data: &Vec<i32>| {
data.into_iter().sum::<i32>() data.iter().sum::<i32>()
})) }))
.depends_on(validate); .depends_on(&validate);
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await .await
.unwrap(); .unwrap();
dag.get(load).unwrap() output.get(load)
}) })
}); });
}); });
+7 -7
View File
@@ -6,7 +6,7 @@ use criterion::Criterion;
pub fn bench_large_scale(c: &mut Criterion) { pub fn bench_large_scale(c: &mut Criterion) {
let rt = tokio::runtime::Runtime::new().unwrap(); let rt = tokio::runtime::Runtime::new().unwrap();
let mut group = c.benchmark_group("large_scale_10k_tasks"); let mut group = c.benchmark_group("comparisons/large_scale_10k_tasks");
group.sample_size(10); group.sample_size(10);
group.measurement_time(std::time::Duration::from_secs(20)); group.measurement_time(std::time::Duration::from_secs(20));
@@ -14,16 +14,16 @@ pub fn bench_large_scale(c: &mut Criterion) {
group.bench_function("dagx", |b| { group.bench_function("dagx", |b| {
b.iter(|| { b.iter(|| {
rt.block_on(async { rt.block_on(async {
use dagx::{task_fn, DagRunner}; use dagx::DagRunner;
use dagx_test::task_fn;
let dag = DagRunner::new(); let mut dag = DagRunner::new();
for i in 0..10_000 { for i in 0..10_000 {
dag.add_task(task_fn(move |_: ()| async move { i * 2 })); dag.add_task(task_fn::<(), _, _>(move |_: ()| i * 2));
} }
dag.run(|fut| { let _output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await .await
.unwrap(); .unwrap();
}) })
+15 -15
View File
@@ -6,36 +6,36 @@ use criterion::Criterion;
pub fn bench_linear_pipeline(c: &mut Criterion) { pub fn bench_linear_pipeline(c: &mut Criterion) {
let rt = tokio::runtime::Runtime::new().unwrap(); let rt = tokio::runtime::Runtime::new().unwrap();
let mut group = c.benchmark_group("linear_pipeline_5_tasks"); let mut group = c.benchmark_group("comparisons/linear_pipeline_5_tasks");
// dagx implementation // dagx implementation
group.bench_function("dagx", |b| { group.bench_function("dagx", |b| {
b.iter(|| { b.iter(|| {
rt.block_on(async { rt.block_on(async {
use dagx::{task_fn, DagRunner}; use dagx::DagRunner;
use dagx_test::task_fn;
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let a = dag.add_task(task_fn(|_: ()| async { 1 })); let a = dag.add_task(task_fn::<(), _, _>(|_: ()| 1));
let b = dag let b = dag
.add_task(task_fn(|x: i32| async move { x + 1 })) .add_task(task_fn::<i32, _, _>(|&x: &i32| x + 1))
.depends_on(&a); .depends_on(&a);
let c = dag let c = dag
.add_task(task_fn(|x: i32| async move { x * 2 })) .add_task(task_fn::<i32, _, _>(|&x: &i32| x * 2))
.depends_on(b); .depends_on(&b);
let d = dag let d = dag
.add_task(task_fn(|x: i32| async move { x - 1 })) .add_task(task_fn::<i32, _, _>(|&x: &i32| x - 1))
.depends_on(c); .depends_on(&c);
let e = dag let e = dag
.add_task(task_fn(|x: i32| async move { x * 3 })) .add_task(task_fn::<i32, _, _>(|&x: &i32| x * 3))
.depends_on(d); .depends_on(&d);
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await .await
.unwrap(); .unwrap();
dag.get(e).unwrap() output.get(e)
}) })
}); });
}); });
+8 -8
View File
@@ -6,26 +6,26 @@ use criterion::Criterion;
pub fn bench_wide_fanout(c: &mut Criterion) { pub fn bench_wide_fanout(c: &mut Criterion) {
let rt = tokio::runtime::Runtime::new().unwrap(); let rt = tokio::runtime::Runtime::new().unwrap();
let mut group = c.benchmark_group("wide_fanout_100_tasks"); let mut group = c.benchmark_group("comparisons/wide_fanout_100_tasks");
// dagx implementation // dagx implementation
group.bench_function("dagx", |b| { group.bench_function("dagx", |b| {
b.iter(|| { b.iter(|| {
rt.block_on(async { rt.block_on(async {
use dagx::{task_fn, DagRunner}; use dagx::DagRunner;
use dagx_test::task_fn;
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let source = dag.add_task(task_fn(|_: ()| async { 42 })); let source = dag.add_task(task_fn::<(), _, _>(|_: ()| 42));
// Create 100 tasks that all depend on source // Create 100 tasks that all depend on source
for i in 0..100 { for i in 0..100 {
dag.add_task(task_fn(move |x: i32| async move { x + i })) dag.add_task(task_fn::<i32, _, _>(move |&x: &i32| x + i))
.depends_on(&source); .depends_on(&source);
} }
dag.run(|fut| { let _output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await .await
.unwrap(); .unwrap();
}) })
-8
View File
@@ -28,12 +28,6 @@ criterion_group! {
targets = basic::creation::bench_dag_creation targets = basic::creation::bench_dag_creation
} }
criterion_group! {
name = execution_benches;
config = configure_criterion();
targets = basic::execution::bench_dag_execution
}
criterion_group! { criterion_group! {
name = scaling_benches; name = scaling_benches;
config = configure_criterion(); config = configure_criterion();
@@ -54,7 +48,6 @@ criterion_group! {
config = configure_criterion(); config = configure_criterion();
targets = targets =
realistic::etl::bench_etl_pipeline, realistic::etl::bench_etl_pipeline,
realistic::config_broadcast::bench_config_broadcast,
realistic::memory_efficiency::bench_memory_efficiency, realistic::memory_efficiency::bench_memory_efficiency,
realistic::mixed::bench_mixed_patterns realistic::mixed::bench_mixed_patterns
} }
@@ -73,7 +66,6 @@ criterion_group! {
criterion_main!( criterion_main!(
creation_benches, creation_benches,
execution_benches,
scaling_benches, scaling_benches,
pattern_benches, pattern_benches,
realistic_benches, realistic_benches,
+21 -18
View File
@@ -1,7 +1,8 @@
//! 10,000 task breakdown benchmarks - construction vs execution //! 10,000 task breakdown benchmarks - construction vs execution
use criterion::Criterion; use criterion::{BatchSize, Criterion};
use dagx::{task_fn, DagRunner}; use dagx::DagRunner;
use dagx_test::task_fn;
pub fn bench_10k_breakdown(c: &mut Criterion) { pub fn bench_10k_breakdown(c: &mut Criterion) {
let rt = tokio::runtime::Runtime::new().unwrap(); let rt = tokio::runtime::Runtime::new().unwrap();
@@ -11,9 +12,9 @@ pub fn bench_10k_breakdown(c: &mut Criterion) {
// a) Adding 10k tasks (construction only) // a) Adding 10k tasks (construction only)
group.bench_function("construction", |b| { group.bench_function("construction", |b| {
b.iter(|| { b.iter(|| {
let dag = DagRunner::new(); let mut dag = DagRunner::new();
for i in 0..10_000 { for i in 0..10_000 {
dag.add_task(task_fn(move |_: ()| async move { i })); dag.add_task(task_fn::<(), _, _>(move |_: ()| i));
} }
}); });
}); });
@@ -22,13 +23,12 @@ pub fn bench_10k_breakdown(c: &mut Criterion) {
group.bench_function("full_execution", |b| { group.bench_function("full_execution", |b| {
b.iter(|| { b.iter(|| {
rt.block_on(async { rt.block_on(async {
let dag = DagRunner::new(); let mut dag = DagRunner::new();
for i in 0..10_000 { for i in 0..10_000 {
dag.add_task(task_fn(move |_: ()| async move { i })); dag.add_task(task_fn::<(), _, _>(move |_: ()| i));
} }
dag.run(|fut| { let _output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await .await
.unwrap(); .unwrap();
}) })
@@ -37,21 +37,24 @@ pub fn bench_10k_breakdown(c: &mut Criterion) {
// c) Execution only (pre-built DAG, just run) // c) Execution only (pre-built DAG, just run)
group.bench_function("execution_only", |b| { group.bench_function("execution_only", |b| {
// Pre-build the DAG outside the measurement b.iter_batched(
let dag = DagRunner::new(); || {
let mut dag = DagRunner::new();
for i in 0..10_000 { for i in 0..10_000 {
dag.add_task(task_fn(move |_: ()| async move { i })); dag.add_task(task_fn::<(), _, _>(move |_: ()| i));
} }
dag
b.iter(|| { },
|dag| {
rt.block_on(async { rt.block_on(async {
dag.run(|fut| { let _output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await .await
.unwrap(); .unwrap();
}) })
}); },
BatchSize::LargeInput,
);
}); });
group.finish(); group.finish();
+9 -10
View File
@@ -1,33 +1,32 @@
//! Diamond pattern benchmarks (fan-out then fan-in) //! Diamond pattern benchmarks (fan-out then fan-in)
use criterion::Criterion; use criterion::Criterion;
use dagx::{task_fn, DagRunner}; use dagx::DagRunner;
use dagx_test::task_fn;
pub fn bench_diamond(c: &mut Criterion) { pub fn bench_diamond(c: &mut Criterion) {
let rt = tokio::runtime::Runtime::new().unwrap(); let rt = tokio::runtime::Runtime::new().unwrap();
c.bench_function("diamond_pattern_50", |b| { c.bench_function("diamond_pattern_50", |b| {
b.iter(|| { b.iter(|| {
rt.block_on(async { rt.block_on(async {
let dag = DagRunner::new(); let mut dag = DagRunner::new();
for i in 0..50 { for i in 0..50 {
let source = dag.add_task(task_fn(move |_: ()| async move { i })); let source = dag.add_task(task_fn::<(), _, _>(move |_: ()| i));
let left = dag let left = dag
.add_task(task_fn(|x: i32| async move { x * 2 })) .add_task(task_fn::<i32, _, _>(|&x: &i32| x * 2))
.depends_on(&source); .depends_on(&source);
let right = dag let right = dag
.add_task(task_fn(|x: i32| async move { x * 3 })) .add_task(task_fn::<i32, _, _>(|&x: &i32| x * 3))
.depends_on(&source); .depends_on(&source);
dag.add_task(task_fn(|(l, r): (i32, i32)| async move { l + r })) dag.add_task(task_fn::<(i32, i32), _, _>(|(l, r): (&i32, &i32)| l + r))
.depends_on((&left, &right)); .depends_on((&left, &right));
} }
dag.run(|fut| { let _output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await .await
.unwrap(); .unwrap();
}) })
+12 -41
View File
@@ -1,8 +1,8 @@
//! Fan-out pattern benchmarks (1 → N dependencies) //! Fan-out pattern benchmarks (1 → N dependencies)
use criterion::Criterion; use criterion::Criterion;
use dagx::{task_fn, DagRunner}; use dagx::DagRunner;
use dagx_test::task_fn;
pub fn bench_fanout(c: &mut Criterion) { pub fn bench_fanout(c: &mut Criterion) {
let rt = tokio::runtime::Runtime::new().unwrap(); let rt = tokio::runtime::Runtime::new().unwrap();
@@ -10,17 +10,16 @@ pub fn bench_fanout(c: &mut Criterion) {
c.bench_function("fanout_1_to_100_i32", |b| { c.bench_function("fanout_1_to_100_i32", |b| {
b.iter(|| { b.iter(|| {
rt.block_on(async { rt.block_on(async {
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let source = dag.add_task(task_fn(|_: ()| async { 42 })); let source = dag.add_task(task_fn::<(), _, _>(|_: ()| 42));
for i in 0..100 { for i in 0..100 {
dag.add_task(task_fn(move |x: i32| async move { x + i })) dag.add_task(task_fn::<i32, _, _>(move |&x: &i32| x + i))
.depends_on(&source); .depends_on(&source);
} }
dag.run(|fut| { let _output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await .await
.unwrap(); .unwrap();
}) })
@@ -31,50 +30,22 @@ pub fn bench_fanout(c: &mut Criterion) {
c.bench_function("fanout_1_to_100_vec", |b| { c.bench_function("fanout_1_to_100_vec", |b| {
b.iter(|| { b.iter(|| {
rt.block_on(async { rt.block_on(async {
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let source = dag.add_task(task_fn(|_: ()| async { let source = dag.add_task(task_fn::<(), _, _>(|_: ()| {
// Create 1000 strings - framework wraps in Arc automatically // Create 1000 strings - framework wraps in Arc automatically
(0..1000).map(|i| format!("Item {}", i)).collect::<Vec<_>>() (0..1000).map(|i| format!("Item {}", i)).collect::<Vec<_>>()
})); }));
for i in 0..100 { for i in 0..100 {
dag.add_task(task_fn(move |data: Vec<String>| async move { dag.add_task(task_fn::<Vec<_>, _, _>(move |data: &Vec<String>| {
// Each consumer gets data extracted from Arc // Each consumer gets data extracted from Arc
data.iter().filter(|s| s.contains(&i.to_string())).count() data.iter().filter(|s| s.contains(&i.to_string())).count()
})) }))
.depends_on(&source); .depends_on(&source);
} }
dag.run(|fut| { let _output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await
.unwrap();
})
});
});
// Arc overhead benchmark - shows cost on Copy types
// For Copy types like usize, Arc adds overhead (atomic refcounting + indirection)
// For heap types like Vec, Arc overhead is negligible compared to clone cost
c.bench_function("arc_overhead_copy_type", |b| {
b.iter(|| {
rt.block_on(async {
let dag = DagRunner::new();
// Source produces a simple Copy type (usize)
// Framework wraps in Arc<usize> internally
let source = dag.add_task(task_fn(|_: ()| async { 42usize }));
// 100 dependents - Arc overhead visible on Copy types
for i in 0..100 {
dag.add_task(task_fn(move |x: usize| async move { x + i }))
.depends_on(&source);
}
dag.run(|fut| {
tokio::spawn(fut);
})
.await .await
.unwrap(); .unwrap();
}) })
-66
View File
@@ -1,66 +0,0 @@
//! Configuration broadcast pattern benchmark - one config shared across many workers
use criterion::{BenchmarkId, Criterion};
use dagx::{task_fn, DagRunner};
use std::collections::HashMap;
use std::hint::black_box;
pub fn bench_config_broadcast(c: &mut Criterion) {
let rt = tokio::runtime::Runtime::new().unwrap();
let mut group = c.benchmark_group("config_broadcast");
for worker_count in [10, 50, 100, 200].iter() {
group.bench_with_input(
BenchmarkId::new("config", worker_count),
worker_count,
|b, &workers| {
b.iter(|| {
rt.block_on(async {
let dag = DagRunner::new();
// Configuration source - framework wraps HashMap in Arc automatically
let config = dag.add_task(task_fn(|_: ()| async {
HashMap::from([
("max_connections".to_string(), "100".to_string()),
("timeout_ms".to_string(), "5000".to_string()),
("batch_size".to_string(), "50".to_string()),
("retry_count".to_string(), "3".to_string()),
("worker_threads".to_string(), "4".to_string()),
("buffer_size".to_string(), "8192".to_string()),
("compression".to_string(), "true".to_string()),
("log_level".to_string(), "info".to_string()),
])
}));
// Workers receive and process based on shared config
for worker_id in 0..workers {
dag.add_task(task_fn(move |cfg: HashMap<String, String>| async move {
// Simulate work based on configuration
let batch_size: usize = cfg
.get("batch_size")
.and_then(|s| s.parse().ok())
.unwrap_or(10);
let timeout: usize = cfg
.get("timeout_ms")
.and_then(|s| s.parse().ok())
.unwrap_or(1000);
// Simulate some computation
black_box(worker_id * batch_size + timeout / 100)
}))
.depends_on(&config);
}
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
})
});
},
);
}
group.finish();
}
+11 -12
View File
@@ -1,8 +1,8 @@
//! ETL (Extract, Transform, Load) pipeline benchmark //! ETL (Extract, Transform, Load) pipeline benchmark
use criterion::{BenchmarkId, Criterion}; use criterion::{BenchmarkId, Criterion};
use dagx::{task_fn, DagRunner}; use dagx::DagRunner;
use dagx_test::task_fn;
pub fn bench_etl_pipeline(c: &mut Criterion) { pub fn bench_etl_pipeline(c: &mut Criterion) {
let rt = tokio::runtime::Runtime::new().unwrap(); let rt = tokio::runtime::Runtime::new().unwrap();
let mut group = c.benchmark_group("etl_scaling"); let mut group = c.benchmark_group("etl_scaling");
@@ -17,10 +17,10 @@ pub fn bench_etl_pipeline(c: &mut Criterion) {
|b, &count| { |b, &count| {
b.iter(|| { b.iter(|| {
rt.block_on(async { rt.block_on(async {
let dag = DagRunner::new(); let mut dag = DagRunner::new();
// Extract: Load JSON-like data (framework wraps in Arc automatically) // Extract: Load JSON-like data (framework wraps in Arc automatically)
let extract = dag.add_task(task_fn(move |_: ()| async move { let extract = dag.add_task(task_fn::<(), _, _>(move |_: ()| {
(0..count) (0..count)
.map(|i| { .map(|i| {
format!( format!(
@@ -37,7 +37,7 @@ pub fn bench_etl_pipeline(c: &mut Criterion) {
// T1: Parse and validate // T1: Parse and validate
let validate = dag let validate = dag
.add_task(task_fn(move |data: Vec<String>| async move { .add_task(task_fn::<Vec<_>, _, _>(move |data: &Vec<String>| {
data.iter() data.iter()
.filter(|s| { .filter(|s| {
s.contains("\"id\"") s.contains("\"id\"")
@@ -50,7 +50,7 @@ pub fn bench_etl_pipeline(c: &mut Criterion) {
// T2: Extract scores // T2: Extract scores
let scores = dag let scores = dag
.add_task(task_fn(move |data: Vec<String>| async move { .add_task(task_fn::<Vec<_>, _, _>(move |data: &Vec<String>| {
data.iter() data.iter()
.filter_map(|s| { .filter_map(|s| {
s.split("\"score\":").nth(1).and_then(|part| { s.split("\"score\":").nth(1).and_then(|part| {
@@ -63,7 +63,7 @@ pub fn bench_etl_pipeline(c: &mut Criterion) {
// T3: Extract names // T3: Extract names
let names = dag let names = dag
.add_task(task_fn(move |data: Vec<String>| async move { .add_task(task_fn::<Vec<_>, _, _>(move |data: &Vec<String>| {
data.iter() data.iter()
.filter_map(|s| { .filter_map(|s| {
s.split("\"name\":\"") s.split("\"name\":\"")
@@ -76,8 +76,8 @@ pub fn bench_etl_pipeline(c: &mut Criterion) {
.depends_on(&extract); .depends_on(&extract);
// Load: Aggregate all transformations // Load: Aggregate all transformations
dag.add_task(task_fn( dag.add_task(task_fn::<(usize, Vec<_>, Vec<_>), _, _>(
move |(valid, scores, names): (usize, Vec<i32>, Vec<String>)| async move { move |(valid, scores, names): (&usize, &Vec<i32>, &Vec<String>)| {
let avg_score = let avg_score =
scores.iter().sum::<i32>() / scores.len().max(1) as i32; scores.iter().sum::<i32>() / scores.len().max(1) as i32;
format!( format!(
@@ -90,9 +90,8 @@ pub fn bench_etl_pipeline(c: &mut Criterion) {
)) ))
.depends_on((&validate, &scores, &names)); .depends_on((&validate, &scores, &names));
dag.run(|fut| { let _output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await .await
.unwrap(); .unwrap();
}) })
+8 -8
View File
@@ -1,7 +1,8 @@
//! Memory efficiency benchmark with large data structures //! Memory efficiency benchmark with large data structures
use criterion::{BenchmarkId, Criterion}; use criterion::{BenchmarkId, Criterion};
use dagx::{task_fn, DagRunner}; use dagx::DagRunner;
use dagx_test::task_fn;
use std::hint::black_box; use std::hint::black_box;
pub fn bench_memory_efficiency(c: &mut Criterion) { pub fn bench_memory_efficiency(c: &mut Criterion) {
@@ -9,23 +10,23 @@ pub fn bench_memory_efficiency(c: &mut Criterion) {
let mut group = c.benchmark_group("memory_efficiency"); let mut group = c.benchmark_group("memory_efficiency");
// Test with different data sizes (in MB) // Test with different data sizes (in MB)
for mb_size in [1, 5, 10].iter() { for mb_size in [1, 10].iter() {
group.bench_with_input( group.bench_with_input(
BenchmarkId::new("large_data_20_consumers", format!("{}MB", mb_size)), BenchmarkId::new("large_data_20_consumers", format!("{}MB", mb_size)),
mb_size, mb_size,
|b, &mb| { |b, &mb| {
b.iter(|| { b.iter(|| {
rt.block_on(async { rt.block_on(async {
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let source = dag.add_task(task_fn(move |_: ()| async move { let source = dag.add_task(task_fn::<(), _, _>(move |_: ()| {
// Create MB of data - framework wraps in Arc automatically // Create MB of data - framework wraps in Arc automatically
vec![0u8; mb * 1024 * 1024] vec![0u8; mb * 1024 * 1024]
})); }));
// 20 consumers share the data via automatic Arc wrapping // 20 consumers share the data via automatic Arc wrapping
for i in 0..20 { for i in 0..20 {
dag.add_task(task_fn(move |data: Vec<u8>| async move { dag.add_task(task_fn::<Vec<_>, _, _>(move |data: &Vec<u8>| {
// Process a small portion (data extracted from Arc) // Process a small portion (data extracted from Arc)
let sample_sum: u32 = data let sample_sum: u32 = data
.iter() .iter()
@@ -38,9 +39,8 @@ pub fn bench_memory_efficiency(c: &mut Criterion) {
.depends_on(&source); .depends_on(&source);
} }
dag.run(|fut| { let _output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await .await
.unwrap(); .unwrap();
}) })
+15 -14
View File
@@ -1,18 +1,18 @@
//! Mixed realistic patterns - fan-out, fan-in, and processing //! Mixed realistic patterns - fan-out, fan-in, and processing
use criterion::Criterion; use criterion::Criterion;
use dagx::{task_fn, DagRunner}; use dagx::DagRunner;
use dagx_test::task_fn;
pub fn bench_mixed_patterns(c: &mut Criterion) { pub fn bench_mixed_patterns(c: &mut Criterion) {
let rt = tokio::runtime::Runtime::new().unwrap(); let rt = tokio::runtime::Runtime::new().unwrap();
c.bench_function("mixed_patterns_realistic", |b| { c.bench_function("mixed_patterns_realistic", |b| {
b.iter(|| { b.iter(|| {
rt.block_on(async { rt.block_on(async {
let dag = DagRunner::new(); let mut dag = DagRunner::new();
// Stage 1: Load data (framework wraps in Arc automatically) // Stage 1: Load data (framework wraps in Arc automatically)
let data = dag.add_task(task_fn(|_: ()| async { let data = dag.add_task(task_fn::<(), _, _>(|_: ()| {
(0..1000) (0..1000)
.map(|i| format!("Record-{:04}", i)) .map(|i| format!("Record-{:04}", i))
.collect::<Vec<_>>() .collect::<Vec<_>>()
@@ -20,39 +20,40 @@ pub fn bench_mixed_patterns(c: &mut Criterion) {
// Stage 2: Fan-out for parallel analysis // Stage 2: Fan-out for parallel analysis
let analysis1 = dag let analysis1 = dag
.add_task(task_fn(|d: Vec<String>| async move { .add_task(task_fn::<Vec<_>, _, _>(|d: &Vec<String>| {
d.iter().filter(|s| s.contains("00")).count() d.iter().filter(|s| s.contains("00")).count()
})) }))
.depends_on(&data); .depends_on(&data);
let analysis2 = dag let analysis2 = dag
.add_task(task_fn(|d: Vec<String>| async move { .add_task(task_fn::<Vec<_>, _, _>(|d: &Vec<String>| {
d.iter().map(|s| s.len()).sum::<usize>() d.iter().map(|s| s.len()).sum::<usize>()
})) }))
.depends_on(&data); .depends_on(&data);
let analysis3 = dag let analysis3 = dag
.add_task(task_fn(|d: Vec<String>| async move { d.len() })) .add_task(task_fn::<Vec<_>, _, _>(|d: &Vec<String>| d.len()))
.depends_on(&data); .depends_on(&data);
// Stage 3: Fan-in to aggregate // Stage 3: Fan-in to aggregate
let summary = dag let summary = dag
.add_task(task_fn(|(a1, a2, a3): (usize, usize, usize)| async move { .add_task(task_fn::<(usize, usize, usize), _, _>(
|(a1, a2, a3): (&usize, &usize, &usize)| {
format!("Matches: {}, Total bytes: {}, Count: {}", a1, a2, a3) format!("Matches: {}, Total bytes: {}, Count: {}", a1, a2, a3)
})) },
))
.depends_on((&analysis1, &analysis2, &analysis3)); .depends_on((&analysis1, &analysis2, &analysis3));
// Stage 4: Fan-out final report to multiple destinations // Stage 4: Fan-out final report to multiple destinations
for i in 0..3 { for i in 0..3 {
dag.add_task(task_fn(move |report: String| async move { dag.add_task(task_fn::<String, _, _>(move |report: &String| {
format!("Destination {}: {}", i, report) format!("Destination {}: {}", i, report)
})) }))
.depends_on(summary); .depends_on(&summary);
} }
dag.run(|fut| { let _output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await .await
.unwrap(); .unwrap();
}) })
-1
View File
@@ -1,6 +1,5 @@
//! Realistic workload benchmarks //! Realistic workload benchmarks
pub mod config_broadcast;
pub mod etl; pub mod etl;
pub mod memory_efficiency; pub mod memory_efficiency;
pub mod mixed; pub mod mixed;
+4 -4
View File
@@ -1,13 +1,13 @@
[package] [package]
name = "dagx-macros" name = "dagx-macros"
version = "0.1.0" version = "0.3.2"
edition = "2021" edition = "2021"
rust-version = "1.78.0" rust-version = "1.81"
authors = ["Stephen Waits <steve@waits.net>"] authors = ["Stephen Waits <steve@waits.net>"]
license = "MIT" license = "MIT"
description = "Procedural macros for dagx" description = "Procedural macros for dagx"
repository = "https://github.com/swaits/dagx" repository = "https://git.swaits.com/swaits/dagx"
homepage = "https://github.com/swaits/dagx" homepage = "https://git.swaits.com/swaits/dagx"
documentation = "https://docs.rs/dagx-macros" documentation = "https://docs.rs/dagx-macros"
readme = "README.md" readme = "README.md"
+54 -153
View File
@@ -4,18 +4,23 @@
//! the `Task` trait by deriving Input and Output types from the `run()` method signature. //! the `Task` trait by deriving Input and Output types from the `run()` method signature.
use proc_macro::TokenStream; use proc_macro::TokenStream;
use proc_macro2::Span;
use quote::quote; use quote::quote;
use syn::{parse_macro_input, FnArg, ImplItem, ItemImpl, Pat, PatType, ReturnType, Type}; use syn::{parse_macro_input, FnArg, Ident, ImplItem, ItemImpl, Pat, PatType, ReturnType, Type};
/// Attribute macro to automatically implement the `Task` trait. /// Attribute macro to automatically implement the `Task` trait.
/// ///
/// Apply this to an `impl` block containing a `run()` method (sync or async). The macro: /// Apply this to an `impl` block containing a `run()` method (sync or async). The macro:
/// - Derives `Input` and `Output` types from the `run()` signature /// - Derives `Input` and `Output` types from the `run()` signature
/// - Automatically implements the `Task` trait /// - Automatically implements the `Task` trait
/// - **Generates type-specific extraction logic** - works with ANY type (Send + Sync + 'static)!
/// - Supports both sync and async run methods /// - Supports both sync and async run methods
/// - Supports stateless (no self) and stateful (&self, &mut self) tasks /// - Supports stateless (no self) and stateful (&self, &mut self) tasks
/// - Handles various input patterns (no inputs, single input, multiple inputs) /// - Handles various input patterns (no inputs, single input, multiple inputs)
/// ///
/// **Key Feature**: Custom types work automatically without implementing any traits!
/// The macro generates inline extraction logic specific to your task's parameter types.
///
/// # Task Patterns /// # Task Patterns
/// ///
/// The `#[task]` macro supports three patterns based on state requirements: /// The `#[task]` macro supports three patterns based on state requirements:
@@ -93,7 +98,7 @@ use syn::{parse_macro_input, FnArg, ImplItem, ItemImpl, Pat, PatType, ReturnType
/// self.value /// self.value
/// } /// }
/// } /// }
/// ```ignore /// ```
/// ///
/// ## Single Input /// ## Single Input
/// ///
@@ -108,7 +113,7 @@ use syn::{parse_macro_input, FnArg, ImplItem, ItemImpl, Pat, PatType, ReturnType
/// input * 2 /// input * 2
/// } /// }
/// } /// }
/// ```ignore /// ```
/// ///
/// ## Multiple Inputs (up to 8) /// ## Multiple Inputs (up to 8)
/// ///
@@ -123,15 +128,13 @@ use syn::{parse_macro_input, FnArg, ImplItem, ItemImpl, Pat, PatType, ReturnType
/// format!("{}: {} ({})", b, a, c) /// format!("{}: {} ({})", b, a, c)
/// } /// }
/// } /// }
/// ```ignore /// ```
/// ///
/// # Requirements /// # Requirements
/// ///
/// - The impl block must contain exactly one `async fn run()` method /// - The impl block must contain exactly one `async fn run()` method
/// - The `run()` method can be stateless (no self parameter) or stateful (`&mut self`) /// - The `run()` method can be stateless (no self parameter) or stateful (`&mut self`)
/// - All input parameters must be references (e.g., `&i32`, not `i32`) /// - All input parameters must be references (e.g., `&i32`, not `i32`)
/// - The macro requires `Task` to be in scope: `use dagx::Task;`
/// - For stateless tasks, the struct must implement `Default` (e.g., unit structs)
/// ///
/// # Generated Code /// # Generated Code
/// ///
@@ -149,13 +152,13 @@ use syn::{parse_macro_input, FnArg, ImplItem, ItemImpl, Pat, PatType, ReturnType
/// } /// }
/// ///
/// // Generated: /// // Generated:
/// impl Task for Add { /// impl Task<(i32, i32)> for Add {
/// type Input = (i32, i32);
/// type Output = i32; /// type Output = i32;
/// ///
/// async fn run(&mut self, input: Self::Input) -> Self::Output { /// async fn run(&mut self, _input: Self::Input) -> Self::Output {
/// let (a, b) = input; /// let (a, _input) = _input.next();
/// Self::run_impl(&a, &b).await /// let (b, _input) = _input.next();
/// Self::run_impl(a, b).await
/// } /// }
/// } /// }
/// ///
@@ -165,38 +168,7 @@ use syn::{parse_macro_input, FnArg, ImplItem, ItemImpl, Pat, PatType, ReturnType
/// a + b /// a + b
/// } /// }
/// } /// }
/// ```ignore /// ```
///
/// For stateful tasks (with &mut self):
///
/// ```ignore
/// // Your code:
/// #[task]
/// impl Counter {
/// async fn run(&mut self, inc: &i32) -> i32 {
/// self.count += inc;
/// self.count
/// }
/// }
///
/// // Generated:
/// impl Task for Counter {
/// type Input = i32;
/// type Output = i32;
///
/// async fn run(&mut self, input: Self::Input) -> Self::Output {
/// let inc = input;
/// self.run_impl(&inc).await
/// }
/// }
///
/// impl Counter {
/// async fn run_impl(&mut self, inc: &i32) -> i32 {
/// self.count += inc;
/// self.count
/// }
/// }
/// ```ignore
#[proc_macro_attribute] #[proc_macro_attribute]
pub fn task(_attr: TokenStream, item: TokenStream) -> TokenStream { pub fn task(_attr: TokenStream, item: TokenStream) -> TokenStream {
let impl_block = parse_macro_input!(item as ItemImpl); let impl_block = parse_macro_input!(item as ItemImpl);
@@ -227,9 +199,6 @@ pub fn task(_attr: TokenStream, item: TokenStream) -> TokenStream {
} }
}; };
// Check if the method is async or sync
let is_async = run_method.sig.asyncness.is_some();
// Extract parameters (excluding self) // Extract parameters (excluding self)
let params_result: Result<Vec<_>, _> = run_method let params_result: Result<Vec<_>, _> = run_method
.sig .sig
@@ -274,42 +243,29 @@ pub fn task(_attr: TokenStream, item: TokenStream) -> TokenStream {
// Extract return type // Extract return type
let output_type = match &run_method.sig.output { let output_type = match &run_method.sig.output {
ReturnType::Default => { ReturnType::Default => syn::parse_quote!(()),
return syn::Error::new_spanned(
&run_method.sig,
"run() method must have an explicit return type\n\n\
Specify the output type: async fn run(...) -> OutputType\n\
For tasks that don't return a value, use '-> ()'.",
)
.to_compile_error()
.into();
}
ReturnType::Type(_, ty) => ty.clone(), ReturnType::Type(_, ty) => ty.clone(),
}; };
// Build Input type based on parameter count // Build Input type based on parameter count
let input_type = match params.len() {
0 => quote! { () },
1 => {
let (_name, ty) = &params[0];
quote! { #ty }
}
_ => {
let types: Vec<_> = params.iter().map(|(_, ty)| ty).collect(); let types: Vec<_> = params.iter().map(|(_, ty)| ty).collect();
quote! { ( #(#types),* ) } let input_type = quote! { ( #(#types,)* ) };
}
};
// Generate parameter destructuring for the wrapper run() method // Generate parameter references for the wrapper run() method
let (param_destructure, param_refs) = if params.is_empty() { let (param_destructure, param_refs) = {
(quote! { _ }, quote! {}) let (destructure, refs): (Vec<_>, Vec<_>) = params
} else if params.len() == 1 { .iter()
let (name, _) = &params[0]; .map(|(ident, _ty)| {
(quote! { #name }, quote! { &#name }) let ident = Ident::new(&format!("__dagx_extract_{}", ident), Span::mixed_site());
} else { (
let names: Vec<_> = params.iter().map(|(name, _)| name).collect(); quote! {
let refs: Vec<_> = params.iter().map(|(name, _)| quote! { &#name }).collect(); let (#ident, _input) = _input.next();
(quote! { ( #(#names),* ) }, quote! { #(#refs),* }) },
ident,
)
})
.unzip();
(quote! { #(#destructure)* }, quote! { #(#refs),* })
}; };
// Clone the run method and rename it to run_impl // Clone the run method and rename it to run_impl
@@ -323,84 +279,29 @@ pub fn task(_attr: TokenStream, item: TokenStream) -> TokenStream {
.iter() .iter()
.any(|arg| matches!(arg, FnArg::Receiver(_))); .any(|arg| matches!(arg, FnArg::Receiver(_)));
// Generate the Task trait implementation based on whether we have self and async/sync let is_async = run_method.sig.asyncness.is_some();
let expanded = if has_self_receiver {
// Stateful task - consumes self but delegates to a method that borrows
if is_async {
// Async with self
quote! {
impl Task for #struct_name {
type Input = #input_type;
type Output = #output_type;
async fn run(mut self, input: Self::Input) -> Self::Output { let run_call = match (has_self_receiver, is_async) {
let #param_destructure = input; (true, true) => quote! { self.run_impl(#param_refs).await },
self.run_impl(#param_refs).await (true, false) => quote! { self.run_impl(#param_refs) },
} (false, true) => quote! { Self::run_impl(#param_refs).await },
} (false, false) => quote! { Self::run_impl(#param_refs) },
impl #struct_name {
#run_impl_method
}
}
} else {
// Sync with self - wrap in async block
quote! {
impl Task for #struct_name {
type Input = #input_type;
type Output = #output_type;
async fn run(mut self, input: Self::Input) -> Self::Output {
let #param_destructure = input;
self.run_impl(#param_refs)
}
}
impl #struct_name {
#run_impl_method
}
}
}
} else {
// Stateless task
if is_async {
// Async stateless
quote! {
impl Task for #struct_name {
type Input = #input_type;
type Output = #output_type;
async fn run(self, input: Self::Input) -> Self::Output {
let #param_destructure = input;
Self::run_impl(#param_refs).await
}
}
impl #struct_name {
#[inline]
#run_impl_method
}
}
} else {
// Sync stateless
quote! {
impl Task for #struct_name {
type Input = #input_type;
type Output = #output_type;
async fn run(self, input: Self::Input) -> Self::Output {
let #param_destructure = input;
Self::run_impl(#param_refs)
}
}
impl #struct_name {
#[inline]
#run_impl_method
}
}
}
}; };
TokenStream::from(expanded) // Generate the Task trait implementation based on whether we have self and async/sync
quote! {
impl ::dagx::Task<#input_type> for #struct_name {
type Output = #output_type;
async fn run(mut self, mut _input: ::dagx::TaskInput<'_, #input_type>) -> Self::Output {
#param_destructure
#run_call
}
}
impl #struct_name {
#run_impl_method
}
}
.into()
} }
+17
View File
@@ -0,0 +1,17 @@
[package]
name = "dagx-test"
version = "0.3.2"
edition = "2021"
rust-version = "1.81"
authors = ["TechnoPorg"]
license = "MIT"
description = "Utilities for dagx's internal tests"
repository = "https://git.swaits.com/swaits/dagx"
homepage = "https://git.swaits.com/swaits/dagx"
documentation = "https://docs.rs/dagx-test"
[lints]
workspace = true
[dependencies]
dagx = { version = "0.3.2", path = "../" }
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2025 Stephen Waits <steve@waits.net>
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+158
View File
@@ -0,0 +1,158 @@
//! Input extraction for tasks from a list of dependencies.
use std::{collections::HashMap, hash::Hash, sync::Arc};
use crate::TaskInput;
/// Helper trait for input extraction from list of dependencies.
pub trait ExtractInput: Sized + Clone {
type Input: Send;
type Retv<'input>;
fn extract_from_task_input<'input>(
input: TaskInput<'input, Self::Input>,
) -> Result<Self::Retv<'input>, String>;
}
// Unit type - no dependencies
impl ExtractInput for () {
type Input = ();
type Retv<'input> = ();
fn extract_from_task_input(_input: TaskInput<'_, Self>) -> Result<Self, String> {
Ok(())
}
}
// Implementations for common primitive and standard library types.
// These allow tasks to receive these types as single inputs without wrapping in tuples.
// Receives Arc<T> from runner, clones the inner value for task consumption.
// For custom types, implement ExtractInput following the same pattern.
macro_rules! impl_extract_single {
($($t:ty),+) => {
$(
impl ExtractInput for $t {
type Input = ($t,);
type Retv<'input> = &'input $t;
fn extract_from_task_input<'input>(input: TaskInput<'input, Self::Input>) -> Result<Self::Retv<'input>, String> {
Ok(input.next().0)
}
}
)+
};
}
impl_extract_single!(
i8, i16, i32, i64, i128, isize, u8, u16, u32, u64, u128, usize, f32, f64, bool, char, String
);
// Implement ExtractInput for Result types to support error handling patterns
impl<T, E> ExtractInput for Result<T, E>
where
T: 'static + Clone + Send + Sync,
E: 'static + Clone + Send + Sync,
{
type Input = (Self,);
type Retv<'input> = &'input Result<T, E>;
fn extract_from_task_input<'input>(
input: TaskInput<'input, (Self,)>,
) -> Result<Self::Retv<'input>, String> {
Ok(input.next().0)
}
}
// Implement ExtractInput for Option types
impl<T> ExtractInput for Option<T>
where
T: 'static + Clone + Send + Sync,
{
type Input = (Self,);
type Retv<'input> = &'input Option<T>;
fn extract_from_task_input<'input>(
input: TaskInput<'input, (Self,)>,
) -> Result<Self::Retv<'input>, String> {
Ok(input.next().0)
}
}
// Implement ExtractInput for Vec types
impl<T> ExtractInput for Vec<T>
where
T: 'static + Clone + Send + Sync,
{
type Input = (Self,);
type Retv<'input> = &'input Vec<T>;
fn extract_from_task_input<'input>(
input: TaskInput<'input, (Self,)>,
) -> Result<Self::Retv<'input>, String> {
Ok(input.next().0)
}
}
// Implement ExtractInput for HashMap types
impl<K, V> ExtractInput for HashMap<K, V>
where
K: 'static + Clone + Send + Sync + Eq + Hash,
V: 'static + Clone + Send + Sync,
{
type Input = (Self,);
type Retv<'input> = &'input HashMap<K, V>;
fn extract_from_task_input<'input>(
input: TaskInput<'input, (Self,)>,
) -> Result<Self::Retv<'input>, String> {
Ok(input.next().0)
}
}
impl<T> ExtractInput for Arc<T>
where
T: 'static + Clone + Send + Sync,
{
type Input = (Self,);
type Retv<'input> = &'input Arc<T>;
fn extract_from_task_input<'input>(
input: TaskInput<'input, (Self,)>,
) -> Result<Self::Retv<'input>, String> {
Ok(input.next().0)
}
}
macro_rules! impl_extract_tuple {
($($T:ident),+) => {
impl<$($T: 'static + Clone + Send + Sync),+> ExtractInput for ($($T,)+) {
type Input = Self;
type Retv<'input> = ($(&'input $T,)+);
fn extract_from_task_input<'input>(input: TaskInput<'input, Self::Input>) -> Result<Self::Retv<'input>, String> {
$(
let ($T, input): (&'input $T, _) = input.next();
)+
let _input: TaskInput<'input, ()> = input;
Ok(($($T,)+))
}
}
};
}
// Generate ExtractInput implementations for tuples of size 2-8.
// Each invocation creates an impl for a specific tuple size.
// Supporting up to 8 elements covers the vast majority of use cases.
#[allow(non_snake_case)]
mod impls {
use super::*;
impl_extract_tuple!(A, B);
impl_extract_tuple!(A, B, C);
impl_extract_tuple!(A, B, C, D);
impl_extract_tuple!(A, B, C, D, E);
impl_extract_tuple!(A, B, C, D, E, F);
impl_extract_tuple!(A, B, C, D, E, F, G);
impl_extract_tuple!(A, B, C, D, E, F, G, H);
}
+52
View File
@@ -0,0 +1,52 @@
//! DAG builder helpers for internal use in dagx tests and benchmarks.
//!
//! This crate is not meant for public use and offers no stability guarantees.
//! The ergonomics are also such that you don't want to use it anyways. **Use the `#[task]` macro instead.**
#![cfg(not(tarpaulin_include))]
use std::marker::PhantomData;
use crate::extract::ExtractInput;
use dagx::{Task, TaskInput};
mod extract;
pub struct TaskFn<I, O, F>
where
for<'input> F: FnMut(I::Retv<'input>) -> O + Send,
I: ExtractInput + Send + Sync + 'static,
O: Send,
{
f: F,
_phantom: PhantomData<fn(I) -> O>,
}
impl<I, O, F> Task<I::Input> for TaskFn<I, O, F>
where
for<'input> F: FnMut(I::Retv<'input>) -> O + Send,
I: ExtractInput + Send + Sync + 'static,
I::Input: Send + Sync + 'static,
O: Send + Sync + 'static,
{
type Output = O;
async fn run(mut self, input: TaskInput<'_, I::Input>) -> O {
(self.f)(I::extract_from_task_input(input).unwrap())
}
}
/// Convenience function to create a task from a closure.
pub fn task_fn<I, O, F>(f: F) -> TaskFn<I, O, F>
where
for<'input> F: FnMut(I::Retv<'input>) -> O + Send,
I: ExtractInput + Send + Sync + 'static,
I::Input: Send + Sync + 'static,
O: Send + Sync + 'static,
{
TaskFn {
f,
_phantom: PhantomData,
}
}
+229
View File
@@ -0,0 +1,229 @@
# Comparison with Similar Projects
The Rust ecosystem offers several DAG execution libraries, each optimized for different use cases. This comparison helps you choose the right tool for your needs.
## Developer Experience: API Comparison
The same task (compute 2 + 3) implemented across different libraries shows how API complexity varies:
**dagx** (Simple: macro + builder):
```rust
#[task]
impl Value { async fn run(&self) -> i32 { self.0 } }
#[task]
impl Add { async fn run(a: &i32, b: &i32) -> i32 { a + b } }
let mut dag = DagRunner::new();
let x = dag.add_task(Value(2));
let y = dag.add_task(Value(3));
let sum = dag.add_task(Add).depends_on((x, y));
dag.run(|fut| async move { tokio::spawn(fut).await.unwrap() }).await?;
```
**dagrs** (Complex: traits + channels + IDs + manual wiring):
```rust
#[async_trait]
impl Action for Value {
async fn run(&self, _: &mut InChannels, out: &mut OutChannels, _: Arc<EnvVar>) -> Output {
out.broadcast(Content::new(self.0)).await;
Output::Out(Some(Content::new(self.0)))
}
}
let mut table = NodeTable::new();
let node1 = DefaultNode::with_action("x".into(), Value(2), &mut table);
let id1 = node1.id();
let node2 = DefaultNode::with_action("y".into(), Value(3), &mut table);
let id2 = node2.id();
let node3 = DefaultNode::with_action("add".into(), Add, &mut table);
let id3 = node3.id();
let mut graph = Graph::new();
graph.add_node(node1); graph.add_node(node2); graph.add_node(node3);
graph.add_edge(id1, vec![id3]); graph.add_edge(id2, vec![id3]);
graph.set_env(EnvVar::new(table)); graph.start().unwrap();
```
**async_dag** (Medium: slots + indices):
```rust
let mut graph = Graph::new();
let x = graph.add_task(|| async { 2 });
let y = graph.add_task(|| async { 3 });
let sum = graph.add_child_task(x, |a: i32| async move { a }, 0)?;
graph.update_dependency(y, sum, 1)?; // Must specify slot index
let sum = graph.add_child_task(sum, |a: i32, b: i32| async move { a + b }, 0)?;
graph.update_dependency(y, sum, 1)?;
```
**Key differences**:
- **dagx**: Type-safe dependencies, automatic wiring, no manual ID tracking, minimal boilerplate
- **dagrs**: Manual channel management, node ID tracking, Content wrapping, Action trait boilerplate
- **async_dag**: Slot indices must be tracked manually, dependencies updated separately
## Quick Comparison
| Project | License | Runtime | Type Safety | API Complexity | Performance vs dagx | Key Features |
| ------------------------------------------------------------- | -------------- | ------------------- | --------------------- | -------------- | ------------------------- | ------------------------------------------------------------------------------------ |
| **dagx** | MIT | Any async runtime | Compile-time | Simple | Baseline (see benchmarks) | Primitives-as-scheduler, inline fast-path, automatic Arc wrapping, up to 8 deps/task |
| [**dagrs**](https://github.com/dagrs-dev/dagrs) (v0.5) | MIT/Apache-2.0 | Creates own runtime | Runtime (async_trait) | Complex | 1-100x slower (see below) | Flow-based Programming, cyclic graphs, loops, conditional nodes, YAML config |
| [**async_dag**](https://github.com/chubei-oppen/async_dag) | MIT | Any async runtime | Compile-time | Medium | No benchmarks | Slot-based dependencies, Graph/TryGraph modes, maximum parallelism |
| [**dag-flow**](https://github.com/makisevon/dag-flow) | MIT/Apache-2.0 | Any async runtime | Runtime (HashMap) | Complex | No benchmarks | Experimental, all tasks run simultaneously, weak dependencies |
| [**RenovZ/dag-runner**](https://github.com/RenovZ/dag-runner) | MIT | Tokio only | Unclear | Simple | No benchmarks | Edge-based API, cycle detection, stops on first error |
| [**tasksitter**](https://github.com/lionkor/tasksitter) | Unspecified | Unclear | Unclear | Medium | No benchmarks | Cyclic graphs, dynamic runtime modification, pause/resume |
**Performance**: As noted in the README, dagx is **1-100x faster** than dagrs across all benchmark patterns. The inline fast-path optimization eliminates spawning overhead for sequential workloads while maintaining excellent parallel performance. Most real-world DAGs mix sequential and parallel patterns. dagx automatically optimizes for both, performing up to 2 orders of magnitude faster than dagrs regardless of workload shape.
_Benchmarks run on Intel i9-13950HX @ 5.5GHz. Run `cargo bench` to test on your hardware._
## Detailed Comparison
### dagrs (Most Mature)
**Best for**: Complex workflows requiring advanced flow control, Tokio-based applications, machine learning pipelines.
**Strengths**:
- Most mature (449 GitHub stars, active community)
- Rich feature set: Flow-based Programming, cyclic graphs, loops, conditional nodes
- Designed for complex orchestration patterns
**Trade-offs**:
- Creates own Tokio runtime internally (not runtime-agnostic, cannot be nested)
- More complex API: `Action` trait, `InChannels`/`OutChannels`, `NodeTable`, `Content` wrappers, manual node ID tracking
- Uses `async_trait` for type erasure (runtime overhead)
- **Slower than dagx** across all benchmark patterns (see comparison benchmarks)
### async_dag (Clean Type Safety)
**Best for**: Runtime flexibility with compile-time type safety, fail-fast workflows.
**Strengths**:
- Runtime-agnostic (works with any async runtime)
- Compile-time type checking on task connections
- Both standard (`Graph`) and fail-fast (`TryGraph`) modes
- Designed for maximum parallelism
**Trade-offs**:
- Medium API complexity: slot-based dependency management
- Must manually specify slot indices (0, 1, etc.) when connecting tasks
- Less mature
**API Style**:
```rust
let mut graph = Graph::new();
let _1 = graph.add_task(|| async { 1 });
let _2 = graph.add_task(|| async { 2 });
// add_child_task with slot index
let _3 = graph.add_child_task(_1, sum, 0).unwrap();
graph.update_dependency(_2, _3, 1).unwrap(); // Specify slot 1
graph.run().await;
```
### dag-flow (Experimental)
**Best for**: Experimental projects, flexible dependency awaiting patterns.
**Strengths**:
- Runtime-agnostic
- All tasks run simultaneously (not in dependency layers)
- Weak dependencies support
- Flexible input awaiting at any point in task execution
**Trade-offs**:
- Explicitly experimental
- Runtime type safety via `HashMap<String, Input>`
- Complex API: implement `Task` trait with `id()`, `dependencies()`, `run()`
- Named dependencies (string-based lookup)
- Very early stage
**API Style**:
```rust
impl Task<String, Bytes> for MyTask {
fn id(&self) -> String { "task_name".into() }
fn dependencies(&self) -> Option<Vec<String>> { Some(vec!["dep1".into()]) }
async fn run(&self, inputs: HashMap<String, Input<'_, Bytes>>) -> Option<Bytes> {
let dep_value = inputs.get("dep1").unwrap().await;
// Process
}
}
```
### RenovZ/dag-runner (Simple Edge-Based)
**Best for**: Simple DAGs with Tokio, straightforward edge-based dependencies.
**Strengths**:
- Simple API: `add_vertex()`, `add_edge()`
- Cycle detection
- Stops on first error
**Trade-offs**:
- Requires Tokio runtime
- Manual channel setup for task communication
- Type safety model unclear
- Very early stage
**API Style**:
```rust
let mut dag = Dag::default();
dag.add_vertex("one", || async move { /* task */ });
dag.add_edge("one", "two");
dag.run().await?;
```
### tasksitter (Dynamic Workflows)
**Best for**: Dynamic workflow modification, cyclic graphs, runtime introspection.
**Strengths**:
- Supports cyclic graphs (not just DAGs)
- Dynamic graph modification at runtime
- Pause/resume capabilities
- Graph introspection
**Trade-offs**:
- Limited documentation
- Runtime and type safety model unclear
- Very early stage
## When to Choose dagx
Choose dagx when you value:
- **Performance**: Faster than dagrs across all workload patterns (see benchmarks)
- **Runtime flexibility**: Works with Tokio, smol, or any async runtime
- **Compile-time safety**: Full type safety with no runtime type errors possible
- **Simple, ergonomic API**: `#[task]` macro, `add_task()`, `depends_on()` - that's it
- **Automatic optimizations**: Arc wrapping, inline execution, adaptive spawning - all transparent
dagx is **not** the right choice if you need:
- Cyclic graphs or dynamic flow control (loops, conditions) → Consider **dagrs** or **tasksitter**
- More than 8 dependencies per task → Consider **dagrs** or **async_dag**
## When to Consider Alternatives
- **Choose dagrs** if you need advanced flow control (loops, conditionals, cyclic graphs), or are already committed to Tokio and want a mature, feature-rich solution
- **Choose async_dag** if you want compile-time type safety with runtime flexibility and the slot-based API appeals to you
- **Choose dag-flow** if you're building experimental projects and the all-tasks-run-simultaneously model fits your use case
- **Choose RenovZ/dag-runner** if you need the simplest possible edge-based API and are already using Tokio
- **Choose tasksitter** if you need dynamic graph modification at runtime or cyclic workflow support
+80
View File
@@ -0,0 +1,80 @@
# Compile-Time Cycle Prevention
**dagx guarantees acyclic graphs at compile time with zero runtime overhead.** Unlike other DAG libraries that detect cycles at runtime, dagx uses Rust's type system to make cycles impossible to express in the first place.
## Why This Matters
Most DAG libraries check for cycles when you execute the graph:
```rust
// Other libraries: cycle detected at RUNTIME
let mut dag = DagRunner::new();
let a = dag.add_task(TaskA);
let b = dag.add_task(TaskB);
a.depends_on(&b);
b.depends_on(&a); // Compiles fine...
dag.run(|fut| async move { tokio::spawn(fut).await.unwrap() })
```
**dagx prevents this at compile time:**
```rust
// dagx: cycle prevented at COMPILE TIME
let mut dag = DagRunner::new();
let a_builder = dag.add_task(TaskA);
let b_builder = dag.add_task(TaskB);
// To make A depend on B, we need B as a TaskHandle.
// But calling depends_on() consumes the builder!
let b_handle = b_builder.depends_on(&some_source);
// Now we can't make B depend on A because b_builder was moved!
// This won't compile:
// a_builder.depends_on(&b_handle); // ✓ OK: A→B
// b_handle.depends_on(&a_handle); // ❌ ERROR: TaskHandle has no depends_on() method!
```
## How It Works: The Type-State Pattern
dagx uses two types to enforce acyclic structure:
1. **`TaskBuilder<T, Deps>`**: Mutable builder that can have dependencies added via `depends_on()`
- Consumed (moved) when you call `depends_on()` or convert to `TaskHandle`
- Can only be used once for wiring
2. **`TaskHandle<T>`**: Immutable reference to a finalized task
- Copy type (just wraps a task ID)
- Can be used as a dependency for other tasks
- **Has no `depends_on()` method** - can't be modified after creation
This creates an impossible ordering constraint for cycles:
```rust
// Attempting a cycle A→B→A:
let a_builder = dag.add_task(TaskA); // TaskBuilder
let b_builder = dag.add_task(TaskB); // TaskBuilder
// To wire A→B, we need B as a TaskHandle
let b_handle = b_builder.into(); // Convert B to TaskHandle (consumes builder)
// To wire B→A, we need A as a TaskHandle
let a_handle = a_builder.depends_on(&b_handle); // A now depends on B
// Now if we try to complete the cycle:
// a_handle.depends_on(...); // ❌ ERROR: no method named `depends_on` found
```
**The type system enforces strict topological ordering!** You can't create a `TaskHandle` until all its dependencies exist as handles, which prevents cycles.
## Proof & Examples
See [`tests/cycle_prevention.rs`](../tests/cycle_prevention.rs) for runtime tests demonstrating the type-state pattern in action.
## Benefits
- ✅ **Zero runtime cost**: No cycle detection code to execute
- ✅ **Catches errors early**: Cycles are caught at compile time, not in production
- ✅ **Compiler-verified**: The type system proves your graph is acyclic
- ✅ **No mental overhead**: You can't accidentally create cycles even if you try
This is a true **zero-cost abstraction**—the type system provides safety guarantees without any runtime checks or performance penalty.
+54
View File
@@ -0,0 +1,54 @@
# Design Philosophy: Primitives as Scheduler
## How It Works
1. **Wire up tasks with primitives**: During DAG construction, tasks are organized into topological layers based on their dependencies
2. **Start everything simultaneously**: When you call `run()`, all tasks in the first layer spawn at once—there's no scheduler deciding when each task should start
3. **Let the primitives handle coordination**: Layers naturally enforce execution order. No custom orchestration needed.
4. **Runtime joins on completion**: The runtime simply spawns all tasks and waits for them to finish. That's it.
## The Implementation
Under the hood, dagx uses:
- **Ownership model**: Tasks take ownership (`self`) and are consumed during execution - no Mutex needed for task state
- **Direct data flow**: Arc-wrapped outputs flow from producer(s) to consumer(s) through a store in the runner
- **Fast-path optimization**: Single-task layers execute inline without spawning overhead
## Inline Execution Fast-Path
**Performance optimization for sequential workloads**: When a layer contains only a single task (common in deep chains and linear pipelines), dagx executes it inline rather than spawning it. This eliminates spawning overhead and context switching, resulting in a 10-100x performance improvements for sequential patterns.
**Panic handling guarantee**: To maintain behavioral consistency between runtimes, as well as inline vs. spawned tasks, panics in tasks are caught using `FutureExt::catch_unwind()` and converted to errors. This ensures your code behaves the same whether a task runs inline or spawned, making dagx's optimizations transparent and predictable.
## Benefits
**Simplicity**: The runtime is straightforward: spawn tasks, coordinate via compile-time dependency guarantees. No complex scheduler code to maintain, debug, or optimize.
**Reliability**: Built on battle-tested primitives from Rust's standard library and the futures crate. These have been used in production by thousands of projects and are orders of magnitude more reliable than custom scheduling logic.
**Bug resistance**: Fewer moving parts means fewer places for bugs to hide. The type system enforces correct wiring at compile time. The ownership model prevents data races. What's left to break?
**Performance**: Near zero-overhead. No locks during execution. Arc reference counting for efficient fanout (atomic operations, not locks). Tasks start as soon as their dependencies complete - maximum parallelism.
**Auditability**: Want to verify correctness? Check the dependency wiring, verify tasks await their inputs, done. No need to trace through complex state machine transitions or wake-up cascades.
## The Insight
The key insight is that **dependencies ARE the schedule**. If task B depends on task A's output, the topological sort naturally enforces that B waits for A. The dependency graph already encodes all the scheduling information—we just need to wire outputs to match it.
This is dagx's core philosophy: leverage the type system for correctness, use primitives for coordination, and let the compiler optimize everything else away.
## Measured Overhead
How much overhead does this approach actually add? Benchmarks on an Intel i9-13950HX show:
**DAG Construction**:
- Empty DAG creation: **~7 nanoseconds**
- Adding tasks: **~5 nanoseconds per task**
- Building a 10,000-task DAG: **~865 microseconds** (100 ns/task)
**Scaling**: Sub-microsecond per-task overhead across all workload patterns. Linear scaling verified to 10k+ tasks.
The primitives-as-scheduler approach with inline fast-path optimization delivers exceptional performance: coordination overhead is sub-microsecond per task, and for real-world workloads where tasks do meaningful work (I/O, computation, etc.), framework overhead is negligible—typically well under 1% of total execution time.
+8 -10
View File
@@ -30,7 +30,7 @@
//! Result: 5 //! Result: 5
//! ``` //! ```
use dagx::{task, DagRunner, Task}; use dagx::{task, DagRunner};
// Step 1: Define a source task (no dependencies) // Step 1: Define a source task (no dependencies)
// //
@@ -77,7 +77,7 @@ impl Add {
#[tokio::main] #[tokio::main]
async fn main() { async fn main() {
// Step 3: Create a new DAG // Step 3: Create a new DAG
let dag = DagRunner::new(); let mut dag = DagRunner::new();
// Step 4: Add source tasks (tasks with no dependencies) // Step 4: Add source tasks (tasks with no dependencies)
// //
@@ -105,20 +105,18 @@ async fn main() {
// 1. Tasks with no dependencies run first (x and y in parallel) // 1. Tasks with no dependencies run first (x and y in parallel)
// 2. Once x and y complete, sum runs (it depends on both) // 2. Once x and y complete, sum runs (it depends on both)
println!("Running DAG...\n"); println!("Running DAG...\n");
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await .await
.unwrap(); .unwrap();
// Step 7: Retrieve results // Step 7: Retrieve results
// //
// Use .get() with a TaskHandle to retrieve the output. // Use .get() with a TaskHandle to retrieve the output.
// Results are cloned, so you can retrieve them multiple times. // Results are not cloned, so you can only retrieve them once.
println!("\nResult: {}", dag.get(sum).unwrap()); println!("\nResult: {}", output.get(sum));
// Note: You can also retrieve intermediate results // Note: You can also retrieve intermediate results
assert_eq!(dag.get(x).unwrap(), 2); assert_eq!(output.get(x), 2);
assert_eq!(dag.get(y).unwrap(), 3); assert_eq!(output.get(y), 3);
assert_eq!(dag.get(sum).unwrap(), 5);
} }
+8 -9
View File
@@ -55,7 +55,7 @@
//! Squared: 100 //! Squared: 100
//! ``` //! ```
use dagx::{task, DagRunner, Task}; use dagx::{task, DagRunner};
// Source task with a value field // Source task with a value field
struct Source { struct Source {
@@ -127,7 +127,7 @@ impl Power {
#[tokio::main] #[tokio::main]
async fn main() { async fn main() {
let dag = DagRunner::new(); let mut dag = DagRunner::new();
// Single source task - constructed with ::new() // Single source task - constructed with ::new()
let source = dag.add_task(Source::new(10)); let source = dag.add_task(Source::new(10));
@@ -140,16 +140,15 @@ async fn main() {
// Run the DAG // Run the DAG
println!("Running fan-out DAG...\n"); println!("Running fan-out DAG...\n");
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await .await
.unwrap(); .unwrap();
// Print all results // Print all results
println!("\nResults:"); println!("\nResults:");
println!(" Source: {}", dag.get(source).unwrap()); println!(" Source: {}", output.get(source));
println!(" Plus One: {}", dag.get(plus_one).unwrap()); println!(" Plus One: {}", output.get(plus_one));
println!(" Times Two: {}", dag.get(times_two).unwrap()); println!(" Times Two: {}", output.get(times_two));
println!(" Squared: {}", dag.get(squared).unwrap()); println!(" Squared: {}", output.get(squared));
} }
+5 -6
View File
@@ -61,7 +61,7 @@
//! Active: true //! Active: true
//! ``` //! ```
use dagx::{task, DagRunner, Task}; use dagx::{task, DagRunner};
struct FetchName; struct FetchName;
@@ -118,7 +118,7 @@ impl BuildProfile {
#[tokio::main] #[tokio::main]
async fn main() { async fn main() {
let dag = DagRunner::new(); let mut dag = DagRunner::new();
// Multiple independent source tasks // Multiple independent source tasks
let name = dag.add_task(FetchName); let name = dag.add_task(FetchName);
@@ -133,12 +133,11 @@ async fn main() {
// Run the DAG // Run the DAG
println!("Running fan-in DAG...\n"); println!("Running fan-in DAG...\n");
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await .await
.unwrap(); .unwrap();
// Print result // Print result
println!("\n{}", dag.get(profile).unwrap()); println!("\n{}", output.get(profile));
} }
+124 -57
View File
@@ -1,61 +1,89 @@
//! # Parallel Computation: Map-Reduce Pattern //! # Proving True Parallelism with Timing
//! //!
//! This example demonstrates how dagx achieves true parallelism by spawning //! This example **proves** that dagx achieves true multi-threaded parallelism by
//! tasks to multiple threads, enabling efficient parallel computation. //! comparing parallel vs sequential execution times. Unlike the basic fan-in example
//! (03_fan_in.rs), this measures and demonstrates actual parallel speedup.
//! //!
//! ## What You'll Learn //! ## What You'll Learn
//! - How to split work across multiple parallel tasks (Map) //! - How to verify true parallelism with timing measurements
//! - How to aggregate results from parallel tasks (Reduce) //! - How to measure parallel speedup (sequential time / parallel time)
//! - How dagx achieves true multi-threaded parallelism //! - Why CPU-bound tasks benefit from true multi-threading
//! - How to verify parallel execution with timing //! - How dagx automatically achieves parallelism with no extra code
//!
//! ## Key Differences from 03_fan_in.rs
//! - **03_fan_in.rs**: Shows the pattern (4 sources → 1 aggregator)
//! - **This example**: Proves the pattern runs in parallel with actual timing
//! //!
//! ## Key Concepts //! ## Key Concepts
//! - **Map**: Split work into independent parallel tasks //! - **Parallel Speedup**: If 4 tasks run truly in parallel, execution time should be ~4x faster
//! - **Reduce**: Aggregate results from parallel tasks into a single output //! - **CPU-bound work**: Computation that keeps the CPU busy (vs I/O-bound waiting)
//! - **True Parallelism**: Tasks spawn to multiple OS threads via the runtime //! - **Multi-threading**: Tasks running simultaneously on different CPU cores
//! - **Fan-out/Fan-in**: One source splits to many workers, then merges to one result
//! //!
//! ## Pattern Diagram //! ## Pattern Diagram
//! ```text //! ```text
//! [Source] → [Sum1] //! [Sum1: 1-250 ] ──┐
//! → [Sum2] → [Aggregate] → Total //! [Sum2: 251-500] ──┼─→ [Aggregate] → Total: 500500
//! → [Sum3] //! [Sum3: 501-750] ──┤
//! → [Sum4] //! [Sum4: 751-1000] ─┘
//!
//! Map Phase Reduce Phase
//! (4 parallel (1 task combines
//! workers) all results)
//! ``` //! ```
//! //!
//! ## Running This Example //! ## Running This Example
//! ```bash //! ```bash
//! cargo run --example parallel_computation //! cargo run --release --example parallel_computation
//! ``` //! ```
//! //!
//! **IMPORTANT**: Run in `--release` mode to see meaningful timing differences!
//!
//! ## Expected Output //! ## Expected Output
//! ```text //! ```text
//! Computing sum of 1-1000 using 4 parallel tasks... //! === Proving True Parallelism ===
//! //!
//! Sum of 1-1000 computed in parallel: 500500 //! Running PARALLEL computation (4 workers on separate threads)...
//! Expected: 500500 //! Parallel result: 500500 (computed in 128ms)
//!
//! Running SEQUENTIAL computation (same work, one at a time)...
//! Sequential result: 500500 (computed in 512ms)
//!
//! Speedup: 4.0x faster with parallel execution!
//! This proves dagx runs tasks on multiple CPU cores simultaneously.
//! ``` //! ```
use dagx::{task, DagResult, DagRunner, Task}; use dagx::{task, DagResult, DagRunner};
// Step 1: Define a worker task that computes a partial sum use std::time::{Duration, Instant};
use tokio::time::sleep;
// Step 1: Define a worker task that does CPU-bound work
// //
// Each worker computes the sum of a range of numbers. // We add artificial delay to make the parallelism measurable.
// These tasks run in parallel on different threads. // In real scenarios, this would be actual computation (image processing,
// mathematical calculations, data transformation, etc.)
struct ComputeSum { struct ComputeSum {
start: i32, start: i32,
end: i32, end: i32,
work_delay_ms: u64, // Simulate CPU-bound work
} }
impl ComputeSum { impl ComputeSum {
fn new(start: i32, end: i32) -> Self { fn new(start: i32, end: i32, work_delay_ms: u64) -> Self {
Self { start, end } Self {
start,
end,
work_delay_ms,
}
} }
} }
#[task] #[task]
impl ComputeSum { impl ComputeSum {
async fn run(&mut self) -> i32 { async fn run(&mut self) -> i32 {
// Simulate CPU-bound work (e.g., complex calculation)
sleep(Duration::from_millis(self.work_delay_ms)).await;
// Compute sum of range [start, end) // Compute sum of range [start, end)
(self.start..self.end).sum() (self.start..self.end).sum()
} }
@@ -77,50 +105,89 @@ impl Aggregate {
#[tokio::main] #[tokio::main]
async fn main() -> DagResult<()> { async fn main() -> DagResult<()> {
println!("Computing sum of 1-1000 using 4 parallel tasks...\n"); println!("=== Proving True Parallelism ===\n");
// Step 3: Build the DAG const WORK_DELAY: u64 = 100; // Each task does 100ms of "work"
let dag = DagRunner::new();
// Map phase: Create 4 workers that run in parallel // ============================================================================
// Each computes sum of 250 numbers (total 1-1000) // PARALLEL EXECUTION: 4 tasks run simultaneously on different threads
let sum1 = dag.add_task(ComputeSum::new(1, 251)); // 1-250 // ============================================================================
let sum2 = dag.add_task(ComputeSum::new(251, 501)); // 251-500 println!("Running PARALLEL computation (4 workers on separate threads)...");
let sum3 = dag.add_task(ComputeSum::new(501, 751)); // 501-750 let (parallel_time, parallel_result) = {
let sum4 = dag.add_task(ComputeSum::new(751, 1001)); // 751-1000 let start = Instant::now();
let mut dag = DagRunner::new();
// Create 4 workers - dagx will run them in parallel
let sum1 = dag.add_task(ComputeSum::new(1, 251, WORK_DELAY));
let sum2 = dag.add_task(ComputeSum::new(251, 501, WORK_DELAY));
let sum3 = dag.add_task(ComputeSum::new(501, 751, WORK_DELAY));
let sum4 = dag.add_task(ComputeSum::new(751, 1001, WORK_DELAY));
// Reduce phase: Aggregate results from all workers
let total = dag let total = dag
.add_task(Aggregate) .add_task(Aggregate)
.depends_on((&sum1, &sum2, &sum3, &sum4)); .depends_on((&sum1, &sum2, &sum3, &sum4));
// Step 4: Execute with true parallelism let mut output = dag
// .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
// tokio::spawn() sends tasks to multiple threads in the thread pool.
// Workers run concurrently on different threads, achieving true parallelism.
dag.run(|fut| {
tokio::spawn(fut);
})
.await?; .await?;
// Step 5: Retrieve and verify results let result = output.get(total);
let result = dag.get(total)?; let elapsed = start.elapsed();
let expected: i32 = (1..1001).sum();
println!("Sum of 1-1000 computed in parallel: {}", result); println!(
println!("Expected: {}", expected); "Parallel result: {} (computed in {}ms)\n",
result,
elapsed.as_millis()
);
assert_eq!(result, 500500);
(elapsed, result)
};
assert_eq!(result, 500500); // Sum of 1-1000 // ============================================================================
// SEQUENTIAL EXECUTION: Run the same 4 tasks one after another
// ============================================================================
println!("Running SEQUENTIAL computation (same work, one at a time)...");
let (sequential_time, sequential_result) = {
let start = Instant::now();
// Note: True parallelism verification // Run tasks sequentially (not using DAG, just raw execution)
// let mut task1 = ComputeSum::new(1, 251, WORK_DELAY);
// If you monitor system resources while running this example, you'll see: let mut task2 = ComputeSum::new(251, 501, WORK_DELAY);
// - Multiple CPU cores being utilized let mut task3 = ComputeSum::new(501, 751, WORK_DELAY);
// - Worker tasks executing simultaneously let mut task4 = ComputeSum::new(751, 1001, WORK_DELAY);
// - Reduced wall-clock time vs sequential execution
// let r1 = task1.run_impl().await;
// This demonstrates that dagx provides REAL parallel execution, let r2 = task2.run_impl().await;
// not just async concurrency on a single thread. let r3 = task3.run_impl().await;
let r4 = task4.run_impl().await;
let result = r1 + r2 + r3 + r4;
let elapsed = start.elapsed();
println!(
"Sequential result: {} (computed in {}ms)\n",
result,
elapsed.as_millis()
);
assert_eq!(result, 500500);
(elapsed, result)
};
// ============================================================================
// PROOF: Compare execution times
// ============================================================================
assert_eq!(parallel_result, sequential_result);
let speedup = sequential_time.as_secs_f64() / parallel_time.as_secs_f64();
println!("Speedup: {:.1}x faster with parallel execution!", speedup);
println!("This proves dagx runs tasks on multiple CPU cores simultaneously.");
if speedup > 2.0 {
println!("\n✓ Parallel speedup confirmed - tasks ran on multiple threads!");
} else {
println!("\n⚠ Warning: Low speedup. Try running with --release flag for accurate timing.");
}
Ok(()) Ok(())
} }
+30 -36
View File
@@ -150,7 +150,8 @@
//! Result 7: Ok("Success") //! Result 7: Ok("Success")
//! ``` //! ```
use dagx::{task, DagRunner, Task}; use dagx::{task, DagRunner};
use std::sync::Arc; use std::sync::Arc;
use std::time::Instant; use std::time::Instant;
use tokio::sync::Mutex; use tokio::sync::Mutex;
@@ -380,7 +381,7 @@ async fn main() {
// Create shared circuit breaker: 3 failures trigger open, 3 second recovery timeout // Create shared circuit breaker: 3 failures trigger open, 3 second recovery timeout
let circuit = Arc::new(CircuitBreaker::new(3, Duration::from_secs(3))); let circuit = Arc::new(CircuitBreaker::new(3, Duration::from_secs(3)));
let dag = DagRunner::new(); let mut dag = DagRunner::new();
// Request 1: Success (circuit closed) // Request 1: Success (circuit closed)
let call1 = dag.add_task(ProtectedServiceCall::new( let call1 = dag.add_task(ProtectedServiceCall::new(
@@ -389,64 +390,60 @@ async fn main() {
false, // success false, // success
)); ));
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await .await
.unwrap(); .unwrap();
println!("Result 1: {:?}\n", dag.get(call1).unwrap()); println!("Result 1: {:?}\n", output.get(call1));
// Request 2: Failure (circuit closed, count = 1) // Request 2: Failure (circuit closed, count = 1)
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let call2 = dag.add_task(ProtectedServiceCall::new( let call2 = dag.add_task(ProtectedServiceCall::new(
"ServiceCall-2", "ServiceCall-2",
circuit.clone(), circuit.clone(),
true, // fail true, // fail
)); ));
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await .await
.unwrap(); .unwrap();
println!("Result 2: {:?}\n", dag.get(call2).unwrap()); println!("Result 2: {:?}\n", output.get(call2));
// Request 3: Failure (circuit closed, count = 2) // Request 3: Failure (circuit closed, count = 2)
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let call3 = dag.add_task(ProtectedServiceCall::new( let call3 = dag.add_task(ProtectedServiceCall::new(
"ServiceCall-3", "ServiceCall-3",
circuit.clone(), circuit.clone(),
true, // fail true, // fail
)); ));
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await .await
.unwrap(); .unwrap();
println!("Result 3: {:?}\n", dag.get(call3).unwrap()); println!("Result 3: {:?}\n", output.get(call3));
// Request 4: Failure (circuit closed, count = 3, opens!) // Request 4: Failure (circuit closed, count = 3, opens!)
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let call4 = dag.add_task(ProtectedServiceCall::new( let call4 = dag.add_task(ProtectedServiceCall::new(
"ServiceCall-4", "ServiceCall-4",
circuit.clone(), circuit.clone(),
true, // fail true, // fail
)); ));
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await .await
.unwrap(); .unwrap();
println!("Result 4: {:?}\n", dag.get(call4).unwrap()); println!("Result 4: {:?}\n", output.get(call4));
// Request 5: Circuit is now open, fails fast with fallback // Request 5: Circuit is now open, fails fast with fallback
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let call5 = dag.add_task(ProtectedServiceCall::new( let call5 = dag.add_task(ProtectedServiceCall::new(
"ServiceCall-5", "ServiceCall-5",
circuit.clone(), circuit.clone(),
@@ -457,13 +454,12 @@ async fn main() {
.add_task(FallbackService::new("ServiceCall-5")) .add_task(FallbackService::new("ServiceCall-5"))
.depends_on(&call5); .depends_on(&call5);
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await .await
.unwrap(); .unwrap();
println!("Result 5: {:?}\n", dag.get(fallback5).unwrap()); println!("Result 5: {:?}\n", output.get(fallback5));
// Example 2: Circuit recovery after timeout // Example 2: Circuit recovery after timeout
println!("Example 2: Circuit recovery after timeout\n"); println!("Example 2: Circuit recovery after timeout\n");
@@ -471,36 +467,34 @@ async fn main() {
sleep(Duration::from_secs(3)).await; sleep(Duration::from_secs(3)).await;
// Request 6: Circuit should be half-open, test succeeds, closes circuit // Request 6: Circuit should be half-open, test succeeds, closes circuit
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let call6 = dag.add_task(ProtectedServiceCall::new( let call6 = dag.add_task(ProtectedServiceCall::new(
"ServiceCall-6", "ServiceCall-6",
circuit.clone(), circuit.clone(),
false, // success false, // success
)); ));
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await .await
.unwrap(); .unwrap();
println!("Result 6: {:?}\n", dag.get(call6).unwrap()); println!("Result 6: {:?}\n", output.get(call6));
// Request 7: Circuit is now closed again, normal operation // Request 7: Circuit is now closed again, normal operation
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let call7 = dag.add_task(ProtectedServiceCall::new( let call7 = dag.add_task(ProtectedServiceCall::new(
"ServiceCall-7", "ServiceCall-7",
circuit.clone(), circuit.clone(),
false, // success false, // success
)); ));
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await .await
.unwrap(); .unwrap();
println!("Result 7: {:?}\n", dag.get(call7).unwrap()); println!("Result 7: {:?}\n", output.get(call7));
} }
println!("=== Circuit Breaker Example Complete ==="); println!("=== Circuit Breaker Example Complete ===");
-217
View File
@@ -1,217 +0,0 @@
//! # Building Complex Workflows: Multi-Layer DAG
//!
//! This example demonstrates how to build complex, real-world DAGs by combining
//! multiple dependency patterns into a cohesive workflow.
//!
//! ## What You'll Learn
//! - How to combine fan-out, fan-in, and diamond patterns in one DAG
//! - How to build multi-layer workflows with intermediate results
//! - How to structure an ETL (Extract-Transform-Load) pipeline
//! - How to pass results through multiple transformation stages
//!
//! ## Workflow Diagram
//! ```text
//! EXTRACT TRANSFORM COMBINE AGGREGATE
//! ┌──────────┐ ┌──────────┐
//! │ LoadData │────►│Transform │────┐
//! │ A │ │ (×2) │ │
//! └──────────┘ └──────────┘ │
//! ├──►┌─────────┐───►┌─────────┐───►┌─────────┐
//! │ │ Combine │ │ Average │ │ Report │
//! └──►└─────────┘ └─────────┘ └─────────┘
//! ┌──────────┐ ┌──────────┐ ┌──► │ │
//! │ LoadData │────►│Transform │────┘ │ │
//! │ B │ │ (×3) │ └──────────────┘
//! └──────────┘ └──────────┘
//! ```
//!
//! ## Key Concepts
//! - **Multi-layer execution**: Tasks organized into logical stages (Extract → Transform → Combine)
//! - **Many-to-many dependencies**: The Combine task depends on both Transform tasks
//! - **Result reuse**: The combined result is used by both Average and Report tasks
//! - **Diamond pattern**: Multiple paths from sources converge and then diverge again
//!
//! ## Use Cases
//! - ETL pipelines with multiple data sources
//! - Multi-stage data processing workflows
//! - Complex business logic with branching and merging
//! - Data aggregation and reporting systems
//!
//! ## Running This Example
//! ```bash
//! cargo run --example complex_dag
//! ```
//!
//! ## Expected Output
//! ```text
//! Building ETL pipeline...
//!
//! Executing pipeline...
//!
//! Loading data from source A
//! Loading data from source B
//! Transforming: 6 * 2 = 12
//! Transforming: 15 * 3 = 45
//! Combining datasets: 12 + 45 = 57
//! Computing average: 57 / 6 = 9.50
//!
//! ETL Pipeline Results:
//! Items: 6
//! Total: 57
//! Average: 9.50
//!
//! Calculation:
//! Source A: sum=6, transformed (x2) = 12
//! Source B: sum=15, transformed (x3) = 45
//! Combined: 6 items, sum=57, avg=9.50
//! ```
use dagx::{task, DagResult, DagRunner, Task};
// Extract: Load data from different sources
struct LoadData {
source: &'static str,
}
impl LoadData {
fn new(source: &'static str) -> Self {
Self { source }
}
}
#[task]
impl LoadData {
async fn run(&mut self) -> i32 {
// Simulate loading and summing data
println!("Loading data from source {}", self.source);
match self.source {
"A" => 6, // sum of [1, 2, 3]
"B" => 15, // sum of [4, 5, 6]
_ => 0,
}
}
}
// Transform: Process data with a multiplier
struct Transform {
multiplier: i32,
}
impl Transform {
fn new(multiplier: i32) -> Self {
Self { multiplier }
}
}
#[task]
impl Transform {
async fn run(&mut self, sum: &i32) -> i32 {
let result = sum * self.multiplier;
println!("Transforming: {} * {} = {}", sum, self.multiplier, result);
result
}
}
// Combine: Merge two datasets
struct Combine;
#[task]
impl Combine {
async fn run(sum_a: &i32, sum_b: &i32) -> i32 {
println!(
"Combining datasets: {} + {} = {}",
sum_a,
sum_b,
sum_a + sum_b
);
sum_a + sum_b
}
}
// Compute average (using a fixed count for simplicity)
struct ComputeAverage {
total_count: i32,
}
impl ComputeAverage {
fn new(total_count: i32) -> Self {
Self { total_count }
}
}
#[task]
impl ComputeAverage {
async fn run(&mut self, sum: &i32) -> f64 {
let avg = *sum as f64 / self.total_count as f64;
println!(
"Computing average: {} / {} = {:.2}",
sum, self.total_count, avg
);
avg
}
}
// Format report
struct FormatReport {
count: i32,
}
impl FormatReport {
fn new(count: i32) -> Self {
Self { count }
}
}
#[task]
impl FormatReport {
async fn run(&mut self, sum: &i32, avg: &f64) -> String {
format!(
"ETL Pipeline Results:\n Items: {}\n Total: {}\n Average: {:.2}",
self.count, sum, avg
)
}
}
#[tokio::main]
async fn main() -> DagResult<()> {
let dag = DagRunner::new();
println!("Building ETL pipeline...\n");
// Extract: Load from two sources (parallel)
let source_a = dag.add_task(LoadData::new("A"));
let source_b = dag.add_task(LoadData::new("B"));
// Transform: Process each source (parallel)
let transform_a = dag.add_task(Transform::new(2)).depends_on(&source_a);
let transform_b = dag.add_task(Transform::new(3)).depends_on(&source_b);
// Combine: Merge the transformed datasets
let combined = dag
.add_task(Combine)
.depends_on((&transform_a, &transform_b));
// Compute average on combined data (we know we have 6 items total)
let avg = dag.add_task(ComputeAverage::new(6)).depends_on(combined);
// Format final report (needs both sum and average)
let report = dag
.add_task(FormatReport::new(6))
.depends_on((&combined, &avg));
// Execute the entire ETL pipeline
println!("Executing pipeline...\n");
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
// Display final report
println!("\n{}", dag.get(report)?);
println!("\nCalculation:");
println!(" Source A: sum=6, transformed (x2) = 12");
println!(" Source B: sum=15, transformed (x3) = 45");
println!(" Combined: 6 items, sum=57, avg=9.50");
Ok(())
}
-150
View File
@@ -1,150 +0,0 @@
//! # Pattern: Conditional Execution with Data Validation
//!
//! This example demonstrates how to implement conditional logic in DAGs by using
//! simple data transformations rather than traditional Result types.
//!
//! ## What You'll Learn
//! - How to implement validation logic in parallel tasks
//! - How to use return values for conditional behavior (e.g., 0 for invalid)
//! - How to categorize and branch based on computed values
//! - How to aggregate results from multiple conditional branches
//!
//! ## Workflow Diagram
//! ```text
//! VALIDATE CATEGORIZE SUMMARIZE
//! ┌─────────────┐ ┌──────────────┐
//! │ Validate(10)│────20──►│Categorize(25)│────"low"────┐
//! └─────────────┘ └──────────────┘ │
//! │
//! ┌─────────────┐ ┌──────────────┐ │
//! │ Validate(-5)│────0───►│Categorize(25)│──"invalid"──┼──►┌────────────┐
//! └─────────────┘ └──────────────┘ │ │ Summarize │
//! │ └────────────┘
//! ┌─────────────┐ ┌──────────────┐ │
//! │ Validate(20)│────40──►│Categorize(25)│────"high"───┘
//! └─────────────┘ └──────────────┘
//! ```
//!
//! ## Key Concepts
//! - **Validation as transformation**: Invalid inputs become sentinel values (0)
//! - **Parallel validation**: All inputs validated simultaneously
//! - **Data-driven branching**: Categorization based on computed values
//! - **Type-safe aggregation**: All branches produce the same type (String)
//!
//! ## Use Cases
//! - Input validation pipelines
//! - Multi-stage data cleaning
//! - Classification and categorization workflows
//! - Quality control systems
//! - Data preprocessing for ML pipelines
//!
//! ## Running This Example
//! ```bash
//! cargo run --example conditional_workflow
//! ```
//!
//! ## Expected Output
//! ```text
//! Processing and categorizing inputs in parallel...
//!
//! Valid input 10: doubling to 20
//! Invalid input -5: returning 0
//! Valid input 20: doubling to 40
//!
//! Final summary: Categories: [low, invalid, high]
//! Expected: Categories: [low, invalid, high]
//! ```
use dagx::{task, DagResult, DagRunner, Task};
// Task that validates and processes input
struct ValidateAndDouble {
value: i32,
}
impl ValidateAndDouble {
fn new(value: i32) -> Self {
Self { value }
}
}
#[task]
impl ValidateAndDouble {
async fn run(&mut self) -> i32 {
if self.value > 0 {
println!("Valid input {}: doubling to {}", self.value, self.value * 2);
self.value * 2
} else {
println!("Invalid input {}: returning 0", self.value);
0 // Return 0 for invalid inputs
}
}
}
// Task that categorizes a number
struct Categorize {
threshold: i32,
}
impl Categorize {
fn new(threshold: i32) -> Self {
Self { threshold }
}
}
#[task]
impl Categorize {
async fn run(&mut self, value: &i32) -> String {
if *value == 0 {
"invalid".to_string()
} else if *value >= self.threshold {
"high".to_string()
} else {
"low".to_string()
}
}
}
// Task that combines categorization results
struct SummarizeResults;
#[task]
impl SummarizeResults {
async fn run(cat1: &String, cat2: &String, cat3: &String) -> String {
format!("Categories: [{}, {}, {}]", cat1, cat2, cat3)
}
}
#[tokio::main]
async fn main() -> DagResult<()> {
let dag = DagRunner::new();
println!("Processing and categorizing inputs in parallel...\n");
// Validate and process multiple inputs in parallel
let processed1 = dag.add_task(ValidateAndDouble::new(10)); // Valid: 20
let processed2 = dag.add_task(ValidateAndDouble::new(-5)); // Invalid: 0
let processed3 = dag.add_task(ValidateAndDouble::new(20)); // Valid: 40
// Categorize each result (also parallel, threshold = 25)
let cat1 = dag.add_task(Categorize::new(25)).depends_on(&processed1); // 20 < 25 = "low"
let cat2 = dag.add_task(Categorize::new(25)).depends_on(&processed2); // 0 = "invalid"
let cat3 = dag.add_task(Categorize::new(25)).depends_on(&processed3); // 40 >= 25 = "high"
// Combine results
let summary = dag
.add_task(SummarizeResults)
.depends_on((&cat1, &cat2, &cat3));
// Execute
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
// Display results
println!("\nFinal summary: {}", dag.get(summary)?);
println!("Expected: Categories: [low, invalid, high]");
Ok(())
}
+5 -6
View File
@@ -75,7 +75,7 @@
//! ============================ //! ============================
//! ``` //! ```
use dagx::{task, DagRunner, Task}; use dagx::{task, DagRunner};
// Data source tasks // Data source tasks
struct LoadValue1; struct LoadValue1;
@@ -178,7 +178,7 @@ impl GenerateReport {
#[tokio::main] #[tokio::main]
async fn main() { async fn main() {
let dag = DagRunner::new(); let mut dag = DagRunner::new();
println!("Building data pipeline DAG...\n"); println!("Building data pipeline DAG...\n");
@@ -220,12 +220,11 @@ async fn main() {
// Execute the pipeline // Execute the pipeline
println!("Executing pipeline...\n"); println!("Executing pipeline...\n");
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await .await
.unwrap(); .unwrap();
// Print final report // Print final report
println!("{}", dag.get(report).unwrap()); println!("{}", output.get(report));
} }
-464
View File
@@ -1,464 +0,0 @@
//! # Debug Tracing and Task Naming
//!
//! This example demonstrates how to add debug metadata and tracing information to tasks
//! for better observability in complex DAG workflows.
//!
//! ## What You'll Learn
//! - How to add task names and IDs for debugging
//! - How to implement correlation IDs for distributed tracing
//! - How to track task execution timing and lifecycle events
//! - How to build traceable task wrappers
//!
//! ## Debugging Patterns
//!
//! **1. Named Tasks** - Add human-readable names to tasks
//! ```rust
//! struct TaskMetadata {
//! id: String,
//! name: String,
//! start_time: Option<Instant>,
//! }
//! ```
//!
//! **2. Correlation IDs** - Track requests across task boundaries
//! ```rust
//! struct TraceContext {
//! correlation_id: String,
//! parent_task: Option<String>,
//! }
//! ```
//!
//! **3. Execution Metrics** - Measure task performance
//! ```rust
//! struct TaskTrace {
//! name: String,
//! duration_ms: u64,
//! status: &'static str,
//! }
//! ```
//!
//! ## Workflow Diagram
//! ```text
//! TRACE: order-12345
//! ┌──────────────────┐
//! │ ValidateOrder │ [order-12345] Start: 0ms
//! │ ID: task-001 │──────► Status: Success
//! └──────────────────┘ Duration: 10ms
//! │
//! ├──────────────────┬──────────────────┐
//! ▼ ▼ ▼
//! ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐
//! │ CheckInventory │ │ ValidatePayment │ │ CalculateShipping│
//! │ ID: task-002 │ │ ID: task-003 │ │ ID: task-004 │
//! └──────────────────┘ └──────────────────┘ └──────────────────┘
//! [order-12345] [order-12345] [order-12345]
//! Duration: 45ms Duration: 120ms Duration: 15ms
//! │ │ │
//! └──────────────────┴──────────────────┘
//! ▼
//! ┌──────────────────┐
//! │ FulfillOrder │ [order-12345]
//! │ ID: task-005 │ Duration: 30ms
//! └──────────────────┘
//! ```
//!
//! ## Key Concepts
//! - **Observability**: Make task execution visible for debugging
//! - **Correlation IDs**: Track a request through multiple tasks
//! - **Structured logging**: Consistent task metadata across logs
//! - **Performance metrics**: Measure task execution time
//! - **Task lifecycle**: Track start, completion, and errors
//!
//! ## Use Cases
//! - Debugging complex workflows in production
//! - Performance analysis and bottleneck identification
//! - Distributed tracing in microservices
//! - SLA monitoring and alerting
//! - Audit logging for compliance
//!
//! ## Running This Example
//! ```bash
//! cargo run --example debug_tracing
//! ```
//!
//! ## Expected Output
//! ```text
//! === Debug Tracing Example ===
//!
//! Starting order processing for order-12345
//!
//! [order-12345] [task-001] ValidateOrder: Starting
//! [order-12345] [task-001] ValidateOrder: Completed in 10ms - Success
//!
//! [order-12345] [task-002] CheckInventory: Starting
//! [order-12345] [task-003] ValidatePayment: Starting
//! [order-12345] [task-004] CalculateShipping: Starting
//! [order-12345] [task-002] CheckInventory: Completed in 45ms - Success
//! [order-12345] [task-004] CalculateShipping: Completed in 15ms - Success
//! [order-12345] [task-003] ValidatePayment: Completed in 120ms - Success
//!
//! [order-12345] [task-005] FulfillOrder: Starting
//! [order-12345] [task-005] FulfillOrder: Completed in 30ms - Success
//!
//! === Execution Trace Summary ===
//! Correlation ID: order-12345
//! Total Tasks: 5
//! Total Duration: 220ms
//! Success: 5, Failed: 0
//!
//! Task Breakdown:
//! ValidateOrder: 10ms
//! CheckInventory: 45ms
//! ValidatePayment: 120ms (slowest)
//! CalculateShipping: 15ms
//! FulfillOrder: 30ms
//! ```
use dagx::{task, DagRunner, Task};
use std::time::Instant;
use tokio::time::{sleep, Duration};
// Stateful task that tracks execution metadata for debugging
struct TracedTask {
task_id: String,
task_name: String,
correlation_id: String,
work_duration_ms: u64,
start_time: Option<Instant>,
}
impl TracedTask {
fn new(
task_id: impl Into<String>,
task_name: impl Into<String>,
correlation_id: impl Into<String>,
work_duration_ms: u64,
) -> Self {
Self {
task_id: task_id.into(),
task_name: task_name.into(),
correlation_id: correlation_id.into(),
work_duration_ms,
start_time: None,
}
}
}
#[task]
impl TracedTask {
async fn run(&mut self) -> Result<TaskTrace, String> {
// Record start
let start = Instant::now();
self.start_time = Some(start);
println!(
"[{}] [{}] {}: Starting",
self.correlation_id, self.task_id, self.task_name
);
// Simulate work
sleep(Duration::from_millis(self.work_duration_ms)).await;
// Record completion
let duration_ms = start.elapsed().as_millis() as u64;
println!(
"[{}] [{}] {}: Completed in {}ms - Success",
self.correlation_id, self.task_id, self.task_name, duration_ms
);
Ok(TaskTrace {
task_id: self.task_id.clone(),
task_name: self.task_name.clone(),
correlation_id: self.correlation_id.clone(),
duration_ms,
status: "Success",
})
}
}
// Task that can fail, demonstrating error tracing
struct TracedTaskWithError {
task_id: String,
task_name: String,
correlation_id: String,
should_fail: bool,
}
impl TracedTaskWithError {
fn new(
task_id: impl Into<String>,
task_name: impl Into<String>,
correlation_id: impl Into<String>,
should_fail: bool,
) -> Self {
Self {
task_id: task_id.into(),
task_name: task_name.into(),
correlation_id: correlation_id.into(),
should_fail,
}
}
}
#[task]
impl TracedTaskWithError {
async fn run(&mut self) -> Result<TaskTrace, TaskTrace> {
let start = Instant::now();
println!(
"[{}] [{}] {}: Starting",
self.correlation_id, self.task_id, self.task_name
);
sleep(Duration::from_millis(50)).await;
let duration_ms = start.elapsed().as_millis() as u64;
if self.should_fail {
println!(
"[{}] [{}] {}: Failed in {}ms - Validation Error",
self.correlation_id, self.task_id, self.task_name, duration_ms
);
Err(TaskTrace {
task_id: self.task_id.clone(),
task_name: self.task_name.clone(),
correlation_id: self.correlation_id.clone(),
duration_ms,
status: "Failed",
})
} else {
println!(
"[{}] [{}] {}: Completed in {}ms - Success",
self.correlation_id, self.task_id, self.task_name, duration_ms
);
Ok(TaskTrace {
task_id: self.task_id.clone(),
task_name: self.task_name.clone(),
correlation_id: self.correlation_id.clone(),
duration_ms,
status: "Success",
})
}
}
}
// Task trace data structure
#[derive(Clone, Debug)]
#[allow(dead_code)]
struct TaskTrace {
task_id: String,
task_name: String,
correlation_id: String,
duration_ms: u64,
status: &'static str,
}
// Task that aggregates traces from multiple upstream tasks
struct TraceAggregator {
correlation_id: String,
}
impl TraceAggregator {
fn new(correlation_id: impl Into<String>) -> Self {
Self {
correlation_id: correlation_id.into(),
}
}
}
#[task]
impl TraceAggregator {
async fn run(
&mut self,
t1: &Result<TaskTrace, String>,
t2: &Result<TaskTrace, String>,
t3: &Result<TaskTrace, String>,
) -> TraceSummary {
let traces = vec![t1, t2, t3];
let mut summary = TraceSummary {
correlation_id: self.correlation_id.clone(),
task_count: traces.len(),
total_duration_ms: 0,
success_count: 0,
failed_count: 0,
task_details: Vec::new(),
};
for trace_result in traces {
match trace_result {
Ok(trace) => {
summary.success_count += 1;
summary.total_duration_ms += trace.duration_ms;
summary.task_details.push(trace.clone());
}
Err(e) => {
summary.failed_count += 1;
eprintln!("[{}] Task failed: {}", self.correlation_id, e);
}
}
}
summary
}
}
// Summary of traced execution
#[derive(Clone, Debug)]
struct TraceSummary {
correlation_id: String,
task_count: usize,
total_duration_ms: u64,
success_count: usize,
failed_count: usize,
task_details: Vec<TaskTrace>,
}
impl TraceSummary {
fn print(&self) {
println!("\n=== Execution Trace Summary ===");
println!("Correlation ID: {}", self.correlation_id);
println!("Total Tasks: {}", self.task_count);
println!("Total Duration: {}ms", self.total_duration_ms);
println!(
"Success: {}, Failed: {}",
self.success_count, self.failed_count
);
if !self.task_details.is_empty() {
println!("\nTask Breakdown:");
let max_duration = self
.task_details
.iter()
.map(|t| t.duration_ms)
.max()
.unwrap_or(0);
for trace in &self.task_details {
let slowest_tag = if trace.duration_ms == max_duration {
" (slowest)"
} else {
""
};
println!(
" {}: {}ms{}",
trace.task_name, trace.duration_ms, slowest_tag
);
}
}
}
}
#[tokio::main]
async fn main() {
println!("=== Debug Tracing Example ===\n");
// Example 1: Successful order processing with full tracing
println!("Example 1: Successful order processing\n");
{
let correlation_id = "order-12345";
println!("Starting order processing for {}\n", correlation_id);
let dag = DagRunner::new();
// Root task: Validate order
let _validate = dag.add_task(TracedTask::new(
"task-001",
"ValidateOrder",
correlation_id,
10,
));
// Parallel validation tasks (independent, no dependencies)
let inventory = dag.add_task(TracedTask::new(
"task-002",
"CheckInventory",
correlation_id,
45,
));
let payment = dag.add_task(TracedTask::new(
"task-003",
"ValidatePayment",
correlation_id,
120,
));
let shipping = dag.add_task(TracedTask::new(
"task-004",
"CalculateShipping",
correlation_id,
15,
));
// Aggregate traces
let summary = dag
.add_task(TraceAggregator::new(correlation_id))
.depends_on((&inventory, &payment, &shipping));
// Run the DAG
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
// Print summary
dag.get(summary).unwrap().print();
}
println!("\n");
// Example 2: Order with failed payment
println!("Example 2: Order with failed payment\n");
{
let correlation_id = "order-67890";
println!("Starting order processing for {}\n", correlation_id);
let dag = DagRunner::new();
let _validate = dag.add_task(TracedTaskWithError::new(
"task-101",
"ValidateOrder",
correlation_id,
false,
));
let _inventory = dag.add_task(TracedTaskWithError::new(
"task-102",
"CheckInventory",
correlation_id,
false,
));
// This task will fail
let payment = dag.add_task(TracedTaskWithError::new(
"task-103",
"ValidatePayment",
correlation_id,
true, // will fail
));
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
// Check results
match dag.get(payment).unwrap() {
Ok(_) => println!("\nPayment validation succeeded"),
Err(trace) => {
println!("\n[{}] Payment validation failed:", correlation_id);
println!(" Task ID: {}", trace.task_id);
println!(" Duration: {}ms", trace.duration_ms);
println!(" Status: {}", trace.status);
}
}
}
println!("\n=== Debug Tracing Example Complete ===");
}
+27 -31
View File
@@ -114,7 +114,7 @@
//! === Error Handling Example Complete === //! === Error Handling Example Complete ===
//! ``` //! ```
use dagx::{task, DagRunner, Task}; use dagx::{task, DagRunner};
// Source task that provides a string input // Source task that provides a string input
struct StringSource(&'static str); struct StringSource(&'static str);
@@ -209,7 +209,7 @@ async fn main() {
// Example 1: Successful path // Example 1: Successful path
println!("Example 1: Valid input"); println!("Example 1: Valid input");
{ {
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let input = dag.add_task(StringSource("42")); let input = dag.add_task(StringSource("42"));
@@ -217,18 +217,17 @@ async fn main() {
let validated = dag let validated = dag
.add_task(ValidateRange { min: 0, max: 100 }) .add_task(ValidateRange { min: 0, max: 100 })
.depends_on(parsed); .depends_on(&parsed);
let processed = dag.add_task(ProcessData).depends_on(validated); let processed = dag.add_task(ProcessData).depends_on(&validated);
let logged = dag.add_task(LogError).depends_on(processed); let logged = dag.add_task(LogError).depends_on(&processed);
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await .await
.unwrap(); .unwrap();
match dag.get(logged).unwrap() { match output.get(logged).as_ref() {
Ok(result) => println!("Final result: {}\n", result), Ok(result) => println!("Final result: {}\n", result),
Err(e) => eprintln!("Final error: {}\n", e), Err(e) => eprintln!("Final error: {}\n", e),
} }
@@ -237,7 +236,7 @@ async fn main() {
// Example 2: Parse error // Example 2: Parse error
println!("Example 2: Invalid input (parse error)"); println!("Example 2: Invalid input (parse error)");
{ {
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let input = dag.add_task(StringSource("not a number")); let input = dag.add_task(StringSource("not a number"));
@@ -245,18 +244,17 @@ async fn main() {
let validated = dag let validated = dag
.add_task(ValidateRange { min: 0, max: 100 }) .add_task(ValidateRange { min: 0, max: 100 })
.depends_on(parsed); .depends_on(&parsed);
let processed = dag.add_task(ProcessData).depends_on(validated); let processed = dag.add_task(ProcessData).depends_on(&validated);
let logged = dag.add_task(LogError).depends_on(processed); let logged = dag.add_task(LogError).depends_on(&processed);
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await .await
.unwrap(); .unwrap();
match dag.get(logged).unwrap() { match output.get(logged).as_ref() {
Ok(result) => println!("Final result: {}\n", result), Ok(result) => println!("Final result: {}\n", result),
Err(e) => eprintln!("Final error: {}\n", e), Err(e) => eprintln!("Final error: {}\n", e),
} }
@@ -265,7 +263,7 @@ async fn main() {
// Example 3: Validation error // Example 3: Validation error
println!("Example 3: Out of range"); println!("Example 3: Out of range");
{ {
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let input = dag.add_task(StringSource("150")); let input = dag.add_task(StringSource("150"));
@@ -273,18 +271,17 @@ async fn main() {
let validated = dag let validated = dag
.add_task(ValidateRange { min: 0, max: 100 }) .add_task(ValidateRange { min: 0, max: 100 })
.depends_on(parsed); .depends_on(&parsed);
let processed = dag.add_task(ProcessData).depends_on(validated); let processed = dag.add_task(ProcessData).depends_on(&validated);
let logged = dag.add_task(LogError).depends_on(processed); let logged = dag.add_task(LogError).depends_on(&processed);
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await .await
.unwrap(); .unwrap();
match dag.get(logged).unwrap() { match output.get(logged).as_ref() {
Ok(result) => println!("Final result: {}\n", result), Ok(result) => println!("Final result: {}\n", result),
Err(e) => eprintln!("Final error: {}\n", e), Err(e) => eprintln!("Final error: {}\n", e),
} }
@@ -293,7 +290,7 @@ async fn main() {
// Example 4: Error recovery with fallback // Example 4: Error recovery with fallback
println!("Example 4: Error recovery with fallback"); println!("Example 4: Error recovery with fallback");
{ {
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let input = dag.add_task(StringSource("invalid")); let input = dag.add_task(StringSource("invalid"));
@@ -301,15 +298,14 @@ async fn main() {
let with_fallback = dag let with_fallback = dag
.add_task(WithFallback { fallback: 0 }) .add_task(WithFallback { fallback: 0 })
.depends_on(parsed); .depends_on(&parsed);
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await .await
.unwrap(); .unwrap();
let result = dag.get(with_fallback).unwrap(); let result = output.get(with_fallback);
println!("Recovered with fallback: {}\n", result); println!("Recovered with fallback: {}\n", result);
} }
-391
View File
@@ -1,391 +0,0 @@
//! # Working with Large DAGs: Performance and Scalability
//!
//! This example demonstrates dagx's performance characteristics when working
//! with large-scale DAGs containing hundreds or thousands of tasks.
//!
//! ## What You'll Learn
//! - How dagx scales with increasing task counts
//! - Performance characteristics of different DAG topologies
//! - How to benchmark DAG construction and execution
//! - Memory efficiency considerations
//!
//! ## DAG Topologies Tested
//!
//! **1. Wide DAG** - Maximum parallelism
//! ```text
//! ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐
//! │Task 1│ │Task 2│ │Task 3│ ... │Task N│
//! └──────┘ └──────┘ └──────┘ └──────┘
//! (1000 independent parallel tasks)
//! ```
//!
//! **2. Deep DAG** - Sequential chain
//! ```text
//! ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐
//! │Task 1│───►│Task 2│───►│Task 3│───► ... ──►│Task N│
//! └──────┘ └──────┘ └──────┘ └──────┘
//! (100-level dependency chain)
//! ```
//!
//! **3. Binary Tree** - Balanced aggregation
//! ```text
//! ┌────┐ ┌────┐ ┌────┐ ┌────┐
//! │ T1 │ │ T2 │ │ T3 │ │ T4 │
//! └─┬──┘ └─┬──┘ └─┬──┘ └─┬──┘
//! └───┬───┘ └───┬───┘
//! │ │
//! ┌──▼──┐ ┌──▼──┐
//! │ C1 │ │ C2 │
//! └──┬──┘ └──┬──┘
//! └───────┬──────┘
//! │
//! ┌──▼──┐
//! │Root │
//! └─────┘
//! ```
//!
//! **4. Diamond Pattern** - Many-to-many
//! ```text
//! Layer 1: 50 sources
//! Layer 2: 50 tasks (each depends on 2 sources)
//! Layer 3: 4 aggregates
//! Layer 4: 1 final result
//! ```
//!
//! ## Performance Metrics
//! - **Build time**: Time to construct the DAG structure
//! - **Execution time**: Time to run all tasks
//! - **Tasks/second**: Throughput metric
//! - **Memory overhead**: Approximate per-task memory usage
//!
//! ## Key Observations
//! - DAG construction is very fast (< 1ms for 1000s of tasks)
//! - Parallel execution scales well with available CPU cores
//! - Deep chains execute sequentially as expected
//! - Memory overhead is minimal (~200 bytes per task)
//!
//! ## Running This Example
//! ```bash
//! cargo run --example large_dag --release # Use --release for accurate benchmarks
//! ```
//!
//! ## Expected Output
//! ```text
//! === Large DAG Performance Example ===
//!
//! Example 1: Wide DAG (1000 parallel tasks)
//! Build time: 347µs
//! Execution time: 12.5ms
//! Tasks/second: 80000
//! Sample sum (first 10): ...
//!
//! Example 2: Deep DAG (100-level chain)
//! Build time: 156µs
//! Execution time: 8.2ms
//! Final result: ...
//!
//! Example 3: Binary Tree DAG (depth 10, 1023 tasks)
//! Build time: 423µs
//! Tree depth: 9
//! Execution time: 15.3ms
//! Root result: ...
//!
//! Example 4: Diamond Pattern (50x50 = 2500 edges)
//! Build time: 234µs
//! Execution time: 11.7ms
//! Total tasks: 105
//! Final result: ...
//!
//! Example 5: Memory Efficiency (100 tasks, measure overhead)
//! i32 sum: ...
//! String total length: ...
//!
//! === Large DAG Performance Example Complete ===
//!
//! Key Observations:
//! - DAG construction is very fast (< 1ms for 1000s of tasks)
//! - Parallel execution scales well with available CPU cores
//! - Deep chains execute sequentially as expected
//! - Memory overhead is minimal (~ 200 bytes per task)
//! ```
use dagx::{task, DagRunner, Task};
use std::time::Instant;
// Simple computation task
struct Compute {
value: i32,
}
#[task]
impl Compute {
async fn run(&mut self) -> i32 {
// Simulate some work
let mut result = self.value;
for _ in 0..100 {
result = result.wrapping_mul(31).wrapping_add(17);
}
result
}
}
// Task that combines two values
struct Combine;
#[task]
impl Combine {
async fn run(a: &i32, b: &i32) -> i32 {
a.wrapping_add(*b)
}
}
// Task that aggregates multiple values
struct Aggregate;
#[task]
impl Aggregate {
async fn run(a: &i32, b: &i32, c: &i32, d: &i32) -> i32 {
a.wrapping_add(*b).wrapping_add(*c).wrapping_add(*d)
}
}
// Task that formats a number as a string
struct FormatNumber {
id: i32,
}
#[task]
impl FormatNumber {
async fn run(&self) -> String {
format!("Result {}", self.id)
}
}
#[tokio::main]
async fn main() {
println!("=== Large DAG Performance Example ===\n");
// Example 1: Wide DAG (many parallel tasks)
println!("Example 1: Wide DAG (1000 parallel tasks)");
{
let start = Instant::now();
let dag = DagRunner::new();
// Create 1000 independent tasks
let tasks: Vec<_> = (0..1000)
.map(|i| dag.add_task(Compute { value: i }))
.collect();
let build_time = start.elapsed();
println!(" Build time: {:?}", build_time);
let exec_start = Instant::now();
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
let exec_time = exec_start.elapsed();
println!(" Execution time: {:?}", exec_time);
println!(" Tasks/second: {:.0}", 1000.0 / exec_time.as_secs_f64());
// Verify a few results
let sum: i32 = tasks
.iter()
.take(10)
.map(|t| dag.get(t).unwrap())
.fold(0i32, |acc, x| acc.wrapping_add(x));
println!(" Sample sum (first 10): {}\n", sum);
}
// Example 2: Deep DAG (long dependency chain)
println!("Example 2: Deep DAG (100-level chain)");
{
let start = Instant::now();
let dag = DagRunner::new();
// Create a deep chain by combining adjacent task pairs
let tasks: Vec<_> = (0..100)
.map(|i| dag.add_task(Compute { value: i }))
.collect();
let mut chain_tasks = vec![];
for i in 0..99 {
let combined = dag.add_task(Combine).depends_on((&tasks[i], &tasks[i + 1]));
chain_tasks.push(combined);
}
let prev = chain_tasks.last().unwrap();
let build_time = start.elapsed();
println!(" Build time: {:?}", build_time);
let exec_start = Instant::now();
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
let exec_time = exec_start.elapsed();
println!(" Execution time: {:?}", exec_time);
println!(" Final result: {}\n", dag.get(prev).unwrap());
}
// Example 3: Binary tree DAG
println!("Example 3: Binary Tree DAG (depth 10, 1023 tasks)");
{
let start = Instant::now();
let dag = DagRunner::new();
// Build binary tree: each level combines pairs from previous level
let current_level: Vec<_> = (0..512)
.map(|i| dag.add_task(Compute { value: i }))
.collect();
// Process all levels reducing by combining pairs
let mut level = 0;
let mut layer: Vec<dagx::TaskHandle<i32>> =
current_level.iter().map(|b| b.into()).collect();
while layer.len() > 1 {
level += 1;
let mut next_layer = Vec::new();
for i in (0..layer.len()).step_by(2) {
if i + 1 < layer.len() {
// We have a pair - combine them
let combined = dag.add_task(Combine).depends_on((&layer[i], &layer[i + 1]));
next_layer.push(combined);
} else {
// Odd one out - just pass it through
next_layer.push(layer[i]);
}
}
layer = next_layer;
}
let root = &layer[0];
let build_time = start.elapsed();
println!(" Build time: {:?}", build_time);
println!(" Tree depth: {}", level);
let exec_start = Instant::now();
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
let exec_time = exec_start.elapsed();
println!(" Execution time: {:?}", exec_time);
println!(" Root result: {}\n", dag.get(root).unwrap());
}
// Example 4: Diamond pattern (many-to-many)
println!("Example 4: Diamond Pattern (50x50 = 2500 edges)");
{
let start = Instant::now();
let dag = DagRunner::new();
// Layer 1: 50 source tasks
let sources: Vec<_> = (0..50)
.map(|i| dag.add_task(Compute { value: i }))
.collect();
// Layer 2: 50 tasks, each depends on all sources
let middle: Vec<_> = (0..50)
.map(|i| {
let combine = dag.add_task(Combine);
// Each middle task depends on two sources
combine.depends_on((
&sources[i % sources.len()],
&sources[(i + 1) % sources.len()],
))
})
.collect();
// Layer 3: Aggregate to 4 groups
let aggregates: Vec<_> = (0..4)
.map(|i| {
let start_idx = i * 12;
dag.add_task(Aggregate).depends_on((
&middle[start_idx % middle.len()],
&middle[(start_idx + 1) % middle.len()],
&middle[(start_idx + 2) % middle.len()],
&middle[(start_idx + 3) % middle.len()],
))
})
.collect();
// Layer 4: Final aggregate
let final_result = dag.add_task(Aggregate).depends_on((
&aggregates[0],
&aggregates[1],
&aggregates[2],
&aggregates[3],
));
let build_time = start.elapsed();
println!(" Build time: {:?}", build_time);
let exec_start = Instant::now();
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
let exec_time = exec_start.elapsed();
println!(" Execution time: {:?}", exec_time);
println!(" Total tasks: {}", 50 + 50 + 4 + 1);
println!(" Final result: {}\n", dag.get(final_result).unwrap());
}
// Example 5: Memory efficiency test
println!("Example 5: Memory Efficiency (100 tasks, measure overhead)");
{
let dag = DagRunner::new();
// Create tasks with various output sizes
let small_tasks: Vec<_> = (0..50)
.map(|i| dag.add_task(Compute { value: i }))
.collect();
let string_tasks: Vec<_> = (0..50)
.map(|i| dag.add_task(FormatNumber { id: i }))
.collect();
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
// Verify results are accessible
let i32_sum: i32 = small_tasks
.iter()
.map(|t| dag.get(t).unwrap())
.fold(0i32, |acc, x| acc.wrapping_add(x));
let string_count = string_tasks
.iter()
.map(|t| dag.get(t).unwrap().len())
.sum::<usize>();
println!(" i32 sum: {}", i32_sum);
println!(" String total length: {}", string_count);
}
println!("\n=== Large DAG Performance Example Complete ===");
println!("\nKey Observations:");
println!(" - DAG construction is very fast (< 1ms for 1000s of tasks)");
println!(" - Parallel execution scales well with available CPU cores");
println!(" - Deep chains execute sequentially as expected");
println!(" - Memory overhead is minimal (~ 200 bytes per task)");
}
-131
View File
@@ -1,131 +0,0 @@
//! # Proof of True Parallelism
//!
//! This example demonstrates that dagx provides genuine parallel execution
//! by running 10,000 tasks that each sleep for 1 second, completing in ~1 second
//! total instead of 10,000 seconds.
//!
//! ## What You'll Learn
//! - How to verify true parallel execution
//! - The difference between concurrency and parallelism
//! - How dagx scales to thousands of parallel tasks
//! - Performance characteristics of parallel execution
//!
//! ## The Proof
//!
//! If tasks ran sequentially:
//! - 10,000 tasks × 1 second each = 10,000 seconds (2.78 hours)
//!
//! With true parallelism:
//! - All 10,000 tasks run simultaneously = ~1 second total
//!
//! This dramatic difference (10,000x speedup) proves that tasks execute in
//! parallel across multiple threads, not sequentially.
//!
//! ## Key Concepts
//!
//! **Concurrency** - Multiple tasks making progress, but not necessarily at the same time
//! **Parallelism** - Multiple tasks executing simultaneously on different CPU cores
//!
//! This example proves dagx provides true parallelism.
//!
//! ## Running This Example
//! ```bash
//! cargo run --example parallelism_proof --release
//! ```
//!
//! ## Expected Output
//! ```text
//! === Parallelism Proof ===
//!
//! Launching 10,000 tasks that each sleep for 1 second...
//!
//! Result: 10,000 tasks × 1s sleep completed in ~1.2s
//!
//! Proof: If tasks ran sequentially, this would take 10,000 seconds (2.78 hours).
//! Actual time: 1.2s
//! Speedup: 8333x
//! This proves TRUE PARALLEL EXECUTION!
//! ```
use dagx::{task, DagRunner, Task};
use std::time::{Duration, Instant};
use tokio::time::sleep;
// Task that sleeps for a duration and returns its ID
struct SleepTask {
id: u32,
duration: Duration,
}
#[task]
impl SleepTask {
async fn run(&self) -> u32 {
sleep(self.duration).await;
self.id
}
}
#[tokio::main]
async fn main() {
println!("=== Parallelism Proof ===\n");
println!("Launching 10,000 tasks that each sleep for 1 second...\n");
let dag = DagRunner::new();
let sleep_duration = Duration::from_secs(1);
let num_tasks = 10_000u32;
// Create 10,000 independent tasks that each sleep for 1 second
let tasks: Vec<_> = (0..num_tasks)
.map(|i| {
dag.add_task(SleepTask {
id: i,
duration: sleep_duration,
})
})
.collect();
// Measure execution time
let start = Instant::now();
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
let elapsed = start.elapsed();
// Verify all tasks completed correctly
for (i, task) in tasks.iter().enumerate() {
assert_eq!(dag.get(task).unwrap(), i as u32);
}
// Calculate what sequential execution would take
let sequential_time = sleep_duration * num_tasks;
let speedup = sequential_time.as_secs_f64() / elapsed.as_secs_f64();
println!(
"Result: {} tasks × {}s sleep completed in {:.1}s\n",
num_tasks,
sleep_duration.as_secs(),
elapsed.as_secs_f64()
);
println!(
"Proof: If tasks ran sequentially, this would take {} seconds ({:.2} hours).",
sequential_time.as_secs(),
sequential_time.as_secs_f64() / 3600.0
);
println!(" Actual time: {:.1}s", elapsed.as_secs_f64());
println!(" Speedup: {:.0}x", speedup);
// The proof: execution time should be close to 1 second, not 10,000 seconds
if elapsed < Duration::from_secs(3) {
println!(" This proves TRUE PARALLEL EXECUTION!");
} else {
panic!(
"FAILED: Tasks appear to be running sequentially! Took {:.1}s instead of ~1s",
elapsed.as_secs_f64()
);
}
println!("\n=== Parallelism Proof Complete ===");
}
-470
View File
@@ -1,470 +0,0 @@
//! # Retry Strategies for Resilient Workflows
//!
//! This example demonstrates various retry patterns for handling transient failures
//! in DAG workflows, such as network errors, rate limits, or temporary service unavailability.
//!
//! ## What You'll Learn
//! - How to implement immediate, linear, and exponential backoff retry strategies
//! - How to distinguish retriable vs non-retriable errors
//! - How to track retry attempts and provide diagnostics
//! - How to implement jitter to prevent thundering herd problems
//!
//! ## Retry Patterns
//!
//! **1. Immediate Retry** - Retry without delay (for quick transient failures)
//! ```text
//! Attempt 1: ✗ Error
//! Attempt 2: ✗ Error (no delay)
//! Attempt 3: ✓ Success
//! ```
//!
//! **2. Linear Backoff** - Fixed delay between retries
//! ```text
//! Attempt 1: ✗ Error
//! Wait 100ms
//! Attempt 2: ✗ Error
//! Wait 100ms
//! Attempt 3: ✓ Success
//! ```
//!
//! **3. Exponential Backoff** - Increasing delays between retries
//! ```text
//! Attempt 1: ✗ Error
//! Wait 100ms
//! Attempt 2: ✗ Error
//! Wait 200ms
//! Attempt 3: ✗ Error
//! Wait 400ms
//! Attempt 4: ✓ Success
//! ```
//!
//! **4. Exponential Backoff with Jitter** - Randomized delays to prevent coordinated retries
//! ```text
//! Attempt 1: ✗ Error
//! Wait 87ms (50-150ms range)
//! Attempt 2: ✗ Error
//! Wait 234ms (100-300ms range)
//! Attempt 3: ✓ Success
//! ```
//!
//! ## Workflow Diagram
//! ```text
//! ┌──────────────────────┐
//! │ ApiCallWithRetry │
//! │ Max attempts: 3 │
//! │ Strategy: Exp │
//! └──────────────────────┘
//! │
//! ├─► Attempt 1: ✗ (Connection timeout)
//! │ Wait 100ms
//! │
//! ├─► Attempt 2: ✗ (Connection timeout)
//! │ Wait 200ms
//! │
//! └─► Attempt 3: ✓ Success
//!
//! ┌──────────────────────┐
//! │ ApiCallWithRetry │
//! │ Max attempts: 3 │
//! │ Strategy: Linear │
//! └──────────────────────┘
//! │
//! ├─► Attempt 1: ✗ (Rate limit)
//! │ Wait 500ms
//! │
//! ├─► Attempt 2: ✗ (Rate limit)
//! │ Wait 500ms
//! │
//! ├─► Attempt 3: ✗ (Rate limit)
//! │
//! └─► Final: ✗ Retry exhausted
//! ```
//!
//! ## Key Concepts
//! - **Idempotency**: Retried operations should be safe to repeat
//! - **Retriable errors**: Network timeouts, rate limits, temporary service errors
//! - **Non-retriable errors**: Authentication failures, invalid input, logical errors
//! - **Backoff strategies**: Balance between recovery speed and system load
//! - **Jitter**: Randomization to avoid thundering herd when many clients retry simultaneously
//! - **Retry budgets**: Limit total retries to prevent cascading failures
//!
//! ## Use Cases
//! - External API calls with transient network failures
//! - Database operations with temporary connection issues
//! - Distributed systems with eventual consistency
//! - Rate-limited API integrations
//! - Cloud service calls with intermittent availability
//!
//! ## Running This Example
//! ```bash
//! cargo run --example retry_strategies
//! ```
//!
//! ## Expected Output
//! ```text
//! === Retry Strategies Example ===
//!
//! Example 1: Immediate retry strategy
//! [ApiCall-1] Attempt 1/3: Calling external API...
//! [ApiCall-1] ✗ Transient error: Connection timeout
//! [ApiCall-1] Retrying immediately (attempt 2/3)
//! [ApiCall-1] Attempt 2/3: Calling external API...
//! [ApiCall-1] ✗ Transient error: Connection timeout
//! [ApiCall-1] Retrying immediately (attempt 3/3)
//! [ApiCall-1] Attempt 3/3: Calling external API...
//! [ApiCall-1] ✓ Success: API response data
//! Result: Ok(RetryResult { attempts: 3, success: true })
//!
//! Example 2: Linear backoff (500ms delay)
//! [ApiCall-2] Attempt 1/4: Calling external API...
//! [ApiCall-2] ✗ Transient error: Rate limit exceeded
//! [ApiCall-2] Waiting 500ms before retry (attempt 2/4)
//! [ApiCall-2] Attempt 2/4: Calling external API...
//! [ApiCall-2] ✓ Success: API response data
//! Result: Ok(RetryResult { attempts: 2, success: true })
//!
//! Example 3: Exponential backoff
//! [ApiCall-3] Attempt 1/5: Calling external API...
//! [ApiCall-3] ✗ Transient error: Service unavailable
//! [ApiCall-3] Waiting 100ms before retry (attempt 2/5)
//! [ApiCall-3] Attempt 2/5: Calling external API...
//! [ApiCall-3] ✗ Transient error: Service unavailable
//! [ApiCall-3] Waiting 200ms before retry (attempt 3/5)
//! [ApiCall-3] Attempt 3/5: Calling external API...
//! [ApiCall-3] ✓ Success: API response data
//! Result: Ok(RetryResult { attempts: 3, success: true })
//!
//! Example 4: Non-retriable error (fails immediately)
//! [ApiCall-4] Attempt 1/3: Calling external API...
//! [ApiCall-4] ✗ Non-retriable error: Authentication failed
//! Result: Err("Authentication failed - not retrying")
//!
//! Example 5: Retry exhausted
//! [ApiCall-5] Attempt 1/3: Calling external API...
//! [ApiCall-5] ✗ Transient error: Connection timeout
//! [ApiCall-5] Waiting 100ms before retry (attempt 2/3)
//! [ApiCall-5] Attempt 2/3: Calling external API...
//! [ApiCall-5] ✗ Transient error: Connection timeout
//! [ApiCall-5] Waiting 200ms before retry (attempt 3/3)
//! [ApiCall-5] Attempt 3/3: Calling external API...
//! [ApiCall-5] ✗ Transient error: Connection timeout
//! [ApiCall-5] All retry attempts exhausted
//! Result: Err("Retry exhausted after 3 attempts: Connection timeout")
//! ```
use dagx::{task, DagRunner, Task};
use tokio::time::{sleep, Duration};
// Retry strategy configuration
#[derive(Clone, Copy, Debug)]
enum RetryStrategy {
Immediate,
Linear { delay_ms: u64 },
Exponential { base_ms: u64, factor: f64 },
}
impl RetryStrategy {
fn calculate_delay(&self, attempt: u32) -> Option<Duration> {
match self {
RetryStrategy::Immediate => None,
RetryStrategy::Linear { delay_ms } => Some(Duration::from_millis(*delay_ms)),
RetryStrategy::Exponential { base_ms, factor } => {
let delay = *base_ms as f64 * factor.powi(attempt as i32 - 1);
Some(Duration::from_millis(delay as u64))
}
}
}
}
// Simulated error types
#[derive(Clone, Debug)]
enum ApiError {
// Retriable errors
ConnectionTimeout,
ServiceUnavailable,
RateLimitExceeded,
// Non-retriable errors
AuthenticationFailed,
#[allow(dead_code)]
InvalidInput,
}
impl ApiError {
fn is_retriable(&self) -> bool {
matches!(
self,
ApiError::ConnectionTimeout
| ApiError::ServiceUnavailable
| ApiError::RateLimitExceeded
)
}
fn message(&self) -> &'static str {
match self {
ApiError::ConnectionTimeout => "Connection timeout",
ApiError::ServiceUnavailable => "Service unavailable",
ApiError::RateLimitExceeded => "Rate limit exceeded",
ApiError::AuthenticationFailed => "Authentication failed",
ApiError::InvalidInput => "Invalid input",
}
}
}
// Result of retry operation
#[derive(Clone, Debug)]
#[allow(dead_code)]
struct RetryResult {
attempts: u32,
success: bool,
final_error: Option<String>,
}
// Task that simulates an API call with configurable failure pattern
struct ApiCallWithRetry {
task_id: String,
max_attempts: u32,
strategy: RetryStrategy,
// Simulated failure pattern: errors to return before success
failure_sequence: Vec<ApiError>,
current_attempt: u32,
}
impl ApiCallWithRetry {
fn new(
task_id: impl Into<String>,
max_attempts: u32,
strategy: RetryStrategy,
failure_sequence: Vec<ApiError>,
) -> Self {
Self {
task_id: task_id.into(),
max_attempts,
strategy,
failure_sequence,
current_attempt: 0,
}
}
async fn attempt_api_call(&mut self) -> Result<String, ApiError> {
println!(
"[{}] Attempt {}/{}: Calling external API...",
self.task_id,
self.current_attempt + 1,
self.max_attempts
);
// Simulate network delay
sleep(Duration::from_millis(50)).await;
// Check if we should fail this attempt
if (self.current_attempt as usize) < self.failure_sequence.len() {
let error = self.failure_sequence[self.current_attempt as usize].clone();
Err(error)
} else {
Ok("API response data".to_string())
}
}
}
#[task]
impl ApiCallWithRetry {
async fn run(&mut self) -> Result<RetryResult, String> {
loop {
self.current_attempt += 1;
match self.attempt_api_call().await {
Ok(data) => {
println!("[{}] ✓ Success: {}", self.task_id, data);
return Ok(RetryResult {
attempts: self.current_attempt,
success: true,
final_error: None,
});
}
Err(error) => {
println!(
"[{}] ✗ {} error: {}",
self.task_id,
if error.is_retriable() {
"Transient"
} else {
"Non-retriable"
},
error.message()
);
// Check if error is retriable
if !error.is_retriable() {
return Err(format!("{} - not retrying", error.message()));
}
// Check if we have attempts remaining
if self.current_attempt >= self.max_attempts {
println!("[{}] All retry attempts exhausted", self.task_id);
return Err(format!(
"Retry exhausted after {} attempts: {}",
self.current_attempt,
error.message()
));
}
// Calculate and apply backoff delay
if let Some(delay) = self.strategy.calculate_delay(self.current_attempt) {
println!(
"[{}] Waiting {}ms before retry (attempt {}/{})",
self.task_id,
delay.as_millis(),
self.current_attempt + 1,
self.max_attempts
);
sleep(delay).await;
} else {
println!(
"[{}] Retrying immediately (attempt {}/{})",
self.task_id,
self.current_attempt + 1,
self.max_attempts
);
}
// Continue loop to retry
}
}
}
}
}
#[tokio::main]
async fn main() {
println!("=== Retry Strategies Example ===\n");
// Example 1: Immediate retry - succeeds on 3rd attempt
println!("Example 1: Immediate retry strategy");
{
let dag = DagRunner::new();
let api_call = dag.add_task(ApiCallWithRetry::new(
"ApiCall-1",
3,
RetryStrategy::Immediate,
vec![
ApiError::ConnectionTimeout,
ApiError::ConnectionTimeout,
// 3rd attempt will succeed
],
));
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
println!("Result: {:?}\n", dag.get(api_call).unwrap());
}
// Example 2: Linear backoff - succeeds on 2nd attempt
println!("Example 2: Linear backoff (500ms delay)");
{
let dag = DagRunner::new();
let api_call = dag.add_task(ApiCallWithRetry::new(
"ApiCall-2",
4,
RetryStrategy::Linear { delay_ms: 500 },
vec![
ApiError::RateLimitExceeded,
// 2nd attempt will succeed
],
));
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
println!("Result: {:?}\n", dag.get(api_call).unwrap());
}
// Example 3: Exponential backoff - succeeds on 3rd attempt
println!("Example 3: Exponential backoff");
{
let dag = DagRunner::new();
let api_call = dag.add_task(ApiCallWithRetry::new(
"ApiCall-3",
5,
RetryStrategy::Exponential {
base_ms: 100,
factor: 2.0,
},
vec![
ApiError::ServiceUnavailable,
ApiError::ServiceUnavailable,
// 3rd attempt will succeed
],
));
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
println!("Result: {:?}\n", dag.get(api_call).unwrap());
}
// Example 4: Non-retriable error - fails immediately
println!("Example 4: Non-retriable error (fails immediately)");
{
let dag = DagRunner::new();
let api_call = dag.add_task(ApiCallWithRetry::new(
"ApiCall-4",
3,
RetryStrategy::Immediate,
vec![ApiError::AuthenticationFailed],
));
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
println!("Result: {:?}\n", dag.get(api_call).unwrap());
}
// Example 5: Retry exhausted - all attempts fail
println!("Example 5: Retry exhausted");
{
let dag = DagRunner::new();
let api_call = dag.add_task(ApiCallWithRetry::new(
"ApiCall-5",
3,
RetryStrategy::Exponential {
base_ms: 100,
factor: 2.0,
},
vec![
ApiError::ConnectionTimeout,
ApiError::ConnectionTimeout,
ApiError::ConnectionTimeout,
// All attempts will fail
],
));
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
println!("Result: {:?}\n", dag.get(api_call).unwrap());
}
println!("=== Retry Strategies Example Complete ===");
}
-303
View File
@@ -1,303 +0,0 @@
//! # Managing Timeouts in Tasks
//!
//! This example demonstrates how to implement timeout patterns in DAG tasks using
//! tokio's timeout utilities for time-bounded execution.
//!
//! ## What You'll Learn
//! - How to implement timeouts using `tokio::time::timeout`
//! - How to build tasks with built-in timeout logic
//! - How to handle timeout errors as Result types
//! - How to apply timeouts at both task-level and DAG-level
//!
//! ## Timeout Patterns
//!
//! **1. Task-Level Timeout** - Individual task with timeout
//! ```rust
//! async fn run(&mut self) -> Result<String, String> {
//! let work = async { /* ... */ };
//! match timeout(Duration::from_millis(self.timeout_ms), work).await {
//! Ok(result) => Ok(result),
//! Err(_) => Err("Task timed out".to_string()),
//! }
//! }
//! ```
//!
//! **2. DAG-Level Timeout** - Entire workflow with timeout
//! ```rust
//! match timeout(dag_timeout, dag.run(spawner)).await {
//! Ok(result) => { /* DAG completed */ },
//! Err(_) => { /* DAG timed out */ },
//! }
//! ```
//!
//! ## Workflow Diagram
//! ```text
//! ┌─────────────┐
//! │ FastTask │ (completes in 100ms)
//! │ (timeout │ ───► Ok("Fast task completed")
//! │ 500ms) │
//! └─────────────┘
//!
//! ┌─────────────┐
//! │ SlowTask │ (takes 1000ms)
//! │ (timeout │ ───► Err("Task timed out after 200ms")
//! │ 200ms) │
//! └─────────────┘
//! │
//! └──────►┌──────────────┐
//! │ProcessResult │ ───► "✗ Task timed out..."
//! └──────────────┘
//! ```
//!
//! ## Key Concepts
//! - **Timeout as error**: Timeouts produce `Err` results that propagate through the DAG
//! - **Error as data**: Use Result<T, E> to represent timeout states
//! - **Graceful degradation**: Downstream tasks can handle timeout errors
//! - **Time-bounded execution**: Enforce SLAs and prevent runaway tasks
//!
//! ## Use Cases
//! - API calls with timeout limits
//! - SLA enforcement in workflows
//! - Preventing resource exhaustion from slow tasks
//! - Circuit breaker patterns
//! - External service integration with timeouts
//!
//! ## Running This Example
//! ```bash
//! cargo run --example timeout
//! ```
//!
//! ## Expected Output
//! ```text
//! === Timeout Example ===
//!
//! Example 1: All tasks complete successfully
//! Fast task: Ok("Fast task completed")
//! Timed task: Ok("Work completed in 100ms")
//!
//! Example 2: Task exceeds timeout
//! Result: ✗ Task timed out after 200ms (work would take 1000ms)
//!
//! Example 3: Parallel tasks with different speeds
//! Fast task 1: Ok("Fast task completed")
//! Fast task 2: Ok("Fast task completed")
//! Slow task: Ok("Slow task completed after 500ms")
//! Timed task 1: Ok("Work completed in 100ms")
//! Timed task 2: Err("Task timed out after 200ms...")
//!
//! Example 4: Downstream tasks handling timeout errors
//! Processed 1: ✗ Task timed out after 100ms (work would take 500ms)
//! Processed 2: ✗ Task timed out after 100ms (work would take 500ms)
//!
//! Example 5: Whole-DAG timeout
//! DAG completed within timeout
//! Task 1: Ok("Slow task completed after 200ms")
//! Task 2: Ok("Slow task completed after 200ms")
//! Task 3: Ok("Slow task completed after 200ms")
//!
//! === Timeout Example Complete ===
//! ```
use dagx::{task, DagRunner, Task};
use tokio::time::{sleep, timeout, Duration};
// Fast task that completes quickly
struct FastTask;
#[task]
impl FastTask {
async fn run() -> Result<String, String> {
sleep(Duration::from_millis(100)).await;
Ok("Fast task completed".to_string())
}
}
// Slow task that takes a long time
struct SlowTask {
duration_ms: u64,
}
#[task]
impl SlowTask {
async fn run(&mut self) -> Result<String, String> {
sleep(Duration::from_millis(self.duration_ms)).await;
Ok(format!("Slow task completed after {}ms", self.duration_ms))
}
}
// Task with built-in timeout
struct TaskWithTimeout {
work_duration_ms: u64,
timeout_ms: u64,
}
#[task]
impl TaskWithTimeout {
async fn run(&mut self) -> Result<String, String> {
let work = async {
sleep(Duration::from_millis(self.work_duration_ms)).await;
format!("Work completed in {}ms", self.work_duration_ms)
};
match timeout(Duration::from_millis(self.timeout_ms), work).await {
Ok(result) => Ok(result),
Err(_) => Err(format!(
"Task timed out after {}ms (work would take {}ms)",
self.timeout_ms, self.work_duration_ms
)),
}
}
}
// Task that processes results (including timeouts)
struct ProcessResult;
#[task]
impl ProcessResult {
async fn run(input: &Result<String, String>) -> String {
match input {
Ok(msg) => format!("✓ {}", msg),
Err(err) => format!("✗ {}", err),
}
}
}
#[tokio::main]
async fn main() {
println!("=== Timeout Example ===\n");
// Example 1: All tasks complete within timeout
println!("Example 1: All tasks complete successfully");
{
let dag = DagRunner::new();
let fast = dag.add_task(FastTask);
let timed = dag.add_task(TaskWithTimeout {
work_duration_ms: 100,
timeout_ms: 500,
});
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
println!("Fast task: {:?}", dag.get(fast).unwrap());
println!("Timed task: {:?}\n", dag.get(timed).unwrap());
}
// Example 2: Task exceeds timeout
println!("Example 2: Task exceeds timeout");
{
let dag = DagRunner::new();
let timed = dag.add_task(TaskWithTimeout {
work_duration_ms: 1000,
timeout_ms: 200,
});
let processed = dag.add_task(ProcessResult).depends_on(&timed);
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
println!("Result: {}\n", dag.get(processed).unwrap());
}
// Example 3: Mixed fast and slow tasks
println!("Example 3: Parallel tasks with different speeds");
{
let dag = DagRunner::new();
let fast1 = dag.add_task(FastTask);
let fast2 = dag.add_task(FastTask);
let slow = dag.add_task(SlowTask { duration_ms: 500 });
let timed1 = dag.add_task(TaskWithTimeout {
work_duration_ms: 100,
timeout_ms: 200,
});
let timed2 = dag.add_task(TaskWithTimeout {
work_duration_ms: 300,
timeout_ms: 200,
});
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
println!("Fast task 1: {:?}", dag.get(fast1).unwrap());
println!("Fast task 2: {:?}", dag.get(fast2).unwrap());
println!("Slow task: {:?}", dag.get(slow).unwrap());
println!("Timed task 1: {:?}", dag.get(timed1).unwrap());
println!("Timed task 2: {:?}\n", dag.get(timed2).unwrap());
}
// Example 4: Cascading timeout errors
println!("Example 4: Downstream tasks handling timeout errors");
{
let dag = DagRunner::new();
let source = dag.add_task(TaskWithTimeout {
work_duration_ms: 500,
timeout_ms: 100,
});
let processed1 = dag.add_task(ProcessResult).depends_on(&source);
let processed2 = dag.add_task(ProcessResult).depends_on(&source);
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
println!("Processed 1: {}", dag.get(processed1).unwrap());
println!("Processed 2: {}\n", dag.get(processed2).unwrap());
}
// Example 5: Implementing whole-DAG timeout
println!("Example 5: Whole-DAG timeout");
{
let dag = DagRunner::new();
let t1 = dag.add_task(SlowTask { duration_ms: 200 });
let t2 = dag.add_task(SlowTask { duration_ms: 200 });
let t3 = dag.add_task(SlowTask { duration_ms: 200 });
// Run the entire DAG with a timeout
let dag_timeout = Duration::from_millis(500);
match timeout(
dag_timeout,
dag.run(|fut| {
tokio::spawn(fut);
}),
)
.await
{
Ok(result) => {
result.unwrap();
println!("DAG completed within timeout");
println!("Task 1: {:?}", dag.get(t1).unwrap());
println!("Task 2: {:?}", dag.get(t2).unwrap());
println!("Task 3: {:?}", dag.get(t3).unwrap());
}
Err(_) => {
eprintln!(
"DAG execution timed out after {}ms",
dag_timeout.as_millis()
);
}
}
}
println!("\n=== Timeout Example Complete ===");
}
+120 -64
View File
@@ -1,12 +1,17 @@
#!/bin/bash #!/bin/bash
# Release Readiness Checker for dagx v0.1.0 # Release Readiness Checker for dagx
# This script performs EXHAUSTIVE checks before publishing to crates.io # This script performs EXHAUSTIVE checks before publishing to crates.io
set -e set -e
echo "========================================" # =============================================================================
echo "dagx v0.1.0 EXHAUSTIVE Release Check" # CONFIGURATION
echo "========================================" # =============================================================================
VERSION="0.3.1"
echo "=================="
echo "dagx Release Check"
echo "=================="
echo "" echo ""
# Color codes # Color codes
@@ -45,13 +50,13 @@ section "PRE-FLIGHT CHECKS"
echo "→ Checking Rust version..." echo "→ Checking Rust version..."
RUST_VERSION=$(rustc --version | awk '{print $2}') RUST_VERSION=$(rustc --version | awk '{print $2}')
echo " Current: $RUST_VERSION" echo " Current: $RUST_VERSION"
REQUIRED_VERSION="1.85.0" MSRV=$(grep '^rust-version' Cargo.toml | cut -d'"' -f2)
echo " Required (MSRV): $REQUIRED_VERSION" echo " Required (MSRV from Cargo.toml): $MSRV"
check_pass "Rust version compatible" check_pass "Rust version compatible"
echo "" echo ""
echo "→ Checking for uncommitted changes..." echo "→ Checking for uncommitted changes..."
if ! git diff --quiet || ! git diff --cached --quiet; then if jj status 2>&1 | grep -q "Working copy changes:"; then
check_warn "Uncommitted changes found - you'll need to commit before publishing" check_warn "Uncommitted changes found - you'll need to commit before publishing"
else else
check_pass "No uncommitted changes" check_pass "No uncommitted changes"
@@ -83,13 +88,67 @@ check_pass "Zero clippy warnings"
echo "" echo ""
echo "→ Checking for TODO/FIXME/XXX/HACK in source..." echo "→ Checking for TODO/FIXME/XXX/HACK in source..."
TODO_COUNT=$(grep -r "TODO\|FIXME\|XXX\|HACK" src/ dagx-macros/src/ 2>/dev/null | wc -l) TODO_COUNT=$(grep -r "TODO\|FIXME\|XXX\|HACK" src/ dagx-macros/src/ dagx-test/src/ 2>/dev/null | wc -l)
if [ "$TODO_COUNT" -gt 0 ]; then if [ "$TODO_COUNT" -gt 0 ]; then
check_warn "Found $TODO_COUNT TODO/FIXME/HACK comments in source" check_warn "Found $TODO_COUNT TODO/FIXME/HACK comments in source"
else else
check_pass "No TODO/FIXME/HACK comments in source" check_pass "No TODO/FIXME/HACK comments in source"
fi fi
# =============================================================================
# MSRV VERIFICATION
# =============================================================================
section "MSRV VERIFICATION"
echo "→ Verifying MSRV with no features..."
cargo msrv verify --no-default-features || check_fail "MSRV check failed (no features)"
check_pass "MSRV verified (no features)"
echo ""
echo "→ Verifying MSRV with all features..."
cargo msrv verify --all-features || check_fail "MSRV check failed (all features)"
check_pass "MSRV verified (all features)"
echo ""
echo "→ Verifying MSRV with default features..."
cargo msrv verify || check_fail "MSRV check failed (default features)"
check_pass "MSRV verified (default features)"
# =============================================================================
# FEATURE COMBINATION TESTING
# =============================================================================
section "FEATURE COMBINATION TESTING"
echo "→ Testing with no features..."
cargo test --lib --no-default-features || check_fail "Tests failed (no features)"
check_pass "Tests pass (no features)"
echo ""
echo "→ Testing with all features..."
cargo test --lib --all-features || check_fail "Tests failed (all features)"
check_pass "Tests pass (all features)"
echo ""
echo "→ Testing with default features..."
cargo test --lib || check_fail "Tests failed (default features)"
check_pass "Tests pass (default features)"
echo ""
echo "→ Clippy with all feature combinations..."
cargo clippy --all-targets --no-default-features -- -D warnings || check_fail "Clippy failed (no features)"
cargo clippy --all-targets --all-features -- -D warnings || check_fail "Clippy failed (all features)"
cargo clippy --all-targets -- -D warnings || check_fail "Clippy failed (default features)"
check_pass "Clippy clean across all feature combinations"
echo ""
echo "→ Cargo check with all feature combinations..."
cargo check --no-default-features || check_fail "Check failed (no features)"
cargo check --all-features || check_fail "Check failed (all features)"
cargo check || check_fail "Check failed (default features)"
check_pass "Cargo check passes across all feature combinations"
# ============================================================================= # =============================================================================
# TESTING # TESTING
# ============================================================================= # =============================================================================
@@ -98,17 +157,21 @@ section "TESTING"
echo "→ Running library unit tests..." echo "→ Running library unit tests..."
cargo test --lib || check_fail "Library unit tests failed" cargo test --lib || check_fail "Library unit tests failed"
check_pass "Library unit tests pass (70 tests)" check_pass "Library unit tests pass"
echo "" echo ""
echo "→ Running integration tests..." echo "→ Running integration tests with all feature combinations..."
cargo test --test '*' 2>&1 | grep "test result" || check_fail "Integration tests failed" cargo test --test '*' --no-default-features || check_fail "Integration tests failed (no features)"
check_pass "Integration tests pass (280+ tests total)" cargo test --test '*' --all-features || check_fail "Integration tests failed (all features)"
cargo test --test '*' || check_fail "Integration tests failed (default features)"
check_pass "Integration tests pass across all feature combinations"
echo "" echo ""
echo "→ Running doc tests..." echo "→ Running doc tests with all features..."
cargo test --doc || check_fail "Doc tests failed" cargo test --doc --all-features || check_fail "Doc tests failed (all features)"
check_pass "Doc tests pass" cargo test --doc --no-default-features || check_fail "Doc tests failed (no features)"
cargo test --doc || check_fail "Doc tests failed (default features)"
check_pass "Doc tests pass across all feature combinations"
# ============================================================================= # =============================================================================
# DOCUMENTATION # DOCUMENTATION
@@ -130,21 +193,44 @@ echo "→ Verifying README.md..."
grep -q "# dagx" README.md || check_fail "README.md title incorrect" grep -q "# dagx" README.md || check_fail "README.md title incorrect"
grep -q "https://crates.io/crates/dagx" README.md || check_fail "README.md crates.io badge incorrect" grep -q "https://crates.io/crates/dagx" README.md || check_fail "README.md crates.io badge incorrect"
grep -q "https://docs.rs/dagx" README.md || check_fail "README.md docs.rs badge incorrect" grep -q "https://docs.rs/dagx" README.md || check_fail "README.md docs.rs badge incorrect"
grep -q "https://github.com/swaits/dagx" README.md || check_fail "README.md GitHub URL incorrect" grep -q "https://git.swaits.com/swaits/dagx" README.md || check_fail "README.md repository URL incorrect"
# Extract major.minor version (e.g., "0.2.1" -> "0.2")
MAJOR_MINOR=$(echo "$VERSION" | cut -d'.' -f1-2)
grep -q "dagx = \"$MAJOR_MINOR\"" README.md || check_fail "README.md dependency version not \"$MAJOR_MINOR\" (found: $(grep 'dagx = ' README.md | head -1))"
check_pass "README.md looks perfect" check_pass "README.md looks perfect"
echo "" echo ""
echo "→ Verifying CHANGELOG.md..." echo "→ Verifying CHANGELOG.md..."
[ -f CHANGELOG.md ] || check_fail "CHANGELOG.md missing" [ -f CHANGELOG.md ] || check_fail "CHANGELOG.md missing"
grep -q "0.1.0" CHANGELOG.md || check_fail "CHANGELOG.md doesn't mention v0.1.0" grep -q "$VERSION" CHANGELOG.md || check_fail "CHANGELOG.md doesn't mention v$VERSION"
check_pass "CHANGELOG.md exists and mentions v0.1.0" check_pass "CHANGELOG.md exists and mentions v$VERSION"
echo "" echo ""
echo "→ Verifying LICENSE..." echo "→ Verifying LICENSE..."
[ -f LICENSE ] || check_fail "LICENSE missing" [ -f LICENSE ] || check_fail "LICENSE missing"
[ -f dagx-macros/LICENSE ] || check_fail "dagx-macros/LICENSE missing" [ -f dagx-macros/LICENSE ] || check_fail "dagx-macros/LICENSE missing"
[ -f dagx-test/LICENSE ] || check_fail "dagx-test/LICENSE missing"
check_pass "LICENSE files present in both crates" check_pass "LICENSE files present in both crates"
# =============================================================================
# COVERAGE CHECK
# =============================================================================
section "COVERAGE CHECK"
echo "→ Running coverage analysis..."
COVERAGE=$(cargo tarpaulin --out Stdout 2>&1 | grep -oP '^\d+\.\d+%' | head -1 | sed 's/%//')
if [ -z "$COVERAGE" ]; then
check_warn "Could not determine coverage percentage"
else
echo " Current coverage: ${COVERAGE}%"
if (($(echo "$COVERAGE < 80" | bc -l))); then
check_warn "Coverage is ${COVERAGE}% (below 80% target)"
else
check_pass "Coverage is ${COVERAGE}% (meets 80% target)"
fi
fi
# ============================================================================= # =============================================================================
# BUILD VERIFICATION # BUILD VERIFICATION
# ============================================================================= # =============================================================================
@@ -174,45 +260,13 @@ echo "→ Verifying benchmarks compile..."
cargo bench --no-run || check_fail "Benchmarks failed to compile" cargo bench --no-run || check_fail "Benchmarks failed to compile"
check_pass "Benchmarks compile successfully" check_pass "Benchmarks compile successfully"
# =============================================================================
# CARGO.TOML METADATA VERIFICATION
# =============================================================================
section "CARGO.TOML METADATA"
echo "→ Checking dagx Cargo.toml..."
grep -q 'name = "dagx"' Cargo.toml || check_fail "Package name incorrect"
grep -q 'version = "0.1.0"' Cargo.toml || check_fail "Version not 0.1.0"
grep -q 'edition = "2021"' Cargo.toml || check_fail "Edition not 2021"
grep -q 'license = "MIT"' Cargo.toml || check_fail "License not MIT"
grep -q 'description = "A minimal, type-safe' Cargo.toml || check_fail "Description missing"
grep -q 'repository = "https://github.com/swaits/dagx"' Cargo.toml || check_fail "Repository URL incorrect"
grep -q 'homepage = "https://github.com/swaits/dagx"' Cargo.toml || check_fail "Homepage URL incorrect"
grep -q 'documentation = "https://docs.rs/dagx"' Cargo.toml || check_fail "Documentation URL incorrect"
grep -q 'readme = "README.md"' Cargo.toml || check_fail "README not specified"
grep -q 'keywords = \["dag"' Cargo.toml || check_fail "Keywords missing"
grep -q 'categories = \["asynchronous"' Cargo.toml || check_fail "Categories missing"
check_pass "dagx Cargo.toml metadata complete and correct"
echo ""
echo "→ Checking dagx-macros Cargo.toml..."
grep -q 'name = "dagx-macros"' dagx-macros/Cargo.toml || check_fail "dagx-macros name incorrect"
grep -q 'version = "0.1.0"' dagx-macros/Cargo.toml || check_fail "dagx-macros version not 0.1.0"
grep -q 'edition = "2021"' dagx-macros/Cargo.toml || check_fail "dagx-macros edition not 2021"
grep -q 'license = "MIT"' dagx-macros/Cargo.toml || check_fail "dagx-macros license not MIT"
grep -q 'description = "Procedural macros for dagx"' dagx-macros/Cargo.toml || check_fail "dagx-macros description missing"
grep -q 'repository = "https://github.com/swaits/dagx"' dagx-macros/Cargo.toml || check_fail "dagx-macros repository URL incorrect"
grep -q 'homepage = "https://github.com/swaits/dagx"' dagx-macros/Cargo.toml || check_fail "dagx-macros homepage URL incorrect"
grep -q 'documentation = "https://docs.rs/dagx-macros"' dagx-macros/Cargo.toml || check_fail "dagx-macros documentation URL incorrect"
grep -q 'readme = "README.md"' dagx-macros/Cargo.toml || check_fail "dagx-macros README not specified"
check_pass "dagx-macros Cargo.toml metadata complete and correct"
echo "" echo ""
echo "→ Verifying version consistency..." echo "→ Verifying version consistency..."
DAGX_VERSION=$(grep '^version = ' Cargo.toml | head -1 | cut -d'"' -f2) DAGX_VERSION=$(grep '^version = ' Cargo.toml | head -1 | cut -d'"' -f2)
MACROS_VERSION=$(grep '^version = ' dagx-macros/Cargo.toml | head -1 | cut -d'"' -f2) MACROS_VERSION=$(grep '^version = ' dagx-macros/Cargo.toml | head -1 | cut -d'"' -f2)
if [ "$DAGX_VERSION" != "$MACROS_VERSION" ]; then TEST_VERSION=$(grep '^version = ' dagx-test/Cargo.toml | head -1 | cut -d'"' -f2)
check_fail "Version mismatch: dagx=$DAGX_VERSION, dagx-macros=$MACROS_VERSION" if [ "$DAGX_VERSION" != "$MACROS_VERSION" ] || [ "$DAGX_VERSION" != "$TEST_VERSION" ]; then
check_fail "Version mismatch: dagx=$DAGX_VERSION, dagx-macros=$MACROS_VERSION, dagx-test=$TEST_VERSION"
else else
check_pass "Versions match: both are $DAGX_VERSION" check_pass "Versions match: both are $DAGX_VERSION"
fi fi
@@ -230,8 +284,8 @@ check_pass "dagx-macros is ready to publish"
echo "" echo ""
echo "→ Testing dagx package (dry-run with --allow-dirty)..." echo "→ Testing dagx package (dry-run with --allow-dirty)..."
echo " Note: This will fail until dagx-macros is published to crates.io" echo " Note: This will fail until dagx-macros is published to crates.io"
if cargo publish --dry-run --allow-dirty 2>&1 | grep -q "no matching package named \`dagx-macros\`"; then if cargo publish --dry-run --allow-dirty 2>&1 | grep -q "failed to select a version for the requirement.*dagx-macros"; then
check_warn "dagx package check shows dependency on unpublished dagx-macros (expected)" check_warn "dagx package check shows dependency on unpublished dagx-macros v$VERSION (expected)"
else else
cargo publish --dry-run --allow-dirty || check_fail "dagx package check failed unexpectedly" cargo publish --dry-run --allow-dirty || check_fail "dagx package check failed unexpectedly"
check_pass "dagx package is ready (dagx-macros must be published first)" check_pass "dagx package is ready (dagx-macros must be published first)"
@@ -259,7 +313,7 @@ fi
section "SECURITY CHECKS" section "SECURITY CHECKS"
echo "→ Checking for sensitive data..." echo "→ Checking for sensitive data..."
if grep -r "api[_-]key\|secret\|password\|token" src/ dagx-macros/src/ 2>/dev/null | grep -v "^Binary" | grep -v "token for a node" | grep -q .; then if grep -r "api[_-]key\|secret\|password\|token" src/ dagx-macros/src/ dagx-test/src 2>/dev/null | grep -v "^Binary" | grep -v "token for a node" | grep -q .; then
check_fail "Potential sensitive data found in source" check_fail "Potential sensitive data found in source"
else else
check_pass "No sensitive data found" check_pass "No sensitive data found"
@@ -284,16 +338,15 @@ echo -e "${GREEN}━━━━━━━━━━━━━━━━━━━━━
echo -e "${GREEN}✓ ALL CHECKS PASSED!${NC}" echo -e "${GREEN}✓ ALL CHECKS PASSED!${NC}"
echo -e "${GREEN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}" echo -e "${GREEN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
echo "" echo ""
echo "dagx v0.1.0 is READY FOR RELEASE!" echo "dagx v$VERSION is READY FOR RELEASE!"
echo "" echo ""
echo -e "${YELLOW}PUBLICATION INSTRUCTIONS:${NC}" echo -e "${YELLOW}PUBLICATION INSTRUCTIONS:${NC}"
echo "" echo ""
echo "1. Commit any uncommitted changes:" echo "1. Commit any uncommitted changes:"
echo " git add ." echo " jj commit -m 'chore: release v$VERSION'"
echo " git commit -m 'Release v0.1.0'"
echo "" echo ""
echo "2. Tag the release:" echo "2. Create a bookmark for the release:"
echo " git tag -a v0.1.0 -m 'Release v0.1.0'" echo " jj bookmark create v$VERSION"
echo "" echo ""
echo "3. Publish dagx-macros FIRST:" echo "3. Publish dagx-macros FIRST:"
echo " cd dagx-macros" echo " cd dagx-macros"
@@ -303,9 +356,12 @@ echo "4. Then publish dagx:"
echo " cd .." echo " cd .."
echo " cargo publish" echo " cargo publish"
echo "" echo ""
echo "5. Push changes and tag:" echo "5. Then publish dagx-test:"
echo " git push origin main" echo " cd dagx-test"
echo " git push origin v0.1.0" echo " cargo publish"
echo ""
echo "6. Push changes and bookmark:"
echo " jj git push"
echo "" echo ""
echo -e "${BLUE}Note: dagx-macros MUST be published before dagx${NC}" echo -e "${BLUE}Note: dagx-macros MUST be published before dagx${NC}"
echo -e "${BLUE}because dagx depends on dagx-macros from crates.io${NC}" echo -e "${BLUE}because dagx depends on dagx-macros from crates.io${NC}"
+140 -37
View File
@@ -1,33 +1,26 @@
//! Task builder with type-state pattern for dependency tracking. //! Task builder with type-state pattern for dependency tracking.
//! //!
//! TaskBuilder uses compile-time type checking to ensure dependencies are wired correctly. //! TaskBuilder uses compile-time type checking to ensure dependencies are wired correctly.
//! The type parameter tracks whether dependencies have been specified yet.
use std::marker::PhantomData;
use crate::deps::DepsTuple; use crate::deps::DepsTuple;
use crate::runner::DagRunner; use crate::runner::DagRunner;
use crate::task::Task; use crate::task::Task;
use crate::types::{NodeId, TaskHandle};
/// Marker trait to identify the unit type at compile time. #[cfg(feature = "tracing")]
/// use tracing::debug;
/// This is used to enforce that tasks with non-unit inputs MUST call `.depends_on()`,
/// while tasks with unit inputs can be used directly.
pub trait IsUnitType {}
/// Only the unit type implements IsUnitType
impl IsUnitType for () {}
/// Node builder that tracks dependency completion via type state. /// Node builder that tracks dependency completion via type state.
/// ///
/// A `TaskBuilder<Tk, Deps>` is returned from [`DagRunner::add_task`] and uses the type-state /// A `TaskBuilder<Tk, Input>` is returned from [`DagRunner::add_task`] for tasks with non-unit inputs.
/// pattern to ensure dependencies are wired correctly at compile time: /// After calling [`TaskBuilder::depends_on`], it becomes a [`TaskHandle`].
/// - `TaskBuilder<Tk, Pending>`: No dependencies specified yet
/// - `TaskBuilder<Tk, Tk::Input>`: Dependencies complete; acts as a [`TaskHandle<Tk::Output>`]
/// ///
/// # Examples /// # Examples
/// ///
/// ```no_run /// ```no_run
/// # use dagx::{task, DagRunner, Task}; /// # use dagx::{task, DagRunner, Task};
/// #
/// // Using tuple struct for simple constants /// // Using tuple struct for simple constants
/// struct Constant(i32); /// struct Constant(i32);
/// ///
@@ -51,41 +44,53 @@ impl IsUnitType for () {}
/// # async { /// # async {
/// let mut dag = DagRunner::new(); /// let mut dag = DagRunner::new();
/// ///
/// // Construct with initial value (tuple struct)
/// let a = dag.add_task(Constant(10)); /// let a = dag.add_task(Constant(10));
/// // a is TaskBuilder<_, Pending> since no dependencies needed (Input = ()) /// // a is TaskHandle since no dependencies needed (Input = ())
/// ///
/// // Construct with ::new() pattern
/// let b = dag.add_task(Multiplier::new(2)); /// let b = dag.add_task(Multiplier::new(2));
/// // b is TaskBuilder<_, Pending> until we call depends_on() /// // b is TaskBuilder until we call depends_on()
/// ///
/// let b = b.depends_on(&a); /// let b = b.depends_on(&a);
/// // Now b is a TaskHandle<i32> /// // Now b is a TaskHandle<i32>
/// ///
/// dag.run(|fut| { tokio::spawn(fut); }).await.unwrap(); ///let mut output = dag.run(|fut| async move { tokio::spawn(fut).await.unwrap() }).await.unwrap();
/// assert_eq!(dag.get(&b).unwrap(), 20); /// assert_eq!(output.get(b), 20);
/// # }; /// # };
/// ``` /// ```
pub struct TaskBuilder<'a, Tk: Task, Deps> { #[must_use]
pub struct TaskBuilder<'a, Input, Tk>
where
Tk: Task<Input>,
Input: Send + Sync + 'static,
{
pub(crate) id: NodeId, pub(crate) id: NodeId,
pub(crate) dag: &'a DagRunner, pub(crate) dag: &'a mut DagRunner,
pub(crate) _phantom: std::marker::PhantomData<(Tk, Deps)>, pub(crate) _phantom: PhantomData<(Tk, Input)>,
} }
impl<'a, Tk: Task, Deps> TaskBuilder<'a, Tk, Deps> { impl<'a, Input, Tk> TaskBuilder<'a, Input, Tk>
where
Tk: Task<Input>,
Input: Send + Sync + 'static,
{
/// Provide all dependencies exactly once as a tuple. /// Provide all dependencies exactly once as a tuple.
/// ///
/// The dependencies must match the task's `Input` type exactly: /// The dependencies must match the task's `Input` type exactly:
/// - `Input = ()`: Do **not** call `depends_on` /// - `Input = A`: Pass `&TaskHandle<A>`
/// - `Input = T`: Pass `&TaskHandle` or `(&TaskHandle,)`
/// - `Input = (A, B, ...)`: Pass `(&TaskHandle<A>, &TaskHandle<B>, ...)` /// - `Input = (A, B, ...)`: Pass `(&TaskHandle<A>, &TaskHandle<B>, ...)`
/// ///
/// The order of dependencies in the tuple must match the order in `Input`. /// The order of dependencies in the tuple must match the order in `Input`.
/// ///
/// # Panics
///
/// This function will cause a panic in [`DagRunner::run`] if called with a [`TaskHandle`] from a `DagRunner` instance
/// other than the one which created this TaskBuilder.
///
/// # Examples /// # Examples
/// ///
/// ```no_run /// ```no_run
/// # use dagx::{task, DagRunner, Task}; /// # use dagx::{task, DagRunner, Task};
/// #
/// // Tuple struct /// // Tuple struct
/// struct Value(i32); /// struct Value(i32);
/// ///
@@ -111,7 +116,7 @@ impl<'a, Tk: Task, Deps> TaskBuilder<'a, Tk, Deps> {
/// } /// }
/// ///
/// # async { /// # async {
/// let dag = DagRunner::new(); /// let mut dag = DagRunner::new();
/// ///
/// let x = dag.add_task(Value(2)); /// let x = dag.add_task(Value(2));
/// let y = dag.add_task(Value(3)); /// let y = dag.add_task(Value(3));
@@ -122,41 +127,139 @@ impl<'a, Tk: Task, Deps> TaskBuilder<'a, Tk, Deps> {
/// // Multiple dependencies: tuple form /// // Multiple dependencies: tuple form
/// let sum = dag.add_task(Add).depends_on((&x, &y)); /// let sum = dag.add_task(Add).depends_on((&x, &y));
/// ///
/// dag.run(|fut| { tokio::spawn(fut); }).await.unwrap(); ///let mut output = dag.run(|fut| async move { tokio::spawn(fut).await.unwrap() }).await.unwrap();
/// # }; /// # };
/// ``` /// ```
#[allow(private_bounds)]
pub fn depends_on<D>(self, deps: D) -> TaskHandle<Tk::Output> pub fn depends_on<D>(self, deps: D) -> TaskHandle<Tk::Output>
where where
D: DepsTuple<Tk::Input>, D: DepsTuple<Input>,
{ {
// Register dependencies in the DAG // Register dependencies in the DAG
let dep_ids = deps.to_node_ids(); let dep_ids = deps.to_node_ids();
let mut edges = self.dag.edges.lock(); #[cfg(feature = "tracing")]
let mut dependents = self.dag.dependents.lock(); debug!(
task_id = self.id.0,
dependency_ids = ?dep_ids.iter().map(|id| id.0).collect::<Vec<_>>(),
dependency_count = dep_ids.len(),
"wiring task dependencies"
);
for &dep_id in &dep_ids { for &dep_id in &dep_ids {
// Add edge from this node to dependency // Add edge from this node to dependency
if let Some(node_edges) = edges.get_mut(&self.id) { if let Some(node_edges) = self.dag.edges.get_mut(&self.id) {
node_edges.push(dep_id); node_edges.push(dep_id);
} }
// Add this node as dependent of the dependency // Add this node as dependent of the dependency
if let Some(node_dependents) = dependents.get_mut(&dep_id) { if let Some(node_dependents) = self.dag.dependents.get_mut(&dep_id) {
node_dependents.push(self.id); node_dependents.push(self.id);
} }
} }
TaskHandle { TaskHandle {
id: self.id, id: self.id,
_phantom: std::marker::PhantomData, _phantom: PhantomData,
} }
} }
} }
/// Opaque node identifier
#[derive(Debug, Clone, Copy, Hash, PartialEq, Eq)]
pub struct NodeId(pub u32);
/// Opaque, typed token for a node's output.
///
/// A `TaskHandle<T>` provides compile-time type-safe access to a task's output.
/// You can:
/// 1. Pass it to [`crate::TaskBuilder::depends_on`] to wire up dependencies
/// 2. Use it with [`crate::DagOutput::get`] to retrieve the output after [`crate::DagRunner::run`]
///
/// # Examples
///
/// ```no_run
/// # use dagx::{task, DagRunner, Task};
/// #
/// # struct LoadValue { value: i32 }
/// # impl LoadValue { pub fn new(v: i32) -> Self { Self { value: v } } }
/// # #[task]
/// # impl LoadValue {
/// # async fn run(&mut self) -> i32 { self.value }
/// # }
/// # async {
/// let mut dag = DagRunner::new();
/// let node = dag.add_task(LoadValue::new(42));
///
///let mut output = dag.run(|fut| async move { tokio::spawn(fut).await.unwrap() }).await.unwrap();
///
/// assert_eq!(output.get(node), 42);
/// # };
/// ```
pub struct TaskHandle<T> {
pub(crate) id: NodeId,
pub(crate) _phantom: PhantomData<fn() -> T>,
}
/// Takes a task and converts it to either a TaskBuilder or a TaskHandle,
/// depending on whether it has inputs or not.
///
/// This is useful to enforce at compile time that a TaskBuilder is never created for task with unit input,
/// and that it can be used directly as a dependency without converting it manually to a TaskHandle.
pub trait TaskWire<Input>: Task<Input> + Sync + 'static
where
Input: Send + Sync + 'static,
{
type Retval<'dag>;
fn new_from_dag<'dag>(id: NodeId, dag: &'dag mut DagRunner) -> Self::Retval<'dag>;
}
impl<Tk> TaskWire<()> for Tk
where
Tk: Task<()> + Sync + 'static,
{
type Retval<'dag> = TaskHandle<Tk::Output>;
fn new_from_dag(id: NodeId, _dag: &mut DagRunner) -> Self::Retval<'static> {
Self::Retval {
id,
_phantom: PhantomData,
}
}
}
/// Macro to implement TaskWire for different tuple sizes.
macro_rules! impl_wire_tuple {
($($T:ident),+) => {
impl<Tk, $($T: Send + Sync + 'static),+> TaskWire<($($T,)+)> for Tk
where
Tk: Task<($($T,)+)> + Sync + 'static
{
type Retval<'dag> = TaskBuilder<'dag, ($($T,)+), Tk>;
fn new_from_dag<'dag>(id: NodeId, dag: &'dag mut DagRunner) -> Self::Retval<'dag> {
Self::Retval {
id,
dag,
_phantom: PhantomData,
}
}
}
};
}
impl_wire_tuple!(T1);
impl_wire_tuple!(T1, T2);
impl_wire_tuple!(T1, T2, T3);
impl_wire_tuple!(T1, T2, T3, T4);
impl_wire_tuple!(T1, T2, T3, T4, T5);
impl_wire_tuple!(T1, T2, T3, T4, T5, T6);
impl_wire_tuple!(T1, T2, T3, T4, T5, T6, T7);
impl_wire_tuple!(T1, T2, T3, T4, T5, T6, T7, T8);
#[cfg(test)] #[cfg(test)]
mod tests; mod tests;
// Note: We cannot safely implement AsRef<TaskHandle<Tk::Output>> for TaskBuilder<Tk, Deps> #[cfg(test)]
// because Handle is Copy and returning a reference to a temporary would be unsound. mod coverage_tests;
// Instead, users should use From/Into conversions or call .into() explicitly.
+64
View File
@@ -0,0 +1,64 @@
//! Coverage tests for types module - specifically targeting the TaskBuilder to TaskHandle conversion
//!
//! These tests ensure we hit line 71 in the From<TaskBuilder> implementation
use crate::{task, DagRunner, TaskHandle};
#[tokio::test]
async fn test_unit_struct_with_state() {
// Even structs with fields can have unit input type
struct StatefulTask {
value: i32,
}
impl StatefulTask {
fn new(value: i32) -> Self {
Self { value }
}
}
#[task]
impl StatefulTask {
async fn run(&self) -> i32 {
self.value * 2
}
}
let mut dag = DagRunner::new();
// Create task with initial state
let builder = dag.add_task(StatefulTask::new(21));
// Explicit conversion - triggers line 71
let handle = builder;
let mut output = dag
.run(|fut| async move { tokio::spawn(fut).await.unwrap() })
.await
.unwrap();
assert_eq!(output.get(handle), 42);
}
#[test]
fn test_taskbuilder_conversion_without_running() {
// Test that conversion works even without running the DAG
struct TestTask;
#[task]
impl TestTask {
async fn run(&self) -> String {
"test".to_string()
}
}
let mut dag = DagRunner::new();
let builder = dag.add_task(TestTask);
let builder_id = builder.id;
// Convert without running the DAG
let handle: TaskHandle<String> = builder;
// Verify the ID was transferred correctly
assert_eq!(handle.id, builder_id);
}
+32 -36
View File
@@ -1,9 +1,25 @@
//! Unit tests for builder module //! Unit tests for builder module
use crate::builder::IsUnitType;
use crate::runner::DagRunner; use crate::runner::DagRunner;
use crate::task::Task;
use crate::types::TaskHandle; // Initialize tracing subscriber for tests (idempotent)
#[cfg(feature = "tracing")]
fn init_tracing() {
use std::sync::Once;
static INIT: Once = Once::new();
INIT.call_once(|| {
tracing_subscriber::fmt()
.with_test_writer()
.with_max_level(tracing::Level::TRACE)
.try_init()
.ok();
});
}
#[cfg(not(feature = "tracing"))]
fn init_tracing() {
// No-op when tracing is disabled
}
struct TestTask { struct TestTask {
value: i32, value: i32,
@@ -11,6 +27,7 @@ struct TestTask {
#[crate::task] #[crate::task]
impl TestTask { impl TestTask {
#[cfg(not(tarpaulin_include))]
async fn run(&self) -> i32 { async fn run(&self) -> i32 {
self.value self.value
} }
@@ -20,6 +37,7 @@ struct TestTaskWithInput;
#[crate::task] #[crate::task]
impl TestTaskWithInput { impl TestTaskWithInput {
#[cfg(not(tarpaulin_include))]
async fn run(input: &i32) -> i32 { async fn run(input: &i32) -> i32 {
input * 2 input * 2
} }
@@ -28,56 +46,34 @@ impl TestTaskWithInput {
#[test] #[test]
fn test_task_builder_from_conversion() { fn test_task_builder_from_conversion() {
// Test line 165 in builder.rs - the From implementation // Test line 165 in builder.rs - the From implementation
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let builder = dag.add_task(TestTask { value: 42 }); let builder = dag.add_task(TestTask { value: 42 });
// Convert builder to TaskHandle // Convert builder to TaskHandle
let handle: TaskHandle<i32> = builder.into(); let handle = builder;
// Check that the ID is preserved // Check that the ID is preserved
assert_eq!(handle.id.0, 0); // First task should have ID 0 assert_eq!(handle.id.0, 0); // First task should have ID 0
} }
#[test]
fn test_is_unit_type_trait() {
// Test that IsUnitType is correctly implemented for ()
fn assert_is_unit_type<T: IsUnitType>() {}
// This should compile - () implements IsUnitType
assert_is_unit_type::<()>();
// We can't test the negative case (non-unit types don't implement it)
// because that would be a compile error
}
#[test]
fn test_task_builder_depends_on_returns_handle() {
let dag = DagRunner::new();
let source = dag.add_task(TestTask { value: 10 });
let dependent = dag.add_task(TestTaskWithInput).depends_on(&source);
// depends_on should return a TaskHandle
let _handle: TaskHandle<i32> = dependent;
}
#[test] #[test]
fn test_task_builder_chain() { fn test_task_builder_chain() {
let dag = DagRunner::new(); let mut dag = DagRunner::new();
// Test chaining multiple tasks // Test chaining multiple tasks
let t1 = dag.add_task(TestTask { value: 1 }); let t1 = dag.add_task(TestTask { value: 1 });
let t2 = dag.add_task(TestTaskWithInput).depends_on(&t1); let t2 = dag.add_task(TestTaskWithInput).depends_on(&t1);
let t3 = dag.add_task(TestTaskWithInput).depends_on(t2); let t3 = dag.add_task(TestTaskWithInput).depends_on(&t2);
// All should return TaskHandles // All should return TaskHandles
let _: TaskHandle<i32> = t1.into(); let _ = t1;
let _: TaskHandle<i32> = t2; let _ = t2;
let _: TaskHandle<i32> = t3; let _ = t3;
} }
#[test] #[test]
fn test_multiple_dependencies() { fn test_multiple_dependencies() {
init_tracing();
struct AddTask; struct AddTask;
#[crate::task] #[crate::task]
@@ -87,7 +83,7 @@ fn test_multiple_dependencies() {
} }
} }
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let a = dag.add_task(TestTask { value: 10 }); let a = dag.add_task(TestTask { value: 10 });
let b = dag.add_task(TestTask { value: 20 }); let b = dag.add_task(TestTask { value: 20 });
@@ -96,12 +92,12 @@ fn test_multiple_dependencies() {
let sum = dag.add_task(AddTask).depends_on((&a, &b)); let sum = dag.add_task(AddTask).depends_on((&a, &b));
// Should return a TaskHandle // Should return a TaskHandle
let _: TaskHandle<i32> = sum; let _ = sum;
} }
#[test] #[test]
fn test_task_builder_stores_correct_id() { fn test_task_builder_stores_correct_id() {
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let builder1 = dag.add_task(TestTask { value: 1 }); let builder1 = dag.add_task(TestTask { value: 1 });
let builder2 = dag.add_task(TestTask { value: 2 }); let builder2 = dag.add_task(TestTask { value: 2 });
+22 -75
View File
@@ -1,114 +1,61 @@
//! Dependency tuple trait and implementations. //! Dependency tuple trait and implementations.
//! //!
//! This module defines the DepsTuple trait that allows specifying dependencies in //! This module defines the DepsTuple trait that allows specifying dependencies in
//! various forms (handles, builders, tuples). It uses macro-generated implementations //! various combinations of handles for tuples of different sizes.
//! for different tuple sizes.
//!
//! This trait is internal and not meant for external implementation. Users should only
//! use the provided implementations.
use crate::builder::TaskBuilder; use crate::builder::{NodeId, TaskHandle};
use crate::task::Task;
use crate::types::{NodeId, TaskHandle};
pub(crate) trait DepsTuple<Input> { pub(crate) trait DepsTuple<Input> {
fn to_node_ids(&self) -> Vec<NodeId>; fn to_node_ids(self) -> Vec<NodeId>;
} }
// Implementation for unit (no dependencies) // Implementation for unit (no dependencies)
impl DepsTuple<()> for () { impl DepsTuple<()> for () {
fn to_node_ids(&self) -> Vec<NodeId> { fn to_node_ids(self) -> Vec<NodeId> {
Vec::new() Vec::new()
} }
} }
// Implementation for single dependency (all forms) // Implementation for single dependency (all forms)
impl<T> DepsTuple<T> for &TaskHandle<T> { impl<O> DepsTuple<(O,)> for &TaskHandle<O> {
fn to_node_ids(&self) -> Vec<NodeId> { fn to_node_ids(self) -> Vec<NodeId> {
vec![self.id] vec![self.id]
} }
} }
impl<T> DepsTuple<T> for TaskHandle<T> {
fn to_node_ids(&self) -> Vec<NodeId> {
vec![self.id]
}
}
impl<T> DepsTuple<T> for (&TaskHandle<T>,) {
fn to_node_ids(&self) -> Vec<NodeId> {
vec![self.0.id]
}
}
impl<T> DepsTuple<T> for (TaskHandle<T>,) {
fn to_node_ids(&self) -> Vec<NodeId> {
vec![self.0.id]
}
}
// Support for &TaskBuilder
impl<'a, Tk: Task, Deps> DepsTuple<Tk::Output> for &TaskBuilder<'a, Tk, Deps> {
fn to_node_ids(&self) -> Vec<NodeId> {
vec![self.id]
}
}
impl<'a, Tk: Task, Deps> DepsTuple<Tk::Output> for (&TaskBuilder<'a, Tk, Deps>,) {
fn to_node_ids(&self) -> Vec<NodeId> {
vec![self.0.id]
}
}
/// Macro to implement DepsTuple for different tuple sizes. /// Macro to implement DepsTuple for different tuple sizes.
/// ///
/// This macro exists because Rust lacks variadic generics - we need separate implementations
/// for each tuple size.
///
/// For each tuple size, generates two implementations:
/// - `(&TaskHandle<A>, &TaskHandle<B>, ...)` → `DepsTuple<(A, B, ...)>`
/// - `(&TaskBuilder<TkA, _>, &TaskBuilder<TkB, _>, ...)` → `DepsTuple<(TkA::Output, TkB::Output, ...)>`
///
/// This allows flexible dependency specification: /// This allows flexible dependency specification:
/// ```ignore /// ```ignore
/// // Using handles /// // Using handles
/// task.depends_on((&handle_a, &handle_b)) /// task.depends_on((&handle_a, &handle_b))
/// ///
/// // Using builders directly /// // Using builders directly
/// task.depends_on((&dag.add_task(TaskA), &dag.add_task(TaskB))) /// task.depends_on((dag.add_task(TaskA), dag.add_task(TaskB)))
/// ``` /// ```
macro_rules! impl_deps_tuple { macro_rules! impl_deps_tuple {
($($T:ident : $Tk:ident : $D:ident),+) => { ($($O:ident),+) => {
// For &TaskHandle tuples impl<$($O),+> DepsTuple<($($O,)+)> for ($(&TaskHandle<$O>,)+)
impl<$($T),+> DepsTuple<($($T,)+)> for ($(&TaskHandle<$T>,)+) { {
#[allow(non_snake_case)] #[allow(non_snake_case)]
fn to_node_ids(&self) -> Vec<NodeId> { fn to_node_ids(self) -> Vec<NodeId> {
let ($($T,)+) = self; let ($($O,)+) = self;
vec![$($T.id,)+] vec![$($O.id,)+]
}
}
// For &TaskBuilder tuples
impl<'a, $($Tk: Task, $D),+> DepsTuple<($($Tk::Output,)+)> for ($(&TaskBuilder<'a, $Tk, $D>,)+) {
#[allow(non_snake_case)]
fn to_node_ids(&self) -> Vec<NodeId> {
let ($($T,)+) = self;
vec![$($T.id,)+]
} }
} }
}; };
} }
// Generate DepsTuple implementations for tuples of size 2-8. // Generate DepsTuple implementations for tuples of size 1-8.
// Each invocation generates 2 implementations (TaskHandle refs and TaskBuilder refs).
// Supporting up to 8 elements covers the vast majority of use cases. // Supporting up to 8 elements covers the vast majority of use cases.
impl_deps_tuple!(A:TkA:DA, B:TkB:DB); impl_deps_tuple!(O1);
impl_deps_tuple!(A:TkA:DA, B:TkB:DB, C:TkC:DC); impl_deps_tuple!(O1, O2);
impl_deps_tuple!(A:TkA:DA, B:TkB:DB, C:TkC:DC, D:TkD:DD); impl_deps_tuple!(O1, O2, O3);
impl_deps_tuple!(A:TkA:DA, B:TkB:DB, C:TkC:DC, D:TkD:DD, E:TkE:DE); impl_deps_tuple!(O1, O2, O3, O4);
impl_deps_tuple!(A:TkA:DA, B:TkB:DB, C:TkC:DC, D:TkD:DD, E:TkE:DE, F:TkF:DF); impl_deps_tuple!(O1, O2, O3, O4, O5);
impl_deps_tuple!(A:TkA:DA, B:TkB:DB, C:TkC:DC, D:TkD:DD, E:TkE:DE, F:TkF:DF, G:TkG:DG); impl_deps_tuple!(O1, O2, O3, O4, O5, O6);
impl_deps_tuple!(A:TkA:DA, B:TkB:DB, C:TkC:DC, D:TkD:DD, E:TkE:DE, F:TkF:DF, G:TkG:DG, H:TkH:DH); impl_deps_tuple!(O1, O2, O3, O4, O5, O6, O7);
impl_deps_tuple!(O1, O2, O3, O4, O5, O6, O7, O8);
#[cfg(test)] #[cfg(test)]
mod tests; mod tests;
+11 -67
View File
@@ -1,9 +1,8 @@
//! Unit tests for deps module //! Unit tests for deps module
use crate::builder::{NodeId, TaskHandle};
use crate::deps::DepsTuple; use crate::deps::DepsTuple;
use crate::runner::DagRunner; use crate::runner::DagRunner;
use crate::task::Task;
use crate::types::{NodeId, TaskHandle};
// Test task for dependency testing // Test task for dependency testing
struct TestTask { struct TestTask {
@@ -12,21 +11,12 @@ struct TestTask {
#[crate::task] #[crate::task]
impl TestTask { impl TestTask {
#[cfg(not(tarpaulin_include))]
async fn run(&self) -> i32 { async fn run(&self) -> i32 {
self.value self.value
} }
} }
struct TestTaskWithInput;
#[crate::task]
impl TestTaskWithInput {
#[allow(dead_code)]
async fn run(input: &i32) -> i32 {
input * 2
}
}
#[test] #[test]
fn test_deps_tuple_unit() { fn test_deps_tuple_unit() {
// Test lines 20-22 in deps.rs - unit type implementation // Test lines 20-22 in deps.rs - unit type implementation
@@ -47,19 +37,6 @@ fn test_deps_tuple_single_handle_ref() {
assert_eq!(node_ids[0], NodeId(42)); assert_eq!(node_ids[0], NodeId(42));
} }
#[test]
fn test_deps_tuple_single_handle_owned() {
// Test lines 32-36 in deps.rs - TaskHandle<T>
let handle: TaskHandle<String> = TaskHandle {
id: NodeId(99),
_phantom: std::marker::PhantomData,
};
let handle_copy = handle; // Copy for use
let node_ids = handle_copy.to_node_ids();
assert_eq!(node_ids.len(), 1);
assert_eq!(node_ids[0], NodeId(99));
}
#[test] #[test]
fn test_deps_tuple_single_handle_tuple_ref() { fn test_deps_tuple_single_handle_tuple_ref() {
// Test lines 38-42 in deps.rs - (&TaskHandle<T>,) // Test lines 38-42 in deps.rs - (&TaskHandle<T>,)
@@ -72,41 +49,30 @@ fn test_deps_tuple_single_handle_tuple_ref() {
assert_eq!(node_ids[0], NodeId(7)); assert_eq!(node_ids[0], NodeId(7));
} }
#[test]
fn test_deps_tuple_single_handle_tuple_owned() {
// Test lines 44-48 in deps.rs - (TaskHandle<T>,)
let handle: TaskHandle<Vec<u8>> = TaskHandle {
id: NodeId(13),
_phantom: std::marker::PhantomData,
};
let handle_copy = handle; // Copy for use
let node_ids = (handle_copy,).to_node_ids();
assert_eq!(node_ids.len(), 1);
assert_eq!(node_ids[0], NodeId(13));
}
#[test] #[test]
fn test_deps_tuple_task_builder_ref() { fn test_deps_tuple_task_builder_ref() {
// Test lines 51-55 in deps.rs - &TaskBuilder // Test lines 51-55 in deps.rs - &TaskBuilder
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let builder = dag.add_task(TestTask { value: 42 }); let builder = dag.add_task(TestTask { value: 42 });
let builder_id = builder.id;
// We need to test the DepsTuple implementation directly // We need to test the DepsTuple implementation directly
// The builder has an id field we can check // The builder has an id field we can check
let node_ids = (&builder).to_node_ids(); let node_ids = builder.to_node_ids();
assert_eq!(node_ids.len(), 1); assert_eq!(node_ids.len(), 1);
assert_eq!(node_ids[0], builder.id); assert_eq!(node_ids[0], builder_id);
} }
#[test] #[test]
fn test_deps_tuple_task_builder_tuple() { fn test_deps_tuple_task_builder_tuple() {
// Test lines 57-61 in deps.rs - (&TaskBuilder,) // Test lines 57-61 in deps.rs - (&TaskBuilder,)
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let builder = dag.add_task(TestTask { value: 100 }); let builder = dag.add_task(TestTask { value: 100 });
let builder_id = builder.id;
let node_ids = (&builder,).to_node_ids(); let node_ids = (&builder,).to_node_ids();
assert_eq!(node_ids.len(), 1); assert_eq!(node_ids.len(), 1);
assert_eq!(node_ids[0], builder.id); assert_eq!(node_ids[0], builder_id);
} }
#[test] #[test]
@@ -139,28 +105,6 @@ fn test_deps_tuple_multiple_handles() {
assert_eq!(node_ids[2], NodeId(3)); assert_eq!(node_ids[2], NodeId(3));
} }
#[test]
fn test_deps_tuple_multiple_builders() {
// Test macro-generated implementations for multiple builders
let dag = DagRunner::new();
let builder1 = dag.add_task(TestTask { value: 10 });
let builder2 = dag.add_task(TestTask { value: 20 });
let builder3 = dag.add_task(TestTask { value: 30 });
// Test 2-tuple of builders
let node_ids = (&builder1, &builder2).to_node_ids();
assert_eq!(node_ids.len(), 2);
assert_eq!(node_ids[0], builder1.id);
assert_eq!(node_ids[1], builder2.id);
// Test 3-tuple of builders
let node_ids = (&builder1, &builder2, &builder3).to_node_ids();
assert_eq!(node_ids.len(), 3);
assert_eq!(node_ids[0], builder1.id);
assert_eq!(node_ids[1], builder2.id);
assert_eq!(node_ids[2], builder3.id);
}
#[test] #[test]
fn test_deps_tuple_large_tuples() { fn test_deps_tuple_large_tuples() {
// Test that larger tuples work (up to 16 elements) // Test that larger tuples work (up to 16 elements)
@@ -176,7 +120,7 @@ fn test_deps_tuple_large_tuples() {
let node_ids = (&handles[0], &handles[1], &handles[2], &handles[3]).to_node_ids(); let node_ids = (&handles[0], &handles[1], &handles[2], &handles[3]).to_node_ids();
assert_eq!(node_ids.len(), 4); assert_eq!(node_ids.len(), 4);
for (i, node_id) in node_ids.iter().enumerate().take(4) { for (i, node_id) in node_ids.iter().enumerate().take(4) {
assert_eq!(*node_id, NodeId(i)); assert_eq!(*node_id, NodeId(i as u32));
} }
// Test 8-tuple // Test 8-tuple
@@ -193,6 +137,6 @@ fn test_deps_tuple_large_tuples() {
.to_node_ids(); .to_node_ids();
assert_eq!(node_ids.len(), 8); assert_eq!(node_ids.len(), 8);
for (i, node_id) in node_ids.iter().enumerate().take(8) { for (i, node_id) in node_ids.iter().enumerate().take(8) {
assert_eq!(*node_id, NodeId(i)); assert_eq!(*node_id, NodeId(i as u32));
} }
} }
+3 -58
View File
@@ -4,62 +4,15 @@
/// Errors that can occur during DAG construction and execution /// Errors that can occur during DAG construction and execution
#[derive(Debug, Clone, PartialEq)] #[derive(Debug, Clone, PartialEq)]
#[non_exhaustive]
pub enum DagError { pub enum DagError {
/// A cycle was detected in the DAG
CycleDetected {
nodes: Vec<usize>,
description: String,
},
/// Invalid dependency: task does not exist
InvalidDependency { task_id: usize },
/// Type mismatch in task dependencies
TypeMismatch {
expected: &'static str,
found: &'static str,
},
/// Task panicked during execution /// Task panicked during execution
TaskPanicked { TaskPanicked { task_id: u32, panic_message: String },
task_id: usize,
panic_message: String,
},
/// Result not found for task
ResultNotFound { task_id: usize },
} }
impl std::fmt::Display for DagError { impl std::fmt::Display for DagError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self { match self {
DagError::CycleDetected { nodes, description } => {
write!(
f,
"Cycle detected in DAG: {}\n\
Nodes involved: {:?}\n\
\n\
A DAG cannot have cycles. Review your dependency graph to identify \
and remove circular dependencies.",
description, nodes
)
}
DagError::InvalidDependency { task_id } => {
write!(
f,
"Invalid dependency: task #{} does not exist.\n\
\n\
Ensure all task handles reference tasks that have been added to the DAG.",
task_id
)
}
DagError::TypeMismatch { expected, found } => {
write!(
f,
"Type mismatch in task dependencies.\n\
Expected: {}\n\
Found: {}\n\
\n\
Verify that dependency types match the task's Input type exactly.",
expected, found
)
}
DagError::TaskPanicked { DagError::TaskPanicked {
task_id, task_id,
panic_message, panic_message,
@@ -72,15 +25,6 @@ impl std::fmt::Display for DagError {
task_id, panic_message task_id, panic_message
) )
} }
DagError::ResultNotFound { task_id } => {
write!(
f,
"Result not found for task #{}.\n\
\n\
Call dag.run() before accessing results with get().",
task_id
)
}
} }
} }
} }
@@ -88,6 +32,7 @@ impl std::fmt::Display for DagError {
impl std::error::Error for DagError {} impl std::error::Error for DagError {}
/// Result type for DAG operations /// Result type for DAG operations
#[cfg(not(tarpaulin_include))]
pub type DagResult<T> = Result<T, DagError>; pub type DagResult<T> = Result<T, DagError>;
#[cfg(test)] #[cfg(test)]
+5 -102
View File
@@ -1,33 +1,6 @@
//! Unit tests for error module //! Unit tests for error module
use crate::error::{DagError, DagResult}; use crate::error::DagError;
#[test]
fn test_dag_error_display_invalid_dependency() {
// Test lines 47-54 in error.rs
let err = DagError::InvalidDependency { task_id: 42 };
let display = format!("{}", err);
assert!(display.contains("Invalid dependency"));
assert!(display.contains("task #42"));
assert!(display.contains("does not exist"));
assert!(display.contains("Ensure all task handles"));
}
#[test]
fn test_dag_error_display_type_mismatch() {
// Test lines 56-65 in error.rs
let err = DagError::TypeMismatch {
expected: "i32",
found: "String",
};
let display = format!("{}", err);
assert!(display.contains("Type mismatch"));
assert!(display.contains("Expected: i32"));
assert!(display.contains("Found: String"));
assert!(display.contains("Verify that dependency types"));
}
#[test] #[test]
fn test_dag_error_display_task_panicked() { fn test_dag_error_display_task_panicked() {
@@ -44,86 +17,16 @@ fn test_dag_error_display_task_panicked() {
assert!(display.contains("entire DAG execution is aborted")); assert!(display.contains("entire DAG execution is aborted"));
} }
#[test]
fn test_dag_error_display_cycle_detected() {
let err = DagError::CycleDetected {
nodes: vec![1, 2, 3, 1],
description: "A->B->C->A".to_string(),
};
let display = format!("{}", err);
assert!(display.contains("Cycle detected"));
assert!(display.contains("A->B->C->A"));
assert!(display.contains("[1, 2, 3, 1]"));
assert!(display.contains("cannot have cycles"));
}
#[test]
fn test_dag_error_display_result_not_found() {
let err = DagError::ResultNotFound { task_id: 7 };
let display = format!("{}", err);
assert!(display.contains("Result not found"));
assert!(display.contains("task #7"));
assert!(display.contains("Call dag.run()"));
}
#[test] #[test]
fn test_dag_error_std_error_impl() { fn test_dag_error_std_error_impl() {
// Test that DagError implements std::error::Error // Test that DagError implements std::error::Error
let err = DagError::InvalidDependency { task_id: 1 }; let err = DagError::TaskPanicked {
task_id: 1,
panic_message: "test panic".to_string(),
};
let err_ref: &dyn std::error::Error = &err; let err_ref: &dyn std::error::Error = &err;
// Should be able to call Error trait methods // Should be able to call Error trait methods
let _ = err_ref.to_string(); let _ = err_ref.to_string();
assert!(err_ref.source().is_none()); // DagError doesn't chain errors assert!(err_ref.source().is_none()); // DagError doesn't chain errors
} }
#[test]
fn test_dag_error_equality() {
let err1 = DagError::InvalidDependency { task_id: 42 };
let err2 = DagError::InvalidDependency { task_id: 42 };
let err3 = DagError::InvalidDependency { task_id: 99 };
assert_eq!(err1, err2);
assert_ne!(err1, err3);
}
#[test]
fn test_dag_error_clone() {
let err = DagError::TaskPanicked {
task_id: 10,
panic_message: "test panic".to_string(),
};
let cloned = err.clone();
assert_eq!(err, cloned);
}
#[test]
fn test_dag_error_debug() {
let err = DagError::TypeMismatch {
expected: "i32",
found: "bool",
};
let debug_str = format!("{:?}", err);
assert!(debug_str.contains("TypeMismatch"));
assert!(debug_str.contains("expected"));
assert!(debug_str.contains("found"));
}
#[test]
fn test_dag_result_type_alias() {
// Test that DagResult works as expected
fn returns_dag_result() -> DagResult<i32> {
Ok(42)
}
fn returns_dag_error() -> DagResult<String> {
Err(DagError::ResultNotFound { task_id: 1 })
}
assert!(returns_dag_result().is_ok());
assert!(returns_dag_error().is_err());
}
-291
View File
@@ -1,291 +0,0 @@
//! Input extraction for task dependencies via oneshot channels.
//!
//! This module handles extracting typed inputs from task dependencies via oneshot channels.
//!
//! # How it works
//!
//! When a task executes, it needs to extract its inputs from type-erased oneshot receivers.
//! Outputs are wrapped in Arc for efficient fanout (cheap Arc clones instead of data clones).
//! This module provides the ExtractInput trait with implementations for:
//! - `()` for tasks with no dependencies
//! - Common types (`i32`, `String`, etc.) for tasks with a single dependency
//! - `(A, B, ...)` tuples for tasks with multiple dependencies (awaited concurrently)
//!
//! # Why macros?
//!
//! Rust lacks variadic generics, so we use macros to generate implementations for:
//! - Common primitive/standard types (one impl per type)
//! - Tuples of different sizes (one impl per size, up to 8 elements)
//!
//! This is standard practice in Rust for working with tuples of different arities.
use std::any::Any;
use std::sync::Arc;
use futures::channel::oneshot;
/// Helper trait for async input extraction from oneshot channels.
///
/// Type erasure occurs only at the ExecutableNode trait boundary - by the time
/// we're in ExtractInput, we know the concrete type and can safely downcast.
pub(crate) trait ExtractInput: Sized + Clone {
fn extract_from_channels(
receivers: Vec<Box<dyn Any + Send>>,
) -> impl std::future::Future<Output = Result<Self, String>> + Send;
}
// Unit type - no dependencies
impl ExtractInput for () {
async fn extract_from_channels(_receivers: Vec<Box<dyn Any + Send>>) -> Result<Self, String> {
Ok(())
}
}
// Implementations for common primitive and standard library types.
// These allow tasks to receive these types as single inputs without wrapping in tuples.
// Receives Arc<T> from channel, clones the inner value for task consumption.
// For custom types, implement ExtractInput following the same pattern.
macro_rules! impl_extract_single {
($($t:ty),+) => {
$(
impl ExtractInput for $t {
async fn extract_from_channels(mut receivers: Vec<Box<dyn Any + Send>>) -> Result<Self, String> {
if receivers.len() != 1 {
return Err(format!("Expected 1 dependency, got {}", receivers.len()));
}
let rx = *receivers.pop()
.unwrap()
.downcast::<oneshot::Receiver<Arc<$t>>>()
.map_err(|_| format!("Type mismatch: expected Arc<{}>", std::any::type_name::<$t>()))?;
let arc_value = rx.await.map_err(|_| "Channel closed before receiving value".to_string())?;
Ok((*arc_value).clone())
}
}
)+
};
}
impl_extract_single!(
i8, i16, i32, i64, i128, isize, u8, u16, u32, u64, u128, usize, f32, f64, bool, char, String
);
// Implement ExtractInput for Result types to support error handling patterns
impl<T, E> ExtractInput for Result<T, E>
where
T: 'static + Clone + Send + Sync,
E: 'static + Clone + Send + Sync,
{
async fn extract_from_channels(
mut receivers: Vec<Box<dyn Any + Send>>,
) -> Result<Self, String> {
if receivers.len() != 1 {
return Err(format!("Expected 1 dependency, got {}", receivers.len()));
}
let rx = *receivers
.pop()
.unwrap()
.downcast::<oneshot::Receiver<Arc<Result<T, E>>>>()
.map_err(|_| {
format!(
"Type mismatch: expected Arc<Result<{}, {}>>",
std::any::type_name::<T>(),
std::any::type_name::<E>()
)
})?;
let arc_value = rx
.await
.map_err(|_| "Channel closed before receiving value".to_string())?;
Ok((*arc_value).clone())
}
}
// Implement ExtractInput for Option types
impl<T> ExtractInput for Option<T>
where
T: 'static + Clone + Send + Sync,
{
async fn extract_from_channels(
mut receivers: Vec<Box<dyn Any + Send>>,
) -> Result<Self, String> {
if receivers.len() != 1 {
return Err(format!("Expected 1 dependency, got {}", receivers.len()));
}
let rx = *receivers
.pop()
.unwrap()
.downcast::<oneshot::Receiver<Arc<Option<T>>>>()
.map_err(|_| {
format!(
"Type mismatch: expected Arc<Option<{}>>",
std::any::type_name::<T>()
)
})?;
let arc_value = rx
.await
.map_err(|_| "Channel closed before receiving value".to_string())?;
Ok((*arc_value).clone())
}
}
// Implement ExtractInput for Vec types
impl<T> ExtractInput for Vec<T>
where
T: 'static + Clone + Send + Sync,
{
async fn extract_from_channels(
mut receivers: Vec<Box<dyn Any + Send>>,
) -> Result<Self, String> {
if receivers.len() != 1 {
return Err(format!("Expected 1 dependency, got {}", receivers.len()));
}
let rx = *receivers
.pop()
.unwrap()
.downcast::<oneshot::Receiver<Arc<Vec<T>>>>()
.map_err(|_| {
format!(
"Type mismatch: expected Arc<Vec<{}>>",
std::any::type_name::<T>()
)
})?;
let arc_value = rx
.await
.map_err(|_| "Channel closed before receiving value".to_string())?;
Ok((*arc_value).clone())
}
}
// Implement ExtractInput for HashMap types
impl<K, V> ExtractInput for std::collections::HashMap<K, V>
where
K: 'static + Clone + Send + Sync + Eq + std::hash::Hash,
V: 'static + Clone + Send + Sync,
{
async fn extract_from_channels(
mut receivers: Vec<Box<dyn Any + Send>>,
) -> Result<Self, String> {
if receivers.len() != 1 {
return Err(format!("Expected 1 dependency, got {}", receivers.len()));
}
let rx = *receivers
.pop()
.unwrap()
.downcast::<oneshot::Receiver<Arc<std::collections::HashMap<K, V>>>>()
.map_err(|_| {
format!(
"Type mismatch: expected Arc<HashMap<{}, {}>>",
std::any::type_name::<K>(),
std::any::type_name::<V>()
)
})?;
let arc_value = rx
.await
.map_err(|_| "Channel closed before receiving value".to_string())?;
Ok((*arc_value).clone())
}
}
// Implement ExtractInput for Arc types - SPECIAL CASE: just clone the Arc, don't unwrap!
// This allows zero-copy data sharing for users who wrap their data in Arc.
impl<T> ExtractInput for Arc<T>
where
T: 'static + Clone + Send + Sync,
{
async fn extract_from_channels(
mut receivers: Vec<Box<dyn Any + Send>>,
) -> Result<Self, String> {
if receivers.len() != 1 {
return Err(format!("Expected 1 dependency, got {}", receivers.len()));
}
let rx = *receivers
.pop()
.unwrap()
.downcast::<oneshot::Receiver<Arc<Arc<T>>>>()
.map_err(|_| {
format!(
"Type mismatch: expected Arc<Arc<{}>>",
std::any::type_name::<T>()
)
})?;
let arc_arc_value = rx
.await
.map_err(|_| "Channel closed before receiving value".to_string())?;
// Return the inner Arc (clone is cheap - just refcount increment)
Ok(Arc::clone(&*arc_arc_value))
}
}
/// Macro to implement ExtractInput for tuple types.
///
/// Generates an implementation that extracts values from multiple dependency channels
/// and combines them into a tuple. All channels are awaited concurrently using join_all.
///
/// We need separate implementations for each tuple size (2-8) because Rust doesn't support
/// variadic generics. This is standard practice when working with tuples in Rust.
///
/// # Compile-time Safety
///
/// The number of dependencies is guaranteed to match the tuple size at compile time through:
/// 1. The `DepsTuple<Tk::Input>` trait bound on `depends_on()` ensures type/arity matching
/// 2. The `IsUnitType` constraint on `From<TaskBuilder>` prevents using tasks without calling `depends_on()`
///
/// Therefore, no runtime validation is needed - if it compiles, the dependency count is correct.
macro_rules! impl_extract_tuple {
($($T:ident : $idx:tt),+) => {
impl<$($T: 'static + Clone + Send + Sync),+> ExtractInput for ($($T,)+) {
async fn extract_from_channels(receivers: Vec<Box<dyn Any + Send>>) -> Result<Self, String> {
// Verify we have the correct number of receivers
let expected_count = [$($idx),+].len();
if receivers.len() != expected_count {
return Err(format!("Expected {} dependencies, got {}", expected_count, receivers.len()));
}
// Convert Vec to iterator so we can consume it
let mut iter = receivers.into_iter();
// Extract each receiver by consuming from the iterator
// Receivers now contain Arc-wrapped values for efficient fanout
// Dereference Box after downcasting to get the receiver itself
// Use type names as variable names to avoid numeric suffix issues
$(
#[allow(non_snake_case)]
let $T = *iter.next()
.ok_or_else(|| format!("Missing receiver at index {}", $idx))?
.downcast::<oneshot::Receiver<Arc<$T>>>()
.map_err(|_| format!("Type mismatch at index {}: expected Arc<{}>",
$idx, std::any::type_name::<$T>()))?;
)+
// Await all channels concurrently - this is a key optimization!
// We can start receiving as soon as ANY dependency completes, not waiting
// for them sequentially.
let arc_results = futures::join!(
$(
async move {
$T.await.map_err(|_| format!("Channel {} closed", $idx))
}
),+
);
// Clone inner values from Arc (cheap for small types, necessary for ownership)
Ok(($(
(*arc_results.$idx?).clone()
,)+))
}
}
};
}
// Generate ExtractInput implementations for tuples of size 2-8.
// Each invocation creates an impl for a specific tuple size.
// Supporting up to 8 elements covers the vast majority of use cases.
impl_extract_tuple!(A:0, B:1);
impl_extract_tuple!(A:0, B:1, C:2);
impl_extract_tuple!(A:0, B:1, C:2, D:3);
impl_extract_tuple!(A:0, B:1, C:2, D:3, E:4);
impl_extract_tuple!(A:0, B:1, C:2, D:3, E:4, F:5);
impl_extract_tuple!(A:0, B:1, C:2, D:3, E:4, F:5, G:6);
impl_extract_tuple!(A:0, B:1, C:2, D:3, E:4, F:5, G:6, H:7);
#[cfg(test)]
mod tests;
-294
View File
@@ -1,294 +0,0 @@
//! Unit tests for extract module
#![allow(dead_code)] // Test helpers generate unused code via macro
use crate::extract::ExtractInput;
use crate::task::Task;
use futures::channel::oneshot;
use std::any::Any;
use std::sync::Arc;
// Test task for error scenarios
#[allow(dead_code)]
struct TestTask {
value: i32,
}
#[crate::task]
#[allow(dead_code)]
impl TestTask {
async fn run(&self) -> i32 {
self.value
}
}
#[allow(dead_code)]
struct TestTaskWithInput;
#[crate::task]
#[allow(dead_code)]
impl TestTaskWithInput {
async fn run(input: &i32) -> i32 {
input * 2
}
}
#[tokio::test]
async fn test_extract_result_type_single_dependency_error() {
// Test Result<T,E> with wrong dependency count
let deps = vec![
Box::new(oneshot::channel::<Result<i32, String>>().1) as Box<dyn Any + Send>,
Box::new(oneshot::channel::<Result<i32, String>>().1) as Box<dyn Any + Send>,
]; // Two dependencies when expecting 1
let result = Result::<i32, String>::extract_from_channels(deps).await;
assert!(result.is_err());
assert!(result.unwrap_err().contains("Expected 1 dependency, got 2"));
}
#[tokio::test]
async fn test_extract_result_type_success() {
// Test Result<T,E> successful extraction
let (tx, rx) = oneshot::channel::<Arc<Result<i32, String>>>();
// Send value in background
tokio::spawn(async move {
let _ = tx.send(Arc::new(Ok(42)));
});
let deps = vec![Box::new(rx) as Box<dyn Any + Send>];
let result = Result::<i32, String>::extract_from_channels(deps).await;
assert!(result.is_ok());
assert_eq!(result.unwrap(), Ok(42));
}
#[tokio::test]
async fn test_extract_option_type_single_dependency_error() {
// Test Option<T> with wrong dependency count
let deps = vec![
Box::new(oneshot::channel::<Option<String>>().1) as Box<dyn Any + Send>,
Box::new(oneshot::channel::<Option<String>>().1) as Box<dyn Any + Send>,
Box::new(oneshot::channel::<Option<String>>().1) as Box<dyn Any + Send>,
]; // Three dependencies when expecting 1
let result = Option::<String>::extract_from_channels(deps).await;
assert!(result.is_err());
assert!(result.unwrap_err().contains("Expected 1 dependency, got 3"));
}
#[tokio::test]
async fn test_extract_option_type_success() {
// Test Option<T> successful extraction
let (tx, rx) = oneshot::channel::<Arc<Option<i32>>>();
// Send value in background
tokio::spawn(async move {
let _ = tx.send(Arc::new(Some(99)));
});
let deps = vec![Box::new(rx) as Box<dyn Any + Send>];
let result = Option::<i32>::extract_from_channels(deps).await;
assert!(result.is_ok());
assert_eq!(result.unwrap(), Some(99));
}
#[tokio::test]
async fn test_extract_vec_type_single_dependency_error() {
// Test Vec<T> with wrong dependency count
let deps = vec![]; // Zero dependencies when expecting 1
let result = Vec::<u8>::extract_from_channels(deps).await;
assert!(result.is_err());
assert!(result.unwrap_err().contains("Expected 1 dependency, got 0"));
}
#[tokio::test]
async fn test_extract_vec_type_success() {
// Test Vec<T> successful extraction
let (tx, rx) = oneshot::channel::<Arc<Vec<i32>>>();
// Send value in background
tokio::spawn(async move {
let _ = tx.send(Arc::new(vec![1, 2, 3]));
});
let deps = vec![Box::new(rx) as Box<dyn Any + Send>];
let result = Vec::<i32>::extract_from_channels(deps).await;
assert!(result.is_ok());
assert_eq!(result.unwrap(), vec![1, 2, 3]);
}
#[tokio::test]
async fn test_extract_primitive_single_dependency_error() {
// Test primitive types with wrong dependency count
let deps = vec![
Box::new(oneshot::channel::<i32>().1) as Box<dyn Any + Send>,
Box::new(oneshot::channel::<i32>().1) as Box<dyn Any + Send>,
]; // Two dependencies when expecting 1
let result = i32::extract_from_channels(deps).await;
assert!(result.is_err());
assert!(result.unwrap_err().contains("Expected 1 dependency, got 2"));
// Test with String
let deps = vec![
Box::new(oneshot::channel::<String>().1) as Box<dyn Any + Send>,
Box::new(oneshot::channel::<String>().1) as Box<dyn Any + Send>,
];
let result = String::extract_from_channels(deps).await;
assert!(result.is_err());
assert!(result.unwrap_err().contains("Expected 1 dependency, got 2"));
// Test with f64
let deps = vec![
Box::new(oneshot::channel::<f64>().1) as Box<dyn Any + Send>,
Box::new(oneshot::channel::<f64>().1) as Box<dyn Any + Send>,
];
let result = f64::extract_from_channels(deps).await;
assert!(result.is_err());
assert!(result.unwrap_err().contains("Expected 1 dependency, got 2"));
}
#[tokio::test]
async fn test_extract_unit_type() {
// Test that unit type always succeeds regardless of dependencies
let deps = vec![];
let result = <()>::extract_from_channels(deps).await;
assert!(result.is_ok());
// Even with dependencies (ignored), unit type succeeds
let deps = vec![
Box::new(oneshot::channel::<i32>().1) as Box<dyn Any + Send>,
Box::new(oneshot::channel::<i32>().1) as Box<dyn Any + Send>,
];
let result = <()>::extract_from_channels(deps).await;
assert!(result.is_ok());
}
#[tokio::test]
async fn test_extract_tuple_basic() {
// Test tuple extraction with channels
let (tx1, rx1) = oneshot::channel::<Arc<i32>>();
let (tx2, rx2) = oneshot::channel::<Arc<i32>>();
// Send values in background
tokio::spawn(async move {
let _ = tx1.send(Arc::new(42));
let _ = tx2.send(Arc::new(100));
});
let deps = vec![
Box::new(rx1) as Box<dyn Any + Send>,
Box::new(rx2) as Box<dyn Any + Send>,
];
// Extract a tuple (i32, i32)
let result = <(i32, i32)>::extract_from_channels(deps).await;
assert!(result.is_ok());
assert_eq!(result.unwrap(), (42, 100));
}
#[tokio::test]
async fn test_extract_channel_closed() {
// Test error when channel is closed
let (tx, rx) = oneshot::channel::<Arc<i32>>();
// Explicitly drop tx to close the channel
drop(tx);
let deps = vec![Box::new(rx) as Box<dyn Any + Send>];
let result = i32::extract_from_channels(deps).await;
assert!(result.is_err());
assert!(result.unwrap_err().contains("Channel closed"));
}
#[tokio::test]
async fn test_extract_type_mismatch() {
// Test type mismatch during downcast
let (tx, rx) = oneshot::channel::<String>();
// Send a String
tokio::spawn(async move {
let _ = tx.send("hello".to_string());
});
// Try to extract as wrong type - put wrong receiver type in deps
let deps = vec![Box::new(rx) as Box<dyn Any + Send>];
let result = i32::extract_from_channels(deps).await;
assert!(result.is_err());
assert!(result.unwrap_err().contains("Type mismatch"));
}
#[tokio::test]
async fn test_extract_all_primitive_types() {
// Test that all primitive types in impl_extract_single! work correctly
macro_rules! test_type {
($t:ty, $val:expr) => {
let (tx, rx) = oneshot::channel::<Arc<$t>>();
tokio::spawn(async move {
let _ = tx.send(Arc::new($val));
});
let deps = vec![Box::new(rx) as Box<dyn Any + Send>];
let result = <$t>::extract_from_channels(deps).await;
assert!(result.is_ok());
assert_eq!(result.unwrap(), $val);
};
}
test_type!(i8, 42i8);
test_type!(i16, 42i16);
test_type!(i32, 42i32);
test_type!(i64, 42i64);
test_type!(String, "test".to_string());
}
#[tokio::test]
async fn test_extract_larger_tuples() {
// Test extraction for larger tuple sizes
let (tx1, rx1) = oneshot::channel::<Arc<i32>>();
let (tx2, rx2) = oneshot::channel::<Arc<i32>>();
let (tx3, rx3) = oneshot::channel::<Arc<i32>>();
let (tx4, rx4) = oneshot::channel::<Arc<i32>>();
let (tx5, rx5) = oneshot::channel::<Arc<i32>>();
// Send values in background
tokio::spawn(async move {
let _ = tx1.send(Arc::new(0));
let _ = tx2.send(Arc::new(1));
let _ = tx3.send(Arc::new(2));
let _ = tx4.send(Arc::new(3));
let _ = tx5.send(Arc::new(4));
});
let deps = vec![
Box::new(rx1) as Box<dyn Any + Send>,
Box::new(rx2) as Box<dyn Any + Send>,
Box::new(rx3) as Box<dyn Any + Send>,
Box::new(rx4) as Box<dyn Any + Send>,
Box::new(rx5) as Box<dyn Any + Send>,
];
// Extract a 5-element tuple
let result = <(i32, i32, i32, i32, i32)>::extract_from_channels(deps).await;
assert!(result.is_ok());
assert_eq!(result.unwrap(), (0, 1, 2, 3, 4));
}
#[tokio::test]
async fn test_extract_arc_type() {
// Test Arc<T> extraction
let (tx, rx) = oneshot::channel::<Arc<Arc<String>>>();
let test_arc = Arc::new("test".to_string());
let test_arc_clone = test_arc.clone();
tokio::spawn(async move {
let _ = tx.send(Arc::new(test_arc));
});
let deps = vec![Box::new(rx) as Box<dyn Any + Send>];
let result = Arc::<String>::extract_from_channels(deps).await;
assert!(result.is_ok());
assert_eq!(*result.unwrap(), *test_arc_clone);
}
+74 -266
View File
@@ -5,21 +5,18 @@
//! //!
//! # Features //! # Features
//! //!
//! - **Compile-time cycle prevention**: The type system makes cycles **impossible**—no runtime
//! cycle detection needed! See [the repository documentation](https://git.swaits.com/swaits/dagx/src/branch/main/docs/CYCLE_PREVENTION.md) for a detailed explanation.
//! - **Compile-time type safety**: Dependencies are validated at compile time through the type //! - **Compile-time type safety**: Dependencies are validated at compile time through the type
//! system. The public API is fully type-safe with no runtime type errors. Internal execution //! system. The public API is fully type-safe with no runtime type errors. Internal execution
//! uses type erasure for heterogeneous task storage, but this is never exposed to users. //! uses type erasure for heterogeneous task storage, but this is never exposed to users.
//! - **Runtime-agnostic**: Works with any async runtime (Tokio, async-std, smol, Embassy, etc.) //! - **Runtime-agnostic**: Works with any async runtime (Tokio, smol, Embassy, etc.)
//! - **Type-state pattern**: The API guides you with compile-time errors if you wire dependencies
//! incorrectly
//! - **Optimal execution**: Topological scheduling with maximum safe parallelism //! - **Optimal execution**: Topological scheduling with maximum safe parallelism
//! - **Zero-cost abstractions**: Leverages generics and monomorphization for minimal overhead
//! - **Error handling**: Result-based error handling with [`DagResult<T>`] for cycle detection
//! and validation
//! //!
//! # Quick Start //! # Quick Start
//! //!
//! ```no_run //! ```no_run
//! use dagx::{task, DagRunner, Task}; //! use dagx::{task, DagRunner, Task, TaskHandle};
//! //!
//! // Source task with read-only state (tuple struct) //! // Source task with read-only state (tuple struct)
//! struct Value(i32); //! struct Value(i32);
@@ -38,7 +35,7 @@
//! } //! }
//! //!
//! # async { //! # async {
//! let dag = DagRunner::new(); //! let mut dag = DagRunner::new();
//! //!
//! // Add source tasks //! // Add source tasks
//! let x = dag.add_task(Value(2)); //! let x = dag.add_task(Value(2));
@@ -48,8 +45,8 @@
//! let sum = dag.add_task(Add).depends_on((&x, &y)); //! let sum = dag.add_task(Add).depends_on((&x, &y));
//! //!
//! // Execute and retrieve results //! // Execute and retrieve results
//! dag.run(|fut| { tokio::spawn(fut); }).await.unwrap(); //!let mut output = dag.run(|fut| async move { tokio::spawn(fut).await.unwrap() }).await.unwrap();
//! assert_eq!(dag.get(sum).unwrap(), 5); //! assert_eq!(output.get(sum), 5);
//! # }; //! # };
//! ``` //! ```
//! //!
@@ -60,12 +57,11 @@
//! dagx automatically optimizes execution for sequential workloads using an inline fast-path. //! dagx automatically optimizes execution for sequential workloads using an inline fast-path.
//! When a layer contains only a single task (common in deep chains and linear pipelines), //! When a layer contains only a single task (common in deep chains and linear pipelines),
//! that task executes inline rather than being spawned. This eliminates spawning overhead, //! that task executes inline rather than being spawned. This eliminates spawning overhead,
//! context switching, and channel creation. //! and context switching.
//! //!
//! **Panic handling**: To maintain consistent behavior between inline and spawned execution, //! **Panic handling**: To maintain consistent behavior between runtimes,
//! panics in inline tasks are caught using `FutureExt::catch_unwind()` and converted to //! panics in tasks are caught using `FutureExt::catch_unwind()` and converted to
//! `DagError::TaskPanicked`. This matches the behavior of async runtimes like Tokio, async-std, //! `DagError::TaskPanicked`. This matches the behavior of async runtimes like Tokio.
//! and smol, which catch panics in spawned tasks and convert them to errors.
//! //!
//! **What this means for you**: //! **What this means for you**:
//! - Tasks behave identically whether executed inline or spawned //! - Tasks behave identically whether executed inline or spawned
@@ -74,8 +70,6 @@
//! - The optimization is completely transparent - no code changes needed //! - The optimization is completely transparent - no code changes needed
//! //!
//! **When inline execution happens**: //! **When inline execution happens**:
//! - Single-task layers (e.g., long sequential chains)
//! - Linear pipelines (A→B→C→D)
//! - Any layer where `layer.len() == 1` after topological ordering //! - Any layer where `layer.len() == 1` after topological ordering
//! //!
//! ## Dependency Limits //! ## Dependency Limits
@@ -85,71 +79,11 @@
//! 2. Use intermediate aggregation tasks //! 2. Use intermediate aggregation tasks
//! 3. Consider if 8+ dependencies indicates a design issue //! 3. Consider if 8+ dependencies indicates a design issue
//! //!
//! ## Task Output Limitations
//!
//! **IMPORTANT**: Tasks **cannot return bare tuples** as output types. This is a technical
//! limitation of the current implementation. If you need to return multiple values, use one
//! of these workarounds:
//!
//! ### Workaround 1: Wrap in Result (Recommended for Fallible Operations)
//!
//! ```
//! use dagx::{task, Task};
//!
//! struct ProcessData;
//! #[task]
//! impl ProcessData {
//! async fn run(input: &String) -> Result<(String, i32, bool), String> {
//! // Wrap the tuple in Result - this works!
//! Ok(("Alice".to_string(), 30, true))
//! }
//! }
//! ```
//!
//! ### Workaround 2: Use a Struct (Recommended for Most Cases)
//!
//! ```
//! use dagx::{task, Task};
//!
//! // Define a struct to hold your multiple values
//! struct UserData {
//! name: String,
//! age: i32,
//! active: bool,
//! }
//!
//! struct ProcessDataStruct;
//! #[task]
//! impl ProcessDataStruct {
//! async fn run(input: &String) -> UserData {
//! // Return the struct - clean and self-documenting
//! UserData {
//! name: "Alice".to_string(),
//! age: 30,
//! active: true,
//! }
//! }
//! }
//! ```
//!
//! ### Why Use Structs Over `Result<(...)>`?
//!
//! While both workarounds solve the technical limitation, **structs are the better choice**
//! for most cases:
//!
//! - **Self-documenting**: `user.name` is clearer than `user.0`
//! - **Refactorable**: Easy to add/remove fields without breaking all dependencies
//! - **Type-safe**: Named fields prevent field order mistakes
//! - **Semantic clarity**: `Result` should indicate fallibility, not just wrap data
//!
//! Use `Result<(...), E>` only when your operation can genuinely fail and you need to
//! return multiple values on success
//!
//! # Core Concepts //! # Core Concepts
//! //!
//! ## Task //! ## Task
//! //!
//! A [`Task`] is a unit of async work with typed inputs and outputs. Use the `#[task]` macro. //! A [`Task`] is a unit of async work with typed inputs and outputs. Use the `#[task]` macro to generate an implementation.
//! //!
//! ### Task Patterns //! ### Task Patterns
//! //!
@@ -208,7 +142,7 @@
//! //!
//! A [`TaskHandle<T>`] is a typed, opaque reference to a task's output. Use it to: //! A [`TaskHandle<T>`] is a typed, opaque reference to a task's output. Use it to:
//! - Wire dependencies between tasks //! - Wire dependencies between tasks
//! - Retrieve results after execution with [`DagRunner::get`] //! - Retrieve results after execution with [`DagOutput::get`]
//! //!
//! ## Dependency Patterns //! ## Dependency Patterns
//! //!
@@ -216,7 +150,7 @@
//! //!
//! ### No Dependencies (Source Tasks) //! ### No Dependencies (Source Tasks)
//! //!
//! Tasks with no dependencies don't call `depends_on()`: //! Tasks with no dependencies return a [`TaskHandle`] directly and don't call `depends_on()`:
//! //!
//! ```no_run //! ```no_run
//! # use dagx::{task, DagRunner, Task}; //! # use dagx::{task, DagRunner, Task};
@@ -226,7 +160,7 @@
//! # async fn run(&self) -> i32 { self.0 } //! # async fn run(&self) -> i32 { self.0 }
//! # } //! # }
//! # async { //! # async {
//! # let dag = DagRunner::new(); //! # let mut dag = DagRunner::new();
//! let source = dag.add_task(Value(42)); //! let source = dag.add_task(Value(42));
//! # }; //! # };
//! ``` //! ```
@@ -248,12 +182,14 @@
//! async fn run(input: &i32) -> i32 { input * 2 } //! async fn run(input: &i32) -> i32 { input * 2 }
//! } //! }
//! # async { //! # async {
//! # let dag = DagRunner::new(); //! # let mut dag = DagRunner::new();
//! # let source = dag.add_task(Value(42)); //! # let source = dag.add_task(Value(42));
//! let doubled = dag.add_task(Double).depends_on(&source); //! let doubled = dag.add_task(Double).depends_on(&source);
//! # }; //! # };
//! ``` //! ```
//! //!
//! To manually implement [`Task`] for a single-dependency type, you must implement [`Task<(T,)>`], not [`Task<T>`].
//!
//! ### Multiple Dependencies //! ### Multiple Dependencies
//! //!
//! Tasks with multiple dependencies receive separate reference parameters (order matters!): //! Tasks with multiple dependencies receive separate reference parameters (order matters!):
@@ -271,7 +207,7 @@
//! async fn run(a: &i32, b: &i32) -> i32 { a + b } //! async fn run(a: &i32, b: &i32) -> i32 { a + b }
//! } //! }
//! # async { //! # async {
//! # let dag = DagRunner::new(); //! # let mut dag = DagRunner::new();
//! # let x = dag.add_task(Value(2)); //! # let x = dag.add_task(Value(2));
//! # let y = dag.add_task(Value(3)); //! # let y = dag.add_task(Value(3));
//! let sum = dag.add_task(Add).depends_on((&x, &y)); //! let sum = dag.add_task(Add).depends_on((&x, &y));
@@ -287,6 +223,7 @@
//! ```no_run //! ```no_run
//! use dagx::{task, DagRunner, Task}; //! use dagx::{task, DagRunner, Task};
//! //!
//!
//! // Source task (tuple struct) //! // Source task (tuple struct)
//! struct Value(i32); //! struct Value(i32);
//! //!
@@ -312,16 +249,16 @@
//! } //! }
//! //!
//! # async { //! # async {
//! let dag = DagRunner::new(); //! let mut dag = DagRunner::new();
//! //!
//! let base = dag.add_task(Value(10)); //! let base = dag.add_task(Value(10));
//! let plus1 = dag.add_task(Add(1)).depends_on(&base); //! let plus1 = dag.add_task(Add(1)).depends_on(&base);
//! let times2 = dag.add_task(Scale(2)).depends_on(&base); //! let times2 = dag.add_task(Scale(2)).depends_on(&base);
//! //!
//! dag.run(|fut| { tokio::spawn(fut); }).await.unwrap(); //!let mut output = dag.run(|fut| async move { tokio::spawn(fut).await.unwrap() }).await.unwrap();
//! //!
//! assert_eq!(dag.get(plus1).unwrap(), 11); //! assert_eq!(output.get(plus1), 11);
//! assert_eq!(dag.get(times2).unwrap(), 20); //! assert_eq!(output.get(times2), 20);
//! # }; //! # };
//! ``` //! ```
//! //!
@@ -332,6 +269,7 @@
//! ```no_run //! ```no_run
//! use dagx::{task, DagRunner, Task}; //! use dagx::{task, DagRunner, Task};
//! //!
//!
//! // Source task for String (tuple struct) //! // Source task for String (tuple struct)
//! struct Name(String); //! struct Name(String);
//! //!
@@ -367,68 +305,16 @@
//! } //! }
//! //!
//! # async { //! # async {
//! let dag = DagRunner::new(); //! let mut dag = DagRunner::new();
//! //!
//! let name = dag.add_task(Name("Alice".to_string())); //! let name = dag.add_task(Name("Alice".to_string()));
//! let age = dag.add_task(Age(30)); //! let age = dag.add_task(Age(30));
//! let active = dag.add_task(Active(true)); //! let active = dag.add_task(Active(true));
//! let result = dag.add_task(FormatUser).depends_on((&name, &age, &active)); //! let result = dag.add_task(FormatUser).depends_on((&name, &age, &active));
//! //!
//! dag.run(|fut| { tokio::spawn(fut); }).await.unwrap(); //!let mut output = dag.run(|fut| async move { tokio::spawn(fut).await.unwrap() }).await.unwrap();
//! //!
//! assert_eq!(dag.get(result).unwrap(), "User: Alice, Age: 30, Active: true"); //! assert_eq!(output.get(result), "User: Alice, Age: 30, Active: true");
//! # };
//! ```
//!
//! ## Many-to-Many Pattern (m ↔ n)
//!
//! Complex DAGs with multiple layers and dependencies:
//!
//! ```no_run
//! use dagx::{task, DagRunner, Task};
//!
//! // Source task (tuple struct)
//! struct Value(i32);
//!
//! #[task]
//! impl Value {
//! async fn run(&self) -> i32 { self.0 }
//! }
//!
//! // Stateless addition (unit struct)
//! struct Add;
//!
//! #[task]
//! impl Add {
//! async fn run(a: &i32, b: &i32) -> i32 { a + b }
//! }
//!
//! // Stateless multiplication (unit struct)
//! struct Multiply;
//!
//! #[task]
//! impl Multiply {
//! async fn run(a: &i32, b: &i32) -> i32 { a * b }
//! }
//!
//! # async {
//! let dag = DagRunner::new();
//!
//! // Layer 1: Sources
//! let x = dag.add_task(Value(2));
//! let y = dag.add_task(Value(3));
//! let z = dag.add_task(Value(5));
//!
//! // Layer 2: Intermediate computations
//! let sum_xy = dag.add_task(Add).depends_on((&x, &y)); // 2 + 3 = 5
//! let prod_yz = dag.add_task(Multiply).depends_on((&y, &z)); // 3 * 5 = 15
//!
//! // Layer 3: Final result
//! let total = dag.add_task(Add).depends_on((&sum_xy, &prod_yz)); // 5 + 15 = 20
//!
//! dag.run(|fut| { tokio::spawn(fut); }).await.unwrap();
//!
//! assert_eq!(dag.get(total).unwrap(), 20);
//! # }; //! # };
//! ``` //! ```
//! //!
@@ -436,7 +322,6 @@
//! //!
//! dagx works with any async runtime. The library has been tested with: //! dagx works with any async runtime. The library has been tested with:
//! - **Tokio** - Most popular async runtime //! - **Tokio** - Most popular async runtime
//! - **async-std** - Alternative async runtime
//! - **smol** - Lightweight async runtime //! - **smol** - Lightweight async runtime
//! //!
//! Examples with different runtimes: //! Examples with different runtimes:
@@ -445,22 +330,17 @@
//! // With Tokio //! // With Tokio
//! #[tokio::main] //! #[tokio::main]
//! async fn main() { //! async fn main() {
//! let dag = DagRunner::new(); //! let mut dag = DagRunner::new();
//! // ... build and run DAG //! // ... build DAG
//! } //! let result = dag.run(|fut| async move { tokio::spawn(fut).await.unwrap() }).await;
//!
//! // With async-std
//! #[async_std::main]
//! async fn main() {
//! let dag = DagRunner::new();
//! // ... build and run DAG
//! } //! }
//! //!
//! // With smol //! // With smol
//! fn main() { //! fn main() {
//! smol::block_on(async { //! smol::block_on(async {
//! let dag = DagRunner::new(); //! let mut dag = DagRunner::new();
//! // ... build and run DAG //! // ... build and run DAG
//! let result = dag.run(|fut| smol::spawn(fut)).await;
//! }); //! });
//! } //! }
//! ``` //! ```
@@ -470,52 +350,34 @@
//! dagx uses [`DagResult<T>`] (an alias for `Result<T, DagError>`) for operations that //! dagx uses [`DagResult<T>`] (an alias for `Result<T, DagError>`) for operations that
//! can fail: //! can fail:
//! //!
//! - [`DagRunner::run`] returns `DagResult<()>` and can fail if a cycle is detected //! - [`DagRunner::run`] returns `DagResult<DagOutput>` and can fail if tasks panic or if multiple
//! - [`DagRunner::get`] returns `DagResult<T>` and can fail if the task hasn't executed or //! concurrent runs are attempted
//! - [`DagOutput::get`] returns `DagResult<T>` and can fail if the task hasn't executed or
//! the handle is invalid //! the handle is invalid
//! //!
//! You can handle errors explicitly or use `.unwrap()` for simple cases:
//!
//! ```no_run //! ```no_run
//! # use dagx::{task, DagRunner, Task}; //! # use dagx::{task, DagRunner, Task};
//! #
//! # struct Value(i32); //! # struct Value(i32);
//! # #[task] //! # #[task]
//! # impl Value { //! # impl Value {
//! # async fn run(&self) -> i32 { self.0 } //! # async fn run(&self) -> i32 { self.0 }
//! # } //! # }
//! # async { //! # async {
//! # let dag = DagRunner::new(); //! # let mut dag = DagRunner::new();
//! # let node = dag.add_task(Value(42)); //! # let node = dag.add_task(Value(42));
//! // Simple approach with .unwrap() //! // Simple approach with .unwrap()
//! dag.run(|fut| { tokio::spawn(fut); }).await.unwrap(); //! let mut output = dag.run(|fut| async move { tokio::spawn(fut).await.unwrap() }).await.unwrap();
//! let result = dag.get(node).unwrap(); //! let result = output.get(node);
//! # let mut dag = DagRunner::new();
//! //!
//! // Or handle errors explicitly //! // Or handle errors explicitly
//! match dag.run(|fut| { tokio::spawn(fut); }).await { //! match dag.run(|fut| async move { tokio::spawn(fut).await.unwrap() }).await {
//! Ok(_) => println!("DAG executed successfully"), //! Ok(_) => println!("DAG executed successfully"),
//! Err(e) => eprintln!("DAG execution failed: {}", e), //! Err(e) => eprintln!("DAG execution failed: {}", e),
//! } //! }
//! # }; //! # };
//! ``` //! ```
//!
//! # Documentation Accuracy
//!
//! This library maintains documentation accuracy through:
//!
//! 1. **Doctests**: All code examples are tested via `cargo test --doc`
//! 2. **Examples**: All files in `examples/` are tested via `cargo build --examples`
//! 3. **Type signatures**: Documentation reflects actual API return types
//! 4. **Error handling**: All examples show proper `DagResult<T>` handling
//! 5. **Safety claims**: All safety guarantees are verified by tests
//!
//! Last audited: 2025-10-06 for v0.1.0
//!
//! Changes that require documentation updates:
//! - API signature changes (`run()`, `get()`, etc.)
//! - New error types or error handling changes
//! - Safety guarantee changes
//! - Example code updates
//!
//! # API Design Principles //! # API Design Principles
//! //!
//! dagx follows these API design principles: //! dagx follows these API design principles:
@@ -523,48 +385,13 @@
//! 1. **Type Safety**: Dependencies validated at compile time via type-state pattern //! 1. **Type Safety**: Dependencies validated at compile time via type-state pattern
//! 2. **Builder Pattern**: Fluent interface with `add_task().depends_on()` //! 2. **Builder Pattern**: Fluent interface with `add_task().depends_on()`
//! 3. **Error Handling**: All fallible operations return `DagResult<T>` //! 3. **Error Handling**: All fallible operations return `DagResult<T>`
//! 4. **Minimal Surface**: Small, focused API with 4 core types and 1 trait //! 4. **Minimal Surface**: Small, focused API
//! 5. **Zero Cost**: Leverages generics for monomorphization
//! 6. **Consistency**:
//! - All types are PascalCase
//! - All methods are snake_case
//! - Mutable operations take `&mut self`
//! - Builder methods consume `self`
//! - Results are always `DagResult<T>`
//! //!
//! # Performance Characteristics //! # Performance Characteristics
//! //!
//! dagx is designed for minimal overhead and optimal parallel execution. //! dagx is designed for minimal overhead and optimal parallel execution.
//! //!
//! ## Benchmarks //! See the [repository](https://git.swaits.com/swaits/dagx) for up-to-date benchmarks.
//!
//! Run benchmarks with:
//! ```bash
//! cargo bench
//! ```
//!
//! View the detailed HTML reports:
//! ```bash
//! # macOS
//! open target/criterion/report/index.html
//!
//! # Linux
//! xdg-open target/criterion/report/index.html
//!
//! # Windows
//! start target/criterion/report/index.html
//!
//! # Or manually open target/criterion/report/index.html in your browser
//! ```
//!
//! Typical performance on modern hardware (AMD 7840U - Zen 4 laptop CPU):
//! - **DAG creation**: ~20ns empty DAG, ~96ns per task added
//! - **Execution overhead**: ~119ns per task (framework coordination only)
//! - **Total overhead** (construction + execution): ~223ns per task
//! - **Scaling**: Linear overhead with task count - 100 tasks in ~123µs, 1000 tasks in ~1.15ms
//!
//! For real-world workloads where tasks perform meaningful work (I/O, computation), the framework
//! overhead is typically well under 1% of total execution time.
//! //!
//! ## Performance Tips //! ## Performance Tips
//! //!
@@ -574,7 +401,8 @@
//! You output `T`, the framework handles the Arc wrapping: //! You output `T`, the framework handles the Arc wrapping:
//! //!
//! ``` //! ```
//! use dagx::{task, DagRunner, Task}; //! # use dagx::{task, DagRunner, Task};
//! #
//! //!
//! // ✅ CORRECT: Just output Vec<String>, framework wraps in Arc internally //! // ✅ CORRECT: Just output Vec<String>, framework wraps in Arc internally
//! struct FetchData; //! struct FetchData;
@@ -586,7 +414,7 @@
//! } //! }
//! //!
//! // Downstream tasks receive &Vec<String> as normal //! // Downstream tasks receive &Vec<String> as normal
//! // Behind the scenes: Arc<Vec<String>> is cloned cheaply, then inner Vec extracted //! // Behind the scenes: Arc<Vec<String>> is cloned cheaply, then a reference to the inner Vec is extracted
//! struct ProcessData; //! struct ProcessData;
//! #[task] //! #[task]
//! impl ProcessData { //! impl ProcessData {
@@ -596,7 +424,7 @@
//! } //! }
//! //!
//! # async { //! # async {
//! let dag = DagRunner::new(); //! let mut dag = DagRunner::new();
//! let data = dag.add_task(FetchData); //! let data = dag.add_task(FetchData);
//! //!
//! // All three tasks get efficient Arc-wrapped sharing automatically //! // All three tasks get efficient Arc-wrapped sharing automatically
@@ -604,80 +432,60 @@
//! let task2 = dag.add_task(ProcessData).depends_on(&data); //! let task2 = dag.add_task(ProcessData).depends_on(&data);
//! let task3 = dag.add_task(ProcessData).depends_on(&data); //! let task3 = dag.add_task(ProcessData).depends_on(&data);
//! //!
//! dag.run(|fut| { tokio::spawn(fut); }).await.unwrap(); //!let mut output = dag.run(|fut| async move { tokio::spawn(fut).await.unwrap() }).await.unwrap();
//! # }; //! # };
//! ``` //! ```
//! //!
//! **How it works:** //! **How it works:**
//! - Your task outputs `T` (e.g., `Vec<String>`) //! - Your task outputs `T` (e.g., `Vec<String>`)
//! - Framework wraps it in `Arc<T>` internally //! - Framework wraps it in `Arc<T>` internally
//! - For fan-out (1→N), Arc is cloned N times (just atomic pointer increments - O(1)) //! - For fan-out (1→N), Arc is cloned N times
//! - Each downstream task receives `&T` after extracting from Arc //! - Each downstream task receives `&T` after extracting from Arc
//! - When you call `dag.get()`, the Arc is cloned and inner value extracted
//!
//! **Performance characteristics:**
//! - **Heap types** (Vec, String, HashMap): Arc overhead is negligible (~few ns)
//! - **Copy types** (i32, usize): Small Arc overhead but usually still negligible
//! - See `cargo bench` for actual measurements on your hardware
//!
//! **Advanced - Zero-copy optimization:**
//! If you want true zero-copy sharing (no extraction), output `Arc<T>` explicitly:
//! ```
//! # use dagx::{task, Task};
//! # use std::sync::Arc;
//! struct ZeroCopyData;
//! #[task]
//! impl ZeroCopyData {
//! // Output Arc<T> directly
//! async fn run() -> Arc<Vec<String>> {
//! Arc::new(vec!["data".to_string(); 10_000])
//! }
//! }
//!
//! struct ZeroCopyProcess;
//! #[task]
//! impl ZeroCopyProcess {
//! // Receive &Arc<T> - just clones the Arc pointer, no data extraction
//! async fn run(data: &Arc<Vec<String>>) -> usize {
//! data.len()
//! }
//! }
//! ```
//! This becomes `Arc<Arc<T>>` internally, but ExtractInput unwraps one layer automatically.
//! Use this when you have many dependents AND want to avoid the clone() of inner data
//! //!
//! ### Other Tips //! ### Other Tips
//! //!
//! 1. **Minimize task count**: Combine small operations into larger tasks //! 1. **Minimize task count**: Combine small operations into larger tasks
//! 2. **Use appropriate granularity**: Don't create tasks for trivial work (< 1µs) //! 2. **Use appropriate granularity**: Don't create tasks for trivial work (< 1µs)
//! 3. **Parallel execution**: dagx automatically maximizes parallelism
//! //!
//! Run `cargo bench` to see comprehensive benchmarks including basic operations, scaling //! Run `cargo bench` to see comprehensive benchmarks including basic operations, scaling
//! characteristics, common patterns (fan-out, diamond), and realistic workloads. //! characteristics, common patterns (fan-out, diamond), and realistic workloads.
//! //!
//! # Optional Tracing Support
//!
//! dagx provides optional observability through the `tracing` crate. It can be enabled with the `tracing` feature flag.
//!
//! ## Log Levels
//!
//! - **INFO**: DAG execution start/completion
//! - **DEBUG**: Task additions, dependency wiring, layer computation
//! - **TRACE**: Individual task execution (inline vs spawned), detailed execution flow
//! - **ERROR**: Task panics, concurrent execution attempts
//!
//! Control log level with the `RUST_LOG` environment variable:
//!
//! ```bash
//! RUST_LOG=dagx=info cargo run # High-level execution info
//! RUST_LOG=dagx=debug cargo run # Task and layer details
//! RUST_LOG=dagx=trace cargo run # All execution details
//! ```
#![allow(private_bounds, private_interfaces)] extern crate self as dagx;
// Module declarations
mod builder; mod builder;
mod deps; mod deps;
mod error; mod error;
mod extract;
mod node; mod node;
mod output;
mod runner; mod runner;
mod task; mod task;
mod types;
// Public re-exports // Public re-exports
pub use builder::TaskBuilder; pub use builder::{TaskBuilder, TaskHandle};
pub use error::{DagError, DagResult}; pub use error::{DagError, DagResult};
pub use output::DagOutput;
pub use runner::DagRunner; pub use runner::DagRunner;
pub use task::Task; pub use task::{Task, TaskInput};
pub use types::{Pending, TaskHandle};
// Re-export the procedural macro // Re-export the procedural macro
#[cfg(feature = "derive")]
pub use dagx_macros::task; pub use dagx_macros::task;
// Internal testing support - not part of the public API
#[doc(hidden)]
pub use task::task_fn;
+35 -83
View File
@@ -8,21 +8,12 @@
use std::any::Any; use std::any::Any;
use std::future::Future; use std::future::Future;
use std::marker::PhantomData;
use std::pin::Pin; use std::pin::Pin;
use std::sync::Arc; use std::sync::Arc;
use futures::channel::oneshot;
use crate::error::DagError; use crate::error::DagError;
use crate::extract::ExtractInput; use crate::task::{Task, TaskInput};
use crate::task::Task;
use crate::types::NodeId;
/// Type-erased channel endpoints (senders or receivers)
type ErasedChannels = Vec<Box<dyn Any + Send>>;
/// Type-erased channel pair (senders, receivers)
type ChannelPair = (ErasedChannels, ErasedChannels);
/// Type-erased async execution result (Arc-wrapped for efficient fanout) /// Type-erased async execution result (Arc-wrapped for efficient fanout)
type ExecuteFuture = type ExecuteFuture =
@@ -34,110 +25,71 @@ type ExecuteFuture =
/// collection. While the public API (Task, Handle, Node) is fully type-safe, internally we /// collection. While the public API (Task, Handle, Node) is fully type-safe, internally we
/// need dynamic dispatch to execute tasks with different input/output types. /// need dynamic dispatch to execute tasks with different input/output types.
/// ///
/// This is the ONLY place where type erasure occurs - channels are created with full type /// This is the ONLY place where type erasure occurs - tasks are created with full type
/// information and only erased at this trait boundary. /// information and only erased at this trait boundary.
///
/// This is an implementation detail and not part of the public API.
pub(crate) trait ExecutableNode: Send { pub(crate) trait ExecutableNode: Send {
/// Create type-erased output channels for this node's dependents. /// Execute the task with type-erased inputs and output
///
/// Returns (senders, receivers) where:
/// - senders: Type-erased senders that this node will use to send outputs
/// - receivers: Type-erased receivers that dependents will use to receive inputs
fn create_output_channels(&self, num_dependents: usize) -> ChannelPair;
/// Execute the task with type-erased input receivers and output senders.
/// ///
/// The TypedNode implementation knows the concrete types and can safely downcast. /// The TypedNode implementation knows the concrete types and can safely downcast.
/// Consumes the node and returns the output value. /// Consumes the node and returns the output value.
fn execute_with_channels( fn execute_with_deps(
self: Box<Self>, self: Box<Self>,
receivers: ErasedChannels, dependencies: Vec<Arc<dyn Any + Send + Sync>>,
senders: ErasedChannels,
) -> ExecuteFuture; ) -> ExecuteFuture;
} }
/// Fully-typed node storage for a single task. /// Fully-typed node storage for a single task.
/// ///
/// Stores a task of type T. Implements ExecutableNode for type erasure, /// Stores a task of type T. Implements ExecutableNode for type erasure,
/// allowing heterogeneous tasks to be stored in a single Vec. /// allowing heterogeneous tasks to be stored together.
/// ///
/// # Ownership Model /// # Ownership Model
/// ///
/// The task is owned directly (no Mutex needed) and consumed during execution. /// The task is owned directly (no Mutex needed) and consumed during execution.
/// Each task executes exactly once, taking ownership and producing an output. /// Each task executes exactly once, taking ownership and producing an output.
/// ///
/// # Oneshot Channels /// The output is also returned from execute_with_deps to be passed to dependents and returned as an output
/// pub(crate) struct TypedNode<Input, T>
/// Outputs are communicated via oneshot channels for inter-task communication.
/// The output is also returned from execute_with_channels for central storage.
pub(crate) struct TypedNode<T: Task> {
pub(crate) id: NodeId,
pub(crate) task: T,
}
impl<T: Task + 'static> TypedNode<T> {
pub(crate) fn new(id: NodeId, task: T) -> Self {
Self { id, task }
}
}
impl<T: Task + 'static> ExecutableNode for TypedNode<T>
where where
T::Input: 'static + Clone + ExtractInput, Input: Send + Sync + 'static,
T::Output: 'static + Clone, T: Task<Input> + 'static,
{ {
fn create_output_channels(&self, num_dependents: usize) -> ChannelPair { pub(crate) task: T,
let mut senders = Vec::new(); phantom: PhantomData<Input>,
let mut receivers = Vec::new();
for _ in 0..num_dependents {
// Create channels that pass Arc-wrapped outputs (efficient for fanout)
let (tx, rx) = oneshot::channel::<Arc<T::Output>>();
senders.push(Box::new(tx) as Box<dyn Any + Send>);
receivers.push(Box::new(rx) as Box<dyn Any + Send>);
} }
(senders, receivers) impl<Input, T> TypedNode<Input, T>
where
T: Task<Input>,
Input: Send + Sync + 'static,
{
pub(crate) fn new(task: T) -> Self {
Self {
task,
phantom: PhantomData,
}
}
} }
fn execute_with_channels( impl<Input, T> ExecutableNode for TypedNode<Input, T>
where
T: Task<Input>,
Input: Send + Sync + 'static,
{
fn execute_with_deps(
self: Box<Self>, self: Box<Self>,
receivers: ErasedChannels, dependencies: Vec<Arc<dyn Any + Send + Sync>>,
senders: ErasedChannels,
) -> ExecuteFuture { ) -> ExecuteFuture {
Box::pin(async move { Box::pin(async move {
// Extract typed input from type-erased receivers // Use the task's run method which has inline extraction logic
// TypedNode<T> knows T::Input, so it can safely downcast // generated by the #[task] macro.
let input = match T::Input::extract_from_channels(receivers).await { let output = self.task.run(TaskInput::new(dependencies.iter())).await;
Ok(input) => input,
Err(_msg) => {
// Failed to extract input - likely channel closed or type mismatch
return Err(DagError::InvalidDependency { task_id: self.id.0 });
}
};
// Execute task (consumes self.task)
let output = self.task.run(input).await;
// Arc Wrapping Strategy: // Arc Wrapping Strategy:
// Wrap output in Arc ONCE for efficient fan-out sharing. // Wrap output in Arc ONCE for efficient fan-out sharing.
// For 1:N dependencies, this enables O(1) sharing (Arc clone = atomic increment) // Downstream tasks receive &T after extraction from Arc.
// instead of O(N) cloning (N deep copies of output data).
// Downstream tasks receive &T after ExtractInput extracts from Arc.
let arc_output = Arc::new(output); let arc_output = Arc::new(output);
// Send Arc to all dependents via their channels
// TypedNode<T> knows T::Output, so downcast is safe
for sender_any in senders {
if let Ok(sender) = sender_any.downcast::<oneshot::Sender<Arc<T::Output>>>() {
// Clone Arc (just atomic refcount increment, NOT data clone)
// This is the key performance optimization for fan-out patterns
// Ignore send errors - dependent might have been cancelled
let _ = sender.send(Arc::clone(&arc_output));
}
}
// Return Arc-wrapped output (for storage by runner) // Return Arc-wrapped output (for storage by runner)
Ok(arc_output as Arc<dyn Any + Send + Sync>) Ok(arc_output as Arc<dyn Any + Send + Sync>)
}) })
+21 -113
View File
@@ -1,8 +1,8 @@
//! Unit tests for node module //! Unit tests for node module
use std::sync::Arc;
use crate::node::{ExecutableNode, TypedNode}; use crate::node::{ExecutableNode, TypedNode};
use crate::task::Task;
use crate::types::NodeId;
// Simple test task // Simple test task
struct TestTask { struct TestTask {
@@ -16,28 +16,6 @@ impl TestTask {
} }
} }
#[test]
fn test_create_output_channels() {
let node = TypedNode::new(NodeId(0), TestTask { value: 42 });
// Create channels for 3 dependents
let (senders, receivers) = node.create_output_channels(3);
assert_eq!(senders.len(), 3);
assert_eq!(receivers.len(), 3);
}
#[test]
fn test_create_output_channels_zero_dependents() {
let node = TypedNode::new(NodeId(0), TestTask { value: 42 });
// Create channels for 0 dependents (sink node)
let (senders, receivers) = node.create_output_channels(0);
assert_eq!(senders.len(), 0);
assert_eq!(receivers.len(), 0);
}
struct TaskWithDependency; struct TaskWithDependency;
#[crate::task] #[crate::task]
@@ -48,68 +26,38 @@ impl TaskWithDependency {
} }
#[tokio::test] #[tokio::test]
async fn test_execute_with_channels_missing_input() { async fn test_execute_with_deps_success() {
// Test when task expects input but doesn't receive it
let node = Box::new(TypedNode::new(NodeId(0), TaskWithDependency));
// TaskWithDependency expects an i32 input, but we provide no receivers
let receivers = vec![];
let senders = vec![];
let result = node.execute_with_channels(receivers, senders).await;
assert!(result.is_err());
if let Err(crate::error::DagError::InvalidDependency { task_id }) = result {
assert_eq!(task_id, 0);
} else {
panic!("Expected InvalidDependency error, got: {:?}", result);
}
}
#[tokio::test]
async fn test_execute_with_channels_success() {
// Test successful execution with channels // Test successful execution with channels
let dependency_node = Box::new(TypedNode::new(NodeId(0), TestTask { value: 21 })); let dependency_node = Box::new(TypedNode::new(TestTask { value: 21 }));
let dependent_node = Box::new(TypedNode::new(NodeId(1), TaskWithDependency)); let dependent_node = Box::new(TypedNode::new(TaskWithDependency));
// Create a channel from dependency to dependent (Arc-wrapped)
let (tx, rx) = futures::channel::oneshot::channel::<std::sync::Arc<i32>>();
// Execute dependency node with sender // Execute dependency node with sender
let dep_senders = vec![Box::new(tx) as Box<dyn std::any::Any + Send>]; let dep_dependencies = vec![];
let dep_receivers = vec![];
// Execute in background // Execute in background
let dep_handle = tokio::spawn(async move { let dep_result =
dependency_node tokio::spawn(async move { dependency_node.execute_with_deps(dep_dependencies).await })
.execute_with_channels(dep_receivers, dep_senders)
.await .await
}); .unwrap()
.unwrap();
// Execute dependent node with receiver // Execute dependent node with receiver
let receivers = vec![Box::new(rx) as Box<dyn std::any::Any + Send>]; let dependencies = vec![dep_result as Arc<dyn std::any::Any + Send + Sync>];
let senders = vec![];
let result = dependent_node let result = dependent_node.execute_with_deps(dependencies).await;
.execute_with_channels(receivers, senders)
.await;
// Wait for dependency to complete
let _ = dep_handle.await;
assert!(result.is_ok()); assert!(result.is_ok());
} }
#[tokio::test] #[tokio::test]
async fn test_execute_with_channels_sink_node() { async fn test_execute_with_deps_sink_node() {
// Test that sink nodes (no dependents) return their output // Test that sink nodes (no dependents) return their output
let node = Box::new(TypedNode::new(NodeId(0), TestTask { value: 42 })); let node = Box::new(TypedNode::new(TestTask { value: 42 }));
// No senders means this is a sink node // No senders means this is a sink node
let receivers = vec![]; let dependencies = vec![];
let senders = vec![];
let result = node.execute_with_channels(receivers, senders).await; let result = node.execute_with_deps(dependencies).await;
assert!(result.is_ok()); assert!(result.is_ok());
// Check output was returned (Arc-wrapped) // Check output was returned (Arc-wrapped)
@@ -119,58 +67,18 @@ async fn test_execute_with_channels_sink_node() {
} }
#[tokio::test] #[tokio::test]
async fn test_execute_with_channels_non_sink_node() { async fn test_execute_with_deps_non_sink_node() {
// Test that non-sink nodes send output via channels // Test that non-sink nodes return their output
let node = TypedNode::new(NodeId(0), TestTask { value: 42 }); let node = TypedNode::new(TestTask { value: 42 });
// Create output channels for one dependent
let (senders, mut receivers) = node.create_output_channels(1);
// Execute with senders (non-sink node) // Execute with senders (non-sink node)
let node = Box::new(node); let node = Box::new(node);
let exec_receivers = vec![]; let result = node.execute_with_deps(vec![]).await;
let result = node.execute_with_channels(exec_receivers, senders).await;
assert!(result.is_ok()); assert!(result.is_ok());
// Verify output was sent through channel (Arc-wrapped) // Verify output was sent through channel (Arc-wrapped)
let rx = receivers let arc_value = result.unwrap().downcast::<i32>().unwrap();
.pop()
.unwrap()
.downcast::<futures::channel::oneshot::Receiver<std::sync::Arc<i32>>>()
.unwrap();
let received_arc = rx.await;
assert!(received_arc.is_ok());
assert_eq!(*received_arc.unwrap(), 42);
// Also verify output was returned (Arc-wrapped) // Also verify output was returned (Arc-wrapped)
let output = result.unwrap();
let arc_value = output.downcast::<i32>().unwrap();
assert_eq!(*arc_value, 42); assert_eq!(*arc_value, 42);
} }
#[tokio::test]
async fn test_execute_with_multiple_dependents() {
// Test that output is sent to multiple dependents
let node = TypedNode::new(NodeId(0), TestTask { value: 100 });
// Create output channels for three dependents
let (senders, receivers) = node.create_output_channels(3);
// Execute node
let node = Box::new(node);
let exec_receivers = vec![];
let result = node.execute_with_channels(exec_receivers, senders).await;
assert!(result.is_ok());
// Verify all three dependents receive the output (Arc-wrapped)
for receiver in receivers {
let rx = receiver
.downcast::<futures::channel::oneshot::Receiver<std::sync::Arc<i32>>>()
.unwrap();
let received_arc = rx.await;
assert!(received_arc.is_ok());
assert_eq!(*received_arc.unwrap(), 100);
}
}
+69
View File
@@ -0,0 +1,69 @@
use std::{any::Any, sync::Arc};
use crate::{builder::NodeId, runner::PassThroughHashMap, TaskHandle};
pub struct DagOutput {
outputs: PassThroughHashMap<NodeId, Arc<dyn Any + Send + Sync + 'static>>,
}
impl DagOutput {
/// Retrieve a task's output after [`crate::DagRunner::run`].
///
/// # Behavior
///
/// All task outputs are stored after execution and can be retrieved via get().
///
/// # Panics
///
/// This function panics if called with a [`TaskHandle`] from a [`crate::DagRunner`] instance other than the
/// one which produced this DagOutput.
///
/// # Examples
///
/// ```no_run
/// # use dagx::{task, DagRunner, Task};
/// #
/// struct Configuration {
/// setting: i32,
/// }
///
/// impl Configuration {
/// fn new(setting: i32) -> Self {
/// Self { setting }
/// }
/// }
///
/// #[task]
/// impl Configuration {
/// async fn run(&mut self) -> i32 { self.setting }
/// }
///
/// # async {
/// let mut dag = DagRunner::new();
///
/// // Construct task with specific setting value
/// let task = dag.add_task(Configuration::new(42));
///
/// let mut output = dag.run(|fut| async move { tokio::spawn(fut).await.unwrap() }).await.unwrap();
///
/// assert_eq!(output.get(task), 42);
/// # };
/// ```
pub fn get<T: 'static + Send + Sync>(&mut self, handle: TaskHandle<T>) -> T {
// Because this consumes the task handle, it can only be retrieved once, so is certain to be there
let arc_output = self.outputs.remove(&handle.id).unwrap();
// Downcast Arc<dyn Any> to Arc<T>
let output = arc_output.downcast::<T>().unwrap();
// Because we never give the user access to anything but a reference to the Arc,
// we can guarantee that the strong reference count is exactly one here
Arc::into_inner(output).unwrap()
}
pub(super) fn new(
outputs: PassThroughHashMap<NodeId, Arc<dyn Any + Send + Sync + 'static>>,
) -> Self {
Self { outputs }
}
}
+232 -344
View File
@@ -2,39 +2,55 @@
//! //!
//! Provides DagRunner for building and executing directed acyclic graphs of async tasks //! Provides DagRunner for building and executing directed acyclic graphs of async tasks
//! with compile-time type-safe dependencies. //! with compile-time type-safe dependencies.
//!
//! Uses Mutex for interior mutability to enable builder pattern (`&self` instead of `&mut self`).
use std::any::Any;
use std::collections::{HashMap, HashSet, VecDeque}; use std::collections::{HashMap, HashSet, VecDeque};
use std::future::Future;
use std::hash::{BuildHasher, Hasher};
use std::panic::AssertUnwindSafe; use std::panic::AssertUnwindSafe;
use std::sync::{ use std::sync::Arc;
atomic::{AtomicBool, Ordering},
Arc,
};
use futures::channel::mpsc; use futures_util::future::BoxFuture;
use futures::future::BoxFuture; use futures_util::{stream::FuturesUnordered, FutureExt, StreamExt, TryFutureExt};
use futures::{FutureExt, StreamExt};
use parking_lot::Mutex;
use crate::builder::TaskBuilder; #[cfg(feature = "tracing")]
use tracing::{debug, error, info, trace};
use crate::builder::{NodeId, TaskWire};
use crate::error::{DagError, DagResult}; use crate::error::{DagError, DagResult};
use crate::extract::ExtractInput;
use crate::node::{ExecutableNode, TypedNode}; use crate::node::{ExecutableNode, TypedNode};
use crate::task::Task; use crate::DagOutput;
use crate::types::{NodeId, Pending, TaskHandle};
// Guard to ensure run_lock is released even on early return or panic /// Fast hasher using values as hashes
struct RunGuard<'a> { #[derive(Default, Clone)]
lock: &'a AtomicBool, pub(crate) struct PassThroughHasher {
hash: u64,
} }
impl<'a> Drop for RunGuard<'a> { impl Hasher for PassThroughHasher {
fn drop(&mut self) { fn finish(&self) -> u64 {
self.lock.store(false, Ordering::SeqCst); self.hash
}
fn write_u32(&mut self, i: u32) {
self.hash = i as u64;
}
fn write(&mut self, _bytes: &[u8]) {
panic!("PassThroughHasher used on invalid type");
} }
} }
impl BuildHasher for PassThroughHasher {
type Hasher = PassThroughHasher;
fn build_hasher(&self) -> Self::Hasher {
PassThroughHasher::default()
}
}
pub(crate) type PassThroughHashMap<K, V> = HashMap<K, V, PassThroughHasher>;
/// Build and execute a typed DAG of tasks. /// Build and execute a typed DAG of tasks.
/// ///
/// A `DagRunner` is the main orchestrator for building and executing a directed acyclic graph /// A `DagRunner` is the main orchestrator for building and executing a directed acyclic graph
@@ -43,15 +59,16 @@ impl<'a> Drop for RunGuard<'a> {
/// # Workflow /// # Workflow
/// ///
/// 1. Create a new DAG with [`DagRunner::new`] /// 1. Create a new DAG with [`DagRunner::new`]
/// 2. Add tasks with [`DagRunner::add_task`] to get [`TaskBuilder`] builders /// 2. Add tasks with [`DagRunner::add_task`] to get builders
/// 3. Wire dependencies with [`TaskBuilder::depends_on`] /// 3. Wire dependencies with [`crate::TaskBuilder::depends_on`]
/// 4. Execute all tasks with [`DagRunner::run`] /// 4. Execute all tasks with [`DagRunner::run`]
/// 5. Optionally retrieve outputs with [`DagRunner::get`] /// 5. Optionally retrieve outputs with [`DagOutput::get`]
/// ///
/// # Examples /// # Examples
/// ///
/// ```no_run /// ```no_run
/// # use dagx::{task, DagRunner, Task}; /// # use dagx::{task, DagRunner, Task};
/// #
/// // Task with state constructed via ::new() /// // Task with state constructed via ::new()
/// struct LoadValue { value: i32 } /// struct LoadValue { value: i32 }
/// ///
@@ -64,7 +81,6 @@ impl<'a> Drop for RunGuard<'a> {
/// async fn run(&mut self) -> i32 { self.value } /// async fn run(&mut self) -> i32 { self.value }
/// } /// }
/// ///
/// // Unit struct - no fields needed
/// struct Add; /// struct Add;
/// ///
/// #[task] /// #[task]
@@ -73,32 +89,24 @@ impl<'a> Drop for RunGuard<'a> {
/// } /// }
/// ///
/// # async { /// # async {
/// let dag = DagRunner::new(); /// let mut dag = DagRunner::new();
/// ///
/// // Construct instances using ::new() pattern /// // Construct instances using ::new() pattern
/// let x = dag.add_task(LoadValue::new(2)); /// let x = dag.add_task(LoadValue::new(2));
/// let y = dag.add_task(LoadValue::new(3)); /// let y = dag.add_task(LoadValue::new(3));
/// let sum = dag.add_task(Add).depends_on((&x, &y)); /// let sum = dag.add_task(Add).depends_on((&x, &y));
/// ///
/// dag.run(|fut| { tokio::spawn(fut); }).await.unwrap(); ///let mut output = dag.run(|fut| async move { tokio::spawn(fut).await.unwrap() }).await.unwrap();
/// ///
/// assert_eq!(dag.get(sum).unwrap(), 5); /// assert_eq!(output.get(sum), 5);
/// # }; /// # };
/// ``` /// ```
///
/// Uses Mutex for interior mutability to enable the builder pattern (`&self` not `&mut self`).
/// This allows fluent chaining of `add_task()` calls.
///
/// Nodes use Option to allow taking ownership during execution.
/// Outputs are Arc-wrapped and stored separately for retrieval via get().
/// Arc enables efficient sharing during fanout without cloning data.
pub struct DagRunner { pub struct DagRunner {
pub(crate) nodes: Mutex<Vec<Option<Box<dyn ExecutableNode>>>>, pub(crate) nodes: Vec<Option<Box<dyn ExecutableNode + Sync>>>,
pub(crate) outputs: Mutex<HashMap<NodeId, std::sync::Arc<dyn std::any::Any + Send + Sync>>>, /// node -> dependencies
pub(crate) edges: Mutex<HashMap<NodeId, Vec<NodeId>>>, // node -> dependencies pub(crate) edges: PassThroughHashMap<NodeId, Vec<NodeId>>,
pub(crate) dependents: Mutex<HashMap<NodeId, Vec<NodeId>>>, // node -> tasks that depend on it /// node -> tasks that depend on it
pub(crate) next_id: Mutex<usize>, pub(crate) dependents: PassThroughHashMap<NodeId, Vec<NodeId>>,
pub(crate) run_lock: AtomicBool, // Ensures only one run() at a time
} }
impl Default for DagRunner { impl Default for DagRunner {
@@ -115,36 +123,27 @@ impl DagRunner {
/// ``` /// ```
/// use dagx::DagRunner; /// use dagx::DagRunner;
/// ///
/// let dag = DagRunner::new(); /// let mut dag = DagRunner::new();
/// ``` /// ```
pub fn new() -> Self { pub fn new() -> Self {
Self { Self {
nodes: Mutex::new(Vec::new()), nodes: Vec::new(),
outputs: Mutex::new(HashMap::new()), edges: HashMap::default(),
edges: Mutex::new(HashMap::new()), dependents: HashMap::default(),
dependents: Mutex::new(HashMap::new()),
next_id: Mutex::new(0),
run_lock: AtomicBool::new(false),
} }
} }
pub(crate) fn alloc_id(&self) -> NodeId {
let mut next_id = self.next_id.lock();
let id = NodeId(*next_id);
*next_id += 1;
id
}
/// Add a task instance to the DAG, returning a node builder for wiring dependencies. /// Add a task instance to the DAG, returning a node builder for wiring dependencies.
/// ///
/// The returned [`TaskBuilder<Tk, Pending>`](TaskBuilder) can be used to: /// If the task has no dependencies, a [`crate::TaskHandle`] will be returned.
/// - Specify dependencies via [`TaskBuilder::depends_on`] /// If not, dependencies should be specified for the returned [`crate::TaskBuilder`]
/// - Used directly as a [`TaskHandle`] to the task's output /// using [`crate::TaskBuilder::depends_on`].
/// ///
/// # Examples /// # Examples
/// ///
/// ```no_run /// ```no_run
/// # use dagx::{task, DagRunner, Task}; /// # use dagx::{task, DagRunner, Task};
/// #
/// // Task with state - shows you construct with specific value /// // Task with state - shows you construct with specific value
/// struct LoadValue { /// struct LoadValue {
/// initial: i32, /// initial: i32,
@@ -178,7 +177,7 @@ impl DagRunner {
/// } /// }
/// ///
/// # async { /// # async {
/// let dag = DagRunner::new(); /// let mut dag = DagRunner::new();
/// ///
/// // Construct task with initial value of 10 /// // Construct task with initial value of 10
/// let base = dag.add_task(LoadValue::new(10)); /// let base = dag.add_task(LoadValue::new(10));
@@ -186,53 +185,57 @@ impl DagRunner {
/// // Construct task with offset of 1 /// // Construct task with offset of 1
/// let inc = dag.add_task(AddOffset::new(1)).depends_on(&base); /// let inc = dag.add_task(AddOffset::new(1)).depends_on(&base);
/// ///
/// dag.run(|fut| { tokio::spawn(fut); }).await.unwrap(); ///let mut output = dag.run(|fut| async move { tokio::spawn(fut).await.unwrap() }).await.unwrap();
/// assert_eq!(dag.get(&inc).unwrap(), 11); /// assert_eq!(output.get(inc), 11);
/// # }; /// # };
/// ``` /// ```
pub fn add_task<Tk>(&self, task: Tk) -> TaskBuilder<'_, Tk, Pending> pub fn add_task<'dag, Input, Tk>(&'dag mut self, task: Tk) -> Tk::Retval<'dag>
where where
Tk: Task + 'static, Tk: TaskWire<Input>,
Tk::Input: 'static + Clone + ExtractInput, Input: Send + Sync + 'static,
Tk::Output: 'static + Clone,
{ {
let id = self.alloc_id(); let id = NodeId(self.nodes.len() as u32);
let node = TypedNode::new(id, task);
self.nodes.lock().push(Some(Box::new(node)));
self.edges.lock().insert(id, Vec::new());
self.dependents.lock().insert(id, Vec::new());
TaskBuilder { #[cfg(feature = "tracing")]
id, debug!(
dag: self, task_id = id.0,
_phantom: std::marker::PhantomData, task_type = std::any::type_name::<Tk>(),
} "adding task to DAG"
);
let node = TypedNode::new(task);
self.nodes.push(Some(Box::new(node)));
self.edges.insert(id, Vec::new());
self.dependents.insert(id, Vec::new());
Tk::new_from_dag(id, self)
} }
/// Run the entire DAG to completion using the provided spawner. /// Run the entire DAG to completion using the provided spawner.
/// ///
/// This method:
/// - Executes tasks in topological order (respecting dependencies) /// - Executes tasks in topological order (respecting dependencies)
/// - Runs ready tasks with maximum parallelism (executor-limited) /// - Runs ready tasks with maximum parallelism (executor-limited)
/// - Executes each task at most once /// - Executes each task at most once
/// - Waits for **all sinks** (tasks with no dependents) to complete
/// - Is runtime-agnostic via the spawner function /// - Is runtime-agnostic via the spawner function
/// ///
/// # Parameters /// # Parameters
/// ///
/// - `spawner`: A function that spawns futures on the async runtime. Examples: /// - `spawner`: A function that spawns futures on the async runtime
/// - Tokio: `|fut| { tokio::spawn(fut); }` /// and returns a handle to the task. This is the only way to run tasks on separate threads. Examples:
/// - Smol: `|fut| { smol::spawn(fut).detach(); }` /// - Tokio: `|fut| { tokio::spawn(fut).await.unwrap() }`
/// - Async-std: `|fut| { async_std::task::spawn(fut); }` /// - Smol: `|fut| { smol::spawn(fut) }`
/// - Single-threaded on invoking runtime: `|fut| fut`
/// - Can be faster in situations where waiting time dominates
/// ///
/// # Errors /// # Errors
/// ///
/// Returns `DagError::CycleDetected` if the DAG contains a cycle. /// - Returns `DagError::TaskPanicked` if any task panics during execution
/// ///
/// # Examples /// # Examples
/// ///
/// ``` /// ```
/// # use dagx::{task, DagRunner, Task}; /// # use dagx::{task, DagRunner, Task};
/// #
/// // Tuple struct /// // Tuple struct
/// struct Value(i32); /// struct Value(i32);
/// ///
@@ -250,319 +253,213 @@ impl DagRunner {
/// } /// }
/// ///
/// # async { /// # async {
/// let dag = DagRunner::new(); /// let mut dag = DagRunner::new();
/// ///
/// let a = dag.add_task(Value(1)); /// let a = dag.add_task(Value(1));
/// let b = dag.add_task(Value(2)); /// let b = dag.add_task(Value(2));
/// let sum = dag.add_task(Add).depends_on((&a, &b)); /// let sum = dag.add_task(Add).depends_on((&a, &b));
/// ///
/// dag.run(|fut| { tokio::spawn(fut); }).await.unwrap(); // Executes all tasks ///let mut output = dag.run(|fut| async move { tokio::spawn(fut).await.unwrap() }).await.unwrap(); // Executes all tasks
/// # }; /// # };
/// ``` /// ```
/// #[inline]
/// # Implementation Note #[cfg_attr(feature = "tracing", tracing::instrument(skip(self, spawner)))]
/// pub async fn run<S, F>(mut self, spawner: S) -> DagResult<DagOutput>
/// Tasks communicate via oneshot channels created fresh during run().
/// This eliminates Mutex contention on outputs and enables true streaming execution.
/// Type erasure occurs only at the ExecutableNode trait boundary - channels are created
/// with full type information and type-erased just before passing to execute_with_channels.
pub async fn run<S>(&self, spawner: S) -> DagResult<()>
where where
S: Fn(BoxFuture<'static, ()>), S: Fn(BoxFuture<'static, DagResult<Arc<dyn Any + Send + Sync>>>) -> F,
F: Future<Output = DagResult<Arc<dyn Any + Send + Sync>>>,
{ {
// Acquire run lock to prevent concurrent executions #[cfg(feature = "tracing")]
// Use atomic compare_exchange to check if already running info!("starting DAG execution");
if self
.run_lock
.compare_exchange(false, true, Ordering::SeqCst, Ordering::SeqCst)
.is_err()
{
return Err(DagError::CycleDetected {
nodes: vec![],
description: "DAG is already running - concurrent execution not supported"
.to_string(),
});
}
// Guard ensures lock is released on drop (even on early return or panic)
let _run_guard = RunGuard {
lock: &self.run_lock,
};
// Build topological layers // Build topological layers
let layers = self.compute_layers()?; let layers = self.compute_layers()?;
let edges = self.edges.lock().clone(); let total_tasks = layers.iter().map(|l| l.len()).sum::<usize>();
// Create all channels upfront #[cfg(feature = "tracing")]
// Map: (producer_id, consumer_id) -> receiver index debug!(
// We store receivers in a Vec per consumer to maintain order layer_count = layers.len(),
let mut consumer_receivers: HashMap<NodeId, Vec<Box<dyn std::any::Any + Send>>> = total_tasks, "computed topological layers"
HashMap::new(); );
let mut producer_senders: HashMap<NodeId, Vec<Box<dyn std::any::Any + Send>>> =
HashMap::new();
// Create channels for each producer-consumer relationship let mut outputs: PassThroughHashMap<NodeId, Arc<dyn Any + Send + Sync>> =
// We need to lock nodes temporarily to call create_output_channels HashMap::with_capacity_and_hasher(total_tasks, PassThroughHasher::default());
{ let mut first_error = None;
let nodes_lock = self.nodes.lock();
for (consumer_id, producer_ids) in &edges { // Panic handling is required to maintain behavioral consistency, as
for &producer_id in producer_ids { // different async runtimes (Tokio, async-std, smol, embassy-rs) handle panics in
// Get the producer node (it's still in the vector at this point) // spawned tasks differently.
if let Some(Some(producer_node)) = nodes_lock.get(producer_id.0) { //
// Ask the producer to create ONE channel for this consumer // This ensures tasks behave in the same way whether executed inline or on the spawner
let (mut senders, mut receivers) = producer_node.create_output_channels(1); // across every runtime.
// Store the sender for the producer
producer_senders
.entry(producer_id)
.or_default()
.push(senders.pop().unwrap());
// Store the receiver for the consumer (order matters!)
consumer_receivers
.entry(*consumer_id)
.or_default()
.push(receivers.pop().unwrap());
}
}
}
}
// Create a channel to collect Arc-wrapped outputs from all tasks
let (output_tx, mut output_rx) = mpsc::unbounded::<(
NodeId,
Result<std::sync::Arc<dyn std::any::Any + Send + Sync>, DagError>,
)>();
// Execute layer by layer
for layer in layers { for layer in layers {
// ============================================================================ #[cfg(feature = "tracing")]
// PERFORMANCE OPTIMIZATION: Inline execution for single-task layers {
// ============================================================================ debug!(task_count = layer.len(), "executing layer");
}
// Performance optimization: Inline execution for single-task layers
// //
// When a layer contains exactly one task (common in deep chains, linear // When a layer contains exactly one task (common in deep chains, linear
// pipelines), we execute it inline rather than spawning it. This provides // pipelines), we execute it inline rather than spawning it. This provides
// 10-100x performance improvements for sequential workloads by eliminating: // 10-100x performance improvements for sequential workloads by eliminating:
// - Task spawning overhead // - Task spawning overhead
// - Channel creation/destruction for layer coordination
// - Context switching to/from the runtime // - Context switching to/from the runtime
//
// CRITICAL: Panic handling is required to maintain behavioral consistency.
// All async runtimes (Tokio, async-std, smol, embassy-rs) catch panics in
// spawned tasks and convert them to errors. We must do the same for inline
// execution to ensure tasks behave identically regardless of execution path.
//
// Without panic catching:
// - Spawned task panics → caught by runtime → becomes error (expected)
// - Inline task panics → bubbles up → crashes program (surprising!)
//
// With panic catching (current implementation):
// - Spawned task panics → caught by runtime → becomes error
// - Inline task panics → caught by us → becomes error (consistent!)
//
// This ensures the optimization is transparent to users.
// ============================================================================
if layer.len() == 1 { if layer.len() == 1 {
let node_id = layer[0]; let node_id = layer[0];
let out_tx = output_tx.clone();
#[cfg(feature = "tracing")]
trace!(
task_id = node_id.0,
"executing task inline (single-task layer optimization)"
);
// Take ownership of the node // Take ownership of the node
let node = { let node = self.nodes[node_id.0 as usize].take();
let mut nodes_lock = self.nodes.lock();
nodes_lock[node_id.0].take()
};
if let Some(node) = node { if let Some(node) = node {
// Take the receivers for this consumer let dependencies: Vec<_> = self.edges[&node_id]
let receivers = consumer_receivers.remove(&node_id).unwrap_or_default(); .iter()
.flat_map(|dep| outputs.get(dep))
// Take the senders for this producer .cloned()
let senders = producer_senders.remove(&node_id).unwrap_or_default(); .collect();
// Execute inline with panic handling. // Execute inline with panic handling.
// let result = AssertUnwindSafe(node.execute_with_deps(dependencies))
// FutureExt::catch_unwind() ensures panics are caught and converted to
// DagError::TaskPanicked, matching the behavior of async runtimes when
// they spawn tasks. This guarantees consistent error handling whether
// a task executes inline (single-task layer) or spawned (multi-task layer).
let result = AssertUnwindSafe(node.execute_with_channels(receivers, senders))
.catch_unwind() .catch_unwind()
.await .await
.unwrap_or_else(|panic_payload| { .unwrap_or_else(|panic_payload| {
// Convert panic to error // Convert panic to error
Err(DagError::TaskPanicked { let panic_message =
task_id: node_id.0, if let Some(s) = panic_payload.downcast_ref::<&str>() {
panic_message: if let Some(s) = panic_payload.downcast_ref::<&str>()
{
s.to_string() s.to_string()
} else if let Some(s) = panic_payload.downcast_ref::<String>() { } else if let Some(s) = panic_payload.downcast_ref::<String>() {
s.clone() s.clone()
} else { } else {
"unknown panic".to_string() "unknown panic".to_string()
}, };
#[cfg(feature = "tracing")]
error!(
task_id = node_id.0,
panic_message = %panic_message,
"task panicked during inline execution"
);
Err(DagError::TaskPanicked {
task_id: node_id.0,
panic_message,
}) })
}); });
// Send the output (ignore send errors - receiver may be dropped) match result {
let _ = out_tx.unbounded_send((node_id, result)); Ok(output) => {
outputs.insert(node_id, output);
}
Err(e) => {
first_error.get_or_insert(e);
}
}
} }
} else { } else {
// Slow path: Multiple tasks require spawning and coordination // Slow path: Multiple tasks require spawning and coordination
// Create a channel to track task completion for this layer
let (tx, mut rx) = mpsc::channel::<()>(layer.len());
// Spawn each task in this layer // Spawn each task in this layer
for &node_id in &layer { let mut futures: FuturesUnordered<_> = layer
let mut task_tx = tx.clone(); .into_iter()
let out_tx = output_tx.clone(); .filter_map(|node_id| {
#[cfg(feature = "tracing")]
trace!(task_id = node_id.0, "spawning task");
// Take ownership of the node // Take ownership of the node
let node = { let node = self.nodes[node_id.0 as usize].take();
let mut nodes_lock = self.nodes.lock();
nodes_lock[node_id.0].take()
};
if let Some(node) = node { if let Some(node) = node {
// Take the receivers for this consumer let dependencies: Vec<_> = self.edges[&node_id]
let receivers = consumer_receivers.remove(&node_id).unwrap_or_default(); .iter()
.flat_map(|dep| outputs.get(dep))
.cloned()
.collect();
// Take the senders for this producer let inner_future = spawner(node.execute_with_deps(dependencies));
let senders = producer_senders.remove(&node_id).unwrap_or_default(); // Spawn the task using the provided spawner
// Create a 'static future by taking ownership
let inner_future = async move { let inner_future = async move {
let task_future = node.execute_with_channels(receivers, senders); let result = inner_future.await?;
Ok((node_id, result))
let result = task_future.await;
// Send the output (ignore send errors - receiver may be dropped)
let _ = out_tx.unbounded_send((node_id, result));
// Signal completion (ignore send errors - receiver may be dropped)
let _ = task_tx.try_send(());
}; };
// Spawn the task using the provided spawner Some(
spawner(Box::pin(inner_future)); AssertUnwindSafe(inner_future)
.catch_unwind()
.unwrap_or_else(move |panic_payload| {
// Convert panic to error
let panic_message =
if let Some(s) = panic_payload.downcast_ref::<&str>() {
s.to_string()
} else if let Some(s) =
panic_payload.downcast_ref::<String>()
{
s.clone()
} else {
"unknown panic".to_string()
};
#[cfg(feature = "tracing")]
error!(
task_id = node_id.0,
panic_message = %panic_message,
"task panicked during inline execution"
);
Err(DagError::TaskPanicked {
task_id: node_id.0,
panic_message,
})
}),
)
} else {
None
}
})
.collect();
while let Some(out) = futures.next().await {
match out {
Ok(output) => {
outputs.insert(output.0, output.1);
}
Err(e) => {
first_error.get_or_insert(e);
}
}
} }
} }
// Drop the original sender so the channel closes when all tasks complete // Return first error if any, aborting execution after this layer
drop(tx);
// Wait for all tasks in this layer to complete
while rx.next().await.is_some() {}
}
}
// Drop the output sender so the channel closes
drop(output_tx);
// Collect all outputs and check for errors
let mut collected = Vec::new();
let mut first_error = None;
while let Some((node_id, result)) = output_rx.next().await {
match result {
Ok(output) => collected.push((node_id, output)),
Err(e) if first_error.is_none() => first_error = Some(e),
Err(_) => {} // Ignore subsequent errors
}
}
// Insert all successful outputs (avoiding holding lock across await)
let mut outputs = self.outputs.lock();
for (node_id, output) in collected {
outputs.insert(node_id, output);
}
drop(outputs);
// Return first error if any
if let Some(err) = first_error { if let Some(err) = first_error {
#[cfg(feature = "tracing")]
error!(?err, "DAG execution failed");
return Err(err); return Err(err);
} }
Ok(())
} }
/// Retrieve a task's output after [`DagRunner::run`]. #[cfg(feature = "tracing")]
/// info!("DAG execution completed successfully");
/// # Behavior
///
/// All task outputs are stored after execution and can be retrieved via get().
///
/// # Errors
///
/// Returns `DagError::ResultNotFound` if:
/// - The task hasn't been executed yet
/// - The handle is invalid
///
/// # Examples
///
/// ```no_run
/// # use dagx::{task, DagRunner, Task};
/// struct Configuration {
/// setting: i32,
/// }
///
/// impl Configuration {
/// fn new(setting: i32) -> Self {
/// Self { setting }
/// }
/// }
///
/// #[task]
/// impl Configuration {
/// async fn run(&mut self) -> i32 { self.setting }
/// }
///
/// # async {
/// let dag = DagRunner::new();
///
/// // Construct task with specific setting value
/// let task = dag.add_task(Configuration::new(42));
///
/// dag.run(|fut| { tokio::spawn(fut); }).await.unwrap();
///
/// assert_eq!(dag.get(task).unwrap(), 42);
/// # };
/// ```
pub fn get<T: 'static + Clone + Send + Sync, H>(&self, handle: H) -> DagResult<T>
where
H: Into<TaskHandle<T>>,
{
let handle: TaskHandle<T> = handle.into();
let outputs = self.outputs.lock();
let arc_output = outputs.get(&handle.id).ok_or(DagError::ResultNotFound { Ok(DagOutput::new(outputs))
task_id: handle.id.0,
})?;
// Downcast Arc<dyn Any> to Arc<T>, then clone the inner value
// The Arc itself is stored for efficient sharing, but get() clones the data
Arc::clone(arc_output)
.downcast::<T>()
.map(|arc| (*arc).clone())
.map_err(|_| DagError::TypeMismatch {
expected: std::any::type_name::<T>(),
found: "unknown",
})
} }
fn compute_layers(&self) -> DagResult<Vec<Vec<NodeId>>> { fn compute_layers(&self) -> DagResult<Vec<Vec<NodeId>>> {
let mut in_degree: HashMap<NodeId, usize> = HashMap::new(); #[cfg(feature = "tracing")]
debug!("computing topological layers");
let mut in_degree: PassThroughHashMap<NodeId, usize> = HashMap::default();
let mut layers = Vec::new(); let mut layers = Vec::new();
// Calculate in-degrees: for each node, count how many dependencies it has // Calculate in-degrees: for each node, count how many dependencies it has
let edges = self.edges.lock();
let total_nodes = edges.len();
for (&node, deps) in edges.iter() { for (&node, deps) in self.edges.iter() {
let degree = deps.len(); let degree = deps.len();
in_degree.insert(node, degree); in_degree.insert(node, degree);
} }
drop(edges); // Release lock early
// Find all nodes with in-degree 0 (sources - nodes with no dependencies) // Find all nodes with in-degree 0 (sources - nodes with no dependencies)
let mut queue: VecDeque<NodeId> = in_degree let mut queue: VecDeque<NodeId> = in_degree
@@ -587,8 +484,7 @@ impl DagRunner {
visited.insert(node); visited.insert(node);
// For each node that depends on the current node // For each node that depends on the current node
let dependents = self.dependents.lock(); if let Some(deps) = self.dependents.get(&node) {
if let Some(deps) = dependents.get(&node) {
for &dependent in deps { for &dependent in deps {
if let Some(degree) = in_degree.get_mut(&dependent) { if let Some(degree) = in_degree.get_mut(&dependent) {
*degree -= 1; *degree -= 1;
@@ -603,35 +499,27 @@ impl DagRunner {
} }
if !current_layer.is_empty() { if !current_layer.is_empty() {
current_layer.sort(); // Deterministic ordering
layers.push(current_layer); layers.push(current_layer);
} }
} }
// Cycle detection: if we haven't visited all nodes, there's a cycle // Cycle detection removed: cycles are impossible via the public API.
if visited.len() != total_nodes { //
let unvisited: Vec<_> = in_degree // The type-state pattern enforces acyclic structure at compile time through:
.iter() // 1. TaskBuilder::depends_on() consumes the builder (move semantics)
.filter(|(node, _)| !visited.contains(node)) // 2. TaskHandle has no methods to add dependencies (immutability)
.map(|(node, _)| node.0) // 3. Strict topological ordering requirement for dependency wiring
.collect(); //
let description = format!( // This eliminates the need for runtime cycle detection, providing both
"{} node(s) could not be processed: {:?}", // safety guarantees and performance benefits (no validation overhead).
unvisited.len(), debug_assert!(!visited.is_empty() || layers.is_empty());
unvisited
); #[cfg(feature = "tracing")]
return Err(DagError::CycleDetected { debug!(layer_count = layers.len(), "topological layers computed");
nodes: unvisited,
description,
});
}
Ok(layers) Ok(layers)
} }
} }
// Note: We cannot implement Default for DagRunner anymore since new() returns Arc<Self>.
// Users should call DagRunner::new() directly.
#[cfg(test)] #[cfg(test)]
mod tests; mod tests;
+101 -155
View File
@@ -1,9 +1,29 @@
//! Unit tests for runner module //! Unit tests for runner module
use crate::error::DagError; use std::collections::HashMap;
use crate::runner::DagRunner; use std::hash::Hasher;
use crate::task::Task;
use crate::types::TaskHandle; use crate::builder::TaskHandle;
use crate::runner::{DagRunner, PassThroughHasher};
// Initialize tracing subscriber for tests (idempotent)
#[cfg(feature = "tracing")]
fn init_tracing() {
use std::sync::Once;
static INIT: Once = Once::new();
INIT.call_once(|| {
tracing_subscriber::fmt()
.with_test_writer()
.with_max_level(tracing::Level::TRACE)
.try_init()
.ok();
});
}
#[cfg(not(feature = "tracing"))]
fn init_tracing() {
// No-op when tracing is disabled
}
struct TestTask { struct TestTask {
value: i32, value: i32,
@@ -19,16 +39,16 @@ impl TestTask {
#[test] #[test]
fn test_dag_runner_new() { fn test_dag_runner_new() {
// Test that DagRunner::new() creates a new instance // Test that DagRunner::new() creates a new instance
let dag = DagRunner::new(); let mut dag = DagRunner::new();
// Initial state should be empty // Initial state should be empty
assert_eq!(dag.nodes.lock().len(), 0); assert_eq!(dag.nodes.len(), 0);
assert_eq!(dag.edges.lock().len(), 0); assert_eq!(dag.edges.len(), 0);
assert_eq!(dag.dependents.lock().len(), 0); assert_eq!(dag.dependents.len(), 0);
// Adding a task works // Adding a task works
dag.add_task(TestTask { value: 42 }); dag.add_task(TestTask { value: 42 });
assert_eq!(dag.nodes.lock().len(), 1); assert_eq!(dag.nodes.len(), 1);
} }
#[test] #[test]
@@ -37,20 +57,21 @@ fn test_dag_runner_default() {
let dag = DagRunner::default(); let dag = DagRunner::default();
// Should behave the same as new() // Should behave the same as new()
assert_eq!(dag.nodes.lock().len(), 0); assert_eq!(dag.nodes.len(), 0);
assert_eq!(dag.edges.lock().len(), 0); assert_eq!(dag.edges.len(), 0);
assert_eq!(dag.dependents.lock().len(), 0); assert_eq!(dag.dependents.len(), 0);
} }
#[tokio::test] #[tokio::test]
#[should_panic]
async fn test_get_wrong_type() { async fn test_get_wrong_type() {
// Test getting with wrong type - this should return TypeMismatch init_tracing();
let dag = DagRunner::new(); // Test getting with wrong type - this should panic
let mut dag = DagRunner::new();
let handle = dag.add_task(TestTask { value: 42 }); let handle = dag.add_task(TestTask { value: 42 });
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await .await
.unwrap(); .unwrap();
@@ -60,159 +81,84 @@ async fn test_get_wrong_type() {
_phantom: std::marker::PhantomData, _phantom: std::marker::PhantomData,
}; };
let result = dag.get(fake_handle); let _result = output.get(fake_handle);
assert!(result.is_err());
// When downcast fails, we get TypeMismatch
match result.unwrap_err() {
DagError::TypeMismatch { expected, .. } => {
assert_eq!(expected, std::any::type_name::<String>());
}
_ => panic!("Expected TypeMismatch error"),
}
}
#[tokio::test]
async fn test_get_result_not_found() {
// Test getting a result before running the DAG
let dag = DagRunner::new();
let handle = dag.add_task(TestTask { value: 42 });
let handle_id = handle.id.0; // Save ID before moving handle
let result = dag.get(handle);
assert!(result.is_err());
match result.unwrap_err() {
DagError::ResultNotFound { task_id } => {
assert_eq!(task_id, handle_id);
}
_ => panic!("Expected ResultNotFound error"),
}
}
#[tokio::test]
async fn test_concurrent_run_protection() {
// Test that run_lock prevents concurrent runs
use std::sync::Arc;
use std::time::Duration;
use tokio::time::sleep;
let dag = DagRunner::new();
// Add a task that takes some time
struct SlowTask;
#[crate::task]
impl SlowTask {
async fn run(&self) -> i32 {
sleep(Duration::from_millis(100)).await;
42
}
}
dag.add_task(SlowTask);
// Wrap in Arc for sharing between tasks
let dag = Arc::new(dag);
let dag1 = Arc::clone(&dag);
let dag2 = Arc::clone(&dag);
// Start two runs concurrently
let handle1 = tokio::spawn(async move {
dag1.run(|fut| {
tokio::spawn(fut);
})
.await
});
let handle2 = tokio::spawn(async move {
// Give the first run a head start
sleep(Duration::from_millis(10)).await;
dag2.run(|fut| {
tokio::spawn(fut);
})
.await
});
// First should succeed, second should fail (concurrent execution not supported)
let result1 = handle1.await.unwrap();
let result2 = handle2.await.unwrap();
assert!(result1.is_ok());
assert!(result2.is_err()); // Concurrent run() returns error
}
#[test]
fn test_add_task_increments_id() {
let dag = DagRunner::new();
let handle1 = dag.add_task(TestTask { value: 1 });
let handle2 = dag.add_task(TestTask { value: 2 });
let handle3 = dag.add_task(TestTask { value: 3 });
// Node IDs should be sequential
assert_eq!(handle1.id.0, 0);
assert_eq!(handle2.id.0, 1);
assert_eq!(handle3.id.0, 2);
// Check that nodes were actually added
assert_eq!(dag.nodes.lock().len(), 3);
}
#[tokio::test]
async fn test_multiple_get_calls() {
// Test that get() can be called multiple times for the same handle
let dag = DagRunner::new();
let handle = dag.add_task(TestTask { value: 100 });
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
// Multiple get calls should all return the same value
let result1 = dag.get(&handle).unwrap();
let result2 = dag.get(&handle).unwrap();
let result3 = dag.get(&handle).unwrap();
assert_eq!(result1, 100);
assert_eq!(result2, 100);
assert_eq!(result3, 100);
}
struct TaskWithDep;
#[crate::task]
impl TaskWithDep {
#[allow(dead_code)]
async fn run(input: &i32) -> i32 {
input * 2
}
} }
#[tokio::test] #[tokio::test]
#[should_panic]
async fn test_invalid_node_id_in_get() { async fn test_invalid_node_id_in_get() {
init_tracing();
// Test getting with an invalid node ID // Test getting with an invalid node ID
let dag = DagRunner::new(); let dag = DagRunner::new();
// Create a handle with an ID that doesn't exist // Create a handle with an ID that doesn't exist
let invalid_handle: TaskHandle<i32> = TaskHandle { let invalid_handle: TaskHandle<i32> = TaskHandle {
id: crate::types::NodeId(999), id: crate::builder::NodeId(999),
_phantom: std::marker::PhantomData, _phantom: std::marker::PhantomData,
}; };
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await .await
.unwrap(); .unwrap();
let result = dag.get(invalid_handle); let _result = output.get(invalid_handle);
assert!(result.is_err()); }
// With centralized output storage, we get ResultNotFound for invalid node IDs #[tokio::test]
match result.unwrap_err() { async fn test_task_panic_in_multi_task_layer() {
DagError::ResultNotFound { task_id } => { init_tracing();
assert_eq!(task_id, 999); // Test that a panic in one task of a multi-task layer is caught and reported
}
_ => panic!("Expected ResultNotFound error"), struct Source;
#[crate::task]
impl Source {
async fn run(&self) -> i32 {
42
} }
} }
struct PanicTask;
#[crate::task]
impl PanicTask {
async fn run(&self, _input: &i32) -> i32 {
panic!("This task panics!");
}
}
struct GoodTask;
#[crate::task]
impl GoodTask {
async fn run(&self, input: &i32) -> i32 {
input * 2
}
}
let mut dag = DagRunner::new();
let source = dag.add_task(Source);
// Create two tasks in the same layer - one panics, one doesn't
let _panic_task = dag.add_task(PanicTask).depends_on(&source);
let _good_task = dag.add_task(GoodTask).depends_on(&source);
// Run the DAG - the panic should be caught
let result = dag
.run(|fut| async move { tokio::spawn(fut).await.unwrap() })
.await;
// The run should fail due to the panic
assert!(result.is_err());
}
#[test]
fn test_passthrough_hasher() {
let mut passthrough_hashmap = HashMap::with_hasher(PassThroughHasher::default());
passthrough_hashmap.insert(64u32, "test string");
}
#[test]
#[should_panic]
fn test_passthrough_hasher_only_u32() {
let mut passthrough_hasher = PassThroughHasher::default();
passthrough_hasher.write(&[0; 4]);
}
+53 -105
View File
@@ -3,72 +3,18 @@
//! This module defines the core Task trait that all DAG nodes must implement, //! This module defines the core Task trait that all DAG nodes must implement,
//! along with convenience functions for creating tasks from closures. //! along with convenience functions for creating tasks from closures.
use std::future::Future; use std::{any::Any, future::Future, marker::PhantomData, slice::Iter, sync::Arc};
/// A unit of async work with typed inputs and outputs. /// A unit of async work with typed inputs and outputs.
/// ///
/// Use the [`task`](crate::task) macro to implement this trait. The macro automatically /// Use the [`crate::task`] macro to implement this trait. The macro automatically
/// derives `Input` and `Output` types from your `run()` method signature. /// derives `Input` and `Output` types from your `run()` method signature and generates
/// extraction logic in `run()`.
/// ///
/// # Task Patterns /// **Custom types work automatically!** As long as your output type implements
/// `Send + Sync + 'static`, the macro handles everything.
/// ///
/// dagx supports three main task patterns: /// **Note:** When implementing this trait manually for a task with a single dependency, the Input type must be (T,) instead of T.
///
/// ## 1. Stateless Tasks (Pure Functions)
///
/// Tasks with no internal state - just transform inputs to outputs:
///
/// ```
/// use dagx::{task, Task};
///
/// struct Add;
///
/// #[task]
/// impl Add {
/// async fn run(a: &i32, b: &i32) -> i32 {
/// a + b
/// }
/// }
/// ```
///
/// ## 2. Stateful Tasks with Shared State (`&self`)
///
/// Tasks that read from internal state but don't modify it:
///
/// ```
/// use dagx::{task, Task};
///
/// struct Multiplier {
/// factor: i32,
/// }
///
/// #[task]
/// impl Multiplier {
/// async fn run(&self, input: &i32) -> i32 {
/// input * self.factor
/// }
/// }
/// ```
///
/// ## 3. Stateful Tasks with Mutable State (`&mut self`)
///
/// Tasks that modify internal state during execution:
///
/// ```
/// use dagx::{task, Task};
///
/// struct Counter {
/// count: i32,
/// }
///
/// #[task]
/// impl Counter {
/// async fn run(&mut self, increment: &i32) -> i32 {
/// self.count += increment;
/// self.count
/// }
/// }
/// ```
/// ///
/// # Input Patterns /// # Input Patterns
/// ///
@@ -76,68 +22,70 @@ use std::future::Future;
/// - `&T`: Single dependency /// - `&T`: Single dependency
/// - `&A, &B, &C, ...`: Multiple dependencies (up to 8, order matters) /// - `&A, &B, &C, ...`: Multiple dependencies (up to 8, order matters)
/// ///
/// # The Underlying Trait
///
/// The `#[task]` macro generates an implementation of this trait:
///
/// # Output Wrapping /// # Output Wrapping
/// ///
/// **Task outputs are automatically wrapped in `Arc<T>` internally** for efficient fan-out. /// **Task outputs are automatically wrapped in `Arc<T>` internally** for efficient fan-out.
/// You define `Output = T`, but the framework wraps it in `Arc<T>` before sending to channels. /// You define `Output = T`, but the framework wraps it in `Arc<T>` before sending to dependents.
/// This enables O(1) sharing for 1:N dependencies - Arc is cloned (cheap), not your data. /// This enables O(1) sharing for 1:N dependencies - Arc is cloned (cheap), not your data.
/// ///
/// Downstream tasks receive `&T` after ExtractInput extracts from the Arc. This is transparent /// Downstream tasks receive `&T` after ExtractInput extracts from the Arc. This is transparent
/// to your task code - you work with `T`, the framework handles Arc wrapping/unwrapping. /// to your task code - you work with `T`, the framework handles Arc wrapping/unwrapping.
pub trait Task: Send { pub trait Task<Input>: Send
type Input: Send; where
type Output: Send + Sync; // Sync required for Arc-wrapping and cross-thread sharing Input: Send + Sync + 'static,
{
type Output: Send + Sync + 'static; // Sync required for Arc-wrapping and cross-thread sharing
fn run(self, input: Self::Input) -> impl Future<Output = Self::Output> + Send; fn run(self, input: TaskInput<Input>) -> impl Future<Output = Self::Output> + Send;
} }
/// Convenience function to create a task from a closure. /// Linear type providing a [`Task`] access to its dependencies.
/// ///
/// **INTERNAL USE ONLY**: This function is for testing purposes only and should /// The framework ensures that all dependencies will be present when this task runs, and
/// not be used in production code. Use the `#[task]` macro instead. /// can be retrieved with [`TaskInput::next`]. The nth call to `next` consumes this TaskInput instances and returns a
#[doc(hidden)] /// (&T, TaskInput) tuple with the nth dependency specified in the call to [`crate::TaskBuilder::depends_on`] and a
pub fn task_fn<I, O, Fut, F>(f: F) -> impl Task<Input = I, Output = O> /// TaskInput with the remaining dependencies.
where ///
F: FnMut(I) -> Fut + Send + 'static, /// No dependencies can be retrieved from an empty TaskInput (Input = ()).
Fut: Future<Output = O> + Send + 'static, pub struct TaskInput<'inputs, Input: Send> {
I: Send + 'static, inputs: Iter<'inputs, Arc<dyn Any + Send + Sync + 'static>>,
O: Send + Sync + 'static, phantom: PhantomData<Input>,
{
struct TaskFn<I, O, Fut, F>
where
F: FnMut(I) -> Fut + Send,
Fut: Future<Output = O> + Send,
I: Send,
O: Send,
{
f: F,
_phantom: std::marker::PhantomData<fn(I) -> O>,
} }
impl<I, O, Fut, F> Task for TaskFn<I, O, Fut, F> impl<'inputs, Input: Send> TaskInput<'inputs, Input> {
where pub(crate) fn new(inputs: Iter<'inputs, Arc<dyn Any + Send + Sync + 'static>>) -> Self {
F: FnMut(I) -> Fut + Send, Self {
Fut: Future<Output = O> + Send, inputs,
I: Send, phantom: PhantomData,
O: Send + Sync, }
{
type Input = I;
type Output = O;
async fn run(mut self, input: I) -> O {
(self.f)(input).await
} }
} }
TaskFn { macro_rules! impl_task_input {
f, ($First:ident $(, $T:ident)*) => {
_phantom: std::marker::PhantomData, impl<'inputs, $First: 'static + Send, $($T: 'static + Send),*> TaskInput<'inputs, ($First, $($T,)*)> {
/// Get a reference to the next dependency from this TaskInput, and a [`TaskInput`] instance with the remaining dependencies.
#[must_use]
pub fn next(mut self) -> (&'inputs $First, TaskInput<'inputs, ($($T,)*)>) {
let value = self.inputs.next().and_then(|value| value.downcast_ref()).unwrap();
let next_inputs = TaskInput {
inputs: self.inputs,
phantom: PhantomData,
};
(value, next_inputs)
} }
} }
};
}
impl_task_input!(A);
impl_task_input!(A, B);
impl_task_input!(A, B, C);
impl_task_input!(A, B, C, D);
impl_task_input!(A, B, C, D, E);
impl_task_input!(A, B, C, D, E, F);
impl_task_input!(A, B, C, D, E, F, G);
impl_task_input!(A, B, C, D, E, F, G, H);
#[cfg(test)] #[cfg(test)]
mod tests; mod tests;
+11 -132
View File
@@ -1,87 +1,20 @@
//! Unit tests for task module //! Unit tests for task module
use crate::task::{task_fn, Task}; use crate::task::{Task, TaskInput};
use std::future::Future; use std::any::Any;
use std::pin::Pin; use std::sync::Arc;
// Test the task_fn function and TaskFn struct
#[tokio::test]
async fn test_task_fn_creation_and_execution() {
// Test line 111 in task.rs - Task implementation for TaskFn
// Create a simple task function
let task = task_fn(|input: i32| async move { input * 2 });
// Test the Task trait implementation
let result = task.run(42).await;
assert_eq!(result, 84);
}
#[tokio::test]
async fn test_task_fn_with_unit_input() {
// Test task_fn with unit input
let task = task_fn(|_: ()| async { 100 });
let result = task.run(()).await;
assert_eq!(result, 100);
}
#[tokio::test]
async fn test_task_fn_with_tuple_input() {
// Test task_fn with tuple input
let task = task_fn(|(a, b): (i32, i32)| async move { a + b });
let result = task.run((10, 20)).await;
assert_eq!(result, 30);
}
#[tokio::test]
async fn test_task_fn_with_string_input() {
// Test task_fn with String input
let task = task_fn(|input: String| {
let len = input.len();
async move { len }
});
let test_string = String::from("hello");
let result = task.run(test_string).await;
assert_eq!(result, 5);
}
#[tokio::test]
async fn test_task_fn_with_complex_output() {
// Test task_fn with complex output type
let task = task_fn(|input: i32| async move { vec![input, input * 2, input * 3] });
let result = task.run(5).await;
assert_eq!(result, vec![5, 10, 15]);
}
#[tokio::test]
async fn test_task_fn_stateful_closure() {
// Test that task_fn can capture state
let multiplier = 3;
let task = task_fn(move |input: i32| async move { input * multiplier });
let result = task.run(7).await;
assert_eq!(result, 21);
}
// Test custom Task implementation // Test custom Task implementation
struct CustomTask { struct CustomTask {
state: i32, state: i32,
} }
impl Task for CustomTask { impl Task<(i32,)> for CustomTask {
type Input = i32;
type Output = i32; type Output = i32;
#[allow(refining_impl_trait)] async fn run(mut self, input: TaskInput<'_, (i32,)>) -> i32 {
fn run(mut self, input: Self::Input) -> Pin<Box<dyn Future<Output = Self::Output> + Send>> { self.state += input.next().0;
Box::pin(async move {
self.state += input;
self.state self.state
})
} }
} }
@@ -89,67 +22,13 @@ impl Task for CustomTask {
async fn test_custom_task_implementation() { async fn test_custom_task_implementation() {
let task = CustomTask { state: 10 }; let task = CustomTask { state: 10 };
let result1 = task.run(5).await; let result1 = task
.run(TaskInput::new(
[Arc::new(5) as Arc<dyn Any + Send + Sync + 'static>].iter(),
))
.await;
assert_eq!(result1, 15); assert_eq!(result1, 15);
// Can't call run again since task was consumed // Can't call run again since task was consumed
// This demonstrates the ownership model // This demonstrates the ownership model
} }
// Test that Task trait bounds are correct
fn assert_task_bounds<T: Task>() {
// This function just checks that T implements Task with the right bounds
fn assert_send<T: Send>() {}
#[allow(dead_code)]
fn assert_send_future<F: Future + Send>(_: &F) {}
// These should compile if bounds are correct
assert_send::<T>();
assert_send::<T::Input>();
assert_send::<T::Output>();
}
#[test]
fn test_task_trait_bounds() {
// Verify that our test tasks meet the Task trait bounds
assert_task_bounds::<CustomTask>();
// We can't directly test TaskFn because it's a private type and needs a concrete closure type,
// but the fact that task_fn works in the tests above proves it's correct
}
#[tokio::test]
async fn test_task_fn_single_call() {
// Test that task_fn works (can only call once due to ownership)
let task = task_fn(|x: i32| async move { x + 1 });
assert_eq!(task.run(1).await, 2);
}
#[tokio::test]
async fn test_task_fn_with_result_type() {
// Test task_fn returning Result
let task = task_fn(|x: i32| async move {
if x > 0 {
Ok(x * 2)
} else {
Err("negative input")
}
});
assert_eq!(task.run(5).await, Ok(10));
}
#[tokio::test]
async fn test_task_fn_with_option_type() {
// Test task_fn returning Option
let task = task_fn(|x: i32| async move {
if x > 0 {
Some(x * 2)
} else {
None
}
});
assert_eq!(task.run(5).await, Some(10));
}
-97
View File
@@ -1,97 +0,0 @@
//! Core type definitions for DAG nodes and handles.
//!
//! This module defines the fundamental types used throughout the DAG system,
//! including node identifiers, task handles, and type-state markers.
use crate::builder::{IsUnitType, TaskBuilder};
use crate::task::Task;
/// Opaque node identifier (internal use)
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
pub(crate) struct NodeId(pub usize);
/// Opaque, typed token for a node's output.
///
/// A `TaskHandle<T>` provides compile-time type-safe access to a task's output.
/// You can:
/// 1. Pass it to [`crate::TaskBuilder::depends_on`] to wire up dependencies
/// 2. Use it with [`crate::DagRunner::get`] to retrieve the output after [`crate::DagRunner::run`]
///
/// Handles are cheap to clone and copy.
///
/// # Examples
///
/// ```no_run
/// # use dagx::{task, DagRunner, Task};
/// # struct LoadValue { value: i32 }
/// # impl LoadValue { pub fn new(v: i32) -> Self { Self { value: v } } }
/// # #[task]
/// # impl LoadValue {
/// # async fn run(&mut self) -> i32 { self.value }
/// # }
/// # async {
/// let dag = DagRunner::new();
/// let node = dag.add_task(LoadValue::new(42));
///
/// dag.run(|fut| { tokio::spawn(fut); }).await.unwrap();
///
/// assert_eq!(dag.get(node).unwrap(), 42);
/// # };
/// ```
pub struct TaskHandle<T> {
pub(crate) id: NodeId,
pub(crate) _phantom: std::marker::PhantomData<fn() -> T>,
}
impl<T> Clone for TaskHandle<T> {
fn clone(&self) -> Self {
*self
}
}
impl<T> Copy for TaskHandle<T> {}
/// Type-level marker for empty dependency list.
///
/// This type appears in the type signature of [`crate::TaskBuilder<Tk, Pending>`](crate::TaskBuilder) before
/// dependencies are specified with [`crate::TaskBuilder::depends_on`].
///
/// You don't interact with this type directly; it's part of the type-state
/// pattern that ensures dependencies are wired correctly at compile time.
pub struct Pending;
// TaskBuilder can be converted to a TaskHandle ONLY if Input is () (unit type)
// This enforces at compile time that tasks with non-unit inputs MUST call .depends_on()
impl<'a, Tk: Task, Deps> From<TaskBuilder<'a, Tk, Deps>> for TaskHandle<Tk::Output>
where
Tk::Input: IsUnitType,
{
fn from(node: TaskBuilder<'a, Tk, Deps>) -> Self {
TaskHandle {
id: node.id,
_phantom: std::marker::PhantomData,
}
}
}
impl<'a, Tk: Task, Deps> From<&TaskBuilder<'a, Tk, Deps>> for TaskHandle<Tk::Output>
where
Tk::Input: IsUnitType,
{
fn from(node: &TaskBuilder<'a, Tk, Deps>) -> Self {
TaskHandle {
id: node.id,
_phantom: std::marker::PhantomData,
}
}
}
// TaskHandle can be converted from &TaskHandle (for .get() calls)
impl<T> From<&TaskHandle<T>> for TaskHandle<T> {
fn from(handle: &TaskHandle<T>) -> Self {
*handle
}
}
#[cfg(test)]
mod tests;
-98
View File
@@ -1,98 +0,0 @@
//! Unit tests for types module
use crate::types::{NodeId, TaskHandle};
use std::marker::PhantomData;
#[test]
fn test_node_id_operations() {
let id1 = NodeId(1);
let id2 = NodeId(2);
let id3 = NodeId(1);
// Test equality
assert_eq!(id1, id3);
assert_ne!(id1, id2);
// Test ordering
assert!(id1 < id2);
assert!(id2 > id1);
// Test Debug
let debug_str = format!("{:?}", id1);
assert!(debug_str.contains("NodeId"));
assert!(debug_str.contains("1"));
// Test Clone
let cloned = id1;
assert_eq!(id1, cloned);
// Test Copy
let copied = id1;
assert_eq!(id1, copied);
// Test Hash (compile-time check that it implements Hash)
use std::collections::HashSet;
let mut set = HashSet::new();
set.insert(id1);
set.insert(id2);
assert_eq!(set.len(), 2);
}
#[test]
fn test_task_handle_clone() {
let handle: TaskHandle<i32> = TaskHandle {
id: NodeId(42),
_phantom: PhantomData,
};
// Test Clone implementation (lines 44-46 in types.rs)
let cloned = handle;
assert_eq!(handle.id.0, cloned.id.0);
// Test that clone is actually a copy (same memory semantics)
let handle2 = handle;
let handle3 = handle2;
assert_eq!(handle3.id.0, 42);
}
#[test]
fn test_task_handle_copy() {
let handle: TaskHandle<String> = TaskHandle {
id: NodeId(99),
_phantom: PhantomData,
};
// Test that TaskHandle implements Copy
let copied = handle;
let copied2 = handle; // This wouldn't compile if not Copy
assert_eq!(copied.id.0, 99);
assert_eq!(copied2.id.0, 99);
}
#[test]
fn test_task_handle_with_different_types() {
// Test that TaskHandle works with various types
let int_handle: TaskHandle<i32> = TaskHandle {
id: NodeId(1),
_phantom: PhantomData,
};
let string_handle: TaskHandle<String> = TaskHandle {
id: NodeId(2),
_phantom: PhantomData,
};
let vec_handle: TaskHandle<Vec<u8>> = TaskHandle {
id: NodeId(3),
_phantom: PhantomData,
};
// Clone and copy should work for all
let _ = int_handle;
let _ = string_handle;
let _ = vec_handle;
let _ = int_handle;
let _ = string_handle;
let _ = vec_handle;
}
+13
View File
@@ -0,0 +1,13 @@
[test_config]
manifest-path = "./Cargo.toml"
skip-clean = true
engine = "Llvm"
# cargo-tarpaulin already reports tracing lines incorrectly when the feature is turned off,
# and this is exacerbated when building with all-features.
all-features = false
run-types = ["Tests"]
command = "Test"
locked = true
frozen = true
fail-under = 80.0
+6 -24
View File
@@ -4,22 +4,6 @@
Integration and performance tests for dagx. Unit tests live in `src/{module}/tests.rs`. Integration and performance tests for dagx. Unit tests live in `src/{module}/tests.rs`.
## Directory Structure
```
tests/
├── common/ # Shared test utilities
├── dependencies/ # Dependency resolution tests
├── execution/ # Task execution and DAG running
├── parallelism/ # Parallel execution verification
├── panic/ # Panic handling and isolation
├── performance/ # Stress tests and benchmarks
├── runtimes/ # Runtime compatibility tests
├── task_patterns/ # Common task pattern tests
├── type_safety/ # Type system validation
└── lib_tests.rs # Main test entry point
```
## Running Tests ## Running Tests
```bash ```bash
@@ -56,19 +40,17 @@ cargo test --release -- --ignored
### Integration Tests (`tests/`) ### Integration Tests (`tests/`)
- **execution**: End-to-end DAG execution - **boundaries**: Edge cases (very large/very small graphs)
- **dependencies**: Complex dependency graphs - **dependencies**: Complex dependency graphs
- **errors**: Error handling
- **execution**: End-to-end DAG execution
- **interleaving**: Layer interleaving patterns
- **parallelism**: Concurrent task execution - **parallelism**: Concurrent task execution
- **panic**: Error propagation and isolation - **runtimes**: Tokio, smol compatibility
- **performance**: Large DAGs and stress tests - **tracing**: Execution observability
- **runtimes**: Tokio, async-std, smol compatibility
- **task_patterns**: Common usage patterns
- **type_safety**: Type system guarantees
## Writing Tests ## Writing Tests
Integration tests use common utilities from `tests/common/mod.rs`.
Unit tests should: Unit tests should:
- Test single units of functionality - Test single units of functionality
-1377
View File
File diff suppressed because it is too large. Load diff
+54 -173
View File
@@ -1,88 +1,26 @@
//! Tests for extremely deep dependency chains //! Tests for extremely deep dependency chains
use crate::common::task_fn;
use dagx::{DagResult, DagRunner}; use dagx::{DagResult, DagRunner};
use dagx_test::task_fn;
use std::sync::atomic::{AtomicUsize, Ordering}; use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::Arc; use std::sync::Arc;
#[tokio::test]
async fn test_1000_level_deep_chain() -> DagResult<()> {
// Test a linear chain of 1000 tasks
let dag = DagRunner::new();
let first = dag.add_task(task_fn(|_: ()| async { 0 }));
let mut current = (&first).into(); // Convert to TaskHandle
for _i in 1..1000 {
current = dag
.add_task(task_fn(move |prev: i32| async move { prev + 1 }))
.depends_on(current);
}
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
assert_eq!(dag.get(current)?, 999);
Ok(())
}
#[tokio::test]
async fn test_deep_chain_with_branches() -> DagResult<()> {
// Test a deep chain where each node has a side branch
let dag = DagRunner::new();
let first = dag.add_task(task_fn(|_: ()| async { 0 }));
let mut main_chain: dagx::TaskHandle<i32> = (&first).into();
let mut branches = Vec::new();
for i in 1..500 {
// Main chain continues
main_chain = dag
.add_task(task_fn(move |prev: i32| async move { prev + 1 }))
.depends_on(main_chain);
// Create a branch at every 10th node
if i % 10 == 0 {
let branch = dag
.add_task(task_fn(move |main_val: i32| async move { main_val * 2 }))
.depends_on(main_chain);
branches.push(branch);
}
}
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
assert_eq!(dag.get(main_chain)?, 499);
// Check some branches
if !branches.is_empty() {
assert_eq!(dag.get(branches[0])?, 20); // 10 * 2
assert_eq!(dag.get(branches[4])?, 100); // 50 * 2
}
Ok(())
}
#[tokio::test] #[tokio::test]
async fn test_multiple_parallel_deep_chains() -> DagResult<()> { async fn test_multiple_parallel_deep_chains() -> DagResult<()> {
// Test 10 parallel chains of depth 500 each // Test 10 parallel chains of depth 500 each
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let mut chain_ends = Vec::new(); let mut chain_ends = Vec::new();
for chain_id in 0..8 { for chain_id in 0..8 {
let first = dag.add_task(task_fn(move |_: ()| async move { chain_id * 1000 })); let first = dag.add_task(task_fn::<(), _, _>(move |_: ()| chain_id * 1000));
let mut current: dagx::TaskHandle<i32> = (&first).into(); let mut current: dagx::TaskHandle<i32> = first;
for _ in 0..500 { for _ in 0..500 {
current = dag current = dag
.add_task(task_fn(|prev: i32| async move { prev + 1 })) .add_task(task_fn::<i32, _, _>(|&prev: &i32| prev + 1))
.depends_on(current); .depends_on(&current);
} }
chain_ends.push(current); chain_ends.push(current);
@@ -90,8 +28,8 @@ async fn test_multiple_parallel_deep_chains() -> DagResult<()> {
// Add a final task that depends on all chains // Add a final task that depends on all chains
let final_task = dag let final_task = dag
.add_task(task_fn( .add_task(task_fn::<(i32, i32, i32, i32, i32, i32, i32, i32), _, _>(
|inputs: (i32, i32, i32, i32, i32, i32, i32, i32)| async move { |inputs: (&i32, &i32, &i32, &i32, &i32, &i32, &i32, &i32)| {
inputs.0 inputs.0
+ inputs.1 + inputs.1
+ inputs.2 + inputs.2
@@ -113,9 +51,8 @@ async fn test_multiple_parallel_deep_chains() -> DagResult<()> {
&chain_ends[7], &chain_ends[7],
)); ));
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await?; .await?;
// Each chain: start + 500 // Each chain: start + 500
@@ -123,7 +60,7 @@ async fn test_multiple_parallel_deep_chains() -> DagResult<()> {
// Chain 1: 1000 + 500 = 1500 // Chain 1: 1000 + 500 = 1500
// ... // ...
// Sum = 500*8 + (0+1000+2000+...+7000) = 4000 + 28000 = 32000 // Sum = 500*8 + (0+1000+2000+...+7000) = 4000 + 28000 = 32000
assert_eq!(dag.get(final_task)?, 32000); assert_eq!(output.get(final_task), 32000);
Ok(()) Ok(())
} }
@@ -131,43 +68,38 @@ async fn test_multiple_parallel_deep_chains() -> DagResult<()> {
#[tokio::test] #[tokio::test]
async fn test_deep_chain_execution_order() -> DagResult<()> { async fn test_deep_chain_execution_order() -> DagResult<()> {
// Verify tasks in a deep chain execute in correct order // Verify tasks in a deep chain execute in correct order
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let order = Arc::new(AtomicUsize::new(0)); let order = Arc::new(AtomicUsize::new(0));
let first = dag.add_task(task_fn({ let first = dag.add_task(task_fn::<(), _, _>({
let order = order.clone(); let order = order.clone();
move |_: ()| { move |_: ()| {
let order = order.clone(); let order = order.clone();
async move {
assert_eq!(order.fetch_add(1, Ordering::SeqCst), 0); assert_eq!(order.fetch_add(1, Ordering::SeqCst), 0);
0 0
} }
}
})); }));
let mut current: dagx::TaskHandle<i32> = (&first).into(); let mut current: dagx::TaskHandle<i32> = first;
for expected in 1..100 { for expected in 1..100 {
current = dag current = dag
.add_task(task_fn({ .add_task(task_fn::<i32, _, _>({
let order = order.clone(); let order = order.clone();
move |prev: i32| { move |&prev: &i32| {
let order = order.clone(); let order = order.clone();
async move {
assert_eq!(order.fetch_add(1, Ordering::SeqCst), expected); assert_eq!(order.fetch_add(1, Ordering::SeqCst), expected);
prev + 1 prev + 1
} }
}
})) }))
.depends_on(current); .depends_on(&current);
} }
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await?; .await?;
assert_eq!(order.load(Ordering::SeqCst), 100); assert_eq!(order.load(Ordering::SeqCst), 100);
assert_eq!(dag.get(current)?, 99); assert_eq!(output.get(current), 99);
Ok(()) Ok(())
} }
@@ -175,31 +107,32 @@ async fn test_deep_chain_execution_order() -> DagResult<()> {
#[tokio::test] #[tokio::test]
async fn test_deep_binary_tree() -> DagResult<()> { async fn test_deep_binary_tree() -> DagResult<()> {
// Test a deep binary tree structure (depth 10 = 1023 nodes) // Test a deep binary tree structure (depth 10 = 1023 nodes)
let dag = DagRunner::new(); let mut dag = DagRunner::new();
fn build_tree(dag: &DagRunner, depth: usize, value: i32) -> dagx::TaskHandle<i32> { fn build_tree(dag: &mut DagRunner, depth: usize, value: i32) -> dagx::TaskHandle<i32> {
if depth == 0 { if depth == 0 {
let task = dag.add_task(task_fn(move |_: ()| async move { value })); let task = dag.add_task(task_fn::<(), _, _>(move |_: ()| value));
(&task).into() // Convert TaskBuilder to TaskHandle task // Convert TaskBuilder to TaskHandle
} else { } else {
let left = build_tree(dag, depth - 1, value * 2); let left = build_tree(dag, depth - 1, value * 2);
let right = build_tree(dag, depth - 1, value * 2 + 1); let right = build_tree(dag, depth - 1, value * 2 + 1);
dag.add_task(task_fn(move |(l, r): (i32, i32)| async move { l + r })) dag.add_task(task_fn::<(i32, i32), _, _>(move |(l, r): (&i32, &i32)| {
l + r
}))
.depends_on((&left, &right)) .depends_on((&left, &right))
} }
} }
let root = build_tree(&dag, 10, 1); let root = build_tree(&mut dag, 10, 1);
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await?; .await?;
// Sum of all values in a complete binary tree with values 1024..2047 // Sum of all values in a complete binary tree with values 1024..2047
// This is sum from 2^10 to 2^11-1 = 1024 * 1023 / 2 + 1024 * 512 = 1047552 // This is sum from 2^10 to 2^11-1 = 1024 * 1023 / 2 + 1024 * 512 = 1047552
let result = dag.get(root)?; let result = output.get(root);
assert!(result > 0); assert!(result > 0);
Ok(()) Ok(())
@@ -208,56 +141,29 @@ async fn test_deep_binary_tree() -> DagResult<()> {
#[tokio::test] #[tokio::test]
async fn test_fibonacci_chain() -> DagResult<()> { async fn test_fibonacci_chain() -> DagResult<()> {
// Test a Fibonacci-like dependency pattern // Test a Fibonacci-like dependency pattern
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let f0_builder = dag.add_task(task_fn(|_: ()| async { 0i64 })); let f0_builder = dag.add_task(task_fn::<(), _, _>(|_: ()| 0i64));
let f1_builder = dag.add_task(task_fn(|_: ()| async { 1i64 })); let f1_builder = dag.add_task(task_fn::<(), _, _>(|_: ()| 1i64));
let mut prev2: dagx::TaskHandle<i64> = (&f0_builder).into(); let mut prev2: dagx::TaskHandle<i64> = f0_builder;
let mut prev1: dagx::TaskHandle<i64> = (&f1_builder).into(); let mut prev1: dagx::TaskHandle<i64> = f1_builder;
for _ in 2..50 { for _ in 2..50 {
let next = dag let next = dag
.add_task(task_fn(|(a, b): (i64, i64)| async move { a + b })) .add_task(task_fn::<(i64, i64), _, _>(|(a, b): (&i64, &i64)| a + b))
.depends_on((&prev2, &prev1)); .depends_on((&prev2, &prev1));
prev2 = prev1; prev2 = prev1;
prev1 = next; prev1 = next;
} }
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await?; .await?;
// 50th Fibonacci number (0-indexed) = 7778742049 // 50th Fibonacci number (0-indexed) = 7778742049
assert_eq!(dag.get(prev1)?, 7778742049); assert_eq!(output.get(prev1), 7778742049);
Ok(())
}
#[tokio::test]
async fn test_deep_chain_with_accumulator() -> DagResult<()> {
// Test a deep chain where each node accumulates a value
let dag = DagRunner::new();
// Use a simpler accumulation pattern - just sum
let first = dag.add_task(task_fn(|_: ()| async { 0i64 }));
let mut current: dagx::TaskHandle<i64> = (&first).into();
for i in 1..=200 {
current = dag
.add_task(task_fn(move |sum: i64| async move { sum + i as i64 }))
.depends_on(current);
}
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
let result = dag.get(current)?;
assert_eq!(result, 20100); // sum of 1..=200
Ok(()) Ok(())
} }
@@ -265,64 +171,39 @@ async fn test_deep_chain_with_accumulator() -> DagResult<()> {
#[tokio::test] #[tokio::test]
async fn test_zigzag_dependencies() -> DagResult<()> { async fn test_zigzag_dependencies() -> DagResult<()> {
// Test a zigzag pattern: two chains that cross-reference each other // Test a zigzag pattern: two chains that cross-reference each other
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let a_first = dag.add_task(task_fn(|_: ()| async { 1 })); let a_first = dag.add_task(task_fn::<(), _, _>(|_: ()| 1));
let b_first = dag.add_task(task_fn(|_: ()| async { 2 })); let b_first = dag.add_task(task_fn::<(), _, _>(|_: ()| 2));
let mut chain_a: dagx::TaskHandle<i32> = (&a_first).into(); let mut chain_a: dagx::TaskHandle<i32> = a_first;
let mut chain_b: dagx::TaskHandle<i32> = (&b_first).into(); let mut chain_b: dagx::TaskHandle<i32> = b_first;
for i in 0..100 { for i in 0..100 {
if i % 2 == 0 { if i % 2 == 0 {
// A depends on B // A depends on B
chain_a = dag chain_a = dag
.add_task(task_fn(move |b_val: i32| async move { b_val + 1 })) .add_task(task_fn::<i32, _, _>(move |b_val: &i32| b_val + 1))
.depends_on(chain_b); .depends_on(&chain_b);
} else { } else {
// B depends on A // B depends on A
chain_b = dag chain_b = dag
.add_task(task_fn(move |a_val: i32| async move { a_val + 2 })) .add_task(task_fn::<i32, _, _>(move |a_val: &i32| a_val + 2))
.depends_on(chain_a); .depends_on(&chain_a);
} }
} }
// Final task depends on both chains // Final task depends on both chains
let final_task = dag let final_task = dag
.add_task(task_fn(|(a, b): (i32, i32)| async move { a + b })) .add_task(task_fn::<(i32, i32), _, _>(|(a, b): (&i32, &i32)| a + b))
.depends_on((&chain_a, &chain_b)); .depends_on((&chain_a, &chain_b));
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await?; .await?;
let result = dag.get(final_task)?; let result = output.get(final_task);
assert!(result > 100); assert!(result > 100);
Ok(()) Ok(())
} }
#[tokio::test]
#[ignore] // This test is extremely resource intensive
async fn test_10000_level_chain_stress() -> DagResult<()> {
// Ultimate deep chain test
let dag = DagRunner::new();
let first = dag.add_task(task_fn(|_: ()| async { 0u64 }));
let mut current: dagx::TaskHandle<u64> = (&first).into();
for _ in 1..10_000 {
current = dag
.add_task(task_fn(|prev: u64| async move { prev + 1 }))
.depends_on(current);
}
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
assert_eq!(dag.get(current)?, 9_999);
Ok(())
}
-249
View File
@@ -1,249 +0,0 @@
//! Large scale stress tests with thousands of nodes and edges
use crate::common::task_fn;
use dagx::{DagResult, DagRunner};
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::Arc;
#[tokio::test]
async fn test_10000_independent_tasks() -> DagResult<()> {
// Test with 10,000 independent tasks
let dag = DagRunner::new();
let counter = Arc::new(AtomicUsize::new(0));
let tasks: Vec<_> = (0..10_000)
.map(|i| {
let counter = counter.clone();
dag.add_task(task_fn(move |_: ()| {
let counter = counter.clone();
async move {
counter.fetch_add(1, Ordering::Relaxed);
i
}
}))
})
.collect();
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
// Verify all tasks completed
assert_eq!(counter.load(Ordering::Relaxed), 10_000);
// Spot check some results
assert_eq!(dag.get(&tasks[0])?, 0);
assert_eq!(dag.get(&tasks[9_999])?, 9_999);
assert_eq!(dag.get(&tasks[5_000])?, 5_000);
Ok(())
}
#[tokio::test]
async fn test_5000_node_pyramid() -> DagResult<()> {
// Create a pyramid structure with ~5000 nodes
// Layer sizes: 2048 -> 512 -> 128 -> 32 -> 8 -> 2 -> 1
let dag = DagRunner::new();
// Bottom layer: 2048 nodes - convert to TaskHandles
let mut current_layer: Vec<dagx::TaskHandle<i32>> = (0..2048)
.map(|i| {
let task = dag.add_task(task_fn(move |_: ()| async move {
if i < 1024 {
1
} else {
0
}
}));
(&task).into()
})
.collect();
let layer_sizes = vec![512, 128, 32, 8, 2];
for layer_size in layer_sizes {
let mut next_layer = Vec::with_capacity(layer_size);
let group_size = current_layer.len() / layer_size;
for i in 0..layer_size {
let start = i * group_size;
let deps: Vec<_> = current_layer[start..start + group_size.min(4)]
.iter()
.take(4)
.collect();
let task = match deps.len() {
1 => dag
.add_task(task_fn(|a: i32| async move { a }))
.depends_on(deps[0]),
2 => dag
.add_task(task_fn(|(a, b): (i32, i32)| async move { a + b }))
.depends_on((deps[0], deps[1])),
3 => dag
.add_task(task_fn(
|(a, b, c): (i32, i32, i32)| async move { a + b + c },
))
.depends_on((deps[0], deps[1], deps[2])),
4 => dag
.add_task(task_fn(|(a, b, c, d): (i32, i32, i32, i32)| async move {
a + b + c + d
}))
.depends_on((deps[0], deps[1], deps[2], deps[3])),
_ => unreachable!(),
};
next_layer.push(task);
}
current_layer = next_layer;
}
// Final layer combines the last 2
let root = dag
.add_task(task_fn(|(a, b): (i32, i32)| async move { a + b }))
.depends_on((&current_layer[0], &current_layer[1]));
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
// We created 1024 ones and 1024 zeros at the bottom
// The sum should propagate up to 1024
let result = dag.get(root)?;
assert!(result > 0, "Expected positive result, got {}", result);
Ok(())
}
#[tokio::test]
async fn test_wide_dag_1000_sources_1000_sinks() -> DagResult<()> {
// Test a very wide DAG with 1000 sources and 1000 sinks
let dag = DagRunner::new();
// Create 1000 source tasks
let sources: Vec<_> = (0..1000)
.map(|i| dag.add_task(task_fn(move |_: ()| async move { i % 10 })))
.collect();
// Create 1000 sink tasks, each depending on 3 random sources
let sinks: Vec<_> = (0..1000)
.map(|i| {
let idx1 = (i * 7) % 1000;
let idx2 = (i * 13) % 1000;
let idx3 = (i * 17) % 1000;
dag.add_task(task_fn(move |(a, b, c): (i32, i32, i32)| async move {
a + b + c + i as i32
}))
.depends_on((&sources[idx1], &sources[idx2], &sources[idx3]))
})
.collect();
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
// Verify some sinks completed correctly
let result0 = dag.get(sinks[0])?;
let result500 = dag.get(sinks[500])?;
let result999 = dag.get(sinks[999])?;
assert!(result0 >= 0);
assert!(result500 >= 500);
assert!(result999 >= 999);
Ok(())
}
#[tokio::test]
async fn test_10000_node_linear_chain_segments() -> DagResult<()> {
// Test 10000 nodes arranged as 100 parallel chains of 100 nodes each
let dag = DagRunner::new();
let counter = Arc::new(AtomicUsize::new(0));
let mut chain_ends = Vec::new();
for chain_id in 0..100 {
let first = dag.add_task(task_fn({
let counter = counter.clone();
move |_: ()| {
let counter = counter.clone();
async move {
counter.fetch_add(1, Ordering::Relaxed);
chain_id * 100
}
}
}));
let mut current: dagx::TaskHandle<i32> = (&first).into();
for _i in 1..100 {
current = dag
.add_task(task_fn({
let counter = counter.clone();
move |prev: i32| {
let counter = counter.clone();
async move {
counter.fetch_add(1, Ordering::Relaxed);
prev + 1
}
}
}))
.depends_on(current);
}
chain_ends.push(current);
}
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
// All 10,000 tasks should have executed
assert_eq!(counter.load(Ordering::Relaxed), 10_000);
// Each chain should end with start + 99
for (chain_id, &chain_end) in chain_ends.iter().enumerate() {
let expected = chain_id as i32 * 100 + 99;
assert_eq!(dag.get(chain_end)?, expected);
}
Ok(())
}
#[tokio::test]
#[ignore] // This test is very resource intensive
async fn test_100000_nodes_stress() -> DagResult<()> {
// Ultimate stress test: 100,000 nodes
let dag = DagRunner::new();
let completed = Arc::new(AtomicUsize::new(0));
// Create 100,000 tasks in batches
let batch_size = 1000;
let num_batches = 100;
for batch in 0..num_batches {
let _: Vec<_> = (0..batch_size)
.map(|i| {
let completed = completed.clone();
let task_id = batch * batch_size + i;
dag.add_task(task_fn(move |_: ()| {
let completed = completed.clone();
async move {
completed.fetch_add(1, Ordering::Relaxed);
task_id
}
}))
})
.collect();
}
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
assert_eq!(completed.load(Ordering::Relaxed), 100_000);
Ok(())
}
-1
View File
@@ -4,6 +4,5 @@
//! by normal usage patterns. //! by normal usage patterns.
mod deep_chains; mod deep_chains;
mod large_scale;
mod type_limits; mod type_limits;
mod zero_tasks; mod zero_tasks;
+24 -251
View File
@@ -1,7 +1,8 @@
//! Tests for type system edge cases and limits //! Tests for type system edge cases and limits
use crate::common::task_fn;
use dagx::{DagResult, DagRunner}; use dagx::{DagResult, DagRunner};
use dagx_test::task_fn;
use std::marker::PhantomData; use std::marker::PhantomData;
#[tokio::test] #[tokio::test]
@@ -15,37 +16,34 @@ async fn test_zero_sized_types_throughout() -> DagResult<()> {
_phantom: PhantomData<T>, _phantom: PhantomData<T>,
} }
let dag = DagRunner::new(); let mut dag = DagRunner::new();
// Task producing ZST // Task producing ZST
let t1 = dag.add_task(task_fn(|_: ()| async { Empty })); let t1 = dag.add_task(task_fn::<(), _, _>(|_: ()| Empty));
// Task that also produces ZST (independent since we can't extract custom types) // Task that also produces ZST (independent since we can't extract custom types)
let t2 = dag.add_task(task_fn(|_: ()| async { 42 })); let t2 = dag.add_task(task_fn::<(), _, _>(|_: ()| 42));
// Task with PhantomData // Task with PhantomData
let _t3 = dag.add_task(task_fn(|_: ()| async { let _t3 = dag.add_task(task_fn::<(), _, _>(|_: ()| PhantomWrapper::<i32> {
PhantomWrapper::<i32> {
_phantom: PhantomData, _phantom: PhantomData,
}
})); }));
// Task that depends on t2 (uses primitive type) // Task that depends on t2 (uses primitive type)
let t4 = dag let t4 = dag
.add_task(task_fn(|val: i32| async move { .add_task(task_fn::<i32, _, _>(|&val: &i32| {
assert_eq!(val, 42); assert_eq!(val, 42);
"success" "success"
})) }))
.depends_on(&t2); .depends_on(&t2);
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await?; .await?;
assert_eq!(dag.get(t1)?, Empty); assert_eq!(output.get(t1), Empty);
assert_eq!(dag.get(t2)?, 42); assert_eq!(output.get(t2), 42);
assert_eq!(dag.get(t4)?, "success"); assert_eq!(output.get(t4), "success");
Ok(()) Ok(())
} }
@@ -59,9 +57,9 @@ async fn test_large_value_types() -> DagResult<()> {
more_data: Vec<u64>, more_data: Vec<u64>,
} }
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let t1 = dag.add_task(task_fn(|_: ()| async { let t1 = dag.add_task(task_fn::<(), _, _>(|_: ()| {
let large = LargeStruct { let large = LargeStruct {
data: [42; 8192], data: [42; 8192],
more_data: vec![1; 1000], more_data: vec![1; 1000],
@@ -71,170 +69,16 @@ async fn test_large_value_types() -> DagResult<()> {
})); }));
let t2 = dag let t2 = dag
.add_task(task_fn(|val: usize| async move { .add_task(task_fn::<usize, _, _>(|&val: &usize| {
val // Just pass through to verify val // Just pass through to verify
})) }))
.depends_on(&t1); .depends_on(&t1);
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await?; .await?;
assert_eq!(dag.get(t2)?, 1042); assert_eq!(output.get(t2), 1042);
Ok(())
}
#[tokio::test]
async fn test_unit_type_chains() -> DagResult<()> {
// Test chains of unit type operations
let dag = DagRunner::new();
let t1 = dag.add_task(task_fn(|_: ()| async {}));
let t2 = dag.add_task(task_fn(|_: ()| async {})).depends_on(&t1);
let t3 = dag.add_task(task_fn(|_: ()| async {})).depends_on(t2);
let t4 = dag
.add_task(task_fn(|_: ()| async { "done" }))
.depends_on(&t3);
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
dag.get(t1)?; // Verify no error
dag.get(t2)?;
dag.get(t3)?;
assert_eq!(dag.get(t4)?, "done");
Ok(())
}
#[tokio::test]
async fn test_nested_option_result_types() -> DagResult<()> {
// Test deeply nested generic types
type ComplexType = Result<Option<Result<Vec<Option<i32>>, String>>, &'static str>;
let dag = DagRunner::new();
let t1 = dag.add_task(task_fn(|_: ()| async {
let val: ComplexType = Ok(Some(Ok(vec![Some(42), None, Some(13)])));
val
}));
let t2 = dag
.add_task(task_fn(|complex: ComplexType| async move {
match complex {
Ok(Some(Ok(vec))) => vec.iter().filter_map(|x| *x).sum::<i32>(),
_ => 0,
}
}))
.depends_on(&t1);
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
assert_eq!(dag.get(t2)?, 55);
Ok(())
}
#[tokio::test]
async fn test_tuple_of_tuples() -> DagResult<()> {
// Test nested tuple types
let dag = DagRunner::new();
let t1 = dag.add_task(task_fn(|_: ()| async { ((1, 2), (3, 4)) }));
let t2 = dag.add_task(task_fn(|_: ()| async { ((5, 6), (7, 8)) }));
let t3 = dag
.add_task(task_fn(
|inputs: (((i32, i32), (i32, i32)), ((i32, i32), (i32, i32)))| async move {
let (((a, b), (c, d)), ((e, f), (g, h))) = inputs;
a + b + c + d + e + f + g + h
},
))
.depends_on((&t1, &t2));
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
assert_eq!(dag.get(t3)?, 36);
Ok(())
}
#[tokio::test]
async fn test_string_types_variety() -> DagResult<()> {
// Test various string types
let dag = DagRunner::new();
let t1 = dag.add_task(task_fn(|_: ()| async { "static str" }));
let t2 = dag.add_task(task_fn(|_: ()| async { String::from("owned string") }));
let t3 = dag.add_task(task_fn(|_: ()| async {
std::borrow::Cow::Borrowed("cow str")
}));
let t4 = dag.add_task(task_fn(|_: ()| async { Box::new("boxed str") }));
let t5 = dag.add_task(task_fn(|_: ()| async { std::sync::Arc::new("arc str 2") }));
let t6 = dag.add_task(task_fn(|_: ()| async { std::sync::Arc::new("arc str") }));
let combined = dag
.add_task(task_fn(
|(s1, s2, s3, s4, s5, s6): (
&str,
String,
std::borrow::Cow<str>,
Box<&str>,
std::sync::Arc<&str>,
std::sync::Arc<&str>,
)| async move { format!("{} {} {} {} {} {}", s1, s2, s3, s4, s5, s6) },
))
.depends_on((&t1, &t2, &t3, &t4, &t5, &t6));
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
let result = dag.get(combined)?;
assert!(result.contains("static str"));
assert!(result.contains("owned string"));
Ok(())
}
#[tokio::test]
async fn test_generic_type_constraints() -> DagResult<()> {
// Test with types that have trait bounds
#[derive(Debug, Clone, PartialEq)]
struct Wrapper<T: Clone + Send + 'static>(T);
let dag = DagRunner::new();
let t1 = dag.add_task(task_fn(|_: ()| async { Wrapper(42) }));
let t2 = dag.add_task(task_fn(|_: ()| async { Wrapper("hello") }));
let t3 = dag.add_task(task_fn(|_: ()| async { Wrapper(vec![1, 2, 3]) }));
let combined = dag
.add_task(task_fn(
|(w1, w2, w3): (Wrapper<i32>, Wrapper<&str>, Wrapper<Vec<i32>>)| async move {
format!("{:?} {:?} {:?}", w1, w2, w3)
},
))
.depends_on((&t1, &t2, &t3));
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
let result = dag.get(combined)?;
assert!(result.contains("Wrapper(42)"));
Ok(()) Ok(())
} }
@@ -244,9 +88,9 @@ async fn test_reference_wrapper_types() -> DagResult<()> {
// Test types that wrap references // Test types that wrap references
use std::sync::{Arc, Mutex}; use std::sync::{Arc, Mutex};
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let t1 = dag.add_task(task_fn(|_: ()| async { let t1 = dag.add_task(task_fn::<(), _, _>(|_: ()| {
let arc = Arc::new(Mutex::new(100)); let arc = Arc::new(Mutex::new(100));
// Extract value instead of passing Arc // Extract value instead of passing Arc
let guard = arc.lock().unwrap(); let guard = arc.lock().unwrap();
@@ -254,85 +98,14 @@ async fn test_reference_wrapper_types() -> DagResult<()> {
})); }));
let t2 = dag let t2 = dag
.add_task(task_fn(|val: i32| async move { val * 2 })) .add_task(task_fn::<i32, _, _>(|&val: &i32| val * 2))
.depends_on(&t1); .depends_on(&t1);
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await?; .await?;
assert_eq!(dag.get(t2)?, 200); assert_eq!(output.get(t2), 200);
Ok(())
}
#[tokio::test]
async fn test_array_types() -> DagResult<()> {
// Test with fixed-size arrays
let dag = DagRunner::new();
let t1 = dag.add_task(task_fn(|_: ()| async { [1, 2, 3, 4, 5] }));
let t2 = dag.add_task(task_fn(|_: ()| async { [6, 7, 8, 9, 10] }));
let t3 = dag
.add_task(task_fn(|(a1, a2): ([i32; 5], [i32; 5])| async move {
let mut sum = 0;
for i in 0..5 {
sum += a1[i] + a2[i];
}
sum
}))
.depends_on((&t1, &t2));
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
assert_eq!(dag.get(t3)?, 55);
Ok(())
}
#[tokio::test]
async fn test_enum_variants() -> DagResult<()> {
// Test with various enum types
#[derive(Debug, Clone, PartialEq)]
#[allow(clippy::large_enum_variant)]
enum MyEnum {
Unit,
Tuple(i32, String),
Struct { x: i32, y: i32 },
LargeVariant([u8; 1024]),
}
let dag = DagRunner::new();
let t1 = dag.add_task(task_fn(|_: ()| async { MyEnum::Unit }));
let t2 = dag.add_task(task_fn(|_: ()| async {
MyEnum::Tuple(42, "test".to_string())
}));
let t3 = dag.add_task(task_fn(|_: ()| async { MyEnum::Struct { x: 10, y: 20 } }));
let t4 = dag.add_task(task_fn(|_: ()| async { MyEnum::LargeVariant([0; 1024]) }));
let combined = dag
.add_task(task_fn(
|(e1, e2, e3, e4): (MyEnum, MyEnum, MyEnum, MyEnum)| async move {
matches!(e1, MyEnum::Unit)
&& matches!(e2, MyEnum::Tuple(42, _))
&& matches!(e3, MyEnum::Struct { x: 10, y: 20 })
&& matches!(e4, MyEnum::LargeVariant(_))
},
))
.depends_on((&t1, &t2, &t3, &t4));
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
assert!(dag.get(combined)?);
Ok(()) Ok(())
} }
+27 -160
View File
@@ -1,115 +1,25 @@
//! Tests for empty DAGs and minimal task configurations //! Tests for empty DAGs and minimal task configurations
use crate::common::task_fn; use dagx::{task, DagResult, DagRunner};
use dagx::{task, DagResult, DagRunner, Task}; use dagx_test::task_fn;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::Arc;
#[tokio::test] use tokio::task::yield_now;
async fn test_empty_dag_execution() -> DagResult<()> {
// Test that an empty DAG can be created and run without issues
let dag = DagRunner::new();
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
// Should complete successfully with no tasks
Ok(())
}
#[tokio::test]
async fn test_single_task_no_deps() -> DagResult<()> {
// Test the absolute minimum: one task with no dependencies
let dag = DagRunner::new();
let task = dag.add_task(task_fn(|_: ()| async { 42 }));
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
assert_eq!(dag.get(task)?, 42);
Ok(())
}
#[tokio::test]
async fn test_single_task_unit_input_output() -> DagResult<()> {
// Test a task that takes unit and returns unit
let dag = DagRunner::new();
let executed = Arc::new(AtomicUsize::new(0));
let exec_clone = executed.clone();
let task = dag.add_task(task_fn(move |_: ()| {
let exec = exec_clone.clone();
async move {
exec.fetch_add(1, Ordering::SeqCst);
}
}));
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
dag.get(task)?; // Just verify it doesn't error
assert_eq!(executed.load(Ordering::SeqCst), 1);
Ok(())
}
#[tokio::test]
async fn test_zero_sized_type_task() -> DagResult<()> {
// Test with zero-sized types
#[derive(Debug, Clone, PartialEq)]
struct ZeroSized;
let dag = DagRunner::new();
let task = dag.add_task(task_fn(|_: ()| async { ZeroSized }));
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
assert_eq!(dag.get(&task)?, ZeroSized);
Ok(())
}
#[tokio::test]
async fn test_empty_dag_multiple_runs() -> DagResult<()> {
// Test that an empty DAG can be run multiple times
let dag = DagRunner::new();
for _ in 0..10 {
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
}
Ok(())
}
#[tokio::test] #[tokio::test]
async fn test_dag_with_only_source_tasks() -> DagResult<()> { async fn test_dag_with_only_source_tasks() -> DagResult<()> {
// Test a DAG with multiple source tasks but no dependencies // Test a DAG with multiple source tasks but no dependencies
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let tasks: Vec<_> = (0..100) let tasks: Vec<_> = (0..100)
.map(|i| dag.add_task(task_fn(move |_: ()| async move { i }))) .map(|i| dag.add_task(task_fn::<(), _, _>(move |_: ()| i)))
.collect(); .collect();
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await?; .await?;
for (i, task) in tasks.iter().enumerate() { for (i, task) in tasks.into_iter().enumerate() {
assert_eq!(dag.get(task)?, i); assert_eq!(output.get(task), i);
} }
Ok(()) Ok(())
@@ -129,81 +39,38 @@ async fn test_single_self_contained_stateful_task() -> DagResult<()> {
} }
} }
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let task = dag.add_task(Counter { value: 21 }); let task = dag.add_task(Counter { value: 21 });
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await?; .await?;
assert_eq!(dag.get(task)?, 42); assert_eq!(output.get(task), 42);
Ok(())
}
#[tokio::test]
async fn test_alternating_empty_and_filled_dag_runs() -> DagResult<()> {
// Test alternating between empty DAG and DAG with tasks
for i in 0..5 {
let dag = DagRunner::new();
if i % 2 == 0 {
// Empty DAG
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
} else {
// DAG with a task
let task = dag.add_task(task_fn(move |_: ()| async move { i }));
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
assert_eq!(dag.get(task)?, i);
}
}
Ok(())
}
#[tokio::test]
async fn test_dag_with_never_type_simulation() -> DagResult<()> {
// Test handling of tasks that conceptually "never" complete
// (simulated by immediate completion with special marker)
#[derive(Debug, Clone, PartialEq)]
enum Never {}
let dag = DagRunner::new();
// This task completes immediately but returns a Result containing "never" type
let task = dag.add_task(task_fn(|_: ()| async { Result::<i32, Never>::Ok(42) }));
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
assert_eq!(dag.get(&task)?, Ok(42));
Ok(()) Ok(())
} }
#[tokio::test] #[tokio::test]
async fn test_minimal_task_with_minimal_async_work() -> DagResult<()> { async fn test_minimal_task_with_minimal_async_work() -> DagResult<()> {
// Test a task that does the absolute minimum async work // Test a task that does the absolute minimum async work
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let task = dag.add_task(task_fn(|_: ()| async { struct YieldTask;
// Just yield once to the runtime
tokio::task::yield_now().await; #[task]
impl YieldTask {
async fn run() -> i32 {
yield_now().await;
1337 1337
})); }
}
dag.run(|fut| { let task = dag.add_task(YieldTask);
tokio::spawn(fut);
}) let mut output = dag
.run(|fut| async move { tokio::spawn(fut).await.unwrap() })
.await?; .await?;
assert_eq!(dag.get(task)?, 1337); assert_eq!(output.get(task), 1337);
Ok(()) Ok(())
} }
-112
View File
@@ -1,112 +0,0 @@
// Common test utilities and helpers for the test suite
// Re-export task_fn for internal testing
// Note: task_fn is #[doc(hidden)] and not part of the documented public API
pub use dagx::task_fn;
// Re-export commonly used test structs
#[allow(dead_code)]
pub mod tasks {
use dagx::{task, Task};
// Simple seed task with mutable state
pub struct Seed {
pub value: i32,
}
impl Seed {
pub fn new(v: i32) -> Self {
Self { value: v }
}
}
#[task]
impl Seed {
async fn run(&mut self) -> i32 {
self.value
}
}
// Stateless increment task
pub struct Inc;
impl Inc {
pub fn new() -> Self {
Self
}
}
#[task]
impl Inc {
async fn run(x: &i32) -> i32 {
x + 1
}
}
// Stateless add task
pub struct Add;
impl Add {
pub fn new() -> Self {
Self
}
}
#[task]
impl Add {
async fn run(a: &i32, b: &i32) -> i32 {
a + b
}
}
// Stateless multiply task
pub struct Mul;
impl Mul {
pub fn new() -> Self {
Self
}
}
#[task]
impl Mul {
async fn run(a: &i32, b: &i32) -> i32 {
a * b
}
}
// Stateful counter task
pub struct Counter {
pub count: i32,
}
impl Counter {
pub fn new(initial: i32) -> Self {
Self { count: initial }
}
}
#[task]
impl Counter {
async fn run(&mut self, input: &i32) -> i32 {
self.count += input;
self.count
}
}
// Task that outputs multiple types combined as string
pub struct OutputResult;
impl OutputResult {
pub fn new() -> Self {
Self
}
}
#[task]
impl OutputResult {
async fn run(s: &String, n: &usize, f: &bool) -> String {
format!("{s}:{n}:{f}")
}
}
}
+110
View File
@@ -0,0 +1,110 @@
// Test that custom types work with the #[task] macro
use dagx::{task, DagRunner};
#[derive(Clone, Debug, PartialEq)]
struct Person {
name: String,
age: u32,
}
#[derive(Clone, Debug, PartialEq)]
struct Company {
name: String,
employees: Vec<Person>,
}
// Source task returning custom type
struct FetchPerson;
#[task]
impl FetchPerson {
async fn run(&self) -> Person {
Person {
name: "Alice".to_string(),
age: 30,
}
}
}
// Task consuming custom type
struct ProcessPerson;
#[task]
impl ProcessPerson {
async fn run(person: &Person) -> String {
format!("{} is {} years old", person.name, person.age)
}
}
// Task transforming custom type to another custom type
struct PersonToCompany;
#[task]
impl PersonToCompany {
async fn run(person: &Person) -> Company {
Company {
name: format!("{}'s Company", person.name),
employees: vec![person.clone()],
}
}
}
#[tokio::test]
async fn test_custom_type_single_dependency() {
let mut dag = DagRunner::new();
let fetch = dag.add_task(FetchPerson);
let process = dag.add_task(ProcessPerson).depends_on(&fetch);
let mut output = dag
.run(|fut| async move { tokio::spawn(fut).await.unwrap() })
.await
.unwrap();
let result = output.get(process);
assert_eq!(result, "Alice is 30 years old");
}
#[tokio::test]
async fn test_custom_type_passthrough() {
let mut dag = DagRunner::new();
let fetch = dag.add_task(FetchPerson);
let transform = dag.add_task(PersonToCompany).depends_on(&fetch);
let process = dag.add_task(ProcessPerson).depends_on(&fetch);
let mut output = dag
.run(|fut| async move { tokio::spawn(fut).await.unwrap() })
.await
.unwrap();
let person_result = output.get(process);
let company_result = output.get(transform);
assert_eq!(person_result.as_str(), "Alice is 30 years old");
assert_eq!(company_result.name, "Alice's Company");
assert_eq!(company_result.employees.len(), 1);
}
#[tokio::test]
async fn test_custom_type_fan_out() {
let mut dag = DagRunner::new();
let fetch = dag.add_task(FetchPerson);
let process1 = dag.add_task(ProcessPerson).depends_on(&fetch);
let process2 = dag.add_task(ProcessPerson).depends_on(&fetch);
let process3 = dag.add_task(ProcessPerson).depends_on(&fetch);
let mut output = dag
.run(|fut| async move { tokio::spawn(fut).await.unwrap() })
.await
.unwrap();
let result1 = output.get(process1);
let result2 = output.get(process2);
let result3 = output.get(process3);
assert_eq!(result1, "Alice is 30 years old");
assert_eq!(result2, "Alice is 30 years old");
assert_eq!(result3, "Alice is 30 years old");
}
+258
View File
@@ -0,0 +1,258 @@
//! Tests demonstrating that the type system prevents cycles at compile-time.
//!
//! These tests show that even if you try to create cycles, the Rust compiler
//! prevents it through the type-state pattern.
use dagx::{task, DagRunner};
/// Demonstrates that TaskBuilder is consumed by depends_on,
/// preventing you from wiring the same task multiple times
#[test]
fn test_builder_consumed_by_depends_on() {
struct Source;
#[task]
impl Source {
async fn run(&self) -> i32 {
42
}
}
struct Consumer;
#[task]
impl Consumer {
async fn run(&self, input: &i32) -> i32 {
*input * 2
}
}
let mut dag = DagRunner::new();
let source = dag.add_task(Source);
let consumer_builder = dag.add_task(Consumer);
// This consumes consumer_builder and returns a TaskHandle
let _consumer_handle = consumer_builder.depends_on(&source);
// We CANNOT use consumer_builder again because it was moved
// This would fail to compile:
// let _consumer2 = consumer_builder.depends_on(&source); // ERROR: use of moved value
}
/// Documents the ordering constraint that prevents cycles
#[test]
fn test_ordering_prevents_cycles() {
struct TaskA;
#[task]
impl TaskA {
async fn run(&self) -> i32 {
42
}
}
struct TaskB;
#[task]
impl TaskB {
async fn run(&self, input: &i32) -> i32 {
input + 2
}
}
let mut dag = DagRunner::new();
let a_builder = dag.add_task(TaskA);
let b_builder = dag.add_task(TaskB);
// To make A depend on B:
// 1. Need B as a TaskHandle
// 2. But B is currently a TaskBuilder
// 3. To get B as TaskHandle, must call b_builder.depends_on(...)
// 4. But then B depends on something else, not A!
// To make B depend on A:
// 1. Need A as a TaskHandle
// 2. But if we finalize A first, we can't make it depend on B later
// This creates an impossible ordering:
// - Can't finalize A before B (need B first for A's dependency)
// - Can't finalize B before A (need A first for B's dependency)
// The ONLY valid ordering is a strict topological order
// So we can only create DAGs, never cycles!
// Let's create a valid DAG instead: A→B (B depends on A)
let a_handle = a_builder; // A has no deps (unit input)
let _b_handle = b_builder.depends_on(&a_handle);
// Now we could try: a_handle.depends_on(&b_handle)?
// But TaskHandle has no such method!
}
/// Shows that even with unit-type tasks, cycles are prevented
#[test]
fn test_unit_tasks_cannot_create_cycles() {
struct Task1;
#[task]
impl Task1 {
async fn run(&self) -> i32 {
1
}
}
struct Task2;
#[task]
impl Task2 {
async fn run(&self) -> i32 {
2
}
}
let mut dag = DagRunner::new();
let task1 = dag.add_task(Task1);
let task2 = dag.add_task(Task2);
// Both have unit input, so we can convert to handles
let _handle1 = task1;
let _handle2 = task2;
// Now we have two handles, but...
// TaskHandle has no methods to add dependencies!
// The ONLY way to add dependencies is via TaskBuilder::depends_on()
// And once you've converted to TaskHandle, you can't go back
// This would fail to compile:
// _handle1.depends_on(_handle2); // ERROR: no method named `depends_on`
}
/// Demonstrates the catch-22 of three-way cycles
#[test]
fn test_three_way_cycle_impossible() {
struct A;
#[task]
impl A {
async fn run(&self, c: &i32) -> i32 {
*c
}
}
struct B;
#[task]
impl B {
async fn run(&self, a: &i32) -> i32 {
*a
}
}
struct C;
#[task]
impl C {
async fn run(&self, b: &i32) -> i32 {
*b
}
}
let mut dag = DagRunner::new();
let _a_builder = dag.add_task(A);
let _b_builder = dag.add_task(B);
let _c_builder = dag.add_task(C);
// To create A→C→B→A:
// 1. a_builder.depends_on(&c_handle) // Need c_handle
// 2. c_builder.depends_on(&b_handle) // Need b_handle
// 3. b_builder.depends_on(&a_handle) // Need a_handle (from step 1!)
//
// But step 3 needs a_handle which doesn't exist until step 1 completes.
// And step 1 needs c_handle which doesn't exist until step 2 completes.
// And step 2 needs b_handle which doesn't exist until step 3 completes.
//
// This creates a circular dependency in the COMPILATION order,
// not just the runtime order. The compiler won't allow it!
}
/// Shows that self-loops are prevented by the move semantics
#[test]
fn test_self_loop_prevented() {
struct SelfReferential;
#[task]
impl SelfReferential {
async fn run(&self, input: &i32) -> i32 {
input + 1
}
}
let mut dag = DagRunner::new();
let _builder = dag.add_task(SelfReferential);
// To create A→A (self-loop):
// builder.depends_on(&builder)
//
// But this would fail because:
// 1. depends_on() takes `self` by value (moves builder)
// 2. So builder is moved in the function call
// 3. We can't also use &builder as an argument because it's being moved
//
// This would fail to compile:
// let _handle = builder.depends_on(&builder); // ERROR: use of moved value
}
/// Comprehensive test showing all the guardrails
#[tokio::test]
async fn test_type_safety_prevents_invalid_graphs() {
// The type system enforces:
// 1. Tasks with dependencies must call depends_on()
// 2. depends_on() consumes the builder (can only be called once)
// 3. TaskHandle is immutable (no way to modify after finalization)
// 4. You need a TaskHandle to wire a dependency
// 5. Getting a TaskHandle consumes the TaskBuilder
//
// Together, these rules make it IMPOSSIBLE to create cycles!
struct Source;
#[task]
impl Source {
async fn run(&self) -> i32 {
10
}
}
struct Transform;
#[task]
impl Transform {
async fn run(&self, x: &i32) -> i32 {
x * 2
}
}
struct Aggregate;
#[task]
impl Aggregate {
async fn run(&self, a: &i32, b: &i32) -> i32 {
a + b
}
}
let mut dag = DagRunner::new();
// Build a valid DAG: two sources, two transforms, one aggregator
let s1 = dag.add_task(Source);
let s2 = dag.add_task(Source);
// Transform depends on sources
let t1 = dag.add_task(Transform).depends_on(&s1);
let t2 = dag.add_task(Transform).depends_on(&s2);
// Aggregate depends on transforms
let result = dag.add_task(Aggregate).depends_on((&t1, &t2));
// Execute and verify
let mut output = dag
.run(|fut| async move { tokio::spawn(fut).await.unwrap() })
.await
.unwrap();
assert_eq!(output.get(result), 40); // (10*2) + (10*2)
// Now if we wanted to add a cycle, we'd need to:
// - Make one of the Source tasks depend on the Aggregate result
// - But the Source builders were already consumed by .into() conversions
// - And even if we had them, Aggregate is already finalized as a TaskHandle
// - And TaskHandle has no method to add it as a dependency
//
// The API makes cycles structurally impossible!
}
-1
View File
@@ -1,5 +1,4 @@
//! Dependency tests module //! Dependency tests module
pub mod resolution; pub mod resolution;
pub mod single;
pub mod tuples; pub mod tuples;
+29 -103
View File
@@ -1,92 +1,89 @@
//! Tests for dependency resolution order and transitive dependencies //! Tests for dependency resolution order and transitive dependencies
use crate::common::task_fn; use dagx::{task, DagResult, DagRunner};
use dagx::{DagResult, DagRunner}; use dagx_test::task_fn;
use std::sync::{Arc, Mutex}; use std::sync::{Arc, Mutex};
#[tokio::test] #[tokio::test]
async fn test_dependency_resolution_order() -> DagResult<()> { async fn test_dependency_resolution_order() -> DagResult<()> {
// Test that dependencies are resolved in the correct order // Test that dependencies are resolved in the correct order
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let execution_log = Arc::new(Mutex::new(Vec::new())); let execution_log = Arc::new(Mutex::new(Vec::new()));
// Create a chain: A -> B -> C -> D -> E struct ATask {
let a = { log: Arc<Mutex<Vec<(&'static str, i32)>>>,
let log = execution_log.clone(); }
dag.add_task(task_fn(move |_: ()| {
let log = log.clone(); #[task]
async move { impl ATask {
log.lock().unwrap().push(("A_start", 0)); async fn run(&self) -> i32 {
self.log.lock().unwrap().push(("A_start", 0));
tokio::time::sleep(tokio::time::Duration::from_millis(10)).await; tokio::time::sleep(tokio::time::Duration::from_millis(10)).await;
log.lock().unwrap().push(("A_end", 1)); self.log.lock().unwrap().push(("A_end", 1));
1 1
} }
})) }
};
// Create a chain: A -> B -> C -> D -> E
let a = dag.add_task(ATask {
log: execution_log.clone(),
});
let b = { let b = {
let log = execution_log.clone(); let log = execution_log.clone();
dag.add_task(task_fn(move |x: i32| { dag.add_task(task_fn::<i32, _, _>(move |&x: &i32| {
let log = log.clone(); let log = log.clone();
async move {
log.lock().unwrap().push(("B_start", x)); log.lock().unwrap().push(("B_start", x));
assert_eq!(x, 1, "B should receive 1 from A"); assert_eq!(x, 1, "B should receive 1 from A");
log.lock().unwrap().push(("B_end", x + 1)); log.lock().unwrap().push(("B_end", x + 1));
x + 1 x + 1
}
})) }))
.depends_on(&a) .depends_on(&a)
}; };
let c = { let c = {
let log = execution_log.clone(); let log = execution_log.clone();
dag.add_task(task_fn(move |x: i32| { dag.add_task(task_fn::<i32, _, _>(move |&x: &i32| {
let log = log.clone(); let log = log.clone();
async move {
log.lock().unwrap().push(("C_start", x)); log.lock().unwrap().push(("C_start", x));
assert_eq!(x, 2, "C should receive 2 from B"); assert_eq!(x, 2, "C should receive 2 from B");
log.lock().unwrap().push(("C_end", x + 1)); log.lock().unwrap().push(("C_end", x + 1));
x + 1 x + 1
}
})) }))
.depends_on(b) .depends_on(&b)
}; };
let d = { let d = {
let log = execution_log.clone(); let log = execution_log.clone();
dag.add_task(task_fn(move |x: i32| { dag.add_task(task_fn::<i32, _, _>(move |&x: &i32| {
let log = log.clone(); let log = log.clone();
async move {
log.lock().unwrap().push(("D_start", x)); log.lock().unwrap().push(("D_start", x));
assert_eq!(x, 3, "D should receive 3 from C"); assert_eq!(x, 3, "D should receive 3 from C");
log.lock().unwrap().push(("D_end", x + 1)); log.lock().unwrap().push(("D_end", x + 1));
x + 1 x + 1
}
})) }))
.depends_on(c) .depends_on(&c)
}; };
let e = { let e = {
let log = execution_log.clone(); let log = execution_log.clone();
dag.add_task(task_fn(move |x: i32| { dag.add_task(task_fn::<i32, _, _>(move |&x: &i32| {
let log = log.clone(); let log = log.clone();
async move {
log.lock().unwrap().push(("E_start", x)); log.lock().unwrap().push(("E_start", x));
assert_eq!(x, 4, "E should receive 4 from D"); assert_eq!(x, 4, "E should receive 4 from D");
log.lock().unwrap().push(("E_end", x + 1)); log.lock().unwrap().push(("E_end", x + 1));
x + 1 x + 1
}
})) }))
.depends_on(d) .depends_on(&d)
}; };
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await?; .await?;
assert_eq!(dag.get(e)?, 5); assert_eq!(output.get(e), 5);
// Verify execution order // Verify execution order
let log = execution_log.lock().unwrap(); let log = execution_log.lock().unwrap();
@@ -103,74 +100,3 @@ async fn test_dependency_resolution_order() -> DagResult<()> {
Ok(()) Ok(())
} }
#[tokio::test]
async fn test_transitive_dependencies() -> DagResult<()> {
// Test that transitive dependencies work correctly
// A -> B -> C (creates transitive A -> C relationship)
let dag = DagRunner::new();
let a = dag.add_task(task_fn(|_: ()| async { 10 }));
let b = dag
.add_task(task_fn(|x: i32| async move { x * 2 }))
.depends_on(&a);
let c = dag
.add_task(task_fn(|x: i32| async move { x + 5 }))
.depends_on(b);
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
assert_eq!(dag.get(&a)?, 10);
assert_eq!(dag.get(b)?, 20);
assert_eq!(dag.get(c)?, 25); // 20 + 5
Ok(())
}
#[tokio::test]
async fn test_deep_chain_50_levels() -> DagResult<()> {
let dag = DagRunner::new();
let first = dag.add_task(task_fn(|_: ()| async { 0 }));
let mut handles = vec![(&first).into()];
for _i in 1..=50 {
let next = dag
.add_task(task_fn(move |x: i32| async move { x + 1 }))
.depends_on(handles.last().unwrap());
handles.push(next);
}
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
assert_eq!(dag.get(handles.last().unwrap())?, 50);
Ok(())
}
#[tokio::test]
async fn test_deep_chain_200_levels() -> DagResult<()> {
let dag = DagRunner::new();
let first = dag.add_task(task_fn(|_: ()| async { 1 }));
let mut current = (&first).into();
for _ in 0..200 {
current = dag
.add_task(task_fn(|x: i32| async move { x + 1 }))
.depends_on(current);
}
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
assert_eq!(dag.get(current)?, 201);
Ok(())
}
-128
View File
@@ -1,128 +0,0 @@
//! Tests for single dependencies and basic dependency data flow
use crate::common::task_fn;
use dagx::{DagResult, DagRunner};
#[tokio::test]
async fn test_single_dependency() -> DagResult<()> {
let dag = DagRunner::new();
let source = dag.add_task(task_fn(|_: ()| async { 42 }));
let dependent = dag
.add_task(task_fn(|x: i32| async move { x + 1 }))
.depends_on(&source);
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
assert_eq!(dag.get(dependent)?, 43);
Ok(())
}
#[tokio::test]
async fn test_dependency_data_flow() -> DagResult<()> {
// Test that data flows correctly through dependencies
let dag = DagRunner::new();
let source = dag.add_task(task_fn(|_: ()| async { vec![1, 2, 3, 4, 5] }));
let sum = dag
.add_task(task_fn(|v: Vec<i32>| async move { v.iter().sum::<i32>() }))
.depends_on(&source);
let product = dag
.add_task(task_fn(
|v: Vec<i32>| async move { v.iter().product::<i32>() },
))
.depends_on(&source);
let final_result = dag
.add_task(task_fn(|(s, p): (i32, i32)| async move { s + p }))
.depends_on((&sum, &product));
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
// Note: sum and product are not sinks (final_result depends on them)
// Only final_result is a sink and can be retrieved
assert_eq!(dag.get(final_result)?, 135); // 15+120
Ok(())
}
#[tokio::test]
async fn test_dependencies_with_different_types() -> DagResult<()> {
// Test that dependencies with different types work correctly
let dag = DagRunner::new();
let int_source = dag.add_task(task_fn(|_: ()| async { 42 }));
let string_source = dag.add_task(task_fn(|_: ()| async { "hello".to_string() }));
let bool_source = dag.add_task(task_fn(|_: ()| async { true }));
let combined = dag
.add_task(task_fn(|(i, s, b): (i32, String, bool)| async move {
format!("{}: {} ({})", s, i, b)
}))
.depends_on((&int_source, &string_source, &bool_source));
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
assert_eq!(dag.get(combined)?, "hello: 42 (true)");
Ok(())
}
#[tokio::test]
async fn test_shared_dependencies() -> DagResult<()> {
// Test that a single node can be a dependency for multiple downstream nodes
let dag = DagRunner::new();
let shared = dag.add_task(task_fn(|_: ()| async { 42 }));
// Create 10 tasks that all depend on the shared node
let dependents: Vec<_> = (0..10)
.map(|i| {
dag.add_task(task_fn(move |x: i32| async move { x * (i + 1) }))
.depends_on(&shared)
})
.collect();
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
// Verify all dependents got the correct value from shared
for (i, handle) in dependents.iter().enumerate() {
assert_eq!(dag.get(handle)?, 42 * (i as i32 + 1));
}
Ok(())
}
#[tokio::test]
async fn test_multiple_source_nodes() -> DagResult<()> {
// Test DAG with multiple source nodes (no dependencies)
let dag = DagRunner::new();
let sources: Vec<_> = (0..20)
.map(|i| dag.add_task(task_fn(move |_: ()| async move { i * 10 })))
.collect();
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
for (i, source) in sources.iter().enumerate() {
assert_eq!(dag.get(source)?, i as i32 * 10);
}
Ok(())
}
+56 -65
View File
@@ -1,131 +1,124 @@
//! Tests for tuple dependencies (2-8 dependencies) //! Tests for tuple dependencies (2-8 dependencies)
use crate::common::task_fn;
use dagx::{DagResult, DagRunner}; use dagx::{DagResult, DagRunner};
use dagx_test::task_fn;
#[tokio::test] #[tokio::test]
async fn test_two_dependencies() -> DagResult<()> { async fn test_two_dependencies() -> DagResult<()> {
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let a = dag.add_task(task_fn(|_: ()| async { 10 })); let a = dag.add_task(task_fn::<(), _, _>(|_: ()| 10));
let b = dag.add_task(task_fn(|_: ()| async { 20 })); let b = dag.add_task(task_fn::<(), _, _>(|_: ()| 20));
let sum = dag let sum = dag
.add_task(task_fn(|(x, y): (i32, i32)| async move { x + y })) .add_task(task_fn::<(i32, i32), _, _>(|(x, y): (&i32, &i32)| x + y))
.depends_on((&a, &b)); .depends_on((&a, &b));
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await?; .await?;
assert_eq!(dag.get(sum)?, 30); assert_eq!(output.get(sum), 30);
Ok(()) Ok(())
} }
#[tokio::test] #[tokio::test]
async fn test_three_dependencies() -> DagResult<()> { async fn test_three_dependencies() -> DagResult<()> {
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let a = dag.add_task(task_fn(|_: ()| async { 1 })); let a = dag.add_task(task_fn::<(), _, _>(|_: ()| 1));
let b = dag.add_task(task_fn(|_: ()| async { 2 })); let b = dag.add_task(task_fn::<(), _, _>(|_: ()| 2));
let c = dag.add_task(task_fn(|_: ()| async { 3 })); let c = dag.add_task(task_fn::<(), _, _>(|_: ()| 3));
let sum = dag let sum = dag
.add_task(task_fn( .add_task(task_fn::<(i32, i32, i32), _, _>(
|(x, y, z): (i32, i32, i32)| async move { x + y + z }, |(x, y, z): (&i32, &i32, &i32)| x + y + z,
)) ))
.depends_on((&a, &b, &c)); .depends_on((&a, &b, &c));
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await?; .await?;
assert_eq!(dag.get(sum)?, 6); assert_eq!(output.get(sum), 6);
Ok(()) Ok(())
} }
#[tokio::test] #[tokio::test]
async fn test_four_dependencies() -> DagResult<()> { async fn test_four_dependencies() -> DagResult<()> {
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let deps: Vec<_> = (1..=4) let deps: Vec<_> = (1..=4)
.map(|i| dag.add_task(task_fn(move |_: ()| async move { i }))) .map(|i| dag.add_task(task_fn::<(), _, _>(move |_: ()| i)))
.collect(); .collect();
let sum = dag let sum = dag
.add_task(task_fn(|(a, b, c, d): (i32, i32, i32, i32)| async move { .add_task(task_fn::<(i32, i32, i32, i32), _, _>(
a + b + c + d |(a, b, c, d): (&i32, &i32, &i32, &i32)| a + b + c + d,
})) ))
.depends_on((&deps[0], &deps[1], &deps[2], &deps[3])); .depends_on((&deps[0], &deps[1], &deps[2], &deps[3]));
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await?; .await?;
assert_eq!(dag.get(sum)?, 10); // 1+2+3+4 assert_eq!(output.get(sum), 10); // 1+2+3+4
Ok(()) Ok(())
} }
#[tokio::test] #[tokio::test]
async fn test_five_dependencies() -> DagResult<()> { async fn test_five_dependencies() -> DagResult<()> {
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let deps: Vec<_> = (1..=5) let deps: Vec<_> = (1..=5)
.map(|i| dag.add_task(task_fn(move |_: ()| async move { i }))) .map(|i| dag.add_task(task_fn::<(), _, _>(move |_: ()| i)))
.collect(); .collect();
let sum = dag let sum = dag
.add_task(task_fn( .add_task(task_fn::<(i32, i32, i32, i32, i32), _, _>(
|(a, b, c, d, e): (i32, i32, i32, i32, i32)| async move { a + b + c + d + e }, |(a, b, c, d, e): (&i32, &i32, &i32, &i32, &i32)| a + b + c + d + e,
)) ))
.depends_on((&deps[0], &deps[1], &deps[2], &deps[3], &deps[4])); .depends_on((&deps[0], &deps[1], &deps[2], &deps[3], &deps[4]));
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await?; .await?;
assert_eq!(dag.get(sum)?, 15); // 1+2+3+4+5 assert_eq!(output.get(sum), 15); // 1+2+3+4+5
Ok(()) Ok(())
} }
#[tokio::test] #[tokio::test]
async fn test_six_dependencies() -> DagResult<()> { async fn test_six_dependencies() -> DagResult<()> {
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let deps: Vec<_> = (1..=6) let deps: Vec<_> = (1..=6)
.map(|i| dag.add_task(task_fn(move |_: ()| async move { i }))) .map(|i| dag.add_task(task_fn::<(), _, _>(move |_: ()| i)))
.collect(); .collect();
let sum = dag.add_task(task_fn( let sum = dag
|(a, b, c, d, e, f): (i32, i32, i32, i32, i32, i32)| async move { .add_task(task_fn::<(i32, i32, i32, i32, i32, i32), _, _>(
a + b + c + d + e + f |(a, b, c, d, e, f): (&i32, &i32, &i32, &i32, &i32, &i32)| a + b + c + d + e + f,
} ))
)).depends_on(( .depends_on((&deps[0], &deps[1], &deps[2], &deps[3], &deps[4], &deps[5]));
&deps[0], &deps[1], &deps[2], &deps[3], &deps[4], &deps[5]
));
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await?; .await?;
assert_eq!(dag.get(sum)?, 21); // 1+2+3+4+5+6 assert_eq!(output.get(sum), 21); // 1+2+3+4+5+6
Ok(()) Ok(())
} }
#[tokio::test] #[tokio::test]
async fn test_seven_dependencies() -> DagResult<()> { async fn test_seven_dependencies() -> DagResult<()> {
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let deps: Vec<_> = (1..=7) let deps: Vec<_> = (1..=7)
.map(|i| dag.add_task(task_fn(move |_: ()| async move { i }))) .map(|i| dag.add_task(task_fn::<(), _, _>(move |_: ()| i)))
.collect(); .collect();
let sum = dag let sum = dag
.add_task(task_fn( .add_task(task_fn::<(i32, i32, i32, i32, i32, i32, i32), _, _>(
|(a, b, c, d, e, f, g): (i32, i32, i32, i32, i32, i32, i32)| async move { |(a, b, c, d, e, f, g): (&i32, &i32, &i32, &i32, &i32, &i32, &i32)| {
a + b + c + d + e + f + g a + b + c + d + e + f + g
}, },
)) ))
@@ -133,26 +126,25 @@ async fn test_seven_dependencies() -> DagResult<()> {
&deps[0], &deps[1], &deps[2], &deps[3], &deps[4], &deps[5], &deps[6], &deps[0], &deps[1], &deps[2], &deps[3], &deps[4], &deps[5], &deps[6],
)); ));
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await?; .await?;
assert_eq!(dag.get(sum)?, 28); // 1+2+3+4+5+6+7 assert_eq!(output.get(sum), 28); // 1+2+3+4+5+6+7
Ok(()) Ok(())
} }
#[tokio::test] #[tokio::test]
async fn test_eight_dependencies() -> DagResult<()> { async fn test_eight_dependencies() -> DagResult<()> {
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let deps: Vec<_> = (1..=8) let deps: Vec<_> = (1..=8)
.map(|i| dag.add_task(task_fn(move |_: ()| async move { i }))) .map(|i| dag.add_task(task_fn::<(), _, _>(move |_: ()| i)))
.collect(); .collect();
let sum = dag let sum = dag
.add_task(task_fn( .add_task(task_fn::<(i32, i32, i32, i32, i32, i32, i32, i32), _, _>(
|(a, b, c, d, e, f, g, h): (i32, i32, i32, i32, i32, i32, i32, i32)| async move { |(a, b, c, d, e, f, g, h): (&i32, &i32, &i32, &i32, &i32, &i32, &i32, &i32)| {
a + b + c + d + e + f + g + h a + b + c + d + e + f + g + h
}, },
)) ))
@@ -160,11 +152,10 @@ async fn test_eight_dependencies() -> DagResult<()> {
&deps[0], &deps[1], &deps[2], &deps[3], &deps[4], &deps[5], &deps[6], &deps[7], &deps[0], &deps[1], &deps[2], &deps[3], &deps[4], &deps[5], &deps[6], &deps[7],
)); ));
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await?; .await?;
assert_eq!(dag.get(sum)?, 36); // Sum of 1..=8 assert_eq!(output.get(sum), 36); // Sum of 1..=8
Ok(()) Ok(())
} }
-175
View File
@@ -1,175 +0,0 @@
//! Tests for different error types and error handling
use crate::common::task_fn;
use dagx::{DagError, DagRunner};
#[tokio::test]
async fn test_result_not_found_error() {
let dag = DagRunner::new();
let t1 = dag.add_task(task_fn(|_: ()| async { 42 }));
// Try to get result before running
let result = dag.get(&t1);
assert!(matches!(result, Err(DagError::ResultNotFound { .. })));
// Run and verify it works after
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
assert_eq!(dag.get(&t1).unwrap(), 42);
}
#[tokio::test]
async fn test_nested_result_error_handling() {
let dag = DagRunner::new();
// Task that returns a Result
let task1 = dag.add_task(task_fn(|_: ()| async {
Result::<i32, String>::Err("Internal error".to_string())
}));
let task2 = dag.add_task(task_fn(|_: ()| async { Result::<i32, String>::Ok(42) }));
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
// Both tasks complete successfully (they return Results)
assert_eq!(dag.get(&task1).unwrap(), Err("Internal error".to_string()));
assert_eq!(dag.get(&task2).unwrap(), Ok(42));
}
#[tokio::test]
async fn test_option_none_not_error() {
let dag = DagRunner::new();
let task = dag.add_task(task_fn(|_: ()| async { None::<i32> }));
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
// None is a valid result, not an error
assert_eq!(dag.get(&task).unwrap(), None);
}
#[tokio::test]
async fn test_custom_error_types() {
#[derive(Debug, Clone, PartialEq)]
enum CustomError {
ValidationError(String),
ProcessingError { code: i32, details: String },
}
let dag = DagRunner::new();
let task1 = dag.add_task(task_fn(|_: ()| async {
Result::<i32, CustomError>::Err(CustomError::ValidationError("Invalid input".to_string()))
}));
let task2 = dag.add_task(task_fn(|_: ()| async {
Result::<i32, CustomError>::Err(CustomError::ProcessingError {
code: 500,
details: "Processing failed".to_string(),
})
}));
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
// Check custom errors are preserved
match dag.get(&task1).unwrap() {
Err(CustomError::ValidationError(msg)) => assert_eq!(msg, "Invalid input"),
_ => panic!("Wrong error type"),
}
match dag.get(&task2).unwrap() {
Err(CustomError::ProcessingError { code, .. }) => assert_eq!(code, 500),
_ => panic!("Wrong error type"),
}
}
#[tokio::test]
async fn test_error_in_tuple_dependency() {
let dag = DagRunner::new();
let ok_task = dag.add_task(task_fn(|_: ()| async { 10 }));
let err_task = dag.add_task(task_fn(|_: ()| async {
panic!("Error in dependency");
}));
let dependent = dag
.add_task(task_fn(|(a, b): (i32, i32)| async move { a + b }))
.depends_on((&ok_task, &err_task));
let result = dag
.run(|fut| {
tokio::spawn(fut);
})
.await;
assert!(result.is_err());
assert!(dag.get(dependent).is_err());
}
#[tokio::test]
async fn test_string_panic_vs_structured_panic() {
let dag = DagRunner::new();
let string_panic = dag.add_task(task_fn(|_: ()| async {
panic!("String panic message");
}));
let formatted_panic = dag.add_task(task_fn(|_: ()| async {
panic!("Formatted panic: value={}, code={}", 42, "ERROR");
}));
let _ = dag
.run(|fut| {
tokio::spawn(fut);
})
.await;
// Both should result in errors
assert!(dag.get(&string_panic).is_err());
assert!(dag.get(&formatted_panic).is_err());
}
#[tokio::test]
async fn test_anyhow_like_error_chain() {
// Test error chains with simple string-based errors
let dag = DagRunner::new();
let task = dag.add_task(task_fn(|_: ()| async {
// Simulate an error chain with nested Results
let _inner: Result<i32, String> = Err("Inner error".to_string());
let outer: Result<Result<i32, String>, String> = Err("Outer error".to_string());
outer
}));
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
// Get the chained error
let result = dag.get(&task).unwrap();
assert!(result.is_err());
if let Err(e) = result {
assert_eq!(e, "Outer error");
}
}
-1
View File
@@ -2,6 +2,5 @@
//! //!
//! Tests for error scenarios and recovery patterns. //! Tests for error scenarios and recovery patterns.
mod error_types;
mod propagation; mod propagation;
mod recovery; mod recovery;
+88 -288
View File
@@ -1,47 +1,38 @@
//! Tests for error propagation through the DAG //! Tests for error propagation through the DAG
use crate::common::task_fn; use dagx::{task, DagError, DagRunner};
use dagx::DagRunner; use dagx_test::task_fn;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::Arc; use std::sync::{Arc, Mutex};
use std::time::Duration;
use tokio::time::sleep;
#[tokio::test] #[tokio::test]
async fn test_linear_error_propagation() { async fn test_linear_error_propagation() {
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let t1 = dag.add_task(task_fn(|_: ()| async { 1 })); let t1 = dag.add_task(task_fn::<(), _, _>(|_: ()| 1));
let t2 = dag let t2 = dag
.add_task(task_fn(|x: i32| async move { .add_task(task_fn::<i32, _, _>(|&x: &i32| -> i32 {
panic!("Error at t2 with input {}", x); panic!("Error at t2 with input {}", x);
})) }))
.depends_on(&t1); .depends_on(&t1);
let t3 = dag let t3 = dag
.add_task(task_fn(|x: i32| async move { x + 1 })) .add_task(task_fn::<i32, _, _>(|&x: &i32| x + 1))
.depends_on(t2); .depends_on(&t2);
let t4 = dag dag.add_task(task_fn::<i32, _, _>(|&x: &i32| x * 2))
.add_task(task_fn(|x: i32| async move { x * 2 })) .depends_on(&t3);
.depends_on(t3);
let _ = dag assert!(matches!(
.run(|fut| { dag.run(|fut| async move { tokio::spawn(fut).await.unwrap() })
tokio::spawn(fut); .await,
}) Err(DagError::TaskPanicked { .. })
.await; ));
// t1 succeeds
assert_eq!(dag.get(&t1).unwrap(), 1);
// t2 fails
assert!(dag.get(t2).is_err());
// t3 and t4 fail due to dependency
assert!(dag.get(t3).is_err());
assert!(dag.get(t4).is_err());
} }
#[tokio::test] #[tokio::test]
async fn test_diamond_error_propagation() { async fn test_diamond_error_propagation() {
let dag = DagRunner::new(); let mut dag = DagRunner::new();
// source // source
// / \ // / \
@@ -49,347 +40,156 @@ async fn test_diamond_error_propagation() {
// \ / // \ /
// sink // sink
let source = dag.add_task(task_fn(|_: ()| async { 100 })); let source = dag.add_task(task_fn::<(), _, _>(|_: ()| 100));
let left = dag let left = dag
.add_task(task_fn(|x: i32| async move { x / 2 })) .add_task(task_fn::<i32, _, _>(|&x: &i32| x / 2))
.depends_on(&source); .depends_on(&source);
let right = dag let right = dag
.add_task(task_fn(|x: i32| async move { .add_task(task_fn::<i32, _, _>(|&x: &i32| {
panic!("Right path fails with {}", x); panic!("Right path fails with {}", x);
})) }))
.depends_on(&source); .depends_on(&source);
let sink = dag dag.add_task(task_fn::<(i32, i32), _, _>(|(l, r): (&i32, &i32)| l + r))
.add_task(task_fn(|(l, r): (i32, i32)| async move { l + r }))
.depends_on((&left, &right)); .depends_on((&left, &right));
let _ = dag assert!(matches!(
.run(|fut| { dag.run(|fut| async move { tokio::spawn(fut).await.unwrap() })
tokio::spawn(fut); .await,
}) Err(DagError::TaskPanicked { .. })
.await; ));
assert_eq!(dag.get(&source).unwrap(), 100);
assert_eq!(dag.get(left).unwrap(), 50);
assert!(dag.get(right).is_err());
assert!(dag.get(sink).is_err()); // Fails due to right dependency
}
#[tokio::test]
async fn test_error_stops_at_boundary() {
let dag = DagRunner::new();
let execution_tracker = Arc::new(AtomicUsize::new(0));
// Branch with error
let error_source = dag.add_task(task_fn({
let tracker = execution_tracker.clone();
move |_: ()| {
let tracker = tracker.clone();
async move {
tracker.fetch_add(1, Ordering::SeqCst);
panic!("Error source");
}
}
}));
let error_dependent = dag
.add_task(task_fn({
let tracker = execution_tracker.clone();
move |_: i32| {
let tracker = tracker.clone();
async move {
tracker.fetch_add(100, Ordering::SeqCst); // Should not execute
42
}
}
}))
.depends_on(&error_source);
// Independent branch
let independent = dag.add_task(task_fn({
let tracker = execution_tracker.clone();
move |_: ()| {
let tracker = tracker.clone();
async move {
tracker.fetch_add(10, Ordering::SeqCst);
10
}
}
}));
let _ = dag
.run(|fut| {
tokio::spawn(fut);
})
.await;
// Error branch: source runs but dependent doesn't
assert!(dag.get(&error_source).is_err());
assert!(dag.get(error_dependent).is_err());
// Independent branch runs
assert_eq!(dag.get(&independent).unwrap(), 10);
// Check execution: 1 (error_source) + 10 (independent) = 11
assert_eq!(execution_tracker.load(Ordering::SeqCst), 11);
}
#[tokio::test]
async fn test_multiple_error_sources_convergence() {
let dag = DagRunner::new();
// Multiple error sources
let error1 = dag.add_task(task_fn(|_: ()| async {
panic!("Error source 1");
}));
let error2 = dag.add_task(task_fn(|_: ()| async {
panic!("Error source 2");
}));
let success = dag.add_task(task_fn(|_: ()| async { 42 }));
// Convergence point depends on all three
let convergence = dag
.add_task(task_fn(|(_, _, _): (i32, i32, i32)| async { 100 }))
.depends_on((&error1, &error2, &success));
let _ = dag
.run(|fut| {
tokio::spawn(fut);
})
.await;
// All error sources fail
assert!(dag.get(&error1).is_err());
assert!(dag.get(&error2).is_err());
// Success task works
assert_eq!(dag.get(&success).unwrap(), 42);
// Convergence fails due to any error dependency
assert!(dag.get(convergence).is_err());
} }
#[tokio::test] #[tokio::test]
async fn test_error_in_wide_fanout() { async fn test_error_in_wide_fanout() {
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let source = dag.add_task(task_fn(|_: ()| async { let source = dag.add_task(task_fn::<(), _, _>(|_: ()| -> i32 {
panic!("Source fails"); panic!("Source fails");
})); }));
// Create wide fanout from failing source // Create wide fanout from failing source
let dependents: Vec<_> = (0..20) for i in 0..20 {
.map(|i| { dag.add_task(task_fn::<i32, _, _>(move |&x: &i32| x + i))
dag.add_task(task_fn(move |x: i32| async move { x + i })) .depends_on(&source);
.depends_on(&source)
})
.collect();
let _ = dag
.run(|fut| {
tokio::spawn(fut);
})
.await;
// Source fails
assert!(dag.get(&source).is_err());
// All dependents fail
for dependent in &dependents {
assert!(dag.get(dependent).is_err());
} }
assert!(matches!(
dag.run(|fut| async move { tokio::spawn(fut).await.unwrap() })
.await,
Err(DagError::TaskPanicked { .. })
));
} }
#[tokio::test] #[tokio::test]
async fn test_selective_error_propagation() { async fn test_selective_error_propagation() {
let dag = DagRunner::new(); let mut dag = DagRunner::new();
// Source that produces a Result // Source that produces a Result
let source = dag.add_task(task_fn(|_: ()| async { let source = dag.add_task(task_fn::<(), _, _>(|_: ()| {
Result::<i32, String>::Err("Source error".to_string()) Result::<i32, String>::Err("Source error".to_string())
})); }));
// Handler that processes the Result // Handler that processes the Result
let handler = dag let handler = dag
.add_task(task_fn(|result: Result<i32, String>| async move { .add_task(task_fn::<Result<_, _>, _, _>(
|result: &Result<i32, String>| {
match result { match result {
Ok(val) => val * 2, Ok(val) => val * 2,
Err(_) => -1, // Default value on error Err(_) => -1, // Default value on error
} }
})) },
))
.depends_on(&source); .depends_on(&source);
// Further processing // Further processing
let final_task = dag let final_task = dag
.add_task(task_fn(|val: i32| async move { .add_task(task_fn::<i32, _, _>(|&val: &i32| {
if val < 0 { if val < 0 {
"Handled error case".to_string() "Handled error case".to_string()
} else { } else {
format!("Success: {}", val) format!("Success: {}", val)
} }
})) }))
.depends_on(handler); .depends_on(&handler);
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await .await
.unwrap(); .unwrap();
// Source returns an Err variant (not a panic) // Source returns an Err variant (not a panic)
assert_eq!(dag.get(&source).unwrap(), Err("Source error".to_string())); assert_eq!(output.get(source), Err("Source error".to_string()));
// Handler processes it // Handler processes it
assert_eq!(dag.get(handler).unwrap(), -1); assert_eq!(output.get(handler), -1);
// Final task handles the error case // Final task handles the error case
assert_eq!(dag.get(final_task).unwrap(), "Handled error case"); assert_eq!(output.get(final_task), "Handled error case");
} }
#[tokio::test] #[tokio::test]
async fn test_error_propagation_timing() { async fn test_error_propagation_timing() {
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let propagation_order = Arc::new(parking_lot::Mutex::new(Vec::new())); let propagation_order = Arc::new(Mutex::new(Vec::new()));
let t1 = dag.add_task(task_fn({ struct T1Task {
let order = propagation_order.clone(); order: Arc<Mutex<Vec<&'static str>>>,
move |_: ()| { }
let order = order.clone();
async move { #[task]
tokio::time::sleep(tokio::time::Duration::from_millis(10)).await; impl T1Task {
order.lock().push("t1_complete"); async fn run(&self) -> i32 {
sleep(Duration::from_millis(10)).await;
self.order.lock().unwrap().push("t1_complete");
1 1
} }
} }
}));
let t1 = dag.add_task(T1Task {
order: propagation_order.clone(),
});
struct T2Task {
order: Arc<Mutex<Vec<&'static str>>>,
}
#[task]
impl T2Task {
async fn run(&self, x: &i32) -> i32 {
self.order.lock().unwrap().push("t2_start");
sleep(Duration::from_millis(10)).await;
panic!("t2 fails with {x}");
}
}
let t2 = dag let t2 = dag
.add_task(task_fn({ .add_task(T2Task {
let order = propagation_order.clone(); order: propagation_order.clone(),
move |x: i32| { })
let order = order.clone();
async move {
order.lock().push("t2_start");
tokio::time::sleep(tokio::time::Duration::from_millis(10)).await;
panic!("t2 fails with {}", x);
}
}
}))
.depends_on(&t1); .depends_on(&t1);
let _t3 = dag let _t3 = dag
.add_task(task_fn({ .add_task(task_fn::<i32, _, _>({
let order = propagation_order.clone(); let order = propagation_order.clone();
move |x: i32| { move |x: &i32| {
let order = order.clone(); let order = order.clone();
async move { order.lock().unwrap().push("t3_should_not_run");
order.lock().push("t3_should_not_run");
x + 1 x + 1
} }
}
})) }))
.depends_on(&t2); .depends_on(&t2);
let _ = dag let _ = dag
.run(|fut| { .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
tokio::spawn(fut);
})
.await; .await;
let order = propagation_order.lock().clone(); let order = propagation_order.lock().unwrap().clone();
// t1 completes, t2 starts and fails, t3 never runs // t1 completes, t2 starts and fails, t3 never runs
assert!(order.contains(&"t1_complete")); assert!(order.contains(&"t1_complete"));
assert!(order.contains(&"t2_start")); assert!(order.contains(&"t2_start"));
assert!(!order.contains(&"t3_should_not_run")); assert!(!order.contains(&"t3_should_not_run"));
} }
#[tokio::test]
async fn test_partial_branch_failure_propagation() {
let dag = DagRunner::new();
// root
// / | \
// a1 b1 c1
// | | |
// a2 b2 c2
// | (err) |
// a3 b3 c3
// \ | /
// sink
let root = dag.add_task(task_fn(|_: ()| async { 100 }));
// Branch A
let a1 = dag
.add_task(task_fn(|x: i32| async move { x + 1 }))
.depends_on(&root);
let a2 = dag
.add_task(task_fn(|x: i32| async move { x + 2 }))
.depends_on(a1);
let a3 = dag
.add_task(task_fn(|x: i32| async move { x + 3 }))
.depends_on(a2);
// Branch B (with error) - use Result instead of panic
let b1 = dag
.add_task(task_fn(|x: i32| async move { Ok(x * 2) }))
.depends_on(&root);
let b2 = dag
.add_task(task_fn(|_x: Result<i32, &str>| async move {
Err("Branch B fails at b2")
}))
.depends_on(&b1);
let b3 = dag
.add_task(task_fn(
|x: Result<i32, &str>| async move { x.map(|v| v * 3) },
))
.depends_on(&b2);
// Branch C
let c1 = dag
.add_task(task_fn(|x: i32| async move { x - 1 }))
.depends_on(&root);
let c2 = dag
.add_task(task_fn(|x: i32| async move { x - 2 }))
.depends_on(c1);
let c3 = dag
.add_task(task_fn(|x: i32| async move { x - 3 }))
.depends_on(c2);
// Sink depends on all branches - handle Result from B
let sink = dag
.add_task(task_fn(
|(a, b, c): (i32, Result<i32, &str>, i32)| async move {
match b {
Ok(b_val) => Ok(a + b_val + c),
Err(e) => Err(e),
}
},
))
.depends_on((&a3, &b3, &c3));
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.ok();
// Branch A succeeds
assert_eq!(dag.get(a3).unwrap(), 106);
// Branch B fails at b2
assert_eq!(dag.get(b1).unwrap(), Ok(200));
assert_eq!(dag.get(b2).unwrap(), Err("Branch B fails at b2"));
assert_eq!(dag.get(b3).unwrap(), Err("Branch B fails at b2")); // Propagated error
// Branch C succeeds
assert_eq!(dag.get(c3).unwrap(), 94);
// Sink fails due to branch B
assert_eq!(dag.get(sink).unwrap(), Err("Branch B fails at b2"));
}
+43 -183
View File
@@ -1,54 +1,21 @@
//! Tests for error recovery patterns //! Tests for error recovery patterns
use crate::common::task_fn;
use dagx::{DagResult, DagRunner}; use dagx::{DagResult, DagRunner};
use dagx_test::task_fn;
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use std::sync::Arc; use std::sync::{Arc, Mutex};
#[tokio::test]
async fn test_fallback_on_error() -> DagResult<()> {
let dag = DagRunner::new();
// Primary task that fails
let primary = dag.add_task(task_fn(|_: ()| async {
panic!("Primary task failed");
#[allow(unreachable_code)]
42
}));
// Fallback task
let fallback = dag.add_task(task_fn(|_: ()| async { 100 }));
// Selector that uses fallback if primary fails
let _selector = dag.add_task(task_fn(|_: ()| async {}));
// Run DAG
let _ = dag
.run(|fut| {
tokio::spawn(fut);
})
.await;
// Primary should fail
assert!(dag.get(&primary).is_err());
// Fallback should succeed
assert_eq!(dag.get(&fallback)?, 100);
Ok(())
}
#[tokio::test] #[tokio::test]
async fn test_retry_pattern_simulation() -> DagResult<()> { async fn test_retry_pattern_simulation() -> DagResult<()> {
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let attempt_counter = Arc::new(AtomicUsize::new(0)); let attempt_counter = Arc::new(AtomicUsize::new(0));
// Task that would fail on first attempts (simulated) // Task that would fail on first attempts (simulated)
let task = dag.add_task(task_fn({ let task = dag.add_task(task_fn::<(), _, _>({
let counter = attempt_counter.clone(); let counter = attempt_counter.clone();
move |_: ()| { move |_: ()| {
let counter = counter.clone(); let counter = counter.clone();
async move {
let attempt = counter.fetch_add(1, Ordering::SeqCst); let attempt = counter.fetch_add(1, Ordering::SeqCst);
// In real retry, this would be retried // In real retry, this would be retried
@@ -60,23 +27,21 @@ async fn test_retry_pattern_simulation() -> DagResult<()> {
Ok(42) Ok(42)
} }
}
})); }));
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await?; .await?;
// Task completes with error on first "attempt" // Task completes with error on first "attempt"
assert_eq!(dag.get(&task)?, Err("Simulated failure")); assert_eq!(output.get(task), Err("Simulated failure"));
Ok(()) Ok(())
} }
#[tokio::test] #[tokio::test]
async fn test_circuit_breaker_pattern() -> DagResult<()> { async fn test_circuit_breaker_pattern() -> DagResult<()> {
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let failure_count = Arc::new(AtomicUsize::new(0)); let failure_count = Arc::new(AtomicUsize::new(0));
let circuit_open = Arc::new(AtomicBool::new(false)); let circuit_open = Arc::new(AtomicBool::new(false));
@@ -85,10 +50,9 @@ async fn test_circuit_breaker_pattern() -> DagResult<()> {
.map(|i| { .map(|i| {
let failures = failure_count.clone(); let failures = failure_count.clone();
let circuit = circuit_open.clone(); let circuit = circuit_open.clone();
dag.add_task(task_fn(move |_: ()| { dag.add_task(task_fn::<(), _, _>(move |_: ()| {
let failures = failures.clone(); let failures = failures.clone();
let circuit = circuit.clone(); let circuit = circuit.clone();
async move {
// Check if circuit is open // Check if circuit is open
if circuit.load(Ordering::SeqCst) { if circuit.load(Ordering::SeqCst) {
return Err("Circuit breaker open"); return Err("Circuit breaker open");
@@ -104,201 +68,97 @@ async fn test_circuit_breaker_pattern() -> DagResult<()> {
} }
Ok(i) Ok(i)
}
})) }))
}) })
.collect(); .collect();
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await?; .await?;
// First few tasks fail and open circuit // First few tasks fail and open circuit
for task in &tasks[0..3] { for task in tasks.into_iter().take(4) {
assert!(dag.get(task)?.is_err()); assert!(output.get(task).is_err());
} }
Ok(()) Ok(())
} }
#[tokio::test]
async fn test_partial_recovery_with_defaults() -> DagResult<()> {
let dag = DagRunner::new();
// Some tasks that might fail
let _tasks: Vec<_> = (0..10)
.map(|i| {
dag.add_task(task_fn(move |_: ()| async move {
if i % 3 == 0 {
panic!("Task {} failed", i);
}
i * 10
}))
})
.collect();
// Aggregator that handles failures gracefully
let aggregator = dag.add_task(task_fn(|_: ()| async {
// In real scenario, would check task results and use defaults
vec![0, 10, 20, 0, 40, 50, 0, 70, 80, 0] // 0 for failed tasks
}));
let _ = dag
.run(|fut| {
tokio::spawn(fut);
})
.await;
let result = dag.get(&aggregator)?;
assert_eq!(result.len(), 10);
Ok(())
}
#[tokio::test] #[tokio::test]
async fn test_error_accumulation_pattern() -> DagResult<()> { async fn test_error_accumulation_pattern() -> DagResult<()> {
let dag = DagRunner::new(); let mut dag = DagRunner::new();
let errors = Arc::new(parking_lot::Mutex::new(Vec::new())); let errors = Arc::new(Mutex::new(Vec::new()));
// Tasks that may produce errors // Tasks that may produce errors
let tasks: Vec<_> = (0..10) let mut tasks: Vec<_> = (0..10)
.map(|i| { .map(|i| {
let errors = errors.clone(); let errors = errors.clone();
dag.add_task(task_fn(move |_: ()| { dag.add_task(task_fn::<(), _, _>(move |_: ()| {
let errors = errors.clone(); let errors = errors.clone();
async move {
if i % 2 == 0 { if i % 2 == 0 {
errors.lock().push(format!("Error from task {}", i)); errors
.lock()
.unwrap()
.push(format!("Error from task {}", i));
return Err(format!("Task {} error", i)); return Err(format!("Task {} error", i));
} }
Ok(i) Ok(i)
}
})) }))
}) })
.collect(); .collect();
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await?; .await?;
// Check accumulated errors // Check accumulated errors
let error_list = errors.lock().clone(); let error_list = errors.lock().unwrap().clone();
assert_eq!(error_list.len(), 5); // Even numbered tasks assert_eq!(error_list.len(), 5); // Even numbered tasks
tasks.truncate(2);
let [first, second] = <[_; 2]>::try_from(tasks).map_err(|_e| ()).unwrap();
// Verify specific results // Verify specific results
assert!(dag.get(&tasks[0])?.is_err()); assert!(output.get(first).is_err());
assert_eq!(dag.get(&tasks[1])?, Ok(1)); assert_eq!(output.get(second), Ok(1));
Ok(())
}
#[tokio::test]
async fn test_graceful_degradation() -> DagResult<()> {
let dag = DagRunner::new();
// Critical task
let critical = dag.add_task(task_fn(|_: ()| async { vec![1, 2, 3, 4, 5] }));
// Optional enhancement that fails - returns Result
let _enhancement = dag.add_task(task_fn(|_: ()| async {
Err::<Vec<i32>, &str>("Enhancement failed")
}));
// Final task that works with what's available
let final_task = dag
.add_task(task_fn(|base: Vec<i32>| async move {
// Would combine with enhancement if available
// Here just uses base
base.iter().sum::<i32>()
}))
.depends_on(&critical);
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
// Critical path works
assert_eq!(dag.get(final_task)?, 15);
Ok(()) Ok(())
} }
#[tokio::test] #[tokio::test]
async fn test_error_boundary_isolation() -> DagResult<()> { async fn test_error_boundary_isolation() -> DagResult<()> {
let dag = DagRunner::new(); let mut dag = DagRunner::new();
// Group A: contains an error - use Result // Group A: contains an error - use Result
let a1 = dag.add_task(task_fn(|_: ()| async { 10 })); let a1 = dag.add_task(task_fn::<(), _, _>(|_: ()| 10));
let a2 = dag let a2 = dag
.add_task(task_fn(|_x: i32| async move { .add_task(task_fn::<i32, _, _>(|_x: &i32| {
Err::<i32, &str>("Group A error") Err::<i32, &str>("Group A error")
})) }))
.depends_on(&a1); .depends_on(&a1);
// Group B: should be isolated from Group A's error // Group B: should be isolated from Group A's error
let b1 = dag.add_task(task_fn(|_: ()| async { 20 })); let b1 = dag.add_task(task_fn::<(), _, _>(|_: ()| 20));
let b2 = dag let b2 = dag
.add_task(task_fn(|x: i32| async move { x * 2 })) .add_task(task_fn::<i32, _, _>(|&x: &i32| x * 2))
.depends_on(&b1); .depends_on(&b1);
// Group C: depends on B but not A // Group C: depends on B but not A
let c1 = dag let c1 = dag
.add_task(task_fn(|x: i32| async move { x + 5 })) .add_task(task_fn::<i32, _, _>(|&x: &i32| x + 5))
.depends_on(b2); .depends_on(&b2);
dag.run(|fut| { let mut output = dag
tokio::spawn(fut); .run(|fut| async move { tokio::spawn(fut).await.unwrap() })
})
.await?; .await?;
// Group A fails (returns Err) // Group A fails (returns Err)
assert_eq!(dag.get(a2)?, Err("Group A error")); assert_eq!(output.get(a2), Err("Group A error"));
// Groups B and C succeed // Groups B and C succeed
assert_eq!(dag.get(b2)?, 40); assert_eq!(output.get(b2), 40);
assert_eq!(dag.get(c1)?, 45); assert_eq!(output.get(c1), 45);
Ok(())
}
#[tokio::test]
async fn test_compensation_action() -> DagResult<()> {
let dag = DagRunner::new();
let compensation_triggered = Arc::new(AtomicBool::new(false));
// Action that fails
let action = dag.add_task(task_fn(|_: ()| async {
// Perform some action that fails
panic!("Action failed");
}));
// Compensation task (runs independently)
let compensation = dag.add_task(task_fn({
let triggered = compensation_triggered.clone();
move |_: ()| {
let triggered = triggered.clone();
async move {
triggered.store(true, Ordering::SeqCst);
"Compensation executed"
}
}
}));
let _ = dag
.run(|fut| {
tokio::spawn(fut);
})
.await;
// Action failed
assert!(dag.get(&action).is_err());
// Compensation ran successfully
assert_eq!(dag.get(&compensation)?, "Compensation executed");
assert!(compensation_triggered.load(Ordering::SeqCst));
Ok(()) Ok(())
} }
-164
View File
@@ -1,164 +0,0 @@
// Basic execution tests for DagRunner
use crate::common::task_fn;
use dagx::*;
#[tokio::test]
async fn test_empty_dag() {
let dag = DagRunner::new();
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap(); // Should complete without error
}
#[tokio::test]
async fn test_single_task() {
let dag = DagRunner::new();
let task = dag.add_task(task_fn(|_: ()| async { 42 }));
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
assert_eq!(dag.get(task).unwrap(), 42);
}
#[tokio::test]
async fn test_deep_chain() {
let dag = DagRunner::new();
let a = dag.add_task(task_fn(|_: ()| async { 1 }));
let b = dag
.add_task(task_fn(|x: i32| async move { x + 1 }))
.depends_on(&a);
let c = dag
.add_task(task_fn(|x: i32| async move { x + 1 }))
.depends_on(b);
let d = dag
.add_task(task_fn(|x: i32| async move { x + 1 }))
.depends_on(c);
let e = dag
.add_task(task_fn(|x: i32| async move { x + 1 }))
.depends_on(d);
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
// Note: a, b, c, d are not sinks (they have dependents in the chain)
// Only e is a sink and can be retrieved
assert_eq!(dag.get(e).unwrap(), 5);
}
#[tokio::test]
async fn test_wide_parallel() {
let dag = DagRunner::new();
let tasks: Vec<_> = (0..10)
.map(|i| dag.add_task(task_fn(move |_: ()| async move { i * 2 })))
.collect();
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
for (i, task) in tasks.iter().enumerate() {
assert_eq!(dag.get(task).unwrap(), i * 2);
}
}
#[tokio::test]
async fn test_diamond_dependency() {
let dag = DagRunner::new();
let a = dag.add_task(task_fn(|_: ()| async { 10 }));
let b = dag
.add_task(task_fn(|x: i32| async move { x + 5 }))
.depends_on(&a);
let c = dag
.add_task(task_fn(|x: i32| async move { x * 2 }))
.depends_on(&a);
let d = dag
.add_task(task_fn(|(x, y): (i32, i32)| async move { x + y }))
.depends_on((&b, &c));
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
// Note: a, b, and c are not sinks (they have dependents)
// Only d is a sink and can be retrieved
assert_eq!(dag.get(d).unwrap(), 35); // 15 + 20
}
#[derive(Clone)]
struct User {
name: String,
age: u32,
}
#[tokio::test]
async fn test_different_output_types() {
let dag = DagRunner::new();
let string_task = dag.add_task(task_fn(|_: ()| async { "hello".to_string() }));
let vec_task = dag.add_task(task_fn(|_: ()| async { vec![1, 2, 3] }));
let struct_task = dag.add_task(task_fn(|_: ()| async {
User {
name: "Alice".to_string(),
age: 30,
}
}));
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
assert_eq!(&dag.get(string_task).unwrap(), "hello");
assert_eq!(dag.get(vec_task).unwrap(), vec![1, 2, 3]);
let user = dag.get(struct_task).unwrap();
assert_eq!(user.name, "Alice");
assert_eq!(user.age, 30);
}
#[tokio::test]
async fn test_multiple_sinks() {
let dag = DagRunner::new();
let source = dag.add_task(task_fn(|_: ()| async { 10 }));
// Branch 1
let branch1 = dag
.add_task(task_fn(|x: i32| async move { x * 2 }))
.depends_on(&source);
let sink1 = dag
.add_task(task_fn(|x: i32| async move { x + 1 }))
.depends_on(branch1);
// Branch 2
let branch2 = dag
.add_task(task_fn(|x: i32| async move { x + 5 }))
.depends_on(&source);
let sink2 = dag
.add_task(task_fn(|x: i32| async move { x * 3 }))
.depends_on(branch2);
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap(); // Should wait for both sinks
assert_eq!(dag.get(sink1).unwrap(), 21); // (10 * 2) + 1
assert_eq!(dag.get(sink2).unwrap(), 45); // (10 + 5) * 3
}
-87
View File
@@ -1,87 +0,0 @@
// Example pattern tests for DagRunner
use crate::common::task_fn;
use crate::common::tasks::*;
use dagx::*;
#[tokio::test]
async fn test_example_a_fan_out() {
let dag = DagRunner::new();
let base = dag.add_task(Seed::new(10)); // Input=() → no depends_on, returns TaskHandle
let plus1 = dag.add_task(Inc::new()).depends_on(&base); // 11
let plus2 = dag.add_task(Inc::new()).depends_on(&base); // 11
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap(); // SPEC API with spawner
assert_eq!(dag.get(plus1).unwrap(), 11);
assert_eq!(dag.get(plus2).unwrap(), 11);
}
#[tokio::test]
async fn test_example_b_fan_in() {
let dag = DagRunner::new();
let s = dag.add_task(task_fn(|_: ()| async { "hi".to_owned() }));
let n = dag.add_task(task_fn(|_: ()| async { 42usize }));
let f = dag.add_task(task_fn(|_: ()| async { true }));
let output = dag.add_task(OutputResult::new()).depends_on((&s, &n, &f));
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
assert_eq!(&dag.get(output).unwrap(), "hi:42:true");
}
#[tokio::test]
async fn test_example_c_many_to_many() {
let dag = DagRunner::new();
let x = dag.add_task(Seed::new(2));
let y = dag.add_task(Seed::new(3));
let z = dag.add_task(Seed::new(5));
let sum_xy = dag.add_task(Add::new()).depends_on((&x, &y)); // 5
let prod_yz = dag.add_task(Mul::new()).depends_on((&y, &z)); // 15
let total = dag.add_task(Add::new()).depends_on((&sum_xy, &prod_yz)); // 20
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
assert_eq!(dag.get(total).unwrap(), 20);
}
#[tokio::test]
async fn test_single_input_forms() {
let dag = DagRunner::new();
let s = dag.add_task(task_fn(|_: ()| async { "hello".to_owned() }));
// Preferred form: &node
let upper1 = dag
.add_task(task_fn(|s: String| async move { s.to_uppercase() }))
.depends_on(&s);
// Also allowed: (&node,)
let upper2 = dag
.add_task(task_fn(|s: String| async move { s.to_uppercase() }))
.depends_on((&s,));
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
assert_eq!(&dag.get(upper1).unwrap(), "HELLO");
assert_eq!(&dag.get(upper2).unwrap(), "HELLO");
}
-364
View File
@@ -1,364 +0,0 @@
//! Integration tests for end-to-end functionality
use dagx::{DagResult, DagRunner, Task};
use crate::common::task_fn;
use std::time::Instant;
use tokio::time::{sleep, Duration};
// === Shared Test Tasks ===
struct LoadValue {
value: i32,
}
impl LoadValue {
fn new(value: i32) -> Self {
Self { value }
}
}
#[dagx::task]
impl LoadValue {
async fn run(&self) -> i32 {
self.value // Read-only state: use &self
}
}
struct Multiply {
factor: i32,
}
impl Multiply {
fn new(factor: i32) -> Self {
Self { factor }
}
}
#[dagx::task]
impl Multiply {
async fn run(&self, input: &i32) -> i32 {
input * self.factor // Read-only state: use &self
}
}
struct Add;
#[dagx::task]
impl Add {
async fn run(a: &i32, b: &i32) -> i32 {
a + b // Stateless: no self needed
}
}
// === Integration Tests ===
#[tokio::test]
async fn integration_test_etl_pipeline() -> DagResult<()> {
// Simulate an Extract-Transform-Load pipeline
let dag = DagRunner::new();
// Extract: Load data from sources
let source1 = dag.add_task(LoadValue::new(10));
let source2 = dag.add_task(LoadValue::new(20));
let source3 = dag.add_task(LoadValue::new(30));
// Transform: Process each source
let transform1 = dag.add_task(Multiply::new(2)).depends_on(&source1);
let transform2 = dag.add_task(Multiply::new(3)).depends_on(&source2);
let transform3 = dag.add_task(Multiply::new(4)).depends_on(&source3);
// Load: Aggregate results
let combine12 = dag.add_task(Add).depends_on((&transform1, &transform2));
let final_result = dag.add_task(Add).depends_on((&combine12, &transform3));
// Execute pipeline
dag.run(|fut| { tokio::spawn(fut); }).await?;
// Verify
let result = dag.get(final_result)?;
assert_eq!(result, (10 * 2) + (20 * 3) + (30 * 4)); // 20 + 60 + 120 = 200
Ok(())
}
#[tokio::test]
async fn integration_test_multi_layer_dag() -> DagResult<()> {
// Complex multi-layer DAG
let dag = DagRunner::new();
// Layer 1: Sources
let a = dag.add_task(LoadValue::new(1));
let b = dag.add_task(LoadValue::new(2));
let c = dag.add_task(LoadValue::new(3));
// Layer 2: First transformations
let ab = dag.add_task(Add).depends_on((&a, &b));
let bc = dag.add_task(Add).depends_on((&b, &c));
// Layer 3: Second transformations
let double_ab = dag.add_task(Multiply::new(2)).depends_on(ab);
let double_bc = dag.add_task(Multiply::new(2)).depends_on(bc);
// Layer 4: Final result
let final_result = dag.add_task(Add).depends_on((&double_ab, &double_bc));
dag.run(|fut| { tokio::spawn(fut); }).await?;
let result = dag.get(final_result)?;
// (1+2)*2 + (2+3)*2 = 6 + 10 = 16
assert_eq!(result, 16);
Ok(())
}
#[tokio::test]
async fn integration_test_multiple_sinks() -> DagResult<()> {
// DAG with multiple output sinks
let dag = DagRunner::new();
let source = dag.add_task(LoadValue::new(10));
// Multiple independent branches
let branch1 = dag.add_task(Multiply::new(2)).depends_on(&source);
let branch2 = dag.add_task(Multiply::new(3)).depends_on(&source);
let branch3 = dag.add_task(Multiply::new(4)).depends_on(&source);
dag.run(|fut| { tokio::spawn(fut); }).await?;
// All branches should complete
assert_eq!(dag.get(branch1)?, 20);
assert_eq!(dag.get(branch2)?, 30);
assert_eq!(dag.get(branch3)?, 40);
Ok(())
}
#[tokio::test]
async fn integration_test_diamond_cascade() -> DagResult<()> {
// Multiple diamond patterns in sequence
let dag = DagRunner::new();
// Diamond 1
let source1 = dag.add_task(LoadValue::new(10));
let left1 = dag.add_task(Multiply::new(2)).depends_on(&source1);
let right1 = dag.add_task(Multiply::new(3)).depends_on(&source1);
let sink1 = dag.add_task(Add).depends_on((&left1, &right1));
// Diamond 2 (depends on Diamond 1)
let left2 = dag.add_task(Multiply::new(2)).depends_on(sink1);
let right2 = dag.add_task(Multiply::new(3)).depends_on(sink1);
let sink2 = dag.add_task(Add).depends_on((&left2, &right2));
dag.run(|fut| { tokio::spawn(fut); }).await?;
let result = dag.get(sink2)?;
// Diamond 1: (10*2) + (10*3) = 50
// Diamond 2: (50*2) + (50*3) = 250
assert_eq!(result, 250);
Ok(())
}
#[tokio::test]
async fn integration_test_wide_and_deep() -> DagResult<()> {
// Combination of wide (parallel) and deep (sequential) patterns
let dag = DagRunner::new();
// Wide: 5 parallel sources
let sources: Vec<_> = (0..5)
.map(|i| dag.add_task(LoadValue::new(i * 10)))
.collect();
// Deep: Chain each source through 3 transformations
let mut final_nodes = Vec::new();
for source in &sources {
let step1 = dag.add_task(Multiply::new(2)).depends_on(source);
let step2 = dag
.add_task(task_fn(|x: i32| async move { x + 5 }))
.depends_on(step1);
let step3 = dag.add_task(Multiply::new(3)).depends_on(step2);
final_nodes.push(step3);
}
dag.run(|fut| { tokio::spawn(fut); }).await?;
// Verify each chain
for (i, node) in final_nodes.iter().enumerate() {
let expected = ((i as i32 * 10) * 2 + 5) * 3;
assert_eq!(dag.get(node)?, expected);
}
Ok(())
}
#[tokio::test]
async fn integration_test_error_recovery() -> DagResult<()> {
// Test that DAG handles normal execution gracefully
let dag = DagRunner::new();
let source = dag.add_task(LoadValue::new(42));
let transform = dag.add_task(Multiply::new(2)).depends_on(&source);
dag.run(|fut| { tokio::spawn(fut); }).await?;
// Should be able to access results
assert_eq!(dag.get(transform)?, 84);
// Getting valid task should return ok
let result = dag.get(transform);
assert!(result.is_ok());
Ok(())
}
#[tokio::test]
async fn integration_test_large_scale() -> DagResult<()> {
// Large-scale integration test
let dag = DagRunner::new();
// Create 100 sources
let sources: Vec<_> = (0..100).map(|i| dag.add_task(LoadValue::new(i))).collect();
// Create 50 aggregation tasks (each combines 2 sources)
let aggregations: Vec<_> = (0..50)
.map(|i| {
dag.add_task(Add)
.depends_on((&sources[i * 2], &sources[i * 2 + 1]))
})
.collect();
// Create 25 second-level aggregations
let second_level: Vec<_> = (0..25)
.map(|i| {
dag.add_task(Add)
.depends_on((&aggregations[i * 2], &aggregations[i * 2 + 1]))
})
.collect();
dag.run(|fut| { tokio::spawn(fut); }).await?;
// Verify some results
assert_eq!(dag.get(aggregations[0])?, 1); // First two
assert_eq!(dag.get(second_level[0])?, 1 + (2 + 3)); // First four
Ok(())
}
#[tokio::test]
async fn integration_test_reused_values() -> DagResult<()> {
// Test that a single source can be used by multiple consumers
let dag = DagRunner::new();
let source = dag.add_task(LoadValue::new(100));
// Use source in multiple different ways
let double = dag.add_task(Multiply::new(2)).depends_on(&source);
let triple = dag.add_task(Multiply::new(3)).depends_on(&source);
let quadruple = dag.add_task(Multiply::new(4)).depends_on(&source);
// Combine some of the results
let combined = dag.add_task(Add).depends_on((&double, &triple));
dag.run(|fut| { tokio::spawn(fut); }).await?;
assert_eq!(dag.get(double)?, 200);
assert_eq!(dag.get(triple)?, 300);
assert_eq!(dag.get(quadruple)?, 400);
assert_eq!(dag.get(combined)?, 500);
Ok(())
}
#[tokio::test]
async fn integration_test_parallel_execution_timing() -> DagResult<()> {
// Test that independent tasks actually run in parallel by measuring execution time
let dag = DagRunner::new();
let sleep_duration = Duration::from_millis(100);
let num_parallel_tasks = 5;
// Create 5 independent tasks that each sleep for 100ms
let tasks: Vec<_> = (0..num_parallel_tasks)
.map(|i| {
dag.add_task(task_fn(move |_: ()| async move {
sleep(sleep_duration).await;
i
}))
})
.collect();
let start = Instant::now();
dag.run(|fut| { tokio::spawn(fut); }).await?;
let elapsed = start.elapsed();
// Verify all tasks completed
for (i, task) in tasks.iter().enumerate() {
assert_eq!(dag.get(task)?, i);
}
// If tasks ran sequentially, it would take 5 * 100ms = 500ms
// With parallelism, it should take ~100ms (plus overhead)
// We'll check it's less than 300ms (generous margin for CI environments)
assert!(
elapsed < Duration::from_millis(300),
"Parallel execution took {:?}, expected < 300ms (sequential would be 500ms)",
elapsed
);
Ok(())
}
#[tokio::test]
async fn integration_test_parallel_vs_sequential_timing() -> DagResult<()> {
// Test mixed parallel and sequential execution patterns
let dag = DagRunner::new();
let sleep_duration = Duration::from_millis(50);
// Layer 1: 4 parallel tasks (should take ~50ms)
let parallel_tasks: Vec<_> = (0..4)
.map(|i| {
dag.add_task(task_fn(move |_: ()| async move {
sleep(sleep_duration).await;
i * 10
}))
})
.collect();
// Layer 2: Sequential chain of 3 tasks depending on the first parallel task
// (should add ~150ms)
let seq1 = dag.add_task(task_fn(move |x: i32| async move {
sleep(sleep_duration).await;
x + 1
})).depends_on(&parallel_tasks[0]);
let seq2 = dag.add_task(task_fn(move |x: i32| async move {
sleep(sleep_duration).await;
x + 1
})).depends_on(seq1);
let seq3 = dag.add_task(task_fn(move |x: i32| async move {
sleep(sleep_duration).await;
x + 1
})).depends_on(seq2);
let start = Instant::now();
dag.run(|fut| { tokio::spawn(fut); }).await?;
let elapsed = start.elapsed();
// Verify results
assert_eq!(dag.get(seq3)?, 3); // 0 + 1 + 1 + 1
// Expected time: ~200ms (4 layers of 50ms each: parallel layer + 3 sequential)
// If everything ran sequentially: 7 * 50ms = 350ms
// We'll check it's less than 300ms
assert!(
elapsed < Duration::from_millis(300),
"Execution took {:?}, expected < 300ms (fully sequential would be 350ms)",
elapsed
);
// Also verify it took more than just the parallel layer
assert!(
elapsed >= Duration::from_millis(150),
"Execution took {:?}, expected >= 150ms (should include sequential chain)",
elapsed
);
Ok(())
}
+176 -4
View File
@@ -1,5 +1,177 @@
// Test module for execution tests // Basic execution tests for DagRunner
use dagx_test::task_fn;
mod basic; use dagx::*;
mod examples;
mod results; #[tokio::test]
async fn test_empty_dag() {
let dag = DagRunner::new();
let _ = dag
.run(|fut| async move { tokio::spawn(fut).await.unwrap() })
.await
.unwrap(); // Should complete without error
}
#[tokio::test]
async fn test_single_task() {
let mut dag = DagRunner::new();
let task = dag.add_task(task_fn::<(), _, _>(|_: ()| 42));
let mut output = dag
.run(|fut| async move { tokio::spawn(fut).await.unwrap() })
.await
.unwrap();
assert_eq!(output.get(task), 42);
}
#[tokio::test]
async fn test_wide_parallel() {
let mut dag = DagRunner::new();
let tasks: Vec<_> = (0..10)
.map(|i| dag.add_task(task_fn::<(), _, _>(move |_: ()| i * 2)))
.collect();
let mut output = dag
.run(|fut| async move { tokio::spawn(fut).await.unwrap() })
.await
.unwrap();
for (i, task) in tasks.into_iter().enumerate() {
assert_eq!(output.get(task), i * 2);
}
}
#[tokio::test]
async fn test_diamond_dependency() {
let mut dag = DagRunner::new();
let a = dag.add_task(task_fn::<(), _, _>(|_: ()| 10));
let b = dag
.add_task(task_fn::<i32, _, _>(|&x: &i32| x + 5))
.depends_on(&a);
let c = dag
.add_task(task_fn::<i32, _, _>(|&x: &i32| x * 2))
.depends_on(&a);
let d = dag
.add_task(task_fn::<(i32, i32), _, _>(|(x, y): (&i32, &i32)| x + y))
.depends_on((&b, &c));
let mut output = dag
.run(|fut| async move { tokio::spawn(fut).await.unwrap() })
.await
.unwrap();
// Note: a, b, and c are not sinks (they have dependents)
// Only d is a sink and can be retrieved
assert_eq!(output.get(d), 35); // 15 + 20
}
#[derive(Clone)]
struct User {
name: String,
age: u32,
}
#[tokio::test]
async fn test_different_output_types() {
let mut dag = DagRunner::new();
let string_task = dag.add_task(task_fn::<(), _, _>(|_: ()| "hello".to_string()));
let vec_task = dag.add_task(task_fn::<(), _, _>(|_: ()| vec![1, 2, 3]));
let struct_task = dag.add_task(task_fn::<(), _, _>(|_: ()| User {
name: "Alice".to_string(),
age: 30,
}));
let mut output = dag
.run(|fut| async move { tokio::spawn(fut).await.unwrap() })
.await
.unwrap();
assert_eq!(&output.get(string_task), "hello");
assert_eq!(output.get(vec_task), vec![1, 2, 3]);
let user = output.get(struct_task);
assert_eq!(user.name, "Alice");
assert_eq!(user.age, 30);
}
#[tokio::test]
async fn test_multiple_sinks() {
let mut dag = DagRunner::new();
let source = dag.add_task(task_fn::<(), _, _>(|_: ()| 10));
// Branch 1
let branch1 = dag
.add_task(task_fn::<i32, _, _>(|&x: &i32| x * 2))
.depends_on(&source);
let sink1 = dag
.add_task(task_fn::<i32, _, _>(|&x: &i32| x + 1))
.depends_on(&branch1);
// Branch 2
let branch2 = dag
.add_task(task_fn::<i32, _, _>(|&x: &i32| x + 5))
.depends_on(&source);
let sink2 = dag
.add_task(task_fn::<i32, _, _>(|&x: &i32| x * 3))
.depends_on(&branch2);
let mut output = dag
.run(|fut| async move { tokio::spawn(fut).await.unwrap() })
.await
.unwrap(); // Should wait for both sinks
assert_eq!(output.get(sink1), 21); // (10 * 2) + 1
assert_eq!(output.get(sink2), 45); // (10 + 5) * 3
}
#[tokio::test]
async fn test_single_task_inline_path() {
// Test single-task layer inline execution path (line 367-410 in runner.rs)
let mut dag = DagRunner::new();
// Single task in its own layer
let t1 = dag.add_task(task_fn::<(), _, _>(|_: ()| 10));
let t2 = dag
.add_task(task_fn::<i32, _, _>(|&x: &i32| x * 2))
.depends_on(&t1);
let t3 = dag
.add_task(task_fn::<i32, _, _>(|&x: &i32| x + 5))
.depends_on(&t2);
let mut output = dag
.run(|fut| async move { tokio::spawn(fut).await.unwrap() })
.await
.unwrap();
assert_eq!(output.get(t3), 25); // (10 * 2) + 5
}
#[tokio::test]
async fn test_multi_consumer_fanout() {
// Test fanout with multiple consumers from one producer
let mut dag = DagRunner::new();
let producer = dag.add_task(task_fn::<(), _, _>(|_: ()| 42));
// Multiple consumers
let c1 = dag
.add_task(task_fn::<i32, _, _>(|&x: &i32| x * 2))
.depends_on(&producer);
let c2 = dag
.add_task(task_fn::<i32, _, _>(|&x: &i32| x + 10))
.depends_on(&producer);
let c3 = dag
.add_task(task_fn::<i32, _, _>(|&x: &i32| x - 5))
.depends_on(&producer);
let mut output = dag
.run(|fut| async move { tokio::spawn(fut).await.unwrap() })
.await
.unwrap();
assert_eq!(output.get(c1), 84);
assert_eq!(output.get(c2), 52);
assert_eq!(output.get(c3), 37);
}
-127
View File
@@ -1,127 +0,0 @@
// Result and error handling tests for DagRunner
use crate::common::task_fn;
use crate::common::tasks::Counter;
use dagx::*;
#[tokio::test]
async fn test_result_type() {
let dag = DagRunner::new();
let success_task = dag.add_task(task_fn(|_: ()| async { Ok::<i32, String>(42) }));
let error_task = dag.add_task(task_fn(|_: ()| async {
Err::<i32, String>("error".to_string())
}));
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
assert_eq!(dag.get(success_task).unwrap(), Ok(42));
assert_eq!(dag.get(error_task).unwrap(), Err("error".to_string()));
}
#[tokio::test]
async fn test_dag_result_type_works() {
let dag = DagRunner::new();
let task = dag.add_task(task_fn(|_: ()| async { 100 }));
// Test that DagResult<()> works with ? operator pattern
async fn run_dag(dag: &DagRunner) -> DagResult<()> {
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
Ok(())
}
let result = run_dag(&dag).await;
assert!(result.is_ok());
assert_eq!(dag.get(task).unwrap(), 100);
}
#[tokio::test]
async fn test_task_not_executed_error() {
let dag = DagRunner::new();
let task = dag.add_task(task_fn(|_: ()| async { 42 }));
// Try to get result before running - should error
let result = dag.get(task);
assert!(result.is_err());
match result {
Err(DagError::ResultNotFound { task_id }) => {
assert_eq!(task_id, 0); // First task has ID 0
}
_ => panic!("Expected ResultNotFound error"),
}
}
#[tokio::test]
async fn test_error_messages_are_helpful() {
// Test ResultNotFound error message
let dag = DagRunner::new();
let task = dag.add_task(task_fn(|_: ()| async { 42 }));
let result = dag.get(task);
assert!(result.is_err());
let err_msg = format!("{}", result.unwrap_err());
assert!(err_msg.contains("Result not found"));
assert!(err_msg.contains("task #0")); // Task ID
assert!(err_msg.contains("Call dag.run()"));
}
#[tokio::test]
async fn test_successful_execution_returns_ok() {
let dag = DagRunner::new();
let task = dag.add_task(task_fn(|_: ()| async { 42 }));
// Should return Ok(())
let result = dag
.run(|fut| {
tokio::spawn(fut);
})
.await;
assert!(result.is_ok());
// Should be able to get value
assert_eq!(dag.get(task).unwrap(), 42);
}
#[tokio::test]
async fn test_stateful_tasks() {
let dag = DagRunner::new();
let input = dag.add_task(task_fn(|_: ()| async { 5 }));
let counter = dag.add_task(Counter::new(10)).depends_on(&input);
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
assert_eq!(dag.get(counter).unwrap(), 15); // 10 + 5
}
// Cycle detection test removed - it required access to private fields
// The public API correctly prevents creating cycles, so this is tested implicitly
#[tokio::test]
async fn test_task_fn_with_captured_state() {
let dag = DagRunner::new();
let multiplier = 7;
let base = dag.add_task(task_fn(|_: ()| async { 5 }));
let scaled = dag
.add_task(task_fn(move |x: i32| async move { x * multiplier }))
.depends_on(&base);
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
assert_eq!(dag.get(scaled).unwrap(), 35);
}
+277
View File
@@ -0,0 +1,277 @@
//! Tests for forced execution interleaving patterns
use dagx::{task, DagResult, DagRunner};
use dagx_test::task_fn;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::{Arc, Mutex};
use tokio::sync::Barrier;
#[tokio::test]
async fn test_forced_interleaving_with_barriers() -> DagResult<()> {
// Force specific interleaving using barriers
let mut dag = DagRunner::new();
let barrier = Arc::new(Barrier::new(3));
let order = Arc::new(Mutex::new(Vec::new()));
struct InterleaveTask {
barrier: Arc<Barrier>,
order: Arc<Mutex<Vec<String>>>,
prefix: &'static str,
value: i32,
}
#[task]
impl InterleaveTask {
async fn run(&self) -> i32 {
self.order
.lock()
.unwrap()
.push(format!("{}_start", self.prefix));
self.barrier.wait().await;
self.order
.lock()
.unwrap()
.push(format!("{}_end", self.prefix));
self.value
}
}
// Three parallel tasks that synchronize at barrier
let t1 = dag.add_task(InterleaveTask {
barrier: barrier.clone(),
order: order.clone(),
prefix: "t1",
value: 1,
});
let t2 = dag.add_task(InterleaveTask {
barrier: barrier.clone(),
order: order.clone(),
prefix: "t2",
value: 2,
});
let t3 = dag.add_task(InterleaveTask {
barrier: barrier.clone(),
order: order.clone(),
prefix: "t2",
value: 3,
});
// Collector
let collector = dag
.add_task(task_fn::<(i32, i32, i32), _, _>(
|(a, b, c): (&i32, &i32, &i32)| a + b + c,
))
.depends_on((&t1, &t2, &t3));
let mut output = dag
.run(|fut| async move { tokio::spawn(fut).await.unwrap() })
.await?;
assert_eq!(output.get(collector), 6);
// Check interleaving pattern
let events = order.lock().unwrap().clone();
assert_eq!(events.len(), 6);
// All starts should come before all ends due to barrier
let start_count = events[0..3]
.iter()
.filter(|e| e.ends_with("_start"))
.count();
assert_eq!(start_count, 3);
Ok(())
}
#[tokio::test]
async fn test_alternating_execution_pattern() -> DagResult<()> {
// Create an alternating execution pattern
let mut dag = DagRunner::new();
let turn = Arc::new(AtomicUsize::new(0));
let events: Arc<Mutex<Vec<String>>> = Arc::new(Mutex::new(Vec::new()));
struct FirstA {
turn: Arc<AtomicUsize>,
events: Arc<Mutex<Vec<String>>>,
}
#[task]
impl FirstA {
async fn run(&self) -> i32 {
while self.turn.load(Ordering::SeqCst) != 0 {
tokio::task::yield_now().await;
}
self.events.lock().unwrap().push("A0".to_string());
self.turn.store(1, Ordering::SeqCst);
0
}
}
let mut chain_a = dag.add_task(FirstA {
turn: turn.clone(),
events: events.clone(),
});
struct FirstB {
turn: Arc<AtomicUsize>,
events: Arc<Mutex<Vec<String>>>,
}
#[task]
impl FirstB {
async fn run(&self) -> i32 {
while self.turn.load(Ordering::SeqCst) != 1 {
tokio::task::yield_now().await;
}
self.events.lock().unwrap().push("B0".to_string());
self.turn.store(2, Ordering::SeqCst);
0
}
}
let mut chain_b = dag.add_task(FirstB {
turn: turn.clone(),
events: events.clone(),
});
struct ChainA {
turn: Arc<AtomicUsize>,
events: Arc<Mutex<Vec<String>>>,
idx: usize,
}
#[task]
impl ChainA {
async fn run(&self, x: &i32) -> i32 {
while self.turn.load(Ordering::SeqCst) != self.idx * 2 {
tokio::task::yield_now().await;
}
self.events.lock().unwrap().push(format!("A{}", self.idx));
self.turn.store(self.idx * 2 + 1, Ordering::SeqCst);
x + 1
}
}
struct ChainB {
turn: Arc<AtomicUsize>,
events: Arc<Mutex<Vec<String>>>,
idx: usize,
}
#[task]
impl ChainB {
async fn run(&self, x: &i32) -> i32 {
while self.turn.load(Ordering::SeqCst) != self.idx * 2 + 1 {
tokio::task::yield_now().await;
}
self.events.lock().unwrap().push(format!("B{}", self.idx));
self.turn.store(self.idx * 2 + 2, Ordering::SeqCst);
x + 1
}
}
for i in 1..5 {
chain_a = dag
.add_task(ChainA {
events: events.clone(),
turn: turn.clone(),
idx: i,
})
.depends_on(&chain_a);
chain_b = dag
.add_task(ChainB {
turn: turn.clone(),
events: events.clone(),
idx: i,
})
.depends_on(&chain_b);
}
let _output = dag
.run(|fut| async move { tokio::spawn(fut).await.unwrap() })
.await?;
// Check alternating pattern
let event_list = events.lock().unwrap().clone();
assert_eq!(
event_list,
vec!["A0", "B0", "A1", "B1", "A2", "B2", "A3", "B3", "A4", "B4"]
);
Ok(())
}
#[tokio::test]
async fn test_layered_interleaving() -> DagResult<()> {
// Test interleaving across DAG layers
let mut dag = DagRunner::new();
let layer_active = Arc::new(AtomicUsize::new(0));
struct Layer1Task {
active: Arc<AtomicUsize>,
idx: usize,
}
#[task]
impl Layer1Task {
async fn run(&self) -> i32 {
assert_eq!(self.active.fetch_add(1, Ordering::SeqCst), self.idx);
tokio::task::yield_now().await;
self.active.fetch_sub(1, Ordering::SeqCst);
self.idx as i32
}
}
// Layer 1: 4 tasks
let layer1: Vec<_> = (0..4)
.map(|i| {
dag.add_task(Layer1Task {
active: layer_active.clone(),
idx: i,
})
})
.collect();
struct Layer23Task {
active: Arc<AtomicUsize>,
}
#[task]
impl Layer23Task {
async fn run(&self, a: &i32, b: &i32) -> i32 {
self.active.fetch_add(1, Ordering::SeqCst);
tokio::task::yield_now().await;
self.active.fetch_sub(1, Ordering::SeqCst);
a + b
}
}
// Layer 2: 2 tasks, each depends on 2 from layer1
let layer2: Vec<_> = (0..2)
.map(|i| {
let idx = i * 2;
dag.add_task(Layer23Task {
active: layer_active.clone(),
})
.depends_on((&layer1[idx], &layer1[idx + 1]))
})
.collect();
// Layer 3: 1 task depends on both layer2
let layer3 = dag
.add_task(Layer23Task {
active: layer_active.clone(),
})
.depends_on((&layer2[0], &layer2[1]));
let mut output = dag
.run(|fut| async move { tokio::spawn(fut).await.unwrap() })
.await?;
// Sum of 0+1+2+3 = 6
assert_eq!(output.get(layer3), 6);
Ok(())
}
+2 -8
View File
@@ -4,17 +4,11 @@
#![allow(clippy::needless_borrow)] #![allow(clippy::needless_borrow)]
#![allow(clippy::redundant_clone)] #![allow(clippy::redundant_clone)]
#[path = "common/mod.rs"]
mod common;
mod boundaries; mod boundaries;
mod dependencies; mod dependencies;
mod errors; mod errors;
mod execution; mod execution;
mod interleaving;
mod parallelism; mod parallelism;
mod patterns;
mod performance;
mod runtimes; mod runtimes;
mod task_patterns; mod tracing;
mod timing;
mod type_safety;
-432
View File
@@ -1,432 +0,0 @@
//! Concurrent access tests
use crate::common::task_fn;
use dagx::DagRunner;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::Arc;
use std::time::Duration;
use tokio::task::JoinSet;
#[tokio::test]
async fn test_concurrent_run_calls() {
// Test that multiple run() calls are handled correctly
let dag = Arc::new(DagRunner::new());
// Add some tasks
let tasks: Vec<_> = (0..10)
.map(|i| dag.add_task(task_fn(move |_: ()| async move { i })))
.collect();
// Try to run the DAG multiple times concurrently
let mut handles = JoinSet::new();
for _ in 0..5 {
let dag = Arc::clone(&dag);
handles.spawn(async move {
dag.run(|fut| {
tokio::spawn(fut);
})
.await
});
}
// Only one run should succeed, others should fail with concurrent execution error
let mut success_count = 0;
let mut error_count = 0;
while let Some(result) = handles.join_next().await {
match result.unwrap() {
Ok(_) => success_count += 1,
Err(_) => error_count += 1,
}
}
// Exactly one should succeed, the rest should fail
assert_eq!(success_count, 1);
assert_eq!(error_count, 4);
// Verify results are consistent from the successful run
for (i, task) in tasks.iter().enumerate() {
assert_eq!(dag.get(task).unwrap(), i);
}
}
#[tokio::test]
async fn test_concurrent_execution_and_building() {
// Test that we can build and execute concurrently
let dag = Arc::new(DagRunner::new());
// Start with some initial tasks
let initial: Vec<_> = (0..10)
.map(|i| dag.add_task(task_fn(move |_: ()| async move { i })))
.collect();
// Execute the initial DAG
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
// Now try to add more tasks while also reading results
let mut handles = JoinSet::new();
// Reader tasks
for task in &initial {
let dag = Arc::clone(&dag);
let task_handle: dagx::TaskHandle<i32> = task.into();
handles.spawn(async move { dag.get(task_handle).unwrap() });
}
// Builder tasks (adding new nodes)
for i in 10..20 {
let dag = Arc::clone(&dag);
handles.spawn(async move {
dag.add_task(task_fn(move |_: ()| async move { i }));
i
});
}
// Wait for all operations
while let Some(result) = handles.join_next().await {
result.unwrap();
}
// Execute again with the new tasks
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
}
#[tokio::test]
async fn test_dag_builder_thread_safety() {
// Test that the DAG builder is thread-safe
let dag = Arc::new(DagRunner::new());
let counter = Arc::new(AtomicUsize::new(0));
let mut handles = JoinSet::new();
// Spawn tasks that build complex sub-graphs concurrently
for thread_id in 0..10 {
let dag = Arc::clone(&dag);
let counter = Arc::clone(&counter);
handles.spawn(async move {
// Each thread creates a small pipeline
let source = dag.add_task(task_fn(move |_: ()| async move { thread_id }));
let transform1 = dag
.add_task(task_fn(move |x: i32| async move { x * 10 }))
.depends_on(&source);
let transform2 = dag
.add_task(task_fn(move |x: i32| async move { x + thread_id }))
.depends_on(transform1);
counter.fetch_add(1, Ordering::SeqCst);
transform2
});
}
let mut final_tasks = vec![];
while let Some(result) = handles.join_next().await {
final_tasks.push(result.unwrap());
}
// Verify all threads completed
assert_eq!(counter.load(Ordering::SeqCst), 10);
// Execute the combined DAG
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
// Verify each pipeline computed correctly
for task in final_tasks.iter() {
let result = dag.get(task).unwrap();
// Each pipeline: thread_id * 10 + thread_id
assert!((0..100).contains(&result));
}
}
#[tokio::test]
async fn test_concurrent_get_operations() {
// Test concurrent get() operations
let dag = Arc::new(DagRunner::new());
// Create and execute tasks
let tasks: Vec<_> = (0..100)
.map(|i| dag.add_task(task_fn(move |_: ()| async move { i })))
.collect();
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
// Now access results concurrently
let mut handles = JoinSet::new();
for task in tasks.iter() {
let dag = Arc::clone(&dag);
let task_handle: dagx::TaskHandle<i32> = task.into();
handles.spawn(async move { dag.get(task_handle).unwrap() });
}
// Collect and verify results
let mut results = vec![];
while let Some(result) = handles.join_next().await {
results.push(result.unwrap());
}
// Results should contain all values 0..100
results.sort();
for (i, val) in results.iter().enumerate() {
assert_eq!(*val, i as i32);
}
}
#[tokio::test]
async fn test_no_race_in_dependency_tracking() {
// Test that concurrent dependency modifications don't cause races
let dag = Arc::new(DagRunner::new());
let source = dag.add_task(task_fn(|_: ()| async { 42 }));
let mut handles = JoinSet::new();
// Multiple threads create tasks depending on the same source
let source_handle: dagx::TaskHandle<i32> = (&source).into();
for i in 0..50 {
let dag = Arc::clone(&dag);
let source_clone = source_handle;
handles.spawn(async move {
dag.add_task(task_fn(move |x: i32| async move { x + i }))
.depends_on(source_clone)
});
}
let mut dependents = vec![];
while let Some(result) = handles.join_next().await {
dependents.push(result.unwrap());
}
// Execute
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
// All dependents should have correct values
for dep in dependents.iter() {
let result = dag.get(dep).unwrap();
assert!((42..92).contains(&result)); // 42 + (0..50)
}
}
#[tokio::test]
async fn test_high_concurrency_stress() {
// Stress test with high concurrency
let dag = Arc::new(DagRunner::new());
let completed = Arc::new(AtomicUsize::new(0));
let mut handles = JoinSet::new();
// 1000 concurrent operations
for i in 0..1000 {
let dag = Arc::clone(&dag);
let completed = Arc::clone(&completed);
handles.spawn(async move {
// Mix of operations
match i % 3 {
0 => {
// Add a source task
dag.add_task(task_fn(move |_: ()| async move { i }));
}
1 => {
// Add a task with a simple dependency
let source = dag.add_task(task_fn(move |_: ()| async move { i }));
dag.add_task(task_fn(move |x: i32| async move { x * 2 }))
.depends_on(&source);
}
2 => {
// Add a more complex subgraph
let a = dag.add_task(task_fn(move |_: ()| async move { i }));
let b = dag.add_task(task_fn(move |_: ()| async move { i + 1 }));
dag.add_task(task_fn(move |(x, y): (i32, i32)| async move { x + y }))
.depends_on((&a, &b));
}
_ => unreachable!(),
}
completed.fetch_add(1, Ordering::SeqCst);
});
}
// Wait for all operations to complete
while let Some(result) = handles.join_next().await {
result.unwrap();
}
assert_eq!(completed.load(Ordering::SeqCst), 1000);
// The DAG should still be executable
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
}
#[tokio::test]
async fn test_lock_contention_measurement() {
// Test to measure lock contention under concurrent load
let dag = Arc::new(DagRunner::new());
let start = tokio::time::Instant::now();
let operations = Arc::new(AtomicUsize::new(0));
let mut handles = JoinSet::new();
// Create many concurrent operations that will contend for locks
for i in 0..100 {
let dag = Arc::clone(&dag);
let ops = Arc::clone(&operations);
handles.spawn(async move {
for j in 0..10 {
dag.add_task(task_fn(move |_: ()| async move { i * 10 + j }));
ops.fetch_add(1, Ordering::SeqCst);
// Small yield to increase contention likelihood
tokio::task::yield_now().await;
}
});
}
// Wait for completion
while let Some(result) = handles.join_next().await {
result.unwrap();
}
let duration = start.elapsed();
let total_ops = operations.load(Ordering::SeqCst);
println!(
"Completed {} operations in {:?} ({:.0} ops/sec)",
total_ops,
duration,
total_ops as f64 / duration.as_secs_f64()
);
// Should complete reasonably quickly even with contention
assert!(
duration < Duration::from_secs(5),
"Too much lock contention"
);
}
#[tokio::test]
async fn test_atomic_dag_operations() {
// Test that DAG operations are atomic (all-or-nothing)
let dag = Arc::new(DagRunner::new());
// Create a base task
let base = dag.add_task(task_fn(|_: ()| async { 100 }));
// Try to create dependent tasks concurrently
let mut handles = JoinSet::new();
let success_count = Arc::new(AtomicUsize::new(0));
let base_handle: dagx::TaskHandle<i32> = (&base).into();
for i in 0..50 {
let dag = Arc::clone(&dag);
let base_clone = base_handle;
let success = Arc::clone(&success_count);
handles.spawn(async move {
let dep = dag
.add_task(task_fn(move |x: i32| async move { x + i }))
.depends_on(base_clone);
success.fetch_add(1, Ordering::SeqCst);
dep
});
}
let mut deps = vec![];
while let Some(result) = handles.join_next().await {
deps.push(result.unwrap());
}
// All operations should have succeeded
assert_eq!(success_count.load(Ordering::SeqCst), 50);
assert_eq!(deps.len(), 50);
// Execute and verify
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
for dep in deps.iter() {
let result = dag.get(dep).unwrap();
assert!((100..150).contains(&result));
}
}
#[tokio::test]
async fn test_no_deadlock_on_circular_waiting() {
// Test that we don't deadlock even with complex concurrent operations
let dag = Arc::new(DagRunner::new());
// Create multiple threads that might cause circular waiting patterns
let mut handles = JoinSet::new();
for i in 0..10 {
let dag = Arc::clone(&dag);
handles.spawn(async move {
// Each thread creates a chain of tasks
let first = dag.add_task(task_fn(move |_: ()| async move { i }));
let mut chain = vec![(&first).into()];
for j in 0..5 {
let next = dag
.add_task(task_fn(move |x: i32| async move { x + j }))
.depends_on(chain.last().unwrap());
chain.push(next);
// Yield to potentially cause interleaving
tokio::task::yield_now().await;
}
*chain.last().unwrap()
});
}
// Set a timeout to detect deadlocks
let result = tokio::time::timeout(Duration::from_secs(5), async {
let mut finals = vec![];
while let Some(result) = handles.join_next().await {
finals.push(result.unwrap());
}
finals
})
.await;
assert!(result.is_ok(), "Deadlock detected - operation timed out");
// Should be able to execute the DAG
dag.run(|fut| {
tokio::spawn(fut);
})
.await
.unwrap();
}
-373
View File
@@ -1,373 +0,0 @@
//! Tests for execution correctness and parallelism
use crate::common::task_fn;
use dagx::{DagResult, DagRunner};
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
use tokio::time::sleep;
#[tokio::test]
async fn test_layers_execute_in_parallel() -> DagResult<()> {
// Test that tasks in the same topological layer execute in parallel
let dag = DagRunner::new();
let layer_timing = Arc::new(Mutex::new(Vec::new()));
// Layer 0: 3 independent sources
let sources: Vec<_> = (0..3)
.map(|i| {
let timing = layer_timing.clone();
dag.add_task(task_fn(move |_: ()| {
let timing = timing.clone();
async move {
let start = Instant::now();
timing.lock().unwrap().push((format!("L0_{}", i), start));
sleep(Duration::from_millis(20)).await;
i * 10
}
}))
})
.collect();
// Layer 1: Tasks depending on layer 0
let layer1: Vec<_> = sources
.iter()
.enumerate()
.map(|(i, source)| {
let timing = layer_timing.clone();
dag.add_task(task_fn(move |x: i32| {
let timing = timing.clone();
async move {
let start = Instant::now();
timing.lock().unwrap().push((format!("L1_{}", i), start));
sleep(Duration::from_millis(20)).await;
x + 1
}
}))
.depends_on(source)
})
.collect();
// Layer 2: Final aggregation
let final_task = {
let timing = layer_timing.clone();
dag.add_task(task_fn(move |(a, b, c): (i32, i32, i32)| {
let timing = timing.clone();
async move {
let start = Instant::now();
timing.lock().unwrap().push(("L2_final".to_string(), start));
a + b + c
}
}))
.depends_on((&layer1[0], &layer1[1], &layer1[2]))
};
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
assert_eq!(dag.get(final_task)?, 33); // (0+1) + (10+1) + (20+1)
// Analyze timing
let timing = layer_timing.lock().unwrap();
// Get layer 0 start times
let l0_times: Vec<Instant> = (0..3)
.map(|i| {
timing
.iter()
.find(|(name, _)| name == &format!("L0_{}", i))
.map(|(_, t)| *t)
.unwrap()
})
.collect();
// Get layer 1 start times
let l1_times: Vec<Instant> = (0..3)
.map(|i| {
timing
.iter()
.find(|(name, _)| name == &format!("L1_{}", i))
.map(|(_, t)| *t)
.unwrap()
})
.collect();
// Layer 0 tasks should all start nearly simultaneously
let l0_spread = l0_times
.iter()
.max()
.unwrap()
.duration_since(*l0_times.iter().min().unwrap());
assert!(
l0_spread < Duration::from_millis(5),
"Layer 0 tasks should start together, spread was {:?}",
l0_spread
);
// Layer 1 tasks should also start nearly simultaneously (after layer 0 completes)
let l1_spread = l1_times
.iter()
.max()
.unwrap()
.duration_since(*l1_times.iter().min().unwrap());
assert!(
l1_spread < Duration::from_millis(5),
"Layer 1 tasks should start together, spread was {:?}",
l1_spread
);
// Layer 1 should start after layer 0
let earliest_l1 = *l1_times.iter().min().unwrap();
let earliest_l0 = *l0_times.iter().min().unwrap();
assert!(
earliest_l1 >= earliest_l0 + Duration::from_millis(15),
"Layer 1 should start after layer 0 completes"
);
Ok(())
}
#[tokio::test]
async fn test_completion_order_matches_dependencies() -> DagResult<()> {
// Verify that tasks complete in dependency order
let dag = DagRunner::new();
let completion_order = Arc::new(Mutex::new(Vec::new()));
// Create a complex DAG with clear ordering requirements
let a = {
let order = completion_order.clone();
dag.add_task(task_fn(move |_: ()| {
let order = order.clone();
async move {
sleep(Duration::from_millis(10)).await;
order.lock().unwrap().push("A");
1
}
}))
};
let b = {
let order = completion_order.clone();
dag.add_task(task_fn(move |_: ()| {
let order = order.clone();
async move {
sleep(Duration::from_millis(5)).await;
order.lock().unwrap().push("B");
2
}
}))
};
let c = {
let order = completion_order.clone();
dag.add_task(task_fn(move |(x, y): (i32, i32)| {
let order = order.clone();
async move {
order.lock().unwrap().push("C");
x + y
}
}))
.depends_on((&a, &b))
};
let d = {
let order = completion_order.clone();
dag.add_task(task_fn(move |x: i32| {
let order = order.clone();
async move {
order.lock().unwrap().push("D");
x * 2
}
}))
.depends_on(c)
};
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
assert_eq!(dag.get(d)?, 6); // (1 + 2) * 2
// Check completion order
let order = completion_order.lock().unwrap();
// A and B can complete in any order (parallel)
assert!(order.contains(&"A"));
assert!(order.contains(&"B"));
// C must come after both A and B
let c_pos = order.iter().position(|name| *name == "C").unwrap();
let a_pos = order.iter().position(|name| *name == "A").unwrap();
let b_pos = order.iter().position(|name| *name == "B").unwrap();
assert!(c_pos > a_pos, "C should complete after A");
assert!(c_pos > b_pos, "C should complete after B");
// D must come after C
let d_pos = order.iter().position(|name| *name == "D").unwrap();
assert!(d_pos > c_pos, "D should complete after C");
Ok(())
}
#[tokio::test]
async fn test_dependent_tasks_respect_ordering() -> DagResult<()> {
// Prove that dependent tasks wait for their dependencies
let dag = DagRunner::new();
let execution_order = Arc::new(Mutex::new(Vec::new()));
// Create a chain: A -> B -> C -> D
let a = {
let order = execution_order.clone();
dag.add_task(task_fn(move |_: ()| {
let order = order.clone();
async move {
sleep(Duration::from_millis(20)).await;
order.lock().unwrap().push("A");
1
}
}))
};
let b = {
let order = execution_order.clone();
dag.add_task(task_fn(move |x: i32| {
let order = order.clone();
async move {
order.lock().unwrap().push("B");
sleep(Duration::from_millis(10)).await;
x + 1
}
}))
.depends_on(&a)
};
let c = {
let order = execution_order.clone();
dag.add_task(task_fn(move |x: i32| {
let order = order.clone();
async move {
order.lock().unwrap().push("C");
sleep(Duration::from_millis(10)).await;
x + 1
}
}))
.depends_on(b)
};
let d = {
let order = execution_order.clone();
dag.add_task(task_fn(move |x: i32| {
let order = order.clone();
async move {
order.lock().unwrap().push("D");
x + 1
}
}))
.depends_on(c)
};
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
// Verify results
assert_eq!(dag.get(a)?, 1);
assert_eq!(dag.get(b)?, 2);
assert_eq!(dag.get(c)?, 3);
assert_eq!(dag.get(d)?, 4);
// Verify execution order
let order = execution_order.lock().unwrap();
let expected = vec!["A", "B", "C", "D"];
assert_eq!(order.as_slice(), expected.as_slice());
Ok(())
}
#[tokio::test]
async fn test_independent_tasks_dont_wait() -> DagResult<()> {
// Prove that independent tasks don't wait for each other
let dag = DagRunner::new();
let task_started = Arc::new(Mutex::new(Vec::new()));
let task_finished = Arc::new(Mutex::new(Vec::new()));
// Create tasks with different durations
let fast1 = {
let started = task_started.clone();
let finished = task_finished.clone();
dag.add_task(task_fn(move |_: ()| {
let started = started.clone();
let finished = finished.clone();
async move {
started.lock().unwrap().push(("fast1", Instant::now()));
sleep(Duration::from_millis(10)).await;
finished.lock().unwrap().push(("fast1", Instant::now()));
"fast1"
}
}))
};
let slow = {
let started = task_started.clone();
let finished = task_finished.clone();
dag.add_task(task_fn(move |_: ()| {
let started = started.clone();
let finished = finished.clone();
async move {
started.lock().unwrap().push(("slow", Instant::now()));
sleep(Duration::from_millis(100)).await;
finished.lock().unwrap().push(("slow", Instant::now()));
"slow"
}
}))
};
let fast2 = {
let started = task_started.clone();
let finished = task_finished.clone();
dag.add_task(task_fn(move |_: ()| {
let started = started.clone();
let finished = finished.clone();
async move {
started.lock().unwrap().push(("fast2", Instant::now()));
sleep(Duration::from_millis(10)).await;
finished.lock().unwrap().push(("fast2", Instant::now()));
"fast2"
}
}))
};
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
// Check results
assert_eq!(dag.get(fast1)?, "fast1");
assert_eq!(dag.get(slow)?, "slow");
assert_eq!(dag.get(fast2)?, "fast2");
// Verify that fast tasks finished before slow task
let finished = task_finished.lock().unwrap();
let fast1_finish = finished.iter().find(|x| x.0 == "fast1").unwrap().1;
let fast2_finish = finished.iter().find(|x| x.0 == "fast2").unwrap().1;
let slow_finish = finished.iter().find(|x| x.0 == "slow").unwrap().1;
assert!(
fast1_finish < slow_finish,
"fast1 should finish before slow"
);
assert!(
fast2_finish < slow_finish,
"fast2 should finish before slow"
);
Ok(())
}
-3
View File
@@ -1,8 +1,5 @@
//! Parallelism tests module //! Parallelism tests module
pub mod concurrent;
pub mod execution;
pub mod proof;
pub mod spawning; pub mod spawning;
pub mod threads; pub mod threads;
pub mod timing; pub mod timing;
-65
View File
@@ -1,65 +0,0 @@
//! Parallelism proof tests
use crate::common::task_fn;
use dagx::{DagResult, DagRunner};
use std::time::{Duration, Instant};
use tokio::time::sleep;
#[tokio::test]
async fn test_parallelism_proof_1000_tasks() -> DagResult<()> {
// Proof that 1000 tasks sleeping 100ms each complete in ~100ms (not 100 seconds)
let dag = DagRunner::new();
let sleep_duration = Duration::from_millis(100);
let num_tasks = 1000u32;
// Create 1000 independent tasks that each sleep for 100ms
let tasks: Vec<_> = (0..num_tasks)
.map(|i| {
dag.add_task(task_fn(move |_: ()| async move {
sleep(sleep_duration).await;
i
}))
})
.collect();
let start = Instant::now();
dag.run(|fut| {
tokio::spawn(fut);
})
.await?;
let elapsed = start.elapsed();
// Verify all tasks completed
for (i, task) in tasks.iter().enumerate() {
assert_eq!(dag.get(task)?, i as u32);
}
// Sequential execution would take: 1000 * 100ms = 100 seconds
let sequential_time = sleep_duration * num_tasks;
println!(
"Parallelism proof: {} tasks × {}ms completed in {:?} (sequential would be {:?})",
num_tasks,
sleep_duration.as_millis(),
elapsed,
sequential_time
);
// Allow generous overhead: should complete in less than 500ms (not 100 seconds!)
assert!(
elapsed < Duration::from_millis(500),
"Tasks appear to be running sequentially! Took {:?} instead of ~100ms",
elapsed
);
// Verify we achieved massive speedup
let speedup = sequential_time.as_secs_f64() / elapsed.as_secs_f64();
assert!(
speedup > 200.0,
"Speedup too low: {:.1}x (expected > 200x)",
speedup
);
Ok(())
}
Loaded 100 of 128 files, more files were not shown because too many files have changed in this diff. Show more