doc: update for next release
This commit is contained in:
TechnoPorg committed
1 parent
a13f148778
commit
766555bef4
24 files changed
+224
-878
No files matched your search
+19
-64
@@ -66,36 +66,18 @@ graph.update_dependency(y, sum, 1)?;
|
||||
|
||||
## Quick Comparison
|
||||
|
||||
| Project | License | Runtime | Type Safety | API Complexity | Performance vs dagx | Key Features |
|
||||
| ------------------------------------------------------------- | -------------- | ------------------- | --------------------- | -------------- | ---------------------------- | ------------------------------------------------------------------------------------ |
|
||||
| **dagx** | MIT | Any async runtime | Compile-time | Simple | Baseline (see benchmarks) | Primitives-as-scheduler, inline fast-path, automatic Arc wrapping, up to 8 deps/task |
|
||||
| [**dagrs**](https://github.com/dagrs-dev/dagrs) (v0.5) | MIT/Apache-2.0 | Creates own runtime | Runtime (async_trait) | Complex | 1.04-129x slower (see below) | Flow-based Programming, cyclic graphs, loops, conditional nodes, YAML config |
|
||||
| [**async_dag**](https://github.com/chubei-oppen/async_dag) | MIT | Any async runtime | Compile-time | Medium | No benchmarks | Slot-based dependencies, Graph/TryGraph modes, maximum parallelism |
|
||||
| [**dag-flow**](https://github.com/makisevon/dag-flow) | MIT/Apache-2.0 | Any async runtime | Runtime (HashMap) | Complex | No benchmarks | Experimental, all tasks run simultaneously, weak dependencies |
|
||||
| [**RenovZ/dag-runner**](https://github.com/RenovZ/dag-runner) | MIT | Tokio only | Unclear | Simple | No benchmarks | Edge-based API, cycle detection, stops on first error |
|
||||
| [**tasksitter**](https://github.com/lionkor/tasksitter) | Unspecified | Unclear | Unclear | Medium | No benchmarks | Cyclic graphs, dynamic runtime modification, pause/resume |
|
||||
| Project | License | Runtime | Type Safety | API Complexity | Performance vs dagx | Key Features |
|
||||
| ------------------------------------------------------------- | -------------- | ------------------- | --------------------- | -------------- | ------------------------- | ------------------------------------------------------------------------------------ |
|
||||
| **dagx** | MIT | Any async runtime | Compile-time | Simple | Baseline (see benchmarks) | Primitives-as-scheduler, inline fast-path, automatic Arc wrapping, up to 8 deps/task |
|
||||
| [**dagrs**](https://github.com/dagrs-dev/dagrs) (v0.5) | MIT/Apache-2.0 | Creates own runtime | Runtime (async_trait) | Complex | 1-100x slower (see below) | Flow-based Programming, cyclic graphs, loops, conditional nodes, YAML config |
|
||||
| [**async_dag**](https://github.com/chubei-oppen/async_dag) | MIT | Any async runtime | Compile-time | Medium | No benchmarks | Slot-based dependencies, Graph/TryGraph modes, maximum parallelism |
|
||||
| [**dag-flow**](https://github.com/makisevon/dag-flow) | MIT/Apache-2.0 | Any async runtime | Runtime (HashMap) | Complex | No benchmarks | Experimental, all tasks run simultaneously, weak dependencies |
|
||||
| [**RenovZ/dag-runner**](https://github.com/RenovZ/dag-runner) | MIT | Tokio only | Unclear | Simple | No benchmarks | Edge-based API, cycle detection, stops on first error |
|
||||
| [**tasksitter**](https://github.com/lionkor/tasksitter) | Unspecified | Unclear | Unclear | Medium | No benchmarks | Cyclic graphs, dynamic runtime modification, pause/resume |
|
||||
|
||||
_GitHub stars (as of 2025): dagrs (449), async_dag (25), dag-flow (2), others (0-2)_
|
||||
**Performance**: As noted in the README, dagx is **1-100x faster** than dagrs across all benchmark patterns. The inline fast-path optimization eliminates spawning overhead for sequential workloads while maintaining excellent parallel performance. Most real-world DAGs mix sequential and parallel patterns. dagx automatically optimizes for both, performing up to 2 orders of magnitude faster than dagrs regardless of workload shape.
|
||||
|
||||
## Performance Benchmarks vs dagrs
|
||||
|
||||
Direct comparison benchmarks (lower is better):
|
||||
|
||||
| Workload | Tasks | dagx | dagrs | Speedup |
|
||||
| ----------------- | ------ | ------- | ------- | ------------------ |
|
||||
| Sequential chain | 5 | 3.0 µs | 385 µs | **129x faster** 🚀 |
|
||||
| Sequential chain | 100 | 79 µs | 703 µs | **8.9x faster** |
|
||||
| Diamond pattern | 4 | 11 µs | 387 µs | **34x faster** |
|
||||
| Fan-out (1→100) | 101 | 155 µs | 595 µs | **3.85x faster** |
|
||||
| Independent tasks | 10,000 | 12.7 ms | 13.3 ms | **1.04x faster** |
|
||||
|
||||
**Summary**: dagx is **1.04-129x faster** than dagrs across all benchmark patterns. The inline fast-path optimization eliminates spawning overhead for sequential workloads while maintaining excellent parallel performance.
|
||||
|
||||
**How does dagx achieve this?** The inline fast-path detects single-task layers (common in sequential chains) and executes them directly without spawning overhead. For parallel workloads, tasks still spawn to maximize concurrency. This adaptive execution strategy combines the best of both worlds: sub-microsecond overhead for sequential work, true parallelism for concurrent work.
|
||||
|
||||
**Key insight**: Most real-world DAGs mix sequential and parallel patterns. dagx automatically optimizes for both, delivering 1.04-129x better performance than dagrs regardless of workload shape.
|
||||
|
||||
_Benchmarks run on AMD Ryzen 7 7840U (Zen 4) @ 3.3GHz. Run `cargo bench` to test on your hardware._
|
||||
_Benchmarks run on Intel i9-13950HX @ 5.5GHz. Run `cargo bench` to test on your hardware._
|
||||
|
||||
## Detailed Comparison
|
||||
|
||||
@@ -107,7 +89,6 @@ _Benchmarks run on AMD Ryzen 7 7840U (Zen 4) @ 3.3GHz. Run `cargo bench` to test
|
||||
|
||||
- Most mature (449 GitHub stars, active community)
|
||||
- Rich feature set: Flow-based Programming, cyclic graphs, loops, conditional nodes
|
||||
- YAML configuration support for declarative workflows
|
||||
- Designed for complex orchestration patterns
|
||||
|
||||
**Trade-offs**:
|
||||
@@ -115,29 +96,7 @@ _Benchmarks run on AMD Ryzen 7 7840U (Zen 4) @ 3.3GHz. Run `cargo bench` to test
|
||||
- Creates own Tokio runtime internally (not runtime-agnostic, cannot be nested)
|
||||
- More complex API: `Action` trait, `InChannels`/`OutChannels`, `NodeTable`, `Content` wrappers, manual node ID tracking
|
||||
- Uses `async_trait` for type erasure (runtime overhead)
|
||||
- **1.04-129x slower than dagx** across all benchmark patterns (see comparison benchmarks)
|
||||
|
||||
**API Style** (v0.5):
|
||||
|
||||
```rust
|
||||
// Define action with channels for communication
|
||||
#[async_trait]
|
||||
impl Action for MyTask {
|
||||
async fn run(&self, input: &mut InChannels,
|
||||
output: &mut OutChannels, env: Arc<EnvVar>) -> Output {
|
||||
let inputs: Vec<T> = input.map(|c| ...).await; // Manually extract
|
||||
output.broadcast(Content::new(result)).await; // Manually wrap
|
||||
Output::Out(Some(Content::new(result)))
|
||||
}
|
||||
}
|
||||
|
||||
let mut table = NodeTable::new();
|
||||
let node = DefaultNode::with_action("name".into(), MyTask, &mut table);
|
||||
let node_id = node.id(); // Must capture ID before moving
|
||||
graph.add_node(node);
|
||||
graph.add_edge(source_id, vec![target_id]);
|
||||
graph.set_env(EnvVar::new(table)); graph.start().unwrap();
|
||||
```
|
||||
- **Slower than dagx** across all benchmark patterns (see comparison benchmarks)
|
||||
|
||||
### async_dag (Clean Type Safety)
|
||||
|
||||
@@ -154,8 +113,7 @@ graph.set_env(EnvVar::new(table)); graph.start().unwrap();
|
||||
|
||||
- Medium API complexity: slot-based dependency management
|
||||
- Must manually specify slot indices (0, 1, etc.) when connecting tasks
|
||||
- Less mature (25 stars)
|
||||
- No performance benchmarks
|
||||
- Less mature
|
||||
|
||||
**API Style**:
|
||||
|
||||
@@ -188,7 +146,7 @@ graph.run().await;
|
||||
- Runtime type safety via `HashMap<String, Input>`
|
||||
- Complex API: implement `Task` trait with `id()`, `dependencies()`, `run()`
|
||||
- Named dependencies (string-based lookup)
|
||||
- Very early stage (2 stars)
|
||||
- Very early stage
|
||||
|
||||
**API Style**:
|
||||
|
||||
@@ -219,7 +177,7 @@ impl Task<String, Bytes> for MyTask {
|
||||
- Requires Tokio runtime
|
||||
- Manual channel setup for task communication
|
||||
- Type safety model unclear
|
||||
- Very early stage (0 stars)
|
||||
- Very early stage
|
||||
|
||||
**API Style**:
|
||||
|
||||
@@ -245,29 +203,26 @@ dag.run().await?;
|
||||
|
||||
- Limited documentation
|
||||
- Runtime and type safety model unclear
|
||||
- Very early stage (0 stars)
|
||||
- Very early stage
|
||||
|
||||
## When to Choose dagx
|
||||
|
||||
Choose dagx when you value:
|
||||
|
||||
- **Performance**: 1.04-129x faster than dagrs across all workload patterns (see benchmarks)
|
||||
- **Runtime flexibility**: Works with Tokio, async-std, smol, or any async runtime
|
||||
- **Compile-time safety**: Full type safety with no runtime type errors in the public API
|
||||
- **Minimal overhead**: ~790 ns per task (sequential), ~1.3 µs per task (parallel)
|
||||
- **Performance**: Faster than dagrs across all workload patterns (see benchmarks)
|
||||
- **Runtime flexibility**: Works with Tokio, smol, or any async runtime
|
||||
- **Compile-time safety**: Full type safety with no runtime type errors possible
|
||||
- **Simple, ergonomic API**: `#[task]` macro, `add_task()`, `depends_on()` - that's it
|
||||
- **Automatic optimizations**: Arc wrapping, inline execution, adaptive spawning - all transparent
|
||||
- **Predictable performance**: Linear scaling, no hidden complexity, consistent sub-µs overhead
|
||||
|
||||
dagx is **not** the right choice if you need:
|
||||
|
||||
- Cyclic graphs or dynamic flow control (loops, conditions) → Consider **dagrs** or **tasksitter**
|
||||
- More than 8 dependencies per task → Consider **dagrs** or **async_dag**
|
||||
- YAML-based configuration → Consider **dagrs**
|
||||
|
||||
## When to Consider Alternatives
|
||||
|
||||
- **Choose dagrs** if you need advanced flow control (loops, conditionals, cyclic graphs), YAML configuration, or are already committed to Tokio and want a mature, feature-rich solution
|
||||
- **Choose dagrs** if you need advanced flow control (loops, conditionals, cyclic graphs), or are already committed to Tokio and want a mature, feature-rich solution
|
||||
- **Choose async_dag** if you want compile-time type safety with runtime flexibility and the slot-based API appeals to you
|
||||
- **Choose dag-flow** if you're building experimental projects and the all-tasks-run-simultaneously model fits your use case
|
||||
- **Choose RenovZ/dag-runner** if you need the simplest possible edge-based API and are already using Tokio
|
||||
|
||||
@@ -26,12 +26,12 @@ let b_builder = dag.add_task(TaskB);
|
||||
|
||||
// To make A depend on B, we need B as a TaskHandle.
|
||||
// But calling depends_on() consumes the builder!
|
||||
let b_handle = b_builder.depends_on(&some_source);
|
||||
let b_handle = b_builder.depends_on(some_source);
|
||||
|
||||
// Now we can't make B depend on A because b_builder was moved!
|
||||
// This won't compile:
|
||||
// a_builder.depends_on(&b_handle); // ✓ OK: A→B
|
||||
// b_handle.depends_on(&a_handle); // ❌ ERROR: TaskHandle has no depends_on() method!
|
||||
// a_builder.depends_on(b_handle); // ✓ OK: A→B
|
||||
// b_handle.depends_on(a_handle); // ❌ ERROR: TaskHandle has no depends_on() method!
|
||||
```
|
||||
|
||||
## How It Works: The Type-State Pattern
|
||||
@@ -58,18 +58,16 @@ let b_builder = dag.add_task(TaskB); // TaskBuilder
|
||||
let b_handle = b_builder.into(); // Convert B to TaskHandle (consumes builder)
|
||||
|
||||
// To wire B→A, we need A as a TaskHandle
|
||||
let a_handle = a_builder.depends_on(&b_handle); // A now depends on B
|
||||
let a_handle = a_builder.depends_on(b_handle); // A now depends on B
|
||||
|
||||
// Now if we try to complete the cycle:
|
||||
// a_handle.depends_on(&...); // ❌ ERROR: no method named `depends_on` found
|
||||
// a_handle.depends_on(...); // ❌ ERROR: no method named `depends_on` found
|
||||
```
|
||||
|
||||
**The type system enforces strict topological ordering!** You can't create a `TaskHandle` until all its dependencies exist as handles, which prevents cycles.
|
||||
|
||||
## Proof & Examples
|
||||
|
||||
See [`src/cycle_prevention.rs`](../src/cycle_prevention.rs) for comprehensive documentation with `compile_fail` tests proving various cycle attempts don't compile.
|
||||
|
||||
See [`tests/cycle_prevention.rs`](../tests/cycle_prevention.rs) for runtime tests demonstrating the type-state pattern in action.
|
||||
|
||||
## Benefits
|
||||
|
||||
@@ -1,16 +1,10 @@
|
||||
# Design Philosophy: Primitives as Scheduler
|
||||
|
||||
dagx takes an unconventional approach to task orchestration: **there is no traditional scheduler**. Instead of complex scheduling logic managing when tasks run, the entire system is built on communication and synchronization primitives that do the scheduling themselves.
|
||||
|
||||
## How It Works
|
||||
|
||||
Traditional DAG executors contain substantial scheduling code—algorithms that track task states, manage dependencies, coordinate execution order, and handle synchronization. This code is complex, error-prone, and difficult to verify.
|
||||
|
||||
dagx eliminates this entirely:
|
||||
|
||||
1. **Wire up tasks with primitives**: During DAG construction, tasks are organized into topological layers based on their dependencies
|
||||
2. **Start everything simultaneously**: When you call `run()`, all tasks spawn at once—there's no scheduler deciding when each task should start
|
||||
3. **Let the primitives handle coordination**: Layers naturally enforce execution order—tasks wait on their dependencies until . No custom orchestration needed.
|
||||
2. **Start everything simultaneously**: When you call `run()`, all tasks in the first layer spawn at once—there's no scheduler deciding when each task should start
|
||||
3. **Let the primitives handle coordination**: Layers naturally enforce execution order. No custom orchestration needed.
|
||||
4. **Runtime joins on completion**: The runtime simply spawns all tasks and waits for them to finish. That's it.
|
||||
|
||||
## The Implementation
|
||||
@@ -18,54 +12,14 @@ dagx eliminates this entirely:
|
||||
Under the hood, dagx uses:
|
||||
|
||||
- **Ownership model**: Tasks take ownership (`self`) and are consumed during execution - no Mutex needed for task state
|
||||
- **Direct data flow**: Outputs flow from producer(s) to consumer(s) through a shared hash map
|
||||
- **Direct data flow**: Arc-wrapped outputs flow from producer(s) to consumer(s) through a store in the runner
|
||||
- **Fast-path optimization**: Single-task layers execute inline without spawning overhead
|
||||
|
||||
The core execution logic in `run()` is remarkably simple:
|
||||
|
||||
```rust
|
||||
let outputs = HashMap::new();
|
||||
|
||||
// Execute layer by layer
|
||||
for layer in layers {
|
||||
if layer.len() == 1 {
|
||||
// Fast path: Execute inline, no spawning overhead
|
||||
// IMPORTANT: Panic handling is required here!
|
||||
let result = task.execute().catch_unwind().await;
|
||||
// Convert panic to error to match spawned task behavior
|
||||
} else {
|
||||
// Parallel path: Spawn all tasks in layer
|
||||
let tasks = layers
|
||||
.map(async {
|
||||
let input = get_inputs(map); // Deps are guaranteed to be in the map
|
||||
let output = task.run(input); // Consumes task
|
||||
output
|
||||
})
|
||||
.collect::<FuturesUnordered<_>>(); // Run all tasks in parallel
|
||||
|
||||
while let Some(out) = tasks.next().await {
|
||||
outputs.insert(out.task_id, out.output); // Make result available for dependents
|
||||
}
|
||||
}
|
||||
// Wait for layer completion
|
||||
}
|
||||
|
||||
// FuturesUnordered coordinates task execution - no custom scheduler needed
|
||||
```
|
||||
|
||||
## Inline Execution Fast-Path
|
||||
|
||||
**Performance optimization for sequential workloads**: When a layer contains only a single task (common in deep chains and linear pipelines), dagx executes it inline rather than spawning it. This eliminates spawning overhead and context switching, resulting in a 10-100x performance improvements for sequential patterns.
|
||||
|
||||
**Panic handling guarantee**: To maintain behavioral consistency between inline and spawned execution, panics in inline tasks are caught using `FutureExt::catch_unwind()` and converted to errors. This matches the behavior of all major async runtimes (Tokio, async-std, smol, embassy-rs), which catch panics in spawned tasks and convert them to `JoinError` or equivalent.
|
||||
|
||||
**Why this matters**:
|
||||
|
||||
- **Spawned tasks** (layer.len() > 1): Runtime catches panics automatically → becomes error
|
||||
- **Inline tasks** (layer.len() == 1): We catch panics manually → becomes error
|
||||
- **Result**: Identical behavior regardless of execution path
|
||||
|
||||
This ensures your code behaves the same whether a task runs inline or spawned, making dagx's optimizations transparent and predictable.
|
||||
**Panic handling guarantee**: To maintain behavioral consistency between runtimes, as well as inline vs. spawned tasks, panics in tasks are caught using `FutureExt::catch_unwind()` and converted to errors. This ensures your code behaves the same whether a task runs inline or spawned, making dagx's optimizations transparent and predictable.
|
||||
|
||||
## Benefits
|
||||
|
||||
@@ -75,7 +29,7 @@ This ensures your code behaves the same whether a task runs inline or spawned, m
|
||||
|
||||
**Bug resistance**: Fewer moving parts means fewer places for bugs to hide. The type system enforces correct wiring at compile time. The ownership model prevents data races. What's left to break?
|
||||
|
||||
**Performance**: Near zero-overhead. No Mutex locks during execution. Arc reference counting for efficient fanout (atomic operations, not locks). Tasks start as soon as their dependencies complete - maximum parallelism.
|
||||
**Performance**: Near zero-overhead. No locks during execution. Arc reference counting for efficient fanout (atomic operations, not locks). Tasks start as soon as their dependencies complete - maximum parallelism.
|
||||
|
||||
**Auditability**: Want to verify correctness? Check the dependency wiring, verify tasks await their inputs, done. No need to trace through complex state machine transitions or wake-up cascades.
|
||||
|
||||
@@ -87,24 +41,14 @@ This is dagx's core philosophy: leverage the type system for correctness, use pr
|
||||
|
||||
## Measured Overhead
|
||||
|
||||
How much overhead does this approach actually add? Benchmarks on an AMD 7840U (Zen 4 laptop CPU) show:
|
||||
How much overhead does this approach actually add? Benchmarks on an Intel i9-13950HX show:
|
||||
|
||||
**DAG Construction**:
|
||||
|
||||
- Empty DAG creation: **~20 nanoseconds**
|
||||
- Adding tasks: **~100 nanoseconds per task**
|
||||
- Building a 10,000-task DAG: **~1.0 milliseconds** (100 ns/task)
|
||||
|
||||
**Execution Overhead** (framework coordination, excluding actual task work):
|
||||
|
||||
- Sequential workloads: **~790 nanoseconds per task** (inline execution fast-path)
|
||||
- Parallel workloads: **~1.3 microseconds per task**
|
||||
- 100-task sequential chain: **~79 microseconds total**
|
||||
- 100 independent tasks: **~122 microseconds total**
|
||||
- 10,000 independent tasks: **~12.7 milliseconds total**
|
||||
- Empty DAG creation: **~7 nanoseconds**
|
||||
- Adding tasks: **~5 nanoseconds per task**
|
||||
- Building a 10,000-task DAG: **~865 microseconds** (100 ns/task)
|
||||
|
||||
**Scaling**: Sub-microsecond per-task overhead across all workload patterns. Linear scaling verified to 10k+ tasks.
|
||||
|
||||
**Comparison**: dagx is **1.04-129x faster** than dagrs (v0.5) across all benchmark patterns (see [comparisons](COMPARISONS.md)).
|
||||
|
||||
The primitives-as-scheduler approach with inline fast-path optimization delivers exceptional performance: coordination overhead is sub-microsecond per task, and for real-world workloads where tasks do meaningful work (I/O, computation, etc.), framework overhead is negligible—typically well under 1% of total execution time.
|
||||
@@ -1,53 +0,0 @@
|
||||
# Tracing Support
|
||||
|
||||
dagx provides optional observability through the `tracing` crate with **zero runtime overhead when disabled**. The tracing instrumentation is conditionally compiled using feature flags, meaning when the feature is off, the logging code doesn't exist in the compiled binary at all.
|
||||
|
||||
## Enabling Tracing
|
||||
|
||||
```toml
|
||||
[dependencies]
|
||||
dagx = { version = "0.3", features = ["tracing"] }
|
||||
tracing-subscriber = "0.3"
|
||||
```
|
||||
|
||||
## Usage Example
|
||||
|
||||
```rust
|
||||
use tracing_subscriber::{fmt, EnvFilter};
|
||||
|
||||
// Initialize tracing subscriber
|
||||
fmt()
|
||||
.with_env_filter(
|
||||
EnvFilter::try_from_default_env()
|
||||
.unwrap_or_else(|_| EnvFilter::new("dagx=info"))
|
||||
)
|
||||
.init();
|
||||
|
||||
// Your DAG code here - tracing will log execution details
|
||||
```
|
||||
|
||||
## Log Levels
|
||||
|
||||
- **INFO**: DAG execution start/completion
|
||||
- **DEBUG**: Task additions, dependency wiring, layer computation
|
||||
- **TRACE**: Individual task execution (inline vs spawned), detailed execution flow
|
||||
- **ERROR**: Task panics, concurrent execution attempts
|
||||
|
||||
Run with different levels:
|
||||
|
||||
```bash
|
||||
RUST_LOG=dagx=info cargo run # High-level execution info
|
||||
RUST_LOG=dagx=debug cargo run # Task and layer details
|
||||
RUST_LOG=dagx=trace cargo run # All execution details
|
||||
```
|
||||
|
||||
## Zero-Cost Guarantee
|
||||
|
||||
When the `tracing` feature is disabled (the default), there is **literally 0ns overhead**:
|
||||
|
||||
- Logging code is removed at compile time via `#[cfg(feature = "tracing")]`
|
||||
- No branches, no function calls, nothing
|
||||
- The `tracing` crate isn't even linked
|
||||
- Benchmarks verify identical performance with/without the feature
|
||||
|
||||
This follows the same pattern used by Tokio, Hyper, and other performance-critical Rust async libraries.
|
||||
Reference in new issue
Block a user