Don't clone task inputs
This commit is contained in:
1 parent
6f7a58e4c5
commit
4ef7e69cdd
10 files changed
+15
-137
No files matched your search
@@ -1,109 +0,0 @@
|
|||||||
# Documentation Update Summary - Custom Type Support
|
|
||||||
|
|
||||||
## Context
|
|
||||||
|
|
||||||
Fixed a significant issue where dagx only worked with types that had `ExtractInput` trait implementations (primitives, String, Vec, HashMap, etc.). Custom user types didn't work.
|
|
||||||
|
|
||||||
**The Fix**: The `#[task]` macro now generates inline `extract_and_run()` methods with type-specific extraction logic. This means **ANY type** implementing `Clone + Send + Sync + 'static` works automatically - no trait implementations required!
|
|
||||||
|
|
||||||
## Files Updated
|
|
||||||
|
|
||||||
### 1. **README.md**
|
|
||||||
- **Features section**: Added "Works with ANY type" as a key feature (line 117)
|
|
||||||
- **Core Concepts**: Added new "Custom Types" section with example (lines 190-218)
|
|
||||||
- **Examples**: Added `custom_types.rs` to advanced examples list (line 232)
|
|
||||||
|
|
||||||
### 2. **CHANGELOG.md**
|
|
||||||
- **[Unreleased] section**: Added comprehensive documentation of this improvement (lines 8-33)
|
|
||||||
- Explained the change from ExtractInput to macro-generated extraction
|
|
||||||
- Highlighted that custom types now work seamlessly
|
|
||||||
- Noted new example file
|
|
||||||
|
|
||||||
### 3. **src/lib.rs** (Main library documentation)
|
|
||||||
- **Features**: Added custom type support as third bullet point (lines 13-15)
|
|
||||||
- **Examples section**: Added new "Custom Types" example showing usage (lines 439-484)
|
|
||||||
|
|
||||||
### 4. **src/task.rs** (Task trait documentation)
|
|
||||||
- **Trait docs**: Updated to mention macro generates extraction logic (lines 10-15)
|
|
||||||
- Added note that custom types work automatically
|
|
||||||
|
|
||||||
### 5. **src/extract.rs** (ExtractInput trait documentation)
|
|
||||||
- **Module docs**: Updated to clarify this is now legacy/internal only (lines 1-33)
|
|
||||||
- Explained modern approach via `#[task]` macro vs legacy ExtractInput trait
|
|
||||||
- Noted ExtractInput only used by internal `task_fn` helper
|
|
||||||
|
|
||||||
### 6. **dagx-macros/src/lib.rs** (Macro documentation)
|
|
||||||
- **Macro docs**: Added emphasis on type-specific extraction generation (lines 15-22)
|
|
||||||
- Highlighted that custom types work without trait implementations
|
|
||||||
|
|
||||||
### 7. **examples/custom_types.rs** (NEW FILE)
|
|
||||||
- **Created comprehensive example** demonstrating custom type support
|
|
||||||
- Shows nested custom types (User, Project, TeamMember, Team)
|
|
||||||
- Demonstrates types flowing through multiple task layers
|
|
||||||
- Full working example with detailed documentation comments
|
|
||||||
- Expected output documented in header
|
|
||||||
|
|
||||||
## Key Messages in Documentation
|
|
||||||
|
|
||||||
### What Changed
|
|
||||||
1. **Before**: Only types with ExtractInput implementations worked (primitives + standard types)
|
|
||||||
2. **After**: ANY type with `Clone + Send + Sync + 'static` works automatically
|
|
||||||
|
|
||||||
### Why This Matters
|
|
||||||
- No trait boilerplate for custom types
|
|
||||||
- Nested structs, collections of custom types all work seamlessly
|
|
||||||
- Simpler mental model - just derive Clone and go!
|
|
||||||
- More ergonomic API
|
|
||||||
|
|
||||||
### How It Works
|
|
||||||
- `#[task]` macro generates custom `extract_and_run()` implementation per task
|
|
||||||
- Type-specific extraction logic inlined at compile time
|
|
||||||
- No runtime overhead compared to ExtractInput approach
|
|
||||||
- ExtractInput now only used internally by test helper
|
|
||||||
|
|
||||||
## Documentation Gaps Remaining
|
|
||||||
|
|
||||||
None found. The following were checked and updated:
|
|
||||||
- ✅ README.md (main documentation)
|
|
||||||
- ✅ CHANGELOG.md (version history)
|
|
||||||
- ✅ src/lib.rs (API documentation)
|
|
||||||
- ✅ src/task.rs (Task trait docs)
|
|
||||||
- ✅ src/extract.rs (ExtractInput trait docs)
|
|
||||||
- ✅ dagx-macros/src/lib.rs (macro documentation)
|
|
||||||
- ✅ examples/ (new example added)
|
|
||||||
- ✅ CONTRIBUTING.md (no changes needed - general guidelines)
|
|
||||||
- ✅ docs/*.md (no changes needed - architectural docs)
|
|
||||||
|
|
||||||
## Example Test Status
|
|
||||||
|
|
||||||
The new `custom_types.rs` example:
|
|
||||||
- Created at `/home/swaits/Code/dagx/examples/custom_types.rs`
|
|
||||||
- Demonstrates User, Project, TeamMember, Team custom types
|
|
||||||
- Shows nested types and collections
|
|
||||||
- Needs to be compiled and tested: `cargo run --example custom_types`
|
|
||||||
|
|
||||||
## Recommendations
|
|
||||||
|
|
||||||
### Before Release
|
|
||||||
1. **Compile and test** the new example: `cargo run --example custom_types`
|
|
||||||
2. **Update lib.rs last audited date** (currently shows 2025-10-06 for v0.1.0)
|
|
||||||
3. **Review CHANGELOG** to ensure version number and date are correct when releasing
|
|
||||||
4. Consider adding a **migration guide** if there are any breaking changes
|
|
||||||
|
|
||||||
### Future Documentation Enhancements
|
|
||||||
1. Consider adding a "Type Requirements" section to README explaining `Clone + Send + Sync + 'static`
|
|
||||||
2. Could add more examples showing:
|
|
||||||
- Enums as task outputs
|
|
||||||
- Generic custom types
|
|
||||||
- Types with lifetimes (showing why 'static is needed)
|
|
||||||
|
|
||||||
## Summary
|
|
||||||
|
|
||||||
All documentation has been updated to accurately reflect the new custom type support. The key improvement - that ANY type works automatically with just `Clone + Send + Sync` - is now prominently featured in:
|
|
||||||
- Main features list
|
|
||||||
- Core concepts with examples
|
|
||||||
- API documentation
|
|
||||||
- Macro documentation
|
|
||||||
- A comprehensive new example
|
|
||||||
|
|
||||||
The documentation emphasizes this is a **major ergonomic improvement** that makes dagx significantly easier to use with custom types.
|
|
||||||
@@ -114,7 +114,7 @@ See [design philosophy](docs/DESIGN_PHILOSOPHY.md) for details.
|
|||||||
|
|
||||||
- **Compile-time cycle prevention**: Type system makes cycles impossible — no runtime checks
|
- **Compile-time cycle prevention**: Type system makes cycles impossible — no runtime checks
|
||||||
- **Compile-time type safety**: Dependencies validated at compile time, no runtime type errors
|
- **Compile-time type safety**: Dependencies validated at compile time, no runtime type errors
|
||||||
- **Works with ANY type**: Custom types work automatically — just `Clone + Send + Sync`, no trait implementations needed
|
- **Works with ANY type**: Custom types work automatically — just `Send + Sync + 'static`, no trait implementations needed
|
||||||
- **Runtime-agnostic**: Works with Tokio, async-std, smol, or any async runtime
|
- **Runtime-agnostic**: Works with Tokio, async-std, smol, or any async runtime
|
||||||
- **True parallelism**: Tasks spawn to multiple threads for genuine parallel execution
|
- **True parallelism**: Tasks spawn to multiple threads for genuine parallel execution
|
||||||
- **Type-state pattern**: API prevents incorrect wiring through compile-time errors
|
- **Type-state pattern**: API prevents incorrect wiring through compile-time errors
|
||||||
@@ -189,10 +189,9 @@ let task = dag.add_task(my_task).depends_on((&upstream1, &upstream2));
|
|||||||
|
|
||||||
### Custom Types
|
### Custom Types
|
||||||
|
|
||||||
**dagx works with ANY type automatically!** As long as your type implements `Clone + Send + Sync + 'static`, it works seamlessly:
|
**dagx works with ANY type automatically!** As long as your type implements `Send + Sync + 'static`, it works seamlessly:
|
||||||
|
|
||||||
```rust
|
```rust
|
||||||
#[derive(Clone)] // Just derive Clone!
|
|
||||||
struct User {
|
struct User {
|
||||||
name: String,
|
name: String,
|
||||||
age: u32,
|
age: u32,
|
||||||
|
|||||||
@@ -13,7 +13,7 @@ use syn::{parse_macro_input, FnArg, Ident, ImplItem, ItemImpl, Pat, PatType, Ret
|
|||||||
/// Apply this to an `impl` block containing a `run()` method (sync or async). The macro:
|
/// Apply this to an `impl` block containing a `run()` method (sync or async). The macro:
|
||||||
/// - Derives `Input` and `Output` types from the `run()` signature
|
/// - Derives `Input` and `Output` types from the `run()` signature
|
||||||
/// - Automatically implements the `Task` trait
|
/// - Automatically implements the `Task` trait
|
||||||
/// - **Generates type-specific extraction logic** - works with ANY type (Clone + Send + Sync)!
|
/// - **Generates type-specific extraction logic** - works with ANY type (Send + Sync + 'static)!
|
||||||
/// - Supports both sync and async run methods
|
/// - Supports both sync and async run methods
|
||||||
/// - Supports stateless (no self) and stateful (&self, &mut self) tasks
|
/// - Supports stateless (no self) and stateful (&self, &mut self) tasks
|
||||||
/// - Handles various input patterns (no inputs, single input, multiple inputs)
|
/// - Handles various input patterns (no inputs, single input, multiple inputs)
|
||||||
|
|||||||
+1
-1
@@ -227,7 +227,7 @@ impl Task<String, Bytes> for MyTask {
|
|||||||
let mut dag = Dag::default();
|
let mut dag = Dag::default();
|
||||||
dag.add_vertex("one", || async move { /* task */ });
|
dag.add_vertex("one", || async move { /* task */ });
|
||||||
dag.add_edge("one", "two");
|
dag.add_edge("one", "two");
|
||||||
dag.run(|fut| tokio::spawn(fut).map(Result::unwrap)).await?;
|
dag.run().await?;
|
||||||
```
|
```
|
||||||
|
|
||||||
### tasksitter (Dynamic Workflows)
|
### tasksitter (Dynamic Workflows)
|
||||||
|
|||||||
@@ -1,20 +1,18 @@
|
|||||||
//! # Custom Types: Using Your Own Structs
|
//! # Custom Types: Using Your Own Structs
|
||||||
//!
|
//!
|
||||||
//! This example demonstrates that **ANY custom type** works in dagx as long as it implements
|
//! This example demonstrates that **ANY custom type** works in dagx as long as it implements
|
||||||
//! `Clone + Send + Sync + 'static`. You don't need to implement any special traits!
|
//! ` Send + Sync + 'static`. You don't need to implement any special traits!
|
||||||
//!
|
//!
|
||||||
//! ## What You'll Learn
|
//! ## What You'll Learn
|
||||||
//! - How to use custom structs as task outputs
|
//! - How to use custom structs as task outputs
|
||||||
//! - How custom types flow through the DAG automatically
|
//! - How custom types flow through the DAG automatically
|
||||||
//! - How one task's output can be consumed by multiple downstream tasks
|
//! - How one task's output can be consumed by multiple downstream tasks
|
||||||
//! - No trait implementations required (just derive Clone!)
|
|
||||||
//!
|
//!
|
||||||
//! ## Key Insight
|
//! ## Key Insight
|
||||||
//!
|
//!
|
||||||
//! **dagx works with ANY type automatically!** The `#[task]` macro generates type-specific
|
//! **dagx works with ANY type automatically!** The `#[task]` macro generates type-specific
|
||||||
//! extraction logic, so you don't need to implement any special traits. Just make sure your
|
//! extraction logic, so you don't need to implement any special traits. Just make sure your
|
||||||
//! type implements:
|
//! type implements:
|
||||||
//! - `Clone` (so it can be extracted from Arc)
|
|
||||||
//! - `Send + Sync` (required for async tasks)
|
//! - `Send + Sync` (required for async tasks)
|
||||||
//! - `'static` (no lifetime constraints)
|
//! - `'static` (no lifetime constraints)
|
||||||
//!
|
//!
|
||||||
@@ -39,8 +37,7 @@
|
|||||||
use dagx::{task, DagRunner, TaskHandle};
|
use dagx::{task, DagRunner, TaskHandle};
|
||||||
use futures::FutureExt;
|
use futures::FutureExt;
|
||||||
|
|
||||||
// Custom type - just derive Clone!
|
#[derive(Debug)]
|
||||||
#[derive(Clone, Debug)]
|
|
||||||
struct User {
|
struct User {
|
||||||
name: String,
|
name: String,
|
||||||
age: u32,
|
age: u32,
|
||||||
|
|||||||
+4
-8
@@ -5,8 +5,8 @@
|
|||||||
//! **NOTE**: This module is now primarily for internal use. The `ExtractInput` trait is only
|
//! **NOTE**: This module is now primarily for internal use. The `ExtractInput` trait is only
|
||||||
//! used by the internal `task_fn` test helper function.
|
//! used by the internal `task_fn` test helper function.
|
||||||
//!
|
//!
|
||||||
//! **For regular tasks**: The `#[task]` macro generates inline `extract_and_run()` methods with
|
//! **For regular tasks**: The `#[task]` macro generates inline `run()` methods with
|
||||||
//! type-specific extraction logic. This means **ANY type** implementing `Clone + Send + Sync`
|
//! type-specific extraction logic. This means **ANY type** implementing `Send + Sync`
|
||||||
//! works automatically without needing `ExtractInput` implementations!
|
//! works automatically without needing `ExtractInput` implementations!
|
||||||
//!
|
//!
|
||||||
//! # How it works
|
//! # How it works
|
||||||
@@ -15,8 +15,8 @@
|
|||||||
//! Outputs are wrapped in Arc for efficient fanout (cheap Arc clones instead of data clones).
|
//! Outputs are wrapped in Arc for efficient fanout (cheap Arc clones instead of data clones).
|
||||||
//!
|
//!
|
||||||
//! **Modern approach (via `#[task]` macro)**:
|
//! **Modern approach (via `#[task]` macro)**:
|
||||||
//! - The macro generates custom extraction logic inline in `extract_and_run()`
|
//! - The macro generates custom extraction logic inline in `run()`
|
||||||
//! - Works with ANY type (just needs `Clone + Send + Sync`)
|
//! - Works with ANY type (just needs `Send + Sync + 'static`)
|
||||||
//! - No trait implementations needed!
|
//! - No trait implementations needed!
|
||||||
//!
|
//!
|
||||||
//! **Legacy approach (via `ExtractInput` trait)**:
|
//! **Legacy approach (via `ExtractInput` trait)**:
|
||||||
@@ -40,10 +40,6 @@ use crate::TaskInput;
|
|||||||
///
|
///
|
||||||
/// Type erasure occurs only at the ExecutableNode trait boundary - by the time
|
/// Type erasure occurs only at the ExecutableNode trait boundary - by the time
|
||||||
/// we're in ExtractInput, we know the concrete type and can safely downcast.
|
/// we're in ExtractInput, we know the concrete type and can safely downcast.
|
||||||
///
|
|
||||||
/// NOTE: This trait is now primarily used by the internal `task_fn` test helper.
|
|
||||||
/// The `#[task]` macro generates inline extraction logic via `extract_and_run()`,
|
|
||||||
/// which allows ANY type to work without requiring ExtractInput implementations.
|
|
||||||
#[allow(dead_code)]
|
#[allow(dead_code)]
|
||||||
pub trait ExtractInput: Sized + Clone {
|
pub trait ExtractInput: Sized + Clone {
|
||||||
type Input: Send;
|
type Input: Send;
|
||||||
|
|||||||
+2
-4
@@ -11,7 +11,7 @@
|
|||||||
//! system. The public API is fully type-safe with no runtime type errors. Internal execution
|
//! system. The public API is fully type-safe with no runtime type errors. Internal execution
|
||||||
//! uses type erasure for heterogeneous task storage, but this is never exposed to users.
|
//! uses type erasure for heterogeneous task storage, but this is never exposed to users.
|
||||||
//! - **Works with ANY type**: Custom types work automatically with the `#[task]` macro—just
|
//! - **Works with ANY type**: Custom types work automatically with the `#[task]` macro—just
|
||||||
//! implement `Clone + Send + Sync`, no trait implementations needed! The macro generates
|
//! implement `Send + Sync`, no trait implementations needed! The macro generates
|
||||||
//! type-specific extraction logic for seamless integration.
|
//! type-specific extraction logic for seamless integration.
|
||||||
//! - **Runtime-agnostic**: Works with any async runtime (Tokio, async-std, smol, Embassy, etc.)
|
//! - **Runtime-agnostic**: Works with any async runtime (Tokio, async-std, smol, Embassy, etc.)
|
||||||
//! - **Type-state pattern**: The API guides you with compile-time errors if you wire dependencies
|
//! - **Type-state pattern**: The API guides you with compile-time errors if you wire dependencies
|
||||||
@@ -442,15 +442,13 @@
|
|||||||
//!
|
//!
|
||||||
//! ## Custom Types
|
//! ## Custom Types
|
||||||
//!
|
//!
|
||||||
//! **dagx works with ANY type automatically!** As long as your type implements `Clone + Send + Sync + 'static`,
|
//! **dagx works with ANY type automatically!** As long as your type is `Send + Sync + 'static`,
|
||||||
//! the `#[task]` macro generates the necessary extraction logic:
|
//! the `#[task]` macro generates the necessary extraction logic:
|
||||||
//!
|
//!
|
||||||
//! ```no_run
|
//! ```no_run
|
||||||
//! # use dagx::{task, DagRunner, Task};
|
//! # use dagx::{task, DagRunner, Task};
|
||||||
//! # use futures::FutureExt;
|
//! # use futures::FutureExt;
|
||||||
//!
|
//!
|
||||||
//! // Just derive Clone - that's all you need!
|
|
||||||
//! #[derive(Clone)]
|
|
||||||
//! struct User {
|
//! struct User {
|
||||||
//! name: String,
|
//! name: String,
|
||||||
//! age: u32,
|
//! age: u32,
|
||||||
|
|||||||
+1
-1
@@ -70,7 +70,7 @@ where
|
|||||||
dependencies: Vec<Arc<dyn Any + Send + Sync>>,
|
dependencies: Vec<Arc<dyn Any + Send + Sync>>,
|
||||||
) -> ExecuteFuture {
|
) -> ExecuteFuture {
|
||||||
Box::pin(async move {
|
Box::pin(async move {
|
||||||
// Use the task's extract_and_run method which has inline extraction logic
|
// Use the task's run method which has inline extraction logic
|
||||||
// generated by the #[task] macro. This allows ANY type to work without
|
// generated by the #[task] macro. This allows ANY type to work without
|
||||||
// requiring ExtractInput trait implementations.
|
// requiring ExtractInput trait implementations.
|
||||||
let output = self.task.run(TaskInput::new(dependencies.iter())).await;
|
let output = self.task.run(TaskInput::new(dependencies.iter())).await;
|
||||||
|
|||||||
@@ -527,9 +527,6 @@ impl DagRunner {
|
|||||||
/// - The task hasn't been executed yet
|
/// - The task hasn't been executed yet
|
||||||
/// - The handle is invalid
|
/// - The handle is invalid
|
||||||
///
|
///
|
||||||
/// Returns `DagError::TaskPanicked` if:
|
|
||||||
/// - A task executing in the
|
|
||||||
///
|
|
||||||
/// # Examples
|
/// # Examples
|
||||||
///
|
///
|
||||||
/// ```no_run
|
/// ```no_run
|
||||||
|
|||||||
+2
-2
@@ -11,10 +11,10 @@ use crate::extract::ExtractInput;
|
|||||||
///
|
///
|
||||||
/// Use the [`task`](crate::task) macro to implement this trait. The macro automatically
|
/// Use the [`task`](crate::task) macro to implement this trait. The macro automatically
|
||||||
/// derives `Input` and `Output` types from your `run()` method signature and generates
|
/// derives `Input` and `Output` types from your `run()` method signature and generates
|
||||||
/// type-specific extraction logic in `extract_and_run()`.
|
/// type-specific extraction logic in `run()`.
|
||||||
///
|
///
|
||||||
/// **Custom types work automatically!** As long as your output type implements
|
/// **Custom types work automatically!** As long as your output type implements
|
||||||
/// `Clone + Send + Sync + 'static`, the macro handles everything. No trait implementations needed!
|
/// `Send + Sync + 'static`, the macro handles everything. No trait implementations needed!
|
||||||
///
|
///
|
||||||
/// # Task Patterns
|
/// # Task Patterns
|
||||||
///
|
///
|
||||||
|
|||||||
Reference in new issue
Block a user