Implements dagx-11: Add comprehensive testing infrastructure including fuzzing, property-based testing, and mutation testing configuration. **Fuzzing Infrastructure:** - Created fuzz/ directory with cargo-fuzz setup - 4 fuzz targets covering different scenarios: - fuzz_dag_execution: Basic DAG execution with various patterns - fuzz_dependencies: Dependency management (linear, diamond, complex) - fuzz_type_safety: Type-safe output retrieval - fuzz_concurrent: Concurrent execution and thread safety - Comprehensive README with usage instructions - Integrated with justfile (fuzz, fuzz-all, fuzz-extended, fuzz-build) **Property-Based Testing:** - Added proptest dependency to Cargo.toml - Created tests/property_tests.rs with 14 property tests: - Value preservation through DAG - Linear chain correctness - Fan-out pattern verification - Diamond pattern correctness - String, Vec, Option, Result handling - HashMap and custom struct preservation - DAG isolation - Concurrent execution scaling - Integrated with justfile (proptest recipe) **Mutation Testing:** - Created .cargo-mutants.toml configuration - Focused on critical functions: - DagRunner::run and ::get - TaskBuilder::depends_on - ExecutableNode::execute_with_listeners - ExtractFromListeners::extract - DepsTuple::to_node_ids - Excluded low-value mutations (fmt, clone, default) - Configured for nextest and parallel execution - Already integrated via existing justfile mutants recipe **Justfile Updates:** - Added fuzzing section with 5 new recipes - Added proptest recipe for property-based tests - All recipes follow existing patterns and naming conventions Similar setup to ../announcement project but tailored for DAG-specific testing.
192 lines
4.8 KiB
Markdown
192 lines
4.8 KiB
Markdown
# Fuzz Testing for dagx
|
|
|
|
This directory contains fuzz testing infrastructure using [cargo-fuzz](https://github.com/rust-fuzz/cargo-fuzz) and libFuzzer.
|
|
|
|
## Overview
|
|
|
|
We have 4 fuzz targets that test different aspects of the `dagx` library:
|
|
|
|
1. **fuzz_dag_execution** - Tests DAG execution with various task configurations
|
|
2. **fuzz_dependencies** - Tests dependency patterns (linear, diamond, fan-out, complex)
|
|
3. **fuzz_type_safety** - Tests type-safe output retrieval with various types
|
|
4. **fuzz_concurrent** - Tests concurrent task execution and thread safety
|
|
|
|
## Prerequisites
|
|
|
|
Fuzzing requires nightly Rust and cargo-fuzz:
|
|
|
|
```bash
|
|
rustup install nightly
|
|
cargo install cargo-fuzz
|
|
```
|
|
|
|
## Running Fuzz Tests
|
|
|
|
Run a single fuzz target:
|
|
|
|
```bash
|
|
cargo +nightly fuzz run fuzz_dag_execution
|
|
```
|
|
|
|
Run with a time limit (recommended for CI):
|
|
|
|
```bash
|
|
cargo +nightly fuzz run fuzz_dag_execution -- -max_total_time=60
|
|
```
|
|
|
|
Run with a specific number of iterations:
|
|
|
|
```bash
|
|
cargo +nightly fuzz run fuzz_dag_execution -- -runs=10000
|
|
```
|
|
|
|
Run all fuzz targets sequentially:
|
|
|
|
```bash
|
|
for target in fuzz_dag_execution fuzz_dependencies fuzz_type_safety fuzz_concurrent; do
|
|
echo "Running $target..."
|
|
cargo +nightly fuzz run $target -- -runs=10000
|
|
done
|
|
```
|
|
|
|
## Continuous Fuzzing
|
|
|
|
For deeper testing, run fuzz targets for extended periods:
|
|
|
|
```bash
|
|
# Run overnight (8 hours)
|
|
cargo +nightly fuzz run fuzz_concurrent -- -max_total_time=28800
|
|
|
|
# Run until crash (or Ctrl+C)
|
|
cargo +nightly fuzz run fuzz_dag_execution
|
|
```
|
|
|
|
## Understanding Output
|
|
|
|
Fuzzer output includes:
|
|
|
|
- `cov: X` - Code coverage (number of edges covered)
|
|
- `ft: X` - Feature coverage (number of unique code paths)
|
|
- `corp: X/Yb` - Corpus size (X inputs, Y bytes total)
|
|
- `exec/s: X` - Executions per second
|
|
|
|
If a crash is found:
|
|
|
|
- Crash input is saved to `artifacts/FUZZ_TARGET/crash-*`
|
|
- Reproduce with: `cargo +nightly fuzz run FUZZ_TARGET artifacts/FUZZ_TARGET/crash-*`
|
|
|
|
## Fuzz Target Details
|
|
|
|
### fuzz_dag_execution
|
|
|
|
Tests basic DAG execution:
|
|
- Simple linear task chains
|
|
- Fan-out patterns (one task → multiple dependents)
|
|
- Various data types (i32, Vec<u8>, custom structs)
|
|
- DAG execution correctness
|
|
- Output value verification
|
|
|
|
### fuzz_dependencies
|
|
|
|
Tests dependency management:
|
|
- Linear dependency chains
|
|
- Diamond patterns (source → two paths → merge)
|
|
- Multiple independent sources
|
|
- Complex dependency graphs
|
|
- Dependency tracking correctness
|
|
|
|
### fuzz_type_safety
|
|
|
|
Tests type system correctness:
|
|
- Various output types (i32, String, Vec, custom structs)
|
|
- Correct type retrieval with dag.get()
|
|
- Option<T> and Result<T, E> handling
|
|
- Custom struct preservation
|
|
- Type mismatch error handling
|
|
|
|
### fuzz_concurrent
|
|
|
|
Tests thread safety and concurrency:
|
|
- Multiple independent concurrent tasks
|
|
- Concurrent dependent tasks (fan-out)
|
|
- Arc<DagRunner> sharing across threads
|
|
- Task timing variations
|
|
- Result access from multiple threads
|
|
|
|
## Adding New Fuzz Targets
|
|
|
|
1. Create a new file in `fuzz_targets/` (e.g., `fuzz_new_feature.rs`)
|
|
2. Use the template:
|
|
|
|
```rust
|
|
#![no_main]
|
|
use libfuzzer_sys::fuzz_target;
|
|
use dagx::{task, DagRunner};
|
|
|
|
fuzz_target!(|data: &[u8]| {
|
|
// Your fuzz test here
|
|
});
|
|
```
|
|
|
|
3. Add to `fuzz/Cargo.toml`:
|
|
|
|
```toml
|
|
[[bin]]
|
|
name = "fuzz_new_feature"
|
|
path = "fuzz_targets/fuzz_new_feature.rs"
|
|
test = false
|
|
doc = false
|
|
bench = false
|
|
```
|
|
|
|
4. Build and run:
|
|
|
|
```bash
|
|
cargo +nightly fuzz build
|
|
cargo +nightly fuzz run fuzz_new_feature
|
|
```
|
|
|
|
## CI Integration
|
|
|
|
Fuzzing can run automatically in CI on every push/PR:
|
|
- Each target runs for 60-120 seconds
|
|
- Targets run 1,000-10,000 iterations minimum
|
|
- Any crashes fail the build
|
|
|
|
See your CI configuration for details.
|
|
|
|
## Corpus Management
|
|
|
|
The fuzzer builds a corpus of interesting inputs in `corpus/FUZZ_TARGET/`. These inputs:
|
|
- Are automatically discovered during fuzzing
|
|
- Can be committed to Git for regression testing
|
|
- Help future fuzzing sessions start with good coverage
|
|
|
|
To minimize the corpus:
|
|
|
|
```bash
|
|
cargo +nightly fuzz cmin fuzz_dag_execution
|
|
```
|
|
|
|
## Tips
|
|
|
|
1. **Start with short runs** - Use `-max_total_time=60` to catch obvious bugs quickly
|
|
2. **Run overnight** - For thorough testing, run for hours or days
|
|
3. **Check coverage** - Monitor `cov:` values to ensure good code coverage
|
|
4. **Save crashes** - Commit crash artifacts as regression tests
|
|
5. **Update regularly** - Re-run fuzzing when adding new features
|
|
6. **Limit data size** - Fuzz targets filter out oversized inputs to avoid timeouts
|
|
|
|
## Performance Notes
|
|
|
|
- DAG execution involves async runtime overhead
|
|
- Each fuzz iteration creates a new Tokio runtime
|
|
- Typical execution rate: 100-1000 runs/second
|
|
- Adjust `-runs` or `-max_total_time` based on your needs
|
|
|
|
## References
|
|
|
|
- [cargo-fuzz book](https://rust-fuzz.github.io/book/cargo-fuzz.html)
|
|
- [libFuzzer documentation](https://llvm.org/docs/LibFuzzer.html)
|
|
- [Fuzzing in Rust](https://rust-fuzz.github.io/book/)
|