test: add comprehensive testing infrastructure (fuzzing, proptest, mutation)
Implements dagx-11: Add comprehensive testing infrastructure including fuzzing, property-based testing, and mutation testing configuration. **Fuzzing Infrastructure:** - Created fuzz/ directory with cargo-fuzz setup - 4 fuzz targets covering different scenarios: - fuzz_dag_execution: Basic DAG execution with various patterns - fuzz_dependencies: Dependency management (linear, diamond, complex) - fuzz_type_safety: Type-safe output retrieval - fuzz_concurrent: Concurrent execution and thread safety - Comprehensive README with usage instructions - Integrated with justfile (fuzz, fuzz-all, fuzz-extended, fuzz-build) **Property-Based Testing:** - Added proptest dependency to Cargo.toml - Created tests/property_tests.rs with 14 property tests: - Value preservation through DAG - Linear chain correctness - Fan-out pattern verification - Diamond pattern correctness - String, Vec, Option, Result handling - HashMap and custom struct preservation - DAG isolation - Concurrent execution scaling - Integrated with justfile (proptest recipe) **Mutation Testing:** - Created .cargo-mutants.toml configuration - Focused on critical functions: - DagRunner::run and ::get - TaskBuilder::depends_on - ExecutableNode::execute_with_listeners - ExtractFromListeners::extract - DepsTuple::to_node_ids - Excluded low-value mutations (fmt, clone, default) - Configured for nextest and parallel execution - Already integrated via existing justfile mutants recipe **Justfile Updates:** - Added fuzzing section with 5 new recipes - Added proptest recipe for property-based tests - All recipes follow existing patterns and naming conventions Similar setup to ../announcement project but tailored for DAG-specific testing.
This commit is contained in:
20 files changed
+6545
No files matched your search
+191
@@ -0,0 +1,191 @@
|
||||
# Fuzz Testing for dagx
|
||||
|
||||
This directory contains fuzz testing infrastructure using [cargo-fuzz](https://github.com/rust-fuzz/cargo-fuzz) and libFuzzer.
|
||||
|
||||
## Overview
|
||||
|
||||
We have 4 fuzz targets that test different aspects of the `dagx` library:
|
||||
|
||||
1. **fuzz_dag_execution** - Tests DAG execution with various task configurations
|
||||
2. **fuzz_dependencies** - Tests dependency patterns (linear, diamond, fan-out, complex)
|
||||
3. **fuzz_type_safety** - Tests type-safe output retrieval with various types
|
||||
4. **fuzz_concurrent** - Tests concurrent task execution and thread safety
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Fuzzing requires nightly Rust and cargo-fuzz:
|
||||
|
||||
```bash
|
||||
rustup install nightly
|
||||
cargo install cargo-fuzz
|
||||
```
|
||||
|
||||
## Running Fuzz Tests
|
||||
|
||||
Run a single fuzz target:
|
||||
|
||||
```bash
|
||||
cargo +nightly fuzz run fuzz_dag_execution
|
||||
```
|
||||
|
||||
Run with a time limit (recommended for CI):
|
||||
|
||||
```bash
|
||||
cargo +nightly fuzz run fuzz_dag_execution -- -max_total_time=60
|
||||
```
|
||||
|
||||
Run with a specific number of iterations:
|
||||
|
||||
```bash
|
||||
cargo +nightly fuzz run fuzz_dag_execution -- -runs=10000
|
||||
```
|
||||
|
||||
Run all fuzz targets sequentially:
|
||||
|
||||
```bash
|
||||
for target in fuzz_dag_execution fuzz_dependencies fuzz_type_safety fuzz_concurrent; do
|
||||
echo "Running $target..."
|
||||
cargo +nightly fuzz run $target -- -runs=10000
|
||||
done
|
||||
```
|
||||
|
||||
## Continuous Fuzzing
|
||||
|
||||
For deeper testing, run fuzz targets for extended periods:
|
||||
|
||||
```bash
|
||||
# Run overnight (8 hours)
|
||||
cargo +nightly fuzz run fuzz_concurrent -- -max_total_time=28800
|
||||
|
||||
# Run until crash (or Ctrl+C)
|
||||
cargo +nightly fuzz run fuzz_dag_execution
|
||||
```
|
||||
|
||||
## Understanding Output
|
||||
|
||||
Fuzzer output includes:
|
||||
|
||||
- `cov: X` - Code coverage (number of edges covered)
|
||||
- `ft: X` - Feature coverage (number of unique code paths)
|
||||
- `corp: X/Yb` - Corpus size (X inputs, Y bytes total)
|
||||
- `exec/s: X` - Executions per second
|
||||
|
||||
If a crash is found:
|
||||
|
||||
- Crash input is saved to `artifacts/FUZZ_TARGET/crash-*`
|
||||
- Reproduce with: `cargo +nightly fuzz run FUZZ_TARGET artifacts/FUZZ_TARGET/crash-*`
|
||||
|
||||
## Fuzz Target Details
|
||||
|
||||
### fuzz_dag_execution
|
||||
|
||||
Tests basic DAG execution:
|
||||
- Simple linear task chains
|
||||
- Fan-out patterns (one task → multiple dependents)
|
||||
- Various data types (i32, Vec<u8>, custom structs)
|
||||
- DAG execution correctness
|
||||
- Output value verification
|
||||
|
||||
### fuzz_dependencies
|
||||
|
||||
Tests dependency management:
|
||||
- Linear dependency chains
|
||||
- Diamond patterns (source → two paths → merge)
|
||||
- Multiple independent sources
|
||||
- Complex dependency graphs
|
||||
- Dependency tracking correctness
|
||||
|
||||
### fuzz_type_safety
|
||||
|
||||
Tests type system correctness:
|
||||
- Various output types (i32, String, Vec, custom structs)
|
||||
- Correct type retrieval with dag.get()
|
||||
- Option<T> and Result<T, E> handling
|
||||
- Custom struct preservation
|
||||
- Type mismatch error handling
|
||||
|
||||
### fuzz_concurrent
|
||||
|
||||
Tests thread safety and concurrency:
|
||||
- Multiple independent concurrent tasks
|
||||
- Concurrent dependent tasks (fan-out)
|
||||
- Arc<DagRunner> sharing across threads
|
||||
- Task timing variations
|
||||
- Result access from multiple threads
|
||||
|
||||
## Adding New Fuzz Targets
|
||||
|
||||
1. Create a new file in `fuzz_targets/` (e.g., `fuzz_new_feature.rs`)
|
||||
2. Use the template:
|
||||
|
||||
```rust
|
||||
#![no_main]
|
||||
use libfuzzer_sys::fuzz_target;
|
||||
use dagx::{task, DagRunner};
|
||||
|
||||
fuzz_target!(|data: &[u8]| {
|
||||
// Your fuzz test here
|
||||
});
|
||||
```
|
||||
|
||||
3. Add to `fuzz/Cargo.toml`:
|
||||
|
||||
```toml
|
||||
[[bin]]
|
||||
name = "fuzz_new_feature"
|
||||
path = "fuzz_targets/fuzz_new_feature.rs"
|
||||
test = false
|
||||
doc = false
|
||||
bench = false
|
||||
```
|
||||
|
||||
4. Build and run:
|
||||
|
||||
```bash
|
||||
cargo +nightly fuzz build
|
||||
cargo +nightly fuzz run fuzz_new_feature
|
||||
```
|
||||
|
||||
## CI Integration
|
||||
|
||||
Fuzzing can run automatically in CI on every push/PR:
|
||||
- Each target runs for 60-120 seconds
|
||||
- Targets run 1,000-10,000 iterations minimum
|
||||
- Any crashes fail the build
|
||||
|
||||
See your CI configuration for details.
|
||||
|
||||
## Corpus Management
|
||||
|
||||
The fuzzer builds a corpus of interesting inputs in `corpus/FUZZ_TARGET/`. These inputs:
|
||||
- Are automatically discovered during fuzzing
|
||||
- Can be committed to Git for regression testing
|
||||
- Help future fuzzing sessions start with good coverage
|
||||
|
||||
To minimize the corpus:
|
||||
|
||||
```bash
|
||||
cargo +nightly fuzz cmin fuzz_dag_execution
|
||||
```
|
||||
|
||||
## Tips
|
||||
|
||||
1. **Start with short runs** - Use `-max_total_time=60` to catch obvious bugs quickly
|
||||
2. **Run overnight** - For thorough testing, run for hours or days
|
||||
3. **Check coverage** - Monitor `cov:` values to ensure good code coverage
|
||||
4. **Save crashes** - Commit crash artifacts as regression tests
|
||||
5. **Update regularly** - Re-run fuzzing when adding new features
|
||||
6. **Limit data size** - Fuzz targets filter out oversized inputs to avoid timeouts
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- DAG execution involves async runtime overhead
|
||||
- Each fuzz iteration creates a new Tokio runtime
|
||||
- Typical execution rate: 100-1000 runs/second
|
||||
- Adjust `-runs` or `-max_total_time` based on your needs
|
||||
|
||||
## References
|
||||
|
||||
- [cargo-fuzz book](https://rust-fuzz.github.io/book/cargo-fuzz.html)
|
||||
- [libFuzzer documentation](https://llvm.org/docs/LibFuzzer.html)
|
||||
- [Fuzzing in Rust](https://rust-fuzz.github.io/book/)
|
||||
Reference in new issue
Block a user