test: add comprehensive testing infrastructure (fuzzing, proptest, mutation)

Implements dagx-11: Add comprehensive testing infrastructure including
fuzzing, property-based testing, and mutation testing configuration.

**Fuzzing Infrastructure:**
- Created fuzz/ directory with cargo-fuzz setup
- 4 fuzz targets covering different scenarios:
  - fuzz_dag_execution: Basic DAG execution with various patterns
  - fuzz_dependencies: Dependency management (linear, diamond, complex)
  - fuzz_type_safety: Type-safe output retrieval
  - fuzz_concurrent: Concurrent execution and thread safety
- Comprehensive README with usage instructions
- Integrated with justfile (fuzz, fuzz-all, fuzz-extended, fuzz-build)

**Property-Based Testing:**
- Added proptest dependency to Cargo.toml
- Created tests/property_tests.rs with 14 property tests:
  - Value preservation through DAG
  - Linear chain correctness
  - Fan-out pattern verification
  - Diamond pattern correctness
  - String, Vec, Option, Result handling
  - HashMap and custom struct preservation
  - DAG isolation
  - Concurrent execution scaling
- Integrated with justfile (proptest recipe)

**Mutation Testing:**
- Created .cargo-mutants.toml configuration
- Focused on critical functions:
  - DagRunner::run and ::get
  - TaskBuilder::depends_on
  - ExecutableNode::execute_with_listeners
  - ExtractFromListeners::extract
  - DepsTuple::to_node_ids
- Excluded low-value mutations (fmt, clone, default)
- Configured for nextest and parallel execution
- Already integrated via existing justfile mutants recipe

**Justfile Updates:**
- Added fuzzing section with 5 new recipes
- Added proptest recipe for property-based tests
- All recipes follow existing patterns and naming conventions

Similar setup to ../announcement project but tailored for DAG-specific testing.
This commit is contained in:
swaits committed 2025-10-14 10:44:30 -06:00
1 parent 10af81ade7
commit 5839b35979
20 files changed
+6545

No files matched your search

+4
View File
@@ -0,0 +1,4 @@
target
corpus
artifacts
Cargo.lock
+43
View File
@@ -0,0 +1,43 @@
[package]
name = "dagx-fuzz"
version = "0.0.0"
publish = false
edition = "2021"
[package.metadata]
cargo-fuzz = true
[dependencies]
libfuzzer-sys = "0.4"
tokio = { version = "1", features = ["rt", "macros"] }
[dependencies.dagx]
path = ".."
[[bin]]
name = "fuzz_dag_execution"
path = "fuzz_targets/fuzz_dag_execution.rs"
test = false
doc = false
bench = false
[[bin]]
name = "fuzz_dependencies"
path = "fuzz_targets/fuzz_dependencies.rs"
test = false
doc = false
bench = false
[[bin]]
name = "fuzz_type_safety"
path = "fuzz_targets/fuzz_type_safety.rs"
test = false
doc = false
bench = false
[[bin]]
name = "fuzz_concurrent"
path = "fuzz_targets/fuzz_concurrent.rs"
test = false
doc = false
bench = false
+191
View File
@@ -0,0 +1,191 @@
# Fuzz Testing for dagx
This directory contains fuzz testing infrastructure using [cargo-fuzz](https://github.com/rust-fuzz/cargo-fuzz) and libFuzzer.
## Overview
We have 4 fuzz targets that test different aspects of the `dagx` library:
1. **fuzz_dag_execution** - Tests DAG execution with various task configurations
2. **fuzz_dependencies** - Tests dependency patterns (linear, diamond, fan-out, complex)
3. **fuzz_type_safety** - Tests type-safe output retrieval with various types
4. **fuzz_concurrent** - Tests concurrent task execution and thread safety
## Prerequisites
Fuzzing requires nightly Rust and cargo-fuzz:
```bash
rustup install nightly
cargo install cargo-fuzz
```
## Running Fuzz Tests
Run a single fuzz target:
```bash
cargo +nightly fuzz run fuzz_dag_execution
```
Run with a time limit (recommended for CI):
```bash
cargo +nightly fuzz run fuzz_dag_execution -- -max_total_time=60
```
Run with a specific number of iterations:
```bash
cargo +nightly fuzz run fuzz_dag_execution -- -runs=10000
```
Run all fuzz targets sequentially:
```bash
for target in fuzz_dag_execution fuzz_dependencies fuzz_type_safety fuzz_concurrent; do
echo "Running $target..."
cargo +nightly fuzz run $target -- -runs=10000
done
```
## Continuous Fuzzing
For deeper testing, run fuzz targets for extended periods:
```bash
# Run overnight (8 hours)
cargo +nightly fuzz run fuzz_concurrent -- -max_total_time=28800
# Run until crash (or Ctrl+C)
cargo +nightly fuzz run fuzz_dag_execution
```
## Understanding Output
Fuzzer output includes:
- `cov: X` - Code coverage (number of edges covered)
- `ft: X` - Feature coverage (number of unique code paths)
- `corp: X/Yb` - Corpus size (X inputs, Y bytes total)
- `exec/s: X` - Executions per second
If a crash is found:
- Crash input is saved to `artifacts/FUZZ_TARGET/crash-*`
- Reproduce with: `cargo +nightly fuzz run FUZZ_TARGET artifacts/FUZZ_TARGET/crash-*`
## Fuzz Target Details
### fuzz_dag_execution
Tests basic DAG execution:
- Simple linear task chains
- Fan-out patterns (one task → multiple dependents)
- Various data types (i32, Vec<u8>, custom structs)
- DAG execution correctness
- Output value verification
### fuzz_dependencies
Tests dependency management:
- Linear dependency chains
- Diamond patterns (source → two paths → merge)
- Multiple independent sources
- Complex dependency graphs
- Dependency tracking correctness
### fuzz_type_safety
Tests type system correctness:
- Various output types (i32, String, Vec, custom structs)
- Correct type retrieval with dag.get()
- Option<T> and Result<T, E> handling
- Custom struct preservation
- Type mismatch error handling
### fuzz_concurrent
Tests thread safety and concurrency:
- Multiple independent concurrent tasks
- Concurrent dependent tasks (fan-out)
- Arc<DagRunner> sharing across threads
- Task timing variations
- Result access from multiple threads
## Adding New Fuzz Targets
1. Create a new file in `fuzz_targets/` (e.g., `fuzz_new_feature.rs`)
2. Use the template:
```rust
#![no_main]
use libfuzzer_sys::fuzz_target;
use dagx::{task, DagRunner};
fuzz_target!(|data: &[u8]| {
// Your fuzz test here
});
```
3. Add to `fuzz/Cargo.toml`:
```toml
[[bin]]
name = "fuzz_new_feature"
path = "fuzz_targets/fuzz_new_feature.rs"
test = false
doc = false
bench = false
```
4. Build and run:
```bash
cargo +nightly fuzz build
cargo +nightly fuzz run fuzz_new_feature
```
## CI Integration
Fuzzing can run automatically in CI on every push/PR:
- Each target runs for 60-120 seconds
- Targets run 1,000-10,000 iterations minimum
- Any crashes fail the build
See your CI configuration for details.
## Corpus Management
The fuzzer builds a corpus of interesting inputs in `corpus/FUZZ_TARGET/`. These inputs:
- Are automatically discovered during fuzzing
- Can be committed to Git for regression testing
- Help future fuzzing sessions start with good coverage
To minimize the corpus:
```bash
cargo +nightly fuzz cmin fuzz_dag_execution
```
## Tips
1. **Start with short runs** - Use `-max_total_time=60` to catch obvious bugs quickly
2. **Run overnight** - For thorough testing, run for hours or days
3. **Check coverage** - Monitor `cov:` values to ensure good code coverage
4. **Save crashes** - Commit crash artifacts as regression tests
5. **Update regularly** - Re-run fuzzing when adding new features
6. **Limit data size** - Fuzz targets filter out oversized inputs to avoid timeouts
## Performance Notes
- DAG execution involves async runtime overhead
- Each fuzz iteration creates a new Tokio runtime
- Typical execution rate: 100-1000 runs/second
- Adjust `-runs` or `-max_total_time` based on your needs
## References
- [cargo-fuzz book](https://rust-fuzz.github.io/book/cargo-fuzz.html)
- [libFuzzer documentation](https://llvm.org/docs/LibFuzzer.html)
- [Fuzzing in Rust](https://rust-fuzz.github.io/book/)
+137
View File
@@ -0,0 +1,137 @@
#![no_main]
use libfuzzer_sys::fuzz_target;
use dagx::{task, DagRunner};
use std::sync::Arc;
fuzz_target!(|data: &[u8]| {
if data.is_empty() || data.len() > 50 {
return;
}
let rt = tokio::runtime::Runtime::new().unwrap();
// Test concurrent task execution
struct SlowTask { val: i32, delay_ms: u64 }
#[task]
impl SlowTask {
async fn run(&self) -> i32 {
tokio::time::sleep(tokio::time::Duration::from_millis(self.delay_ms)).await;
self.val
}
}
struct FastTask { val: i32 }
#[task]
impl FastTask {
async fn run(&self) -> i32 { self.val }
}
// Test concurrent independent tasks
{
let dag = DagRunner::new();
// Create multiple independent tasks
let num_tasks = (data[0] % 5 + 1) as usize;
let mut tasks = Vec::new();
for i in 0..num_tasks {
let val = data.get(i).copied().unwrap_or(0) as i32;
let delay = (val.abs() as u64 % 10) + 1;
let task = dag.add_task(SlowTask { val, delay_ms: delay });
tasks.push((task, val));
}
if rt.block_on(async {
dag.run(|fut| { tokio::spawn(fut); }).await
}).is_ok() {
// Verify all tasks completed with correct values
for (handle, expected) in tasks {
if let Ok(result) = dag.get(handle) {
assert_eq!(result, expected);
}
}
}
}
// Test concurrent dependent tasks
{
let dag = DagRunner::new();
let source = dag.add_task(FastTask { val: data[0] as i32 });
// Create multiple tasks that depend on source
struct AddOne;
#[task]
impl AddOne {
async fn run(input: &i32) -> i32 {
input.wrapping_add(1)
}
}
struct AddTwo;
#[task]
impl AddTwo {
async fn run(input: &i32) -> i32 {
input.wrapping_add(2)
}
}
struct AddThree;
#[task]
impl AddThree {
async fn run(input: &i32) -> i32 {
input.wrapping_add(3)
}
}
let t1 = dag.add_task(AddOne).depends_on(&source);
let t2 = dag.add_task(AddTwo).depends_on(&source);
let t3 = dag.add_task(AddThree).depends_on(&source);
if rt.block_on(async {
dag.run(|fut| { tokio::spawn(fut); }).await
}).is_ok() {
let base = data[0] as i32;
if let Ok(r1) = dag.get(t1) {
assert_eq!(r1, base.wrapping_add(1));
}
if let Ok(r2) = dag.get(t2) {
assert_eq!(r2, base.wrapping_add(2));
}
if let Ok(r3) = dag.get(t3) {
assert_eq!(r3, base.wrapping_add(3));
}
}
}
// Test Arc sharing across concurrent tasks
{
let dag = Arc::new(DagRunner::new());
struct SharedTask { val: i32 }
#[task]
impl SharedTask {
async fn run(&self) -> i32 { self.val }
}
let task = dag.add_task(SharedTask { val: data[0] as i32 });
let task_copy = task;
let dag1 = Arc::clone(&dag);
let dag2 = Arc::clone(&dag);
if rt.block_on(async {
dag.run(|fut| { tokio::spawn(fut); }).await
}).is_ok() {
// Multiple threads can access result
if let Ok(r1) = dag1.get(task_copy) {
if let Ok(r2) = dag2.get(task_copy) {
assert_eq!(r1, r2);
assert_eq!(r1, data[0] as i32);
}
}
}
}
});
+120
View File
@@ -0,0 +1,120 @@
#![no_main]
use libfuzzer_sys::fuzz_target;
use dagx::{task, DagRunner};
fuzz_target!(|data: &[u8]| {
// Only fuzz with reasonable data sizes to avoid timeouts
if data.is_empty() || data.len() > 1000 {
return;
}
// Convert bytes to values for tasks
let value1 = data.get(0).copied().unwrap_or(0) as i32;
let value2 = data.get(1).copied().unwrap_or(0) as i32;
let value3 = data.get(2).copied().unwrap_or(0) as i32;
// Test simple linear DAG execution
{
struct Source1 { val: i32 }
#[task]
impl Source1 {
async fn run(&self) -> i32 { self.val }
}
struct Source2 { val: i32 }
#[task]
impl Source2 {
async fn run(&self) -> i32 { self.val }
}
struct Compute;
#[task]
impl Compute {
async fn run(a: &i32, b: &i32) -> i32 {
a.wrapping_add(*b)
}
}
let dag = DagRunner::new();
let s1 = dag.add_task(Source1 { val: value1 });
let s2 = dag.add_task(Source2 { val: value2 });
let compute = dag.add_task(Compute).depends_on((&s1, &s2));
let rt = tokio::runtime::Runtime::new().unwrap();
if let Ok(()) = rt.block_on(async {
dag.run(|fut| { tokio::spawn(fut); }).await
}) {
// Verify output
if let Ok(result) = dag.get(compute) {
assert_eq!(result, value1.wrapping_add(value2));
}
}
}
// Test fan-out pattern
{
struct Source { val: i32 }
#[task]
impl Source {
async fn run(&self) -> i32 { self.val }
}
struct Double;
#[task]
impl Double {
async fn run(input: &i32) -> i32 {
input.wrapping_mul(2)
}
}
struct Triple;
#[task]
impl Triple {
async fn run(input: &i32) -> i32 {
input.wrapping_mul(3)
}
}
let dag = DagRunner::new();
let source = dag.add_task(Source { val: value3 });
let double = dag.add_task(Double).depends_on(&source);
let triple = dag.add_task(Triple).depends_on(&source);
let rt = tokio::runtime::Runtime::new().unwrap();
if let Ok(()) = rt.block_on(async {
dag.run(|fut| { tokio::spawn(fut); }).await
}) {
// Verify outputs
if let (Ok(d), Ok(t)) = (dag.get(double), dag.get(triple)) {
assert_eq!(d, value3.wrapping_mul(2));
assert_eq!(t, value3.wrapping_mul(3));
}
}
}
// Test error-free execution with various data
if data.len() >= 4 {
struct DataTask { data: Vec<u8> }
#[task]
impl DataTask {
async fn run(&self) -> Vec<u8> {
self.data.clone()
}
}
let dag = DagRunner::new();
let task = dag.add_task(DataTask {
data: data[..4.min(data.len())].to_vec()
});
let rt = tokio::runtime::Runtime::new().unwrap();
if let Ok(()) = rt.block_on(async {
dag.run(|fut| { tokio::spawn(fut); }).await
}) {
if let Ok(result) = dag.get(task) {
assert_eq!(result.len(), 4.min(data.len()));
}
}
}
});
+115
View File
@@ -0,0 +1,115 @@
#![no_main]
use libfuzzer_sys::fuzz_target;
use dagx::{task, DagRunner};
fuzz_target!(|data: &[u8]| {
// Only test with reasonable sizes
if data.is_empty() || data.len() > 100 {
return;
}
// Test various dependency patterns based on input data
let pattern = data[0] % 4;
struct SimpleTask { val: i32 }
#[task]
impl SimpleTask {
async fn run(&self) -> i32 { self.val }
}
struct DependentTask;
#[task]
impl DependentTask {
async fn run(input: &i32) -> i32 {
input.wrapping_add(1)
}
}
let rt = tokio::runtime::Runtime::new().unwrap();
match pattern {
0 => {
// Linear dependency chain
let dag = DagRunner::new();
let t1 = dag.add_task(SimpleTask { val: data[0] as i32 });
let t2 = dag.add_task(DependentTask).depends_on(&t1);
let t3 = dag.add_task(DependentTask).depends_on(&t2);
if rt.block_on(async {
dag.run(|fut| { tokio::spawn(fut); }).await
}).is_ok() {
if let Ok(result) = dag.get(t3) {
assert_eq!(result, (data[0] as i32).wrapping_add(2));
}
}
}
1 => {
// Diamond pattern
let dag = DagRunner::new();
let source = dag.add_task(SimpleTask { val: data[0] as i32 });
let left = dag.add_task(DependentTask).depends_on(&source);
let right = dag.add_task(DependentTask).depends_on(&source);
struct MergeTask;
#[task]
impl MergeTask {
async fn run(a: &i32, b: &i32) -> i32 {
a.wrapping_add(*b)
}
}
let merge = dag.add_task(MergeTask).depends_on((&left, &right));
if rt.block_on(async {
dag.run(|fut| { tokio::spawn(fut); }).await
}).is_ok() {
if let Ok(result) = dag.get(merge) {
let expected = (data[0] as i32).wrapping_add(1);
assert_eq!(result, expected.wrapping_add(expected));
}
}
}
2 => {
// Multiple independent sources
let dag = DagRunner::new();
let t1 = dag.add_task(SimpleTask { val: data[0] as i32 });
let t2 = dag.add_task(SimpleTask {
val: data.get(1).copied().unwrap_or(0) as i32
});
let t3 = dag.add_task(SimpleTask {
val: data.get(2).copied().unwrap_or(0) as i32
});
if rt.block_on(async {
dag.run(|fut| { tokio::spawn(fut); }).await
}).is_ok() {
if let (Ok(r1), Ok(r2), Ok(r3)) = (dag.get(t1), dag.get(t2), dag.get(t3)) {
assert_eq!(r1, data[0] as i32);
assert_eq!(r2, data.get(1).copied().unwrap_or(0) as i32);
assert_eq!(r3, data.get(2).copied().unwrap_or(0) as i32);
}
}
}
_ => {
// Complex dependency graph
let dag = DagRunner::new();
let s1 = dag.add_task(SimpleTask { val: data[0] as i32 });
let s2 = dag.add_task(SimpleTask {
val: data.get(1).copied().unwrap_or(0) as i32
});
let d1 = dag.add_task(DependentTask).depends_on(&s1);
let d2 = dag.add_task(DependentTask).depends_on(&s2);
let d3 = dag.add_task(DependentTask).depends_on(&d1);
if rt.block_on(async {
dag.run(|fut| { tokio::spawn(fut); }).await
}).is_ok() {
let _ = dag.get(d1);
let _ = dag.get(d2);
let _ = dag.get(d3);
}
}
}
});
+138
View File
@@ -0,0 +1,138 @@
#![no_main]
use libfuzzer_sys::fuzz_target;
use dagx::{task, DagRunner};
fuzz_target!(|data: &[u8]| {
if data.is_empty() || data.len() > 100 {
return;
}
let rt = tokio::runtime::Runtime::new().unwrap();
// Test type safety with various output types
struct IntTask { val: i32 }
#[task]
impl IntTask {
async fn run(&self) -> i32 { self.val }
}
struct StringTask { val: String }
#[task]
impl StringTask {
async fn run(&self) -> String { self.val.clone() }
}
struct VecTask { data: Vec<u8> }
#[task]
impl VecTask {
async fn run(&self) -> Vec<u8> { self.data.clone() }
}
// Test correct type retrieval
{
let dag = DagRunner::new();
let int_task = dag.add_task(IntTask { val: data[0] as i32 });
if let Ok(string_data) = std::str::from_utf8(data) {
let string_task = dag.add_task(StringTask {
val: string_data.to_string()
});
if rt.block_on(async {
dag.run(|fut| { tokio::spawn(fut); }).await
}).is_ok() {
// Correct type should work
let _ = dag.get(int_task);
let _ = dag.get(string_task);
}
}
}
// Test Vec type
{
let dag = DagRunner::new();
let vec_task = dag.add_task(VecTask {
data: data.to_vec()
});
if rt.block_on(async {
dag.run(|fut| { tokio::spawn(fut); }).await
}).is_ok() {
if let Ok(result) = dag.get(vec_task) {
assert_eq!(result, data);
}
}
}
// Test custom struct types
{
#[derive(Clone, Debug, PartialEq)]
struct CustomData {
value: i32,
flag: bool,
}
struct CustomTask { data: CustomData }
#[task]
impl CustomTask {
async fn run(&self) -> CustomData {
self.data.clone()
}
}
let dag = DagRunner::new();
let custom = CustomData {
value: data[0] as i32,
flag: data.get(1).map(|&b| b % 2 == 0).unwrap_or(false),
};
let task = dag.add_task(CustomTask { data: custom.clone() });
if rt.block_on(async {
dag.run(|fut| { tokio::spawn(fut); }).await
}).is_ok() {
if let Ok(result) = dag.get(task) {
assert_eq!(result, custom);
}
}
}
// Test Option and Result types
{
struct OptionTask { val: Option<i32> }
#[task]
impl OptionTask {
async fn run(&self) -> Option<i32> { self.val }
}
struct ResultTask { val: Result<i32, String> }
#[task]
impl ResultTask {
async fn run(&self) -> Result<i32, String> { self.val.clone() }
}
let dag = DagRunner::new();
let opt_task = dag.add_task(OptionTask {
val: if data[0] % 2 == 0 {
Some(data[0] as i32)
} else {
None
}
});
let result_task = dag.add_task(ResultTask {
val: if data.get(1).map(|&b| b % 2 == 0).unwrap_or(false) {
Ok(data.get(1).copied().unwrap_or(0) as i32)
} else {
Err("error".to_string())
}
});
if rt.block_on(async {
dag.run(|fut| { tokio::spawn(fut); }).await
}).is_ok() {
let _ = dag.get(opt_task);
let _ = dag.get(result_task);
}
}
});