dagx/PERFORMANCE_REGRESSION_ANALYSIS.md

Performance Regression Analysis: ETL Benchmark

Summary

The ETL scaling benchmark regressed by ~17% (3.24ms -> 3.30ms for 10k records) due to expensive data cloning when using the announcement crate.

Root Cause

In both dagx-macros/src/lib.rs:327 and src/task.rs:253:

let _ = announcer_typed.announce((*output_arc).clone());

This dereferences the Arc and clones the actual data, which is extremely expensive for large types.

For ETL benchmark with 10k strings:

Current (Broken) Flow:

1. Task produces Vec<String> (10k items)
2. Wrap in Arc<Vec<String>>
3. Announce (*arc).clone() → Full Vec clone #1 (EXPENSIVE!)
4. Announcement stores cloned Vec in OnceLock<Vec<String>>
5. Each of 3 listeners calls .listen() → Vec clone #2, #3, #4 (EXPENSIVE!)

Total: 4 full Vec clones = 40k string clones

Why other benchmarks improved:

Most benchmarks use primitive types (i32, etc.) where cloning is cheap. The announcement crate's broadcast pattern is more efficient than multiple oneshot channels for small types.

The Fix

Change announcements to use Arc<T> instead of T:

New (Correct) Flow:

1. Task produces Vec<String> (10k items)
2. Wrap in Arc<Vec<String>>
3. Announce Arc::clone(&arc) → Arc clone (CHEAP!)
4. Announcement stores Arc in OnceLock<Arc<Vec<String>>>
5. Each of 3 listeners calls .listen() → Arc clone #2, #3, #4 (CHEAP!)

Total: 4 Arc clones = just refcount increments (~40ns total)

Required Changes

1. dagx-macros/src/lib.rs

// Line 327: Change announcement call
- let _ = announcer_typed.announce((*output_arc).clone());
+ let _ = announcer_typed.announce(Arc::clone(&output_arc));

// Line 325: Change announcer type
- .downcast::<announcement::Announcer<#output_type>>()
+ .downcast::<announcement::Announcer<Arc<#output_type>>>()

// Line 352: Change announcement creation
- let (announcer, announcement) = announcement::Announcement::<#output_type>::new();
+ let (announcer, announcement) = announcement::Announcement::<Arc<#output_type>>::new();

// Line 257: Change listener type in parameter extraction
- .downcast_ref::<announcement::Listener<#ty>>()
+ .downcast_ref::<announcement::Listener<Arc<#ty>>>()

// Line 361: Change announcement downcast
- .downcast_ref::<announcement::Announcement<#output_type>>()
+ .downcast_ref::<announcement::Announcement<Arc<#output_type>>>()

2. src/task.rs

// Line 253: Change announcement call
- let _ = announcer_typed.announce((*output_arc).clone());
+ let _ = announcer_typed.announce(Arc::clone(&output_arc));

// Line 249: Change announcer type
- .downcast::<announcement::Announcer<O>>()
+ .downcast::<announcement::Announcer<Arc<O>>>()

// Line 261: Change announcement creation
- let (announcer, announcement) = announcement::Announcement::<O>::new();
+ let (announcer, announcement) = announcement::Announcement::<Arc<O>>::new();

// Line 270: Change announcement downcast
- .downcast_ref::<announcement::Announcement<O>>()
+ .downcast_ref::<announcement::Announcement<Arc<O>>>()

// Lines 34, 57, 67, 84, 102, 125: Change all listener downcasts from:
- .downcast::<announcement::Listener<T>>()
+ .downcast::<announcement::Listener<Arc<T>>>()

// Then unwrap the Arc after listening:
let arc_value = listener.listen().await?;
let value = (*arc_value).clone(); // Only clone primitives, Arc is already cheap for complex types

3. Handle Arc unwrapping in ExtractFromListeners

For each type implementation, extract from Listener<Arc<T>> and then dereference:

impl ExtractFromListeners for Vec<T> {
    async fn extract(mut listeners: Vec<Box<dyn Any + Send>>) -> Result<Self, String> {
        let listener = *listeners.pop().unwrap()
            .downcast::<announcement::Listener<Arc<Vec<T>>>>() // Now Arc<Vec<T>>
            .map_err(...)?;
        let arc_vec = listener.listen().await.map_err(...)?;
        Ok((*arc_vec).clone()) // Clone the Vec out of the Arc
    }
}

Actually, a better approach: keep the Arc! For types that are expensive to clone, just return the Arc:

// For expensive types, use Arc directly in function signatures
async fn run(&self, data: &Arc<Vec<String>>) -> Result { ... }

Or even simpler: change ExtractFromListeners to extract as the Arc type directly, and modify the macro's parameter extraction to work with Arc types.

Expected Performance Impact

• ETL 10k benchmark: Should improve by ~17% back to baseline (or better)
• Other benchmarks: Should remain the same or slightly improve (Arc clone overhead is minimal)
• Memory efficiency: Better! Arc refcounting instead of deep copies

Why This Wasn't Caught

• Most tests use small primitive types where clone() is fast
• The regression only shows up with large data structures (Vec, HashMap, etc.)
• The ETL benchmark with 10k records is the perfect stress test for this issue