diff --git a/examples/compare-results.md b/examples/compare-results.md index cbbcd9a..4406884 100644 --- a/examples/compare-results.md +++ b/examples/compare-results.md @@ -17,7 +17,9 @@ in particular fell ~2.7× from the `hypervolume_nd` rework. This refresh also adds three **combinatorial / sequencing** problems — TSP, job-shop scheduling, and a bi-objective knapsack — which exercise the permutation and bitstring operators and a different algorithm roster (the -real-vector methods can't run them). +real-vector methods can't run them) — and three **many-objective** +problems (DTLZ at 4, 10, and 8 objectives) that push past where Pareto +dominance still discriminates. Wall-clock numbers are from the development machine and will vary; the *relative* numbers across algorithms are the interesting part. @@ -238,4 +240,94 @@ bitstrings. No closed-form optimum; scored by hypervolume vs reference | RandomSearch | 1118233.1 ± 34150.3 | 9 | 17 | The three Pareto EAs land within ~1% of each other; random search finds a -front of only ~9 points and trails badly. +front of only ~9 points and trails badly. Note IBEA — which dominates the +*continuous* multi-objective tables — is only mid-pack here: its +continuous-MO edge does not transfer to a binary combinatorial encoding. + +--- + +## Many-objective (4+ objectives) + +The curse of dimensionality for multi-objective optimizers: as objective +count climbs, almost every pair of solutions becomes mutually +non-dominated, so Pareto rank stops discriminating. NSGA-II's whole +population collapses into front 0 and only crowding distance is left to +steer. Reference-point (NSGA-III), decomposition (MOEA/D), +reference-vector (RVEA), grid (GrEA), and indicator (IBEA, HypE) methods +are built for this regime. Scored by mean distance to the true front +(lower better). + +### DTLZ2 4-objective (dim=13, 40000 evals/run × 10 seeds) + +DTLZ2 scaled to 4 objectives — the entry point to many-objective. Front +is still the unit-hypersphere octant (`Σf² = 1`). Already hard: with 4 +objectives most random solution pairs are mutually non-dominated, so +Pareto rank alone barely discriminates. + +| algorithm | mean dist ↓ | front | ms | +|---|---|---|---| +| **HypE** | **0.0005 ± 0.0004** | 56 | 292 | +| MOEA/D | 0.0019 ± 0.0004 | 46 | 33 | +| GrEA | 0.0023 ± 0.0021 | 56 | 75 | +| IBEA | 0.0043 ± 0.0008 | 56 | 135 | +| RVEA | 0.0193 ± 0.0040 | 56 | 58 | +| NSGA-III | 0.0312 ± 0.0046 | 56 | 100 | +| AGE-MOEA | 0.0457 ± 0.0113 | 56 | 239 | +| NSGA-II | 0.1149 ± 0.0249 | 56 | 74 | +| RandomSearch | 0.4720 ± 0.0122 | 887 | 1960 | + +NSGA-II already trails the specialists by ~230× — and its "front" is the +whole population (56), the first sign of dominance resistance. Random +search's front balloons to ~887: nothing it sampled dominates anything +else. + +### DTLZ2 10-objective (dim=19, 40000 evals/run × 10 seeds) + +DTLZ2 at 10 objectives — the curse of dimensionality in full. In 10-D +objective space almost *every* pair of solutions is mutually +non-dominated. + +| algorithm | mean dist ↓ | front | ms | +|---|---|---|---| +| **HypE** | **0.0007 ± 0.0005** | 55 | 555 | +| MOEA/D | 0.0029 ± 0.0022 | 48 | 57 | +| GrEA | 0.0066 ± 0.0145 | 55 | 146 | +| RVEA | 0.0094 ± 0.0066 | 41 | 74 | +| IBEA | 0.0118 ± 0.0033 | 55 | 171 | +| AGE-MOEA | 0.1812 ± 0.0523 | 55 | 529 | +| NSGA-III | 0.3064 ± 0.0327 | 55 | 220 | +| RandomSearch | 0.6326 ± 0.0044 | 4592 | 16131 | +| NSGA-II | 2.0096 ± 0.0540 | 55 | 186 | + +**The headline result.** NSGA-II is *dead last — worse than random +search* (2.01 vs 0.63). Its crowding distance in 10-D doesn't just fail +to help, it actively misleads. The indicator (HypE, IBEA), decomposition +(MOEA/D) and grid (GrEA) methods barely notice the objective-count jump +from 4 to 10; AGE-MOEA and NSGA-III degrade noticeably but still beat +random. + +### DTLZ1 8-objective (dim=12, 40000 evals/run × 10 seeds) + +DTLZ1 at 8 objectives — the brutal one: many-objective dominance collapse +*plus* DTLZ1's deceptive multimodal `g`-term (a huge number of local +fronts). The true front is the linear simplex `Σf = 0.5`; reaching it at +all is the achievement. + +| algorithm | mean dist ↓ | front | ms | +|---|---|---|---| +| **GrEA** | **1.5441 ± 0.3844** | 98 | 183 | +| MOEA/D | 2.2867 ± 2.0553 | 94 | 37 | +| RVEA | 2.4016 ± 1.3780 | 51 | 116 | +| IBEA | 7.9615 ± 3.6041 | 101 | 283 | +| NSGA-III | 26.6956 ± 7.3771 | 120 | 295 | +| HypE | 26.8702 ± 5.5660 | 120 | 375 | +| AGE-MOEA | 43.9530 ± 15.4464 | 120 | 591 | +| RandomSearch | 172.6562 ± 6.8456 | 700 | 2553 | +| NSGA-II | 281.4563 ± 11.9140 | 120 | 277 | + +**GrEA wins** — consistent with the 3-objective DTLZ1 table, where it +also won: grid-based niching matches a linear/simplex front at any +objective count. The other striking result is **HypE's reversal**: #1 on +both DTLZ2 tables, but #6 here — Monte-Carlo hypervolume is a poor +discriminator on the deceptive simplex. NSGA-II again finishes last, +worse than random by ~1.6×. diff --git a/examples/compare.rs b/examples/compare.rs index 0141ee4..0588666 100644 --- a/examples/compare.rs +++ b/examples/compare.rs @@ -2847,6 +2847,320 @@ fn run_knapsack_comparison() { print_table(&["algorithm", "hypervolume", "front", "ms"], &table); } +// ============================================================================= +// Many-objective problems (4+ objectives) +// +// The "curse of dimensionality" for multi-objective optimizers: as the +// objective count climbs, the fraction of mutually non-dominated solution +// pairs rushes toward 1, so Pareto rank alone stops discriminating. +// NSGA-II's whole population collapses into front 0 and only crowding +// distance is left to steer. Reference-point (NSGA-III), decomposition +// (MOEA/D), reference-vector (RVEA), grid (GrEA) and indicator (IBEA, +// HypE) methods are built to survive this regime. +// +// Both DTLZ structs above are already parameterised by objective count, +// and the DTLZ1/DTLZ2 distance metrics generalise to any M, so a single +// `ManySpec` + generic runners cover every objective count. +// ============================================================================= + +/// A DTLZ instance at an arbitrary objective count, plus the budget and +/// the reference-set sizing the many-objective algorithms need. +#[derive(Clone, Copy)] +struct ManySpec { + objectives: usize, + dim: usize, + budget: usize, + /// Population for the fixed-population algorithms; also the + /// Das-Dennis weight count MOEA/D derives from `reference_divisions`. + population: usize, + /// Das-Dennis divisions for NSGA-III / RVEA / MOEA/D reference sets. + reference_divisions: usize, + /// `true` = DTLZ1 (deceptive multimodal, linear-simplex front); + /// `false` = DTLZ2 (unit-hypersphere-octant front). + is_dtlz1: bool, + /// Per-axis HypE hypervolume reference coordinate. + hype_ref: f64, +} + +/// One DTLZ problem type so the generic runners have a single `Problem` +/// to hand to `run` regardless of which front geometry is in play. +enum ManyDtlz { + D1(Dtlz1), + D2(Dtlz2), +} +impl Problem for ManyDtlz { + type Decision = Vec; + fn objectives(&self) -> ObjectiveSpace { + match self { + ManyDtlz::D1(p) => p.objectives(), + ManyDtlz::D2(p) => p.objectives(), + } + } + fn evaluate(&self, x: &Vec) -> Evaluation { + match self { + ManyDtlz::D1(p) => p.evaluate(x), + ManyDtlz::D2(p) => p.evaluate(x), + } + } +} + +impl ManySpec { + fn problem(&self) -> ManyDtlz { + if self.is_dtlz1 { + ManyDtlz::D1(Dtlz1 { + num_objectives: self.objectives, + dim: self.dim, + }) + } else { + ManyDtlz::D2(Dtlz2 { + num_objectives: self.objectives, + dim: self.dim, + }) + } + } + fn bounds(&self) -> Vec<(f64, f64)> { + vec![(0.0, 1.0); self.dim] + } + fn variation(&self) -> CompositeVariation { + let b = self.bounds(); + CompositeVariation { + crossover: SimulatedBinaryCrossover::new(b.clone(), 30.0, 1.0), + mutation: PolynomialMutation::new(b, 20.0, 1.0 / self.dim as f64), + } + } + fn mean_dist(&self, front: &[Candidate>]) -> f64 { + if self.is_dtlz1 { + mean_distance_to_dtlz1_front(front) + } else { + mean_distance_to_dtlz2_front(front) + } + } +} + +fn many_random(spec: ManySpec, seed: u64) -> MoRun { + let problem = spec.problem(); + let mut opt = RandomSearch::new( + RandomSearchConfig { + iterations: spec.budget, + batch_size: 1, + seed, + }, + RealBounds::new(spec.bounds()), + ); + let t0 = Instant::now(); + let result = opt.run(&problem); + MoRun { + front: result.pareto_front, + wall_ms: t0.elapsed().as_millis(), + } +} + +fn many_nsga2(spec: ManySpec, seed: u64) -> MoRun { + let problem = spec.problem(); + let mut opt = Nsga2::new( + Nsga2Config { + population_size: spec.population, + generations: spec.budget / spec.population, + seed, + }, + RealBounds::new(spec.bounds()), + spec.variation(), + ); + let t0 = Instant::now(); + let result = opt.run(&problem); + MoRun { + front: result.pareto_front, + wall_ms: t0.elapsed().as_millis(), + } +} + +fn many_nsga3(spec: ManySpec, seed: u64) -> MoRun { + let problem = spec.problem(); + let mut opt = Nsga3::new( + Nsga3Config { + population_size: spec.population, + generations: spec.budget / spec.population, + reference_divisions: spec.reference_divisions, + seed, + }, + RealBounds::new(spec.bounds()), + spec.variation(), + ); + let t0 = Instant::now(); + let result = opt.run(&problem); + MoRun { + front: result.pareto_front, + wall_ms: t0.elapsed().as_millis(), + } +} + +fn many_moead(spec: ManySpec, seed: u64) -> MoRun { + let problem = spec.problem(); + let mut opt = Moead::new( + MoeadConfig { + generations: spec.budget / spec.population, + reference_divisions: spec.reference_divisions, + neighborhood_size: 20, + seed, + }, + RealBounds::new(spec.bounds()), + spec.variation(), + ); + let t0 = Instant::now(); + let result = opt.run(&problem); + MoRun { + front: result.pareto_front, + wall_ms: t0.elapsed().as_millis(), + } +} + +fn many_rvea(spec: ManySpec, seed: u64) -> MoRun { + let problem = spec.problem(); + let mut opt = Rvea::new( + RveaConfig { + population_size: spec.population, + generations: spec.budget / spec.population, + reference_divisions: spec.reference_divisions, + alpha: 2.0, + seed, + }, + RealBounds::new(spec.bounds()), + spec.variation(), + ); + let t0 = Instant::now(); + let result = opt.run(&problem); + MoRun { + front: result.pareto_front, + wall_ms: t0.elapsed().as_millis(), + } +} + +fn many_grea(spec: ManySpec, seed: u64) -> MoRun { + let problem = spec.problem(); + let mut opt = Grea::new( + GreaConfig { + population_size: spec.population, + generations: spec.budget / spec.population, + grid_divisions: 10, + seed, + }, + RealBounds::new(spec.bounds()), + spec.variation(), + ); + let t0 = Instant::now(); + let result = opt.run(&problem); + MoRun { + front: result.pareto_front, + wall_ms: t0.elapsed().as_millis(), + } +} + +fn many_ibea(spec: ManySpec, seed: u64) -> MoRun { + let problem = spec.problem(); + let mut opt = Ibea::new( + IbeaConfig { + population_size: spec.population, + generations: spec.budget / spec.population, + kappa: 0.05, + seed, + }, + RealBounds::new(spec.bounds()), + spec.variation(), + ); + let t0 = Instant::now(); + let result = opt.run(&problem); + MoRun { + front: result.pareto_front, + wall_ms: t0.elapsed().as_millis(), + } +} + +fn many_hype(spec: ManySpec, seed: u64) -> MoRun { + let problem = spec.problem(); + let mut opt = Hype::new( + HypeConfig { + population_size: spec.population, + generations: spec.budget / spec.population, + reference_point: vec![spec.hype_ref; spec.objectives], + mc_samples: 1_000, + seed, + }, + RealBounds::new(spec.bounds()), + spec.variation(), + ); + let t0 = Instant::now(); + let result = opt.run(&problem); + MoRun { + front: result.pareto_front, + wall_ms: t0.elapsed().as_millis(), + } +} + +fn many_age_moea(spec: ManySpec, seed: u64) -> MoRun { + let problem = spec.problem(); + let mut opt = AgeMoea::new( + AgeMoeaConfig { + population_size: spec.population, + generations: spec.budget / spec.population, + seed, + }, + RealBounds::new(spec.bounds()), + spec.variation(), + ); + let t0 = Instant::now(); + let result = opt.run(&problem); + MoRun { + front: result.pareto_front, + wall_ms: t0.elapsed().as_millis(), + } +} + +fn run_many_objective_comparison(title: &str, blurb: &[&str], spec: ManySpec) { + println!(); + println!("== {title} =="); + for line in blurb { + println!("{line}"); + } + println!("sorted best-first by mean dist to the true front (lower is better)"); + println!(); + + type Runner = fn(ManySpec, u64) -> MoRun; + let runners: &[(&str, Runner)] = &[ + ("RandomSearch", many_random), + ("NSGA-II", many_nsga2), + ("NSGA-III", many_nsga3), + ("MOEA/D", many_moead), + ("RVEA", many_rvea), + ("GrEA", many_grea), + ("IBEA", many_ibea), + ("HypE", many_hype), + ("AGE-MOEA", many_age_moea), + ]; + + let mut rows: Vec<(f64, Vec)> = Vec::new(); + for (name, runner) in runners { + let runs: Vec = (0..SEEDS).map(|s| runner(spec, s)).collect(); + let dist: Vec = runs.iter().map(|r| spec.mean_dist(&r.front)).collect(); + let fs: Vec = runs.iter().map(|r| r.front.len() as f64).collect(); + let ms: Vec = runs.iter().map(|r| r.wall_ms as f64).collect(); + let (d_m, d_s) = mean_std(&dist); + let (fs_m, _) = mean_std(&fs); + let (ms_m, _) = mean_std(&ms); + rows.push(( + d_m, + vec![ + name.to_string(), + format!("{d_m:.4}+/-{d_s:.4}"), + format!("{fs_m:.0}"), + format!("{ms_m:.0}"), + ], + )); + } + rows.sort_by(|a, b| a.0.partial_cmp(&b.0).unwrap_or(std::cmp::Ordering::Equal)); + let table: Vec> = rows.into_iter().map(|(_, r)| r).collect(); + print_table(&["algorithm", "mean dist", "front", "ms"], &table); +} + fn main() { run_zdt1_comparison(); run_zdt3_comparison(); @@ -2858,4 +3172,66 @@ fn main() { run_tsp_comparison(); run_jss_comparison(); run_knapsack_comparison(); + + // ---- Many-objective (4+) ---- + run_many_objective_comparison( + "DTLZ2 4-objective (dim=13, 40000 evals/run × 10 seeds)", + &[ + "DTLZ2 scaled to 4 objectives -- the entry point to many-objective.", + "Front is still the unit-hypersphere octant (Σf² = 1). Already hard:", + "with 4 objectives most random pairs of solutions are mutually", + "non-dominated, so Pareto rank alone barely discriminates. Optimum:", + "mean dist -> 0.", + ], + ManySpec { + objectives: 4, + dim: 13, + budget: 40_000, + population: 56, + reference_divisions: 5, + is_dtlz1: false, + hype_ref: 3.0, + }, + ); + run_many_objective_comparison( + "DTLZ2 10-objective (dim=19, 40000 evals/run × 10 seeds)", + &[ + "DTLZ2 scaled to 10 objectives -- the curse of dimensionality in full.", + "In 10-D objective space almost EVERY pair of solutions is mutually", + "non-dominated, so NSGA-II's whole population collapses into front 0", + "and crowding distance is the only signal left. Reference-point,", + "decomposition and indicator methods are built for exactly this.", + "Watch the 'front' column: it pins to the population size because", + "nothing dominates anything. Optimum: mean dist -> 0.", + ], + ManySpec { + objectives: 10, + dim: 19, + budget: 40_000, + population: 55, + reference_divisions: 2, + is_dtlz1: false, + hype_ref: 3.0, + }, + ); + run_many_objective_comparison( + "DTLZ1 8-objective (dim=12, 40000 evals/run × 10 seeds)", + &[ + "DTLZ1 scaled to 8 objectives -- the brutal one. Stacks the", + "many-objective dominance collapse on top of DTLZ1's deceptive", + "multimodal g-term (a huge number of local fronts). The true front", + "is the linear simplex Σf = 0.5; reaching it at all is the", + "achievement. Expect large mean-dist values and wide spreads -- this", + "is near the edge of what the catalogue does at this budget.", + ], + ManySpec { + objectives: 8, + dim: 12, + budget: 40_000, + population: 120, + reference_divisions: 3, + is_dtlz1: true, + hype_ref: 1.0, + }, + ); }