swaits/heuropt
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

refactor(compare): realign + sort tables, add combinatorial problems

Browse Source 
The terminal output was misaligned: headers and data were right-aligned
with hardcoded column widths and separator lengths, and the mean ± std
cells contained the non-ASCII `±` (plus `ε`, `↑`, `↓`) -- on any terminal
that renders those at a non-1 column width the columns drift, and the
hardcoded `-`.repeat(n) separators didn't match the real table width
anyway.

Changes:
- New `print_table` helper: column widths derived from the actual cell
  contents (header + every row), separator length computed to match.
- All table cells are now ASCII: `+/-` instead of `±`, `eps-MOEA`
  instead of `ε-MOEA`, arrows dropped from headers.
- Every table is sorted best-first by its primary quality metric.
- Added three combinatorial / sequencing problems with their own
  (permutation- / bitstring-native) algorithm rosters: a convex-position
  ring TSP (known optimum), FT06 job-shop makespan (known optimum 55),
  and a bi-objective 0/1 knapsack scored by hypervolume.
- Expanded every problem's preamble: what it is, why it's hard, and the
  best-known / optimal result.
- Regenerated examples/compare-results.md to match.

Continuous-problem quality metrics are unchanged (bit-identical to prior
snapshots); only ms columns and row order move.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
swaits
2026-05-14 07:51:12 -06:00
co-authored by Claude Opus 4.7
parent c50e390969
commit 2b453464e7
2 changed files with 1130 additions and 178 deletions
Download Patch File Download Diff File Expand all files Collapse all files
examples/compare-results.md
+167 -75
View File
@@ -1,142 +1,234 @@
# `compare` example — reference output # `compare` example — reference output
Snapshot from `cargo run --release --example compare` after the v0.4.0 Snapshot from `cargo run --release --example compare`, refreshed 2026-05-14
perf pass landed (2026-05-05). 10 seeds per algorithm per problem. for heuropt v0.10.0. 10 seeds per algorithm per problem.
The **quality metrics** (hypervolume / spacing / mean L2 / mean dist / Each table is **sorted best-first** by its primary quality metric. The
front size) are bit-identical to the v0.3.0 snapshot — the v0.4.0 live terminal output uses ASCII `+/-` for the mean ± std cells (so column
optimization work was strictly CPU-time, never algorithmic. The **ms alignment can't be broken by a terminal that renders `±` at an odd
columns** reflect the v0.4.0 numbers; total compare-harness wall-clock width); this doc uses `±` since markdown renders it fine.
dropped from ~18.6 s to ~5.7 s (3.27× faster).
Wall-clock numbers are from the development machine and will vary; The **continuous-problem quality metrics** are bit-identical to the
the *relative* numbers across algorithms are the interesting part. v0.3.0–v0.4.0 snapshots — every optimization pass so far (including the
Phase B CPU work) has been verified bit-identical by the `run()` snapshot
tests. The **ms columns** are the post-Phase-B numbers; SMS-EMOA on DTLZ2
in particular fell ~2.7× from the `hypervolume_nd` rework.
This refresh also adds three **combinatorial / sequencing** problems —
TSP, job-shop scheduling, and a bi-objective knapsack — which exercise the
permutation and bitstring operators and a different algorithm roster (the
real-vector methods can't run them).
Wall-clock numbers are from the development machine and will vary; the
*relative* numbers across algorithms are the interesting part.
--- ---
## ZDT1 (dim=30, 25000 evals/run × 10 seeds) ## ZDT1 (dim=30, 25000 evals/run × 10 seeds)
Two-objective benchmark with a smooth Pareto front along Zitzler-Deb-Thiele 2-objective benchmark: 30 real variables, one smooth
`f₂ = 1 − √f₁`. Hypervolume reference point: `[11, 11]`. convex Pareto front `f₂ = 1 − √f₁`. Hard because 29 of 30 variables must
collapse to 0 before the front is even reachable, and only then can the
population spread along it. Optimum: mean L2 → 0 (the front is known
exactly). Sorted by hypervolume (reference `[11, 11]`).
| algorithm | hypervolume ↑ | spacing ↓ | mean L2 ↓ | front | ms | | algorithm | hypervolume ↑ | spacing ↓ | mean L2 ↓ | front | ms |
|---|---|---|---|---|---| |---|---|---|---|---|---|
| RandomSearch | 99.5691 ± 0.94 | 0.0937 ± 0.03 | 2.3621 ± 0.14 | 28 | 94 | | MOPSO | **120.6149 ± 0.0529** | 0.0125 ± 0.0025 | **0.0005 ± 0.0001** | 100 | 80 |
| PAES | 104.1887 ± 0.90 | 0.0351 ± 0.01 | 1.3195 ± 0.06 | 33 | 30 | | IBEA | 120.0167 ± 0.3112 | 0.0130 ± 0.0027 | 0.0448 ± 0.0168 | 73 | 130 |
| MOPSO | **120.6149 ± 0.05** | 0.0125 ± 0.00 | **0.0005 ± 0.00** | 100 | 89 | | MOEA/D | 119.9450 ± 0.4953 | 0.0118 ± 0.0013 | 0.0065 ± 0.0020 | 96 | 27 |
| SPEA2 | 118.0823 ± 0.60 | 0.0111 ± 0.00 | 0.2408 ± 0.05 | 97 | 234 | | PESA-II | 119.3670 ± 0.3261 | **0.0095 ± 0.0011** | 0.0802 ± 0.0354 | 100 | 67 |
| PESA-II | 119.3670 ± 0.33 | **0.0095 ± 0.00** | 0.0802 ± 0.04 | 100 | 73 | | eps-MOEA | 118.8742 ± 0.6835 | 0.0167 ± 0.0058 | 0.0493 ± 0.0227 | 45 | 46 |
| ε-MOEA | 118.8742 ± 0.68 | 0.0167 ± 0.01 | 0.0493 ± 0.02 | 45 | 50 | | NSGA-II | 118.3336 ± 0.7750 | 0.0112 ± 0.0022 | 0.1891 ± 0.0599 | 96 | 40 |
| IBEA | 120.0167 ± 0.31 | 0.0130 ± 0.00 | 0.0448 ± 0.02 | 73 | 138 | | SPEA2 | 118.0823 ± 0.5973 | 0.0111 ± 0.0023 | 0.2408 ± 0.0509 | 97 | 226 |
| HypE | 105.6489 ± 0.98 | 0.0266 ± 0.01 | 1.4820 ± 0.10 | 72 | 38 | | NSGA-III | 115.1612 ± 0.4745 | 0.0139 ± 0.0029 | 0.4314 ± 0.0582 | 86 | 47 |
| SMS-EMOA | 102.8871 ± 1.05 | 0.0263 ± 0.00 | 1.4937 ± 0.12 | 40 | 67 | | RVEA | 111.7151 ± 1.8195 | 0.0308 ± 0.0099 | 0.8399 ± 0.1569 | 47 | 62 |
| RVEA | 111.7151 ± 1.82 | 0.0308 ± 0.01 | 0.8399 ± 0.16 | 47 | 65 | | HypE | 105.6489 ± 0.9789 | 0.0266 ± 0.0053 | 1.4820 ± 0.1003 | 72 | 30 |
| NSGA-II | 118.3336 ± 0.78 | 0.0112 ± 0.00 | 0.1891 ± 0.06 | 96 | 67 | | PAES | 104.1887 ± 0.8953 | 0.0351 ± 0.0067 | 1.3195 ± 0.0558 | 33 | 27 |
| NSGA-III | 115.1612 ± 0.47 | 0.0139 ± 0.00 | 0.4314 ± 0.06 | 86 | 70 | | SMS-EMOA | 102.8871 ± 1.0543 | 0.0263 ± 0.0039 | 1.4937 ± 0.1192 | 40 | 54 |
| MOEA/D | 119.9450 ± 0.50 | 0.0118 ± 0.00 | 0.0065 ± 0.00 | 96 | 28 | | RandomSearch | 99.5691 ± 0.9383 | 0.0937 ± 0.0347 | 2.3621 ± 0.1428 | 28 | 88 |
**MOPSO and MOEA/D dominate** convergence (mean L2 to true front ≤ 0.01). **MOPSO and MOEA/D dominate** convergence (mean L2 to true front ≤ 0.01).
PESA-II edges spacing. PESA-II edges spacing.
## ZDT3 (dim=30, 25000 evals × 10 seeds) ## ZDT3 (dim=30, 25000 evals × 10 seeds)
Disconnected Pareto front; tests an algorithm's ability to maintain Zitzler-Deb-Thiele 2-objective with a **disconnected** front: five
spread across gaps. separate arcs rather than one curve. Hard because an algorithm has to
discover and populate every arc while not stranding solutions in the
dominated gaps between them.
| algorithm | hypervolume ↑ | spacing ↓ | front | ms | | algorithm | hypervolume ↑ | spacing ↓ | front | ms |
|---|---|---|---|---| |---|---|---|---|---|
| NSGA-II | 123.1826 ± 1.58 | 0.0092 ± 0.00 | 98 | 68 | | **IBEA** | **126.2072 ± 1.2280** | 0.0164 ± 0.0036 | 48 | 131 |
| MOEA/D | 125.2413 ± 2.16 | 0.0198 ± 0.00 | 92 | 28 | | MOEA/D | 125.2413 ± 2.1647 | 0.0198 ± 0.0043 | 92 | 27 |
| **IBEA** | **126.2072 ± 1.23** | 0.0164 ± 0.00 | 48 | 135 | | NSGA-II | 123.1826 ± 1.5829 | **0.0092 ± 0.0020** | 98 | 40 |
| AGE-MOEA | 119.5132 ± 1.27 | 0.0136 ± 0.00 | 90 | 199 | | AGE-MOEA | 119.5132 ± 1.2732 | 0.0136 ± 0.0023 | 90 | 171 |
## DTLZ2 (3-obj, dim=12, 30000 evals × 10 seeds) ## DTLZ2 (3-obj, dim=12, 30000 evals/run × 10 seeds)
Spherical Pareto front. Mean dist = `|‖f‖ − 1|`. Deb-Thiele-Laumanns-Zitzler 3-objective; the Pareto front is the
unit-sphere octant (`Σf² = 1, all f ≥ 0`) — a curved 2-D surface embedded
in 3-D objective space. `mean dist = |‖f‖ − 1|`, so 0 means perfectly on
the sphere (the known optimum).
| algorithm | mean dist ↓ | spacing ↓ | front | ms | | algorithm | mean dist ↓ | spacing ↓ | front | ms |
|---|---|---|---|---| |---|---|---|---|---|
| RandomSearch | 0.3949 ± 0.02 | 0.0797 ± 0.01 | 239 | 520 | | **IBEA** | **0.0014 ± 0.0002** | 0.0607 ± 0.0047 | 87 | 148 |
| MOPSO | 0.0566 ± 0.00 | 0.0687 ± 0.01 | 100 | 71 | | MOEA/D | 0.0037 ± 0.0003 | 0.0886 ± 0.0024 | 78 | 23 |
| NSGA-II | 0.0332 ± 0.01 | 0.0577 ± 0.01 | 92 | 104 | | HypE | 0.0113 ± 0.0033 | **0.0269 ± 0.0172** | 80 | 41 |
| SPEA2 | 0.0368 ± 0.00 | **0.0288 ± 0.00** | 92 | 534 | | NSGA-III | 0.0197 ± 0.0015 | 0.0735 ± 0.0052 | 92 | 91 |
| PESA-II | 0.0395 ± 0.00 | 0.0616 ± 0.01 | 100 | 396 | | eps-MOEA | 0.0325 ± 0.0104 | 0.0572 ± 0.0170 | 136 | 88 |
| ε-MOEA | 0.0325 ± 0.01 | 0.0572 ± 0.02 | 136 | 89 | | NSGA-II | 0.0332 ± 0.0068 | 0.0577 ± 0.0109 | 92 | 60 |
| **IBEA** | **0.0014 ± 0.00** | 0.0607 ± 0.00 | 87 | 156 | | SPEA2 | 0.0368 ± 0.0021 | 0.0288 ± 0.0038 | 92 | 530 |
| HypE | 0.0113 ± 0.00 | 0.0269 ± 0.02 | 80 | 53 | | PESA-II | 0.0395 ± 0.0033 | 0.0616 ± 0.0051 | 100 | 372 |
| SMS-EMOA | 0.0484 ± 0.01 | 0.0764 ± 0.01 | 40 | 1218 | | SMS-EMOA | 0.0484 ± 0.0134 | 0.0764 ± 0.0081 | 40 | 483 |
| RVEA | 0.0510 ± 0.00 | 0.0631 ± 0.00 | 68 | 73 | | RVEA | 0.0510 ± 0.0044 | 0.0631 ± 0.0024 | 68 | 66 |
| NSGA-III | 0.0197 ± 0.00 | 0.0735 ± 0.01 | 92 | 137 | | MOPSO | 0.0566 ± 0.0048 | 0.0687 ± 0.0084 | 100 | 66 |
| MOEA/D | 0.0037 ± 0.00 | 0.0886 ± 0.00 | 78 | 24 | | RandomSearch | 0.3949 ± 0.0152 | 0.0797 ± 0.0083 | 239 | 530 |
**IBEA wins decisively** (15× closer to the true front than NSGA-III). **IBEA wins decisively** (14× closer to the true front than NSGA-III).
SMS-EMOA's wall-clock fell ~2.7× from the v0.4.0 snapshot — the
`hypervolume_nd` rework.
## DTLZ1 (3-obj, dim=7, 30000 evals × 10 seeds) ## DTLZ1 (3-obj, dim=7, 30000 evals × 10 seeds)
Linear simplex Pareto front (`Σf = 0.5`). Deb-Thiele-Laumanns-Zitzler 3-objective; the Pareto front is the linear
simplex `Σf = 0.5` in the positive octant. Hard because a deceptive
multimodal `g` term riddles the approach with a huge number of local
fronts — only fully-converged runs land on the simplex.
| algorithm | mean dist ↓ | spacing ↓ | front | ms | | algorithm | mean dist ↓ | spacing ↓ | front | ms |
|---|---|---|---|---| |---|---|---|---|---|
| NSGA-III | 5.9130 ± 2.82 | 0.4375 ± 0.22 | 92 | 133 | | **GrEA** | **1.7725 ± 0.9897** | **0.0719 ± 0.0438** | 72 | 62 |
| MOEA/D | 2.8022 ± 1.78 | 0.2279 ± 0.22 | 78 | 21 | | MOEA/D | 2.8022 ± 1.7807 | 0.2279 ± 0.2247 | 78 | 22 |
| AGE-MOEA | 4.5395 ± 2.21 | 0.3930 ± 0.29 | 90 | 247 | | AGE-MOEA | 4.5395 ± 2.2114 | 0.3930 ± 0.2864 | 90 | 193 |
| **GrEA** | **1.7725 ± 0.99** | **0.0719 ± 0.04** | 72 | 104 | | NSGA-III | 5.9130 ± 2.8212 | 0.4375 ± 0.2212 | 92 | 81 |
**GrEA shines on linear fronts** — the grid-based niching matches the **GrEA shines on linear fronts** — the grid-based niching matches the
geometry better than reference points. geometry better than reference points.
## Rastrigin (dim=5, 50000 evals/run × 10 seeds) ## Rastrigin (dim=5, 50000 evals/run × 10 seeds)
Multimodal trap. Global minimum f = 0 at the origin. Highly multimodal trap: `f = 10n + Σ(xᵢ² − 10·cos(2π·xᵢ))`. Hard because a
near-quadratic global bowl is overlaid with ~10⁵ regularly spaced local
minima — any greedy step lands in the nearest dimple. Global optimum
`f = 0` at the origin.
| algorithm | best f | ms | | algorithm | best f | ms |
|---|---|---| |---|---|---|
| RandomSearch | 1.1064e1 ± 2.54 | 14 | | **(1+1)-ES** | **0.0000e0 ± 0.00e0** | 4 |
| HillClimber | 1.5966e1 ± 6.25 | 6 | | **DE** | **0.0000e0 ± 0.00e0** | 6 |
| **(1+1)-ES** | **0.0000e0 ± 0.00** | 4 | | GA | 7.0913e-8 ± 5.50e-8 | 15 |
| SimulatedAnneal | 3.8540e0 ± 1.48 | 7 | | NSGA-II | 4.9270e-5 ± 5.04e-5 | 60 |
| PAES | 1.5966e1 ± 6.25 | 10 | | IPOP-CMA-ES | 1.3423e-1 ± 2.71e-1 | 61 |
| GA | 7.0913e-8 ± 5.50e-8 | 16 |
| PSO | 7.9598e-1 ± 8.67e-1 | 5 | | PSO | 7.9598e-1 ± 8.67e-1 | 5 |
| NSGA-II | 4.9270e-5 ± 5.04e-5 | 83 | | CMA-ES | 2.3453e0 ± 1.49e0 | 10 |
| **DE** | **0.0000e0 ± 0.00** | 6 | | SimulatedAnneal | 3.8540e0 ± 1.48e0 | 7 |
| CMA-ES | 2.3453e0 ± 1.49 | 11 | | RandomSearch | 1.1064e1 ± 2.54e0 | 14 |
| **IPOP-CMA-ES** | 1.3423e-1 ± 2.71e-1 | 66 | | HillClimber | 1.5966e1 ± 6.25e0 | 6 |
| PAES | 1.5966e1 ± 6.25e0 | 10 |
(1+1)-ES and DE tie for f = 0. **IPOP-CMA-ES drops vanilla CMA-ES from (1+1)-ES and DE tie for `f = 0`. **IPOP-CMA-ES drops vanilla CMA-ES from
2.35 → 0.13** — the restart logic does what it should. 2.35 → 0.13** — the restart logic does what it should.
## Rosenbrock (dim=5, 30000 evals × 10 seeds) ## Rosenbrock (dim=5, 30000 evals × 10 seeds)
Smooth non-convex valley. Rosenbrock's banana valley: `f = Σ(100·(xᵢ₊₁ − xᵢ²)² + (1 − xᵢ)²)`. Hard
because the minimum sits in a long, bent, near-flat valley — easy to
enter, very slow to crawl along to the tip. Global optimum `f = 0` at the
all-ones point.
| algorithm | best f | ms | | algorithm | best f | ms |
|---|---|---| |---|---|---|
| **Nelder-Mead** | **0.0000e0 ± 0.00e0** | 1 |
| CMA-ES | 3.6207e-29 ± 2.35e-29 | 5 |
| TLBO | 1.8458e-3 ± 1.91e-3 | 1 |
| DE | 3.3345e-1 ± 3.01e-1 | 2 | | DE | 3.3345e-1 ± 3.01e-1 | 2 |
| PSO | 8.2124e-1 ± 1.58e0 | 2 | | PSO | 8.2124e-1 ± 1.58e0 | 2 |
| **CMA-ES** | **3.6207e-29 ± 2.35e-29** | 5 |
| TLBO | 1.8458e-3 ± 1.91e-3 | 1 |
| (1+1)-ES | 2.2115e0 ± 2.70e0 | 1 | | (1+1)-ES | 2.2115e0 ± 2.70e0 | 1 |
| **Nelder-Mead** | **0.0000e0 ± 0.00** | 1 | | BO (60 evals) | 3.1725e3 ± 2.92e3 | 39 |
| BO (60 evals) | 3.1725e3 ± 2.92e3 | 40 |
Nelder-Mead **= 0 exactly**, CMA-ES at machine epsilon. BO at only 60 Nelder-Mead **= 0 exactly**, CMA-ES at machine epsilon. BO at only 60
evaluations is honestly bad on 5-D Rosenbrock (no kernel evaluations is honestly bad on 5-D Rosenbrock (no kernel hyperparameter
hyperparameter tuning) — included as a reminder that BO needs more tuning) — included as a reminder that BO needs more evaluations than a
evaluations than a smooth problem actually requires for these other smooth problem actually requires for these other methods.
methods.
## Ackley (dim=5, 30000 evals × 10 seeds) ## Ackley (dim=5, 30000 evals × 10 seeds)
Smoother multimodal landscape than Rastrigin. Ackley's function: a near-flat outer plateau with shallow ripples
surrounding a single deep, narrow global basin. Hard because the gradient
is almost zero far from the optimum, giving local search little to
follow. Global optimum `f = 0` at the origin.
| algorithm | best f | ms | | algorithm | best f | ms |
|---|---|---| |---|---|---|
| DE | 4.4409e-16 ± 0.00 | 4 | | **DE** | **4.4409e-16 ± 0.00e0** | 3 |
| PSO | 1.5099e-15 ± 1.63e-15 | 3 | | PSO | 1.5099e-15 ± 1.63e-15 | 3 |
| CMA-ES | 1.5099e-15 ± 1.63e-15 | 6 | | CMA-ES | 1.5099e-15 ± 1.63e-15 | 5 |
| TLBO | 2.2204e-15 ± 1.78e-15 | 2 | | TLBO | 2.2204e-15 ± 1.78e-15 | 2 |
| BO (60 evals) | 1.9622e1 ± 1.23 | 40 | | BO (60 evals) | 1.9622e1 ± 1.23e0 | 38 |
All conventional methods reach machine precision. BO at 60 evals All conventional methods reach machine precision. BO at 60 evals
struggles — same caveat as Rosenbrock. struggles — same caveat as Rosenbrock.
---
## TSP ring-15 (8000 evals/run × 10 seeds)
15 equally-spaced cities on the unit circle; minimize the closed tour
length. The space is `(15−1)!/2` distinct tours, but cities in convex
position have no 2-opt local optima — so this instance cleanly separates
methods with good neighbourhood moves (inversion = 2-opt) from blind
recombination / sampling. Known optimum (the polygon perimeter):
**6.2374**.
| algorithm | tour length ↓ | ms |
|---|---|---|
| **HillClimber** | **6.2374 ± 0.0000** | 0 |
| **SimulatedAnneal** | **6.2374 ± 0.0000** | 0 |
| **TabuSearch** | **6.2374 ± 0.0000** | 0 |
| **AntColony** | **6.2374 ± 0.0000** | 8 |
| GA | 7.0133 ± 0.6725 | 2 |
| RandomSearch | 12.1474 ± 0.6797 | 1 |
Every local-search method (and Ant Colony) hits the exact optimum — as
theory predicts for convex-position TSP under 2-opt. The GA's order
crossover drifts off the optimum, and random sampling is hopeless.
## JSS FT06 (8000 evals/run × 10 seeds)
Fisher & Thompson 1963 6-job × 6-machine job-shop; minimize makespan.
Hard because every job has a fixed machine order, so swapping two
operations can ripple delays across the whole schedule. Known optimum:
**55**.
| algorithm | makespan ↓ | ms |
|---|---|---|
| **SimulatedAnneal** | **55.2000 ± 0.6000** | 1 |
| TabuSearch | 55.9000 ± 1.4457 | 1 |
| GA | 56.0000 ± 1.5492 | 4 |
| RandomSearch | 58.5000 ± 1.2042 | 4 |
| HillClimber | 62.5000 ± 4.3186 | 0 |
Simulated annealing gets within 0.4% of the known optimum on average;
greedy hill-climbing stalls in operation-order local optima.
## Knapsack (30 items, bi-objective, 20000 evals/run × 10 seeds)
Zitzler-Thiele style 0/1 knapsack: two profit vectors, one capacity (half
the total weight). Hard because the two profit objectives conflict and
the capacity constraint carves feasible regions out of the `2³⁰`
bitstrings. No closed-form optimum; scored by hypervolume vs reference
`[0, 0]` (higher is better).
| algorithm | hypervolume ↑ | front | ms |
|---|---|---|---|
| **NSGA-II** | **1360468.3 ± 11619.6** | 100 | 39 |
| SPEA2 | 1355615.5 ± 9266.8 | 100 | 213 |
| IBEA | 1352595.5 ± 10183.6 | 99 | 101 |
| NSGA-III | 1346446.0 ± 6922.1 | 100 | 39 |
| RandomSearch | 1118233.1 ± 34150.3 | 9 | 17 |
The three Pareto EAs land within ~1% of each other; random search finds a
front of only ~9 points and trails badly.
examples/compare.rs
+935 -75
View File
File diff suppressed because it is too large Load Diff