feat(algorithms): add Tpe (Tree-structured Parzen Estimator)

Bergstra et al. 2011: sample-efficient sequential optimizer that's the
workhorse of Hyperopt and Optuna. Different surrogate from BO's
Gaussian process — TPE models p(x | y < y*) with one KDE and
p(x | y >= y*) with another, then samples candidates from the 'good'
KDE and ranks by the ratio l(x) / g(x). The acquisition is implicit
in the ratio (a closed-form analog of Expected Improvement).

Implementation:
- 1-D Gaussian KDE per axis, with bandwidth chosen by Scott's rule
- Per-step:
  - Evaluate observations into 'good' (top γ fraction by target) and
    'bad'
  - Sample n_candidates from the good distribution (independent per
    axis) and pick the one with the largest l(x)/g(x)
  - Evaluate it, append to history

Vec<f64> only, single-objective only. Compared with BayesianOpt:
- Cheaper per-step (no GP factorization)
- Doesn't need kernel hyperparameter tuning to work well
- Naturally extends to mixed/categorical decision types (future work)
- Generally less sample-efficient than well-tuned BO on smooth
  continuous problems, but more robust out of the box

Tests cover convergence on 1-D Sphere within a tight budget,
deterministic reruns, panic on multi-objective.
This commit is contained in:
2026-05-05 09:55:01 -06:00
parent e7355ebb8a
commit 358e441b36
3 changed files with 358 additions and 1 deletions
+1 -1
View File
@@ -32,6 +32,6 @@ pub use crate::algorithms::{
ParticleSwarmConfig, RandomSearch, RandomSearchConfig, Rvea, RveaConfig,
SeparableNes, SeparableNesConfig, SimulatedAnnealing,
SimulatedAnnealingConfig, SmsEmoa, SmsEmoaConfig, Spea2, Spea2Config, TabuSearch,
TabuSearchConfig, Tlbo, TlboConfig, Umda,
TabuSearchConfig, Tlbo, TlboConfig, Tpe, TpeConfig, Umda,
UmdaConfig,
};