Bergstra et al. 2011: sample-efficient sequential optimizer that's the
workhorse of Hyperopt and Optuna. Different surrogate from BO's
Gaussian process — TPE models p(x | y < y*) with one KDE and
p(x | y >= y*) with another, then samples candidates from the 'good'
KDE and ranks by the ratio l(x) / g(x). The acquisition is implicit
in the ratio (a closed-form analog of Expected Improvement).
Implementation:
- 1-D Gaussian KDE per axis, with bandwidth chosen by Scott's rule
- Per-step:
- Evaluate observations into 'good' (top γ fraction by target) and
'bad'
- Sample n_candidates from the good distribution (independent per
axis) and pick the one with the largest l(x)/g(x)
- Evaluate it, append to history
Vec<f64> only, single-objective only. Compared with BayesianOpt:
- Cheaper per-step (no GP factorization)
- Doesn't need kernel hyperparameter tuning to work well
- Naturally extends to mixed/categorical decision types (future work)
- Generally less sample-efficient than well-tuned BO on smooth
continuous problems, but more robust out of the box
Tests cover convergence on 1-D Sphere within a tight budget,
deterministic reruns, panic on multi-objective.
swaits
358e441b36