feat(algorithms): add BayesianOpt — GP-based Bayesian Optimization

The first sample-efficient algorithm in heuropt. Bayesian optimization
maintains a Gaussian-process surrogate of the objective and at each
step picks the next decision by maximizing an acquisition function on
that surrogate, so the evaluation budget is used surgically.

Implementation:
- **Kernel**: anisotropic RBF (squared-exponential) with per-axis
  length scales, signal variance, and a small noise/jitter floor.
  Hyperparameters are exposed in the config; a future version can add
  marginal-likelihood maximization.
- **Posterior**: standard formulation. Cholesky factorizes K (using
  the new internal helper); mean and variance predictions follow.
- **Acquisition**: Expected Improvement against the best observed
  feasible point. Optimized by best-of-N random sampling — simple,
  predictable cost, no inner-optimizer footgun.
- **Initial design**: `initial_samples` uniform-random points in
  bounds before the BO loop starts.
- **Constraints**: feasibility-aware EI — best observed value uses
  only feasible points; infeasible candidates are penalized.

Vec<f64> decisions, single-objective only. Targets the regime no
existing heuropt algorithm covers: 50–500 evaluations on an
expensive black-box function (CFD sim, ML training run, real-world
measurement).

Tests cover convergence on the 1-D sphere within a tight evaluation
budget (~30 evals get to f < 1e-6 — vs population-based methods
needing thousands), deterministic reruns, and panic on
multi-objective + dim mismatches.
This commit is contained in:
2026-05-05 09:51:12 -06:00
parent 284f1143de
commit a70500406c
4 changed files with 423 additions and 2 deletions
+5 -1
View File
@@ -15,9 +15,11 @@ pub(crate) fn cholesky(a: &[Vec<f64>]) -> Result<Vec<Vec<f64>>, &'static str> {
}
debug_assert!(a.iter().all(|row| row.len() == n));
let mut l = vec![vec![0.0_f64; n]; n];
#[allow(clippy::needless_range_loop)] // body indexes both `a` and `l` rows.
for i in 0..n {
for j in 0..=i {
let mut sum = a[i][j];
#[allow(clippy::needless_range_loop)]
for k in 0..j {
sum -= l[i][k] * l[j][k];
}
@@ -90,9 +92,11 @@ mod tests {
assert!(approx_eq(l[1][0], 1.0, 1e-12));
assert!(approx_eq(l[1][1], 2.0, 1e-12));
// L · L^T should reconstruct A.
#[allow(clippy::needless_range_loop)]
for i in 0..2 {
for j in 0..2 {
let mut s = 0.0;
#[allow(clippy::needless_range_loop)]
for k in 0..2 {
s += l[i][k] * l[j][k];
}
@@ -111,7 +115,7 @@ mod tests {
];
let l = cholesky(&a).unwrap();
// Choose a vector and check A · x = b round trip.
let x_truth = vec![1.0, -2.0, 0.5];
let x_truth = [1.0, -2.0, 0.5];
let b: Vec<f64> = (0..3)
.map(|i| (0..3).map(|j| a[i][j] * x_truth[j]).sum())
.collect();