Skip to main content
Back to timeline
arXivSource publication:

STaSBO hands strategy, not points, to an LLM: best average rank of 2.17 and 2.60 across 22 benchmarks

Synopsis

Bayesian optimization depends heavily on how its surrogate model and acquisition function are paired, and a fixed pairing transfers poorly across problems. STaSBO gives that pairing decision to a large language model acting as an online meta-controller: it first plans the search intent, then jointly picks a configuration using surrogate fit evidence and prospective candidate diagnostics, while numerical Bayesian optimization still handles modeling and candidate generation. Across 12 synthetic mathematical tasks and 10 scientific and engineering tasks it reaches average ranks of 2.17 and 2.60, against 4.08 and 4.10 for the strongest baseline in each suite.

Source-provided article image: STaSBO: LLM-Guided Strategy Planning and Configuration Selection for Bayesian Optimization
arXiv

Interpretation

The language model no longer proposes the next evaluation point directly; it acts as a high-level scheduler for Bayesian optimization, deciding what behavior the search should pursue this round and which surrogate-acquisition pair should realize it. Earlier approaches either had the model propose candidate points from problem descriptions and observation history, or adapted a single component, such as evolving kernels under a fixed acquisition rule or selecting acquisitions under a fixed surrogate family. On 12 synthetic mathematical tasks and 10 scientific and engineering tasks with 50 objective evaluations per run, STaSBO's average task ranks are 2.17 and 2.60, against 4.08 and 4.10 for the strongest baseline in each suite.

The controller separates search intent from configuration choice, so the strategy is made explicit before a configuration is selected and can be retained or revised in later iterations. Whether a configuration suits depends on the problem and the current optimization state, and a configuration's edge on a source problem does not guarantee transfer to a target problem. Across six settings of Ackley, Levy and Rastrigin in 5D and 15D, the 30 fixed configurations each have a different best pairing, and the Ackley-5D oracle ranks only 26th on Ackley-15D.

A decision-aware memory links prior plans, selected configurations, candidate behavior and observed outcomes into one feedback chain, letting the controller distinguish whether a candidate matched its plan from whether the objective actually improved. Watching only whether the objective dropped cannot tell whether an exploratory move was executed as intended, which makes it hard to decide whether to continue or turn. In one Ackley-8D trace, the iteration-27 intermediate move matched its plan but did not improve the incumbent; at iteration 28 the controller switched to an isotropic Matérn surrogate with EI and cut incumbent regret from 12.35 to 11.24.

Configuration shortlisting first narrows the field with surrogate fit evidence, then compares the not-yet-evaluated candidate diagnostics to see what action each configuration would actually propose. Static configuration descriptions alone do not reveal where a configuration will query under the current observations, so numerical evidence has to enter the selection step. In an ablation on Ackley-8D over ten seeds, full STaSBO reaches a mean final simple regret of 8.25, while the reported variants range from 9.47 to 13.89.

Perspective

The demonstrated scope is 12 synthetic mathematical tasks and 10 scientific and engineering tasks under a shared budget of five initial observations plus 45 iterations, 50 objective evaluations in total, with all language-model methods using the same model and reasoning effort. The nine application objectives are deterministic functions interpolated from archived experimental measurements, and Lunar Lander comes from a simulator; neither involves executing real experiments.

The benefit of switching configurations depends on the problem, the switch time and the destination: on Levy-15D, 20 of 29 alternatives switched after 30 iterations beat the reference, whereas on Rastrigin-5D, 13 improve when switched after 15 iterations and only eight after 30, and both switch groups have higher branch-averaged final regret than the reference. The configuration-selection patterns describe aggregate preferences across tasks rather than a common schedule for every task.