Skip to main content
Back to timeline
arXivSource publication:

Control policies stop being only weight updates: a fixed table of high-scoring code lifts held-out episodes in four tasks

Synopsis

EliteTable writes control policies as executable Python programs and uses a fixed-capacity table of high-scoring programs to guide fresh proposals, edits, and remixes, running ten searches on each of six Gymnasium control tasks. New-candidate score medians rose in all six tasks and selected-winner search-score medians rose in five; on MountainCar-v0, LunarLander-v3, BipedalWalker-v3, and CarRacing-v3, all ten searches improved from first to final selected winner on the same 100 held-out episodes. Six of ten Acrobot-v1 searches declined on held-out episodes with a negative median change, so rising search scores do not imply better new episodes.

AI-generated editorial illustration: EliteTable: Self-Improving Program Search for Executable Control Policies

Interpretation

Control-policy improvement can happen at the source-code level: the search repeatedly rewrites executable programs instead of only updating neural-network weights. Earlier program-search systems retained different solution sets and generated candidates differently, for example maintaining multiple island populations or evolving heuristic descriptions together with their implementations. Six Gymnasium control tasks each ran ten independent searches of ten generations with fifty candidate slots per generation; between the first and final generations, new-candidate score medians rose in all six tasks and selected-winner search-score medians rose in five, with CartPole-v1 winners already at the score ceiling in generation one.

The search keeps a fixed-capacity table of high-scoring programs, and new candidates compete against retained solutions, with three operators producing candidates. Generation one contains only fresh proposals; later generations allocate twenty percent fresh proposals, forty percent edits, and forty percent remixes, where an edit receives one parent's source and a remix combines information from up to three parents. Candidates pass interface, syntax, and duplicate checks, and failures enter a repair loop capped at five repairs per slot; 29,959 of 30,000 candidate slots reached evaluated status, or 99.9 percent, and the logs contain 38,289 model calls, of which 8,287 are repair calls.

Held-out episode performance does not always track search scores, and rising search scores can accompany regression on new episodes. Winners are selected by search score alone, and held-out episodes do not influence generation, repair, promotion, stopping, or winner selection. Each distinct generation winner was evaluated once per held-out seed across 100 seeds; all ten paired endpoints improved in MountainCar-v0, LunarLander-v3, BipedalWalker-v3, and CarRacing-v3, six of ten Acrobot-v1 searches declined with a negative median paired change, and BipedalWalker-v3 and CarRacing-v3 improved without a final winner reaching the reporting target.

Perspective

The report covers six Gymnasium control tasks, each with ten independent searches of ten generations and fifty candidate slots per generation, with selected winners evaluated on 100 held-out episode seeds. It suits researchers and engineers working on program search, language-model-guided code improvement, and reinforcement-learning control. The author proposes next steps: budget-matched comparisons with other algorithms, complete ablations including a history-independent generation baseline, and tests across language models, longer budgets, and new episode panels.

Rising search scores do not guarantee held-out improvement: Acrobot-v1 search scores rose while the median paired held-out change was negative, and the author says the cause needs further examination. Held-out evaluation covers each generation's selected winner rather than every retained elite, and search used ten episode seeds against 100 held-out seeds. The author notes that ten searches per environment characterize variation under fixed episode panels but do not establish robustness across episode distributions, language models, prompts, or search configurations; the tasks are well known, so the model may have encountered descriptions or solutions during training; and time-based deadlines and hosted-provider variation can change search trajectories.