Skip to main content
Back to timeline
arXivSource publication:

Self-driving labs drop the Q-learning rate: heteroscedastic Gaussian processes set a noise level per target and near the best return in fewer episodes across five benchmarks

Synopsis

Each campaign in a self-driving laboratory is a real reactor or culture run, and tens to hundreds of episodes leave no room to tune a learning rate as well. Heteroscedastic Gaussian Process Q-learning (HGPQ) places a Gaussian process directly on the Q-function, uses the Kalman gain as the learning rate, and estimates each Bellman target's noise variance with a second heteroscedastic Gaussian process so noisy targets move the estimate less. Pooled over five simulated benchmarks that include a chemical reactor and a bioprocess, HGPQ approaches the best return found by any method in fewer episodes than GP, kernel and deep RL baselines.

Source-provided article image: Heteroscedastic Gaussian Process Q-learning for Sample Efficient Experimentation
arXiv

Interpretation

Q-learning no longer needs a per-problem learning rate: the size of each update comes from the Gaussian process posterior, and every Bellman target is weighted by how reliable it is. Earlier GP Q-learning either tuned the rate itself or replaced it with a single noise level, while laboratory noise varies with instrument, material batch and operating condition and is rarely known in advance. Pooled over five simulated benchmarks, HGPQ reaches 69% of the thresholds within 150 episodes and 77% within 300, more than every baseline after its full budget of 750 or 2000 episodes except Ensemble-DQN; GPQ-SLR and GPQ-CLR reach about half the thresholds by the end of their 300-episode budget.

The method treats each Bellman target as a noisy measurement of the Q-function, takes the learning rate to be the Kalman gain, and the whole update is equivalent to one Gaussian process regression. The Kalman gain is near one for an uncertain estimate and a precise target and near zero for a precise estimate or a noisy target, and it spreads the update to nearby state-input pairs through the posterior covariance without a decay schedule. Target variances come from a second heteroscedastic Gaussian process fitted to leave-one-out residuals; on the semi-batch reactor, HGPQ's final return is 0.059 against 0.007 for a tuned constant variance.

Per-target variances are not decoration: randomly permuting them among targets makes the method markedly worse. The permuted variances keep their values but no longer correspond to their own targets. The final cost is multiplied by 6.5 on the pendulum and by 3.2 on target tracking, over twenty paired seeds.

Perspective

The conclusions rest on five simulated benchmarks and two synthetic MDPs, covering a semi-batch reactor, photosynthetic biomass production, an inverted pendulum, target tracking and a non-stationary linear MDP, with budgets of 300 episodes for HGPQ and 750 to 2000 for the kernel and deep RL baselines. The theoretical guarantee covers a fixed kernel and per-target variances assigned when a target is collected, not the re-estimated variances and refitted kernel of the algorithm. The benchmarks penalise state-constraint violations rather than prevent them, and future work plans to handle constraints with feasibility backups.

The bound assumes a fixed kernel and variances set when a target is collected, whereas the algorithm refits the kernel and re-estimates variances each episode; the burn-in in the bound grows at least exponentially with dimension. On the non-stationary linear MDP, whose value is linear, KOVI, KTD and KFQL do about as well as HGPQ. Sparse HGPQ needs 35 episodes on target tracking at ten times the process noise, where SAC needs 31. On the reactor the best final returns still come from deep RL after 2000 episodes.