Sampling-based MPC stops resampling every step: PPO-MPPI reuses old trajectories, matching DIAL-MPC on a quadruped and quadrotor at about a quarter of its per-update time
Synopsis
Sampling-based model predictive control (MPC) picks control inputs by repeatedly forward-simulating candidate trajectories, which is expensive. PPO-MPPI and SPO-MPPI bring proximal policy optimization from reinforcement learning into this framework, taking several updates from one batch of already-evaluated trajectories without extra simulation. On Unitree Go2 quadruped velocity tracking at 1.5 m/s, the two reach average velocities of about 1.28–1.29 m/s versus 1.006 m/s for Vanilla-MPPI; on Skydio X2 quadrotor heart-shaped path tracking, SPO-MPPI's mean position error of 0.0854 m is slightly below DIAL-MPC's 0.0866 m, at roughly a quarter of DIAL-MPC's per-update computation time.
Interpretation
Each planning update in sampling-based MPC regenerates and forward-simulates a batch of candidate control sequences, and trajectory evaluation dominates computation with high-fidelity simulators. Conventional MPPI and the broader preconditioned gradient update must draw fresh trajectories after every proposal update, capping the optimization iterations possible within a fixed compute budget. Tracking a 1.5 m/s body-frame forward velocity on a Unitree Go2 quadruped, PPO-MPPI and SPO-MPPI reach average velocities of 1.2887 and 1.2802 m/s against 1.0056 m/s for Vanilla-MPPI, while their computation times of 0.0334 s stay close to Vanilla-MPPI's 0.0321 s.
The approach writes trajectory optimization as KL-regularized free-energy minimization over a distribution of open-loop control sequences, then improves that distribution several times from one batch of evaluated trajectories. It borrows the performance-difference identity from reinforcement learning, defining a signed exponential advantage that expresses free-energy decrease as an advantage-weighted expectation under the current sampling distribution. PPO-MPPI builds a clipped likelihood-ratio surrogate and SPO-MPPI instead applies a quadratic penalty on likelihood-ratio deviation; both give population-level monotonic improvement guarantees under fixed-covariance Gaussian sampling, and PPO-MPPI is proved to converge to first-order stationarity.
On quadrotor path tracking, the two trajectory-reuse methods cut per-update MPC computation to about a quarter of DIAL-MPC's. DIAL-MPC runs five stages of trajectory-level annealing, whereas PPO-MPPI and SPO-MPPI spend the sampling budget on multiple optimization updates over the same trajectories. Tracking a heart-shaped reference of given width, constant altitude and period with a Skydio X2, PPO-MPPI shows a mean position error of 0.1012 m and SPO-MPPI 0.0854 m, against 0.0866 m for DIAL-MPC; the two take about 0.0217–0.0218 s per update versus 0.0811 s for DIAL-MPC.
Perspective
The demonstrated scope is two simulated tasks: a Unitree Go2 quadruped tracking a 1.5 m/s body-frame forward velocity, and a Skydio X2 tracking a heart-shaped reference path, with all trajectories simulated in parallel in MJX on a single GPU and 1,024 trajectories evaluated per sampling stage. The theoretical results are population-level, and the convergence-to-first-order-stationarity proof additionally requires fixed-covariance Gaussian sampling distributions.
The monotonic-improvement guarantee is population-level and does not cover Monte Carlo estimation error at finite samples; the convergence result relies on fixed-covariance Gaussian sampling and bounded cost. Clipping zeroes the gradient once the likelihood ratio passes its threshold, which can permit large distribution shifts, and SPO-MPPI's quadratic penalty is introduced to counter that. Experiments report simulation results only.
