A 5-tonne skid-steer loader learns two-wheel acrobatics: reinforcement learning policies swing, balance and cross bumps on the real machine
Synopsis
A 5-tonne John Deere 332G skid-steer loader trained with reinforcement learning in simulation and transferred to the real machine can rock itself up onto its front wheels, balance on two wheels and track velocity commands. Against a model predictive control baseline, the two perform comparably at static balancing and velocity tracking, but the baseline loses balance beyond a target speed of 3.5 and cannot cross 75-millimetre undulations, while the learned policy can. The pipeline needs no prior machine model; an actuator network trained on measured data closes the sim-to-real gap.
Interpretation
A full-size 5-tonne John Deere 332G skid-steer loader can, without an operator, perform maneuvers previously achieved only by a small number of elite operators in official world-record performances: it accumulates momentum by rocking back and forth to swing up onto its two front wheels, then drives, steers and spins while balanced there. Earlier reinforcement learning and sim-to-real transfer on heavy machinery addressed statically stable tasks such as breaking rocks with an impact hammer, soil-adaptive excavation and grading with hydraulic excavator arms; highly dynamic maneuvers involving underactuated balancing and frequent contact changes were left to model-based control. On the real machine the swing-up controller reliably performed the maneuver for arm configurations seen in training; when the arm was higher than in training, the bucket touched the ground in 3 of 14 trials.
The control policies are trained entirely in simulation without expert demonstrations or a preexisting machine-specific model; the simulation combines a rigid-body model with an actuator network learned from real-machine data that reproduces hydraulic delays and nonlinearities. Hydraulic actuators involve fluids, flexible conduits and substantial delays, which are laborious to model from first principles and hard to model accurately enough, while model error directly undermines sim-to-real transfer. The actuator network predicts the next wheel speed from a history of control inputs and wheel speeds, trained on step, sinusoidal, ramp and constant-zero inputs with amplitudes up to 100% of the maximum command over a total of 1.5, later augmented with 20 of data collected during balancing.
While balancing, the learned policy tracks velocity commands up to its training maximum and crosses 75-millimetre undulations despite being trained only on flat terrain; the model predictive control baseline fell forward onto the bucket or backward onto the rear wheels for velocities beyond 3.5. Model predictive control had shown strong performance on hydraulic systems for trajectory tracking and vibration control, but contact changes and nonconvex optimization require task-specific extensions to the model. Under spin commands the learned policy stayed balanced at maximum rotational speed but oscillated more, while the baseline oscillated less at high spin rates; on undulating terrain the baseline stayed balanced but could not traverse the bumps even when the target velocity was increased.
Perspective
The demonstrated scope is one John Deere 332G skid-steer loader and one Super Mega Bot electric platform, with maneuvers composed from four separately trained sub-skills switched by hand-crafted conditions; the authors propose hierarchical reinforcement learning to replace the switching logic and list LiDAR or camera perception as a future direction.
The authors attribute the residual sim-to-real gap to effects including a 2-degree smaller average balancing pitch angle on the real machine and lower oscillation amplitudes under arm disturbance than in simulation; bucket ground contact when the arm was higher than in training is attributed to the increased inertia of the system.
