RLRT turns the actor–learner coupling into a tunable refresh bound: under 1% deadline misses on large policies across two Jetson boards
Synopsis
A team at Nanyang Technological University presents RLRT, an on-device reinforcement learning runtime that makes the coupling between a periodic actor and its learner an explicit, tunable refresh bound, coordinating actor, learner and a third-party tenant with a GPU permit and slack-aware admission atop the platform's real-time scheduler. On an Orin AGX and an Orin Nano under the real refresh cost, a M-parameter actor policy stays below 1% deadline misses, against 11.8% to 99.9% for a runtime-free actor, at lower learner throughput.
Interpretation
In on-device reinforcement learning a periodic actor must deliver actions by its control deadline while a learner updates the policy on the same embedded GPU, and RLRT turns their coupling into a runtime parameter: a refresh bound capping how many actor releases may pass between refresh attempts. Earlier designs fixed the coupling at one end or left it implicit: lockstep synchronises the actor with every published update, throttling the learner to the actor's rate, measured below what the device sustains, and missing essentially every deadline once the refresh or learner step no longer fits the actor's slack; free-running never refreshes and drifts thousands of updates behind. Run under the real refresh cost on an Orin AGX and an Orin Nano, a M-parameter replay-based actor-critic policy kept measured misses below 1% under RLRT, where a runtime-free actor missed 11.8% to 99.9% of deadlines, at lower learner throughput.
A GPU permit serialises work from the actor, learner and third-party tenant, a slack-aware admission rule fills the actor's spare time with learner steps, and updates too long to fit are split into minibatch gradient-accumulation passes and optimiser pieces. The permit grants one step at a time and each step synchronises its CUDA stream before releasing it, trading concurrency for predictability; admission gives ready real-time requests priority in base-policy order and otherwise starts a learner request only when its predicted cost plus a safety margin still falls before the next periodic release. On the AGX and the Nano, measured learner-step costs land near their predicted values; on the Nano's large network a millisecond-scale update progressed by splitting into eight sample passes of gradient accumulation and three optimiser pieces, where without splitting that configuration's learner throughput is markedly lower.
Perspective
The demonstrated scope is six reinforcement learning algorithms, two control environments, six base schedulers, a swept actor deadline and actor networks from k to M parameters on two Jetson boards (Orin AGX and Orin Nano), plus a feasibility demonstration on a stationary Duckiebot whose Jetson Nano 2GB runs the complete ROS2 visual pipeline and a 2.24M-parameter ACNet.
Admission is empirical rather than a guarantee; beyond the model-size range where runtime behaviour stays unchanged, the contended tail on the AGX stretches so far that no refresh bound recovers the deadline, and the learner state no longer fits in memory past the respective sizes on the Nano and the AGX. With double buffering the refresh bound mainly sets update lag, while on the copy path it mainly sets refresh frequency.
