Robot action-head weights rise in precision across flow steps: 2 to 4 bits keep closed-loop success, and inference runs 1.7 times faster than NVFP4 on Thor
Synopsis
A robot vision-language-action model's diffusion action head re-reads all its weights at every step but applies each to only about fifty tokens, so inference is bound by memory bandwidth. Because later flow steps attenuate the quantization errors of earlier ones, weight precision can rise across the steps; with plain per-channel rounding and no calibration or training, a [2,3,3,4] schedule preserves the closed-loop success of three policies while transferring 25% fewer bits than 4-bit weights at every step. Current GPU kernels cannot exploit this because of fixed overheads, so a programmable accelerator scales latency and energy with the transferred weight bits and runs the GR00T N1.6 action head in 8.02 ms per inference under LPDDR5X timing simulation.
Interpretation
An action head's weight precision can be scheduled to rise across flow steps instead of using one width at every step. Prior work quantized current policies to uniform 4-bit weights with the same width at every flow step; below 4 bits, uniformly rounded weights lose closed-loop success with and without calibration. Among 12 nominal bit-step schedules for GR00T N1.6, only the rising [2,3,3,4] preserves success on both screening benchmarks, while uniform 3-bit and the falling [4,3,3,2] lose success on both; [2,3,3,4] preserves success on five of eight full benchmark suites.
The accelerator brings per-inference latency and energy close to the floor set by transferring weight bits, because it keeps the DRAM link saturated at every width. On Jetson AGX Thor, going from [4,4,4,4] to [2,3,3,4] cuts weight bytes by 25% but time by only 9%, with attention, LayerNorm and activation rounding taking about 3.2 ms at every width. An RTL implementation simulated with LPDDR5X timing runs the GR00T N1.6 action head in 8.02 ms per inference with 2, 3, 3 and 4 bits in its four flow steps; with a copy per width and 50% of Thor's memory bandwidth it is 1.7 times faster than Thor in NVFP4 at 9.1 times less energy and equal closed-loop success.
The same bit-serial sub-arrays run action heads of different families and sizes as well as a vision-language model. The width changes per step and the attention's share varies with model and context, so a fixed partition leaves one side idle while the other lacks sub-arrays. A run-time allocator divides eight sub-arrays between the projections and the attention based on how full the weight FIFO is, making whole inferences of the three action heads 1.1% to 2.3% faster than static allocation; the same RTL runs GR00T N1.5 at [2,3,3,4] in 5.16 ms and another action head at [2,5] in 2.70 ms.
Perspective
All closed-loop results come from simulated environments, covering GR00T N1.6, GR00T N1.5 and another open policy on benchmarks including LIBERO, RoboCasa and SimplerEnv; the accelerator's latency, area and energy come from RTL and gate-level simulation synthesized in TSMC 7 nm, with DRAM timing from an LPDDR5X-8533 model.
Quantized policies that hold success in simulation have lost success on real robots; Quantization-in-DRAM needs a DRAM mode that no commercial part offers; chip energy comes from netlists with a clock tree but before placement and routing, and DRAM energy comes from a model.
