Skip to main content
Back to timeline
arXivSource publication:

FP4 reinforcement learning stops quantizing each path alone: TRACE shares rounding across training and rollout for up to 5.4x faster generation

Synopsis

Reinforcement learning for mixture-of-experts language models can generate rollouts in FP4 to save compute and memory, but quantizing the training and rollout paths separately sends the same activation to different FP4 codewords, amplifying error into policy mismatch and unstable training. TRACE feeds the FP4 quantization outcomes actually produced during rollout back to the training side to choose how training activations are rounded, and caches only mantissa and scale information from deeper layers to keep transfer costs down. Across four Qwen mixture-of-experts models it supports FP4 weights, activations and KV cache together, matches BF16 rollout on average score, and speeds rollout by up to 5.4x.

Source-provided article image: TRACE: Rollout-Guided Quantization-Aware Training for FP4 Reinforcement Learning of MoE Language Models
arXiv

Interpretation

TRACE lets mixture-of-experts models run reinforcement learning training with weights, activations and KV cache all in FP4 while final task performance matches BF16 rollout. Earlier FP4 reinforcement learning methods reduced quantization error separately on the training and rollout paths rather than directly shrinking the gap between the two quantized execution paths. On Qwen3.5-35B-A3B reasoning tasks, TRACE averages 75.3 across four benchmarks against 74.9 for BF16 rollout, while QAT, QaRL and QUADS reach 59.6, 58.6 and 68.8.

During rollout generation it records the quantization outcomes of FP4 routed-expert activations and FP4 KV states, then uses those records to constrain rounding choices on the training side. The standard approach is round-to-nearest, which looks only at which FP4 codeword the training activation is closest to and ignores which codeword rollout actually produced. For each training activation, TRACE picks whichever of its two neighboring FP4 E2M1 codewords is closer to the rollout-side codeword, so under the same candidate set it does not increase the local train-rollout discrepancy relative to round-to-nearest.

To limit transfer overhead it keeps only mantissa and scale information from the later layers instead of complete quantized values. Keeping complete rollout-side quantization information would, by the paper's estimate for Qwen3.5-35B-A3B, produce about 51 TB of records per RL step, making serial storage I/O a bottleneck. On Qwen3.5-35B-A3B, retaining one mantissa bit plus amortized scale metadata for the last 20 layers caches about 7.5 KB per generated token, roughly 6x smaller than caching full quantized activations, and end-to-end RL step time rises from 664 to 713, a 7.4% overhead.

Perspective

The demonstrated scope is reinforcement-learning post-training of four Qwen mixture-of-experts language models, Qwen3.5-35B-A3B, Qwen3.5-122B-A10B, Qwen3.8-Flash-Next and Qwen3.8-2.4T-A95B, covering reasoning, coding and long-horizon tasks, with rollout quantizing only routed experts in MoE layers and the KV cache in softmax-attention layers while other modules stay in BF16.

The local guarantee only says the captured sites are no worse than round-to-nearest under the same candidate set; it does not guarantee a monotonic drop in full-network discrepancy or convergence of policy-level divergence. The training policy can be newer than the policy that generated the rollout, so TRACE does not remove activation differences from policy staleness, only the part amplified by inconsistent rounding. The caching scheme approximates full guidance using partial code information.