Async RL post-training gets an offline scheduler: ASP cuts iteration time by up to 76.8% against MindSpeed-RL
Synopsis
Once LLM reinforcement-learning post-training splits generation from training across machines, multi-instance rollout still wastes time on uneven output lengths and on length-sorted dispatch that makes the training queue fill and then drain. ASP runs one offline preprocessing pass before training: it predicts each prompt's output length, uses dynamic programming to build decoding minibatches of similar length, then reorders them into batch groups with balanced predicted load. On Ascend 910B3, complete ASP reduces mean iteration time by up to 76.8% relative to MindSpeed-RL and by up to 62.5% relative to verl.
Interpretation
Multi-instance asynchronous RL carries a scheduling problem that was not previously handled on its own: each decoding minibatch should stay full, while the rate at which samples reach the trainer should stay steady. Separating generation from training and running several generation instances does not remove the early scaling plateau of single-instance inference; sorting prompts by predicted length reduces within-minibatch length spread but sends short minibatches first and long ones later, so the training queue first builds up and then drains. On 39,000 measured Qwen3-32B responses, sorting by predicted length raised batch-group load coefficient of variation from 0.103 to 0.403 and group-load max/min from 1.68 to 9.88; balanced ordering brought the same minibatches to 0.070 while the padding rate stayed at 0.224.
Scheduling becomes a single offline preprocessing pass that takes no online decisions during rollout. Length sorting alone addressed only local efficiency, and dynamic programming over minibatch boundaries barely changed the temporal imbalance. Dynamic programming moved mean padding rate from 0.226 to 0.224 while load CV stayed at 0.401; simulated-annealing reordering then cut CV to 0.070, whereas random shuffling reached only 0.162.
The complete system shortens iteration time and raises throughput in real training. Turning scheduling on and off inside the same ASP architecture isolates what scheduling itself contributes, beyond the gains of disaggregation. In complete Qwen3-30B-A3B training, enabling scheduling reduced mean iteration time from 238 to 212 s (10.9%), raised end-to-end throughput from 47 to 58 tokens/s/NPU (23.4%), and cut the first iteration from 1042 to 724 s.
Perspective
The demonstrated setting is a MindSpeed-RL implementation on Ascend 910B3 NPUs, with GRPO, end-to-end comparisons on Qwen3-30B-A3B and Qwen3-32B, scheduling ablations and predictor-sensitivity experiments on Qwen3-8B, and scalability testing up to 128 NPUs. The intended reader is a systems or algorithm engineer building asynchronous RL post-training pipelines.
The output-length predictor stays fixed throughout training, and mean response length drifted only 1.6% in the main run, so longer runs, stronger policy shifts or curriculum changes may call for renewed prediction. Convergence was checked only on Maze-Reasoning-GRPO-v0.1 over 30 GRPO iterations, at a measured policy lag of about 3.3 trainer steps. Direct comparisons with StreamRL, AsyncFlow, AReaL and Laminar were not conducted. The internal telecom deployment is reported only as qualitative evidence because its workflow and data are confidential.
