Skip to main content
Back to timeline
arXivSource publication:

A 100 ms VLA inference delay can nearly halve success on moving targets, and VLALAGBENCH turns latency into a predictable success curve

Synopsis

Researchers from the National University of Singapore and collaborators released VLALAGBENCH, which varies task type, target speed, execution mode, VLA policy and inference latency on MuJoCo LIBERO tasks to measure how latency affects robot manipulation success. At 0.1 m/s, a 100 ms synchronous inference delay cut set-down success from 71.5% to 40.5%, while grasping fell only from 86.5% to 74.0%. An empirical model fitted to these measurements predicts success at unseen latencies from limited calibration data, with 5.7 percentage points root-mean-square error for π0.5.

AI-generated editorial illustration: 1Introduction

Interpretation

Robot manipulation success drops markedly as VLA inference latency grows, and the size of the drop depends on the task and the target speed rather than on one uniform latency threshold. Common manipulation benchmarks such as LIBERO and RoboCasa use static environments, where longer latency lengthens completion time without changing the outcome; VLA-Perf treats 10 Hz and 100 Hz as acceptable and high-performance reference points without evaluating how those rates translate into task success. With π0.5 under synchronous execution at a target speed of 0.1 m/s, 100 ms of latency lowered set-down from 71.5% to 40.5% and grasping from 86.5% to 74.0%; on nearly static targets at 0.001 m/s, 200 ms of latency changed success by only a few percentage points.

Latency costs success through two paths: stale observations make the robot act on a position the target has already left, and pauses during synchronous inference disrupt tasks that need continuous motion. Set-down requires placing the bowl within 3 cm of the plate center, whereas release tolerates drops anywhere across a 12.2 cm basket opening; at 0.1 m/s, 200 ms of extra staleness lets the plate move 2 cm, two-thirds of set-down's tolerance. On release, 200 ms of synchronous latency reduced π0.5's average approach hand speed by 34%, so the arm fell behind the moving basket; asynchronous execution keeps the robot moving during inference and restores release and articulated-task success close to the zero-latency baseline at both 50 and 200 ms.

An empirical model taking observation staleness and idle time as inputs predicts success across latencies, execution modes and execution horizons from limited calibration data, and uses those predictions to pick a configuration for a given inference system. Exhaustively evaluating every combination of execution mode and horizon requires repeated task executions per candidate and is prohibitively expensive; the model instead reads a measured success curve at an adjusted target speed. Fitted on 0, 100 and 200 ms runs, the model predicted π0.5's held-out 50, 150 and 250 ms curves with 5.7 percentage points RMS error and MolmoAct2's with 7.8 points; across four deployment scenarios the selected configurations matched the measured best on all four, with prediction errors of 2.5% to 4.6% on Systems B, C and D.

Perspective

The findings rest on four tasks adapted from LIBERO in MuJoCo, with targets moving along fixed closed paths at 0.001 to 0.2 m/s, a 20 Hz control rate and execution horizon H = 5, using public checkpoints of π0.5, GR00T N1.7 and MolmoAct2; DOM tasks in Isaac Sim and SO-101 real-robot experiments serve as supplementary verification.

The model departs from measurements on π0.5's fastest targets: at 0.2 m/s grasping, synchronous execution beats both asynchronous modes at every latency from 100 to 250 ms, while the model predicts synchronous slightly below lazy execution; at 250 ms, lazy and continuous execution also lose success on nearly static targets. GR00T N1.7 is sensitive to initial target positions, and its fitted coefficients are negative for staleness on set-down and idle time on grasping, which limits how well speed scaling describes it.