Streaming video agents learn when to stay silent: BLIP-Live's PAUC-F curbs repeat firing, and its Qwen model averages 49.08 on ProactiveVideoQA, 44% above the prior best
Synopsis
BLIP-Live reframes proactive streaming video as a causal decision to speak when evidence suffices and stay silent otherwise, and aligns data, metric, inference harness and training around that goal. It builds BLIP-Live-780K, a dataset of 784,944 human-verified responses annotated for both content and timing intervals, and proposes PAUC-F, a metric that balances response timeliness with firing precision. The trained BLIP-Live-Qwen model reaches state-of-the-art results on four proactive streaming benchmarks, averaging 49.08 on ProactiveVideoQA against 34.01 for the prior best MMDuet2, a 44% gain.
Interpretation
A streaming video agent is no longer judged only on answering correctly; at every second it must decide whether to speak or stay silent, and repeated or premature firing carries a cost. The earlier PAUC metric counted only recall inside the gold time window, so repeated in-window responses did not affect the score and out-of-window fires were discarded; ESTP-F1 added precision on out-of-window fires but ignored in-window repetition and needed a task-dependent timing term for recall. In a simulated excessive-firing comparison, one policy plus 1, 5, 10 and 100 out-of-window fires kept PAUC fixed at 0.2575, while PAUC-F fell from 0.4095 to 0.3399, 0.2024, 0.1344 and 0.0191.
PAUC-F takes the harmonic mean of the existing quality-weighted timeliness and a firing-precision term, so trading extra speech for recall no longer pays. The precision denominator counts every emitted response, including out-of-window fires; each gold turn contributes at most one in-window fire to the numerator, and additional fires only enlarge the denominator. After tuning two inference-harness variants separately toward PAUC and PAUC-F, 10 raters preferred the PAUC-F variant on 94% of 20 samples from the WEB split and 97% of 20 samples from the EGO split of ProactiveVideoQA.
One streaming interface serves both the inference harness for frozen models and supervised finetuning, with the model emitting one of three states per second: silence, response or hold. Earlier inference harnesses were mostly hand-crafted and served only to adapt offline models to streaming settings, without being optimized for proactivity itself. With the Qwen3.5-A3B backbone, training recipe and inference harness held fixed, training on BLIP-Live-780K raised mean PAUC-F across the four splits from 42.94 to 49.08.
Perspective
The demonstrated setting is streaming video question answering at a 1 Hz decision rate: harness tuning and calibration for frozen models, plus supervised finetuning on BLIP-Live-780K, evaluated on the four ProactiveVideoQA splits, ESTP-Bench, the PO split of StreamingBench and the CRR task of OVOBench. Training and inference ran on H200 nodes, and the authors state that data, code and model checkpoints will be released publicly.
The authors note that the human preference study tests only whether excessive firing is undesirable, not whether the metric ranks a broad set of model pairs consistently; convergence of the two backbone searches shows cross-backbone agreement, not stability across random validation subsets. The decision rate is fixed at 1 Hz to match prior work, and higher rates might improve temporal resolution but were not evaluated; longer visual windows did not improve the frozen-model harness, while finetuned models benefited from context up to 32 frames.
