Skip to main content
Back to timeline
arXivSource publication:

FOCUS calls the video model only when the scene changes: 25.4 calls match 55 on EgoSchema

Synopsis

Streaming video understanding does not require a vision-language model call on every frame. FOCUS measures patch-embedding dissimilarity with a smaller model from the same family, triggers a burst of inference only when the scene changes meaningfully, and feeds decoder attention back as the spatial prior for the next burst. On the 500-video EgoSchema subset it reaches 74.8% accuracy with 25.4 calls per clip, against 74.6% with 55 calls for uniform sampling.

Source-provided article image: FOCUS: Closed-Loop Attention Feedback for Efficient Vision-Language Understanding
arXiv

Interpretation

The inference cost of video vision-language models shifts from a call on every frame to a call only when the scene changes in a semantically meaningful way, with each call steered toward task-relevant regions of the image. Earlier approaches either sub-sampled frames at a fixed interval, ignoring content and risking missed transient events, or pruned visual tokens inside the model to cut per-call cost while still invoking it on every sampled frame. On the 500-video EgoSchema subset in streaming mode, with every method sharing the Qwen3-VL-8B backbone at a matched cadence, FOCUS reaches 74.8% accuracy with 25.4 calls per clip, uniform sampling reaches 74.6% with 55 calls, and BOLT reaches 74.2% with 55 calls.

The system works in three layers: decide whether a frame should be processed at all, decide where in the frame the model should look, and use the model's own attention to correct where it looks next. Gating computes cosine dissimilarity between consecutive frames in the patch-embedding space of Qwen2-VL-2B and opens a burst of consecutive frames only when the threshold is exceeded, so the decision rests on the semantic features the model uses rather than on pixels. Per-patch dissimilarity is max-pooled over each burst and fused with the attention prior fed back from the previous burst into a spatial focus map, which scales the values of vision tokens in the decoder value cache without removing any token; on a 25-video egocentric captioning benchmark, FOCUS with feedback issues 222 calls against 516 for uniform sampling.

Perspective

The demonstrated scope is streaming egocentric video: the 500 three-minute EgoSchema clips for question answering and 25 videos from Ego4D, EPIC-Kitchens, Ego-Exo4D and GR-1 teleoperation recordings plus one in-house recording for captioning, with every method sharing the Qwen3-VL-8B backbone and the same cadence. The authors explicitly do not recommend it as the sole perception path where a missed event carries safety consequences.

The authors state that the EgoSchema accuracy differences fall within one to three questions on a 500-item benchmark, which five runs cannot resolve, so no accuracy win is claimed; the captioning study has only 25 videos and is model-judged, with gaps that do not reach significance under paired tests, so it serves as a per-criterion diagnostic. Gating is a decision not to look: an event whose inter-frame dissimilarity stays below the threshold never triggers a burst, and the timeout bounds how long the system goes unobserved without guaranteeing that any particular event is captured.