Skip to main content
Back to timeline
arXivSource publication:

Egocentric video captions learn which hand does what: BiScopeNet lifts NVILA action recognition from 14.1% to 45.5%

Synopsis

Egocentric action captions usually say only that someone is picking up a pan, without saying which hand holds the pan and which holds the lid. BiScopeNet leaves the vision encoder and language model untouched, adding a text-guided attention module called FocusAlign at the projection stage between them and supervising training with MetaLoss, which masks metadata spans and upweights object words, alongside the video-level BiScopeCap dataset of hand roles and spatial metadata. On the full BiScopeCap set, the NVILA backbone with BiScopeNet reaches 45.5% action recognition, up from 14.1%, right-hand role recognition 44.9% from 17.2%, left-hand 41.2% from 28.7%, and object recognition 58.9% from 39.7%.

Source-provided article image: Sub-actions in Action: Text-Guided Hand-Role Alignment for Sub-Action Recognition
arXiv

Interpretation

Egocentric video captions now state explicitly what each hand is doing, rather than giving only a single high-level action description. Earlier action recognition and captioning work stayed at the high-level action layer: EPIC-KITCHENS and Ego4D lack hand-role labels, HOI-QA has sparse hand annotations on selected frames only, and HD-Epic has richer captions but does not unify high-level actions with per-hand roles in video-level descriptions. On the full BiScopeCap set, the NVILA backbone with BiScopeNet reaches 45.5% action recognition, 44.9% right-hand role, 41.2% left-hand role and 58.9% object recognition, against 44.0%, 43.0%, 39.1% and 58.0% for the same NVILA backbone under standard supervised fine-tuning.

The change sits only in the projection stage between the vision encoder and the language model, and both backbones keep their vision encoder and language model frozen. The usual approach feeds visual features through a linear or MLP projector, or compresses visual tokens into a fixed set of latent queries with something like Q-Former. FocusAlign exchanges information between video and text tokens through attention, keeps the original visual representation through a residual connection with a learnable scalar gate, and realigns the fused features to the language model's embedding space with a feed-forward network; MetaLoss deterministically masks the < meta > spans from the annotation structure and upweights object words, with no auxiliary reference model.

The same modules still help on the weaker VideoChatGPT backbone and add almost no compute. On the full BiScopeCap set, the original VideoChatGPT architecture reaches 14.3% action recognition and 30.1% object recognition and produces 488 unstructured outputs. With BiScopeNet and the CLIP vision encoder, VideoChatGPT rises to 30.7% action recognition and 47.9% object recognition while unstructured outputs fall to 7; with a DINOv3 encoder it reaches 33.4% action recognition and 53.5% object recognition. FocusAlign adds fewer than 0.3B parameters and less than 1% additional TFLOPs over linear or MLP projectors.

Perspective

The demonstrated scope is trimmed clip-level egocentric sub-action recognition: the model samples a fixed number of frames per clip and assumes left- and right-hand roles are concurrent within that clip rather than performing temporal action localisation. The dataset is curated from EPIC-KITCHENS, EPIC-VISOR, Ego4D and HD-Epic, and the raw videos remain under their original licences.

Fixed frame sampling limits modelling of hand roles that evolve over long videos; repetitive low-amplitude hand motions such as successive up-and-down movements introduce ambiguity, and very short clips combined with high camera motion sharply degrade caption quality, a limitation seen across all evaluated vision-language models. On the data side, a manual audit of 200 randomly sampled training clips found minor descriptive inaccuracies and left- and right-hand hallucinations.