EgoCaptioner turns first-person video into timed steps, not one-line summaries, reaching 75.51 PSJ on EgoHome step alignment
Synopsis
First-person video captioning has forced a trade-off between detail and timing: general captioners cover the scene without binding the narrative to the wearer's actions, while dense paragraphs leave unclear when each action begins. EgoCaptioner splits a caption into a video-level overview and timed steps, builds 50,000 training examples with a coarse-to-fine annotation pipeline that fixes event structure before expanding step descriptions, and refines the model with rewards for factual content, temporal alignment and style. Across five egocentric benchmarks it leads the evaluated open-source models in caption quality, caption-conditioned QA and temporal alignment, scoring 75.51 PSJ on EgoHome step segmentation and 78.84 Overall on PhysCapBench, above Gemini 3.1 Pro.
Interpretation
A first-person video caption now carries both an overview of the whole activity and ordered steps with start and end times, with key moments marked at specific timestamps. General captioners describe the visible scene broadly without binding the narrative to the wearer's actions, and existing egocentric captioners mostly give brief narration or specialize in hand-object manipulation, leaving navigation and social interaction less fully described. Across five egocentric benchmarks, EgoCaptioner leads the evaluated open-source models in caption quality, caption-conditioned QA and temporal alignment, surpasses all evaluated proprietary models on Minerva-Ego, and beats Gemini 3.1 Pro on PhysCapBench.
Training data is produced by two multimodal large language models in sequence: Gemini writes the overview, Seed2.1 pro cuts events with their boundaries, types and step outlines, and Gemini then expands each outline into timestamped steps. Single-pass annotation segments and describes each step in one reading, giving coarser boundaries and fewer step-level facts; the two-stage pipeline fixes event structure first, then describes each step from its event crop with up to 1 s of context on each side. With the same source videos, training volume, output format and hyperparameters, coarse-to-fine supervision improves every metric over single-pass annotation, raising EgoTempo accuracy from 26.60 to 36.40, Minerva-Ego from 7.14 to 18.57 and EgoHome PSJ from 62.89 to 71.63.
Reinforcement learning constrains generation with three rewards: a fact reward for faithful and complete description, a temporal reward for step boundary and granularity agreement, and a style reward that penalizes fragmented or redundant steps. Removing the fact reward drops all metrics below the supervised fine-tuning starting point, because the remaining rewards supervise timing and style but place no constraint on content; removing the temporal reward lowers PSJ below supervised fine-tuning while METEOR is unaffected. GRPO runs on 8,000 video-instruction pairs balanced across the three captioning tasks; the full objective reaches METEOR 20.40, SODA-d 56.38 and PSJ 75.51, while dropping the style reward gives METEOR 20.41, SODA-d 55.23 and PSJ 73.22, so METEOR is slightly higher but SODA-d and PSJ are lower.
Perspective
Evaluation covers five egocentric benchmarks, EgoMe, EgoHome, PhysCapBench, Minerva-Ego and EgoTempo, with Minerva-Ego and EgoTempo adapted to a caption-to-QA protocol in which the judge answers from the generated caption and the question alone; the model starts from Qwen3.5-9B, is fine-tuned with LoRA and then trained with GRPO, and inference samples frames uniformly at a target 4 fps capped at 240 frames.
In the caption-to-QA protocol Gemini 3.8 Flash answers using only the generated caption and the question, and the same model judges the answer against the reference, so results reflect how usable the caption is as a text interface; pose-derived motion facts enter only for videos with sensor recordings, as additional reference facts. The ablation shows that removing the style reward gives METEOR 20.41, slightly above the full objective's 20.40, while SODA-d falls from 56.38 to 55.23 and PSJ from 75.51 to 73.22.
