Skip to main content
Back to timeline
arXivSource publication:

HubEgo-100K opens with 10,124 hours of first-person skilled work: 12,227 contributors across 15,548 environments, built for diversity

Synopsis

HubEgo-100K releases v1.0, 10,124 hours of 1080p egocentric video in which 12,227 contributors record unscripted skilled work in their own workplaces and homes, spanning 15,548 environments, 11 setting types, 30 work categories and 56 task types, with 83% of hours in workplace settings. The dataset is designed around published findings that imitation-learning generalization grows with environment and object diversity, and pairs each episode with synchronized roughly 100 Hz inertial data and machine-generated action and language annotations under CC BY 4.0. The authors state that this version does not yet train baseline policies, leaving the effect of this diversity on policy performance to later releases.

AI-generated editorial illustration: HubEgo-100K: In-the-Wild Egocentric Video of Skilled Work for Robot Learning

Interpretation

Robot imitation policies generalize poorly outside the environments and objects seen in training, and published data-scaling studies find that generalization grows with the diversity of training environments and objects rather than with more demonstrations per environment. Robot data is costly to collect across many environments: DROID gathered 350 hours across 564 scenes over 12 months with 50 data collectors, and existing egocentric video datasets mostly record daily life, kitchens or scripted lab tasks, with little skilled work in real workplaces. Among the datasets in Table 1 that report people counts, the others list 9 to 2,087 people while v1.0 has 12,227 contributors; the other datasets report 45 to 610 kitchens, locations, scenes or rooms, against 15,548 environments here.

Contributors record a named but unscripted task with a head-mounted phone during their own work, in their own workplace or home, keeping their own lighting, clutter, tools and materials. The pipeline first segments each recording in time with a vision-language model and keeps only segments where the contributor's hands perform the named task, then a panel of two vision-language models labels hand visibility frame by frame, drops recordings where hands are mostly out of frame, and cuts frames flagged for children, other people's faces or private text. By model estimate, at least one hand appears in 94.7% of sampled frames and both hands in 76.3%, with an on-task share of 96.9%; 83.0% of hours lie in episodes showing a hand in at least 90% of sampled frames.

The dataset reports a count and an evenness of hours for every diversity axis, and shows that new contributors and new words keep appearing when episodes inside v1.0 are taken in random order. Earlier datasets rarely report people counts, place counts and how evenly hours spread over those axes together, while this version reports each axis separately for contributors, environments, setting types, work categories, task types and action families. No contributor accounts for more than 0.22% of hours and the effective number of contributors is 5,666; the last 10% of hours adds 356 contributors, 570 environments and about 84 noun lemmas on average.

Perspective

The scope of this report and of the dataset description is v1.0: 10,124 hours, 72,754 episodes, 12,227 contributors and 15,548 environments, with 1080p 30 fps monocular video and accelerometer and gyroscope streams at a native median rate of 99.98 Hz. The data targets robot learning and perception research and is released under CC BY 4.0 on Hugging Face, with catalog and annotation tables readable without downloading video. The authors' stated next step is to train baselines on the data, including pretrained visual encoders and policies co-trained with robot data, evaluated on contributor-disjoint splits.

Annotations are machine-generated and have not been audited by human reviewers, and hand visibility and on-task share are model estimates; native inertial rates differ by operating system, about 100 Hz on iOS and a median of about 497 Hz on Android; camera calibration is not provided and the residual time offset between camera and inertial sensors has not been measured; task types were rebalanced, so their hour shares do not reflect how often the tasks occur in the world.