Five trackers and a pair of gloves capture whole-body manipulation: EgoExplorer releases 100 hours of vision-motion-touch data with 18.0 mm body-point error
Synopsis
The wearable capture system EgoExplorer aligns first-person video, whole-body motion, finger articulation and tactile records on one timeline, covering five environments including laboratories, factories and homes across 2,715 recordings and 100 hours. Earlier datasets gave only part of this picture: egocentric video leaves the body unobserved, wearable body tracking lacks finger and contact measurements, and hand-focused sets often omit whole-body motion. To recover whole-body motion from five sparse trackers, the pipeline first calibrates body dimensions and sensor mounts, then completes unobserved joints with a temporal prior and refines under layered constraints; against tracker-fitted reference labels, body-point error is 18.0 mm, a 71.9% reduction relative to the strongest baseline.
Interpretation
EgoExplorer aligns first-person video, reconstructed whole-body motion, glove-measured finger articulation and tactile observations within the same continuous recordings, together with step-level text descriptions covering approaching targets, adjusting posture, manipulating objects and relocating. Earlier public data covered only part of this: egocentric video preserves visual context but leaves much of the body unobserved, wearable body tracking often lacks fine finger articulation and physical contact measurements, and hand-focused datasets typically omit the accompanying whole-body movement. The dataset contains 2,715 egocentric recordings totaling 100 hours across ten task templates and five environments, with an emphasis on professional activities such as laboratories and factories; laboratory activity accounts for 41% of recordings and 46% of duration.
To recover whole-body motion from five sparse trackers, the capture pipeline first separates fixed body-sensor geometry from time-varying joint motion, then completes the joints that were not observed. Direct fitting is ambiguous: each tracker pose depends jointly on the attachment transform and the body's kinematic chain, so different combinations of mounting offsets and joint configurations can explain the same sparse measurements, and mount errors get absorbed into the estimated motion. Calibration matches a prescribed movement sequence to a reference motion library by phase and fits body shape, scale and sensor-to-anchor transforms once per mounting; a temporal motion prior then produces an initial whole-body trajectory from the five anchors, which is refined jointly under skeletal measurement agreement, body-surface clearance and ground non-penetration.
Against tracker-fitted reference labels, EgoExplorer reaches 18.0 mm body-point error and 20.3 mm at non-tracked points, below the three baseline settings listed in the table. The five-anchor AvatarPoser∗ baseline reports 64.1 mm body-point error, and the three-anchor public baselines range from 114.7 to 235.3 mm; relative to the strongest baseline, body-point error falls by 71.9%. Evaluation runs on a purpose-built Evaluation Subset whose reference labels come from the same five-tracker refinement pipeline and were reviewed frame by frame, agreeing with the trackers to within 5 mm, so this agreement is not independent optical ground-truth accuracy.
Gloves measure finger articulation and contact directly, and in matched-configuration hand tests they hold poses more stably than video-based reconstruction methods. The video-based methods HaWoR and HandFlow report hold RMS of 4.5 mm and 11.3 mm, while the glove setting reports 0.4 mm and reaches 100% kinematic target recovery. In regional contact evaluation the glove setting reaches 0.915 IoU, against 0.621 for HACO and 0.324 for hand-pose-only inference; these are separately recorded matched configurations, not synchronized hand-pose references.
Perspective
The demonstrated scope is 2,715 recordings totaling 100 hours across five environments and ten task templates, with an emphasis on professional activities such as laboratories and factories; the body-accuracy comparison runs on a purpose-built Evaluation Subset whose reference labels come from the same five-tracker refinement pipeline. The authors present multimodal generation, conditional generation and simulated human-to-robot motion transfer as preliminary studies and state that benefits for real-world humanoid learning remain unverified.
Reconstruction cannot guarantee exact recovery of all joints, and tactile sensing has variable reliability; expert review of model-generated step text still finds factual errors in 11.09% of reviewed segments and unsupported claims in 2.46%, so human verification and correction precede release. The simulated human-to-robot transfer replays training clips, establishing neither generalization to unseen tasks nor hardware deployment.
