WinDEX turns in-the-wild egocentric stereo video into metric hand trajectories, with offline reconstruction at 43.61 FPS
Synopsis
WinDEX converts hands recorded by a head-worn fisheye stereo camera in everyday scenes into metric, gravity-aligned hand motion in a world frame for robot teleoperation and reinforcement learning. Such data previously required optical motion capture, multi-camera rigs or gloves, or came from monocular video without absolute scale. The system generates annotations through geometric-temporal optimization, trains the end-to-end fisheye-native WinDEX-Net, and distills offline WinDEX-Window and causal streaming WinDEX-Stream; on held-out data, offline reconstruction reaches 14.5 pixels reprojection error and, with visual-inertial odometry, 37.29 mm world-frame joint error.
Interpretation
WinDEX gives hands captured in the wild metric scale, world-frame placement and a scalable annotation source at once, without gloves on the hands or a fixed laboratory rig. Optical motion capture and multi-camera datasets depend on instrumented environments, gloves and exoskeletons support portable capture but still keep hardware on the hands, and monocular video leaves depth and absolute scale ambiguous because a single view does not determine them. On held-out human-annotated in-the-wild 2D references, offline reconstruction reaches 14.5 pixels reprojection error versus 22.42 to 26.72 pixels for fine-tuned baselines; against independent optical motion capture, world-frame joint error with visual-inertial odometry is 37.29 mm.
Reconstruction runs through a fisheye-native end-to-end network that combines calibrated stereo geometry, local hand detail and temporal context in one model and outputs hand pose with metric wrist position. Monocular image and video estimators rely on appearance and motion priors to resolve single-view ambiguity, while multi-view methods target calibrated camera arrays rather than a short-baseline head-worn fisheye stereo pair with camera and hand moving together. Supervision comes from geometric-temporal optimization: hands are fitted to manually reviewed 2D keypoints, 3D labels are filtered by reprojection error, and visual-inertial SLAM places trajectories in the world frame; model predictions then initialize further annotation in a data-model loop, and the training corpus reaches 1,000 hours of in-the-wild annotations.
Perspective
Evaluation is confined to held-out WinDEX-Hand recordings, with 3D accuracy referenced to independent optical motion capture and image consistency referenced to human-annotated in-the-wild 2D keypoints; policy-learning results come from simulated grasp-lift-place trials, with policy transfer to physical robots also reported.
Prolonged occlusion removes visual constraints on finger articulation and position, and persistent tracking and recovery still need stronger temporal reasoning and scene-interaction context; training covers only CyberCap recordings, so cross-device generalization remains to be assessed; the authors also list combining object geometry, pose and intended interaction as future work.
