Skip to main content
Back to timeline
arXivSource publication:

Ropedia Xperience 2 registers scene scans and egocentric motion in one frame, so a single capture becomes simulation environments and robot trajectories

Synopsis

Ropedia released Xperience 2, which registers metric 3D scans of real scenes and egocentric human-motion recordings into one coordinate frame, so a single capture serves both as an evaluation reference and as a source of simulation environments and robot trajectories. It contains 1,589 released scans plus 4,230 recordings of 74 tasks totaling 131 hours, collected with a handheld LiDAR scanner and a head-mounted camera and requiring neither robots nor motion-capture infrastructure. Across the shared 20 evaluation scenes, three benchmarks on depth and geometry, 3D object detection and spatial reasoning show existing models falling short in different ways; in 3D object detection, for instance, no method exceeds 7 AP50.

Source-provided article image: Ropedia Xperience 2
arXiv

Interpretation

Geometric scans of real scenes and human-motion recordings now share one metric coordinate frame, so a single on-site capture can serve both as an evaluation reference and as a source of robot data. Earlier scene datasets emphasized multi-view observation of real environments, while egocentric datasets emphasized activity and motion; the two resources were captured and used separately. The release comprises 1,589 released scans together with 4,230 recordings of 74 tasks totaling 131 hours, and scans and recordings are linked to the same scene frame by a registration pipeline.

Capture proceeds in two stages: a handheld LiDAR scanner with fisheye and wide-angle cameras scans the scene once, then an operator wearing a four-fisheye head-mounted recorder performs the assigned tasks one by one. A processing pipeline reconstructs meshes and 3D Gaussian splatting models from the scan and estimates camera trajectories, depth, hand and body pose from the recordings, then generates hierarchical language annotations with a large multimodal model. Registration localizes recording frames with SIFT matching, RANSAC and PnP, fits the recording-to-scene transform by robust Umeyama alignment, and assigns PASS, WEAK or FAIL labels from held-out error, gravity consistency and an ICP check of depth against the mesh.

Each capture is reused along four robot-learning routes: conversion into an interactive simulation scene, retargeting of human motion to a wheeled humanoid, virtual teleoperation in simulation, and automated long-horizon trajectory generation. These routes offer lower-cost paths to training environments and demonstrations at different levels of fidelity, complementing real-robot teleoperation, and capture cost scales with operators rather than with robots. One converted scene yields 217 separate objects; retargeting a 10.3-minute recording gives median fingertip errors of 0.40–0.55 cm; and for one eight-subtask task, all 30 rollouts from 30 held-out start states complete every subtask.

Perspective

The released scans are dominated by home interiors, which account for 1,389 scans (87.4%); all three benchmarks share the same 20 scanned scenes, 15 indoor and 5 outdoor. The demonstrations cover a kettle grasp and transfer in one converted kitchen scene, 10.3 minutes of motion retargeted across four recordings in one scene, and one eight-subtask household task.

Hand and body motion are estimates rather than motion-capture ground truth, unobserved parts such as the legs are inferred from a learned motion prior, and activity captions are model-generated. A single recording-to-scene transform cannot correct SLAM drift within a recording, and recordings made in place with a trajectory span below about 2 m are registered rigidly. Release grades incorporate policy overrides and should not be read as a uniform visual or geometric validation of all retained scans.