Skip to main content
Back to timeline
arXivSource publication:

Human touch data can teach robots how hard to press: FACT beats T-Rex by up to 60 points with 10 robot trajectories per task

Synopsis

FACT pretrains on about 2,000 hours of human demonstrations that carry tactile force together with 100 hours of robot teleoperation data, then fine-tunes on ten real contact-rich tasks. It first plans coarse actions from predicted future video and visual tracks, then jointly refines the action with predicted future fingertip force. Fine-tuned with as few as 10 robot trajectories plus 90 human trajectories per task, FACT outperforms T-Rex and the baseline on all ten tasks, with success-rate gains of up to 60 percentage points; ablations show force supervision in human pretraining data matters most downstream.

Source-provided article image: FACT: A Multimodal Force-Action-Track Foundation Model for Human-to-Robot Dexterity
arXiv

Interpretation

Dexterous robot manipulation can now be pretrained from human demonstrations that carry contact force, rather than relying only on costly robot data collection. Most human manipulation datasets record vision and hand motion without measuring contact force, so force enters a policy only as an input. On ten real contact-rich tasks, fine-tuned with 10 robot trajectories plus 90 human trajectories per task, FACT exceeds T-Rex by up to 60 percentage points in success rate and outperforms both T-Rex and the baseline across all ten tasks.

The model splits action generation into a planning stage and a force-aware refinement stage, letting predicted future fingertip force and action shape each other during refinement. T-Rex feeds force to its refiner only as vector-quantized observation codes and never predicts it, leaving action-contact dynamics implicit. Evaluated on a bimanual YAM station with Sharpa Wave hands over ten rollouts per task with randomized object placement; removing future-force prediction drops test-tube cap opening from 80% to 40% and clean harp notes from 6.7 to 0.

Force supervision inside human pretraining data is the main source of downstream robot performance. Disabling human force supervision and masking human tactile inputs in the same human-robot mixture degrades contact-sensitive tasks sharply. Test-tube cap opening falls from 80% to 0%, pipetting from 40% to 20%, pipette-dial engagement from 5/10 to 2/10 with mean error rising from 6.4 to 20.5 microliters, and xylophone clean notes from 5.4 to 2.2.

Perspective

Evaluation is confined to ten real contact-rich tasks on a bimanual YAM station with Sharpa Wave hands, with ten randomized-placement rollouts per task and shared scenes and objects across methods. Pretraining mixes about 2,000 hours of EgoForce human demonstrations with 100 hours of T-Rex robot teleoperation data, and fine-tuning uses 10 robot trajectories plus 90 human trajectories per task.

In the pipette-volume task only five of ten trials engage the dial, so failures arise mainly from engaging rather than from adjusting once engaged; four of the five engaged readings land within 4 microliters of the target. Raising the robot-to-human ratio in fine-tuning from 10:90 to 50:50 lifts flip-cap opening from 50% to 60%, multi-object grasping from 40% to 50% and dial engagement from 5/10 to 6/10, while pipette-volume error stays similar across the three mixtures.