WiTac stops feeding touch to policies alone: fusing wrist views into fingertip contact lifts average success by 28.8% across four real robot tasks
Synopsis
Sharpa Robotics with Peking University, Shanghai Jiao Tong University and other institutions built WiTac, a pretrained contact encoder that fuses ten-finger tactile history, hand kinematics and dual wrist-camera views into one contact token per finger before action generation. With the encoder frozen, WiTac raises average success on four real-world contact-rich tasks by 28.8% and 15.0% over vision-only base policies on GR00T N1.7 and a second VLA backbone, and by 18.8% and 7.5% over a wrist-free tactile encoder. Pretraining uses nearly 300 hours of synchronized SharpaDex v1.0 dexterous manipulation data, and none of the four evaluation tasks appears in that dataset.
Interpretation
In contact-rich manipulation, similar fingertip loading can accompany different hand–object–target configurations and call for different corrective motions; WiTac gives tactile features the spatial reference from wrist cameras before they enter the policy. Earlier visuotactile policies supplied standalone tactile features alongside visual streams, modulated visual features with touch, or encoded residuals from visually predicted touch, leaving local touch ungrounded in surrounding visual geometry. On GR00T N1.7, WiTac reaches 66.3% average success across the four tasks, against 37.5% for the vision-only base policy and 47.5% for the wrist-free WiTac-NW; on the second VLA backbone the same three settings reach 58.8%, 43.8% and 51.3%.
The encoder produces one contact token per finger, anchoring that finger's tactile history on its wrist-frame fingertip pose and then querying wrist image patches for visual context through cross-attention. On the tactile side, an eight-frame deformation history is compressed by a spatial backbone and a causal temporal encoder; on the visual side, a frozen DINOv3 ViT-B/16 extracts patch features from both wrist views, and two Transformer blocks run finger self-attention followed by cross-modal cross-attention. Pretraining runs on nearly 300 hours of synchronized tactile, wrist-image, robot-state and fingertip force–torque recordings from SharpaDex v1.0, supervised by five objectives covering contact-state and history reconstruction, masked wrist-feature recovery and action-conditioned wrench-response prediction, with no task-success labels.
During downstream policy learning WiTac stays frozen and enters the VLA action expert only through a lightweight gated cross-attention interface, leaving the policy's native visual, language and state inputs unchanged. A shared gate predictor averages valid contact tokens, applies layer normalization and an MLP, and passes the result through a rescaled sigmoid to produce one scalar gate per injected block, scaling up the contact residual during physical contact; gate supervision uses the active-pixel fraction on the primary fingers as a contact proxy. Ablations on GR00T show average success falling from 66.3% to 53.8% without representation pretraining, to 58.8% without masked visual recovery, and to 50.0% when tactile and wrist features are injected separately instead of fused beforehand.
Perspective
Evaluation uses a Sharpa North robot with two Sharpa Wave dexterous hands on four bimanual contact-rich tasks: battery insertion, bowl wiping, USB reinsertion and cabinet opening, with 20 trials per method per task and policies fine-tuned on 200 exoskeleton-teleoperated demonstrations per task; none of the four tasks is in the pretraining dataset.
Vision and touch are combined only at the feature level through cross-attention, without depth-based alignment in a shared 3D space; tactile history is paired only with current wrist images, without explicitly modeling their joint temporal evolution; transfer across hand morphologies, tactile sensors and camera configurations remains to be validated. In offline analyses, joint encoding does not consistently beat simple concatenation of wrist features and wrist-free tactile features, leading by 4.86% on the 20-class confirmation split while the other three settings differ by at most 1.03%. On the second VLA backbone, action error under noisy tactile input shows small regressions on battery insertion and bowl wiping.
