Skip to main content
Back to timeline
arXivSource publication:

Frozen models get a sense of touch: ForceHarness lifts GPT-6 Astra from 3.3% to 90.0% on a real arm with no wrist force sensor

Synopsis

A frozen frontier model reached only through an API cannot read raw force streams or decide at the speed contact evolves, so force has mostly entered manipulation through demonstration-trained force-aware policies. ForceHarness places three layers between model and robot: hand force calibrated from a wrist sensor or joint torques alone, self-terminating force primitives that regulate force and return at a contact event, and a translation of each force trace into physical quantities with uncertainty. On a Franka FR3 without a wrist force sensor, GPT-6 Astra's zero-shot success rate rises from 3.3% to 90.0%; in simulation it rises from 70.0% to 87.5% on ForceBench and from 62.1% to 96.4% on the seven-task HiddenPhys.

Source-provided article image: ForceHarness: Turning Force into Action and Physics for Frozen Frontier Models
arXiv

Interpretation

Frozen frontier multimodal models can now control robots zero-shot from images, but tasks such as driving a screw or sorting bottles by weight depend on contact forces that images cannot show, and an API-only model can neither read force nor decide as fast as contact changes. Earlier work brought force in by training force-aware policies with wrist force as a network input, which needs demonstrations for every task and weight updates; existing harnesses for frozen models mostly pass raw wrench or tactile values, or have the model write force-control code. On a Franka FR3 without a wrist force sensor, six contact-rich tasks ran 10 episodes each under the same model and instruction: the no-force control succeeded in 2 of 60 episodes and ForceHarness in 54, while bottle sorting and soft-versus-hard probing both stayed at 0/10 without force.

Force is closed-loop regulated beneath the model and translated into physical quantities above it, and the model calls one primitive per turn and reads the result. The bottom layer estimates the force at the flange by least squares from a wrist force/torque sensor or from external joint torques alone, tares at rest and re-zeroes before each empty-hand primitive to cancel a zero point that drifts with pose. The middle layer packages twelve contact skills as self-terminating tools that apply force through the impedance controller's target lead and integrate force error into target velocity by admittance control, stopping on a contact event, a force cap or a timeout; the top layer converts each force trace into payload mass, contact height, friction and similar quantities with uncertainties and status flags using means and least-squares fits.

Perspective

The demonstrated scope is single-arm tabletop manipulation with a parallel gripper: one Franka FR3 without a wrist force sensor plus the ForceBench and HiddenPhys simulated benchmarks, with models from two API providers. On HiddenPhys the model reads noisy object poses as text rather than images, while camera input is covered by the real-robot experiments and ForceBench.

Querying a remote model adds seconds of latency between primitives, which suits quasi-static contact tasks but precludes fast reactive replanning during dynamic motion; the primitive menu and physics estimators are hand-specified, each task relies on a hand-written instruction, and component ablations run only in simulation. On ForceBench, tools and perception thresholds were tuned on eight of ten seeds, Stove Press and Nut Thread were revised after initial runs, and the public evaluator for Faucet Turn, Stove Turn and Drawer Pull omits checks such as release and clearance.