Incomplete scans become simulation-ready articulated assets: an LLM reads the mesh in meters and cuts joint-axis error to 6.8 degrees
Synopsis
Robot policies trained in simulation only transfer if the simulated asset matches the real object. USDCraft has a pretrained LLM turn a source mesh into a metric text description, write an executable asset program, and revise it against that same representation without task-specific training. On USDCraft-bench it reaches 84.83% part-matching F1 and 6.833 degrees joint-axis error; on Lightwheel it leads part matching and joint location but trails mesh-segmentation methods on whole-object Chamfer distance.
Interpretation
A simulation-ready articulated asset can now be built from an incomplete scanned mesh by having an LLM write an executable program and compile it, rather than only partitioning the surface already present. Earlier mesh-based methods split a static mesh into functional parts and estimate joint axes, which breaks down when surfaces are missing or parts are fused, and cannot restore missing geometry or openings. On the 60 assets of USDCraft-bench, USDCraft–Astra reaches 84.83% part-matching F1, 0.694 static gIoU and 6.833 degrees joint-axis error, beating all four baselines ArtLLM, SIMART, Particulate and Articraft on every metric.
The mechanism turns the mesh into metric text the agent can read, then repeatedly aligns the program with that text. An LLM cannot read a mesh directly, rendered views lose metric scale, and raw vertex lists exceed the context budget, so an intermediate representation that keeps metric size is needed. Source geometry analysis samples surface support onto a regular grid, serializes ordered axial slices, and marks unobserved cells as unknown rather than empty or solid; iterative geometric rechecking re-encodes each candidate on the same grid and uses coverage gaps and first-surface depth differences to locate which program edit is needed.
The same program representation also serves generation without a mesh and has produced a large asset library. In generation there is no source mesh, so visual feedback becomes the main check, and materials, textures and PhysX physical parameters are authored directly in the program. On 50 text-conditioned and 50 image-conditioned requests per setting, USDCraft–Astra scores highest on all five judged dimensions with the shortest time, a 4.9-minute median; the resulting USDCraft-10k spans more than 500 everyday object categories.
Perspective
The setting is reconstruction from a static mesh plus a reference image, producing USD assets that load into Isaac Sim with collision geometry, masses, contact parameters and joint limits. Real-robot tests cover drawer opening, toaster-lever pressing and toaster switching, with 20 simulation and 20 real trials per task and policies trained only on simulation demonstrations.
Fine lattice structures such as racket strings and shopping-cart wire frames remain hard to reconstruct faithfully, with wire spacing and local connections differing from the source. Deformable objects are supported only in annotated states and rest geometry, and their bending and contact response has not been validated in Isaac Sim. On Lightwheel, whole-object Chamfer distance still trails methods that segment the supplied mesh, because rebuilding every part accumulates small surface deviations.
