TraceDance auto-builds 107 behavior benchmarks from 252,557 real deployment traces, and nine frontier models pass only 26.7% on average
TraceDance is an agent system that constructs targeted benchmarks from real deployment traces for user-specified undesirable behaviors, using Anchor-and-Confirm (programmable retrieval plus candidate-level confirmation by a Flash LLM) and an Anchor Synthesis Loop to generate behavior specifications, and using decision-point continuation so an evaluated LLM produces its next turn at a recorded decision point and is graded by a behavior-specific rubric; experiments on 252,557 sessions from Claude Code and OpenClaw produce 107 benchmarks with 4,125 instances, fulfilling 95.3% of build-target requests, with both human annotators confirming the requested behavior in 84% of sampled instances, while nine frontier LLMs achieve a mean pass rate of only 26.7%.