Skip to main content
Back to timeline
arXivSource publication:

Passing one reference solution is not enough: TestPrism gives each task 10 implementations, and the best coding agent reaches only 28.00% joint success

Synopsis

A Nanjing University team built TestPrism, a benchmark of 300 test tasks and 3000 labeled candidate implementations, to check whether generated tests actually separate correct code from incorrect code. Its main metric, the Joint Success Function, requires a test to fail on the initial program, accept every valid implementation and reject every invalid one; across fourteen baseline coding-agent configurations the best score is 28.00%, while single-reference success reaches 59.67%. The team also introduces TestHelix, which combines heterogeneous test-and-repair pairs, peer cross validation and recursive self improvement to raise that metric.

Interpretation

The criterion for test quality moves from passing one reference solution to accepting several valid implementations while rejecting several invalid ones. The common practice was single-reference evaluation: a suite counted as successful if it failed on the initial program and passed on one gold solution, even though a requirement can admit multiple valid implementations and plausible incorrect ones. On the 300 TestPrism tasks, the highest Joint Success Function among fourteen baseline coding-agent configurations is 28.00%, against 59.67% single-reference success for the same configurations.

TestHelix generates several test-and-repair pairs first, then uses peer repairs to cross-check those tests. Heterogeneous synthesis assigns isolated agents distinct exploration strategies, each producing a test patch and a repair patch, and requires the test to reproducibly fail on the initial program and pass on its own repair so that each pair is internally consistent. Peer cross validation executes tests against peer repairs, has an independent reviewer check assertions against public requirements, and corrects pairs with counterexamples; across GPT 5.6 sol and Kimi K3 it raises the Joint Success Function by 11.67 to 12.00 percentage points over single-pair generation under matched inference budget caps.

Perspective

The evaluation scope is the 300 TestPrism test tasks, drawn from 17 sources across six programming languages and 67 repositories and projects, with each task fixed to one reference, four valid alternatives and five invalid implementations; the TestHelix comparison runs on GPT 5.6 sol and Kimi K3 under the same inference budget caps.

The Joint Success Function cannot exceed single-reference success, and the gap includes failures from missing or erroneous execution outcomes; the metric falls as the candidate panel grows, so results depend on which implementations the panel contains. The failure analysis samples 300 failed trajectories, 50 from the highest-Joint-Success-Function configuration in each of six model families, and analyzes 286 after excluding 14 timeouts.