Skip to main content
Back to timeline
arXivSource publication:

Safe decoys for AI vulnerability hunters: confirmed real bugs fall 38.7% to 60.4% across five models

Synopsis

In autonomous vulnerability discovery, an AI agent forms hypotheses cheaply but spends most of its budget verifying them. RedHerring inserts certifiably safe decoy paths into a repository to divert that verification effort away from real vulnerabilities. Across 33 OSS-Fuzz projects, 70 evaluation instances and five models under matched budgets, confirmed real vulnerabilities fall by 38.7% to 60.4%; even when the agent is told decoys may be present, the reduction is still 37.2% relative to an informed baseline.

Interpretation

A defender can reduce how many real vulnerabilities an AI agent finds by consuming its verification budget, without knowing where those vulnerabilities are. Earlier patching, obfuscation and deception defenses either require the defender to locate the protected code in advance or act only after a vulnerability is found, during exploitation or runtime interaction. On 33 OSS-Fuzz projects and 70 evaluation instances, confirmed real vulnerabilities under matched budgets fell from 149 to 59, 90 to 40, 98 to 54, 84 to 50 and 124 to 76 across the five models, reductions of 60.4%, 55.6%, 44.9%, 40.5% and 38.7%.

Each decoy pairs a vulnerability chain derived from a real CVE, which attracts verification, with a false bridge that keeps its dangerous sink unreachable. The false bridge is a predicate no input satisfies, for example one built on quadratic residuosity: the defender holds large primes p and q as a private certificate, publishes N = pq and a non-residue A, and the sink is reached only if x² ≡ A (mod N). The private certificate lets the defender decide efficiently that the predicate is unsatisfiable, while deciding it from public parameters alone is computationally hard; each decoy is also rewritten with the target repository's types, call structure and naming so it reads as ordinary input processing.

The verification effort decoys absorb tracks the size of the reduction, indicating that diversion of verification effort explains it. In RedHerring runs, agents spend 30.6% to 51.5% of completion tokens and an estimated 32.5% to 49.9% of runtime verifying decoys. The five models follow the same order by decoy effort share and by reduction, with a Pearson correlation of 0.99; in the ablation the full configuration reduces discovery by 60.4%, removing the chain gives 25.5%, a simple always-false gate gives 32.9%, and harmless code gives 14.1%.

Perspective

The evaluation is limited to 70 instances from 33 OSS-Fuzz projects, with five decoys inserted per instance, Claude Code as the agent scaffold, and a budget of 3 hours and 300 agent rounds; all five models are frontier open-weight models, while Claude and GPT were excluded because of safety-alignment refusals.

Each configuration is run once, and robustness comes from breadth across models and instances; behavior preservation is checked by native test suites, differential testing and targeted fuzzing, without a formal equivalence proof; effort shares rely on turn labels from an annotation model combined with symbol matching; the target repositories are public, so models may have seen some vulnerabilities in training, which applies to both conditions and makes the measured reduction a conservative estimate.