Black-box processor probing moves off expert hands: ProcReverser's multi-agent workflow reaches 81.3% Pass@1 on the XuanTie C910
Synopsis
When processor implementation details are unavailable, security researchers can only infer hardware properties such as branch prediction and speculative execution from software experiments, redesigning tests, writing programs and interpreting results for every chip. ProcReverser hands that loop to five AI agents with separate roles, which use execution feedback to revise experimental plans and test programs and must check measurement evidence before a conclusion is issued. On 16 reference tasks for the XuanTie C910, five independent attempts gave 81.3% Pass@1 and 93.8% Pass@5; on 20 tasks for the gem5 RISC-V O3 model, 83.0% and 100.0%.
Interpretation
Black-box microarchitectural reverse engineering moves from experts hand-designing experiments per processor to a closed loop in which multiple agents design experiments, generate programs, diagnose failures and interpret results. Previously researchers had to design a test for each property, write and debug programs, establish controls, adapt measurements to the available interfaces, and often change assumptions, measurement methods and decision criteria when moving to another processor. On 16 reference tasks for the XuanTie C910, 65 of 80 independent attempts passed scoring for 81.3% Pass@1, and 15 tasks succeeded at least once for 93.8% Pass@5; on 20 gem5 RISC-V O3 tasks, 83 of 100 attempts passed for 83.0% Pass@1, and all 20 succeeded at least once for 100.0% Pass@5.
Roles are split: a Designer turns a property question into a plan with hypotheses, controls and quantitative decision criteria, a Reviewer checks whether the plan can distinguish competing explanations, a Generator implements the test program, a Diagnostician proposes repairs from compiler messages and execution logs, and an Analyst writes the property report. A single agent carrying all duties in one context reached only 17.5% and 18.0% Pass@1 on C910 and O3, against 68.8% and 72.0% for the five-role configuration. With the evidence-checking mechanism Discriminative Revalidation enabled, the five-role configuration's Pass@1 rose by 12.5 percentage points on C910 (95% CI [6.25, 21.25]) and by 11.0 percentage points on gem5 O3 (95% CI [4.00, 18.00]), with intervals from 10,000 paired bootstrap resamples of tasks within each platform.
The evidence-checking mechanism registers candidate explanations, the comparisons needed to separate them, predicted outcomes and quantitative criteria before execution, computes those comparisons directly from raw samples afterward, and routes specific gaps back to diagnosis for targeted retests. On the gem5 O3 hardware-prefetching task with the mechanism disabled, the system ran three rounds that returned only scan timings and misread the persistent timing gap as evidence of hardware prefetching. With it enabled, the system added first-access sampling of a probe cache line; the discovery phase gave 3 and 2 hits over 200 trials per condition, the confirmation batch gave 2 and 3, and both batches met the preregistered criterion of at most 5% hit rate per condition, yielding the conclusion that no hardware prefetching was detected under the tested access patterns.
Perspective
Evaluation covers two commercial RISC-V processors, the XuanTie C910 and SiFive P550, and the gem5 RISC-V O3 CPU model, over a 26-task suite focused on speculative execution, branch prediction, memory dependence and cache behavior; on the P550 the system reports property conclusions for 23 tasks, leaves R2, R3 and R7 undetermined for insufficient evidence, and some conclusions cover only the subproperties the measurements support.
Some properties may remain undetermined when target-side measurement signals are weak or the results cannot distinguish competing explanations; all experimental configurations use the same language model, and how model choice affects experimental design and identification accuracy needs controlled experiments; applicability to processors implementing other instruction sets such as Arm and x86 remains to be evaluated.
