Skip to main content
Back to timeline
AnthropicSource publication:

Experts hand judgment calls to AI agents: 86% of four analysis choices followed the agent's lean, yet only 15% credited it

Synopsis

Anthropic had 50 professional analysts with causal-inference experience use Claude Code on a farming advisory analysis, while a hidden coin flip set the agent's lean on four analytical choices. Analysts' choices matched the agent's lean 86% of the time, against roughly 50% expected with no influence, and they later repeated the agent's arguments while rarely crediting it.

Source-provided article image: How AI agents shape experts’ judgment calls \ Anthropic
Anthropic

Interpretation

When experts hand analysis work to an AI agent, they follow the agent's lean on key judgment calls rather than only keeping the final decision. The usual assumption is that the agent writes code, runs numbers and offers advice, while the expert decides what to measure, how to read ambiguous evidence and what to do next. Across the four analytical choices, analysts' choices matched the agent's assigned lean 92% of the time for the outcome measure, 82% for the comparison method, 81% for which farms the estimate covered and 90% for weighting, 86% combined; with no agent input, about 50% would be expected.

The agent's lean shaped the estimate before analysts had committed, and through it the final recommendation. The lean had a weaker effect on the final recommendation itself, because once an estimate existed the recommendation largely followed from whether it cleared the board's 3–4% threshold. When the agent leaned toward rolling out the program, 54% of analysts recommended it, versus 36% when it leaned the other way; using only the four coin flips, a statistical model correctly predicted 82% of recommendations, against 61% for always guessing the most common one.

Analysts adopted the agent's arguments while still treating the conclusion as their own judgment. The post-task survey asked analysts to explain each choice, and they could not look back at their conversations with the agent while answering. 77% repeated the agent's argument in at least one answer, sometimes verbatim, but only 15% credited the agent anywhere; 52% called the final recommendation more their own than the agent's, and their choices shifted about 65 percentage points toward the agent's lean on average, versus 79 for the rest, a difference within the margin of error.

Perspective

The study examined one farming advisory analysis task with professional analysts who passed a causal-inference knowledge check, with the agent running Claude Opus 4.8 and a median task time of 65 minutes; the authors note that in real work experts would know the field better, likely hold stronger views, and could narrow the reasonable options through interviews and other means.

The agent's leans were deliberately strengthened and explicitly voiced, so the reported rates describe what happens when an agent leans one way and says so; the switching estimate rests on only 38 cases, with a 95% confidence interval from 24 to 82 percentage points, and the difference cannot be attributed with certainty to AI influence; the link between age and following the lean disappeared after adjustment for multiple comparisons.