Agents are now software's users: Vercel's Harbor coffee-order eval shows pass@5 near 99% but pass5 near 8%
Synopsis
Vercel frames AI agents as real users of software and proposes evaluating them across discovery, selection, access and success, concentrating on whether the task actually gets done. Using the open-source Harbor framework to run whole tasks in containers and score each constraint, a coffee-ordering command-line example has Claude Code with Opus 5.5 pass three of five trials: pass@5 is about 99% while pass5 is about 8%, showing that doing it once and doing it every time are different questions.
Interpretation
The object of software testing is widening from people to AI agents: agents call APIs, command-line tools, MCP servers or SDKs on a user's behalf, so measuring how well that work gets done requires tests that are run repeatedly and scored. Earlier tests only answered whether a call succeeded, and a normally returned request looked fine in the logs; an agent can still order a hot latte when the user asked for iced. The coffee example makes the gap concrete: the agent orders at 24th St Beans with the right drink and pickup time but a $9.27 total including tax and tip, over the $8 budget, and the verifier scores the budget constraint as 0.
The approach layers evals by how many large-language-model calls they need: deterministic tests call no model, one-turn evals check a single decision, and end-to-end evals let the agent work through the whole task in an environment before checking the final result. A one-turn eval only sees whether one step looks reasonable, while success depends on the whole chain of searching cafes, filtering on user criteria and placing the order, where each step can look sensible and the outcome still be wrong. The example task makes the environment part real and part fake: the agent uses the real sfbeans command-line tool, but cafe data and the clock are frozen at 8:30am so every trial starts on the same morning, and the verifier scores drink, time and budget separately, awarding 1 only if all hold.
Perspective
The example focuses on the funnel's final stage, success, meaning whether the agent completes the user's task, and names the tool directly in the instruction to skip discovery and selection; evals run in container sandboxes, with cafe data and a frozen clock forming the controlled environment.
The four-variant scores are labeled illustrative; with five trials, 100% means no failures seen yet, not that the agent never fails; evals saturate as models and products improve, and a saturated eval still catches regressions but no longer shows what to improve next.
