Agents are now software's users: Vercel's Harbor coffee-order eval shows pass@5 near 99% but pass5 near 8%
Vercel frames AI agents as real users of software and proposes evaluating them across discovery, selection, access and success, concentrating on whether the task actually gets done. Using the open-source Harbor framework to run whole tasks in containers and score each constraint, a coffee-ordering command-line example has Claude Code with Opus 5.5 pass three of five trials: pass@5 is about 99% while pass5 is about 8%, showing that doing it once and doing it every time are different questions.