Skip to main content
Back to timeline
langchain.comSource publication:

LangSmith now reviews whole conversations in annotation queues, so multi-turn agent errors no longer hide behind the last reply

Synopsis

LangSmith's annotation queues can now treat a full multi-turn session as one review item, not just a single run or step. Multi-turn agent errors often begin several turns earlier and only become visible in the final reply, so step-level review misses the earlier messages and tool calls that explain the answer. Reviewers score the conversation and give their reasoning, and those labels are used to align LLM judges, fix online evaluators and build golden datasets.

Source-provided article image: Why human insight is a strategic advantage for building better agents
langchain.com

Interpretation

The unit of agent review moves from a single run to the whole conversation, so a reviewer sees the interaction the user actually had rather than one step cut out of it. Previously an annotation-queue item could be one run, a single step or span; seeing only one step misses earlier messages, tool calls and decisions that explain the final answer. Fetch Rewards reviewers assess individual turns of the AI Rewards Assistant, an input and an output, but often need to inspect trace metadata to confirm which tools the assistant called and whether its answer matches the offers returned.

Reviewers leave not only a pass or fail but the reason why, and that reasoning is stored as reusable evaluation data. When an automated evaluator or online evaluator flags a trace, the flag is a signal rather than an explanation and usually is not enough to know how to change the agent. In an annotation queue the expert perspective is stored on the run or thread as a score plus the reviewer's reasoning, so builders can turn a single expert review into a clearer definition of quality for the next iteration.

Reviewers read a trajectory view while the queue item remains the thread, because the thing being judged is the conversation. A trajectory shows messages and tool calls in the order they happened with nested run-tree and duplicate information removed, so domain experts need not first learn to read a trace tree. Reviewers read the trajectory as they annotate the thread, which keeps the review next to the interaction a person actually had.

Perspective

The approach targets teams building and evaluating multi-turn agents, demonstrated in conversation review within LangSmith annotation queues and in Fetch Rewards' turn-by-turn assessment of the AI Rewards Assistant.

The text calls human review an important part of building better agents, which is the team's own position; the Fetch Rewards case describes how reviewers check tool calls and offer availability but reports no change in evaluation metrics.