Boundary tests for conversational agents stop guessing: conditions pulled from replies lift explicitly valid tests to 62.7%–83.3%
Synopsis
Boundary testing of conversational agents gets a new route: instead of inferring expected behavior from workflow-graph nodes and edges, statements describing one policy condition and the behavior expected when it holds are extracted from the agent's replies in exploratory conversations, then paired with test transformation instructions. Across four domains, 62.7% to 83.3% of tests are valid boundary tests whose expected behavior is explicitly defined by the prompt, tool code, or knowledge base, above the 47.7% to 68.1% of prior AgentEval suites; transformed tests add 13 to 46 explicit boundaries.
Interpretation
Boundary test generation now targets natural-language statements, each naming one policy condition and the behavior the agent should show when that condition holds, so the generator receives the expected behavior directly instead of inferring it. The prior route mined a workflow graph from discovery traces and prompted a generator to perturb user messages toward the graph's nodes and edges; a node or edge merges several activities, so one target may contain several boundaries and the generator had to supply the expected behavior. Across four domains, 62.7% to 83.3% of tests are explicitly valid boundary tests, meaning every expected behavior follows from the agent's prompt, tool code, or knowledge base; AgentEval's suites reach 47.7% to 68.1%.
A statement is paired with a test transformation instruction that changes one factor of a test, such as the wording of a request, a value in it, or the conversation state, moving the test to the other side of the statement's boundary or to another boundary. A statement alone covers only the side it describes, so pairing adds a second dimension of variation rather than rewriting messages toward a single target. Transformed tests contribute explicit boundaries that plain tests miss in every domain, 13 to 46 of them; they are valid about as often as plain tests in the airline and banking domains and more often in retail and telecom.
The number of pairs far exceeds the attempts a run can try, so a selector picks each pair's statement by novelty and its instruction with the multi-armed bandit algorithm Bayes-UCB, learning from the outcomes of pairs already tried. Prior target selection fixed all targets before executing any test, so early outcomes could not guide later choices. Feedback comes from two signals: whether the generator writes a transformed test, and the LLM judge's pass, fail, or inconclusive verdict on the executed test; only 3% of attempts in the airline domain and 2% in retail end as not applicable.
As regression tests, the suites detect all eight seeded policy faults in the airline domain and four of eight in retail; AgentEval's suites detect five and two, with fewer than a third as many tests. The detection gap traces to coverage: for the three airline faults AgentEval misses, 0, 3, and 1 of its tests cover an affected boundary, against 7, 13, and 13 of AdaT2's tests. Faults are predefined policy flips that rewrite one policy passage to require different behavior; a test detects a fault when it passes on the original agent, fails on the faulty agent, and the auditor attributes the failure to that fault.
Perspective
The reported scope is black-box boundary testing on four text domains: airline, banking, retail, and telecom, with the -bench agent run on DeepSeek V4 Flash 0731, GPT-6 Luna serving as driver, statement extractor, generator, and judge, and GPT-6.1 Sol as auditor, all domains sharing one catalog of 91 instructions. Validity, boundary, and side counts come from the auditor, which does not see which suite a test belongs to, how it was generated, or the judge's verdicts. Regression testing covers only the airline and retail domains, and the seeded faults rewrite only policy text in the agent's prompt.
A boundary has two sides, and in each domain both methods cover fewer than a quarter of their explicit boundaries on both sides; a test covering only one side cannot show whether the agent distinguishes the two sides. Because each run needs thousands of LLM calls, RQ2 uses three runs per configuration and RQ1, RQ3, and RQ4 use one test suite per method and domain, too few for significance tests. Inconclusive judge verdicts are more frequent in AdaT2's suites, mostly on plain tests, lowering accuracy in banking, retail, and telecom. Validity labels, boundary and side counts, and most execution labels come from the auditor, so absolute rates depend on the auditor model, while comparisons within a domain share the same auditor.
