Skip to main content
Search

# LLM benchmark evaluation

1 related articles

Subscribe to updates
  1. AI2's XYBench tests 8,000+ real questions to see whether AI assistants catch a user's mistaken premise

    AI2, with Carnegie Mellon, NVIDIA and Johns Hopkins, released XYBench, a benchmark of 8,115 real questions drawn from Stack Overflow, Stack Exchange, hand-curated examples and WikiHow, testing whether AI assistants recognize the mistaken premise behind a user's request. On Stack Overflow questions, the five evaluated models offered a solution to the stated request in 75–86% of responses but a solution to the underlying goal in only 59–71%; even when the user's intent was supplied, misconception identification did not reliably rise.

You’re all caught up