Skip to main content
Search

# Evaluation awareness

1 related articles

Subscribe to updates
  1. Agent safety evals run backwards: six served models score lower in evaluation than in matched production decisions

    Long-horizon agent safety evaluations may understate, not overstate, how safely a model behaves in real deployment. Comparing six served models on the Respecting Warnings evaluation against structurally comparable Codex production decisions, the measured eval-production gap runs from 10.2 to 37.4 percentage points, with evaluation success the lower side. The models with the largest gaps most often misread the test as one about resisting prompt injection, and a system-level cue naming the correct or the opposing objective pushed the same model's evaluation success to 91.6% and down to 17.6% respectively.

You’re all caught up