Skip to main content
Search

# Design-Based Cross-Validation (DB-CV)

1 related articles

Subscribe to updates
  1. arXiv

    Bringing survey small-area estimation into AI evaluation: PP-S and PP-TS improve point and interval estimates on a benchmark and on deployed traffic, while DB-CV picks as well as an independent validation sample from one sample

    The work frames disaggregated AI evaluation as finite-population survey sampling, proposes prediction-powered smoothing (PP-S), a Bayesian model fit to each domain's prediction-powered estimate, plus a taxonomy extension (PP-TS) that borrows strength along a nested reporting hierarchy, and derives an approximately unbiased design-based cross-validation score (DB-CV); on the Open LLM Leaderboard (9,324 questions, 34 task types) and PRISM deployed traffic (68,371 rated responses, 21 LLMs by 3 conversation types), where every outcome is observed, the smoothed estimators beat direct estimators in point and interval estimation with near-nominal 95% coverage, and DB-CV selects as well as an independent validation sample at the same budget while estimating the chosen estimator's error far more ac

You’re all caught up