Skip to main content
Search

# Summarization Evaluation

2 related articles

Subscribe to updates
  1. EViSE scores summaries on faithfulness, coherence and relevance in one encoder pass, reaching 0.806 overall Spearman on held-out News

    EViSE splits summary evaluation into faithfulness, coherence and relevance and returns all three scores from a single non-autoregressive encoder pass, training score regression together with a within-source ordering constraint over candidate summaries. On 80,856 human-reviewed Vietnamese document–summary instances, EViSE-mmBERT reaches an overall Spearman correlation of 0.806 on the held-out News test set and the compact EViSE-ViDeBERTa reaches 0.737; adding the ranking term to the same architecture raises the three criterion Spearman values by 0.045 to 0.061 over regression alone.

  2. Information Satisfaction: A Reader-Centered Axis for Summarization Evaluation

    This work introduces information satisfaction—a query paired with a reader persona—as an axis of summarization evaluation, and through five perturbation tests plus an expert human evaluation finds that traditional metrics such as ROUGE and BERTScore and LLM-as-judge metrics such as Llama-3.3-70B and Prometheus-7B mostly fail basic perturbation checks and agree with reader preferences at near-chance levels, indicating that existing metrics are insufficient measures of how well a summary serves a specific reader's informational needs.

You’re all caught up