Skip to main content
Search

# Human Evaluation

4 related articles

Subscribe to updates
  1. arXiv

    Ten frontier models each trained a language model: six days produced working assistants, and the best matched NVIDIA's own post-trained release

    RSI Arena hands language-model post-training to ten frontier models: each agent starts from the same 30-billion-parameter base, picks its own data, writes its own training code and nominates checkpoints within six days and 1,000 GPU-hours. All ten produced working assistants, and the model trained by GPT-6 Astra closed a median 76% of the base-to-Instruct gap across 12 capability benchmarks; with answer length and formatting controlled, its human-preference rating is level with NVIDIA Instruct, 1,095 against 1,094. A second stage returned each model's blind battle records to its agent, and eight of the ten models improved their style-controlled rating relative to NVIDIA Instruct, while benchmark gains stayed modest and showed no clear link to preference gains.

  2. 400 ratings from 20 music-domain experts show LLM judges align only moderately with humans (up to r=0.55) yet far exceed BLEU and ROUGE-L reference baselines

    This work presents the first user study assessing the reliability of LLM-as-a-judge for evaluating conversational recommendation system (CRS) responses: 20 multi-turn sessions were sampled from TalkPlayData-Challenge, candidate responses were generated by four instruction-tuned LLMs, and 20 music-domain experts produced n=400 ratings on Personalization Quality and Explanation Quality; bootstrapped correlation analysis over 10,000 iterations found moderate positive alignment between LLM judges and human assessments (highest r=0.55 for Gemini-3.1Pro on personalization), outperforming all reference-based baselines.

  3. Embedding-Based Evaluation and Benchmarking Framework for Optimizing LLM Post-Processing of Medical Transcriptions from a Local Whisper System

    This study proposes a reference-free evaluation framework that combines embedding-based transcript-summary semantic similarity, stability analysis across repeated generations, and structured human evaluation to benchmark LLM post-processing models for clinical transcripts produced by a locally deployed Whisper system, applying it to 26 German physician-patient conversations in which four LLMs generated 416 summaries, and finding that GPT-OSS-120B and MedGemma-27B achieved the highest similarity scores across most embedding models, that cosine similarities across repeated runs typically exceeded 0.90, and that embedding-based similarity signals only partially agreed with human judgments of summary quality.

  4. Information Satisfaction: A Reader-Centered Axis for Summarization Evaluation

    This work introduces information satisfaction—a query paired with a reader persona—as an axis of summarization evaluation, and through five perturbation tests plus an expert human evaluation finds that traditional metrics such as ROUGE and BERTScore and LLM-as-judge metrics such as Llama-3.3-70B and Prometheus-7B mostly fail basic perturbation checks and agree with reader preferences at near-chance levels, indicating that existing metrics are insufficient measures of how well a summary serves a specific reader's informational needs.

You’re all caught up