Skip to main content
Search

# LLM-as-a-Judge

39 related articles

Subscribe to updates
  1. arXiv

    Agent training data no longer needs pure expert demos: JudgeMix judges each step and lifts pass@1 by up to 8 points on AppWorld

    Supervised fine-tuning of AI agents on expert trajectories leaves training data mismatched with the target model's inference-time state distribution, so the model is brittle once it deviates from the ideal path. JudgeMix uses a judge to decide at each step whether the target model's proposed action is semantically correct, hands control to the expert only when the model struggles, and reweights expert supervision by that judgment. On the AppWorld and AutomationBench tool-use benchmarks, JudgeMix reaches the highest pass@1 for both Qwen3.5-2B and Qwen3.5-9B, up to 8 and 10 points over pure expert demonstrations.

  2. arXiv

    An independent LLM judge audits PET depolymerisation records: 67.5% of 3,942 pass as written, with fixes concentrated in catalyst and solvent mass

    Experimental data on homogeneous catalytic PET depolymerisation is scattered across prose, tables and supporting information, and manual consolidation does not scale. This pipeline has an extractor tag each record with the unique identifiers of its source text chunks, then an independent LLM judge reads the full paper, checks each field and proposes cited corrections. On 295 hand-curated ionic-liquid PET glycolysis experiments the judge agreed with human evaluation on 85.7% of reviewed records; across 3,942 records from 561 papers it accepted 67.5% as written, with corrections concentrated in quantitative fields such as catalyst and solvent mass.

  3. arXiv

    A 4B multimodal model reads and writes mobile UIs: fine-grained perception lifts generation score to 4.92

    UI-UG 2 is a 4B-parameter multimodal large language model that trains mobile UI understanding and generation in one pipeline. It annotates over 30,000 screenshots from RICO, VINS, Android in the Wild and Android Control with an 80-category two-level taxonomy aligned to front-end component semantics, paired with more than 5,000 requirement–screenshot–JSON tuples. Generation is optimized by online render-aware GRPO: sampled DSL outputs are rendered by a Playwright server-side pipeline, then scored by an LLM judge and contrastive preference ranking. On its own 80-category benchmark it gives the strongest referring and dense-grounding results among evaluated models, and its composite generation score of 4.92 is the highest among compared models.

  4. arXiv

    Mixed-quality judge pools can still be aggregated: AGORA learns trust weights with item-response theory and holds its edge down to a few hundred items

    LLM judges are a common substitute for costly human evaluation, and aggregating several judges is the standard remedy for any single judge's bias, but judge pools mix very different reliabilities while the human-labeled sets used to validate them hold a few hundred items or fewer. AGORA places every item on one latent quality scale and describes each judge by a sensitivity parameter and ordered rating thresholds, so judge reliability is learned jointly from all items without human labels. Across three benchmarks, AGORA matches or exceeds Dawid-Skene on large evaluation sets, and its advantage widens as the set shrinks toward a few hundred items.

  5. arXiv

    Delivered memory is not used memory: MemUse traces each delivery and finds most entries irrelevant and most relevant ones ignored

    MemUse moves the unit of evaluation from the task outcome to the individual memory delivery: across 8,000 SWE-bench Verified agent episodes, three memory systems stayed within 0.4 percentage points of a no-memory baseline on resolve rate, yet per-delivery judgments show most delivered entries were irrelevant, 51.3%–79.7% of relevant entries went unused in each system, and 23.2% of episodes with beneficial memory use still failed.

  6. A weak judge's taste is readable before training: the probe named Africa, and all 128 final answers said it

    Language models are often trained against a learned judge instead of the truth, but what that judge rewards could previously be seen only after training, from the policy's behavior. Three judges were trained from a 1B-parameter model on 500, 2,000 and 8,000 labeled answers, and a single-edit probe measured each judge's lift on wrong answers before any training. The probe predicted a climb onto flagged text against the weakest judge; after training, 115 of 128 final responses were one Sahara sentence ending "Answer: Africa", and the judge's mean score rose from 0.20 to 0.82. Against the strongest judge it predicted nothing climbable, and that run was not exploited.

  7. Literary translation now rewards a large model's scores: reinforcement learning on source sentences alone lifts a 7B model past QwQ-32B-preview on metaphor-heavy text

    Literary translation now gets its reinforcement-learning reward from a large model scoring against written criteria, instead of human preference labels or paired reference translations. Earlier DRT reasoning translation models were supervised on synthesized long thought and translation data, and reinforcement-learning routes such as R1-T1 and MT-R1 used machine translation metrics like COMET and CometKiwi as reward. Built on Qwen2.5-7B, DeepTrans-7B improves over its backbone by 16.3%, 13.8% and 9.0% on GEA5, GEA100 and GRF on MetaphorTrans, and beats sub-30B baselines on most metrics.

  8. arXiv

    Customer-support guidance without a re-ranker: a portable skill hits 99.2% procedural equivalence on 132 real cases

    A portable skill with no model weights, made of a natural-language operating manual and deterministic scripts, produced guidance procedurally equivalent to human-approved answers on 99.2% of 132 real customer-support cases, with 83.3% evidence sufficiency.

  9. cloud.google.com

    Antigravity's model picker gains two Claude models, letting developers switch mid-thread without losing workspace state

    Google Antigravity's model picker now includes Claude Opus 5.5 and Claude Sonnet 5.5 for Google Cloud customers, selectable by developers once an administrator enables them. Previously the models available inside Antigravity came from Google's own Gemini line, including Gemini 3.8 Flash and Gemini 4 Argon; Opus 5.5 targets complex work while Sonnet 5.5 is a faster, lower-cost complement. Both apply across Antigravity 2.0, the Antigravity CLI and the IDE extensions for VS Code, JetBrains IntelliJ and Visual Studio, and the active model can be switched between turns in one conversation thread while preserving workspace state, open artifacts and terminal context.

  10. arXiv

    Customer-support agent benchmark scores 17 models with five judges: GPT-5.4 wins everywhere, judges agree only moderately

    A 500-scenario zero-shot benchmark built from real customer-support transcripts scores 17 models with five independent judges, and GPT-5.4 ranks first under every judge while judges agree only moderately on pass/fail.