Skip to main content
Search

# Benchmark Construction and Evaluation

14 related articles

Subscribe to updates
  1. Pothole Detectors Score 0.619 on the Published Split and 0.467 Without Near-Duplicates: A Video-Disjoint Re-Split of BharatPotHole

    The published split of the public road-defect dataset BharatPotHole overlaps between training and test: 38.8% of test images have a near-duplicate in training and 99.4% come from videos also used for training. After a deduplicated, video-disjoint re-split, the same YOLO26n detector reaches AP50 0.619 on the published test set and 0.467 on its near-duplicate-free subset, a paired difference of 0.152 (95% CI 0.090–0.234). In a controlled contamination test, replacing unseen test images with copies of training frames raised a video-disjoint model's AP50 from 0.133 to 0.494, while copies of unseen frames left it near 0.15.

  2. arXiv

    SkillMorph evolves agent skills via trajectory-guided fault localization, raising pass3 on SWE-Skills-Bench from 67.96% to 88.37%

    The work proposes SkillMorph, a skill-evolution approach based on trajectory-guided fault localization: it abstracts raw trajectories into state-transition sequences, aggregates failure and success coverage evidence across repeated runs, evolution loops, and tasks to identify suspicious actions, then traces those actions back to edit sites that may span multiple skills and artifacts before generating revisions; on SWE-Skills-Bench and CannBot the evolved skills achieve higher pass@3, trial-level accuracy, and pass3 than the original skills and four existing skill-evolution methods, and a deployment with an AI operator-development team produced six accepted skill-revision pull requests.

  3. arXiv

    MAPE-K dual-AI framework lets a quadruped robot turn natural-language intentions into executable code, robust on finite tasks while open-ended exhaustive search stays cognitively limited

    The work introduces a MAPE-K-inspired dual-AI architecture in which an LLM translates high-level natural-language intentions into executable Python code constrained to a robotic function library, while a VLM provides open-ended semantic grounding through a distillation process, with geometric and semantic thresholds triggering replanning; benchmarked on quadruped exploration tasks in Gazebo across three frontier models, finite tasks show high success rates while open-ended exhaustive tasks degrade, with failures concentrated in semantic grounding, completeness checking, and termination control rather than low-level action execution.

  4. arXiv

    APEX moves indirect prompt-injection defense to the execution boundary, cutting attack success to 0% on five of six benchmarks and 0.56% on the sixth

    The work presents APEX, which shifts indirect prompt-injection defense from recognizing attack patterns to checking each proposed effect at the execution boundary: before untrusted execution it compiles the user's authorization into a contract, WRAP admits an effect only when the contract and runtime evidence reconstruct its authorization, and PLANT places probes on the contract's endorsed dependencies so unendorsed use of runtime information reveals itself before commitment; across six benchmarks and 13 baselines APEX reaches 0% attack success on five benchmarks and 0.56% on the sixth, and holds 0% under three adaptive attacks spanning all three capability-unit types.

  5. arXiv

    EchoChat recasts empathetic spoken dialogue as a three-stage perception–mental-state–response pipeline, lifting EchoEval perception from 6.96 to 8.23 and reasoning from 6.61 to 8.43

    The work introduces EchoChat, which reformulates empathetic spoken dialogue as a structured cognitive reasoning process spanning perception, mental-state reasoning, and response generation, supported by the EchoDialogue-400K dataset of 400K single-turn samples plus 4K multi-turn sessions, Acoustic-Anchored Attention, and a stage-aware RL objective with Step-Decomposed Credit Assignment, together with the 1K-instance expert-annotated EchoEval benchmark; experiments report gains over 12 compared models with perception 8.23, reasoning 8.43, and response 9.26, and an average Spearman correlation of 0.83 between LLM-based scores and human judgments.

  6. arXiv

    Semantic entropy flags small-model errors on GSM8K at AUROC 0.871, but its synthetic-arithmetic counterpart is fully matched by a regex difficulty baseline

    The work tests semantic entropy as an escalation signal for routing queries from a small to a large language model across three benchmarks and two model families, finding that on GSM8K it reliably distinguishes the small model's mistakes (AUROC 0.871) and improves routed accuracy over random escalation by up to nine points at matched cost, while an equally strong-looking synthetic-arithmetic result is almost exactly matched by a model-free rule based only on how hard the question looks on the page; it then offers a checklist of controls, including a difficulty baseline, three escalation targets, an oracle-headroom pre-flight check, per-generation cost accounting, and a difficulty probe that predicted in advance a retrieval shortcut's collapse from AUROC 0.908 to 0.518.

  7. arXiv

    COSMI composes 222k multi-object interaction sequences from single-object captures and trains a diffusion model that generates one to five objects

    COSMI composes multi-object human-object interaction data from single-object captures by extracting contact-consistent clips, mirroring them to balance the hands, and admitting only semantically and physically plausible pairings via a language model and geometric checks, yielding a dataset of 222k sequences and 275 hours with up to five objects (nearly thirty times the largest multi-object capture); it then trains a text-driven diffusion transformer whose object slots share weights and whose objects are predicted relative to the body joints that drive them, outperforming HIMO and multi-object adaptations of MDM and PriorMDM in text alignment and contact accuracy on a benchmark with an unseen object and unseen combinations, with the largest margin on the unseen object.

  8. arXiv

    FairRSFM evaluates three remote sensing foundation models by biome group and finds aggregate metrics mask ecological performance gaps

    The authors introduce FairRSFM, which maps georeferenced samples from four remote sensing downstream datasets (m-EuroSAT, m-BigEarthNet, m-SA-Crop-Type, and MMEarth20K) to 14 terrestrial biome classes consolidated into six macro-groups, and evaluates Prithvi-EO-2.0, SatMAE, and DOFA under a unified frozen-backbone protocol across three random seeds, showing that aggregate metrics consistently mask biome-dependent disparities, for example Prithvi-EO-2.0 reaching high overall macro-F1 on m-EuroSAT while its worst-group score is markedly lower, and compares BOLP, DBR, and GroupDRO as mitigation baselines whose effectiveness is model- and task-dependent.

  9. arXiv

    CRAFT fills spreadsheet forms with template-aware reflection and local repair, raising pair accuracy by 8.51 and 23.38 points on FormFillBench

    The authors propose CRAFT, a template-aware agent framework for spreadsheet form filling that couples reflective validation with constrained local repair: it localizes suspicious regions, restores overwritten template content when needed, re-grounds plausible writable slots with a Rectangle-Aware Slot Grounder (RASG), and constrains later edits via label-slot hints and protected regions; they also release FormFillBench with 327 forms across Instruction-Only and Multi-File tracks, where CRAFT improves pair accuracy over the strongest baselines by 8.51 and 23.38 percentage points respectively, with component-removal experiments supporting structural adjudication and slot re-grounding and a second backbone retaining the relative advantage.

  10. arXiv

    BIRD Unifies Iterative Reward Design Evaluation: Simple Design Choices Are Most Robust, and ERA-U/ERA-S Rank First and Second Under Matched Budgets

    The authors introduce BIRD, which expresses 14 published iterative reward design (IRD) methods as configurations of one loop, compares 6 published methods and ablates 30 design choices across MuJoCo, Meta-World, Assistax and HumanoidBench under matched feedback conditions and policy-training budgets, finding that most choices (including the authors' own proposals) have small or inconclusive effects while a few simple choices (such as greedily retaining checkpoints and summarizing feedback) help consistently; a small hill-climb from RDA yields ERA-U and ERA-S, which take first and second place in the pooled Bradley–Terry ranking, though a small human study shows high benchmark scores do not imply natural behavior.