Skip to main content
Search

# Position Bias

5 related articles

Subscribe to updates
  1. arXiv

    Option order stops steering the answer: iterated cyclic reordering drives flip rate to zero across five benchmarks and five models

    System One decision models answer a choice question in one forward pass, but the answer shifts with the order in which options are listed, and the probabilities carry a prior over answer labels. IJev starts from an order that ignores the input order, repeatedly reorders options by a prior-corrected cyclic average, and averages the round distributions instead of returning the last round. Across five benchmarks and five models, every cell has flip rate zero; the default uniform round average lowers negative log-likelihood in 24 of 25 cells while accuracy stays comparable to a single cyclic average.

  2. Self-consistency backfires on long context: seven of eight samples agree on the same wrong answer

    On long-context tasks, self-consistency, the practice of sampling several answers and aggregating them, no longer improves performance and often lowers it, because positional bias makes the samples repeat the same mistake and the vote turns that mistake into a consensus.

  3. arXiv

    BayesJudge models conflicting human and LLM judgments as posterior distributions, raising expert prediction accuracy to 65.3% on SummEval

    The authors propose BayesJudge, an online Bayesian meta-evaluation layer for conflicting human-LLM judgment streams that outputs a posterior verdict distribution per comparison while estimating rater-specific human confusion matrices and LLM presentation-order bias, using tie-open labels and paired AB/BA order swaps as protocol-level identification instruments; synthetic experiments recover prespecified evaluator parameters, and on SummEval it detects LLM position effects, separates expert from crowdworker behavior signatures without rater metadata, and yields posterior uncertainty that correlates with human disagreement.

  4. arXiv

    Pre-verdict residual stream activations predict LLM judge position flips with AUROC .621–.850 and transfer to .685–.853

    Using nested grouped cross-validation on 534 JudgeBench pairs and 1,802 MT-Bench comparisons, this work trains L2-regularized linear probes on residual stream activations recorded immediately before the verdict to predict position flips, where swapping candidate order changes the judge's selected response identity; AUROCs reach .621–.850 across four judges, exceeding a combined baseline of verbalized confidence, verdict logits, length differences, and the initial choice by .062–.113, and frozen probes transfer to MT-Bench at .685–.853.

  5. arXiv

    Sparse Autoencoders Localize the Clinical Triage Format Effect: Medical Features Go Inactive at the Decision Token, Scaffold Features Account for Over 91% of Gemma Attribution

    Using sparse-autoencoder (SAE) features in Gemma 3 4B/12B IT and Qwen3-8B on clinical triage vignettes, this study finds that medical features fire on the shared clinical narrative under both formats but are inactive at the multiple-choice decision token; emergency-tier information is linearly decodable from vignette representations (ROC-AUC 0.95–1.00) with no significant format difference yet is attenuated at the decision token, and scaffold-peaking features account for over 91% of unsigned attribution in both Gemma models, placing the strongest correlates of the format effect at answer selection.

You’re all caught up