Skip to main content
Back to timeline
arXivSource publication:

A multiple-choice bias score splits in two: flip rate and positional skew, so two mitigations stop looking alike

Synopsis

Under cyclic reorderings that show every option at every position equally often, the pooled selection-rate standard deviation factors exactly into two label-free quantities: the flip rate, how often the chosen option changes with order, and the positional skew, how unevenly the changed answers land on letters. Across 41 open-weight checkpoints and 12 benchmarks, the flip rate tracks accuracy and output uncertainty, while the skew tracks the model's option-symbol prior. Prior correction mainly lowers the skew and aggregating an ordering with its reversal mainly lowers the flip rate; both narrow the accuracy range across correct-answer letters by similar amounts, but aggregation leaves fewer model-rank reversals and settles more items on a wrong option.

Source-provided article image: One Bias Score, Two Factors: How Mitigations Affect Option Bias in Multiple-Choice Evaluation
arXiv

Interpretation

The option bias that multiple-choice evaluation usually compresses into one number now splits into two interpretable quantities: how often the answer changes with option order, and which letters the changed answers favour. Previously selection bias was pooled over a benchmark's answers and order sensitivity counted per item, reported as separate statistics, and a single pooled score could not say which behaviour a model or mitigation had changed. When every option of each question appears equally often at each position, the pooled selection-rate standard deviation is exactly the product of the flip rate and the positional skew; the identity holds across 41 open-weight checkpoints and 12 benchmarks and extends to balanced RStd, which is read over the letters holding the correct option.

Two common mitigations act on different factors: prior correction changes where the changed answers land, and multiplying a content's probabilities under an ordering and its reversal changes whether answers change at all. Both methods narrow the range of accuracy across correct-answer letters by similar amounts, so a single pooled score makes them look equivalent. On the same checkpoints and items, prior correction lowers balanced RStd from 0.064 to 0.036 and the reversal product also reaches 0.036, yet the product lowers the flip rate far more; prior correction barely moves the flip rate and does not reduce rank reversals, while the product cuts the share of checkpoint pairs whose ranking reverses from 15.4% to 10.6%.

Applying prior correction before the reversal product lowers both factors and narrows the accuracy range across correct-answer letters to 5.3 percentage points. Either method alone leaves the other factor nearly unchanged and narrows the placement range only to 9.6 or 8.2 percentage points. On the same checkpoint–benchmark cells with at least three options, prior correction followed by the reversal product narrows the placement range to 5.3 percentage points while rank reversals stay at the product's 10.6%; the same table gives batch calibration followed by the reversal product 4.6 percentage points, and the text calls that composition the narrowest, which disagrees with the table, so only the 5.3 entry is reported here. The product and its compositions raise the stable-wrong share from 0 to at least 6.2 percentage points while accuracy rises 2.9 points.

Perspective

The analysis covers 41 open-weight checkpoints from seven model families on 12 benchmarks with two, four or ten options, with answers read as the highest next-token probability over option letters and five-shot, direct-answer and chain-of-thought prompting as checks; mitigation comparisons use the checkpoint–benchmark cells with at least three options, with priors estimated on a held-out 5% of each cell's items.

The analysis characterises the observable effects of option order, not the model-internal mechanisms that produce them, and the associations with accuracy and output uncertainty do not establish causal explanations. Lowering either factor does not guarantee better predictions: consistently wrong answers can remain stable, and a lower positional skew need not improve placement robustness or ranking stability. Experiments focus on deterministic readouts of mostly English multiple-choice benchmarks and specific mitigation configurations.