Counterfactual rewriting gets its own exam: nine models top out at 61.34%, frontier systems average 58.05%
Synopsis
Columbia's Causal Artificial Intelligence Lab released CoBe, a benchmark for conversational counterfactual text editing: given only an everyday-language hypothetical and no causal graph or editing steps, a model must keep facts unaffected by the change and rewrite only their causal downstream consequences. The benchmark holds 2,000 questions across healthcare, science, engineering, business and everyday scenarios; across nine tested models the highest accuracy is 61.34%, and frontier systems average 58.05%. The authors report that models often fall back on associational generation and that performance does not scale consistently with model size or output length.
Interpretation
Language models are asked to perform a kind of text revision that had not been measured on its own: given a narrative and a hypothetical change phrased in the subjunctive, the model must decide which facts the change leaves untouched and must preserve them, and which consequences must be rewritten. Earlier counterfactual-text datasets did not clearly separate counterfactual editing from associational generation, which produces text statistically matched to the altered condition and may revise background facts along with it. CoBe checks each rewrite against expert-verified criteria for preserved facts, required changes and causal connectors, and a response counts as correct only if all three pass; the LLM judge agrees with three human experts at a Cohen's κ of 0.812.
The approach turns counterfactual reasoning into three decidable checks rather than a comparison against one reference answer. Similarity scores such as BLEU and ROUGE do not directly test which facts should stay invariant or change under an intervention, whereas CoBe admits multiple valid rewrites with different wording and outcomes. The 2,000 questions come from 106 core examples expanded by an LLM into 25 variations each, with 650 of 2,650 candidates removed, and each question carries three differently phrased prompts whose scores are averaged.
The tested models commonly treat correlation as causation, and their most frequent error is editing non-descendant facts that should have stayed fixed. Errors from failing to update downstream variables and from wrong connectors are comparatively less common, so the failures concentrate on judging which facts are unrelated to the hypothetical change. Scores differ negligibly across the three query phrasings for the nine models; stronger conversational prompts raise frontier models by about 5% on average, and one explicit preserve-and-update instruction adds about 3%.
Perspective
The benchmark tests only conversational counterfactual editing of narrative text, and the authors state explicitly that passing it should not be read as achieving general counterfactual reasoning; the evaluation covers nine models with three query phrasings per question, and the authors propose extending the benchmark to domains such as vision as a next direction.
In the implementation, the required-change check accepts deleting a listed fact, so an incomplete rewrite can pass by omission; the authors report no systematic use of this shortcut in manual inspection and do not claim the criteria resist deliberate evasion. The authors also note possible author bias in sample curation and address it with domain and phrasing diversity plus independent human quality checks.
