Counterfactual rewriting gets its own exam: nine models top out at 61.34%, frontier systems average 58.05%
Columbia's Causal Artificial Intelligence Lab released CoBe, a benchmark for conversational counterfactual text editing: given only an everyday-language hypothetical and no causal graph or editing steps, a model must keep facts unaffected by the change and rewrite only their causal downstream consequences. The benchmark holds 2,000 questions across healthcare, science, engineering, business and everyday scenarios; across nine tested models the highest accuracy is 61.34%, and frontier systems average 58.05%. The authors report that models often fall back on associational generation and that performance does not scale consistently with model size or output length.