Skip to main content
Search

# Candidate-Independent Block-Causal Attention

1 related articles

Subscribe to updates
  1. arXiv

    Candidate-independent block-causal attention makes decision models insensitive to candidate ordering while retaining decision quality

    The work introduces candidate-independent block-causal attention, which preserves causal computation within the shared context and each candidate while blocking cross-candidate information flow and resetting candidate positions; across Gemma 3 1B, Qwen3 1.7B, and Qwen3 4B backbones, compared with standard causal attention and complementary invariant baselines, this architecture consistently reduces permutation sensitivity while retaining competitive decision quality, with ablations indicating that candidate isolation is the primary source of the effect and position resetting completing the intended symmetry, and a further Qwen3-4B study examines the architecture's behavior with substantially more training data.

You’re all caught up