No training needed to decide: frozen Qwen3.5-4B reaches 81.4% on public JevBench items, matching training-free readouts on the same backbone
Synopsis
A Microsoft team presents LLM-as-Jev, which turns off-the-shelf causal language models into decision models that return a probability distribution over candidate options without changing architecture or adding classification heads. Frozen Qwen3.5-4B scores 81.4% on the 231 public JevBench items, matching or slightly exceeding the training-free SemIf readout (80.5%) on the same backbone and beating the fine-tuned reflex 4B (79.7%). Fine-tuning brings clear gains mainly for the 0.6B model and high-cardinality routing, and KL anchors keep generation behavior from degrading.
Interpretation
A general-purpose LLM can already serve as a decision model without task-specific training: list the candidate options in the prompt, and the model's normalized continuation probabilities over them form a categorical distribution. Earlier community Jev-style models either added classification heads or reserved tokens to the backbone and retrained per backbone, or read logits for single option letters, capping options at 16 or 26. On the same Qwen3.5-4B backbone, training-free LLM-as-Jev scores 81.4% on public JevBench items, versus 80.5% for SemIf and 79.7% for the fine-tuned reflex 4B; on external benchmarks its macro accuracy is 3.2 points above SemIf, with 313 wins against 154 losses.
The method prefills a fixed prefix in the assistant turn so each option corresponds to a numbered suffix ending in a closing bracket, and all suffixes are scored in parallel after one shared prompt prefill. Scoring options one at a time needs a number of forward passes proportional to the option count, whereas numbered suffixes are never prefixes of one another, so the option count is no longer bounded by alphabet size. Training-free accuracy reaches 69.0% on the 77-way Banking77; with digit-level tokenizers such as Qwen's the candidate trie is at most ⌊log10 K⌋+2 deep, so 150-plus candidates need one prefill plus one to three short forward steps.
Perspective
The demonstrated scope is two backbones, Qwen3.5-4B and Qwen3-0.6B, evaluated on the 231 public JevBench items, seven external classification and reasoning datasets, general language tasks, the MMBench and MMStar multimodal sets, and conversational behavior checks on Dolly-15k and MT-Bench; the sealed JevBench split is not included.
The authors treat the JevBench evaluations as exploratory because later data iterations were motivated by error analysis on public JevBench samples, although JevBench was excluded from hyperparameter tuning and checkpoint selection. Training-free 4B latency has a median of 194 ms on a single RTX A5500, and the reference implementation uses FP32 without speed optimization; the authors state that latency optimization is outside the scope of the work.
