Skip to main content
Back to timeline
arXivSource publication:

A reasoning model gains a no-token switch: HDM reads one forward pass's hidden state to decide whether to write a chain of thought

Synopsis

The hybrid decision model HDM puts token-free answering and full autoregressive reasoning inside one frozen backbone, with a lightweight self-routing block choosing between them before any token is generated. The block estimates the value of thinking, the accuracy gain the slow mode is expected to add over the fast one, and escalates only when that gain exceeds a price set per request. On MiniWoB, HDM reaches a reward of 0.469 against always-thinking's 0.519, cutting thinking tokens by 80% and wall-clock by 67%.

Source-provided article image: The Value of Thinking: Learning When to Reason in Hybrid Decision Models
arXiv

Interpretation

One frozen backbone can now answer a question either directly from a single forward pass that generates no token, or by writing out a full reasoning chain before answering. Earlier adaptive-reasoning methods only shorten the written reasoning or switch between modes, and the answer is still generated token by token; routing methods choose among several models, and every branch generates an answer. On AGIEval, HDM on the Qwen3.8-27B backbone gives up 0.9 accuracy points of always-thinking accuracy at one price for 30% fewer tokens; on MMLU-Pro, the same backbone reaches 0.890 accuracy at the 22%-fewer-tokens setting, against 0.902 for always thinking.

The routing signal comes from hidden states the fast mode has already computed, so the decision itself costs no extra forward pass and no token. The self-routing block reads the backbone's hidden state at the readout position of one layer plus eight distribution statistics, and returns a distribution over the four joint outcomes of the two modes, estimating both modes' utility at once. An ablation shows the hidden state carries the main signal: the layer-24 raw configuration reaches a cross-validation routing AUROC of 0.778 against 0.671 for a block reading only the output distribution, and 0.762 against 0.701 on the three held-out subject categories.

In episodic tasks without per-decision labels, CDPO trains the same routing block from the return of a whole trajectory. Single-turn questions have gold answers, so running both modes constructs four-way labels for supervision; multi-turn agentic tasks decide success through successive actions and lack per-decision labels. On ALFWorld, HDM reaches a reward of 0.560 at one price, between the fast-only 0.463 and the slow-only 0.642, while spending 52% of the thinking tokens; on MiniWoB it reaches 0.469 against always-thinking's 0.519, cutting thinking tokens by 80% and wall-clock by 67%.

Perspective

The demonstrated scope is two frozen backbones, Qwen3.5-4B and Qwen3.8-27B, with greedy decoding for the slow mode; under the thinking budget cap some Qwen3.5-4B questions and a few Qwen3.8-27B questions are truncated, so the corresponding token counts and accuracies are lower bounds. Training draws on eleven MMLU-Pro subject categories, with chemistry, economics and psychology held out as the test set, and AGIEval and GPQA-Diamond used to measure further transfer.

The authors state that what they claim against published systems is competitiveness at lower token cost rather than dominance: on BANKING77 two compared configurations answer more items correctly than HDM, and latencies quoted from other work were measured on other backends. CDPO optimises a surrogate rather than the episode-level objective itself, the per-decision importance ratio introduces bias, and credit assignment is the weakest assumption in the recipe. Reading the final layer for routing is chosen for generality, while the ablation shows layer 24 is better on a single seed.