Skip to main content
Back to timeline
arXivSource publication:

Unified multimodal models now draw the concept behind an answer: no decoder training, pronoun visualization rises from 2–11% to 86–94%

Synopsis

Unified multimodal models combine understanding and generation in one architecture, so the image generator can read hidden states formed while processing text. V-lens centers and rescales a chosen token's hidden state, inserts it into a drawing request, and lets the model's own generator draw it without training a decoder. On BAGEL, SenseNova-U1 and HunyuanImage-3.0, lens tuning with purely visual supervision raises correct visualization of pronoun referents from 2–11% to 86–94%, and improves text answers on supervised stories in Show-o2 and HunyuanImage-3.0.

Source-provided article image: The Elephant in the Question: Unified Models Picture the Concepts Behind Their Answers
arXiv

Interpretation

When a unified multimodal model answers a question, it first forms concepts the question never names and then shifts toward the output token; V-lens makes those intermediate concepts visible as images without training a separate decoder. Earlier routes to interpretable intermediate states were mainly textual, reading a hidden state through the vocabulary or into natural-language descriptions; V-lens instead uses the model's own image generator as the readout channel. Across six unified models, the same token yields different images in different contexts; on the two-meaning test BAGEL scores 12.5/20 and Show-o2, SenseNova-U1 and HunyuanImage-3.0 each score 15/20, all far above the label-swap chance estimate.

The procedure carries a token's hidden state from one layer to an insertion layer, then lets it replace thing in a drawing request and has the model draw with its usual sampler. The transport subtracts the source layer's mean state, rescales each coordinate by its spread at the insertion layer, and places the result at the usual radius of insertion-layer states; these statistics come from plain-text passages unrelated to the test items, so nothing is fitted on pictures. On a prespecified validation set this coordinate scaling beats unchanged copies, final normalization and Jacobian maps, and stays best on fresh test items; removing centering or the scaling lowers recovery on the two-hop questions.

V-lens passes gradients, so images can directly supervise hidden states, a procedure called lens tuning. Untrained, BAGEL, SenseNova-U1 and HunyuanImage-3.0 visualize both referents in only 0/20, 4/20 and 0/20 pronoun text pairs respectively. Across 120 unseen test pairs and three training seeds, lens tuning raises pair-level success from 0.02 to 0.86 in BAGEL, 0.11 to 0.94 in SenseNova-U1 and 0.06 to 0.86 in HunyuanImage-3.0.

Purely visual supervision also changes text answers: in Show-o2 and HunyuanImage-3.0, training with pictures of the correct state raises two-option answer accuracy on supervised stories. Untrained Show-o2 answers 36.3% of stories correctly, and training on pictures of the incorrect state leaves it at 34.6%, so the gain is not from image training in general. With correct-state pictures Show-o2 rises to 49.6% while its correct visualizations rise from 52.5% to 73.3%; HunyuanImage-3.0 rises from 40.0% to 50.8%, but its incorrect-state training also reaches 47.5%.

Perspective

The demonstrated scope is objects, colors and visible states that the generator can draw, tested on 16 two-hop questions, 30 description pairs, 20 two-meaning words, 20 pronoun pairs, 16 two-state pairs, 40 state scenarios and 28 interchange scenarios; each reading requires generating a full image, taking 9–110 seconds in this setup.

A missing picture does not establish that a hidden state lacks the concept; recovery of unnamed objects is modest and not significantly above chance in BAGEL or SenseNova-U1; lens tuning trains and evaluates through the same lens, and the state interchange transfers whole hidden states without isolating the pictured content; general-capability checks use one training seed and their intervals still allow losses of several percentage points.