Skip to main content
Back to timeline
arXivSource publication:

Nunchi reads the room in multi-party conversations: writing speakers and transcripts into the prompt lifts attended-person accuracy to 86.7%

Synopsis

In multi-party conversations, gaze geometry alone becomes ambiguous about whom an observer is looking at as more people appear. Nunchi writes the gaze heatmap, speaker scores and transcript into one text prompt, letting a vision-language model weigh these cues and score every candidate in a single forward pass. On the new SORBET benchmark it reaches 86.7% identity accuracy, 4.8 percentage points above the strongest existing method fine-tuned on SORBET, and 76.0% with five or more candidates, 8.0 points above the next best method.

Source-provided article image: Nunchi: Ears Help Eyes in Multi-Party Conversations
arXiv

Interpretation

Deciding whom an observer looks at in a multi-party conversation can now draw on the conversation itself, not only on the geometry linking gaze direction to people's positions. Earlier gaze-following and social-gaze methods relied on gaze geometry alone, and with more people several candidates often lie in a similar direction from the observer, so a small direction error changes the selected person; on VAT, Gaze-LLE's accuracy drops from 83.9% with two other people to 64.3% with five or more. On the SORBET test set Nunchi reaches 86.7%, against 81.9% for Gaze-LLE with its decoder fine-tuned on SORBET, a gap of 4.8 points; with five or more candidates Nunchi reaches 76.0%, 8.0 points above Qwen3-VL given the conversational cues but no gaze prior.

The method writes every cue as text in one prompt and lets a vision-language model score each candidate in a single forward pass. Gaze geometry comes as a heatmap from a frozen Gaze-LLE, the speaker cue as scores from the frozen active speaker detector LR-ASD, and the transcript from the source datasets; all three are tied to candidates by shared labels drawn on the image and passed to Qwen3-VL-8B as separate prompt paragraphs, with LoRA fine-tuning applied only to the language model. On SORBET the speaker cue raises accuracy from 83.3% to 85.2% and adding the transcript raises it to 86.7%; the same speaker scores give MTGS 0.8 points, whereas Nunchi gains 1.9.

The gain comes from conversational cues rather than from adding more cues in general. Adding any of four visual cues to the base input, namely depth, reciprocal gaze, a gaze graph and an arrow, fails to improve accuracy, and reciprocal gaze lowers it from 83.3% to 81.1%. Conversational cues help across three vision-language models, adding 5.0 to 6.7 points alone and 10.5 to 14.3 points together with the gaze prior; replacing human transcripts with Whisper transcripts costs only 0.3 points on SORBET.

Perspective

The demonstrated scope is the SORBET benchmark drawn from three TV series plus cross-dataset evaluation on VGS and AVA-LAEO; each sample uses a single keyframe and assumes the observer looks at one of the candidates.

SORBET comes from scripted TV series, where acting and camera framing follow a script; gaze shifts and gaze at objects or off-screen people are not covered; mapping names in the transcript to faces requires the character identities of the people in the scene; on VGS the base input reaches 48.9%, below zero-shot Gaze-LLE at 51.5%, so a model trained on SORBET does not always transfer.