Neos wires retrieved image evidence into search decisions, lifting its 9B and 35B models by 17.4 and 15.5 points on average across ten benchmarks
Synopsis
Multimodal deep-search agents often treat retrieved images as one-off inputs and struggle to let intermediate visual evidence steer later queries. Neos embeds a trainable Multimodal Evidence Interface in its search tools, uses reference-guided credit assignment to separate critical errors from their downstream consequences, and alternates policy reinforcement learning with interface fine-tuning. Across ten benchmarks, Neos-9B and Neos-35B beat their respective base workflows by 17.4 and 15.5 points on average.
Interpretation
Multimodal search agents begin to treat retrieved image evidence as an intermediate variable that shapes the next query, not just material for initial recognition or final attribute extraction. Earlier training and evaluation tasks often did not require newly acquired visual evidence to influence later search, and captions, metadata or indexed text could bypass intermediate visual reasoning, so agents could succeed without internalizing that loop. Neos trains on tasks with intermediate visual dependencies and compares against three baseline groups across ten benchmarks: direct answering, agentic workflows over the same tool framework, and agents trained specifically for multimodal search.
Retrieved results are first filtered and summarized by a lightweight vision-language model conditioned on the current search goal, then handed to the search policy as ordinary tool observations. Retrieved content was commonly converted into observations by fixed, large or proprietary auxiliary models, which added repeated inference cost and could not adapt to the policy's evolving evidence needs. The Multimodal Evidence Interface sits inside web visit, image search and reverse image search, returns structured text and image references, and leaves query formulation and on-demand image inspection under policy control.
Training signal is assigned per turn, so genuinely useful evidence gathering or correction inside a failed search still receives positive credit. Supervision from final-answer correctness alone is too global to identify critical errors or distinguish them from downstream consequences. A credit judge assesses failed trajectories against verified successful references, labeling critical errors, their downstream consequences and productive or corrective decisions, while unannotated later turns inherit the severity and confidence of an active error.
Perspective
The demonstrated scope is multimodal deep-search evaluation on ten benchmarks spanning factual visual question answering, multi-step information search and reasoning over online visual evidence; evaluation allows up to 50 interaction turns per question, and the vision encoder and multimodal merger stay frozen during training.
Training of the evidence interface draws on calls selected from online interactions, and failed trajectories contribute only calls labeled productive or corrective; in credit assignment, unannotated turns inherit the severity and confidence of an active error, a propagation rule that is itself a design choice.
