Multimodal agents learn to call new specialist tools: COTEX adds only lightweight adapters and barely disturbs old tool choices
Synopsis
Multimodal agents mostly rely on generic tools such as cropping, search and code execution, leaving specialist vision models outside their tool space. COTEX confines each new tool's adapter to a shared tool-call subspace and keeps its update directions orthogonal to those governing existing tool selection, so old tools survive the addition. Across two agents, DeepEyesV2 and Metis, and three specialist tools for OCR, object detection and depth estimation, the abstract reports gains on nine vision-language benchmarks with at most a 0.3-point drop on existing tools, while the introduction's contribution list says ten benchmarks, so the two counts disagree.
Interpretation
A multimodal agent can take on new specialist vision tools after training, without rerunning full training and without losing its use of existing tools. The two prior routes each carry a cost: full training grows with every added tool, while an adapter trained only on the new tool overwrites the directions the agent relies on for existing tools. After sequentially adding OCR, object detection and depth estimation to DeepEyesV2, V*Bench holds at 80.6, versus 75.9, 78.0 and 74.3 for naive fine-tuning, O-LoRA and KeepLoRA.
Each new tool is written as a low-rank adapter that occupies only the directions for when to call a tool, not the directions for which tool to call. COTEX applies singular value decomposition to the weight displacement between the agent and its base multimodal model, ranks the resulting directions by gradient sensitivity on a calibration set, and forms the shared tool-call subspace from directions sensitive to call initiation. Within that subspace, directions sensitive to existing tool selection are marked protected and the new adapter is kept orthogonal to them; training trajectories come from ICDAR 2015, Ref-Adv-s and UniQA-3D annotations, verified by rules to yield roughly one thousand trajectories per tool.
Trajectories for a new tool need no teacher model: the agent attempts examples from the specialist's native dataset and a task verifier keeps the correct attempts. A specialist model's own annotations say what a correct output is, but not how an agent should invoke the tool and use its output to answer. The verifier retains only trajectories whose final prediction matches the original annotation, and these trajectories then feed supervised fine-tuning followed by reinforcement learning.
Perspective
The demonstrated setting is two post-trained multimodal agents, DeepEyesV2 and Metis, sequentially extended with three frozen vision specialist tools for OCR, object detection and depth estimation, evaluated on benchmarks for visual perception, text understanding, spatial reasoning and relative depth.
Detection accuracy falls after the depth tool is added; in the reinforcement-learning stage depth accuracy drops from 93.5 to 74.0 even though the depth tool's invocation rate stays at 100.0%. The abstract and the introduction disagree on the benchmark count, with the abstract saying nine vision-language benchmarks and the introduction's contribution list saying ten.
