Skip to main content
Search

# Vision-Language Model

13 related articles

Subscribe to updates
  1. arXiv

    Traffic lights skip the perception pipeline: Traffic-VL reads raw camera views and cuts average queues 10.4% over three weeks at ten intersections

    Traffic signal control has relied on structured traffic states from dedicated perception pipelines, or on converting simulator images into intermediate representations before deciding. Traffic-VL is an eight-billion-parameter vision-language model that maps synchronized multi-view images and natural-language phase descriptions directly to phase choices, trained with SynTra-V, which turns physics-based simulation into semantically audited photorealistic surveillance imagery. It matches the structured-state Traffic-R1 on five closed-loop benchmarks, reaches 82.1% phase-selection accuracy on real surveillance images, and over three weeks at ten intersections reduced average queue length by 10.4% relative to historical manual operation.

  2. arXiv

    Uni-Loc draws 3D CT lesion maps from scan-level labels, skipping voxel-by-voxel annotation

    Localizing pathology in 3D CT normally requires experts to outline lesions voxel by voxel, which is costly and hard to extend across findings. Uni-Loc replaces the classification head with similarity scores between a pathology-conditioned query and spatial image features, so the same scores yield both a scan-level diagnosis and a 3D localization map, with gradient-guided thresholding outlining lesion regions. Across five localization tasks covering 355 annotated CT scans, it reaches macro-averaged voxel-level AUROC/AUPR of 96.1%/34.5%, against 80.7%/13.9% for relevance propagation and 88.7%/8.1% for gradient saliency.

  3. arXiv

    A CT foundation model built for cancer patients' chest-abdomen-pelvis scans lifts AUC to 0.767 from 0.695

    CAPTOR, a vision-language foundation model pretrained on 25,512 chest-abdomen-pelvis CT-report pairs from cancer-focused exams, outperformed existing CT foundation models in its target setting. Disease classification reached an AUC of 0.767 versus 0.695 for the next-best model on the full chest-abdomen-pelvis subset, and 0.784 versus 0.734 on the oncologic subset. It encodes organ-centered crops of whole exams but makes study-level, not lesion-level, judgments.

  4. arXiv

    MAPE-K dual-AI framework lets a quadruped robot turn natural-language intentions into executable code, robust on finite tasks while open-ended exhaustive search stays cognitively limited

    The work introduces a MAPE-K-inspired dual-AI architecture in which an LLM translates high-level natural-language intentions into executable Python code constrained to a robotic function library, while a VLM provides open-ended semantic grounding through a distillation process, with geometric and semantic thresholds triggering replanning; benchmarked on quadruped exploration tasks in Gazebo across three frontier models, finite tasks show high success rates while open-ended exhaustive tasks degrade, with failures concentrated in semantic grounding, completeness checking, and termination control rather than low-level action execution.

  5. arXiv

    Using a vision-language model's own judgment as the sole reward, an offline language-to-intervention interface for xenobots reaches 80.0% held-out instruction accuracy

    Treating an existing archive of interventions and their already-observed outcomes as a fixed offline dataset, the work uses a vision-language model to judge whether an archived outcome matches a natural-language description and uses that judgment as the sole training reward to learn a language-to-intervention mapping for a xenobot, a synthetic multicellular construct with no nervous system; the mapping generalizes to entirely new instructions, reaching 80.0% held-out accuracy on archive data withheld from training versus a 66.7% chance baseline and matching a network trained directly on ground-truth labels.

  6. arXiv

    STAVE replaces in-context demos with two task vectors, beating per-layer interventions on six multimodal models at zero-shot inference cost

    The work proposes STAVE, which replaces in-context demonstrations with two task-specific vectors added to the input embeddings of a frozen LLM or LMM, a readout vector updating the answer-producing tokens and a context vector updating the other structural token groups, both trained with answer labels on prompts with and without demos; across six LMMs and five LLMs, STAVE matches or outperforms per-layer interventions with far fewer task parameters, surpasses 15-shot ICL and prior task vectors on 18 text tasks, and runs at zero-shot inference cost.

  7. arXiv

    Foresight lets a frozen Qwen3-VL-8B plan its own future perception in streaming video, reaching 23.0 joint F1 on OmniPro Online and beating the strongest trained baseline by 9.5%

    The authors introduce Foresight, a training-free dual-stream architecture with two Siamese LLMs sharing weights, input encoders, and a KV cache, where one copy continuously ingests the stream while the other runs ahead over the same causal state to write a plan deciding when to reason next, what to check then, and how densely to sample; with a frozen Qwen3-VL-8B backbone it reaches 23.0 mean joint F1 on OmniPro Online (versus 13.5 for the strongest trained baseline, MiniCPM-o 4.5), improves the backbone by 6.7 points on StreamingBench with a best overall score, and by 15.4 points on OVO-Bench, with the largest gain of 18.7 points on Forward Active Responding where evidence arrives later in the stream.

  8. arXiv

    CRAFT fills spreadsheet forms with template-aware reflection and local repair, raising pair accuracy by 8.51 and 23.38 points on FormFillBench

    The authors propose CRAFT, a template-aware agent framework for spreadsheet form filling that couples reflective validation with constrained local repair: it localizes suspicious regions, restores overwritten template content when needed, re-grounds plausible writable slots with a Rectangle-Aware Slot Grounder (RASG), and constrains later edits via label-slot hints and protected regions; they also release FormFillBench with 327 forms across Instruction-Only and Multi-File tracks, where CRAFT improves pair accuracy over the strongest baselines by 8.51 and 23.38 percentage points respectively, with component-removal experiments supporting structural adjudication and slot re-grounding and a second backbone retaining the relative advantage.

  9. arXiv

    CrashSim generates crash scenarios from real-world crash priors, builds the 4,000+ scenario nuCrash dataset, and widens measured safety gaps across five planners

    The authors present CrashSim, a human behavior-informed crash scenario generation framework that uses a vision-language model to interpret naturalistic driving scenes, hierarchically retrieves real-world crash priors, and converts their relative-motion patterns and behavior-stage timings into anchor guidance for a diffusion generator pretrained on nuScenes; results show CrashSim reproduces real-world pre-impact behavior, collision dynamics, collision geometry and crash-type distributions more closely than Strive and LD-scene, and it is used to build nuCrash, a dataset of over 4,000 crash and near-crash scenarios on which five planners show larger closed-loop performance differences than on nuScenes, with an LLM evaluation agent providing capability-level diagnoses and improvement guidance.

  10. arXiv

    Dream4ACT unifies multi-embodiment video-action modeling with fixed-view action views, reaching 88.98% average success on RoboTwin 2.0

    Dream4ACT renders target joint configurations as "action views" under four fixed virtual cameras, jointly models physical-camera observations and action views with a shared video autoencoder and diffusion transformer under masked flow matching, and recovers executable joint targets by training-free, URDF-constrained multiview matching; it reaches 90.50% (clean) and 87.46% (randomized) success on RoboTwin 2.0, averaging 88.98%, a TriWorldBench score of 65.66, and closed-loop evaluation across five simulated embodiments and four real-world platforms.