An 8B model learns to click precisely in professional software: GroundGUI pairs 586K screenshots with two-stage training to match closed-data models
Synopsis
GUI agents must turn a sentence into a specific pixel on screen, and high-quality cross-platform annotations have been scarce. GroundGUI releases both the ScreenRef dataset, with 586K screenshots and 633K referring expressions across desktop, web and mobile, and a two-stage training method, D2RL: a teacher that sees the ground-truth box distills token by token on the student's own rollouts, then RLOO reinforcement learning optimizes the click reward. Built on Qwen3-VL-8B-Instruct, GroundGUI-8B reaches 95.2% on ScreenSpot-V2 and 63.9% on the professional high-resolution ScreenSpot-Pro benchmark.
Interpretation
GUI grounding models now have a public cross-platform training resource: ScreenRef holds 586K desktop, web and mobile screenshots paired with 633K referring expressions, spanning resolutions from 0.1 to 8.3M pixels. Strong GUI agents have mostly relied on proprietary large-scale data that is not released, while open datasets such as GroundCUA and ScaleCUA are limited by diversity or annotation accuracy, and models trained on them perform suboptimally. With supervised fine-tuning alone on the same Qwen3-VL-8B base, the ScreenRef-trained model averages 76.4 across four benchmarks, above ScaleCUA's 72.9 and GroundCUA's 73.7, and leads them by 3.2 and 5.3 points on OSWorld-G.
The training pipeline makes the starting point for reinforcement learning a separate stage: distill first, then reinforce, so hard samples still carry a learning signal. Supervised fine-tuning tends to overfit specific domains, while reinforcement learning such as GRPO suffers sparse signals on hard samples such as small icons or spatial-relation queries, where all rollouts receive zero reward. Starting from the same SFT checkpoint and the same post-training data, full D2RL averages 69.9 across five benchmarks, above SFT-only at 68.0, SFT plus OPSD at 69.1 and SFT plus RLOO at 68.9.
The final model, GroundGUI-8B, reports comparable scores on five GUI grounding benchmarks and surpasses several 32B-scale models within the 8B class. Against the Qwen3-VL-8B base it gains 3.5 points on ScreenSpot-V2, 5.5 on MMBench-GUI L2, 14.0 on ScreenSpot-Pro, 10.6 on OSWorld-G and 20.8 on UI-Vision. GroundGUI-8B scores 95.2% on ScreenSpot-V2, 63.9% on ScreenSpot-Pro, 65.4% on OSWorld-G, 86.8% on MMBench-GUI L2 and 38.3% on UI-Vision, and beats Step-GUI-8B by 1.3 points on ScreenSpot-Pro.
Perspective
The demonstrated scope is GUI element grounding on desktop, web and mobile screenshots, evaluated on ScreenSpot-V2, ScreenSpot-Pro, OSWorld-G, MMBench-GUI L2 and UI-Vision; D2RL post-training uses 12K grounding samples derived from GroundCUA, excluding screenshots identical to those in UI-Vision. ScreenRef is released under a gated, non-commercial research-only license that prohibits redistribution.
The dataset is built from live interface screenshots and public tutorial videos, so frames may incidentally contain personal or account information, and a systematic human privacy audit has not yet been performed. The authors also state that prompts, filtering rules, per-stage filtering statistics and the full training configuration, including the position-aware weighting strength, will be added to the code repository. Refusal samples make up 3.5% of ScreenRef, peaking at 5.2% on desktop.
