Reasoning segmentation stops scoring whole trajectories: HISD assigns token-level credit from target hints and cuts reasoning spans by over 80%
Synopsis
Reasoning segmentation asks a model to pick the object a query refers to among visually similar ones, but a single reward shared by the whole response cannot say which tokens supply the discriminating evidence. Hint-Informed Self-Distillation (HISD) has an external model generate structured Segment Hints offline, then uses a hint-conditioned teacher to score the student's own trajectories token by token and rescales only the magnitudes of positive advantages. Under a unified training protocol, HISD raises ReasonSeg test gIoU and cIoU by 3.00 and 3.55 percentage points over a reproduction of DPAD, and shortens valid reasoning spans by 80.17–91.83% on average across five evaluated splits.
Interpretation
The training signal for reasoning segmentation moves from one score per response to per-token update magnitudes shaped by target-disambiguation evidence. DPAD previously built a binary reward by contrasting an anchored description against the target region and the full image, while credit still landed on the whole response; self-distillation methods such as OPSD instead supply teacher distributions along student trajectories. Under a unified protocol with the same Qwen2.5-VL-7B-Instruct backbone, the same 3,000 RefCOCOg training examples and one epoch, HISD improves ReasonSeg test gIoU and cIoU by 3.00 and 3.55 percentage points over the reproduced DPAD.
The teacher sees target hints during training while the student sees only the image and query, and inference needs neither hints nor a teacher. Segment Hints are generated offline by an external model from training images, queries and annotated boxes, organizing target identity, discriminative attributes, confusable objects and an anchor-to-target path, generated once per sample as fixed teacher context. The teacher is an exponential moving average of the student; a teacher-to-student evidence ratio is computed over a support set of the teacher's top-k tokens plus the sampled token, symmetrically clipped, and applied only to positive advantages; across the five splits of ReasonSeg test, validation and the RefCOCO family, HISD averages 5.66–15.34 reasoning tokens, below SegZero and DPAD.
Perspective
The demonstrated setting is Qwen2.5-VL-7B-Instruct as the trainable policy, frozen SAM2-Large as the segmentation module, 3,000 RefCOCOg samples for one epoch, and evaluation on five splits of ReasonSeg and the RefCOCO family; the authors state that the independent contribution of individual hint fields and generalization to other generators, model sizes and data domains remain to be evaluated.
The authors note that the independent contribution of individual hint fields has not been evaluated and that generalization to other hint generators, model sizes and data domains remains to be tested; gains on the RefCOCO family are smaller than on ReasonSeg, which the authors attribute to short, direct referring expressions placing lower demands on target disambiguation.
