Text-to-image reward models get a new pairing axis: same image, different prompts lifts GenEval by 4.0% in best-of-n and 5.5% under RL
Synopsis
Text-to-image reward models must judge prompt adherence and human aesthetics at once, yet widely used models sit near chance on compositional, spatial and counting reasoning. A Stanford team converts visual-reasoning data into preference pairs that share one image but differ in prompt, mixes them with conventional human-preference data, and improves visual understanding and preference prediction together. On GenEval's compositional prompts, the resulting reward model raises generation by an average of 4.0% in inference-time best-of-n selection and 5.5% in DiffusionNFT reinforcement-learning post-training.
Interpretation
Reward-model training data no longer consists only of two images under one prompt; it now includes pairs where a single image is paired with one correct and one incorrect description, so the model learns directly whether image and text correspond. Conventional preference data entangles prompt adherence with aesthetics, because the two images often differ at once in style, artifacts and semantic correctness, letting a model score well by exploiting aesthetic shortcuts. On ImageRewardDB, fine-tuning Qwen3-2B on conventional preference pairs alone reaches 66.3% preference accuracy but drops average visual understanding from the base model's 85.4% to 78.4%; adding shared-image pairs keeps preference accuracy at 66.3% and restores average visual understanding to 85.3%.
The construction rewrites existing visual question-answering and multiple-choice data into preference pairs: the image is held fixed, the correct description is preferred and a perturbed description is dispreferred, and these are mixed with human preference data for training. Such shared-image pairs are already valid Bradley-Terry comparisons, differing along the text axis rather than the image axis, so they use the same loss and regression readout head with no auxiliary objective. The final mixture holds 157,209 pairs, 76.2% of them from ImageRewardDB and the rest from CLEVR, SpatialSense, TallyQA, ColorSwap and VSR; training runs three epochs on the last 9 layers of Qwen3-2B plus a linear scoring head.
Stronger visual understanding carries over to better generation guidance, and GenEval prompts that require precise rendering of objects, attributes, spatial relations and counts benefit consistently in both best-of-n selection and RL post-training. On SD-Turbo at n=50 the GenEval accuracy is 64.5% versus 62.4% for the next-best method; on FLUX.1-Schnell it is 78.1% versus 75.7%. When DiffusionNFT post-trains Stable Diffusion 3.5 Medium and SANA, the reward model raises GenEval by an average of 5.5%, while the best-of-n setting averages 4.0%.
Perspective
The findings rest on two vision-language backbones, Qwen3-2B and SmolVLM2-2.2B, on ImageRewardDB preference data and a fixed mixture of the listed visual-reasoning sources, evaluated on GenEval, T2I-CompBench and the MJ-Bench alignment split, and used both for best-of-n selection and DiffusionNFT post-training.
How the shared-image-to-preference ratio should scale with preference dataset size or backbone capacity is not characterized, and the current mix reflects natural source sizes rather than a tuned recipe; whether the supervision transfers to generative or pairwise reward models is also untested.
