Skip to main content
Back to timeline
arXivSource publication:

Alibaba's PURE splits identity matching into 14 diagnostics: ten multimodal models span 47.1% to 86.0%, with rule judgment weakest

Synopsis

Alibaba researchers released PURE, a diagnostic benchmark for multi-granularity multimodal identity matching, with 1,457 multiple-choice questions across 14 tasks in four dimensions: Perception, Understanding, Reasoning and Judgment. Questions come from real e-commerce record pairs and were verified by two domain experts. Across ten recent multimodal large language models, overall scores range from 47.1% to 86.0%, with Understanding highest on average and Judgment lowest, showing that aggregate matching accuracy hides whether models struggle to interpret evidence or to apply matching criteria.

Source-provided article image: PURE: A Diagnostic Benchmark for Multi-Granularity Multimodal Identity Matching
Figure 1 · arXiv · Page 2

Interpretation

The answer to an identity match depends on the target granularity: one pair of product records can match at the base level yet differ at the variant level, and a single final matching accuracy cannot show which step a model gets wrong. Earlier multimodal benchmarks either assess visual comparison alone or treat identity matching as one of several e-commerce tasks, without evaluating the supporting capabilities separately under two nested base-level and variant-level criteria. PURE divides matching into 14 tasks across Perception, Understanding, Reasoning and Judgment, draws its questions from real e-commerce record pairs, and yields overall scores from 47.1% to 86.0% across ten multimodal models.

Models are comparatively reliable at extracting and interpreting attributes, and clearly weaker at localizing visual differences and applying variant-level rules. Understanding is the strongest dimension for every evaluated model and Judgment the weakest for most; textual core attribute extraction and attribute equivalence score consistently high, while visual difference localization and variant-level rule judgment are less reliable. Across the ten models' dimension scores, Understanding has the highest mean and Judgment the lowest; the Qwen3.5 series from 9B to 397B-A17B spans 47.1% to 59.2% overall, and the gains from scale appear more in Perception and Understanding than in Reasoning and Judgment.

Multimodal evidence helps overall, but textual evidence can also lead models to treat non-discriminative wording differences as identity conflicts. On cross-product attribute conflict detection, removing images lowers mean accuracy by 8.8 percentage points while removing titles raises it by 2.7 points, with six of the ten models showing the increase. On base-level rule judgment, removing images lowers mean accuracy by 17.5 points and removing titles raises it by 8.2 points, a pattern holding for all ten models; grouped by correct answer, removing titles lifts mean accuracy from 48.3% to 71.7% when the answer is no applicable rule.

In direct matching, models more reliably recognize matching pairs at the base level, and the reverse holds at the variant level. The ten models' recall bias points in opposite directions at the two levels, with the mean recall gap moving from +17.2 percentage points at the base level to −15.4 points at the variant level. The direct-matching set contains 330 base-level pairs and 211 variant-level pairs with balanced classes; PURE overall scores show a Spearman rank correlation of 0.952 with direct-matching balanced accuracy.

Perspective

PURE evaluates only explicit responses from generative multimodal large language models and does not directly characterize embedding-based retrieval or end-to-end matching systems; its questions are instantiated from product records in one e-commerce setting, including images, titles and attribute descriptions of the selected variant.

The source pool is drawn from difficult cases that both an embedding-based matcher and a generative multimodal model misclassified, so it may inherit the coverage biases of those two mechanisms; generalization across platforms, languages, product categories and non-commerce identity domains remains to be established.