Skip to main content
Back to timeline
arXivSource publication:

Fashion retrieval evaluation stops judging pairs by hand: FASHION-RQ builds queries and graded relevance from catalog hierarchy codes

Synopsis

Open-ended fashion queries often match many valid items, while manual graded set-valued labels are costly and tied to one catalog snapshot. FASHION-RQ treats the semantic-ID prefixes learned by an RQ-VAE as an automatic annotation structure: a prefix defines a relevance set and shared prefix length gives graded relevance, and a language model then verbalizes cluster attributes into shopper-style queries, yielding FASHION-RQBENCH with 588 clusters and 5,380 queries on the public Myntra catalog. Off-the-shelf retrieval models expose differences that aggregate ranking scores hide: Qwen3-Embedding-4B leads ranking, relevance classification and paraphrase robustness, while SPLADE++ leads sibling discrimination.

Source-provided article image: FASHION-RQ: Mining Hierarchical Semantic Structure for Automated Evaluation of Fashion Retrieval
arXiv

Interpretation

Evaluation labels for fashion retrieval can now be read directly from a catalog's hierarchical codes, without human judgments of query-product relevance. Earlier fashion retrieval datasets typically pair a query with a single target item and penalize other valid results; graded set-valued labels such as ESCI are feasible but cost 2.6M judgments for 130K queries, and manual labels are tied to a particular catalog snapshot and query distribution. On the public Myntra Fashion Product Images catalog the pipeline yields 588 benchmark clusters (17, 135 and 436 at the three levels) and 5,380 attribute-enriched queries, 5,305 of them unique strings, with relevance inherited deterministically from the semantic-ID structure rather than scored by humans or a language model.

The hierarchy comes from semantic IDs learned by a residual-quantized autoencoder: each product becomes a sequence of discrete codes from coarse to fine, and products sharing a prefix form a semantic cluster. A vanilla RQ-VAE does not guarantee that one code corresponds to a single semantic attribute and can produce heterogeneous clusters, so an attribute exclusivity loss is added to make each quantization level specialize to a coherent attribute. Exclusivity supervision uses masterCategory at level 1, articleType at level 2, and articleType-anchored joint attributes of color, gender, usage, season, age group, price tier and fashion type at level 3; clusters are then filtered by articleType purity, and each surviving cluster is compared with its parent and siblings, keeping attribute differences of at least 20 percentage points as exclusivity constraints.

Queries are written in shopper style by a language model from a cluster's attribute distribution and its differences from parent and sibling clusters, but the model only verbalizes and does not decide relevance. Earlier synthetic query methods are mostly document-conditioned, generating from one document and inheriting a single positive target; here the condition is the whole cluster, which is itself a multi-item relevance set. In a study with 32 participants, FASHION-RQBENCH queries scored 3.62 on naturalness and 3.91 on intent clarity, versus 3.52 and 2.83 for real user queries and 2.84 and 3.76 for unconditioned language-model queries; pairwise differences on intent clarity were all statistically significant, while naturalness was comparable to real user queries.

The benchmark splits evaluation into ranking, hierarchical relevance classification, sibling discrimination and paraphrase robustness, and adds an attribute-weighted query relevance score plus per-attribute scores. A single aggregate ranking score hides differences: two systems can reach similar nDCG while one retrieves the right garment in the wrong color and the other the right color for the wrong garment. Among off-the-shelf retrieval models evaluated without fine-tuning, Qwen3-Embedding-4B reaches nDCG@10 of 0.8719 and Recall@10 of 0.8279 on ranking, 0.5145 accuracy and 0.5146 macro-F1 on four-way relevance classification, and 0.8373 consistency@10 under paraphrase; SPLADE++ leads sibling discrimination with nDCG@3 of 0.6662; attribute scores are high for product type and gender and markedly lower for material and style.

Perspective

The demonstrated scope is construction and evaluation on the public Myntra Fashion Product Images catalog: 588 clusters and 5,380 queries across four tasks, with off-the-shelf retrieval models evaluated without fine-tuning on the benchmark. The authors list human validation, broader domains and attributes, richer semantic relations and adaptive benchmark generation as future work.

The authors state that the method depends on catalog quality, the underlying multimodal representation and language-model-generated queries, which may introduce coverage and linguistic biases, and that the automatically derived relevance structure approximates rather than replaces human judgments. A codebook-size ablation in the appendix also shows clusters fragmenting as the codebook grows from 64 to 256, with average coarsest-level cluster size falling from 823.9 to 166.5 items, so a trade-off between semantic purity and cluster density is required. The main text and appendix disagree on the number of retrieval models evaluated, describing ten off-the-shelf models in one place and 12 baselines in another.