Skip to main content
Back to timeline
arXivSource publication:

Text embeddings get a question hierarchy: deep yes/no questions collapse into shallow ones, cutting human checking without losing task performance

Synopsis

QA-based text embeddings map each dimension to a yes/no question, which makes them readable but forces people to inspect a large number of questions. The new method arranges those questions in a hierarchy, with deeper levels more specific and more numerous, and aligns deep questions to shallow ones through entailment so an interpreter reads only a few general questions. On semantic similarity, text clustering and information retrieval, this replacement lowers the human checking load while holding task performance, shifting the Pareto front between performance and cognitive load.

Source-provided article image: Interpretable Text Embeddings Based on Hierarchical Questions
arXiv

Interpretation

Each dimension of a text embedding can now sit on a question tree, so interpreting a comparison means reading a few shallow summary questions rather than every yes/no item. Earlier QA-based embeddings gave every dimension a readable question, but the sheer number of questions meant a person had to examine many of them to explain a similarity or difference. On semantic similarity, text clustering and information retrieval, replacing deep questions with shallow ones moved the Pareto front between task performance and cognitive load clearly beyond the single-level CQG-MBQA baseline.

The hierarchy is built in three steps, recursive clustering, contrastive question generation and question alignment, with deeper levels more specific and roughly five times more numerous per level. Contrastive prompting has a large language model write yes/no questions that separate positive from negative samples, while recursive clustering lets shallow questions cover broader semantic regions. On the roughly 6.8 million-text MEDI2 corpus, level one yielded 40 clusters and 213 questions, and level four yielded 4,996 clusters and 13,914 questions, against 5,000 clusters and 16,419 questions for the single-step baseline.

Whether an alignment holds is checked with a natural language inference model: questions are rewritten as statements, and the deep statement is tested for entailing the shallow one. Alignment pairs questions by entailment probability rather than by surface similarity. Across many level pairs, more than 67% of lower-level questions entailed their aligned upper-level questions, while the reverse entailment was rare, indicating that shallow questions summarize deep ones.

Perspective

The demonstrated scope is English general-purpose tasks: semantic similarity, text clustering and information retrieval over MTEB datasets, with questions generated by gpt-4o-mini, UAE-large-v1 as the black-box embedder and mDeBERTa-base-v3 as the natural language inference model. A reader weighing transparent retrieval in high-stakes settings can use this to judge the level trade-off, but multilingual settings were not evaluated.

The number of generated questions remains large, so manually inspecting all of them is still difficult, and the qualitative analysis covers only sampled questions; a more comprehensive human evaluation is still needed. The method improves the trade-off between performance and cognitive load rather than directly raising task performance. When the threshold is too low, the gap between the model's and the human's standards, such as a person reading a model's yes as no, may widen and reduce reliability. Whether question quality and granularity hold across languages remains unclear.