Skip to main content
Back to timeline
arXivSource publication:

Similarity needs more than one ruler: BASIS infers which criterion fits the instance from weak feedback, lifting six-probe concept recovery from 56.0% to 68.1%

Synopsis

Language models repeatedly compare similarity at test time to pick exemplars, candidates and revisions, yet lexical, semantic, structural and stylistic resemblance can point to different candidates, and the task-aligned notion can shift from instance to instance. BASIS places the choice of candidate and the choice of criterion in one Bayesian posterior, updates it from scalar scores, pairwise preferences or verifier outputs, and picks each next probe by expected information gain. On concept recovery, six probes give 68.1% Success@1 against 56.0% for a tuned static mixture; reasoning accuracy is 73.1% versus 72.3% and refinement win rate 63.8% versus 61.4%, with the largest gains under criterion mismatch and larger candidate pools.

Source-provided article image: Bayesian Test-Time Inference of Task-Aligned Similarity from Weak Interactive Feedback
arXiv

Interpretation

Test-time candidate selection can now infer both which candidate is useful and which similarity criterion governs the current instance, instead of committing in advance to one scoring rule. Earlier practice fixed an embedding or a set of mixture weights tuned on a development set and then retrieved or reranked under that same rule; when the instance-relevant criterion switches, shared weights cannot satisfy both instances' rankings. In a controlled concept-recovery task with 500 episodes, 50 candidates and up to 12 probes, BASIS reaches 68.1% Success@1 after six probes, the tuned static mixture 56.0%, and the oracle that uses the true criterion 79.5%.

The method needs no training at test time: a frozen encoder supplies a criterion-conditioned similarity for each candidate criterion, and the posterior contracts over the joint space of candidates and criteria. The criterion library is a finite menu supplied in advance, spanning semantic, lexical, entity, structural, style and answer-consistency scores; each criterion is a projection or lightweight scorer, and the static mixture, oracle diagnostic and BASIS share the same criterion scores, differing only in whether weights are fixed, known or inferred online. Reasoning exemplar selection is an equal-weight macro-average over GSM8K, StrategyQA and three BIG-Bench Hard tasks, where BASIS scores 73.1% and the tuned mixture 72.3%; refinement over 1,000 prompts each from HH, OpenAssistant and UltraFeedback gives 63.8% versus 61.4%.

The next probe is chosen by expected information gain, meaning the candidate or candidate pair expected to reduce uncertainty about the joint candidate-criterion state the most. Scalar scores update through a Gaussian likelihood, pairwise preferences through Bradley–Terry, and verifier outputs through a logistic model; disabling active probing lowers reasoning and refinement to 72.0% and 62.4%. On a cached-score benchmark with pools of 64, 256, 1024, 4096 and 10000, posterior updates grow from 0.07 to 5.68 ms per round while probe selection stays between 110 and 117 ms per round; at 10000 candidates exact posterior marginalization over the full pool is retained.

Perspective

Evaluation covers controlled concept recovery, reasoning exemplar selection on GSM8K, StrategyQA and three BIG-Bench Hard tasks, and response refinement on HH, OpenAssistant and UltraFeedback, all with one frozen retriever (Qwen3-Embedding-4B), generator (Qwen3-8B with thinking disabled) and verifier (Ministral-3-14B-Reasoning); the reasoning results describe this matched setting and are not directly comparable to benchmark scores reported with default thinking enabled.

The method only selects among supplied criteria and does not discover new ones; when a relevant criterion is missing or corrupted the posterior may favor an incorrect explanation, and the authors' fallback routes low-fit cases to the tuned mixture but cannot establish that the library is complete. Most feedback comes from offline verifiers or rubric scores, live human feedback and preference drift remain untested, and generalization across models and serving systems is still open.