Black-box LLM uncertainty splits in two: a two-level ensemble separates vague prompts from missing knowledge
Large language model uncertainty has been collapsed into a single consistency score that cannot say whether the model misread the question or lacks the answer. A two-level ensemble treats repeated stochastic decodes of one prompt as the inner level and meaning-preserving rewrites as the outer level, then uses the law of total covariance in sentence-embedding space to split aleatoric from epistemic uncertainty using only black-box sampling. Across five short QA benchmarks and four instruction-tuned models it is competitive with strong white-box baselines, and paired interventions make the two components respond separately to prompt ambiguity and knowledge gaps.