Retrieval-augmented answers read well but resist tracing: retrieving implicit questions lifts textbook adherence by 34% on average
Synopsis
Retrieval-augmented generation (RAG) lets large language models answer from textbooks, yet their explanations often mix in prior knowledge that cannot be traced back to the retrieved text. Across 90 Stack Overflow questions and three programming textbooks, plain RAG reached only 22%–40% median source adherence, versus 0% without retrieval. The authors introduce chain-of-illocution prompting (CoI): implicit explanatory questions are extracted from the textbook, evidence is retrieved for a small set of them, and the explanation is then generated; adherence rose significantly for most models, by 34% on average, though absolute levels remain moderate.
Interpretation
Retrieval-augmented explanations still adhere weakly to their sources: even when prompted to cite the textbook sentence by sentence, models add content absent from the retrieved passages. Models without retrieval had 0% median source adherence, and standard RAG pipelines reached only 22%–40%, by model: GPT-4o 22.2%, Llama 3-70B 33.3%, Llama 3-8B 33.3%, Mistral 23.6%, Mixtral 29.3%, GPT-3.5-turbo 40%. The benchmark covers six models on 90 popular Stack Overflow questions tied to three programming textbooks (Java, Python, Pharo), with adherence judged by embedding similarity of at least 0.7 between generated and textbook clauses.
Retrieving evidence for the implicit questions an explanation must answer keeps more of the generated content on textbook ground. Chain-of-illocution prompting (CoI) first uses Mistral to extract implicit question-answer pairs from textbook paragraphs, retrieves 25 candidate questions per query, filters them to 5 by evidence support, and sends them with the original question into generation. Adherence gains were statistically significant for most models, from +8.1% for GPT-4o (p=0.180, not significant) to +63.7% for Llama 3-8B (p<0.001); mean adherence rose 34.08% while semantic similarity rose only 5.1%.
The adherence gain did not come at the cost of how users rated the explanations. A user study comparing RAG with RAG+CoI found no significant decrease in satisfaction, relevance, or perceived correctness. 220 participants were recruited via Prolific and 165 retained; they rated explanations from three systems under GPT-4o and GPT-3.5-turbo on a 0–10 scale.
Perspective
Evaluation is confined to programming education: three open-access textbooks (Java, Python, Pharo) and 90 corresponding popular Stack Overflow questions, with six models from the GPT, Llama 3 and Mistral families. Source adherence is operationalized as traceability to a single authoritative text and does not cover other explanation-quality dimensions such as completeness, brevity, or pedagogical usefulness.
Even with CoI, the best median adherence is only about 0.55, and several models remain between roughly 0.33 and 0.46; the GPT-4o adherence gain (p=0.180) and Mixtral's gains in adherence and semantic similarity (p=0.282) are not significant. The authors note that larger models lean more on prior knowledge, while smaller models follow the prompt more strictly but are more vulnerable to retrieval errors; the implicit-question set only approximates what a particular learner needs.
