Skip to main content
Back to timeline
arXivSource publication:

Knowledge base grows from human decisions: iRAG cuts final-decision error below a controller-free baseline under 40% drift

Synopsis

iRAG replaces a pre-compiled knowledge base with a precedent store that accumulates each human decision, and lets the system move from silent observation to evidence-grounded contestation and then to conditional autonomy as its reliability estimate rises. On simulated support tickets it matches an independent controller-free RAG-with-defer baseline under 10% drift and lowers final-decision error under 40% drift, while retrieval decay trades stable-ticket accuracy for drift-ticket accuracy rather than giving a general advantage.

Source-provided article image: Incremental Retrieval-Augmented Generation for Human-in-the-Loop Decision Support
Figure 1 · arXiv

Interpretation

The system no longer depends on a static knowledge base compiled before deployment; each input and its final decision is appended as a retrievable precedent record, so knowledge grows out of human judgments. Conventional retrieval-augmented generation assumes the knowledge base is built offline and stays fixed, which does not fit domains where knowledge can only accumulate case by case from expert decisions. On synthetic customer-support tickets simulating four quarters of a CRM-like product, final-decision error is close to an independently rolled-out controller-free RAG-with-defer baseline under 10% drift and lower for every iRAG configuration under 40% drift.

The relationship between system and user changes over time: it first observes human decisions silently, then presents disagreements with retrieved evidence once reliability passes a threshold, and finally takes autonomous decisions while abstaining when evidence is insufficient. Hybrid decision-making previously either had the machine lead and defer to humans when uncertain, or had humans lead while a model acted as a surrogate; moving between the two had only been proposed conceptually, not tested empirically. Reliability is a fading empirical accuracy, the agreement between model and final human decision, and the experiments use three simulated decision-makers, a CEO, a domain expert and an intern, with an auxiliary model judging whether a candidate answer covers the reference decision.

Retrieval relevance decays exponentially with the number of records inserted after a given precedent, making superseded decisions harder to retrieve while keeping them in the store for audit. Non-decaying retrieval treats outdated precedents as equal to still-valid evidence, adding interference when decision criteria change. Under 10% drift, retrieval decay lowers drift-ticket error by several points while raising stable-ticket error by several points, with all four paired confidence intervals above zero; drift-ticket gains remain significant under 40% drift.

Perspective

The evaluation runs on a synthetic benchmark of simulated customer-support tickets covering four quarters of a CRM-like product, with human decision-makers simulated by perturbing gold answers into three profiles, so the conclusions are limited to system-level behaviour under controlled conditions. The system targets decision-support settings where the knowledge base must be built incrementally and humans keep the final say, with retrieval and reliability decay half-lives of roughly one quarter and five hundred reliability observations. The next step is a planned workshop with human participants to test the transition and contestation mechanisms in practice and begin participatory design of the interface.

Human oversight is an architectural provision of the workflow, and its usability with real users remains to be evaluated. Reliability is measured as agreement with final human decisions rather than objective correctness, inheriting the early-decision bias of skeptical learning, which the experiments mitigate with a CEO bootstrap and a review window of the first hundred tickets each quarter. During autonomous operation human labels appear only after deferral or review, so the reliability estimate may go stale. State thresholds are user-specified and potentially arbitrary, and final states are threshold-sensitive, so final-state frequencies are treated as threshold-dependent operational outcomes. The auxiliary judge shares a model family with the main model; the audit found no evidence of blanket self-preference, but a fully independent judge would strengthen it. The experiments assign a substantial share of tickets to an always-correct decision-maker, and future work should vary that share to see how iRAG behaves with a fallible bootstrap expert or narrower review windows.