Agent-retrieved covariates need three checks: date filtering alone lets encoder weights carry future facts
Synopsis
Language-model agents can retrieve fast-moving local context that survey baselines miss, but whether such covariates are usable for causal analysis needs separate verification. A pilot study anchored on Uganda's Youth Opportunities Program compares two retrieval workflows across 11 datasets and 47 treatment-outcome contrasts, with checks on retrieval timing, out-of-sample ranking and pretrained encoder weights. Neither workflow dominates, and swapping the encoder alone shifts downstream heterogeneity rankings.
Interpretation
Local text retrieved by agents needs a verification protocol independent of task completion before it is used as a pretreatment covariate, because such covariates enter the conditioning set of individual treatment effects directly. Existing agent benchmarks score task completion, instruction following and tool use, not whether agent-produced data can support causal estimation. The protocol has three layers: retrieval admissibility by pretreatment date filtering, heterogeneity validation by out-of-sample rank-weighted average treatment effect ratios with cross-fitting, and encoder admissibility by chronologically vintaged encoders, future-fact probes, explicit-label controls and weight perturbations.
The two retrieval workflows differ in whether they share one corpus or run isolated sessions: one builds a date-verified shared corpus per study, the other opens an ephemeral session per study context. A shared corpus reduces per-unit search noise, but a post-treatment leak in a shared source contaminates every unit mapped to it. Across 11 datasets the workspace teammate has a higher mean ratio in 6 datasets for text alone and 7 for combined features; of 188 paired tests, only 7 differences survive multiplicity adjustment, 5 favoring the workspace teammate and 2 favoring session-isolated retrieval.
Even when input text predates the intervention, a pretrained encoder can carry future facts learned during pretraining into the embedding, so temporal filtering is necessary but not sufficient. Date filtering at retrieval constrains document provenance only, not the retrieval policy or the encoder weights. Chronologically vintaged probes detect no post-event shift on clean historical text, yet explicit-label controls show sharp sensitivity to token pooling; in a controlled Gaussian perturbation experiment, probe accuracy falls from 92.8% without noise to 52.1% at a larger noise scale, while held-out token negative log-likelihood rises from 4.82 to 12.20 over the same range.
Perspective
The protocol runs on 11 datasets and 47 treatment-outcome contrasts, anchored on Uganda's Youth Opportunities Program and spanning preference-conjoint, field and behavioral experiments; the conclusions concern the retrieval workflows and encoder audits under these settings.
The two retrieval workflows trade wins in paired tests, with neither universally dominant; the hosted workspace teammate relies on unpinned model weights and a shared corpus that still needs direct audit; observational leakage probes cover specific encoders, token-pooling rules and historical events, and the controlled experiment studies one invented fact under noisy adapter perturbation.
