Skip to main content
Back to timeline
arXivSource publication:

Agent training drops pre-filtered subgraphs: RISE queries all of Freebase and leads multi-hop reasoning across nineteen benchmarks

Synopsis

RISE, a reinforcement-learning training environment for large language model agentic reasoning, replaces pre-extracted per-question subgraphs with the unpruned full Freebase knowledge graph, so path selection, candidate pruning and stopping become the policy's own decisions. Prior subgraph environments compressed the candidate space by five orders of magnitude, and a question-blind degree heuristic surfaced gold answers in the top-10; across nineteen benchmarks and three model backbones, RISE improves in-domain and unseen-schema knowledge-graph reasoning and transfers zero-shot to open-domain passage retrieval.

Source-provided article image: Interactive Structured Environment Scaling for Large Language Model Agentic Reasoning
arXiv

Interpretation

The training environment for agentic reinforcement learning moves from pre-extracted per-question subgraphs to an unpruned full knowledge graph, so the search path, candidate pruning and stopping time are again decided by the policy at each turn. Existing structured and retrieval environments pre-filter upstream before the policy acts: they inject a compact subgraph around the gold answer into the context, or retrieve a fixed passage list first. An audit of the released per-question WebQSP subgraphs, taken exactly as published, shows a median subgraph of only a few triples, five orders of magnitude smaller than full Freebase; a heuristic that never reads the question and ranks nodes by in-subgraph degree surfaces a gold answer in the top-10 far above a uniform random baseline.

The policy emits only natural-language queries with bracketed entity mentions, and a deterministic five-stage typed intent adapter translates them into graph-database operations, so the same weights can drive a static knowledge graph, a temporal graph or a passage retriever. Binding the action space to SPARQL syntax or graph-specific atomic primitives conflates search planning with syntax generation and blocks zero-shot transfer to new tools. The adapter executes symmetrically over outgoing and incoming edges because the audit found that a substantial share of candidate relations exist only as incoming edges; training and evaluation share the same adapter and receipt template, and evaluation runs unconditionally on full official splits.

Learning-Frontier Scaling assigns each question one of four deterministic structural tiers parsed from its gold SPARQL topology and uses a self-evolving curriculum bandit to adjust batch composition online by group-level pass-rate variance. Under full-graph exploration, group-normalized objectives suffer within-group variance collapse: a question whose rollouts all earn the same reward contributes zero gradient, which covers both mastered and unreachable questions. On Qwen2.5-7B over the same question pool, an easy-to-hard curriculum outperforms uniform sampling and also outperforms the reversed hard-to-easy order; closed-loop variance feedback scores higher than uniform sampling on WebQSP, CWQ, GrailQA, MATH500 and AMC23.

Perspective

The demonstrated setting is knowledge-graph question answering trained on Freebase, evaluated on nineteen benchmarks spanning structured knowledge, cross-tool passage retrieval and tool-free mathematical and logical reasoning, with Qwen2.5-7B, Qwen2.5-14B and Llama-3.1-8B backbones. The intended readers are engineers and researchers building multi-turn tool-use reinforcement-learning environments.

Under a finite call budget, questions whose surface forms fail entity resolution, or that need complex semantic bridging beyond dense relation ranking, remain unreachable. The curriculum experiments fix the training horizon at one epoch, and each arm uses a single training run, so multi-epoch scheduling and multi-seed estimates are not yet covered.