Skip to main content
Back to timeline
arXivSource publication:

Grid cells as a spectral basis for successor features: when the goal moves, one linear regression re-adapts the agent

Synopsis

In continual reinforcement learning, successor features separate environment dynamics from task reward, but a poor feature basis forces relearning whenever the task changes. Spectral-SF takes the basis from eigenvectors of the transition matrix of a fixed reference policy, so each successor feature equals its base feature times a known constant and the long-term return shares the same features as the immediate reward, with no successor-feature network or replay buffer. In a two-goal gridworld with the uniform random walk as reference policy, Spectral-SF maintains near-maximal return across every goal change, whereas SimpleSF and DQN lose most of their return at each change.

Source-provided article image: Grid Cells as Spectral Successor Features: A Reusable Basis for Continual Reinforcement Learning
arXiv

Interpretation

An agent changing goals need not relearn the environment dynamics; refitting the reward weights alone keeps return near its maximum after each goal switch. Previously successor features were typically learned by a neural network with temporal-difference updates and had to reorganise when the policy or task changed; SimpleSF and DQN drop sharply in return at each goal change. On the two-goal sequence in the Inverted-LWalls gridworld, both the offline and online Spectral-SF variants approach maximum return within a small fraction of the first task and show only brief drops at each subsequent goal change.

The construction takes the feature basis from eigenvectors of the transition matrix of a fixed reference policy, which makes the successor features closed-form. In general the successor features at a state-action pair cannot be computed from that pair's own base features, which is why successor features usually have to be learned. When the span of the base features is invariant under the transition matrix, the successor features are a linear combination of that pair's base features with closed-form coefficients; with eigenvectors, each successor feature is its base feature times the corresponding eigenvalue. The experiments use the uniform random walk as reference policy, whose basis is the low-frequency graph Laplacian eigenvectors, the grid cells.

Perspective

The demonstrated setting is a single small deterministic gridworld with a fixed layout and two goal locations, whose state-action graph has a finite number of nodes; the offline variant takes its basis and constants from a random walk covering every transition, while the online variant builds its graph from observed transitions but needs the cell identity at every step rather than pixels.

The comparison with SimpleSF and DQN is not on equal information and should be read as a bound rather than a head-to-head result; both Spectral-SF variants plateau slightly below maximum return, which the authors attribute to approximations including the symmetrised walk, the span of retained eigenvectors and the online estimate of reward weights. When rewards of both signs coexist, acting greedily on the reference policy's action values can trap the agent, which is why the lava tiles were removed from the environment.