External advice no longer used once: Encoder BPA retrieves old advice through latent state encoding, speeding early training
Synopsis
In interactive reinforcement learning, external advice is often used once and discarded; Broad-Persistent Advising reuses it across similar states via K-means clustering, but high-dimensional robot environments allow few clusters, so behaviorally different states can be merged. Encoder BPA instead trains an encoder to map recent state sequences into latent representations and retrieves stored advice by cosine similarity, top-k aggregation and a consistency condition. On Cart-Pole, Inverted Pendulum, MuJoCo Ant and a partially observable Ant, it outperforms pure RL, non-persistent, persistent advising and original BPA early in training, while in continuous-action settings later episodes can decline when inappropriate advice is reused.
Interpretation
Advice from an external trainer can now be reused across states instead of being discarded after each step, with reuse keyed to a latent representation of the state sequence rather than to a cluster ID of the raw state. Original BPA compressed states into a few K-means clusters; in the Domestic Robot environment each state is an RGB image, and the elbow method selected only four clusters, so behaviorally different states could land in the same cluster. Across Cart-Pole, Inverted Pendulum, MuJoCo Ant and POMDP Ant, Encoder BPA's early-training curves outperform pure RL, non-persistent advice, persistent advising and original BPA.
The encoder is first trained on transition data collected by a teacher network to predict teacher actions from recent state sequences, then frozen for retrieval. Retrieval compares the current latent with latents stored in persistent memory by cosine similarity, filters by a similarity threshold, forms a candidate action by top-5 majority vote or averaging, and uses a consistency threshold to decide reuse. Spearman correlations for both LSTM and Mamba encoders are strongly negative in all four environments, for example -0.986 and -0.967 on Inverted Pendulum, meaning more similar latents correspond to smaller teacher-action distances.
Reuse frequency exceeds original BPA and state-based persistent advising in most environments. State-based persistent advising rarely meets an identical state in continuous spaces, averaging 18 reuses on Cart-Pole, 1 on Inverted Pendulum, and 0 on Ant and POMDP. Original BPA averaged 6032, 5892, 141952 and 73990 reuses across the four environments, the LSTM encoder 28025, 16553, 119721 and 63288, and the Mamba encoder 27737, 17813, 127132 and 99505.
Perspective
The demonstrated scope is four simulated environments: Cart-Pole, Inverted Pendulum, MuJoCo Ant and a POMDP Ant with two torso-velocity state components masked, with advice from simulated advisors at frequency 0.5 and accuracy 1.0 or 0.9, evaluated by average reward over the most recent 100 episodes and by advice-reuse counts.
In continuous-action environments, later episodes can decline when the averaging strategy folds no-longer-appropriate advice into the candidate action; lowering the consistency threshold from 0.5 to 0.3 substantially reduces reuse; with advisor accuracy at 0.9 the method is sensitive to noisy advice like other interactive approaches, since the encoder retrieves advice but does not evaluate its quality; small changes in the similarity or consistency threshold can greatly change reuse frequency.
