Skip to main content
Back to timeline
arXivSource publication:

Nonlinear system identification stops picking excitation signals: a reinforcement learning agent learns to fill the state space itself

Synopsis

Excitation design for nonlinear system identification has relied on predefined signal classes and repeated optimization, and classical design mainly minimized parameter variance for linear models. Here space-filling input design is recast as a reinforcement learning problem in which an agent issues amplitude-bounded inputs step by step from system observations, with no input class to choose and no frequency band to tune. On two nonlinear benchmarks, a mass-spring-damper with nonlinear stiffness and a Van der Pol oscillator, the learned policy gives the best overall space-filling performance under the same amplitude bound and distributes samples more uniformly than a state-of-the-art space-filling multisine design.

Source-provided article image: A Reinforcement Learning Approach to Space-Filling Input Design for Nonlinear System Identification
arXiv

Interpretation

The excitation is no longer chosen from a predefined signal class; the agent generates it step by step from the system's current observations, and the amplitude bound holds by construction. Earlier space-filling methods relied on predefined input signal classes, required solving an explicit optimization problem for the system at hand, and had to be re-optimized for each set of initial conditions. On two benchmarks, a nonlinear mass-spring-damper system and a Van der Pol oscillator, the learned policy is compared with a frequency-tuned swept sine, a crest-factor-optimized multisine and the space-filling multisine design of Kiss et al., and achieves the best overall space-filling performance on both systems.

The agent discretizes the state space into a grid and is rewarded for entering a cell it has never visited, while revisits earn only a smaller bonus. The reward combines a novelty term that directly encodes the coverage objective with a revisit bonus that keeps feedback informative when no new region is reached. Training uses proximal policy optimization in parallelized environments, with observations of the current state, the past state, the past action and a compressed representation of the visited tensor produced by an offline-trained autoencoder.

The learned inputs take physically meaningful forms: resonance excitation with amplitude modulation on the mechanical system, and a single long amplitude decay on the Van der Pol system. Baselines only tune parameters of a signal class chosen in advance, whereas the agent decides the form of the excitation itself and adapts it to each system. On the mechanical system the learned input reaches 71.72% domain coverage with a coefficient of variation of 0.52, below the other three inputs; on the Van der Pol system it reaches 30.46% coverage with a coefficient of variation of 1.28, the lowest of the four.

Perspective

The demonstrated scope is simulation of two two-state nonlinear benchmark systems, a mass-spring-damper with nonlinear stiffness and a Van der Pol oscillator, with the input amplitude restricted to a continuous interval and the design carried out offline on the exact system equations.

The design stage replaces the surrogate model with the exact system equations to isolate the input design problem from surrogate model error; identifying nonlinear models from the datasets each input generates and comparing prediction errors on independent validation signals, and applying the learned inputs to a real system under model mismatch and measurement noise, are left to future work.