Skip to main content
Back to timeline
arXivSource publication:

Train with a centralized safety filter, deploy without it: a three-agent policy scales zero-shot to twenty agents at constant per-agent cost

Synopsis

Multi-agent reinforcement learning can stay collision-free in training with a centralized control-barrier-function filter, but that filter solves a program whose constraint count grows quadratically with team size and cannot be kept at deployment. This approach extracts the policy from executed safe actions with implicit Q-learning, then distills the training filter into the policy at synthetic near-collision states so the filter-free policy still respects the constraints. Local-observation policies had no collisions in test episodes on their training settings, a three-agent policy transferred zero-shot to twenty agents at constant per-agent computation, and stayed collision-free up to eight agents.

Source-provided article image: Distilling Safety Filters for Filter-free Multi-agent RL
arXiv

Interpretation

Multi-agent reinforcement learning can be trained collision-free under a centralized safety filter and deployed as a policy that sees only local observations and does not communicate. Earlier multi-agent methods with control-barrier-function filters still solve a joint quadratic program online at deployment, with a constraint count that grows quadratically with team size and requires every agent's state and action. On the hardest setting, double-integrator dynamics with local observations, 3 agents and 4 obstacles, the method collides in 0.0% of test episodes without the filter, against 98.3% for multi-agent SAC trained with the same filter.

The policy learns values only from the safe actions actually executed, instead of from filtered data that hides the cost of unsafe proposals. The filter maps many raw actions to the same safe action, so actor-critic methods overvalue unsafe proposals whose consequences were corrected; SAC trained with the filter collides in nearly every episode once it is removed. On 3 agents and 4 obstacles, implicit Q-learning alone without distillation collides in 8.0% of episodes, and adding near-collision-state distillation brings it to 0.0%.

The training filter also serves as a teacher, labeling synthetic near-collision states that the replay buffer never contains. Because the filter prevents collisions, the buffer lacks the near-collision states an unfiltered policy may reach, so in-sample learning alone cannot cover them. In the zero-shot 16-agent, 2-obstacle setting, implicit Q-learning alone collides in 100.0% of episodes, adding distillation gives 7.0%, and distillation without the reinforcement-learning term gives 2.0%.

The deployed policy depends only on local observations and its parameters do not depend on team size, so a three-agent policy carries over to larger teams. The joint quadratic program takes milliseconds at three agents and grows to a third of the control period at twenty, while each agent's local policy takes a constant time. A three-agent double-integrator policy deployed without retraining on up to twenty agents has no colliding episode up to eight agents in either layout, stays below a stated collision level up to sixteen agents in random layouts and reaches a stated level at twenty, while goal reaching exceeds a stated level.

Perspective

The demonstrated scope is disc-shaped agents with fixed radius moving under single- or double-integrator dynamics in an arena with circular obstacles, trained with a centralized control-barrier-function filter and deployed with local observations only, no communication and no filter; hardware validation covers up to five Crazyflie quadrotors at fixed height with motion-capture positioning and a ground computer running the policy.

Under double-integrator dynamics there is no discrete-time training guarantee, the filter keeps the quadratic program feasible with a penalized slack, and the discretization term in the bound can consume most of the safety margin at the control step used, so double-integrator safety is measured empirically; zero-shot scaling shows collisions as teams grow, with the circle stress test colliding beyond twelve agents, mostly with obstacles.