A CLI agent's risk hides in its middle layers: Agent MechSuits cuts attack success from 85.99% to 13.36% with a 10-dimensional subspace
Synopsis
Command-line agents accumulate risk across multi-turn execution: a single step can look harmless while later steps assemble it into a harmful chain, and external guardrails can only check outside the model against predefined rules. Agent MechSuits instead reads each step's hidden representation, uses a linear probe to judge whether the current state is harmful, and on a hit adds a steering vector inside a 10-dimensional subspace at the same layer without changing model weights. Across the white-box agents LLaMA-3.1-8B, Qwen-2.5-7B and Gemma-2-9B, current-step risk detection averages 97.32% accuracy and mean attack success falls from 85.99% to 13.36%, at the cost of a higher generation-collapse rate.
Interpretation
The safety decision moves from outside the model to internal representations during execution, where a lightweight linear probe reads whether each step's hidden state is harmful. Prior agent defenses relied mainly on external guardrails that depend on predefined or automatically generated safety rules, which limits fine-grained behavioral control at each step. On the MAS benchmark, split by task so no test task appears in probe training, the probe reaches 99.15% accuracy and 99.17% F1 on LLaMA-3.1-8B, 97.46% and 96.25% on Qwen-2.5-7B, and 95.35% and 86.67% on Gemma-2-9B.
Once risk is detected, intervention happens only along a sparse direction in a single layer, adding a steering vector to the hidden representation to push the agent away from harmful commands without modifying model weights. The steering direction is built from the 10 dimensions with the largest class-discriminative weight difference in the same probe, broadcast to all token positions at that layer, with a strength coefficient searched by an external LLM judge trading safety against utility. In risk mitigation, attack success falls from 73.80% to 16.70% on LLaMA-3.1-8B, from 91.30% to 4.35% on Qwen-2.5-7B, and from 92.86% to 14.29% on Gemma-2-9B, all below the RepE and Self Reminder baselines.
The safety gain comes with a generation-quality cost, since strong steering can degrade output rather than produce a clean refusal. The unguarded vanilla agent has an average collapse rate of 2.24%, RepE 7.25%, and Self Reminder 3.69%. With steering enabled the average collapse rate rises to 18.54%, split as 28.30% on LLaMA-3.1-8B, 13.04% on Qwen-2.5-7B, and 14.29% on Gemma-2-9B.
Perspective
Evaluation is limited to three open-weight white-box models, ReAct-style command-line agents, and 194 tasks executed inside Docker containers, with both detection and steering depending on access to hidden states. The authors state explicitly that the method should not be viewed as a standalone production safety system, and that practical deployment should combine it with sandboxing, permission control, command-level policy checks, and human or model review for high-risk cases.
Lookahead risk-detection samples are naturally sparse under current open-weight command-line agents, Gemma-2-9B has no lookahead subset, and the authors frame lookahead results as feasibility evidence for representation-level anticipation rather than a robustness guarantee. In the Safe-OS prompt-injection stress test, the current probe does not reliably classify injected states as harmful, and steering is applied after the injection step is externally triggered, dropping attack success from 86.7% to 6.7%; this is described as a detection-control gap rather than an end-to-end defense.
