Turning Safety into Competence:
Minimally Exploitable Robot Policies via Safety-Filtered Reinforcement Learning

Ruihan Wu* Rui Yang* Donggeon David Oh Duy Nguyen† Haimin Hu†
*Equal contribution†Equal advising
Paper Video
S2C and safe RL baselines in the hardware touchdown game against a human opponent

S2C enforces robot safety without sacrificing competitive performance. A quadrupedal robot (orange trajectory) plays a touchdown game against a human opponent (grey trajectory). Top: as attacker, S2C feints past the human to score a touchdown; as defender, it tracks rapid human direction changes and holds the line. Bottom: safe RL baselines fail under competitive opponent strategies — the ET attacker exits the field while turning, the CPO attacker collides with the human, and the ET defender falls while pursuing the human.

TL;DR Safety can be a source of competence: separating safety from task objectives in competitive RL substantially reduces policy exploitability without compromising safety.

Abstract

Robots deployed for competitive tasks must outmaneuver their opponents without sacrificing safety. Existing approaches, including safe reinforcement learning (RL), train a single policy to achieve task success and avoid failures simultaneously. This coupling can complicate training and leave the learned policy exploitable by deliberate attacks. We propose Safety to Competence (S2C), a two-stage RL framework that separates safety synthesis from competitive task learning. We formulate competitive robot interactions as safety-critical Markov games and prove that perfect filtering preserves policy non-exploitability when all players commit to safe maneuvers. S2C learns a robust safety filter via adversarial RL, embeds it in the environment during task policy training, and retains the same filter at deployment. In simulated quadruped touchdown games, S2C outperforms four safe RL baselines and their filtered variants, achieving the highest win rate and Elo rating, and the lowest empirical exploitability. Hardware stress tests against a human opponent confirm S2C’s competence.

Method

S2C training and deployment pipeline

Overview of the S2C training and deployment pipeline, illustrated with the asymmetric touchdown game.

Pseudocode

Algorithm 1 S2C: Safety-Filtered Competitive RL

Input: margins gi, gi,c, task reward r, initial task policy πiinit, robust filter synthesis algorithm Filter_Synthesis, multi-agent RL algorithm MARL

Training Stage 1: Robust Filter Synthesis learn the safety filter once
  1. (Q, π1, π2) ← Filter_Synthesis(g1, g1,c, g2, g2,c)
  2. Robust_Filter ← (Q, π1, π2)
Training Stage 2: Filtered Multi-Agent RL learn to compete under the filter
  1. Initialize policy pool: P ← ∅
  2. Initialize task policies: (π1, π2) ← (π1init, π2init)
  3. for k = 1, 2, … do
  4. Collect rollouts with Robust_Filter; each player faces its opponent using π−i or, with probability p, a policy in P
  5. Train π1, π2 with MARL
  6. Periodically update P ← P ∪ {π1, π2}
Deployment for Ego Robot keep the same filter at runtime
  1. πe ← π1 or π2
  2. aet ← Robust_Filter(st, πe)

Results

Symmetric touchdown game

Symmetric game round-robin matrix, Elo ratings and empirical exploitability

Each cell corresponds to 3900 trials. S2C outperforms all baselines, achieving the highest Elo rating and lowest empirical exploitability. Left: round-robin win-rate matrix; each cell reports the row method’s win rate against the column method, averaged over all pairs of training seeds. Top right: Elo ratings computed jointly from all matches (higher is better). Bottom right: empirical exploitability, the net win rate of the strongest evaluated opponent, averaged across seeds (lower is better). † marks baselines deployed with the S2C safety filter.

Simulation

S2C self-play in the asymmetric game

The attacker starts on the left and must reach the green touchdown region before timeout; the defender wins by holding the line until time runs out. A robot loses immediately if it falls, leaves the field, or initiates a collision. Here both robots run S2C: the three rounds end in a timeout, a touchdown, and another timeout.

Simulation clips play at 1× and slow down at the end of each round.

Symmetric game

S2C vs. Nom

Both robots race for the touchdown line on the opponent’s side. S2C starts on the left and gets there first in both rounds.

Asymmetric game

S2C attacker vs. ET defender

S2C dodges the defender and reaches the touchdown region.

Asymmetric game

ET attacker vs. S2C defender

S2C successfully defends against the opponent’s attack. In both rounds, ET initiates a collision and loses.

Real-world experiments

Unitree Go2 against a human opponent

We deploy the policies on a Unitree Go2 in the asymmetric game against a human opponent, who moves a tracked surrogate representing the opposing robot. The human’s strategies were never seen during training. Top: S2C as attacker and as defender. Bottom: safe RL baselines fail under competitive human strategies.

Touchdown

S2C attacker

S2C feints past the human and reaches the touchdown line.

Timeout

S2C defender

S2C tracks the human’s rapid direction changes and holds the line until timeout.

Out of bounds

ET attacker

ET exits the field while turning.

Collision

CPO attacker

CPO initiates a collision with the human.