string+doiurl[1] \addbibresourcereferences.bib
Gameplay Filters: Safe Robot Walking through Adversarial Imagination
Abstract
Ensuring the safe operation of legged robots in uncertain, novel environments is crucial to their widespread adoption. Despite recent advances in safety filters that can keep arbitrary task-driven policies from incurring safety failures, existing solutions for legged robot locomotion still rely on simplified dynamics and may fail when the robot is perturbed away from predefined stable gaits. This paper presents a general approach that leverages offline game-theoretic reinforcement learning to synthesize a highly robust safety filter for high-order nonlinear dynamics. This gameplay filter then maintains runtime safety by continually simulating adversarial futures and precluding task-driven actions that would cause it to lose future games (and thereby violate safety). Validated on a 36-dimensional quadruped robot locomotion task, the gameplay safety filter exhibits inherent robustness to the sim-to-real gap without manual tuning or heuristic designs. Physical experiments demonstrate the effectiveness of the gameplay safety filter under perturbations, such as tugging and unmodeled irregular terrains, while simulation studies shed light on how to trade off computation and conservativeness without compromising safety.
I Introduction
Increasingly, autonomous robots are being deployed beyond controlled environments and required to operate reliably in uncertain, unforeseen conditions \citepkumar2021rma,zhuang2023robot,hsuzen2022sim2lab2real,margolis2022rapid,kostrikov2023demonstrating. This has resulted in a growing need for robot safety frameworks that can scale with system complexity and generalize gracefully to novel environments.
Model-based approaches developed by the robotics and control communities offer a principled treatment of safe decision-making under uncertainty. Unfortunately, computing global safety \removecontrol \newfallback strategies for high-dimensional, nonlinear robot dynamics remains an open problem. State-of-the-art numerical safety methods only scale to 5–6 state variables [bansal2017hamilton, bui2021realtime], woefully short of the 12 needed to accurately model the flight of a drone in free space, let alone the 30–50 required for most legged robots. Analytical approaches like Lyapunov controllers and CBFs rely on hand-design, structural assumptions, and reduced-order models [nguyen2022robust, molnar2022modelfree], restricting their use to a local operating envelope, such as a predefined stable walking gait. As a result, legged robots are notorious for falling easily, especially on irregular terrain or when externally perturbed (pushed, tugged, or tripped).
Data-driven approaches grounded in machine learning address the scalability challenge by automatically distilling efficient representations from the robot’s prior experience or, more recently, from web-scale data \citepdeepmind2023rtx. In practice, however, learned models for robot control, including deep reinforcement learning and imitation learning, are often trained in simulated environments due to hardware constraints and poor sample complexity (requiring millions of training episodes that can much more easily be procured by at-scale simulation). The discrepancy between training and deployment conditions, or sim-to-real gap, can result in deteriorated operational performance and, in extreme cases, catastrophic safety failures (e.g., damaging the robot or hurting nearby people) [hsuzen2022sim2lab2real]. Additionally, end-to-end approaches often require re-training for different task specifications, which presents technical challenges in balancing safety objectives with task-specific goals, especially avoiding situations where a robot may unexpectedly prioritize task performance over safety.
A recent line of work breaks down the safety–performance trade-off through variations of a supervisory control mechanism known as a safety filter, which monitors the autonomous system’s safety at runtime and intervenes when necessary by adjusting the original performance-oriented control to avert catastrophic failures \citephsu2023safety,fisac2019AGS,ames2017cbf,wabersich2018linear,bastani2021safe,kumar2023cbfddp. While some efforts have been made to synthesize safety filters for legged robot locomotion, these typically rely on simplified low-order dynamics to maintain tractability, and they lack a systematic treatment of uncertainty and reality gap \citephsuzen2022sim2lab2real,yang2022safe.
This paper introduces a novel type of safety filter that brings together the scalability of learning-based representations and the reliability of model-based safety analysis, enabling highly robust and minimally disruptive safety assurance for arbitrary robot task policies. Unlike most general safety filter techniques, the approach scales readily to robot dynamics with tens of state dimensions, which allows us to focus on its use in the dynamic legged locomotion domain. \newFurther, our safety filter can monitor a closed-loop policy and address the associated computational latency, while existing safety filters only handle a single control input.
A preliminary offline stage leverages game-theoretic reinforcement learning to \removesystematicallysynthesize \newcontrol and disturbance policies, which can be systematically used to construct safety filters for general nonlinear, high-dimensional dynamic systems \newat runtime. At every control cycle, the online gameplay safety filter assesses safety risks based on an imagined game between the control and adversarial disturbance policy trained in offline gameplay learning. This imagined gameplay aims to simulate the worst-case realization of the uncertainty in the system, either from a sim-to-real error or perturbations from the environment. If dangerous conditions emerge, the filter steps in to prevent task-driven actions that could lead the robot to lose in the subsequent safety-oriented gameplay.
The effectiveness of the proposed gameplay safety filter is validated in a legged robot locomotion task with a 36-dimensional state space and a 12-dimensional control space.111See https://saferobotics.princeton.edu/research/gameplay-filter for supplementary material. Our results demonstrate that the gameplay safety filter is inherently robust to the sim-to-real gap, operating in a “zero-shot” manner without requiring manual design or hyperparameter tuning during deployment. Moreover, the gameplay safety filter achieves a high safety rate without being overly conservative, avoiding frequent interventions in the performance-oriented control policy. Importantly, the gameplay safety filter synthesis remains independent of the performance-oriented policy, making it modular and adaptable to any performance-oriented policy at runtime. Our evaluation includes real-world experiments on different terrains with perturbations (see Figure 1) and a comprehensive simulation study on the relative importance of design choices.
II Related Work
Learning for Locomotion. Conventionally, legged locomotion has been addressed through model-based techniques, including model-predictive control \citepbledt2018mit and trajectory optimization \citepwinkler2018gait. However, recent advancements in deep learning offer the opportunity to learn directly from interactions with environments and feedback in the form of a reward signal, bypassing the need for intricate dynamics modeling and extensive domain knowledge. \citetkostrikov2023demonstrating demonstrated the direct training of locomotion policies across various terrains in the real world through reinforcement learning by carefully formulating the problem with consideration for state space, action space, and reward function. Despite the success of reinforcement learning, it relies on trial and error during training. In safety-critical environments, learning from scratch can lead to catastrophic safety failures. An alternative approach involves initially training control policies in simulation and then bridging the simulation-to-real gap through methods such as domain randomization \citeptobin2017domain, task-driven adaptation \citepkumar2021rma, ren2023adaptsim, and system identification \citepfabio2019bayessim.
Safety Filters. While learning-based policies discussed earlier exhibit practical utility, they primarily focus on task-oriented performance metrics. However, ensuring their safe operation in unforeseen, uncertain, and unforgiving environments is of paramount importance. A recent line of work aims at inducing safety awareness and even guarantees for learning-based policies through a safety filter. The runtime operation of every safety filter can be conceptualized as two interrelated functions: monitoring and intervention \citephsu2023safety. The safety filter continually monitors the robot’s planned actions to assess the level of safety risk. Subsequently, the filter may intervene by modulating or entirely overriding the robot’s intended control input to guarantee the preservation of safety. Many safety filters incorporate monitoring and intervention procedures guided by a safety-oriented control strategy, which the filter views as a viable fallback.
One important family of safety filters is built on Hamilton-Jacobi (HJ) reachability analysis, which computes a global safe value function through dynamic programming \citepmitchell2008flexible,fisac2015reachavoid. The resulting value function encodes the maximal safe set and optimal safety \removepolicy \newfallback, and thus, a least-restrictive safety filter can be synthesized by a switch-type intervention \citepfisac2019AGS. Although systematic and powerful, HJ methods have poor scalability and are limited to no more than 6 state dimensions \citepbui2022optimizeddp.
On the other hand, control barrier functions (CBFs) \citepames2017cbf,ames2019control no longer encode or approximate the maximal safe set. Instead, CBFs, if found, provide a sufficient condition to keep the system safe forever, akin to control Lyapunov functions \citepsontag1983lyapunov. Another critical feature of CBFs is their usage of optimization-type intervention, which finds minimal modulation to the task-oriented control that still keeps the system safe, and thus CBFs allow a smooth intervention mechanism. However, finding a CBF for general dynamics is usually not trivial, and CBF is only local and not robust to model mismatch.
For high-dimensional dynamics, computing global optimal value functions (HJ) is computationally prohibitive, and finding a valid CBF is often heuristic. Instead of relying on value functions, model predictive safety filters aim to certify the system safety in real time by forward simulating (“rolling out”) or trajectory optimization,222Some recent efforts have been made to synthesize CBFs based on model predictive methods \citepchen2021backup,kumar2023cbfddp, while the concerned dynamics has no more than 5 state dimensions. \citepwabersich2018linear,bastani2021safe,hsunguyen2023isaacs which closely link to this work. \citethsunguyen2023isaacs consider the forward-reachable set (FRS) of the system trajectories. However, the use of FRS brings two challenges for general high-dimensional dynamics: 1) FRS needs to be tight to make safety filters not overly conservative, and 2) the computation of FRS needs to be quick to satisfy real-time constraints. Instead, \citetbastani2021safe assume disturbance distribution is known, by which a sufficient number of trajectories are sampled and a statistical guarantee is derived; nonetheless, disturbance distribution may be difficult to obtain in practice.
Safety filters have been applied to ensure the safe operation of learning-based locomotion \citephsuzen2022sim2lab2real,yang2022safe. \citethsuzen2022sim2lab2real introduce a safety monitor based on a value function and fine-tune the corresponding safety filter using a two-stage reinforcement learning framework, providing statistical safety guarantees. However, they consider the uncertainty distribution as a whole, while our work focuses on robustly safeguarding against the worst-case realization of uncertainty. \citetyang2022safe propose a safety monitor criterion based on a heuristically defined safety-triggered set, checking if rollouts activate the criterion. In contrast, our work determines such a safety-triggered set through gameplay rollouts. Additionally, these methods employ a simplified dynamics model and only consider velocity control instead of torque or joint position control directly.
III Preliminaries
III-A Scalable Safety Analysis via Reinforcement Learning
We consider discrete-time, uncertain robot dynamics
(1) |
where, at each time step , is the state of the system, is the control input (typically from a control policy ), and is the disturbance input, unknown a priori. \newThe disturbance bound defines the operational design domain (ODD), under which we must ensure autonomous systems function safely and effectively. We further assume we are given a specification of the failure set of all conditions the system state should never reach. Safety analysis aims to determine the largest possible safe set , from which there exists a control policy that can maintain system safety against all admissible uncertainty realizations (encoded by a disturbance policy )
(2) |
where and is the system trajectory starting from and following dynamics Eq. 1 with control and disturbance inputs from control policy and disturbance policy , respectively.
Hamilton-Jacobi-Isaacs (HJI) reachability analysis leverages the level set representations to transform the binary outcome, or game-of-kind as formulated in Eq. 2, into a continuous outcome, or game-of-degree, by a (Lipschitz-continuous) margin function such that 333An example of a margin function is the signed distance function to the failure set. \citepmitchell2008flexible,bansal2017hamilton
(3) |
where is the concerned control horizon. Consistent with the identifiers in Eq. 2, we compute the lower value of the game , which gives the disturbance policy information advantage \citepisaacs1954differential. Additionally, this value function is the fixed-point solution of the Isaacs equation, which can be solved by dynamic programming
(4a) | ||||
(4b) |
If a nonempty safe set is present in the context of the differential game, the value function converges and becomes time-independent within this set as . Consequently, we can eliminate the dependence on , resulting in . We can recover the (maximal) safe set by the superzero level set of the value function and the optimal policies from the optimizers of Eq. 4.
However, in practice, it is difficult to know the sufficient control horizon a priori. \newAlso, finding the maximal safe set is usually difficult to find in complex, high-dimensional dynamics. Instead, reach–avoid analysis simplifies the safety analysis by checking whether a control policy exists to guide the system into a (robust controlled-invariant) target set () in time steps without entering the failure set previously. \removeInstead, we can extend the safety analysis to reach–avoid analysis, requiring the system to reach specific conditions in time steps without entering the failure set previously. The reach–avoid set is defined by
(5) |
Importantly, we choose the target set to be a known robust controlled-invariant set, where there exists a policy that can maintain the system state to stay in forever under all disturbance realizations. \newSince is a robust controlled-invariant set, there exists a policy that can maintain the system state to stay in forever under all disturbance realizations. After the reach–avoid policy safely guides the system into , we can switch to to keep the system in forever. Thus, reach–avoid analysis simplifies the safety control design by requiring only the assurance of control invariance for a small subset of states , which\remove This property is sufficient for the reach–avoid set to be a safe set.
An auxiliary game of degree can be similarly formulated by introducing another margin function with respect to the target set such that . We consider the reach–avoid outcome \citephsu2021safety
(6) |
The reach–avoid value function can be solved by the following Isaacs equation
(7a) | ||||
(7b) |
Similarly, the reach–avoid set can be recovered by .
The computation complexity and memory requirement of solving Eq. 4 and Eq. 7 grows exponentially with respect to the dimension of continuous states, which limits its applicability to no more than six state dimensions for general dynamics \citepbui2022optimizeddp. In recent work, \citethsunguyen2023isaacs proposed an adversarial reinforcement learning framework ISAACS to find approximation solutions to the Isaacs equations, where the state-action value function, or Q-function, , \removecontrol policy \newreach–avoid control policy , and disturbance policy are parameterized by neural networks , respectively.444In reinforcement learning literature, is also called critic as it evaluates the quality of the action, while are called actors since they determine which action to take. In each iteration of ISAACS, it simulates adversarial safety games to collect state-action sequences, performs gradient updates of neural networks, and determines the policies to sample from in the next simulated games.
III-B Value-Based and Rollout-Based Safety Filters
This section introduces safety filters and \new555With a slight abuse of notation, we highlight here the safety filter is composed of a safety fallback policy and a safety monitor. based on switch-type intervention, which can generally be formulated \citephsu2023safety
(10) |
where is an arbitrary performance-oriented task policy, , and is a safety-aware or even safety-guaranteed fallback policy. \new is a safety monitor if it satisfies that its positive output indicates that the input policy can keep the system safe from the input state. Therefore, the safety filter in Eq. 10 maintains the system’s safety, following the Theorem 1 in \citephsu2023safety. Our approach introduces a more general and novel safety filter and monitor, considering the task policy as a function rather than just a singular proposed task control.
Previous work has used neural-network-parameterized Q-function for safety monitor with a threshold \citephsuzen2022sim2lab2real,thananjeyan2021recovery
(11) |
Also, the safety fallback policy can be constructed by or directly when the state is outside of the target set. \newOn the other hand, when the state is in the target set , serves as a fallback to keep the state inside the target set. \newIn other words, the safety fallback policy is defined by a switching rule
(14) |
Although this critic filter has shown practical utility, it does not readily provide safety guarantees. In addition, the value threshold needs to be carefully tuned, which may be difficult for safety-critical applications.
In contrast, the safety monitor can be built on model predictive rollouts. \citetbastani2021safe-acc assumes the dynamical model is perfectly accurate (disturbance-free) and checks if, after executing performance-oriented control, rollout based on the fallback policy satisfies the reach–avoid criterion
(15) |
with , , and , . \citethsunguyen2023isaacs tackle the model mismatch by employing robust rollout with FRS
(16) |
with , , and , . In this work, we instead rely on rollouts of the \removetrained safety policy and disturbance policy \newlearned reach–avoid control and disturbance actors to assess system safety risk.
IV Safe Walking by Adversarial Gameplay
This section introduces a systematic way to construct a safety filter for nonlinear, high-dimensional dynamics. In this paper, we specifically consider the task of quadruped walking, but we stress that the method presented is general for different robots and tasks. We start with careful problem formulation by defining state space, control space, uncertainty modeling, and safety specifications. Then, we elucidate offline gameplay learning, \removewhich exploits robot symmetry in simulated games to prevent safety control policy from overfitting to attacks from a specific direction \newwhere a disturbance actor is jointly trained with the reach–avoid control actor to generate the worst-case realization of uncertainty to attack the system adversarially. We close the section \removewith a constructive synthesis for \newby systematically constructing an online gameplay filter using the trained control and disturbance policies \newsynthesized from offline learning.
IV-A State and Action Spaces
The robot’s state and control input are defined by | ||||
(17a) | ||||
(17b) | ||||
\newwith the robot pose, the angular joint position, the rate of these variables and the commanded angular increment of the robot’s joint. Appendix A illustrates the state and action space details. Since the robot has three joints per leg, we end up with a 36-D state space and a 12-D control space. |
We model the sim-to-real gap via a 6-D adversarial force pushing or pulling the robot with a magnitude of 5\new0 N
(17c) |
with represents the force vector applied at position defined by \remove\new in the body coordinates, \remove, \remove. We further assume the optimal force is bang-bang with . \removeThe adversarial disturbance policy takes in the robot’s state and control input , i.e., . The red \removelines\newarrows in the imagined gameplay of Figure 2 show examples of learned adversarial disturbance.
IV-B Safety Specifications
We consider the failure set of states \newwhere the defined critical points (of the robot’s body) are very close to the ground. \removeWe consider the failure set of states where the robot body is very close to the ground or the robot stands with its knees. The safety margin function is defined as
with denotes the desired magnitude.
Also, we consider the target set of states where the concerned variables are within a small box around the target pose and velocity. The target set is designed so that the robot is known to be robustly stable with a simple stance controller. The target margin function is then defined as
Appendix A illustrates the safety specifications details.
IV-C Offline Gameplay Learning
We introduce an offline gameplay learning scheme, which builds upon ISAACS \citephsunguyen2023isaacs. At each iteration, the learning algorithm collects interactions with environments via simulated adversarial safety games, updates neural-network-parameterized Q-function and policies, and determines which control and disturbance policies are used for the next iteration’s simulated gameplay. \removeFurthermore, we utilize the symmetry of legged locomotion to prevent safety control policy from overfitting to attacks from one direction.
Simulated Adversarial Safety Games. At every time step of games, we store the transition in the replay buffer , with , and . The control and disturbance inputs are selected from the policies either trained concurrently or fixed after pre-training. \removeWe notice that in some training runs, the disturbance policy always attacks from a fixed direction (e.g., the positive direction of the y-axis), which results in the safety control policy being vulnerable to the attack in the other direction (e.g., the negative direction of the y-axis). To prevent this overfitting, we utilize the symmetry in robot locomotion by flip** the disturbance inputs in the collected interactions and re-simulate the new disturbance sequences, i.e., .
Policy and Critic Networks Update The core of the proposed offline gameplay learning is to find approximate solutions to the Isaacs equation Eq. 7. We employ the Soft Actor-Critic (SAC) \citephaarnoja2018sac framework to update the critic and actor networks with the following loss functions.
We update the critic to reduce the deviation from the Isaacs target666Deep reinforcement learning typically involves training an auxiliary target critic , with parameters that undergo slow adjustments to align with the critic parameters . This process aims to stabilize the regression by maintaining a fixed target within a relatively short timeframe. | ||||
(18a) | ||||
with , . We update control and disturbance policies following the policy gradient induced by the critic and entropy loss: | ||||
(18b) | ||||
(18c) | ||||
where , , and are hyperparameters encouraging higher entropy in the stochastic policies for more exploration, which decay gradually in magnitude through the training. |
We can directly train the critic and control and disturbance actors from scratch through Eq. 18. On the other hand, we can also utilize a three-level training curriculum with two additional pre-training stages (L1 and L2). In L1, we only train the \removesafety control policy \newreach–avoid control actor without considering adversarial disturbance inputs, which is a special case of Eq. 18 when . Then, in L2, we fix the control policy trained in L1 and train the disturbance policy instead. Since there is only one policy to optimize in L1 and L2, we can use standard SAC directly. At the beginning of the gameplay learning (L3), we can then initialize actor and critic networks with pre-trained weights, i.e., the control actor (L1), disturbance actor (L2), and the safety critic (L2).
Furthermore, L2 training can be viewed as finding the best adversary to attack the associated control policies, or simply best response . \newAfter ISAACS training, we additionally use L2 training to fine-tune against frozen . We combine the resulting into our gameplay filter. On the other hand, we utilize L2 training to perform a \newbespoke ultimate stress test (BUST) for safety policies and safety filters under various design choices in Table III.
Policy Selection. During the L3 training, we also maintain a finite leaderboard of control and disturbance \removepolicies \newactors from past iterations of training. Periodically, the leaderboard is updated by performing simulated gameplays between the current control and disturbance policies and the previous leaders. If the capacity of the leaderboard is reached, we remove the \removecontrol policy \newcontrol actor checkpoints with the lowest safe rate (and the disturbance \removepolicy \newactor checkpoints with the highest safe rate). At the next iteration’s simulated adversarial safety games, we randomly select control and disturbance \removepolicies \newactors from the leaderboard to generate action inputs, which prevents the control \removepolicy \newactor updates from excessively fitting into a single disturbance \removepolicy \newactor \citepvinitsky2020robust.
IV-D Online Gameplay Safety Filter
This section illustrates that the offline game-theoretic reinforcement learning scheme outputs can systematically synthesize \newThis section demonstrates how the reach–avoid control actor and disturbance actor synthesized offline through game-theoretic reinforcement learning can be systematically used at runtime to construct highly effective safety filters for general nonlinear, high-dimensional dynamic systems. \removeWe employ the rollout-based safety monitor as the value-based safety monitor requires tuning the value threshold, which is difficult to perform offline. \new A predictive (rollout-based) safety monitor is employed to prevent tuning the value threshold as the value-based safety monitor, which is difficult to perform before deployment. Also, a simple switching intervention scheme in the form of Eq. 10 is used, although optimization-based schemes like CBF–QP are also possible..
However, \newThe state-of-the-art predictive safety monitors face scalability and robustness challenges. For example, the nominal rollout in Section III-B can result in an overly optimistic filter, while the FRS-based robust rollout in Section III-B can be computationally intensive for high-dimensional dynamics. To tackle \removescalability and robustness \newthese challenges, we propose using \newa novel adversarial gameplay rollout between the fallback and disturbance policy from offline gameplay learning.
Since the safety policy from ISAACS only aims to reach the target set safety, we need to utilize to keep the state inside the target set once the system reaches the target set. Therefore, the fallback policy is defined by the switching rule \removewhere we use to denote safety control policy for clarity.
The adversarial gameplay then checks if the fallback policy can safely reach the target set after executing the performance-oriented controls, with the whole rollout under attacks from the ISAACS disturbance policy. \newThe adversarial gameplay begins with applying a control from task policy and fallback policy afterward, with the whole rollout under attacks from the ISAACS disturbance policy . This gameplay monitor returns success if the state trajectory safely reaches the target set:
(19a) | ||||
with , , and | ||||
(19d) |
Finally, the (real-time) gameplay filter is constructed by the fallback policy in Eq. 14, the gameplay monitor in Eq. 19, and a switch-type intervention scheme in Eq. 10. Algorithm 1 illustrates the proposed gameplay filter, and Appendix Appendix A summarizes the terminology (and the symbols) of the modules in safety filters.
Additionally, we consider a longer horizon instead of potentially switching between steps. \newHowever, the computation time in the gameplay rollout may require multiple time steps. To resolve this latency issue, we verify the task policy by a longer execution horizon of steps instead of within one step, as in Algorithm 1. The longer foresight \removeaddresses measurement latency in real deployment but also smoothens out undesired oscillations close to the boundary of the reach–avoid set. The gameplay-based safety monitor \newwith latency is formulated below
(20a) | ||||
with , , , and | ||||
(20e) | ||||
where \remove\new is the \removepolicy being \newverified gameplay filter and is executed\new, and thus immutable, during the wait for simulated adversarial gameplay. \newIn other words, if the (trained) disturbance policy captures the worst-case realization of the uncertainty and sim-to-real gap, executing is guaranteed safe. However, we may not have the optimal disturbance policy due to the parameterization of neural networks. Instead, we still switch to the fallback if there is a safety failure during the first steps in the imagined gameplay. |
Algorithm 2 summarizes the -step gameplay safety filter.
Figure 2 illustrates the operation of the gameplay safety filter \newwith the -step gameplay safety monitor. For example, at monitor cycle , since the safety monitor check is successful at , . On the other hand, at monitor cycle , since the safety monitor check is failed at , .
V Experiments
Through extensive simulation study and hardware experiments, we aim to answer the following questions: Can our offline game-theoretic learning and gameplay safety filter
-
(1)
\remove
provide a systematic synthesis method for general nonlinear, high-dimensional systems, which achieves a high safe rate without overly frequent filter intervention? \newachieve robust safety for general nonlinear, high-dimensional systems without obstructing task execution?
-
(2)
enable the robot to operate safely in various deployment conditions that are different from the training conditions in a “zero-shot” manner?
-
(3)
outperform reward-based learning, non-game-theoretic learning, and value-based (critic) safety filters?
Additionally, we analyze the relative importance of our design choices, including (a) gameplay filter with reach–avoid criteria versus avoid-only, (b) three-level training curriculum versus L3 directly, and (c) symmetric exploitation in offline learning versus without.
V-A Experiment Setup
Robot \newand sensors. We use Spirit 40 from Ghost Robotics for the robot platform as shown in Figure 1 and the PyBullet physics engine \citepcoumans2021pybullet to construct the simulated environment. We use the internal motor encoder of the robot to obtain joint absolute position \new and velocity \remove,, and built-in IMU for roll \new and pitch \removeangle \new, body axial rotational rate \new and velocity \new\remove, and a motion capture system for robot body height . There is no force sensor or contact sensing capability enabled, meaning that ground contact can only be implicitly inferred.
Physical setup. \newGameplay filter. To implement a gameplay safety filter on a physical robot, we create a gameplay rollout server, \removewhich is a ROS service that takes in the current physical robot state and proposed control action. The server then runs the gameplay rollout for a fixed horizon and returns the filtered safe action. Using the reach–avoid criterion for the gameplay rollout terminal condition, we observe a flat change in elapsed time (from request to response) when the rollout horizon increases (from 10 to 300 steps), yielding an average cycle of .
Task and perturbations. We construct two different terrains for physical experiments: flat terrain with tugging forces and unmodeled irregular terrain. The robot’s task is to traverse safely across from the same state initialization to reach the goal on the other side of the terrain.
To \removesimulate \newemulate adversarial \newtugging forces on the robot, we \removemanually create a tugging force on the robot by mounting \newmount a rope to the robot on one end and a motion-tracked dynamometer on the other \newend to monitor the force magnitude and direction. The dynamometer has a rated capacity of and a resolution of , tethered to a computer via RS232C. The sampling rate is to record both constant pulling and force pulses. \removeAs the rope is attached to the body of the robot, the range of is different from the simulated environment, with the arm length from the robot’s center of mass to the mounting point being , resulting in a net moment force of comparable to applying at .
We construct two different terrains for physical experiments: unmodeled irregular terrain and flat terrain with tugging forces. The irregular terrain is a area with a 15-degree incline along one edge, and two mounds emerged in the middle using memory foam, with \removesize length width height of and , positioned away from each other. \removeFor both types of terrain, the goal of the robot is to traverse safely across from the same state initialization to reach the goal on the other side of the terrain.
Baselines. To evaluate the effectiveness of margin-based feedback signal and uncertainty-aware offline learning, we consider three prior reinforcement learning algorithms: (1) standard SAC \citephaarnoja2018sac with reward defined as
(2) non-game-theoretic reach–avoid reinforcement learning (RARL) \citephsu2021safety, \newRARL with the domain randomization (DR), and (4) adversarial SAC with the reward feedback signal. For the critic filter, we conduct a parameter sweep to find the best value threshold in the simulation and use the same threshold in physical experiments directly.
Policy. We handcraft a task policy by using an inverse kinematics gait planner for forward walking and sideways walking. We parameterize all policies by neural networks of 3 fully connected layers with 256 neurons, and critics have 3 layers with 128 neurons. The gameplay filter uses horizon and . We use a low-level PD position controller that outputs torques \remove to the robot motor controller, with the target angular position, the current position, the angular velocity and the PD gains.\new to the robot motor controller with the proportional and derivative gains.
\TstrutPolicy | Tugging Force | Bumpy Terrain | |||||||||||
Successful Runs | Failed Runs | Successful Runs | |||||||||||
Safe Rate | Filter Freq. | Safe Rate | Filter Freq. | ||||||||||
7/10 | 0.17 | 26.3 | 67.5N | 70.5N | 59.8N | 52.7N | 10/10 | 0.19 | 41.2 | ||||
4/10 | 0.10 | 26.8 | 73.7N | 80.9N | 53.6N | 40.0N | 5/10 | 0.22 | 33.5 | ||||
0/10 | n/a | n/a | n/a | n/a | 56.5N | 41.4N | 5/10 | n/a | 16.4\Bstrut | ||||
\TstrutAlgorithm | Maximum Force | ||||
Left | Right | ||||
Low | High | Low | High | ||
87.1N∗ | 61.1N∗ | 99.3N∗ | 59.1N∗ | ||
100.5N∗ | 150.3N∗ | 121.6N∗ | 121.9N∗ | ||
\newRARL + DR | 46.4N | 43N | 57.2N | 72.1N∗ | |
83.2N | 96.9N | 82.8N∗ | 59N \Bstrut | ||
151.9N∗ | 173.7N∗ | 140.3N∗ | 142.6N∗ | ||
-
Safety policies from reward-based reinforcement learning and ISAACS with the avoid-only objective fail immediately before applying force.
-
The policy can withstand this magnitude of force. Since the policy can make the quadruped move toward the tugging direction, we cannot add more force in 10 pull attempts.
V-B Physical Results
Safe walking on different terrains. We answer Questions (1) and (3) by evaluating physical robots walking on flat terrain with tugging force and bumpy terrain. We compare our proposed gameplay safety filter with the task policy and critic safety filter. We record the number of runs that the quadruped can safely reach the goal. Also, for those successful runs, we also report the frequency of filter intervention and the time to reach the goal. We additionally report the maximum and average (adversarial) force in the walk for the experiment of flat terrain with tugging forces. Table I shows the result of the experiment.777There is one test of critic safety filter on bumpy terrains failing to reach the goal but remaining safe. We do not include this run’s filter frequency and elapsed time for the average. Our proposed gameplay safety filter has the highest safe rate on both flat terrain with tugging force and unmodeled irregular terrain. Even for those failed trials, the gameplay filter withstands higher tugging force before it violates the safety constraints. Further, the gameplay filter does not unduly intervene with the task-oriented actions as it has a similar filter frequency as the critic filter.
Figure 1 shows the quadruped walking by applying the proposed gameplay filter versus the performance-oriented task policy. When there are imminent safety failures after executing candidate performance-oriented controls, e.g., with airborne legs or loss of balance when climbing, the gameplay filter intervenes and stretches the legs of the quadruped to fight against persistent forces and bumpy terrain.
Maximum withstandable force. To answer Question (3), we test the maximum tugging force withstandable for the safety policies trained by ISAACS, SAC with reward, \removenon-game-theoretic reach–avoid reinforcement learning \newwith domain randomization (RARL+DR), and adversarial SAC with reward. We pull the quadruped from different directions, where the tugging angle for “low” is always between , and the tugging angle for “high” is always between .
Table II shows that the employed can withstand more than about from all directions, but the non-game-theoretic counterpart (\removereach–avoid RL\newRARL+DR) is vulnerable to the tugging from the left \newand can only withstand . \newThis observation suggests that DR struggles to capture the worst-case realization of disturbances accurately. This limitation arises from the inherent nature of DR, where the control actor is optimized for average disturbance behavior. As the dimension of disturbance input increases, the likelihood of the random policy simulating the worst-case disturbance decreases exponentially. This underscores the importance of employing adversarial game-theoretic learning techniques over DR approaches.
Further, we notice the reward-based reinforcement learning baselines and ISAACS with the avoid-only objective fail almost immediately before applying the force since they overreact and thus flip the robot. We find that reach–avoid policies generalize better since they can bring the robot to a stable stance. We also include tests for task policy and the fixed-pose policy (used when the state is in the target set). We observe that ISAACS \removesafety policy \newcontrol actor is strictly better than and is comparable to .
V-C Simulated Results
Bespoke ultimate stress test (BUST). We further answer Questions (1) and (3) by running more exhaustive case studies comparing the following policies: task policy , ISAACS \removesafety policy \newcontrol actor , critic safety filter , and proposed gameplay safety filter . In order to test their robustness when taken to the limit, we learn, for each of the above control schemes, a specialized adversarial disturbance policy explicitly trained (via L2) to exploit its safety vulnerabilities. \newWe also compare these policies against random perturbations sampled uniformly from the disturbance set or from extreme points (e.g., ) .
In L2 training, the disturbance actor must face a time-independent optimal control problem, where the control policy (including the appropriate safety filter) is queried during environment simulation. We note that while the internally simulated gameplay rollout considers a time-varying policy, the executed safety-filtered policy remains time-independent. Specifically, selects either the task control or the safety fallback control based on the outcome of the gameplay rollout, with the rollout dependent solely on the initial state but not when this state is visited. Therefore, can be considered part of the time-invariant environment, meeting the requirements of L2 training.
Table III shows the result of the \newBUST. \removeWe first look at the first two columns and find that is easily exploitable, so we end up with a that is very effective against , but not against the others. On the other hand, the specialized adversary against the ISAACS controller remains effective in attacking and . Finally, we observe that safe filters are not as exploitable, so their s do not really learn to do more damage than the “universal worst-case” . \newWe first note that is vulnerable to all , while the proposed gameplay filters can only be exploited by its associated . Further, because is very robust, this helps attack effectively against other policies, where the third column has the lowest safe rates compared to other columns.
The last 2 columns show the safe rate under random disturbance. Except for , both the reach–avoid control actor and safety filters remain at high safe rates. This observation suggests that our L2 training method effectively establishes a superior safety benchmark for policies compared to DR, even when we improve the sampling from uniformly within the set to extreme cases .
\Tstrut | \Bstrut | |||||
0.37 | 0.38 | 0.17 | 0.44 | 0.88 | 0.85 | |
0.0 | 0.0 | 0.0 | 0.0 | 0.03 | 0.03 | |
0.42 | 0.35 | 0.03 | 0.45 | 0.84 | 0.89 | |
0.37 | 0.34 | 0.10 | 0.44 | 0.86 | 0.86\Bstrut | |
Sensitivity analysis: reach–avoid criteria vs. avoid-only. We evaluate the significance of using reach–avoid criteria in the gameplay filter by performing a sensitivity analysis of the horizon in the imagined gameplay. Figure 3 shows that the gameplay filter with reach–avoid criteria still remains 100 safe rate even when the gameplay horizon is short (. However, the gameplay filter with avoid-only criteria, which simplifies Eq. 20
(21) |
has more safety violations than task policy when . The difference is due to shorter imagined gameplay resulting in more frequent filter intervention for reach–avoid criteria but overly optimistic monitoring for avoid-only criteria (ignore the upcoming failure). Further, as the gameplay horizon increases, the filter frequency of using reach–avoid criteria goes down, i.e., if
(22) |
This observation indicates that reach–avoid criteria are preferred in physical deployment as it is difficult to know the sufficient horizon a priori.
Sensitivity analysis: three-level training curriculum. We testify to the need to use a three-level curriculum by gameplay results against a specialized adversary for the \removesafety policy \newreach–avoid control actor trained with the curriculum, i.e., . Figure 4 shows the safe rate of gameplay results between the model checkpoints stored along the training and . We observe that two pre-training stages in the curriculum do not significantly improve training performance. In contrast, directly learning in the L3 stage requires similar steps in gameplay learning to reach a decent safety performance.
VI Conclusion
This work presents a game-theoretic learning approach to synthesize safety filters for high-order, nonlinear dynamics. The proposed gameplay safety filter monitors the risk of system safety through imagined games between its best-effort safety \removecontrol \newfallback policy and a learned virtual adversary, aiming to realize the worst-case uncertainty in the system. We validate our approach on a physical quadruped robot under strong tugging forces and unmodeled irregular terrain while maintaining zero-shot safety. An exhaustive simulation study is performed to compare with state-of-the-art safety \removecontrol synthesis methods \newfallback policies, safety filters, and \newunderstand the relative importance of design choices.
Appendix A Implementation Details
The state and action space are defined as:
with the position of the body center, the velocity of the robot in the body frame coordinate, the roll, pitch, and yaw of the robot, the body axial rotational rate, and the angle, angular velocity, and commanded angular increment of the robot’s joint.888In this work, we specifically consider walking locomotion, so the policies ignore .
We define the critical points as the body corners and \removeelbows\newknees of the robot. The safety margin is defined as:
with the distance to ground of robot body corner and the distance to ground of robot knee .
The target margin function is defined as
with the distance to ground of robot toes . denotes the desired magnitude. Table IV shows the threshold used to define the safety and target margin functions for quadruped walking.
\TstrutNotation | Magnitude |
0.1 m | |
0.05 m | |
0.4 m | |
0.05 m | |
10 deg/s | |
0.2 m/s \Bstrut | |
LABEL:tab:term summarizes the terminology used in safety filter design, which also highlights the modularity of the proposed gameplay filter.