F EASIBLE A DVERSARIAL R OBUST R EINFORCEMENT L EARNING FOR U NDERSPECIFIED E NVIRONMENTS
A BSTRACT
Robust reinforcement learning (RL) considers the problem of learning policies that perform well in the worst case among a set of possible environment parameter values.
In real-world environments, choosing the set of possible values for robust
RL can be a difficult task.
When that set is specified too narrowly, the agent will be left vulnerable to reasonable parameter values unaccounted for.
When specified too broadly, the agent will be too cautious.
In this paper, we propose Feasible
Adversarial Robust RL (FARR), a novel problem formulation and objective for automatically determining the set of environment parameter values over which to be robust.
FARR implicitly defines the set of feasible parameter values as those on which an agent could achieve a benchmark reward given enough training resources.
By formulating this problem as a two-player zero-sum game, optimizing the FARR objective jointly produces an adversarial distribution over parameter values with feasible support and a policy robust over this feasible parameter set.
We demonstrate that approximate Nash equilibria for this objective can be found using a variation of the PSRO algorithm.
Furthermore, we show that an optimal agent trained with FARR is more robust to feasible adversarial parameter selection than with existing minimax, domain-randomization, and regret objectives in a parameterized gridworld and three MuJoCo control environments.
I NTRODUCTION
Recent advancements in deep reinforcement learning (RL) show promise for the field’s applicability to control in real-world environments by training in simulation (OpenAI et al., 2018;
Hu et al., 2021; Li et al., 2021).
In such deployment scenarios, details of the test-time environment layout and dynamics can differ from what may be experienced at training time.
It is important to account for these potential variations to achieve sufficient generalization and test-time performance.
Robust RL methods train on an adversarial distribution of difficult environment variations to attempt to maximize worst-case performance at test-time.
This process can be formulated as a two-player zero-sum game between the primary learning agent, the protagonist , which is a maximizer of its environment reward, and a second agent, the adversary , which alters and affects the environment to minimize the protagonist’s reward (Pinto et al., 2017).
By finding a Nash equilibrium in this game, the protagonist maximizes its worst-case performance over the set of realizable environment variations.
This work aims to address the growing challenge of specifying the robust adversarial RL formulation in complex domains.
On the one hand, the protagonist’s worst-case performance guarantee only applies to the set of environment variations realizable by the adversary.
It is therefore desirable to allow an adversary to represent a large uncertainty set of environment variations.
On the other hand, care has to be taken to prevent the adversary from providing unrealistically difficult conditions.
If an adversary can pose an insurmountable problem in the protagonist’s training curriculum, under a standard robust RL objective, it will learn to do so, and the protagonist will exhibit overly cautious behavior at test-time (Ma et al., 2018).
As the complexity of environment variations representable in simulation increases, the logistic difficulty of well-specifying the limits of the adversary’s abilities is exacerbated.
For example, consider an RL agent for a warehouse robot trained to accomplish object manipulation and retrieval tasks in simulation.
It is conceivable that the developer cannot accurately specify the distribution of environment layouts, events, and object properties that the robot can expect to encounter after deployment.
A robust RL approach may be well suited for this scenario to, during training, highlight difficult and edge-case variations.
However, given the large and complex uncertainty set of environment variations, it is likely that an adversary would be able to find and over-represent unrealistically difficult conditions such as unreachable item locations or a critical hallway perpetually blocked by traffic.
We likely have no need to maximize our performance lower bound on tasks harder than those we believe will be seen in deployment.
However, with an increasingly complex simulation, it may become impractical to hand-design rules to define which variation is and is not too difficult.
To avoid the challenge of precisely tuning the uncertainty set of variations that the adversary can and cannot specify, we consider a new modified robust RL problem setting.
In this setting, we define an environment variation provided by an adversary as feasible if there exists any policy that can achieve at least λ return (i.e. policy success score) under it and infeasible otherwise.
The parameter λ describes the level of worse-case test-time return for which we wish our agent to train.
Given an underspecified environment, i.e. an environment parameterized by an uncertainty set which can include unrealistically difficult, infeasible conditions, our goal is to find a protagonist robust only to the space of feasible environment variations.
With this new problem setting, we propose Feasible Adversarial Robust Reinforcement learning (FARR), in which a protagonist is trained to be robust to the space of all feasible environment variations by applying a reward penalty to the adversary when it provides infeasible conditions.
This penalty is incorporated into a new objective, which we formulate as a two-player zero-sum game.
Notably, this formulation does not require a priori knowledge of which variations are feasible or infeasible.
We compare a near-optimal solution for FARR against that of a standard robust RL minimax game formulation, domain randomization, and an adversarial regret objective similarly designed to avoid unsolvable tasks (Dennis et al., 2020; Parker-Holder et al., 2022).
For the two-player zero-sum game objectives of FARR, minimax, and regret, we approximate Nash equilibria using a variation of the
Policy Space Response Oracles (PSRO) algorithm
(Lanctot et al., 2017).
We evalaute in a gridworld and three MuJoCo environments.
Given underspecified environments where infeasible variations can be selected by the adversary, we demonstrate that FARR produces a protagonist policy more robust to the set of feasible task variations than existing robust RL minimax, domain-randomization, and regret-based objectives.
To summarize, the primary contributions of this work are:
• We introduce FARR, a novel objective designed to produce an agent that is robust to the feasible set of tasks implicitly defined by a threshold on achievable reward.
• We show that this FARR objective can be effectively optimized using a variation of the PSRO algorithm.
• We empirically validate that a near-optimal solution to the FARR objective results in higher worst-case reward among feasible tasks than solutions for other objectives designed for similar purposes: standard robust RL minimax, domain randomization, and regret, in a parameterized gridworld and three MuJoCo environments.
R ELATED W ORK 2. D OMAIN R ANDOMIZATION
Domain randomization methods train in simulation on a distribution of environment variations that is believed to generalize to the real environment.
The choice of a distribution for training-time simulation parameters plays a major role in the final test performance of deployed agents (Vuong et al., 2019).
While domain randomization has been used with much success in sim-to-real settings (OpenAI et al., 2018;
Tobin et al., 2017; Hu et al., 2021; Li et al., 2021), its objective is typically the average-case return over the simulation uncertainty set rather than the worst-case.
This can be desirable in applications where average training-time performance is known to generalize to the test environment or where this can be validated.
However, average-case optimization can be unacceptable in many real-world scenarios where safety, mission-criticality, or regulation require the agent to perform well in test environments whose weight in the average would be small and where sufficient validation is unavailable.
2. R OBUST R EINFORCEMENT L EARNING
Robust reinforcement learning methods optimize reward in the worst-case and provide a promising approach for sim-to-real.
Robust Adversarial Reinforcement Learning (RARL) (Pinto et al., 2017), which this work builds upon, optimizes return under the worst-case environment variations by optimizing a two-player zero-sum game between a task-performing protagonist and an environmentcontrolling adversary.
Numerous mechanisms by which the adversary affects the environment have been explored including perturbing forces and varying environment dynamics (Pinto et al., 2017;
Mandlekar et al., 2017; Nakao et al., 2021), disturbances to actions (Tessler et al., 2019; Vinitsky et al., 2020; Tan et al., 2020), and attacks on agent observations (Pattanaik et al., 2017; Gleave et al., 2019;
Zhang et al., 2020; 2021; Kumar et al., 2021).
However, each of these works makes the assumption that the uncertainty set from which perturbations and variations may be sampled is well-specified and tuned such that an agent robust to a minimax selection over the entire set will perform optimally in deployment.
FARR relaxes this assumption and intends to produce a robust agent to the real environment, even when the adversary can select unrealistically hard variations.
2. A UTOMATIC C URRICULUM D ESIGN
Similar to finding the worst-case distribution of environment variations is finding the best-case distribution for improving an existing agent.
Automatic curricula seek to find environment parameters that are challenging but not too difficult for a training agent.
Racaniere et al.
, Campero et al.
(2020), and Florensa et al.
(2018) generate useful curricula for a single learning agent to solve difficult tasks by proposing appropriately challenging goals for the agent’s current abilities.
POET (Wang et al., 2019; 2020) co-evolves a population of tasks and associated agents to produce an increasingly complex set of tasks with coupled agents capable of solving their associated task.
While each of these methods can create increasingly capable agents, they offer no guarantees for final agent robustness.
Asymmetric self-play (Sukhbaatar et al., 2018; OpenAI et al., 2021) facilitates two agents with similar capabilities to compete in a two-player game where one attempts to reach goals that are achievable but difficult to the other agent.
PAIRED (Dennis et al., 2020) and subsequent improvements to optimization (Parker-Holder et al., 2022; Du et al., 2022) extend this concept beyond goals to the generation of parameterized environments by introducing a two-player zero-sum regret objective between an adversary and a protagonist.
The adversary learns to specify environment conditions in which regret is highest such that the protagonist’s performance most differs from estimated optimal performance.
Although the regret and FARR objectives both involve estimating optimal performance with respect to the adversary, optimizing regret does not necessarily provide the hardest possible tasks like robust RL and instead selects tasks where improvements to behavior most affect incurred reward.
Regret is useful for training a broadly capable agent given no preference over tasks.
Robust RL, which we are interested in extending, optimizes performance under a worst-case distribution of conditions, and FARR does the same while applying constraints on the difficulty of said conditions.
B ACKGROUND 3. U NDERSPECIFIED P ARTIALLY -O BSERVABLE M ARKOV D ECISION P ROCESSES
We adapt Underspecified Partially-Observable Markov Decision Processes (UPOMDPs) from Dennis et al.
(2020) where there exists a parameter of variation θ ∈ Θ that is hidden from the agent.
This parameter θ controls some aspect of the environment that can potentially be discovered through interaction.
For example, θ could control the friction of a robot arm, the mass of a cartpole, or the layout of a room.
We model a UPOMDP as a tuple M = ⟨ A, S, O, Θ , T , ρ, I , R , γ ⟩ where A is the set of actions,
S is the set of states, O is the set of observations, and γ is the discount factor.
The choice of parameter θ ∈ Θ controls the conditions in this environment, namely the transition distribution function T : S ×
A × Θ −→ ∆( S ) , the initial state distribution ρ : Θ −→ ∆( S ) , the observation function I : S × Θ −→ O , and the reward function R : S × Θ −→ R .
Given an observable history h ∈ H
=
( O × A ) ∗ ×
O , a protagonist policy π p : H → ∆( A ) decides at each time step t on the distribution of the action a t after seeing h = o 0 , a 0 , . . .
, o t .
Jointly with a UPOMDP M and a specific environment parameter θ , the policy induces a distribution p θπ p over the states, actions, observations, and rewards in an interaction episode.
We define the protagonist’s utility U p ( π p , θ ) =
E p θπp [ (cid:80) t γ t r t ] as the expected episode discounted return for a protagonist policy π p and choice of θ .
3. P OLICY S PACE R ESPONSE O RACLES
Policy Space Response Oracles (PSRO) (Lanctot et al., 2017) is a deep RL method for calculating approximate Nash equilibria (NE) in zero-sum two-player games.
It extends the normal-form
Double-Oracle algorithm (McMahan et al., 2003) to games with sequential interaction.
At a highlevel, PSRO iteratively adds new policies for each player to a population until a normal-form mixedstrategy solution to the restricted game induced by selecting population policies closely approximates a NE in the full game.
PSRO operates by maintaining a population of policies
Π i for each player i and a normal-form mixed strategy σ i ∈ ∆(Π i ) .
This mixed strategy σ i represents a distribution over policies π
i ∈ Π i to sample from at the beginning of each episode, and upon algorithm termination, σ = ( σ 1 , σ 2 ) is the final output of PSRO.
The utility of player i of playing a mixed strategy σ i against an opponent’s policy π − i is therefore U i ( σ i , π − i ) =
E π i ∼ σ i
[ U i ( π i , π − i )] , and likewise, U i ( σ i , σ − i ) =
E π i ∼ σ i ,π − i ∼ σ −
i
[ U i ( π i , π − i )] .
In each iteration of PSRO, new policies for each player i are added to its population Π i .
In the case of sequential interaction, this is typically an RL best-response BR ( σ − i ) =
arg max π
i U i ( π i , σ − i ) that maximally exploits the opponent mixed-strategy.
However, this choice of new policy is not a requirement, and PSRO maintains NE convergence guarantees so long as novel policies are continuously added for each player.
After adding new policies, utilities U Π ( π 1 , π 2 ) between each pairing of player policies π 1 ∈ Π 1 and π 2 ∈ Π 2 are empirically estimated using rollouts to create a normal-form restricted game in which population policies are the strategies.
A new NE mixed-strategy that solves the restricted game, a restricted NE σ = ( σ 1 , σ 2 ) , is then cheaply calculated for each player.
This process is repeated until no new policies are added to either player’s population or the process is externally stopped.
As the number of strategies in each player’s population grows, a NE solution to the restricted game asymptotically converges to a NE solution to the full game.
While other potentially suitable methods for solving extensive-form games exist, for example NFSP (Heinrich & Silver, 2016) and Deep-CFR (Brown et al., 2019), PSRO was chosen out of practicality because it can be implemented as an additional logic layer on top of existing reinforcement learning software stacks.
Although PSRO-based methods are competitive in sample-efficiently solving twoplayer zero-sum games (Lanctot et al., 2017; Vinyals et al., 2019; McAleer et al., 2021; 2022b;a;
Liu et al., 2022), the purpose of experiments in this work is not focused on improving the speed at which we might reach optima.
Rather, we are interested in ensuring that we can reliably reach approximate NE for each objective we test by using PSRO in order to compare their near-optimal solutions on even ground.
F EASIBLE A DVERSARIAL R OBUST R EINFORCEMENT L EARNING 4. M OTIVATION
We begin our discussion of the FARR method with a motivating example.
Consider a cartpole environment where the pole is subject to perturbing forces of an unknown magnitude.
To maximize our worse-case performance, we could formulate a robust RL game in which the adversary provides the most difficult possible mixed strategy of force magnitudes.
By learning a best response strategy to this adversary, the protagonist will maximize its worst-case performance at test-time when presented with an unknown distribution of forces.
Importantly, to achieve this process, we would need to allow the adversary to specify a sufficiently wide range of force magnitudes such that the real environment is believed to be in this range.
However, we would not want to allow the adversary to specify force magnitudes so large that the task becomes impossible, because the adversary would then always select overly difficult parameters and the protagonist would only train on impossible task variations, failing at test time.
In this cartpole example, it is possible to manually adjust the range of allowed force values until the widest possible uncertainty set is found that still avoids the learning of a degenerate protagonist strategy.
However, if this setting were scaled up to specifying the allowed values of many coefficients, level layouts, or environmental events where the adversary has complex, high-dimensional interactions with the protagonist, hand tuning these limits may no longer be viable.
To remove the need for human expert tuning of the adversary limits, we instead create a game where the adversary is allowed to specify impossible environment variations but is heavily penalized for doing so, using the method we describe below.
4. F EASIBILITY
We define an environment parameter θ ∈ Θ as λ -feasible if a best-response to θ can achieve an expected return of at least λ ∈ R .
Heuristically, the value for λ could be set as the lowest average return that we would expect an agent to receive across variations in deployment if it could act optimally with respect to each variation.
We define the feasible set F λ of environment parameters as:
[Equation]
F λ matches our motivation for considering feasible parameters when either of two conditions is satisfied.
First, we may have a lower bound on the achievable performance in the real environment, and we can set λ at or below that bound.
If the real environment is feasible, θ ∗ ∈ F λ , then we are justified in avoiding training the protagonist on infeasible environments.
Second, there may be a performance threshold such that only above it we have a preference over agent behaviors.
For example, we may only care where a warehouse robot navigates if it has a valid path to its target shelf.
The set of feasible variations F λ is generally unknown.
As part of optimizing the FARR objective, the adversary will learn an approximation of F λ and use it to guide the selection of a mixed strategy over discovered feasible environment variations.
We note that, in order to measure robustness to F λ in this paper, we focus on environments where we can, in fact, calculate F λ for the purpose of test-time evaluation.
It is reasonable to expect our findings to carry over to some domains where F λ is truly unknown and where the method cannot be directly evaluated.
4. FARR O BJECTIVE
We wish to optimize the standard zero-sum robust adversarial game through PSRO with the additional constraint that the support of the adversary mixed strategy σ θ contains only strategies in the feasible set F λ .
Define supp ( σ θ ) = [Equation] θ [Equation]
Θ |
[Equation]
θ [Equation] θ ) > 0 } . Our intended FARR objective is to solve the game:
[Equation]
An adversary mixed strategy σ θ with support that is a subset of F λ will only provide feasible environment variations to the protagonist.
In order to provide flexibility in how novel adversary strategies that satisfy this constraint could be optimized, we can replace the hard constraint with a sufficiently large penalty C to the adversary when it violates the constraint.
An equivalent zero-sum FARR objective would then be to optimize:
[Equation]
[Figure]
where the FARR utility function U λp ( π p , θ ) is unchanged from the original game if the adversary’s provided environment parameter is feasible, and where otherwise a large constant adversary penalty
C is applied:
[Equation]
4. M ATRIX G AME E XAMPLE
We demonstrate the effect of the FARR utility function (equation 4) through a matrix game example shown in Figure 1.
The adversary specifies the location of a bowl among three cabinets, where the middle cabinet is more difficult for the protagonist to access than the left, and the right cabinet is locked and inaccessible.
Without observing the bowl’s location, the protagonist must choose whether or not to attempt to find and grab the bowl, receiving a penalty for a failed retrieval attempt.
In deployment, we expect that the item will always be placed in a feasible, accessible cabinet, either the left or the middle, so ideal behavior for the protagonist would be to always attempt to grab the bowl.
For this game, we define the feasibility threshold over task variations as λ = 1 .
The original game, shown in matrix-form in Figure 1(b), contains an infeasible adversary pure strategy (bowl in locked right cabinet).
Because of the presence of this infeasible adversary strategy, the protagonist will never attempt to grab the bowl in any of the game’s Nash equilibria (one pure strategy NE is displayed in green).
If we now replace the original game’s utility function U p with the FARR utility function U λp using C = 500 , the adversary receives a penalty of − 500 for placing the bowl in the infeasible locked right cabinet because there exists no protagonist strategy that can achieve a utility of a least λ = 1 under that environment variation.
Instead, in the FARR transformed game, a NE adversary places the bowl in the feasible but difficult middle cabinet, and the protagonist learns, facing a selection of feasible tasks, to always attempt to retrieve the bowl.
This new protagonist NE strategy for the FARR game is optimal with respect to anticipated feasible deployment conditions.
4. O PTIMIZING WITH PSRO
Shown in algorithm 1, we use PSRO to solve for an approximate NE of the FARR transformed game with the penalty-based objective defined in equation (3).
To optimize FARR with PSRO, only the scalar values λ and C need to be provided, where λ is the maximum difficulty expected among tasks in deployment and C is an arbitrarily large positive value.
We represent environment parameters θ as strategies in the adversary population Π θ with an output mixed-strategy σ θ over Π θ .
Similarly, Compute Calculate do Calculate
Π θ = Π θ ∪
} end for until terminated early or no novel policies can be added
Output: current Protagonist restricted NE strategy σ p we represent protagonist RL agent policies as strategies π p ∈
Π p with an output mixed-strategy σ p over Π p .
In each PSRO iteration, to best-respond to the current adversary restricted NE σ θ , a new protagonist policy is trained using RL with a fixed environment experience budget.
One or more random novel adversary pure strategies are added in each PSRO iteration.
For wall-time parallelization, we add 3 in each iteration.
For each adversary strategy θ , we also train an evaluator RL policy π θe with the same hyperpameters as the protagonist to estimate BR ( θ ) and feasibility for the FARR utility function U λp .
E XPERIMENTS
To illustrate the utility of filtering out infeasible tasks in a robust RL setting, we perform experiments in environments where overly difficult or unsolvable task variations are possible while measuring performance under feasible conditions.
In a goal-based gridworld environment and three perturbed
MuJoCo (Todorov et al., 2012) control environments, we compare the performance of FARR with three alternative objectives: a standard minimax robust adversarial RL objective, domain randomization, and the regret objective as proposed in Dennis et al.
Given a threshold for feasibility λ , we evaluate worst-case performance within the set F λ of feasible environment parameters.
FARR outperforms each of these objectives because it is able to provide an adversarial training distribution of environment variations limited only to instances that are discovered to be feasible, while other methods provide overly difficult or otherwise mismatched training distributions for worst-case feasible conditions.
We optimize FARR and other baseline objectives with PSRO and identical protagonist RL bestresponse algorithms in order to compare final performance given guarantees of asymptotically reaching an approximate NE for each zero-sum objective.
We describe each baseline objective below:
Minimax The standard objective for robust adversarial RL using unmodified U p ( σ p , σ θ ) .
When infeasible tasks are allowed to the adversary, the standard minimax robust RL objective will focus on such tasks, resulting in overly cautious protagonist behavior or failed learning.
Domain Randomization (DR) With domain randomization, we train a single protagonist policy π DRp =
BR ( σ DRθ ) to saturation against a uniform mixture of all possible environment variations σ DRθ =
U (Θ) .
Domain randomization can result in an exploitable agent if the relevant feasible part of configuration space is underrepresented by the uniform measure.
This objective is optimized by training a single RL policy rather than with PSRO.
Regret Matching the objective used by PAIRED (Dennis et al., 2020), we use PSRO to approximately solve for NE using the objective min σ θ max σ p E θ ∼ σ θ
[ U p ( σ p , θ ) − U p ( BR ( θ ) , θ )] .
BR ( θ ) is estimated using the same method as with FARR by training an evaluation agent π θe against each θ .
While designed to provide a distribution of tasks where the protagonist is known to be able to positively affect its performance through optimal behavior, this curriculum learning objective does not generally provide a task distribution suitable for robust learning to a specific set of tasks such as F λ .
[Figure]
To calculate the worst-case episode reward within the set F λ of feasible environment parameters, we use our knowledge of F λ to enumerate a comprehensive set of feasible tasks on which every baseline’s performance is measured.
For the gridworld environment, the entire feasible set F λ is calculated analytically.
For MuJoCo, a discretization of the continuous parameter space Θ is calculated.
Feasibility for each θ in the discretized space is then measured by training an RL best-response BR ( θ ) to completion and averaging final utility U p ( BR ( θ ) , θ ) over 7 seeds.
F λ is then determined using equation (1).
We measure feasible worst-case episode reward as min θ ∈F λ U p ( σ p , θ ) , averaging over 100 episodes for each value of θ .
We use this as our metric for robustness to the feasible set F λ .
5. L AVA W ORLD
The gridworld task “Lava World” consists of a small platform surrounded by lava, as depicted in
Figure 2 (a).
The adversary specifies a goal location θ that the protagonist needs to reach, however the protagonist does not observe the goal, and the goal can be placed in lava.
With an episode horizon of 20, the protagonist receives a reward of -1 for every timestep that it does not reach the goal.
If the protagonist moves into lava, the episode ends, and it receives a reward of -15 even if the goal is at that location.
The feasibility threshold for this task is λ = − 10 , thus making a parameter for this environment infeasible if the goal is put in lava and feasible otherwise.
We use DDQN (Van Hasselt et al., 2016) to train protagonist RL policies.
Worst-case protagonist episode reward among all values in the feasible set θ ∈ F λ is shown as a function of PSRO iterations for FARR and baseline objectives in Figure 2 (c).
The minimax objective fails because the adversary learns to always suggest infeasible lava goals, and the protagonist learns to immediately jump in lava rather than waste time searching for a goal in non-lava cells.
Likewise, domain-randomization fails because the majority of goals are infeasible lava goals, so in order to optimize the average-case, the protagonist learns the same suboptimal behavior as it does with minimax.
The regret objective produces a distribution of both feasible and infeasible goals where non-lava goals are the majority, however this distribution is not an adversarial robust NE with respect to the set of feasible goals, so worst-case performance with the regret objective is still not optimal.
FARR penalizes the adversary for suggesting the infeasible lava goals and otherwise provides base robust adversarial RL utilities for feasible goals, thus resulting in a protagonist that maximizes worst-case reward among the actual feasible non-lava goal set F λ .
5. M U J O C O
We compare FARR and other objectives on three MuJoCo control tasks, HalfCheetah, Walker2D, and Hopper using PPO (Schulman et al., 2017) to train protagonist RL policies.
In each of these environments, the adversary specifies parameters θ =
( α, β ) where α ∈ (0 , 10] , β ∈ (0 , 10] for a beta distribution B ( α, β ) used to sample from and generate 1D horizontal perturbing forces every
[Figure]
timestep that are applied to the torso of the protagonist’s robot.
The adversary has the ability to specify infeasible distributions of forces which make accruing reward in each task virtually impossible.
We conduct experiments with each MuJoCo environment using three different feasibility threshold values for λ , representing three different assumptions regarding the difficulty of test-time conditions that we wish to prepare for.
For HalfCheetah, in Figure 3 (a) we show worst-case protagonist reward among environment parameters in the feasible set F λ as a function of PSRO iterations with λ values {− 1000 , 0 , 1000 } .
The same is shown for Hopper with λ ∈ { 200 , 400 , 600 } in Figure 3 (b), and for Walker2D with λ ∈ { 200 , 400 , 600 } in the appendix due to space limitations.
Across different values for λ in each of these environments, we see that FARR is able to train an agent which maximizes worst-case reward under the ground-truth F λ by penalizing the adversary to prevent it from providing infeasible variations.
Domain randomization and regret provide training distributions unconditioned on λ or any notion of F λ , which are only sometimes appropriate for robust performance as seen for λ
= 600 in Figure 3 (b) with domain randomization and regret and for λ = 1000 in Figure 3 (a) with regret.
Otherwise, domain randomization and regret result in agents exploitable to some configuration in F λ .
Likewise, minimax consistently provides insurmountable conditions to the protagonist, resulting in failed learning and highlighting the need for methods like
FARR to automatically limit adversary abilities in robust RL.
D ISCUSSION AND F UTURE W ORK
We present FARR, a novel robust RL problem formulation and two-player zero-sum game objective in which we consider an underspecified environment allowing infeasible conditions and we train a protagonist to be robust only to the tasks which are feasible.
By solving for approximate Nash equilibrium under the FARR objective using PSRO, we demonstrate that this method can produce a robust agent even when the adversary is allowed to specify parameters which make sufficient performance at a task impossible.
A limitation and avenue for future work is that our current method for optimizing FARR does not directly optimize the adversary, instead relying on random search and the PSRO restricted game solution to provide an optimal mixed strategy.
In future work, if high-dimensional joint adversary best-responses with feasibility estimates can be sample-efficiently optimized, FARR can provide a prescribable solution to avoid the manual creation of complex rules to limit robust RL adversaries in higher-dimensional sim-to-real configuration spaces.