ERMAS: L EARNING P OLICIES R OBUST TO R EALITY G APS IN M ULTI -A GENT S IMULATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Policies for real-world multi-agent problems, such as optimal taxation, can be learned in multi-agent simulations with AI agents that emulate humans.
However, simulations can suffer from reality gaps as humans often act suboptimally or optimize for different objectives (i.e., bounded rationality).
We introduce (cid:15) -
Robust Multi-Agent Simulation (ERMAS), a robust optimization framework to learn AI policies that are robust to such multi-agent reality gaps.
The objective of ERMAS theoretically guarantees robustness to the (cid:15) -Nash equilibria of other agents – that is, robustness to behavioral deviations with a regret of at most (cid:15) .
ERMAS efﬁciently solves a ﬁrst-order approximation of the robustness objective using meta-learning methods.
We show that ERMAS yields robust policies for repeated bimatrix games and optimal adaptive taxation in economic simulations, even when baseline notions of robustness are uninformative or intractable.
In particular, we show ERMAS can learn tax policies that are robust to changes in agent risk aversion, improving policy objectives (social welfare) by up to 15% in complex spatiotemporal simulations using the AI Economist (Zheng et al., 2020).
I NTRODUCTION
Reinforcement learning (RL) offers a tool to optimize policy decisions affecting complex, multiagent systems; for example, to improve trafﬁc ﬂow or economic productivity.
In practice, the need for efﬁcient policy evaluation necessitates training on simulations of multi-agent systems (MAS).
Agents in these systems can be emulated with ﬁxed behavioral rules, or by optimizing for a reward function using RL (Zheng et al., 2020).
For instance, the impact of economic policy decisions are often estimated with agent-based models (Holland & Miller, 1991; Bonabeau, 2002).
This commonly introduces a reality gap as the reward function and resulting behavior of simulated agents might differ from those of real people (Simon & Schaeffer, 1990).
This becomes especially problematic as the complexity of the simulation grows, for example, when increasing the number of agents, or adding agent affordances (Kirman, 1992; Howitt, 2012).
As a result, policies learned in imperfect simulations need to be robust against reality gaps in order to be effective in the real world.
We introduce (cid:15) -Robust
Multi-Agent Simulation (ERMAS), a robust optimization framework for training robust policies, termed planners , that interact with real-world multi-agent systems.
ERMAS trains robust planners by simulating multi-agent systems with RL and sampling worst-case behaviors from the worst-case agents.
This form of multi-agent robustness poses a very challenging multilevel (e.g., max-min-min) optimization problem.
Existing techniques which could be applied to
ERMAS’s multi-agent robustness objective, e.g., naive adversarial robustness (Pinto et al., 2017) and domain randomization (Tobin et al., 2017; Peng et al., 2018), are intractable as they would require an expensive search through a large space of agent reward functions.
Alternative frameworks improve robustness, e.g., to changes in environment dynamics, observation or action spaces (Pinto et al., 2017; Li et al., 2019; Tessler et al., 2019), but do not address reality gaps due to reward function mismatches, as they use inappropriate metrics on the space of adversarial perturbations.
To solve this problem, ERMAS has three key features: 1) It formulates a multi-agent robustness objective equivalent to ﬁnding the worst case (cid:15) -Nash equilibria.
2) It optimizes a tractable dual problem to the equivalent objective.
3) It approximates the dual problem using local solution concepts and ﬁrst-order meta-learning techniques (Nichol et al., 2018;
Finn et al., 2017).
ERMAS ultimately yields policies that are robust to other agents’ behavioral deviations, up to a regret of (cid:15) .
We show that ERMAS learns robust policies in repeated bimatrix games by ﬁnding the worst-case reality gaps, corresponding to highly adversarial agents, which in turn leads to more robust planners.
We further consider a challenging, large-scale spatiotemporal economy that features a social planner that learns to adjust agent rewards.
In both settings, we show policies trained by ERMAS are more robust by testing them in perturbed environments with agents that have optimized for reward functions unused during ERMAS training.
This generalization error emulates the challenge faced in transferring policies to the real world.
In particular, we show ERMAS can ﬁnd AI Economist tax policies that achieve higher social welfare across a broad range of agent risk aversion objectives.
In all, we demonstrate ERMAS is effective even in settings where baselines fail or become intractable.
Contributions To summarize, our contributions are:
• We derive a multi-agent adversarial robustness problem using (cid:15) -Nash equilibria, which poses a challenging nested optimization problem.
• We describe how ERMAS efﬁciently solves the nested problem using dualization, trustregions, and ﬁrst-order meta-learning techniques.
• We empirically validate ERMAS by training robust policies in two multi-agent problems: sequential bimatrix games and economic simulations.
In particular, ERMAS scales to complex spatiotemporal multi-agent simulations.
R OBUSTNESS AND R EALITY G APS IN M ULTI - AGENT E NVIRONMENTS
Informally, we seek to learn a policy π p , which we term the planner , that interacts with an environment featuring N other agents.
The planner’s objective depends both on its own policy and the behavior of the agents in response to that policy, leading to a multi-agent RL problem in which the planner and agents co-adapt.
In practice, evaluating (and optimizing)
π p requires use of a simulation with agents that emulate those in the environment of interest (i.e. the real world), which might contain agents whose reward function differs from those used in the simulation.
Our goal is to train planner policies that are robust to such reality gaps.
Formally, we build on partially-observable multi-agent Markov Games (MGs) (Sutton & Barto, 2018), deﬁned by the tuple M :=
( S, A, r, T , γ, o, I ) , where S and A are the state and action spaces, respectively, and I are agent indices.
Since the MG played by the agents depends on the choice of planner policy, we denote the MG given by π p as M [ π p ] .
MGs proceed in episodes that last H + 1 steps (possibly inﬁnite), covering H transitions.
At each time t ∈ [0 , H ] , the world state is denoted s t .
Each agent i = 1 , . .
. , N receives an observation o i,t , executes an action a i,t and receives a reward r i,t .
The environment transitions to the next state s t +1 , according to the transition distribution T ( s t +1 | s t , a t ) .
1 Each agent observes o i,t , a part of the state s t .
Agent policies π
i are parameterized by θ i while the planner policy π p is parameterized by θ p .
The Nash equilibria of M [ π p ] are agent policies where any unilateral deviation is suboptimal:
ANE
[Equation]
where J i ( π , π p ) :
= E π ,π p (cid:104)(cid:80)
Ht
=
γ t r ( i ) t (cid:105) denotes the objective of agent i .
Hence, a rational agent would not unilaterally deviate from π ∈ ANE ( π p ) .
To evaluate a ﬁxed planner policy π p , we simply sample outcomes using policies π ∈ ANE ( π p ) .
Also optimizing π p introduces a form of two-level learning .
Under appropriate conditions, this can be solved with simultaneous gradient descent (Zheng et al., 2020; Fiez et al., 2019).
Robustness Objective As noted before, we wish to learn planner policies π p that are robust to reality gaps arising from changes in agent reward functions, e.g., when agents are boundedly rational.
We develop a robustness objective for the planner by formalizing such reality gaps as perturbations ξ i ∈ Ξ to agent objectives, where the uncertainty set Ξ :[-Π N +
-]
( S, A ) H → R is the space of possible perturbations.
We extend ANE ( π p , ξ ) to condition the Nash equilibria on perturbations ξ :
[Equation]
where τ i is a trajectory (sequence of state-action pairs).
Following Morimoto & Doya (2001), a robust planner optimizes its reward, subject to agents playing a perturbed Nash equilibrium ANE ( π p , ξ ) that maximally penalizes the planner:
[Equation]
Note that agent policies π ∈ ANE ( π p , ξ ) describes agents that optimize their own reward function, and we assume an adversary chooses ξ .
Bounded Uncertainty Set
There are two challenges with Equation 4.
First, if the adversary can arbitrarily set Ξ , the worst case is uninformative.
3 Second, depending on the complexity of Π , the uncertainty set Ξ may be high-dimensional and intractable to search.
We address these issues by upper-bounding the L ∞ norm of ξ i ∈ Ξ :
[Equation]
This uncertainty set is equivalent to the (cid:15) -equilibria of M [ π p ] :
[Equation]
Intuitively, (cid:15) parameterizes the size of the reality gap to which the planner is robust.
With this, the robustness objective over a bounded uncertainty set is:
[Equation]
Using ANE ( π p , (cid:15) ) replaces the problem of intractably searching through Ξ with searching through
ANE ( π p , (cid:15) ) , and thus merges the two nested min operations in Equation 4.
Conceptually, this transfers the worst-case search problem to the agents: Agent-Adv-Search agents ﬁnd an adversarial equilibrium; Planner-OPT optimizes the planner given adversarial agents.
ERMAS: R OBUST P OLICIES IN M ULTI -A GENT S IMULATIONS
We now introduce ERMAS, an efﬁcient optimization framework to solve the robustness problem in
Equation 7.
ERMAS proceeds in three steps.
First, it dualizes Equation 7 following constrained RL.
Second, it deﬁnes a trust-region for the uncertainty set ANE ( π p , (cid:15) ) , approximating the dual problem.
Finally, it uses ﬁrst-order meta-learning with trust-regions to solve the approximate dual problem.
See Appendix A.1 for the detailed Algorithm description.
Dualizing Agent-Adv-Search
The agent search problem in Equation 7 can be formulated similar to a constrained RL problem (Paternain et al., 2019), where the primary objective of the agents is to minimize the planner’s reward and the secondary objective is to maximize their own reward.
While conventional constrained RL enforces a constant lower bound in the constraint, e.g.,
J i ( π , π p ) ≥ C , we enforce a dynamic one: ∀ i ∈ [Equation]
N : J i ( π , π p ) ≥ J i ( π ∗ i , π − i , π p ) − (cid:15) , where π ∗ i is the optimal unilateral deviation for agent i : π [Equation] i
[Equation] , π [Equation]
Speciﬁcally, we dualize Agent-Adv-Search , i.e., Equation 4, as:
[Equation]
[Figure]
where λ are Lagrange multipliers.
We abuse notation to denote θ :=
[ θ 1 , . .
, θ N , θ p ] and J p ( θ ) :=
J p ( π , π p ) where θ are the parameters of π .
To solve Equation 8, the agents apply gradients:
[Equation]
(9) where θ (cid:48)
i ( θ ) is the parameters of the optimal unilateral deviation π ∗ i for agent i , i.e. the parameters that minimize local regret, which depends on the current policy parameters θ i .
λ i is updated as:
[Equation] [Equation]
Equation 8 still poses a challenge through the J i ( π ∗ i , π − i , π p ) terms, which correspond to unknown agent regret.
We now detail the efﬁcient approximation of the value and derivative of agent regret using local and meta-learning approximations, respectively.
Trust Regions using Local (cid:15) -equilibria
Estimating regret requires knowledge of the optimal unilateral deviation for agent i .
We can simplify this problem by proposing a reﬁnement of (cid:15) -equilibria inspired by the notion of local Nash equilibria in differentiable games (Ratliff et al., 2014).
Deﬁnition 3.1.
A strategy π is a local (cid:15) -Nash equilibrium if there exists open sets W i ⊂ Π N such that π i ∈ W i and for each i ∈ { 1 , . . .
, N } we have that J i ( π (cid:48)
i , π − i ) ≤ J i ( π ) + (cid:15) (cid:48) for all π (cid:48)
i ∈
W i \ { π i } , where (cid:15) (cid:48) :
= (cid:15) sup π (cid:48)
i ∈
W i KL ( π i || π (cid:48) i ) .
By instead performing Agent-Adv-Search on the local (cid:15) -Nash equilibria, we can limit the set of unilateral deviations to consider to a small trust region, Π η ( π ) :
[Equation] [Equation]
(12) where η > 0 deﬁnes the size of the trust region.
For small η , algorithms such as TRPO (Schulman et al., 2017) can be used to efﬁciently approximate optimal local deviations of π i , affording reasonable approximations of J i ( π ∗ i , π − i , π p ) .
First-Order Meta Learning Approximation
The full gradient in Equation 9 is also complicated by the need to estimate the derivative of local regret ∇ θ i
[ J i ( θ (cid:48)
i ( θ ) , θ − i ) − J i ( θ )] .
The second term maximizes the performance of the agent’s policy and is simply found with policy gradient.
The ﬁrst term is less straightforward: it minimizes the performance of the best agent policy in the current trust region.
We note that this ﬁrst term corresponds to a meta-learning gradient.
We follow REPTILE (Nichol et al., 2018) to obtain a ﬁrst-order approximation of a M -step meta-learning gradient:
[Equation]
[Figure]
where g i denotes the i th policy gradient in the direction of J i .
In practice, we scale this meta-learning term with the hyperparameter β as β < 1 incorporates a helpful inductive bias where maximizing agent reward leads to local maxima.
We can alternatively apply this gradient update periodically, to both mimic β < 1 and reduce computation overhead.
ERMAS
By solving the dual problem (Eq.
8), ERMAS yields robustness to (cid:15) -equilibria and, equivalently, uncertainty in agent objectives.
ERMAS solves the dual problem by combining trust regions and meta-learning techniques to estimate and differentiate agent regret.
Algorithm 1 and(Appendix A.1) describe an efﬁcient implementation of this procedure for nested policy learning.
E XPERIMENTAL V ALIDATION IN M ULTI -A GENT S IMULATIONS 4. ERMAS S OLVES THE I NNER L OOP : A GENT -A DV -S EARCH
Constrained Repeated Bimatrix Game We analyze the agent behaviors learned by
Agent-Adv-Search in a modiﬁed repeated bimatrix game (Figure 1a).
At each timestep t a row and a column player simultaneously choose from a ﬁnite number of actions.
Each pair of actions ( i, j ) corresponds to a cell in the agent payoff matrices r 1 ( i, j ) , r 2 ( i, j ) .
At the initial timestep t = 0 , the agents play a random pair of actions, corresponding to a random cell in the matrix.
At each subsequent timestep t > 0 , the agents can only choose an action adjacent to their previously selected action.
This can be interpreted as “movement” through the payoff matrix with the row player (Agent 1) controlling up-down movement and the column player (Agent 2) controlling left-right movement.
The planner observes the game and receives a payoff r p ( i, j ) .
Discovering (cid:15) -Equilibria
Due to its simplicity, we can explicitly detail the social dilemmas and game-theoretic dynamics in this game.
Figure 2 (left) shows that agents using independent RL discover the Nash equilibrium located in the top left (high planner reward, low agent reward).
There also exists a space of (cid:15) -equilibria characterized by high agent reward r , low agent regret
J i ( π ∗ i , π − i ) − J i ( π ) , and low planner reward J p ( π ) .
As (cid:15) increases, the agents’ objective provides more “slack” for the agents to deviate from the stable, Nash equilibrium in order to penalize the planner.
In essence, and as described by Equation 7, this adversarial behavior is key to the goal of learning a more robust planner.
However, these (cid:15) -equilibria are inherently unstable, in this case due to the social dilemmas embedded in the reward structure.
Agent regret, which tends to increase with (cid:15) (Figure 2, Middle), is a signature of this instability.
[Figure]
4. A LL THE C OMPONENTS OF ERMAS ARE N ECESSARY Dynamic or Frozen Lagrange multipliers λ i
The Lagrange multipliers λ balance the dual objectives of seeking stable agent equilibria and minimizing the planner’s welfare.
The λ are updated using local estimates of agent regret, as described in Equation 8.
Are these updates necessary?
Figure 2 (right) addresses this question.
It shows the equilibria learned by ERMAS across varying values of frozen λ .
Freezing λ reduces Equation 8 to agents learning while sharing rewards.
This can dramatically affect the equilibria discovered by ERMAS.
Larger values of λ mean agents antagonize the planner less and improve agent and planner utility.
However, the bottom right quadrant and its (cid:15) -equilibrium are never reached.
In fact, certain (cid:15) -equilibria are only reachable with dynamic λ .
Hence, with frozen λ the agents would be less effective in robustifying the planner.
First-order Approximation Using the ﬁrst order approximation of Equation 13 might slow learning, since we step “away” from the direction that the standard policy gradient suggests.
In contrast, we ﬁnd that the ERMAS update can signiﬁcantly speed up convergence.
Figure 3 depicts the learning curve of variants of ERMAS with varying weights, β , on the meta-learning term.
For instance, when β = 0 , the meta-learning term is completely dropped.
We ﬁnd that lines corresponding to larger values of β reach improved agent rewards half an order of magnitude faster than lower values of β .
However, larger values of β fail to reach the desirable region of adversarial (cid:15) -equilibria.
In both cases, very large ( β > 1 N ) and very small ( β ∼ 0 ) fail to fully reach and remain at the adversarial region.
However, an appropriate range for β gives rapid convergence and good asymptotic behavior.
4. S OLVING P LANNER -OPT U SING A GENT -A DV -S EARCH
We now show that ERMAS yields planners π p which are more robust to uncertainty in the agent’s reward function.
Speciﬁcally, we empirically validate that ERMAS can ﬁnd strong solutions to the nested optimization problems Agent-Adv-Search and Planner-OPT , in two simulations: repeated bimatrix games and economic simulations (see Figure 1).
Evaluation Workﬂow To measure planner robustness, we evaluate trained planners in test environments with a reality gap (i.e. reward parameterizations that differ from the training environment), containing test agents that are optimized for the test environment’s reward functions and that have not been seen by the planner.
We proceed as follows: 1) Train the agents and planner with ERMAS.
2) Fix the planner and transfer it to a test environment, then train new agents from scratch in the test environment with reward functions unseen by the planner.
3) Report the value of the planner objective after the new agents have converged.
[Figure]
Augmented Repeated Bimatrix Games We modify the bimatrix game to let the planner observe the game state, set an integer a ≥ 0 , and get reward ar p ( i, j ) − a 2 .
a scales the reward but adds a cost a 2 , which counters greed (always choosing large a ).
Since the agents control i, j , they can (intentionally) reduce the planner’s objective.
During evaluation, the agents receive r (cid:48) i
= r i − Qr p where Q ≤ 1 is a ﬁxed parameter that models a reality gap.
Q = 0 means no train/test reality gap.
Figure 4 shows the test performance of ERMAS against vanilla multi-agent RL policies learned without a reality gap ( Q train = 0 ), and a pessimistic setting Q train = 0 .
Naturally, MARL
Q train = 0 .
5 underperforms Q train = 0 at Q test < 0 .
3 , but keeps consistent performance throughout.
Agent-Adv-Search successfully ﬁnds the worst-case perturbation, and yields a planner policy at least as good as MARL Q train = 0 .
This shows that ERMAS can produce policies which are robust to multi-agent uncertainty and is a tractable adversarial robustness algorithm.
The baselines, domain randomization (DR) and risk robustness (RR), do not improve over vanilla robust multi-agent RL.
For DR, agents receive r (cid:48) i
= r i + σ , where σ ∼ U [ − 1 , 1] (recall r i ∈ [0 , 1] ).
Even in a simple 4-by-4 bimatrix game, there are 2304 =
16 states × 16 states × 9 actions possible reward values to perturb; a 2304-dimensional space is too sparse to cover uniformly.
Furthermore, in contrast to DR of visual inputs or dynamics, there is a natural latency to the effect of DR of agent rewards: randomization only has an effect if agents learn to adapt to their perturbed rewards.
RR makes the planner’s reward more concave, e.g., log r p .
However, RR does not yield the multi-agent robustness relevant to our problem.
4. ERMAS FOR L ARGE -S CALE T WO -L EVEL L EARNING : AI E CONOMIST
We train a robust AI Economist: a planner for optimal taxation that acts in a spatiotemporal economic simulation, see Zheng et al.
(2020) for a detailed description.
This setting is very challenging given the scale of the simulation and agent affordances.
In particular, adversarial and randomization baselines do not scale to this setting.
In this simulation, agents optimize their expected utility
E π ,π p (cid:104)(cid:80) Tt =
r
i,t (cid:105) , where the utility r i,t is a function of labor l i,t and post-tax endowments ˜ x i,t : ˜
z
i,t = z i,t − T ( z i,t ) , ˜ x i,t = (cid:88) t (cid:48) ≤ t ˜ z
i,t , r i,t (˜ x i,t , l i,t ) =
˜ x
1 − η i,t −
[Equation]
where η sets the degree of risk aversion (higher η means higher risk aversion).
Agents earn income z i,t and pay taxes T ( z i,t ) , which are set by the planner.
The planner optimizes social welfare swf :
[Equation]
a combination of equality (Gini, 1912) and productivity.
Figure 5 shows the performance of ERMAS tax policies along with baseline tax policies (Saez,
US Federal) and an AI Economist (MARL).
All models are trained on agents with η = 0 . 23 .
We
[Figure]
replicate Zheng et al.
(2020): MARL outperforms the Saez tax.
However, MARL fails to outperform
Saez when testing with η < 0 .
22 and η > 0 . 28 .
In contrast, ERMAS yields consistent gains across the depicted range of η .
This shows that ERMAS’s tax policy can improve swf even if agent risk aversion signiﬁcantly increases or decreases (e.g., due to economic cycles or exogenous shocks).
In fact, ERMAS outperforms the baseline AI Economist for the original setting of η = 0 .
This suggests the robustness and performance do not necessarily pose a zero-sum game: ERMAS can ﬁnd equilibria with high performance and strong generalization.
Single-agent robust reinforcement learning algorithms similarly yield performance improvements even in the absence of environment perturbations—robust learning simply yields better policies Pinto et al.
The Sim2Real problem considers how reality gaps, i.e., mismatches between a simulation and reality, can (negatively) impact RL policies.
Robust performance across reality gaps has been studied when applying deep RL to robotic control with visual inputs (James et al., 2019; Christiano et al., 2016; Rusu et al., 2016; Tzeng et al., 2015).
To close reality gaps, domain adaptation aims to transfer a simulated distribution (of the world) to the real world (Higgins et al., 2017; Kim et al., 2019).
In comparison, the study of reality gaps in multi-agent RL is relatively nascent (Suh et al., 2019; Nachum et al., 2019).
Multi-Agent Robustness Morimoto & Doya (2001) proposed robust RL for the worst-case adversarial objective (i.e., a max-min optimization problem), inspired by H ∞ control, and provided analytical solutions in the linear setting.
The robust RL problem can be solved using an adversarial player that chooses perturbations.
Rather than the worst-case, Pinto et al.
(2017) optimized for the α -quantile of reward, using a conditional value-at-risk (CVAR) interpretation of adversarial robustness.
We consider reward perturbations.
Other work has studied perturbing transition matrices, observation spaces, and action probabilities, which are relevant to robotics and control applications (Pinto et al., 2017; Tessler et al., 2019;
Hou et al., 2020; Li et al., 2019).
The robust RL literature has not considered the nested RL setting at the scale of our work.
In the multi-agent setting, agents do not necessarily optimize for the same (robustness) objective.
We use one-sided robustness in a multi-level setting: only the planner aims to be robust.
This setting is signiﬁcantly more difﬁcult: previous work mostly assumed all agents act robustly (Li et al., 2019).
See Appendix A.2 for additional related work.
F UTURE W ORK
Future work could extend ERMAS to reality gaps due to different causes, such as changes in transition dynamics, agent affordances, number of agents, etc, which might require more reﬁned approximations to the planner’s robustness objective.
Furthermore, it would be interesting to apply ERMAS to problems that feature more general solution concepts, e g. Bayesian Nash equilibria, correlated equilibria, Walrassian equilibria, and others.
} A. A DDITIONAL R ELATED W ORK
Robust Mechanism Design Mechanism design aims to design reward functions that have desirable equilibrium features, such as stability (Nash equilibria), truthfulness (second-price auctions), and others (Myerson, 2016).
Two-level RL can be seen as designing a mechanism (reward function set by a planner) with adaptive agents.
As such, in the two-level RL setting ERMAS learns a robust mechanism.
Robust mechanism design (Bergemann & Morris, 2005) has studied how mechanisms can be effective even when the environment is imperfectly known.
However, it is often very hard to derive analytical solutions for (robust) mechanisms.
Instead, D¨utting et al.
(2019) used deep learning to learn (close to) optimal auctions, while learning optimal taxes by the AI Economist (Zheng et al., 2020) can be seen as adaptive mechanism design using two-level RL.
In this sense, ERMAS learns a robust adaptive mechanism.
Behavioral Economics Behavioral economics has studied human reward functions and how human behavior differs from that of idealized agents optimizing simple utility functions (Pesendorfer, 2006).
Simon (1976) decomposed bounded rationality into substantive and procedural rationality.
Substantively rational agents have incorrectly reward function estimates (e.g., humans might be more risk-averse than a simulated agent) or who require richer games to describe their them.
Procedurally rational agents are irrational agents which violate rationality assumptions or do not maximize for their reward function (Simon & Schaeffer, 1990).