G ENERALIZATION G AMES IN R EINFORCEMENT L EARNING
A BSTRACT
Many subﬁelds have emerged in reinforcement learning (RL) to understand how distributions of training tasks affect an RL agent’s ability to transfer learned experiences to one or more evaluation tasks.
While the ﬁeld is extensive and evergrowing, recent research has underlined that variability among the different methods is not as signiﬁcant.
We leverage this intuition to demonstrate how current methods for generalization in RL are specializations of a general framework.
We obtain the fundamental aspects of this formulation by rebuilding a Markov Decision Process (MDP) from the ground up by resurfacing the game-theoretic framework of games against nature.
The two-player game that arises from considering nature as a complete player on this formulation explains how existing approaches rely on learned and randomized dynamics and initial state distributions.
We develop this result further by drawing inspiration from mechanism design theory to introduce the role of a principal as a third player that can modify the payoff functions of the decision-making agent and nature.
The main contribution of our work is the complete description of the Generalization Games for Reinforcement Learning, a multiagent, multiplayer, game-theoretic formal approach to study generalization methods in RL. The games induced by playing against the principal extend our framework to explain how learned and randomized reward functions induce generalization in RL agents.
We offer a preliminary ablation experiment of the different components of the framework and demonstrate that a more simpliﬁed composition of the objectives that we introduce for each player leads to comparable, and in some cases superior, zero-shot generalization performance than state-of-the-art methods while requiring almost two orders of magnitude fewer samples.
I NTRODUCTION
In reinforcement learning (RL), the term generalization has denoted the practice of introducing function approximation to reduce the intractability of problems with large state and action spaces (Sutton, 1996; Pack Kaelbing et al., 1996) or to designate RL agents’ capacity to transfer learned experiences to one or more evaluation tasks (Whiteson et al., 2011).
Recently, the ﬁelds of curriculum learning Portelas et al.
(2020), open-ended learning (Wang et al., 2019), continual learning (Khetarpal et al., 2020), meta-learning (Duan et al., 2016), simulation-to-reality transfer (Sadeghi & Levine, 2016), or unsupervised environment design (Dennis et al., 2020) have emerged to understand how distributions of training tasks affect an RL agent’s performance in unseen environments.
While the ﬁeld is extensive and ever-growing, recent surveys have pointed out that the amount of variability among the different approaches is not as signiﬁcant (Portelas et al., 2020; Kirk et al., 2021).
For instance, Portelas et al.
(2020) managed to classify more than 30 approaches along three axes of variation: the intended use of the algorithm, aspects of the problem the algorithm controls, and the optimization objective it proposes.
While the ﬁrst axis is subjective and depends on the intended application of the algorithm, variations on the degree of control over the transition dynamics, initial state distributions, and reward functions in Markov Decision Process (MDP) problems indicate the existence of a shared underlying structure of the problem.
We leverage this intuition to develop our core hypothesis that current methods for generalization in RL are specializations of a general framework .
To uncover the fundamental aspects of this problem, we rebuild from the ground up the MDP framework (Puterman, 2005) by resurfacing and expanding games against nature (Milnor, 1951; Papadimitriou, 1985), a game-theoretic approach to decision-making under uncertainty.
This formulation acknowledges the existence of a nature player that may either be a single entity or the composition of multiple ones.
Those entities may possess agency and the capacity to learn and adapt and while doing so, can reshape the problem landscape of a decision-making agent (Leibo et al., 2019; Baker et al., 2020).
We show that, under some assumptions, an MDP problem is a particular case of a repeated game a decision-making agent plays against a selﬂess nature player with unknown payoffs.
At this juncture, we diverge from the traditional literature to make nature’s player a ﬁrst-class citizen within a complete game against nature . In this two-player game, both players aim to maximize their respective non-constant payoff (or reward) functions.
The introduction of nature’s payoff function prompts, under the same assumptions as before, an MDP for nature’s player when it repeatedly interacts with the decisionmaking agent.
We argue that this dual formulation explains how existing algorithms rely on learned or randomized variations of transition dynamics and initial state distributions of MDP problems to induce generalization in RL agents.
Moreover, drawing inspiration from the ﬁeld of mechanism design (Vickrey, 1961), we consider the emergence of complete games against nature that assume transformations on the payoff functions of both nature and the decision-making agent.
To account for these variations, we introduce the ﬁgure of the principal (Myerson, 1983), a third player that can manipulate both nature and the decisionmaking agent payoff structures.
We ﬁrst consider the role played by mechanisms where the principal is an entity external to the design process.
In this context, we argue that the extensive collection of RL problems (Bellemare et al., 2012; Brockman et al., 2016) are, under the equivalences we establish, mechanism-induced repeated games against nature.
Consequently, we reason that problem designers recreate the role of the principal by predeﬁning the payoff structure of the decision-making agent.
Then, to form a complete game against nature from these existing problems, it sufﬁces to choose the appropriate payoff functions for nature’s player.
Inspired by the types of evolutionary pressures nature poses over living agents and in line with existing methods, we formulate cooperative and competitive complete agent-nature games.
Furthermore, inspired by adaptive approaches to mechanism design (Conitzer & Sandholm, 2002; Vorobeychik et al., 2007; Baumann et al., 2018), we describe games against the principal , a threeplayer game between the decision-making agent, nature, and the principal.
In these games, the principal’s objective is to discover the optimal outcome function from a restricted set that maximizes either player’s total payoff.
We propose to set the payoff structure of the principal such that it adopts a dictatorial role (Sandholm, 2003) and sides with either the decision-making agent or nature’s player.
In combination with the cooperative and competitive games induced by manuallydesign mechanism over RL problems, the optimization of the dictatorial principal objective explains how existing algorithms rely on learned or randomized reward functions to induce generalization in RL agents.
Our proposal of a general framework for generalization emerges from considering the consequences rendered over a RL agent by the non-stationarity induced by nature and the principal players learning to optimize their respective payoff functions.
In our presentation of the framework of Generalization Games for Reinforcement Learning , we concentrate on the three RL problems structured as repeated non-differentiable games against the principal, where each agent learns and execute a mixed and parametric strategy.
The core contribution of this work is to demonstrate how this multiagent, multiplayer, game-theoretic framework provides a formal framework to study existing or propose novel methods leading to in RL.
Notably, we show that the axes of control and optimization in the taxonomy introduced by Portelas et al.
(2020) innately emerge from our formulation.
We leverage the experimental setting proposed in the context of unsupervised environment design (Dennis et al., 2020).
Furthermore, we include an exploratory ablation experiment that explores the effects that coalitions with a dictatorial principal have on the ﬁnal zero-shot generalization of an RL agent under adversarial and cooperative objectives.
Through these experiments, we show how the more simpliﬁed composition of the objectives in our ablation study leads to comparable and, in some cases, superior zero-shot generalization than existing state-of-the-art methods (Jiang et al., 2021) at almost two orders of magnitude fewer samples.
G AMES A GAINST N ATURE AND R EINFORCEMENT L EARNING
In this section, we resurface and expand an alternative formulation of the problem of decisionmaking under uncertainty, games against nature (Milnor, 1951; Papadimitriou, 1985; Biswas, 1997;
LaValle, 2006).
We describe how the MDP framework (Puterman, 2005) is a particular case of a repeated game against nature under the assumptions of perfect information and sequential natureagent interactions.
Likewise, we show that from a game-theoretic perspective, a repeated game against nature is an incomplete game where nature does not have a known payoff (or reward) function.
In consequence, we introduce complete games against nature to consider that nature’s player may also optimize for a non-constant payoff.
We show that each complete game against nature then also induces a dual MDP problem for nature’s player.
2. G AMES A GAINST N ATURE
The game against nature framework presents a game-theoretic approach to decision theory and decision-making problems where a selﬁsh and rational agent, the decision-making agent (the agent ), aims to maximize its payoff against a disinterested opponent ( nature ) that plays at random.
We derive a more formal deﬁnition of a game against nature from the works of Milnor (1951), Papadimitriou (1985), Biswas (1997), and LaValle (2006) as follows:
Deﬁnition 1 (Game Against Nature)
A game against nature is an asymmetric two-player game
G = (cid:104)S , A , (cid:96) π , (cid:96) ρ (cid:105) between a player π (the agent) and player ρ (nature) with action spaces a ∈ A and s ∈ S .
In this game, the player π maximizes its payoff function (cid:96) π : S ×
A → R against a nature player ρ that plays at random with disinterested and constant payoff (cid:96) ρ : S ×
A → c .
The asymmetry in this game arises from the different action spaces of the two players.
Furthermore, each player outcome is determined by their strategies π ∈ ∆( A ) and ρ ∈ ∆( S ) deﬁned in the space of discrete probability distributions ∆( · ) over their respective action spaces A and S .
The traditional formulation of games against nature identiﬁes several principles of choice that prescribe how the decision-making agent should select its strategy under uncertainty over nature’s strategy (Milnor, 1951; Biswas, 1997; LaValle, 2006).
From the different principles, the principles of insufﬁcient reason Milnor (1951), minimax regret (Savage, 1951) and expected utility (Von Neumann & Morgenstern, 1944; Savage, 1954) are the more widespread.
In our discussion, we focus on the principle of expected utility as it is most commonly used in the modern literature in decision-making .
Under this principle, the decision-making agent π should select actions a ∈ A that maximize its expected payoff [Equation] nature’s mixed strategy ρ .
However, seldom if ever, games are played once.
In many instances, players repeatedly interact on the same base or stage game, a situation modelled by repeated games .
Similarly, a decision-making agent and nature may continually interact, leading to a repetition of the base game G .
We formally deﬁne a repeated game against nature as follows:
Deﬁnition 2 (Repeated Game Against Nature)
A repeated game against nature G t is a iterated version of a base or stage game against nature G , with t ≥ 0 interactions between an agent π and the nature player ρ .
In this game, the history of interactions τ = { s t − 1 , a t − 1 , . . . , a 0 , s 0 } ∈ ( S × A ) t conditions the players’ payoffs functions (cid:96) tπ : ( S × A )
t → R and (cid:96) tρ : ( S × A ) t → c , respectively.
Under the expected utility principle, the decision-making agent expected outcome L π should be deﬁned for repeated games against nature such that:
[Equation]
where p ( τ | π, ρ ) denotes the probability distribution over feasible t -long interaction histories τ ∈ ( S × A ) t conditioned also by the nature player mixed strategy ρ .
There are several sensitive choices for payoff functions (cid:96) t ( τ ) for repeated players interactions in ﬁnite and inﬁnite repeated games including average, limit of means, and discounted formulations (Abreu, 1988).
We present a more detailed discussion of these functions in Appendix B.1.
2. MDP AND R EPEATED G AMES A GAINST N ATURE
We are interested in connecting the games against nature formalism with the MDP framework employed in RL literature.
To do so, we extend the deﬁnition of repeated game against nature in Def. 2 to consider perfect information and sequential nature-agent interactions.
First, repeated agent-nature interactions enable considering the amount of information available to each player’s decision-making.
We are interested in perfect information games where players can observe their past actions and those of their opponents to introduce repeated perfect information games against nature G t where both the agent and nature strategies are conditioned on past interactions.
Formally,
Deﬁnition 3 (Repeated Perfect Information Game Against Nature)
A repeated perfect information game against nature G t is a repeated game against nature where each player strategy π : ( S ×
A ) t → ∆( A ) and ρ : ( S × A ) t → ∆( S ) is conditioned on the history τ ∈ ( S × A ) t such that, at interaction time t , their actions are drawn from their mixed strategies a t ∼ π ( ·| τ ) and s t ∼ ρ ( ·| τ ) , respectively.
In general, repeated games against nature G t model a game between the decision-making agent and nature where both players choose their action simultaneously.
Simultaneous repeated games against nature with perfect information is a suitable realization to study real-time RL problems (Ramstedt & Pal, 2019).
However, we focus on the more widespread practical assumption that in sequential decision-making problems nature moves ﬁrst .
For these cases, we extend repeated perfect information games against nature to consider sequential agent-nature interactions where nature’s player starts the game, and the decision-making agent acts upon observing nature’s previous action.
More formally,
Deﬁnition 4 A sequential perfect information game against nature is a perfect information game against nature where nature plays ﬁrst with action s 0 ∼ ρ drawn from initial mixed strategy ρ 0 ∈ ∆( S ) .
Thereafter, nature and the agent plays with strategies ρ : ( S × A ) t → ∆( S ) and π :
S × ( S × A ) t → ∆( A ) deﬁned such that, at interaction time t , the agent actions are drawn from the mixed strategy a t ∼ π ( ·| s t , τ ) according to nature’s last action s t ∼ ρ ( ·| τ ) and the history τ of past interactions.
The assumptions of perfect information and sequential interaction allow us to present Markov Decision Processes MDP as a particular repeated games against nature.
Note that, an MDP problem
M = (cid:104)S , A , ρ, r, ρ 0 (cid:105) with state space S , action space A , transition dynamics ρ : S ×
A → P ( S ) , reward function r : S ×
A → R and initial state distribution ρ 0 : P ( S ) is a sequential game a decision-making agent π plays against nature’s strategy ρ .
If ρ 0 denotes nature’s player unconditioned strategy at interaction time t = 0 (nature’s ﬁrst move) the decision-making agent plays with strategy or policy π : S → P ( A ) .
Under Markovian assumptions (Puterman, 2005), nature’s actions s t represent sufﬁcient statistics of past nature-agent interactions τ .
Thus, the agent actions are drawn from its mixed strategy a t ∼ π ( ·| s t ) conditioned on nature’s last action s t .
Furthermore, nature’s player actions for interaction times t ≥ 1 are drawn from a mixed strategy s t ∼ ρ ( s t | s t − 1 , a t − 1 ) (the transition dynamics) and assumed independent of the rest of the interaction history τ given the last step actions s t − 1 and a t − 1 .
Likewise, under this formulation, the reward function r ( s, a ) is equivalent to one-shot payoff function (cid:96) π ( s, a ) of the base game G .
The objectives for an average or discounted reward MDP problem (Puterman, 2005) are easily derived from the repeated payoff structures in inﬁnitely repeated games against nature we deﬁne on Sec.
B.1.
2. C OMPLETE G AMES A GAINST N ATURE
The framework of games against nature that we have described thus far proposes games that are incomplete in the sense that nature’s player payoff function is constant and unknown.
We extend the original formulation to consider a nature player that optimizes for a non-constant payoff function under one of principles of choice for decision-making under uncertainty.
More formally,
Deﬁnition 5 (Complete Game Against Nature)
A complete game against nature is an extension of a game against nature G =
(cid:104)S , A , (cid:96) π , (cid:96) ρ (cid:105) where the nature player ρ plays with strategy ρ : A → ∆( S ) and maximizes a non-constant payoff (cid:96) ρ : S ×
A → R .
As before, we focus on the principle of expected utility principle and deﬁne nature’s expected payoff to be L ρ = E a ∼ π s ∼ ρ [ (cid:96) ρ ( s, a )] against the mixed strategy π of the decision-making agent.
Repeated nature-agent interactions prompt a repeated game for which equivalent MDP and RL problems emerge for nature’s player under a similar expected objective:
[Equation]
Holding the assumptions of perfect information and sequential nature-agent interactions as before, a nature player sequential, repeated, perfect information game against the agent induces the following
MDP problem:
Deﬁnition 6 (Nature MDP)
A nature MDP problem N =
(cid:104)A , S , π, r, π 0 (cid:105) is an MDP with state space A , action space S , transition dynamics π : S → ∆( A ) , reward function r : A × S → R , and initial state distribution π 0 :
∆( A ) is the sequential game a nature agent
ρ plays against a decision-making agent π .
Thus, every complete game induces a pair ( M , N ) of dual MDPs (see Fu et al.
In this case, the initial state distribution π 0 of N is deﬁned in terms of the initial state distribution ρ 0 of the decision-making agent MDP M such that π 0 ( a 0 ) =
(cid:80) s p 0 ( s 0 )
π ( a 0 | s 0 ) .
Moreover, nature’s policy ρ is the transition dynamics ρ ( s t | s t − 1 , a t − 1 ) of the decision-making agent’s
MDP and, conversely, the decision-making agent policy or strategy π is the transition function π ( a t | s t ) of nature’s
MDP.
Also, the reward functions on each MDP are the payoff functions (cid:96) ρ and (cid:96) π on the complete base game G .
M ECHANISM I NDUCED G AMES A GAINST N
ATURE
In this section, drawing inspiration from the ﬁeld of mechanism design (Vickrey, 1961) (see
Sec.
B.2), we consider the emergence of complete games against nature that assume transformations on the payoff functions of both nature and the decision-making agent.
We introduce the ﬁgure of the principal . This third player can manipulate nature and the decision-making agent payoff structures.
Furthermore, motivated by adaptive approaches to mechanism design, we describe games against the principal , a three-player game between the decision-making agent, nature, and the principal.
In these games, the principal’s objective is to discover the optimal outcome function from a restricted set that maximizes either player’s total payoff.
3. M ECHANISM D ESIGN FOR G AMES A GAINST N ATURE
A mechanism design problem for games against nature , a principal deﬁnes the base game G , both players action spaces S and A , and their modiﬁed outcomes ˜ (cid:96) ρ and ˜ (cid:96) π .
We restrict our interest to cardinal outcomes (e.g., money, years in prison, or scalar reward) and structured outcome spaces
O = O 1 × . . .
× O n .
We assume that utilities (cid:96) i : O i → R are of the form (cid:96) i
= o i where the i − th player maximizes the outcome o i provided by the mechanism.
Under these assumptions, we deﬁne the problem of designing a mechanism for nature-agent game as follow:
Deﬁnition 7 (Mechanism of a Game Against Nature) Let R =
( r ρ , r π ) be the set of possible outcomes for nature and the decision-making agent, the mechanism of a game against nature for
R is a tuple F =
(cid:104)R , A × S , r (cid:105) where A and S are the decision-making and nature player action spaces and r : A × S → R is the mechanism outcome function.
Thus, any mechanism F deﬁned this way will induce a nature-agent game G F with the following structure:
Deﬁnition 8 (Mechanism-Induced Game Against Nature)
Given a mechanism of a game against nature F =
(cid:104)R , A × S , r (cid:105) and the utility functions (cid:96) π : R → R and (cid:96) ρ : R → R , a mechanisminduced game against nature is a tuple G F = (cid:68) A × S , ˜ (cid:96) π , ˜ (cid:96) ρ (cid:69) where A and S are the action spaces of the decision-making agent and nature, and the utility functions ˜ (cid:96) π : R → R and ˜ (cid:96) π : R → R are deﬁned such as: [Equation]
[Equation]
for outcomes r π and r ρ and the outcome function r of the mechanism F .
This deﬁnition assumes that both players have linear utility functions that ignore the opponent’s outcome.
Furthermore, the notation for the outcome function r : A×S → R is intentionally chosen to highlight that, in a mechanism-induced game against nature, the outcome function establishes the payoff (or reward) functions
˜ (cid:96) ρ and ˜ (cid:96) π for a stage complete game against nature G F
the induced dual
MDP problems.
3.1. M ANUALLY -D ESIGNED M ECHANISMS
At ﬁrst, we are interested in mechanisms where the principal is an entity external to the design process. Our interest stems from the connection among the extensive collection of RL problems (Bellemare et al., 2012; Brockman et al., 2016), their underlying MDP, and equivalence to repeated games against nature we introduce in Sec. We reason that, in this context, problem designers fulﬁll the role of the principal.
They deﬁne the action spaces for both nature and the decision-making agent and the payoff structure of the latter. If one establishes an appropriate payoff function for nature, it is possible to convert every single-agent MDP into a complete game against nature. Inspired by the types of evolutionary pressures nature poses over living agents, in line with existing methods and with the principles of choice of minimax and maximax (Milnor, 1951; LaValle, 2006), we formulate cooperative and competitive complete agent-nature games.
Deﬁnition 9 (Adversarial Game Against Nature) An adversarial game against nature G − =
(cid:104)S , A , (cid:96) π , (cid:96) ρ (cid:105) is a zero-sum game the decision-making agent plays against nature such as nature non-constant maximizes a payoff function (cid:96) ρ = − (cid:96) π , or in other words, the total outcome of the game G − ( π, ρ )
=
(cid:96) π
+ (cid:96) ρ = 0 .
In this adversarial, zero-sum game both player’s optimization objectives can be succinctly written down as:
[Equation]
On the other hand, we can also deﬁne a game where both players cooperate to maximize the payoff function of the decision-making agent:
Deﬁnition 10 A cooperative game against nature G + =
(cid:104)S , A , (cid:96) π , (cid:96) ρ (cid:105) is a non-zero game where the decision-making agent plays with nature’s player that aims to maximize a non-constant payoff (cid:96)
ρ = (cid:96) π .
This cooperative game objective can be succinctly written down as:
[Equation]
3.1. A UTOMATICALLY -D ESIGNED M ECHANISMS
Motivated by previous adaptive approaches to mechanism design (Vorobeychik et al., 2007; 2012;
Baumann et al., 2018), we are interested in incorporating the principal as a player, with strategy ν , a payoff function (cid:96) v (see Sec.
B.2).
We construct a three-player game between the decision-making agent, nature’s player, and a principal.
More formally,
Deﬁnition
A game against a principal is a tuple G ν = (cid:68) R , A , S , (cid:96) ν ,
˜ (cid:96) π , ˜ (cid:96) ρ (cid:69) that denotes an asymmetric three-player game between a decision-making agent π with strategy π : S → ∆( A ) , nature’s player ρ : A → ∆( S ) , and a principal player ν with strategy r : A × S → R that can manipulate the other players payoff functions ˜ (cid:96) π and ˜ (cid:96) ρ in beneﬁt of maximizing its own payoff function (cid:96) ν whose objective is to ﬁnd the optimal mechanism r ∗ ∈ R over a space of feasible mechanisms R , such that:
[Equation]
Coalitional Games Similarly, we explore adversarial and cooperative games against a dictatorial principal ( (cid:96) ν
= min i ˜ (cid:96) i = k ) (Sandholm, 2003) who forms a coalition with either nature or the decision making-agent.
From the perspective of the decision-making agent, the core of RL and MDP problems, in a cooperative nature-principal game , nature and the principal form a coalitional [Figure] both methods with almost two orders of magnitude fewer training samples.
The performance on Maze and Labyrinth degrades, thus requiring further investigation.
We describe the full experimental setup in Appendix A game (Myerson, 1991) where they cooperate to maximize nature’s player payoff ˜ (cid:96) ρ .
We can write down this objective more succinctly as:
[Equation]
On the other hand, in a cooperative agent-principal game , the decision-making agent and the principal player form a coalition that cooperates to maximize the decision-making payoff function ˜ (cid:96) π with the joint objective:
[Equation]
In the context of the three-player game against a principal G ν , coalitions formed with the principal provide a strategic advantage for the allied player.
For instance, in a cooperative mechanism-induced nature-principal game, if nature’s objective is adversarial (Eq. 5), a game against principal would have an objective where the nature-principal coalition minimizes the decision-making agent payoff:
[Equation]
On the other hand, if the nature-agent game is cooperative, a nature-principal coalition would result in an objective that maximizes the decision-making agent payoff:
[Equation]
While other combinations of coalitions are possible, we focus in the remainder of this work on the objectives in Eq.
10 and Eq.
11 to show the effects that concentrating the principal power on the nature’s player have over generating mechanism-induced game against nature, their equivalent RL and MDP problems, and decision-making adaptation of strategy learned on these coalitional games. 3. P RELIMINARY R ESULTS
In Appendix A, we include an exploratory ablation experiment that explores the effects that coalitions with a dictatorial principal have on the ﬁnal zero-shot generalization of an RL agent under cooperative and cooperative objectives.
We leverage the experimental setting proposed in the context of unsupervised environment design (Dennis et al., 2020).
In this context, we further show (see Figure 1) how the more simpliﬁed composition of the objectives in our ablation study leads to comparable, and in some cases, superior zero-shot generalization to existing, more cumbersome state-of-the-art methods (Dennis et al., 2020; Jiang et al., 2021) with almost two orders of magnitude fewer samples.
G ENERALIZATION G AMES FOR R EINFORCEMENT L EARNING 4. L EARNING IN G AMES A GAINST N ATURE
Players’ iterations at playing a game allow them to adapt their strategies to face opponents.
In games against nature, a decision-making agent can adapt its strategy against an unknown nature’s player whether nature’s strategy remains stationary (same player’s strategy) throughout their repeated interactions or not.
Speciﬁcally, in repeated games against nature G t , the optimal strategy π ∗ of the decision-making agent should maximize the cumulative payoff functions over t -iterations of the base game, an objective that we can state as:
[Equation]
(12) where the payoff (cid:96) tπ is deﬁned as described in Sec.
B.1.
Note that the objective in Eq.
12 is the traditional RL objective with the exception that we make explicit the dependency on the nature player strategy.
Similarly, nature’s player optimal strategy in a repeated game against a decision-making agent G t is the cumulative payoff function (cid:96) tρ : ( S × A ) t → R similarly selected in consequence to the ﬁniteness of interaction length t (see Sec.
B.1).
Nature’s learning objective in complete games (Eq. 2) leads to ﬁnding an optimal strategy ρ ∗ as:
[Equation]
The objective is equivalent to an RL problem in nature’s
MDP.
A similar argument can be made for the principal agent.
4. G ENERALIZATION IN G AMES A GAINST N ATURE
Assume that the strategies of the nature’s player ρ φ : S ×
A → S , the decision-making agent and π θ : S → A and a principal r ω : S ×
A → R are parametrized by parameters φ, θ, ω respectively.
Further, consider the consequences rendered over a RL agent by the non-stationarity induced by nature and the principal players learning to optimize their respective payoff functions.
Our proposal of a Generalization Games for Reinforcement Learning concentrates on the three RL problems structured as repeated non-differentiable games against the principal, where each agent learns and execute a mixed and parametric strategy.
Deﬁnition 12 ( Generalization Game for Reinforcement Learning )
A Generalization Game for RL is a repeated non-differentiable optimization game against a principal G t =
(cid:68) R , A , S , (cid:96) ν , ˜ (cid:96) π , ˜ (cid:96) ρ (cid:69) with objective J ( φ, θ, ω ) among the nature player ρ φ , the principal r ω , the decision-making agent π θ were each may simultaneously optimize for non-constant cumulative payoff functions such that:
[Equation]
reﬂects the update rules equations of each strategy possibly optimized using Monte Carlo gradient estimation techniques (Mohamed et al., 2020).
Notice that the update function of the decision-making agent strategy π θ is the traditional RL problem.
In contrast, the updates rule for nature’s strategy ρ φ and that of the principal r ω introduce the variability over transition dynamics, initial state distribution (nature’s initial strategy), and reward functions (the principal strategy).
Moreover, an alternative for learning is to induce non-stationarity by drawing samples from distributions φ k ∼ p ( φ ) and ω m ∼ p ( ω ) for nature and the principal strategies (Tobin et al., 2017; Wang et al., 2019).
Combined with the adversarial and cooperative objectives we introduced in Sec. B.2, we account for the axis of optimization in Portelas et al.
taxonomy. Thus, the non-stationarity of learned or randomized strategies in these games makes the aspects of current generalization methods innately emerge from our formulation.
C ONCLUSIONS AND F UTURE W ORK
We studied the problem of generalization in RL and introduced the Generalization Games for Reinforcement Learning by leveraging and expanding the game-theoretic framework of games against nature, turning the traditional MDP formulation into a multiagent problem.
Future work will include larger-scale empirical results, including extending our approach to more complex environments possibly with continuous state and actions spaces (Brockman et al., 2016).
A E XPERIMENTS A. E XPERIMENTAL G OALS
Our experiments aim to understand better the principal’s inﬂuence on environment generation and the subsequent consequences on a learner agent’s generalization ability
Speciﬁcally, we are interested in out-of-distribution zero-shot (OODZ) generalization.
This metric informs us of the learner agent’s capacity to adapt to novel environments, directly linked to the environments presented during training (as conﬁrmed by our experiments).
Intuitively, some reward signals lend themselves to vastly more powerful OODZ generalization than others.
In the same vein, some reward signals lead to degenerate solutions on nature’s side. We explore a series of reward signals for the nature coalition.
We ﬁrst study the extrinsic rewards traditionally used for the environment generation problem setting. Then we study rewards derived from an Intrinsic Curiosity Module (ICM) as described in (Pathak et al., 2017) as a way to cut extrinsic rewards out of the equation. Finally, we investigate a mix of ICM signals and extrinsic rewards. Where applicable, we present both adversarial and cooperative coalitions.
In every case, the learner agent receives extrinsic rewards. Finally, as a small investigation into limiting the nature coalition’s powers, we present results for which the coalition was not allowed to choose the placement of the maze’s walls (or their amount) but was allowed to choose the goal and the agent’s placement. Instead, blocks are uniformly sampled.
A. E XPERIMENTAL S ETUP
We evaluate our series of training regimes in the popular environment Minigrid, described in Chevalier-Boisvert et al.
Minigrid is a partially observable maze environment where the decision-making agent must take discrete actions to reach a goal.
When the said goal is reached, the agent receives a sparse reward that decreases with the number of steps.
The learner’s observation is a 7-by-7 pixel image representing the maze directly in front of it, including its current position.
Its vision does not penetrate walls. Nature’s (and the principal’s) observation is the entire grid it is currently building. During training, we utilize 15-by-15 grids, and in them, up to 50 blocks can be placed. Evaluation grid sizes vary by environment.
We train each scheme on three different seeds for 3M steps, except for the Adversarial ICM
Mixing scheme, which instead utilizes seven seeds.
For each seed, we evaluate the learner agent’s zero-shot transfer in a list of challenging OOD environments shown in Figure 5 for ten evaluation runs each.
All learning agents are Proximal Policy Optimization agents using an Long Short Term
Memory-based recurrent policy (see Schulman et al. (2017) and Hochreiter & Schmidhuber (1997), respectively).
We present aggregations of the OODZ generalization performance in the evaluation environment in this section and C.
Additionally, to ensure Minigrid’s reward function does not bias our results, we also explore a variation of it.
This new reward function gives a negative signal whenever the agent tries to walk into a wall.
This densiﬁes the training signal but introduces a new mechanic into training: nature’s actions can now create negative signals for the learner agent.
A. R ESULTS
Results for which blocks were randomly placed are marked with ”RB”.
Results involving the altered
Minigrid extrinsic reward signal are marked with ”NB”.
[Figure]
[Figure]
generalization environment
Our best performing learners overall involve coalitions learning from extrinsic rewards without being able to place blocks. Both in the cooperative and the adversarial setting, they are consistently both the fastest to train and the best at generalization.
The exception is when using the altered extrinsic signal: in that case, allowing the coalition to pick the placement of the blocks leads to the best
[Figure]
S h
o r t
e s t P a t h
L e n g t h
[Figure]
generalization across evaluation environments. However, the former scheme still performs better in the most complex environments (labyrinth and maze). Preventing the coalition from placing the blocks discouraged it from making trivial (too-hard) environments in the cooperative (adversarial) scheme. This induces more environment diversity for the learner.
As for ICM, interestingly, mixing in extrinsic rewards yields close to the same generalization prowess as using the direct ICM signal, for both the original and the altered extrinsic rewards and the cooperative and adversarial schemes (but in both cases, the adversarial mixing does do better than the cooperative one). This suggests that ICM, or other intrinsic signals, could be used as a replacement for the nature coalition’s extrinsic-based training signal.
As they are entirely independent of the extrinsic reward signal, these intrinsic objectives could be used to build much more robust training regimes for environment generators. The relation between adversarial and cooperative schemes is interesting. In almost every scheme and environment, adversarial schemes tend to do slightly better than their counterparts.
In the case of a nature coalition with no power over block placement, the reason is apparent: as seen in Figure 4, the paths generated by the adversarial scheme are longer than the cooperatives.
This explains the gap in OODZ generalization for that scheme: learners trained in the adversarial scheme are tasked with solving much longer mazes, which directly translates into experiencing more varied environments. For other schemes, the gap is less obvious.
Still, the adversarial (cooperative) coalitions ﬁnd alternative ways to make difﬁcult (easy) environments: consistently, when allowed to choose the amount and placement of blocks, adversarial schemes place more blocks than cooperative schemes.
See Figure 10 for more details. Notice in Figure 2 that many of our training regimes yield decision-making agents that beat the reported OODZ generalization performance of PLR ⊥ and PAIRED (as described in Jiang et al.
(2021) and Dennis et al. (2020), respectively)
an order of magnitude earlier than both of those methods on the Sixteenroom environment.
Our learners need less than 3M steps to reach 85% solve rate in Sixteenroom, while PLR ⊥ and PAIRED require 250M to reach about the same.
Speciﬁcally, the agent trained with a nature coalition that cannot pick blocks (using Minigrid’s original extrinsic signal) reaches an almost 95% solve rate in only 1.9M steps.
More comparisons of this sort are available in appendix C.
We have also experimented with the altered Minigrid reward function on PAIRED.
When PAIRED uses said reward, it produces a learner agent that is utterly incapable of solving SixteenRooms, Maze, and Labyrinth.
Its regret formulation explains this: placing a block penalizes the antagonist; thus, the adversary is incentivized to make the simplest tasks possible.
The more blocks there are, the more the antagonist bumps into them and thus, the more negative the reward signal for the adversary.
All of our schemes do better than PAIRED at this altered reward function, implying that serious thought should be given to extrinsic rewards and their unreliability for environment generation.
B A DDITIONAL B ACKGROUND B. P AYOFF S TRUCTURES IN R EPEATED G AMES
In repeated games, the decision-making payoff function (cid:96) tπ ( τ ) is deﬁned differently depending on ﬁniteness of the length or horizon t of repetitions of the base game.
In ﬁnite repeated games , a game
G T with a ﬁnite and bounded number of interactions T < ∞ , the cumulative payoffs functions (cid:96) Tπ : ( S × A ) T → R for can be deﬁned as the sum of individual base game G payoffs (cid:96) π , such that:
[Equation]
However, when the number of interactions is inﬁnite and leads to inﬁnitely repeated nature-agent games G ∞ , there are two main choices for payoffs structures.
When the decision-making agent is indifferent as to when its one-shot payoffs (cid:96) π ( s t , a t ) is received during the interaction, we can deﬁne an average payoff (limit of means) such that
[Equation]
Alternatively, if the agent express preferences for one-shot payoffs (cid:96) π received earlier on the interaction history, we can introduce a discount factor 0 < γ < 1 (Abreu, 1988) such that:
[Equation]
For discount values of γ < 1 , these inﬁnite games are equivalent to a ﬁnite game with an effective number of interactions T = 11 − γ after which no base game payoff is considered ( γ t = 0 ).
B. I NTRODUCTION TO M ECHANISM D ESIGN
Mechanism design Vickrey (1961) aim to elicit desirable behavior (e.g., Pareto optimal Nash equilibrium) from the players involved in a game.
We introduce the concepts of game form or mechanism and that of the n-player game induced by a mechanism.
Deﬁnition
Let O be a set of possible outcomes, a game form or mechanism for O is a tuple
F = (cid:104)O , A , ϕ (cid:105) , where
A =
A 1 × . . .
×
A n is a cross product of action or strategy spaces for n players and ϕ :
A → O is an outcome rule or function.
Utility functions (cid:96) i : O → R generally convert player’s preference over ordinal outcomes into a cardinal utility values (Von Neumann & Morgenstern, 1944)
Deﬁnition
Given a mechanism F and a set (cid:96) of utility functions (cid:96) i :
O → R for each player, the n-player game G F induced by the mechanism F is a pair G F =
(cid:104)A , (cid:96) ϕ (cid:105) where
A =
A 1 × . . .
×A n is a cross product of action or strategy spaces for n players and (cid:96) ϕ is the set of modeﬁed utility functions ˜ (cid:96)
i for each player, such as:
[Equation]
20) deﬁnes the payoff received by the i-th agent under a joint strategy ( a 1 , . .
. , a n ) ∈
A deﬁned by the outcome function ϕ ( a 1 , . . . , a n ) .
Automated Mechanism Design.
The ﬁeld of automated mechanism design approaches design of the mechanism outcome function ϕ : A → O manually or automated.
The approach to automated mechanism design (Conitzer & Sandholm, 2002; Sandholm, 2003) leverages the ﬁgure of the principal or designer, a player ν whose objective (cid:96) ν : Φ × (cid:96) ϕ
→ R is deﬁned over the space of outcome functions ϕ ∈ Φ and players modiﬁed utilities (cid:96) ϕ .
Possible objective functions for the principal ν to maximize may include social welfare (utilitarian) (cid:96) ν
= (cid:80) i ˜ (cid:96) i , the worst utility (egalitarian) of any player (cid:96) ν
= min i ˜ (cid:96) i or dictatorial (cid:96) ν =
˜ (cid:96) i
=
k , among others (Sandholm, 2003).
The principal’s objective is selected according to an equilibrium concept required for players in the induced game (Conitzer & Sandholm, 2003).
In general, the automated design of a mechanism is reduced to ﬁnding, after the outcome and action space are manually deﬁned, the optimal outcome function ϕ ∗ , such that the objective: [Equation]
[Equation]
enforces the preferences of the principal player ν over the game G F induced by the mechanism F .
C A DDITIONAL P LOTS
Here are presented the rest of the zero-shot OODZ generalization plots.
Results for which blocks were randomly placed are marked with ”RB”.
Results involving the altered Minigrid extrinsic reward signal are marked with ”NB”.
[Figure]
o l v e R a t e
[Figure]
[Figure] [Figure]
S o l v e R a t e
[Figure]
[Figure]
[Figure]
N u m b e r o f B l o c
k s P l a c e d
[Figure]