Status-quo policy gradient in Multi-Agent Reinforcement Learning
Abstract
Individual rationality, which involves maximizing expected individual return, does not always lead to high-utility individual or group outcomes in multi-agent problems.
For instance, in multi-agent social dilemmas, Reinforcement Learning (RL) agents trained to maximize individual rewards converge to a low-utility mutually harmful equilibrium.
In contrast, humans evolve useful strategies in such social dilemmas.
Inspired by ideas from human psychology that attribute this behavior to the status-quo bias, we present a status-quo loss ( SQLoss ) and the corresponding policy gradient algorithm that incorporates this bias in an RL agent.
We demonstrate that agents trained with SQLoss learn high-utility policies in several social dilemma matrix games (Prisoner’s Dilemma, Stag Hunt matrix variant, Chicken
Game). We show how SQLoss outperforms existing state-of-the-art methods to obtain high-utility policies in visual input non-matrix games (Coin Game and Stag Hunt visual input variant) using pre-trained cooperation and defection oracles.
Finally, we show that SQLoss extends to a 4-agent setting by demonstrating the emergence of cooperative behavior in the popular Braess’ paradox.
Introduction
In sequential social dilemmas, individually rational behavior can lead to outcomes that are suboptimal for each individual in the group Hardin [1968], Ostrom [1990], Ostrom et al.
[1999], Dietz et al.
Current state-of-the-art Multi-Agent Deep Reinforcement Learning (MARL) methods that train agents independently lead to agents that play individualistically, thereby receiving poor rewards, even in simple social dilemmas [Foerster et al., 2018, Lerer and Peysakhovich, 2017]. [Figure] a [Figure]
n v i r o [Figure] n [Figure]
To illustrate why it is challenging to learn optimal policies in such dilemmas, we consider the Coin
Game, which was first explored in [Foerster et al., 2018].
Each agent can play either selfishly (pick all coins) or cooperatively (pick only coins of its color).
Regardless of the other agent’s behavior, the individually rational choice for an agent is to play selfishly, either to minimize losses (avoid being exploited) or to maximize gains (exploit the other agent).
However, when both agents behave rationally, they try to pick all coins and achieve an average long term reward of − 0 .
In contrast, if both cooperate, then the average long term reward for each agent is 0 .
Therefore, when agents cooperate, they are both better off.
Training Deep RL agents independently in the Coin Game using state-of-the-art methods leads to mutually harmful selfish behavior (Section 2.2).
The problem of how independently learning agents develop optimal behavior in social dilemmas has been studied by researchers through human studies and simulation models [Fudenberg and Maskin, 1986, Green and Porter, 1984, Fudenberg et al., 1994, Kamada and Kominers, 2010, Abreu et al., 1990].
A large body of work has looked at the mechanism of evolution of cooperation through reciprocal behaviour and indirect reciprocity [Trivers, 1971, Axelrod, 1984, Nowak and Sigmund, 1992, 1993, 1998], through variants of reinforcement using aspiration
[Macy and Flache, 2002], attitude [Damer and Gini, 2008] or multi-agent reinforcement learning [Sandholm and Crites, 1996, Wunder et al., 2010], and under specific conditions [Banerjee and Sen, 2007] using different learning rates [de Cote et al., 2006] similar to WoLF
[Bowling and Veloso, 2002] as well as using embedded emotion [Yu et al., 2015], social networks [Ohtsuki et al., 2006, Santos and Pacheco, 2006].
However, these approaches do not directly apply to Deep RL agents [Leibo et al., 2017].
Recent work in this direction [Kleiman-Weiner et al., 2016, Julien et al., 2017, Peysakhovich and Lerer, 2018] focuses on letting agents learn strategies through interactions with other agents in a multi-agent setting.
Leibo et al.
[2017] defines the problem of social dilemmas in the Deep RL framework and analyzes the outcomes of a fruit-gathering game [Julien et al., 2017].
They vary the abundance of resources and the cost of conflict in the fruit environment to generate degrees of cooperation between agents.
Hughes et al.
[2018] defines an intrinsic reward (inequality aversion) that attempts to reduce the difference in obtained rewards between agents.
The agents are designed to have an aversion to both advantageous (guilt) and disadvantageous (unfairness) reward allocation.
This handcrafting of loss with mutual fairness develops cooperation, but it leaves the agent vulnerable to exploitation.
LOLA [Foerster et al., 2018] uses opponent awareness to achieve high cooperation levels in the Coin Game and the Iterated Prisoner’s Dilemma game.
However, the LOLA agent assumes access to the other agent’s network architecture, observations, and learning algorithms.
This access level is analogous to getting complete access to the other agent’s private information and therefore devising a strategy with full knowledge of how they are going to play.
Wang et al.
[2019] proposes an evolutionary Deep RL setup to learn cooperation.
They define an intrinsic reward that is based on features generated from the agent’s past and future rewards, and this reward is shared with other agents.
They use evolution to maximize the sum of rewards among the agents and thus learn cooperative behavior.
However, sharing rewards in this indirect way enforces cooperation rather than evolving it through independently learning agents.
Interestingly, humans develop individual and socially optimal strategies in such social dilemmas without sharing rewards or having access to private information.
Several ideas in human psychology [Samuelson and Zeckhauser, 1988, Kahneman et al., 1991, Kahneman, 2011, Thaler and Sunstein, 2009] have attributed this cooperative behavior to the status-quo bias [Guney and Richter, 2018].
The status-quo bias is a decision-making bias that encourages humans to stick to the status-quo unless doing so causes significant harm.
This preference in humans for sticking to ‘comfortable’ states rather than seeking short-term exploitative gains is one explanation for emergent cooperative behavior in human groups.
Inspired by this idea, we present the status-quo loss ( SQLoss ) and the corresponding status-quo policy gradient formulation for RL.
Agents trained with SQLoss learn optimal policies in multi-agent social dilemmas without sharing rewards, gradients, or using a communication channel.
Intuitively, SQLoss encourages an agent to stick to past actions provided these actions did not cause significant harm.
Therefore, mutually cooperating agents stick to cooperation since the status-quo yields higher individual reward, while unilateral defection by any agent leads to the other agent also switching to defection since the status-quo is very harmful for the exploited agent.
Subsequently, for each agent, the short-term reward of exploitation is overcome by the long-term cost of mutual defection, and agents gradually switch to cooperation.
To apply SQLoss to games where a sequence of non-trivial actions determines cooperation and defection, we present GameDistill , an algorithm that reduces a dynamic game with visual input to a matrix game.
GameDistill uses self-supervision and clustering to extract distinct policies from a sequential social dilemma game automatically.
Our key contributions can be summarised as:
We introduce a Status-Quo loss ( SQLoss , Sec. 2.3) and an associated policy gradient-based algorithm to learn optimal behavior in a decentralized manner for agents playing iterated matrix games where agents can choose between a cooperative and selfish policy at each step.
We empirically demonstrate that agents trained with SQLoss learn optimal behavior in several social dilemma iterated matrix games (Sec. 4).
We extend SQLoss to social dilemma game with visual observations (Sec. 2.4) using GameDistill .
We empirically demonstrate that GameDistill extracts cooperative and selfish policies for the Coin Game and Stag Hunt (Sec. 4.2).
We further show that incorporating GameDistill in LOLA accelerates learning and achieves higher cooperation.
We also demonstrate that SQLoss extends in a straight forward manner to games with more than two agents due to its ego-centric nature (Sec. 6).
Approach
2.1 Social Dilemmas modeled as Iterated Matrix Games
To remain consistent with previous work, we adopt the notations from Foerster et al.
We model social dilemmas as general-sum Markov (simultaneous move) games.
A multi-agent Markov game is specified by G = ⟨ S , A , U , P , r , n , γ ⟩ .
S denotes the state space of the game.
n denotes the number of agents playing the game.
At each step of the game, each agent a ∈ A , selects an action u a ∈ U .
⃗u denotes the joint action vector that represents the simultaneous actions of all agents.
The joint action ⃗u changes the state of the game from s to s ′ according to the state transition function
P ( s ′ | ⃗u, s ) : S × U × S → [0 , 1] .
At the end of each step, each agent a gets a reward according to the reward function r a ( s, ⃗u ) :
S × U → R .
The reward obtained by an agent at each step is a function of the actions played by all agents.
For an agent a , the discounted future return from time t is defined as
R at = (cid:80) ∞
l =
γ l r at + l , where γ ∈ [0 , 1) is the discount factor.
Each agent independently attempts to maximize its expected discounted return.
Matrix games are the special case of two-player perfectly observable Markov games [Foerster et al., 2018].
Table 1 shows examples of matrix games that represent social dilemmas.
Consider the
Prisoner’s Dilemma game in Table 1a.
Each agent can either cooperate ( C ) or defect ( D ).
Playing D is the rational choice for an agent, regardless of whether the other agent plays C or D .
Therefore, if both agents play rationally, they each receive a reward of − 2 .
However, if each agent plays C , then it will obtain a reward of − 1 .
This fact that individually rational behavior leads to a sub-optimal group (and individual) outcome highlights the dilemma.
In Infinitely Iterated Matrix Games, agents repeatedly play a particular matrix game against each other.
In each iteration of the game, each agent has access to actions played by both agents in the previous iteration.
Therefore, the state input to an RL agent consists of both agents’ actions in the previous iteration of the game.
We adopt this state formulation as is typically done in such games [Press and Dyson, 2012, Foerster et al., 2018].
The infinitely iterated variations of the matrix games in
Table 1 represent sequential social dilemmas.
We refer to infinitely iterated matrix games as iterated matrix games in subsequent sections for convenience.
2.2 Learning Policies in Iterated Matrix Games: The Selfish Learner
The standard method to model agents in iterated matrix games is to model each agent as an RL agent that independently attempts to maximize its expected total discounted reward.
Several approaches to model agents in this way use policy gradient-based methods [Sutton et al., 2000, Williams, 1992].
Policy gradient methods update an agent’s policy, parameterized by θ a , by performing gradient ascent on the expected total discounted reward E
[ R a 0 ] .
Formally, let θ a denote the parameterized version of an agent’s policy π a and V aθ 1 ,θ 2 denote the total expected discounted reward for agent a .
Here,
V a is a function of the policy parameters ( θ 1 , θ 2 ) of both agents.
In the i th iteration of the game, each agent updates θ ai to θ ai +1 , such that it maximizes it’s
total expected discounted reward.
θ ai +1 is computed using Eq. 1.
For agents trained using reinforcement learning, the gradient ascent rule to update θ {+
+}
i +1 is given by Eq. 2.
[Equation]
where δ is the step size of the updates.
In the Iterated Prisoner’s Dilemma (IPD) game, agents trained with the policy gradient update method converge to a sub-optimal mutual defection equilibrium (Figure 3, Lerer and Peysakhovich [2017]).
This sub-optimal equilibrium attained by Selfish Learners motivates us to explore alternative methods that could lead to a desirable cooperative equilibrium.
We denote the agent trained using policy gradient updates as a Selfish Learner ( SL ).
2.3 Learning Policies in Iterated Matrix Games: The Status-Quo Aware Learner ( SQLoss ) 2.3.1 SQLoss : Motivation and Theory
The status-quo bias instills in humans a preference for the current state provided the state is not harmful to them.
Inspired by this idea, we introduce a status-quo loss ( SQLoss ) for each agent, derived from the idea of imaginary game-play (Figure 2).
The SQLoss encourages an agent to
Saaa
Status-Quo aware
Learner (SQLearner) (a 0 ) 𝜅 (a 1 ) 𝜅 rrr(a 2 )
𝜅
S S
S S
S
Srrra 3 r
I m a g
i n e d g a m e p l a y
I m a g
i n e d g a m e p l a y
I m a g
i n e d g a m e p l a y imagine a future episode where the status-quo (current situation) is repeated for several steps.
If an agent has been exploited in the previous iteration of the game (state DC ), then SQLoss will encourage the agent to imagine a continued risk of exploitation and subsequently switch to defection and move to state DD .
Hence, for the exploiting agent, the short-term gain from exploitation ( DC ) is overcome by the long-term loss from mutual defection ( DD ).
Conversely, if both agents cooperated in the previous iteration of the game (state CC ), then SQLoss will encourage the agent to imagine a continued gain from mutual cooperation and subsequently stick to state CC .
Since state CC is more beneficial to both agents than DD , and exploitation ( DC and CD ) also leads to DD , agents move away from defection and converge to mutual cooperation ( CC ).
Appendix C describes in detail how cooperation emerges between SQLoss agents.
2.3.2 SQLoss : Formulation
We describe below the formulation of SQLoss with respect to agent 1.
The formulation for agent 2 is identical to that of agent 1.
Let τ a =
( s 0 , u 10 , u 20 , r 10 , · · · s T , u 1 T , u 2 T , r 1 T ) denote the collection
C
D C (-1,
[Table] [Table]
, 0) D (0,
[Table] [Table] [Table]
H
T
H (+1, -
[Table]
) (-1, +
[Table]
T (-1, +1) (+1, -
[Table] [Table] MP)
C
D
C (0, 0) ( [Table]
[Table] D (-1,
[Table]
[Table] [Table]
[Table]
of an agent’s experiences after T time steps. Let R 1 t ( τ 1 ) = (cid:80) Tl =
t γ l − t r 1 l denote the discounted future return for agent 1 starting at s t in actual game-play.
Let ˆ τ 1 denote the collection of an agent’s imagined experiences. For a state s t , where t ∈ [0 , T ] , an agent imagines an episode by starting at s t and repeating u 1 t − 1 , u 2 t − 1 for κ t steps. This is equivalent to imagining a κ t step repetition of already played actions.
We sample κ t from a Discrete Uniform distribution U { 1 , z } where z is a hyper-parameter ≥ 1 .
To simplify notation, let ϕ t ( s t , κ t ) denote the ordered set of state, actions and rewards starting at time t and repeated κ t times for imagined game play.
Let
ˆ
[Table]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
V 1 θ 1 ,θ 2 and ˆ V 1 θ 1 ,θ 2 are approximated by E
[ R 10 ( τ 1 )] and E
[ ˆ R 10 (ˆ τ 1 )] respectively.
These V values are the expected rewards conditioned on both agents’ policies ( π 1 , π 2 ).
For agent 1, the regular gradients and the Status-Quo gradient-like correction term, ∇ θ 1 E
[ R 10 ( τ 1 )] and f 1 sqcor , can be derived from the policy gradient formulation as
[Equation]
Here, f 1 sqcor in Eq.
7 is a biased term that has policy gradient-like structure and incorporates the prior action taken through the u t − 1 | s t term.
It should be noted that the f 1 sqcor is not a gradient of the expected return of the imagined trajectory in Eq. 5, but the form of this expression is arrived at considering the policy gradient expression.
Furthermore, b ( s t ) is a baseline-like term added for variance reduction in the implementation.
The update rule f sql,pg for the policy gradient-based Status-Quo Learner (SQL-PG) is defined by,
[Equation]
[Equation]
where α , β are the loss scaling factor for REINFORCE and imaginative game-play respectively.
2.4 Learning policies in Dynamic Non-Matrix Games using SQLoss and GameDistill
The previous section focused on learning optimal policies in iterated matrix games that represent sequential social dilemmas.
In such games, an agent can take one of a discrete set of policies at each step.
For instance, in IPD, an agent can either cooperate or defect at each step.
However, in social dilemmas such as the Coin Game (Appendix A), cooperation and defection policies are composed of a sequence of state-dependent actions.
SQLoss , proposed above, works for matrix games but is not directly applicable to games with visual input to yield mutual cooperation.
To apply the Status-Quo policy gradient to these games, we present GameDistill , a self-supervised algorithm that reduces a dynamic infinitely iterated game with visual input to a matrix game.
GameDistill takes as input game-play episodes between agents with random policies and learns oracles (or policies) that lead to distinct outcomes.
GameDistill (Figure 1) works as follows.
We initialize agents with random weights and play them against each other in the game.
In these random game-play episodes, whenever an agent receives a reward, we store the sequence of states along with the rewards for both agents.
This collection of state sequences is used to train the GameDistill network, which is a self-supervised trajectory encoder .
It takes as input a sequence of states and predicts the rewards of both agents during training.
We now cluster the embeddings extracted from the penultimate layer of the trained GameDistill network using Agglomerative Clustering [Friedman et al., 2001].
Each embedding is a finite-dimensional representation of the corresponding state sequence.
Each cluster represents a collection of state sequences or transitions, that lead to a consistent outcome (w.r t rewards).
For CoinGame, when we use the number of clusters as two, we observe that one cluster consists of transitions that represent cooperative behavior (cooperation cluster) while the other contains transitions that lead to defection (defection cluster).
Using the state sequences in each cluster, we train an oracle to predict the next action given the current state.
For the Coin Game, the oracle trained on state sequences from the cooperation cluster predicts the cooperative action for a given state.
Similarly, the oracle trained on the defection cluster predicts the defection action for a given state.
Each agent uses GameDistill independently to extract a cooperation and a defection oracle. Figure
Appendix J) illustrates the cooperation and defection
oracles extracted by
the Red agent. During game-play, an agent can consult either oracle at each step.
In CoinGame, this is equivalent to either cooperating (consulting the cooperation oracle) or defecting (consulting the defection oracle).
In this way, an agent reduces a dynamic game to its matrix equivalent using GameDistill .
We then utilize the Status-Quo policy gradient to learn optimal policies in the reduced matrix game.
For the Coin Game, this leads to agents who cooperate by only picking coins of their color (Figure 5).
It is important to note that for games like CoinGame, we could have learned cooperation and defection oracles by training agents using the sum-of-rewards for both agents and individual reward, respectively [Lerer and Peysakhovich, 2017].
However, GameDistill learns distinct policies without using hand-crafted reward functions.
GameDistill is applicable only in games where cooperation and defection policies are clearly defined and lead to distinct payoffs (rewards) for both players.
Algorithm 1 in Appendix provides the pseudo-code for GameDistill with additional details in
Appendix B.3.
Appendix B.2 provides additional details about the architecture and the different components of GameDistill .
Further, we have verified GameDistill empirically on the Coin Game and the Stag Hunt (results in Appendix G).
Additional clustering visualizations for the trajectories and the experimental plots are provided in Appendix F and J.
Experimental Setup
In order to compare our results to previous work, we use the Normalized Discounted Reward or
NDR = (1 − γ ) (cid:80) Tt =
γ t r t .
A higher NDR implies that an agent obtains a higher reward in the environment.
We compare our approach (Status-Quo Aware Learner or SQLearner ) to Learning with Opponent-Learning Awareness (Lola-PG)
[Foerster et al., 2018] and the Selfish Learner (SL) agents.
For all experiments, we perform 20 runs and report average NDR , along with variance across runs.
The bold line in all the figures is the mean, and the shaded region is the one standard deviation region around the mean.
3.1 Iterated Matrix Game Social Dilemmas
For our experiments with social dilemma matrix games, we use the Iterated Prisoners Dilemma (IPD)
[Luce and Raiffa, 1989], Iterated Matching Pennies (IMP)
[Lee and Louis, 1967], and the
Iterated Stag Hunt (ISH)
[Fang et al., 2002].
Each matrix game in Table 1 represents a different dilemma.
In the Prisoner’s Dilemma, the rational policy for each agent is to defect, regardless of the other agent’s policy.
However, when each agent plays rationally, each is worse off.
In Matching
Pennies, if an agent plays predictably, it is prone to exploitation by the other agent.
Therefore, the optimal policy is to randomize between H and T , obtaining an average NDR of 0 .
The Stag Hunt game represents a coordination dilemma.
In the game, given that the other agent will cooperate, an agent’s optimal action is to cooperate as well.
However, each agent has an attractive alternative at each step, that of defecting and obtaining a guaranteed reward of − 1 .
Therefore, the promise of a safer alternative and the fear that the other agent might select the safer choice could drive an agent to select the safer alternative, thereby sacrificing the higher reward of mutual cooperation.
In iterated matrix games, at each iteration (iter), agents take an action according to a policy and receive the rewards in Table 1.
To simulate an infinitely iterated game, we let agents play 200 iters of game against each other, and do not provide an agent with any information about the number of remaining iters.
In an iter, state for an agent is the actions played by both agents in the previous iter.
3.2 Iterated Dynamic Game Social Dilemmas
For our experiments on a social dilemma with extended actions, we use the Coin Game (Figure 7)
[Foerster et al., 2018] and the non-matrix variant of the Stag Hunt (Figure 8).
We provide details of the games in Appendix A due to space considerations.
Results 4.1 Learning optimal policies in Iterated Matrix Dilemmas
Iterated Prisoner’s Dilemma (IPD):
We train different learners to play the IPD game.
Figureshows the results.
For all learners, agents initially defect and move towards an NDR of − 2 . 0 .
This initial bias towards defection is expected, since, for agents trained with random game-play episodes, the benefits of exploitation outweigh the costs of mutual defection.
For Selfish Learner (SL) agents, the bias intensifies, and the agents converge to mutually harmful selfish behavior (NDR of − 2 . 0 ).
Lola-PG agents learn to predict each other’s behavior and realize that defection is more likely to lead to mutual harm.
They subsequently move towards cooperation, but occasionally defect (NDR of − 1 . 2 ).
In contrast, SQLearner agents quickly realize the costs of defection, indicated by the small initial dip in the NDR curves.
They subsequently move towards close to 100% cooperation, with an
NDR of − 1 . 0 .
Finally, it is important to note that SQLearner agents have close to zero variance, unlike other methods where the variance in NDR across runs is significant.
Iterated Matching Pennies (IMP): We train different learners to play the IMP game.
Figure 4 shows the results.
SQLearner agents learn to play optimally and obtain an NDR close to 0 .
Interestingly,
Selfish Learner (SL) and Lola-PG agents converge to an exploiter-exploited equilibrium where one agent consistently exploits the other agent.
This asymmetric exploitation equilibrium is more pronounced for SL agents than for Lola-PG agents.
As before, we observe that SQLearner agents have close to zero variance across runs, unlike other methods where the variance in NDR is significant.
Iterated Chicken Game (ICG): Appendix H shows additional results for the ICG game.
4.2 Learning Optimal Policies in Iterated Dynamic Dilemmas
GameDistill:
To evaluate the clustering step in GameDistill , we make two t-SNE [Maaten and
Hinton, 2008] plots of the 100-dimensional feature vectors extracted from the penultimate layer of the trained GameDistill network in Figure 9 of Appendix F.
In the first plot, we color each point (or state sequence) by the rewards obtained by both agents in the format r 1 | r 2 .
In the second, we color each point by the cluster label output by the clustering technique.
GameDistill correctly discovers two clusters, one for transitions that represent cooperation (Red cluster) and the other for transitions that represent defection (Blue cluster).
We experiment with different values for feature vector dimensions and clustering techniques, and obtain similar results (see Appendix B).
Results on
Stag Hunt using GameDistill are presented in Appendix F and G. To evaluate the trained oracles that represent cooperation and a defection policy, we modify the CoinGame environment to contain only a single agent (the Red agent).
We then play two variations of the game.
In the first variant, the
SQLearner agents achieve a cooperation rate close to 1.0 while Lola-PG
SQLearner agents (w/
GameDistill oracles) achieve close to 100% cooperation.
Lola-PG agents eventually (after roughlyepochs) achieve close to 80% cooperation. Interestingly, Lola-PG agents trained w/ GameDistill oracles converge to higher cooperation rate of 85% in 300 epochs. Red agent executes the actions as suggested by the first oracle.
We observe that Red agent picks only 8 .
4% of Blue coins, indicating a high cooperation rate, which represents a cooperation policy.
In the second variant, Red agent executes the actions suggested by the second oracle.
We observe that Red agent picks 99 .
4% of Blue coins, indicating high defection rate, which represents a defection policy.
SQLoss: During game-play, at each step, an agent follows either the action suggested by its cooperation oracle or defection oracle.
We compare approaches using the degree of cooperation between agents, measured by the probability that an agent will pick the coin of its color [Foerster et al., 2018].
Figure 5 shows the results.
The probability that an SQLearner agent will pick the coin of its color is close to 1 . 0 .
This high probability indicates that the other SQLearner agent is cooperating with this agent and only picking coins of its color.
In contrast, the probability that a Lola-PG agent will pick a coin of its color is close to 0 .
8 , indicating higher defection rates.
As expected, the probability of an agent picking its own coin is the smallest for Selfish Learner.
To evaluate the impact of GameDistill , we compare SQLearner agents to Lola-PG agents trained using GameDistill oracles.
Figure 6 shows the results.
Interestingly, Lola-PG agents trained with
GameDistill oracles converge to a higher cooperation rate ( 0 . 88 ) than Lola-PG agents trained directly on visual input.
SQLoss for social dilemma matrix games
For a given matrix game, SQLoss will work on an equivalent version of the game in which all rewards have been transformed to non-positive values.
For the matrix games described in our paper, we have used their variants with negative rewards to remain consistent with the LOLA paper.
For a general matrix game, we can subtract the maximum reward (or any number larger than it) from each reward value to make rewards negative and then use SQLoss .
We consider the social dilemma class of matrix games from Leibo et al.
[2017], (cid:18)
C
D C
[Equation] R [Equation]
T D
T,S
P,P (cid:19) ,where the first row corresponds to cooperation for the first player and the first column corresponds to cooperation for the second player.
Leibo et al.
[2017] define the following rules that describe different categories of social dilemmas: (i)
If R > P , then mutual cooperation is preferred to mutual defection.
(ii)
If R > S , then mutual cooperation is preferred to being exploited by a defector.
(iii) If 2 R > T + S , this ensures that mutual cooperation is preferred to an equal probability of unilateral cooperation and defection.
(iv) Either greed ( T > R : Exploit a cooperator preferred over mutual cooperation) or fear ( P > S : Mutual defection preferred over being exploited) should hold.
These rules have been reproduced from Leibo et al.
When “greed” ( T > R ) as well as “fear” ( P > S ) conditions hold, we have T > R > P > S .
The Iterated Prisoner’s Dilemma (IPD) is an example of this game.
If we subtract T from each reward, we get the matrix game,
[Equation]
,where all entries are non-positive and therefore Lemma 3 and Lemma 4 hold as before.
When “greed” holds but not “fear” we have T > R > S > = P .
The Chicken Game (CG) is an example of this game.
If we subtract T from each reward as before, we get the equivalent matrix game with non-positive entries and Lemmas 3 and 4 hold.
When “fear” holds but not “greed” we have R > = T > P > S .
The Iterated Stag Hunt is an example of this game.
If we subtract R from each reward, we get the equivalent matrix game with non-positive entries then Lemmas 3 and 4 (Appendix C) hold.
In our experiments, we have considered games from each of the classes mentioned above, and the use of SQLoss leads to cooperation in all these examples.
Games with more than 2 players
Our formulation of SQLoss has the distinct advantage of being fully ego-centric, i.e., the learning agent does not require any information regarding its opponents.
This feature enables a straightforward extension of SQLoss beyond the two agent setting, without any change in each agent’s learning algorithm.
In order to test this extension of SQLoss beyond 2-players, we consider as an example, the problem described in the popular Braess’ paradox[Braess, 1968], which is a well-known extension of the Prisoner’s Dilemma problem to more than 2 agents.
We performed additional experiment in this game with 4 agents.
We simulated the result when all agents are selfish learners (the SL agent(s)) and also when all agents use SQLoss (the SQLearner (s)).
We observe that when using selfish learners, all agents converge to Defection and when using SQLoss , all agents converge to Cooperation.
Detailed explanation of the game, experimental setup and results are shown in
Appendix D.
Conclusion
We presented a status-quo policy gradient that encourages an agent to imagine the consequences of sticking to the status quo.
We demonstrated how SQLoss outperforms LOLA on standard benchmark matrix games.
To work with dynamic games, we further proposed GameDistill , an algorithm that reduces a dynamic game with visual input to a matrix game.
We combined GameDistill and
SQLoss to demonstrate how agents learn optimal policies in dynamic social dilemmas with visual observations.
We empirically demonstrated that SQLoss obtains near-optimal rewards in various social-dilemma games such as IPD, IMP, Chicken, Stag-Hunt and Coin Game (both matrix and variant with visual observations).
[-For all the otherclaims we provide multiple empirical studies in Sections 3 and 
-]