C ONSEQUENTIALIST CONDITIONAL COOPERATION IN SOCIAL DILEMMAS WITH IMPERFECT INFORMATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Social dilemmas, where mutual cooperation can lead to high payoffs but participants face incentives to cheat, are ubiquitous in multi-agent interaction.
We wish to construct agents that cooperate with pure cooperators, avoid exploitation by pure defectors, and incentivize cooperation from the rest.
However, often the actions taken by a partner are (partially) unobserved or the consequences of individual actions are hard to predict.
We show that in a large class of games good strategies can be constructed by conditioning one’s behavior solely on outcomes (ie.
one’s past rewards).
We call this consequentialist conditional cooperation.
We show how to construct such strategies using deep reinforcement learning techniques and demonstrate, both analytically and experimentally, that they are effective in social dilemmas beyond simple matrix games.
We also show the limitations of relying purely on consequences and discuss the need for understanding both the consequences of and the intentions behind an action.
I NTRODUCTION
Deep reinforcement learning (RL) is concerned with constructing agents that start as blank slates and can learn to behave in optimal ways in complex environments.
1 A recent stream of research has taken a particular interest in social dilemmas, situations where individuals have incentives to act in ways that undermine socially optimal outcomes (Leibo et al., 2017; Perolat et al., 2017; Lerer &
Peysakhovich, 2017; Kleiman-Weiner et al., 2016).
In this paper we consider RL-based strategies for social dilemmas in which information about a partner’s actions or the underlying environment is only partially observed.
The simplest social dilemma is the Prisoner’s Dilemma (PD) in which two players choose between one of two actions: cooperate or defect.
Mutual cooperation yields the highest payoffs, but no matter what one’s partner is doing, one can get a higher reward by defecting.
A well studied strategy for maintaining cooperation when the PD is repeated is tit-for-tat (TFT, Axelrod (2006)).
TFT behaves by copying the prior behavior of their partner, rewarding cooperation today with cooperation tomorrow.
Thus, if an agent commits to TFT it makes cooperation the best strategy for the agent’s partner.
TFT has proven to be a heavily studied strategy because it has intuitive appeal: 1) it is easily explainable, 2) it begins cooperating, 3) it rewards a cooperative partner, 4) it avoids being exploited, 5) it is forgiving
Game theorists have studied the emergence of cooperation in bilateral relationships under both perfect and imperfect observation (Green & Porter, 1984; Fudenberg & Maskin,
Fudenberg
Kamada & Kominers, 2010; Abreu et al., 1990).
However, this research program almost exclusively studies repeated matrix games such as the two action, stochastically repeated Prisoner’s Dilemma (PD) and focuses mostly on proving the existence of equilibria which maintain cooperation rather than on constructing simple strategies that do well across many complex situations.
We take some of the insights developed here and use them in the service of agent design.
In Markov games cooperation and defection are not single actions, but rather temporally extended policies.
Recent work has considered expanding TFT to more complex Markov games either as a heuristic, by learning cooperative and selﬁsh policies and switching between them as needed (Lerer & Peysakhovich, 2017), or as an outcome of an end-to-end procedure (Foerster et al., 2017c).
TFT is an example of a conditionally cooperative strategy - that is, it cooperates when a certain condition is fulﬁlled (ie.
the partner’s last period action was cooperative).
Standard TFT requires perfect observability of a partner’s behavior and perfect understanding of each action’s future consequences.
Our main contribution is to use RL methods to construct conditionally cooperative strategies for games with imperfect information.
When information is imperfect, the agent must use what they can observe to try to estimate whether a partner is acting cooperatively (or not) and determine how to respond.
We show that when the game is ergodic, observed rewards can be used as a summary statistic - if the current total (or time averaged) reward is above a time-dependent threshold (where the threshold values are computed using RL and a form of self play) the agent cooperates, otherwise the agent does not 3 .
We call this consequentialist conditional cooperation (CCC).
We show analytically that this strategy cooperates with cooperators, avoids exploitation, and guarantees a good payoff to the CCC agent in the long run.
We study CCC agents in a partially observed Markov game which we call Fishery.
In Fishery two agents live on different sides of a lake in which ﬁsh appear.
The game has partial information because agents cannot observe what happens across the lake.
Fish spawn randomly, starting young and swim to the other side and become mature.
Agents can catch ﬁsh on their side of the lake.
Catching any ﬁsh yields payoff but mature ﬁsh are worth more.
Therefore, cooperative strategies are those which leave young ﬁsh for one’s partner.
However, there is always a temptation to defect and catch both young and mature ﬁsh.
We show that CCC agents cooperate with cooperators, avoid exploitation, and get high payoffs when matched with themselves.
Second, we show that CCC is an efﬁcient strategy for more complex games where implementing conditional cooperation by fully modeling the effect of an action on future rewards (eg.
amTFT (Lerer & Peysakhovich, 2017)) is computationally demanding.
To demonstrate this we follow the method of Tampuu et al.
(2017) to construct a version of Atari Pong which makes the game into a social dilemma.
In what we call the Pong Player’s Dilemma (PPD) when an agent scores they gain a reward of 1 but the partner receives a reward of − 2 .
Thus, in the PPD the only (jointly) winning move is not to play, but selﬁsh agents are again tempted to defect and try to score points even though this decreases total social reward.
We see that CCC is a successful, robust, and simple strategy in this game.
Conditioning on only outcomes has limitations.
We consider a version of the Pong Players’ Dilemma where when a player scores, instead of their partner losing 2 points deterministically they lose 2 /p points with probability p .
Here the expected rewards of non-cooperation are the same as in the PPD and so expected-future-reward based methods (eg. amTFT) will act act identically.
However, when p is low it may take a long time for consequentialist agents to detect a defector.
Empirically we see that in short risky PPD games CCC agents can be exploited by defectors but that amTFT agents cannot.
We close by discussing limitations and progress towards agents that can effectively use both intention and outcome information effectively in navigating the world.
C ONSEQUENTIALIST C ONDITIONALLY C OOPERATIVE S TRATEGIES
We work with partially observed Markov games (POMG), which are multi-agent generalizations of partially observed Markov decision problems:
Deﬁnition 1 A (two-player, ﬁnite) partially observed Markov game (POMG) consists of: a ﬁnite set of states S ; a set of actions for each player
A 1 , A 2 ; a transition function τ : S ×
A 1 × A 2 → ∆( S ) ; an observation function that tells us what each player observes i : S ×
A 1 × A 2 → ∆(Ω i ) where Ω i is a set of possible observations; and a reward function that maps states and actions to each player’s reward R i : S ×
A 1 × A 2 → ∆( R ) .
We assume that per turn rewards are bounded above and below.
Agents choose a policy
[Equation]
which takes as input the observation and outputs a probability distribution on actions - this is similar to the notion of ‘belief free strategies’ in the study of repeated games (Ely et al., 2005).
Given a pair of policies, one for each agent, and a starting state, we deﬁne each player’s value function as the average (undiscounted) reward if players start in state s and follow policies π 1 , π 2 , i.e.
[Equation]
Note that while we will prove results in the undiscounted setting, for δ sufﬁciently close to 1, the optimal policy is the same in the undiscounted and discounted setting, so standard discounted policy gradient techniques can still be used (Schwartz, 1993).
We restrict ourselves to reward-ergodic POMGs:
Assumption 1 (Reward Ergodicity in POMG)
Say a POMG is reward-ergodic if for any pair of policies π 1 , π 2 the long-run reward has a well deﬁned rate independent of the starting state almost surely.
Formally, this means for any starting state s , and either player i , there exists a rate ρ iπ 1 πsuch that
[Equation]
Any pair of policies applied to a POMG creates a Markov chain of underlying states.
If for any pair of policies that Markov chain is ‘unichain’, that it, it has a single positive recurrent chain (Puterman, 2014) then the POMG will be reward ergodic (Meyn & Tweedie, 2012)[Thm.
The unichain assumption is often used in applications of RL methods in the undiscounted RL problem (Schwartz, 1993).
Deﬁnition 2 Cooperative Policies are those that maximize the joint rate of reward:
[Equation]
Let Π C be the set of such tuples.
We look at the class of POMGs which have two restrictions:
Assumption 2 (Social Dilemma)
For any player i and any ( π C 1 , π C 2 )
∈
Π C we have that
[Equation]
Assumption 3 (Exchangeability of Cooperative Strategies)
For any ( π 1 , π 2 ) ∈
Π C and ( π (cid:48) 1 , π (cid:48)
) ∈
Π C we have that ( π (cid:48)
i , π j ) ∈ Π C .
Note that if this exchangeability assumption is not satisﬁed we have both a cooperation problem (should agents defect?) and also a coordination problem (if we choose to cooperate, how do we choose among the multiple potential ways to cooperate?)
We point the reader to Kleiman-Weiner et al.
(2016) for further discussion of this issue.
Solving the coordination problem (eg. by introducing communication) is beyond the scope of this paper though it is an important avenue for future work.
To construct a CCC agent we need access to a policy pair ( π D 1 , π D 2 ) that forms a Nash equilibrium of the game.
We assume that these strategies generalize two properties of defection in the Prisoner’s
Dilemma: 1) the have lower rates of payoff than the socially optimal strategies in the long run for both players and 2) if we take a mixed policy π C + D which behaves according to π C at some periods and π D at others then V i ( π C
+ D i , π Cj ) ≥
V i ( π Ci , π Cj ) .
This last condition is essentially saying that π D is a selﬁsh policy that, even if used some of the time, still increases the payoffs of the player choosing it (while decreasing total social efﬁciency).
We explain a weaker condition in the appendix.
To behave according to CCC our agent maintains a persistent state at each time period C it which is the current time-averaged reward it has received.
The agent plays according to π C if C it > T t and π D otherwise.
Let ρ CC be the rate associated with both players behaving according to π C and let ρ CD be the rate associated with our agent playing according to π C and the other agent behaving according to π D .
Let T =
(1 − α )
ρ CC + αρ CD , where 0 < α < 1 is a slack parameter that speciﬁes the agent’s leniency.
We present the following result:
Theorem 1 Consider a strategy where agent 1 acts according to π C 1 if C Ti > T and π D 1 otherwise.
This gives two guarantees in the long run:
Cooperation Wins: If agent 2 acts according to π C 2 then for both agents lim t →∞ 1 t (cid:80) r it
= ρ iCC .
Defecting Doesn’t Pay: If agent 2 acts according to a policy that gives agent 1 a payoff of less than T in the long run then lim t →∞ 1 t (cid:80) r 2 t ≤ ρ 2 DD < ρ 2 CC .
Thus, a simple threshold based strategy for player 1 makes cooperation a better strategy than defection in the long-run for player 2 in any ergodic game.
CCC satisﬁes the desiderata we set out at the beginning: it is simple to understand, cooperates with cooperators, does not get exploited by pure defectors, incentivizes rational partners to cooperate, and, importantly, gives a way to return to cooperation if it has failed in the past.
The CCC strategy also provides a payoff guarantee against rational learning agents: if one’s partner is a learning agent who best responds in the long-run then, since π D forms an equilibrium, a CCC agent can always guarantee themselves a payoff of at least ρ DD in the long-run.
Unfortunately, CCC does not give any adversarial guarantees in general.
Extending conditionally cooperative strategies to be robust to not only selﬁsh agents trying to cheat but adversarial agents trying to actively destroy value is an interesting direction for future work.
However, we note that the simple construction above only gives long-run guarantees.
We now focus on generalizing this strategy to work well in ﬁnite time as well as how to use RL methods to compute its components.
Finally, we note that this strategy can be extended to some non-payoff ergodic games.
For example, if there is a persistent but unknown state which affects both rewards equally (say, multiplies them by some factor) then the amount of inequality (eg.
the ratio or difference of rewards or more complex function such as those used by Fehr & Schmidt (1999)) can be used as a summary statistic.
RL I MPLEMENTATION OF CCC
To construct the ˆ π C and ˆ π D policies used by CCC, we follow the training procedure of Lerer &
Peysakhovich (2017).
We perform self-play with modiﬁed reward schedules to compute the two policies:
Selﬁsh - here both players get rewards equal to their own reward.
This is the standard self-play paradigm used in multi-agent zero-sum settings.
We refer to the learned policies here as ˆ π D 2.
Prosocial - here both players get rewards at each time step not only for their own reward, but also for the reward their partner receives.
We refer to the learned polices as ˆ π C
Our agents are set up as standard deep RL agents which take game state (e.g. pixels) as input and pass them through a convolutional neural network to compute a distribution over actions.
The architectures of the agents as well as the training procedures are standard and we put them in the appendix.
We note that learning policies via RL in POMDPs has unique challenges (Jaakkola et al., 1995).
The correct choice of learning algorithm will depend on the situation; policy gradient is preferred for
POMDPs because optimal policies may be nondeterministic, and a common approach is to perform a variant of policy gradient with a function approximator augmented with an RNN ‘memory’ that keeps track of past states (Heess et al., 2015).
In our Fishery game, the policy (but not the value) is independent of the unobserved state, so RNN augmentation was unnecessary; however, since our policy was stateless we had to avoid value-based methods (e.g. actor-critic) because the aliasing of values prevents these methods from ﬁnding good policies.
Having computed the policies, we need to compute thresholds for conditional cooperation.
There are 3 important sources of ﬁnite time variance that a threshold needs to account for: ﬁrst, a partner’s π C may not be the same as our agent’s due to function approximation.
Second, initial states of the game may matter in ﬁnite time.
Third, there may be inherent stochasticity in the rewards.
We compute the per-turn threshold ˆ T it as follows:
ﬁrst we take our learned ˆ π C and ˆ π D and perform k rollouts of the full game assuming cooperate (we call the resulting per period cumulative payoffs to our agent ˆ R tkCC where k corresponds to the iteration and t to the time).
We also compute batches of rollouts of a paired cooperator and defector ˆ R tkCD .
We let ¯ R tCC be the bottom q th percentile of these sample paths and we deﬁne our time dependent threshold in terms of cumulative reward as
[Equation]
If the CCC agent’s current cumulative reward is above ˆ T t they behave according to ˆ π C , otherwise they use ˆ π D .
This process gives us slack to account for the three sources of error described above.
Tuning the parameters q, α allows us to trade off between the importance of false positives (detecting defection when one hasn’t occurred) and false negatives (missing a defecting opponent).
The algorithm is formalized into pseudocode below.
We also show an example of threshold computation as well as associated precision/recall with actual opponents in the example below.
Algorithm 1 CCC as Agent
Input: ˆ π C , ˆ π D , α, q, k for b in range (0 , k ) do s CC [ b ] ← NewGame () s CD [ b ] ← NewGame () while Game do for b in range (0 , k ) do s CC [ b ] , R CC [ b ] ← Step ( s CC [ b ] , ˆ π C 1 , ˆ π C 2 ) (cid:46)
Step returns next state and total reward s CD [ b ] , R CD [ b ] ← Step ( s CD [ b ] , ˆ π C 1 ,
ˆ π D 2 ) ¯ R CC ← quantile ( R CC , q ) ¯
R CD ← mean ( R CD )
T ← (1 − α ) ¯ R CC + α ¯ R CD if CurrentTotalReward < T then
Choose a = ˆ π D 1 ( o ) else
Choose a = ˆ π C 1 ( o )
E XPERIMENTS 4. E XPERIMENT : F ISHERY
Our ﬁrst example is a common pool resource game which we call Fishery.
In Fishery two agents live on different sides of a lake in which ﬁsh appear.
Each side of the lake is instantiated as a 5 × 5 grid and agents can walk in all cardinal directions.
Fish spawn randomly, starting young and swim to the other side and become mature.
Agents can catch ﬁsh on their side of the lake by walking over them.
Catching young ﬁsh yields 1 reward while mature ﬁsh yield a reward of 3.
In Fishery cooperative strategies are those which leave young ﬁsh for one’s partner.
However, there is always a temptation to defect and catch both young and mature ﬁsh.
Fishery is an imperfect observation game because agents cannot see the behavior of their partners across the lake.
Figure 1 shows an example of a threshold in our Fishery game with q = .
1 and α = .
05 , we see that CC
[Figure]
trajectories remain mostly above the threshold (meaning low false positives) and CD trajectories mostly lie below even after a short time period (meaning low false negatives).
The game as well as the experimental results are shown in Figure 1.
We train 50 pairs of agents under the selﬁsh and prosocial reward schemes using a policy gradient method with simple CNN policy approximators (see Appendix for details).
We see that selﬁsh training leads to agents that defect and choose greedy, suboptimal strategies whereas prosocial training ﬁnds good policies.
We then compute thresholds as described above and implement CCC agents.
To see how CCC would perform in a mixed population we consider a tournament where we draw random policies from our trained pool of 50 and have them play a 1000 time step Fishery game (we use ﬁxed lengths to normalize the payoffs across games).
We ﬁnd that CCC agents cooperate with cooperators, are not exploited by defectors, and do well when matched with other CCC agents.
4. E XPERIMENT : P ONG P LAYERS ’ D ILEMMA
CCC can also be used in games of perfect information such as the Pong Players’ Dilemma (PPD).
In the PPD we alter the reward structure of Atari Pong so that whenever an agent scores a point they receive a reward of 1 and the other player receives − 2 (Tampuu et al., 2017).
In the PPD the only (jointly) winning move is not to play.
However, selﬁsh agents are again tempted to defect and try to score points even though this decreases total social reward.
[Figure]
We train 18 pairs of agents under the selﬁsh and prosocial reward schemes using randomly determined game lengths using a standard A3C implementation (Mnih et al.
; see Appendix for complete details).
As with the games above we see that selﬁsh training leads to selﬁsh agents that try hard to score every point while prosocial training leads to cooperative agents that hit the ball directly back and forth.
We construct CCC agents as in the experiments above and see how they perform against other strategies in ﬁxed length PPD games.
Again we ﬁnd that CCC cooperates with cooperators (and itself) but does not get exploited by defectors.
An alternative way to construct a conditionally cooperative agent is to be forward looking rather than outcome based.
This is the construction employed by approximate Markov Tit-for-Tat (amTFT
Lerer & Peysakhovich (2017)).
amTFT conditions its cooperation on a counterfactual future rewardthe amTFT agent sees the actions taken by their partner and, if the action is not the one suggested by π C uses the game’s Q function (which is learned or simulated via rollouts) to estimate the one shot-gain to the partner from taking this action.
The agent keeps track of the total ‘debit’ a partner has accrued over time and if that crosses a threshold the amTFT agent then behaves according to π D for enough periods such that the partner’s debit is wiped out.
We call this type of strategy intention-based because it computes, at the time of the action, the expected future consequences rather than waiting for those consequences to occur as CCC agents do.
Though amTFT is more computationally expensive because it requires the use of a Q function (which can be hard to train) or rollouts (which are expensive to compute) we follow the procedure in Lerer &
Peysakhovich (2017) to construct amTFT agents for the PPD.
We see that CCC agents, which are much simpler, perform just as well.
However, CCC is not always as effective as forward looking strategies.
In particular, intentionbased strategies can be effective on much shorter timescales than CCC.
To demonstrate this we modify the reward structure of the PPD so that when a player scores, instead of their partner losing 2 points deterministically they lose 2 /p points with probability p .
Here the expected rewards of non-cooperation are the same as in the PPD and so amTFT acts identically (though now we require large batches if using rollouts).
However, we see that in intermediate length games ( 1000 time steps) and p = .
1 CCC agents can be exploited by defectors.
6 A screenshot of the game as well as experimental results are reported in Figure 2.
C ONCLUSION , L IMITATIONS AND F UTURE W ORK
In this work we have introduced consequentialist conditionally cooperative strategies and shown that they are useful heuristics in social dilemmas, even in those where information is imperfect either due to the structure of the game or due to the fact that we cannot perfectly forecast a partner’s future actions.
We have shown that using one’s own reward stream as a summary statistic for whether to cooperate (or not) in a given period is guaranteed to work in the limit as long as the underlying game is ergodic.
Note that this sometimes (but not always) gives good ﬁnite time guarantees.
In particular, the time scale for a CCC agent to detect exploitation is related to the mixing time of the POMG and the stochasticity of rewards; if these are large, then correspondingly long games are required for CCC to perform well.
We have also compared consequentialist and forward-looking models.
As another simple example of the difference between the two we can consider the random Dictator Game (rDG) introduced by
Cushman et al.
In the rDG, individuals are paired, one (the Dictator) is given an amount of money to split with a Partner, and chooses between one of two dice, a ‘fair’ die which yields a 50 −split with a high probability and an unfair split with a low probability and an ‘unfair’ die which yields a 50 − 50 split with low probability.
Consequentialist conditional cooperators would label a partner a defector if an unfair outcome came up (regardless of die choice) whereas intention-based cooperators would look at the choice of die, not the actual outcome.
For RL trained agents, conditioning purely on intentions (eg. amTFT) has advantages in that it is forward looking and doesn’t require ergodicity assumptions but it is an expensive strategy that is complex (or impossible) to implement for POMDPs and requires very precise estimates of potential outcomes.
CCC is simple, works in POMDPs and requires only information about payoff rates (rather than actual policies), however it may take a long time to converge.
Each has unique advantages and disadvantages.
Therefore constructing agents that can solve social dilemmas will require combining consequentialist and intention-based signals.
Interestingly, experimental evidence shows that while humans combine both intentions and outcomes, we often rely much more heavily on consequences than ‘optimal’ behavior would demand.
For example, experimental subjects rely heavily on the outcome of the die throw rather than die choice in the rDG (Cushman et al., 2009).
This is evidence for the notion that rather than acting optimally in each situation, humans have social heuristics which are tuned to work across many environments (Rand et al., 2014; Hauser et al., 2014; Ouss & Peysakhovich, 2015; Arechar et al., 2016; Mao et al., 2017; Niella et al., 2016).
There is much discussion of hybrid environments that include both artiﬁcial agents and humans (eg.
Shirado & Christakis (2017); Crandall et al. (2017)).
Constructing artiﬁcial agents that can do well in such environments will require going beyond the kinds of optimality theorems and experiments highlighted in this and related work.
To be successful, AI researchers will have to understand human social heuristics and construct agents that are in tune with human moral and social intuitions (Bonnefon et al., 2016; Greene, 2014).
T ECHNICAL A PPENDIX 6. P ROOF OF M AIN T HEOREM
We will use this basic property of almost sure convergence.
If a sequence of random variables X n converges to X almost surely then
[Equation]
To make notation easier we can write that ρ CC > 0 and ρ CD = 0 .
The intuition behind the proof is as follows:
ﬁrst, we show that if the CCC agent’s partner chooses according to π C then for any (cid:15) > 0 and any slack s > 0 the CCC agent receives a payoff of at least
R init =
s + tT with probability 1 − (cid:15) .
Intuitively this is because the rate T is lower than ρ CC which is weakly lower than the rate guaranteed to a CCC agent whose partner behaves according to π C always.
Second, we will show that for any (cid:15) > 0 , if the CCC agent’s partner behaves according to π C then we can ﬁnd an R init such that if a CCC agent’s total payoff is at period t is at least R init then with probability (1 − (cid:15) )
the CCC agent also only plays π C from that point on (this is given by almost sure convergence).
Together this implies that if the partner plays according to π C then with probability (1 − 2 (cid:15) )
the CCC agent behaves according to π D only a ﬁnite amount of times and thus the rates of payoffs for both agents converge to ρ CC .
(Part 1)
Let R t be the total reward of the CCC agent at time t and pick some R init .
We assumed that ρ C + D,C ≥ ρ CC > T .
Therefore if the partner plays π C then R t t → ρ CCC,C > T .
We set δ =
ρ CC − Tand use almost sure convergence to say that
[Equation]
s ,
[Equation]
(cid:19) > 1 − (cid:15).
Let t (cid:48) s = max (cid:16) t s ,
R init ( ρ CC − T ) /(cid:17) .
Then after t (cid:48) s turns the CCC agents total reward will be above tT by at least R init with probability at least 1 − (cid:15) .
(Part
[Equation]
If we start at some R init and assume that both players choose π C forever then by reward ergodicity and the construction that ρ CC > T we have
[Equation]
Assuming bounded rewards the CCC agent’s accumulated reward in the ﬁrst t 0 turns is no less than − t 0 r max .
So, if the CCC agent starts with t 0 r max + tT , then with probability 1 − (cid:15) , their time average reward will never fall below tT and thus they will always play π C .
This validates the initial assumption that both players play π C .
Putting it all together, for any (cid:15) > 0 there exists a R init that guarantees that if the CCC agent starts with total reward given by R init then both players play π C and get average rewards according to ρ CC .
That initial balance will be achieved with probability 1 − (cid:15) after t (cid:48)
s periods.
Therefore, with probability at least 1 − 2 (cid:15) , both players will achieve average reward of ρ CC .
Defecting Doesn’t Pay:
Suppose Agent 1’s average reward converges to ρ < T .
Then for some t 0 we have that R t t < T for all t > t 0 .
Starting at t 0 , CCC plays a ﬁxed (stateless) policy π D 1 .
Since π D forms an equilibrium agent 2 can achieve average reward of at most ρ DD .
6. A N OTE ON π C + D PAYOFFS
While constructing CCC strategies, we make the assumption that the payoff of the pair ( π C + D , π C ) to agent 1 is at least ρ CC .
This is an overly restrictive condition, because it precludes methods of cooperation that take more than one period to execute.
For example, if cooperation is walking to the left side of the room and defection is walking to the right side of the room, then a mix of the two policies may leave me in the middle gaining no reward.
In practice, CCC likely works even when this assumption doesn’t strictly hold (e.g. in Pong and
Coins), but with a slight modiﬁcation to CCC we can use a much weaker condition: Let π kC + kD be a policy that switches between π C and π D at most once every k periods.
Then we must only assume that there exists k such that the payoff to agent 1 of ( π kC + kD , π C ) is at least ρ CC .
In other words, as long as the Agent plays long enough stretches of π C or π D against π C , their reward will be at least ρ C C .
The modiﬁed CCC uses two thresholds: T D =
(1 − α D ) ρ CC + α D ρ CD is the threshold to switch from π C to π D , and T C =
(1 − α C ) ρ CC + α C ρ CD is the threshold to switch back, with α D > α C .
These thresholds grow linearly as t → ∞ so the intervals between CCC switching policies has to grow linearly as well given bounded rewards.
6. F ISHERY AND C OINS T RAINING
The Fishery game has a 5 × 10 state space, where each agent is conﬁned to, and observes, only their 5 × 5 half of the game.
Coins is identical to Lerer & Peysakhovich (2017) but with the board size expanded to 8 × 8 .
We follow the model architecture and training setup from Lerer & Peysakhovich (2017), with the modiﬁcation that models are trained via policy gradient.
We found that value-based methods (e.g. using a neural network baseline, or actor-critic) were unable to train prosocial policies in Fishery when using a stateless model.
We suspect this is because states of very different prosocial value - ‘I just ate the ﬁsh’ vs. ‘I let the ﬁsh swim to my partner’ - are aliased in the observation space.
Policy gradient with standard batch baselining produced converged policies for both Coins and Fishery.
We trained Coins for 100,000 games and Fishery for 40,000 games.
For the tournament, agents play against opponents they have never trained against.
The CCC statistics are computed at test time by rolling out a batch of 32 games for CC and 32 games for CD, updating the rollouts by one step per game step.
Alternatively, the statistics (in particular, the defect threshold) could be pre-computed once for the longest game length of interest, and used in subsequent interactions.
6. P ONG P LAYER D ILEMMA T RAINING
We use the ALE environment modiﬁed for 2-player play as proposed in Tampuu et al.
(2017), with modiﬁed rewards of +1 for scoring a point and -2 for being scored on.
We train policies directly from pixels, using the pytorch-a3c package https://github.com/ ikostrikov/pytorch-a3c .
Policies are trained directly from pixels via A3C (Mnih et al., 2016).
Inputs are rescaled to 42x and normalized, and we augment the state with the difference between successive frames.
We use 38 threads for A3C, over a total of 38,000 games (1,000 per thread).
We use the default settings from pytorch-a3c: a discount rate of 0 . 99 , learning rate of 0 . 0001 , 20-step returns, and entropy regularization weight of 0 . 01 .
The policy is implemented as a convolutional neural network with four layers, following pytorch-a3c.
Each layer uses a 3x3 kernel with stride 2, followed by ELU.
The network has two heads for the actor and critic.
We elide the LSTM layer used in the pytorch-a3c library, as we found it to be unnecessary.
[Figure]
D e f e c t i on s
T o t a l P a y o ff
Number Games Played
Training
Prosocial
Selfish (b) Training Curves
[Figure]
6. E XPERIMENT : C OINS
We show that CCC maintains cooperation in the Coins game from Lerer & Peysakhovich (2017).
In
Coins two agents, Red and Blue, live on an 8 × 8 grid and move around in all cardinal directions 7 .
Coins randomly appear on the board and each have a color.
agent picks up (moves over) coin, they receive up a the
In Coins, the cooperative strategy is to pick up coins of one’s color only.
However, there is always a temptation to grab the other agents’ coins.
We train 10 copies of each strategy type under our two reward schemes and again compare payoffs of various policy combinations (see Figure 3).
Note that Coins is a fully observed Markov game, however we still see that CCC (which has only limit guarantees and throws away all ‘forward’ information contained in an action) is just as effective as the intention based TFT strategies (which incentivize cooperation at all time periods) either built using Q functions (Lerer & Peysakhovich, 2017) or learned via an end-to-end procedure (Foerster et al., 2017c).