Original: S IMPLIFIED A CTION D ECODER FOR D EEP M ULTI -A GENT R EINFORCEMENT L EARNING
Revised: S IMPLIFIED A CTION D ECODER FOR D EEP M ULTI -A GENT R EINFORCEMENT L EARNING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In recent years we have seen fast progress on a number of benchmark problems in
Revised: In recent years we have seen fast progress on a number of benchmark problems in

Original: AI, with modern methods achieving near or super human performance in Go, Poker and Dota.
Revised: AI, with modern methods achieving near or super human performance in Go, Poker and Dota.

Original: One common aspect of all of these challenges is that they are by design adversarial or, technically speaking, zero-sum.
Revised: One common aspect of all of these challenges is that they are by design adversarial or, technically speaking, zero-sum.

Original: In contrast to these settings, success in the real world commonly requires humans to collaborate and communicate with others, in settings that are, at least partially, cooperative.
Revised: In contrast to these settings, success in the real world commonly requires humans to collaborate and communicate with others, in settings that are, at least partially, cooperative.

Original: In the last year, the card game Hanabi has been established as a new benchmark environment for AI to ﬁll this gap.
Revised: In the last year, the card game Hanabi has been established as a new benchmark environment for AI to ﬁll this gap.

Original: In particular, Hanabi is interesting to humans since it is entirely focused on theory of mind , i.e. , the ability to effectively reason over the intentions, beliefs and point of view of other agents when observing their actions.
Revised: In particular, Hanabi is interesting to humans since it is entirely focused on theory of mind , i.e. , the ability to effectively reason over the intentions, beliefs and point of view of other agents when observing their actions.

Original: Learning to be informative when observed by others is an interesting challenge for Reinforcement
Revised: Learning to be informative when observed by others is an interesting challenge for Reinforcement

Original: Learning (RL): Fundamentally, RL requires agents to explore in order to discover good policies.
Revised: Learning (RL): Fundamentally, RL requires agents to explore in order to discover good policies.

Original: However, when done naively, this randomness will inherently make their actions less informative to others during training.
Revised: However, when done naively, this randomness will inherently make their actions less informative to others during training.

Original: We present a new deep multi-agent RL method, the Simpliﬁed Action Decoder (SAD), which resolves this contradiction exploiting the centralized training phase.
Revised: We present a new deep multi-agent RL method, the Simpliﬁed Action Decoder (SAD), which resolves this contradiction exploiting the centralized training phase.

Original: During training SAD allows other agents to not only observe the ( exploratory ) action chosen, but agents instead also observe the greedy action of their team mates.
Revised: During training SAD allows other agents to not only observe the ( exploratory ) action chosen, but agents instead also observe the greedy action of their team mates.

Original: By combining this simple intuition with best practices for multi-agent learning, SAD establishes a new SOTA for  2 - 5 players on the self-play part of the Hanabi challenge.
Revised: By combining this simple intuition with best practices for multi-agent learning, SAD establishes a new SOTA for learning methods for 2 - 5 players on the self-play part of the Hanabi challenge.

Original: Our ablations show the contributions of SAD compared with the best practice components.
Revised: Our ablations show the contributions of SAD compared with the best practice components.

Revised: All of our code and trained agents are available at https://github.com/facebookresearch/Hanabi_SAD .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Humans are highly social creatures and spend vast amounts of time coordinating, collaborating and communicating with others.
Revised: Humans are highly social creatures and spend vast amounts of time coordinating, collaborating and communicating with others.

Original: In contrast to these, at least partially,  most progress on AI in games has been in zero-sum games where agents compete against each other, typically rendering communication futile.
Revised: In contrast to these, at least partially, cooperative settings most progress on AI in games has been in zero-sum games where agents compete against each other, typically rendering communication futile.

Original: This includes examples such as Go (Silver et al., 2016; 2017; 2018), poker (Brown & Sandholm, 2017; Morav ˇ cík et al., 2017; Brown & Sandholm, 2019) and chess (Campbell et al., 2002).
Revised: This includes examples such as Go (Silver et al., 2016; 2017; 2018), poker (Brown & Sandholm, 2017; Moravˇcík et al., 2017; Brown & Sandholm, 2019) and chess (Campbell et al., 2002).

Original: This narrow focus is unfortunate, since communication and coordination require unique abilities.
Revised: This narrow focus is unfortunate, since communication and coordination require unique abilities.

Original: In order to enable smooth and efﬁcient social interactions of groups of people, it is commonly required to reason over the intents, points of views and beliefs of other agents from observing their actions.
Revised: In order to enable smooth and efﬁcient social interactions of groups of people, it is commonly required to reason over the intents, points of views and beliefs of other agents from observing their actions.

Original: For example, a driver can reasonably infer that if a truck in front of them is slowing down when approaching an intersection, then there is likely an obstacle ahead.
Revised: For example, a driver can reasonably infer that if a truck in front of them is slowing down when approaching an intersection, then there is likely an obstacle ahead.

Original: Furthermore, humans are both able to interpret the actions of others and can act in a way that is informative when their actions are being observed by others, capabilities that are commonly called theory of Mind (ToM), (Baker et al., 2017).
Revised: Furthermore, humans are both able to interpret the actions of others and can act in a way that is informative when their actions are being observed by others, capabilities that are commonly called theory of Mind (ToM), (Baker et al., 2017).

Original: Importantly, in order to carry out this kind of reasoning, an agent needs to consider why a given action is taken and what this decision indicates about the state of the world.
Revised: Importantly, in order to carry out this kind of reasoning, an agent needs to consider why a given action is taken and what this decision indicates about the state of the world.

Original: Simply observing what other agents are doing is not sufﬁcient.
Revised: Simply observing what other agents are doing is not sufﬁcient.

Original: While these abilities are particularly relevant in partially observable fully cooperative multi-agent settings, ToM reasoning clearly matters in a variety of real world scenarios.
Revised: While these abilities are particularly relevant in partially observable, fully cooperative multi-agent settings, ToM reasoning clearly matters in a variety of real world scenarios.

Original: For example, autonomous cars will likely need to understand the point of view, intents and beliefs of other trafﬁc participants in order to deal with highly interactive settings such as 4-way crossing or dense trafﬁc in cities.
Revised: For example, autonomous cars will likely need to understand the point of view, intents and beliefs of other trafﬁc participants in order to deal with highly interactive settings such as 4-way crossing or dense trafﬁc in cities.

Original: Hanabi is a fully cooperative partially-observable card game that has recently been proposed as a new benchmark challenge problem for AI research (Bard et al., 2019) to ﬁll the gap around ToM.
Revised: Hanabi is a fully cooperative, partially-observable card game that has recently been proposed as a new benchmark challenge problem for AI research (Bard et al., 2019) to ﬁll the gap around ToM.

Original: In
Revised: In

Original: Hanabi, players need to ﬁnd conventions that allow them to effectively exchange information from their local observations through their actions, taking advantage of the fact that actions are observed by all team mates.
Revised: Hanabi, players need to ﬁnd conventions that allow them to effectively exchange information from their local observations through their actions, taking advantage of the fact that actions are observed by all team mates.

Original: Most prior state-of-the-art agents for Hanabi were developed using handcrafted algorithms, which beat off-the-shelf deep multi-agent RL methods by a large margin.
Revised: Most prior state-of-the-art agents for Hanabi were developed using handcrafted algorithms, which beat off-the-shelf deep multi-agent RL methods by a large margin.

Original: This makes intuitive sense: Beyond the “standard” multi-agent challenges of credit assignment, non-stationarity and joint exploration, learning an informative policy presents an additional fundamentally new conﬂict.
Revised: This makes intuitive sense: Beyond the “standard” multi-agent challenges of credit assignment, nonstationarity and joint exploration, learning an informative policy presents an additional fundamentally new conﬂict.

Original: On the one hand, an RL agent needs to explore in order to discover good policies through trial and error.
Revised: On the one hand, an RL agent needs to explore in order to discover good policies through trial and error.

Original: On the other hand, when carried out naively, this exploration will add noise to the policy of the agent during the training process, making their actions strictly less informative to their team mates.
Revised: On the other hand, when carried out naively, this exploration will add noise to the policy of the agent during the training process, making their actions strictly less informative to their team mates.

Original: One possible solution to this is to explore in the space of deterministic partial policies, rather than actions, and sample these policies from a distribution that conditions on a common knowledge
Revised: One possible solution to this is to explore in the space of deterministic partial policies, rather than actions, and sample these policies from a distribution that conditions on a common knowledge

Original: Bayesian belief.
Revised: Bayesian belief.

Original: This is successfully carried out in the Bayesian Action Decoder (BAD) (Foerster et al., 2019), the only previous Deep RL method to achieve a state of the art in Hanabi.
Revised: This is successfully carried out in the Bayesian Action Decoder (BAD) (Foerster et al., 2019), the only previous Deep RL method to achieve a state-of-the-art in Hanabi.

Original: While this is a notable accomplishment, it comes at the cost of simplicity and generality.
Revised: While this is a notable accomplishment, it comes at the cost of simplicity and generality.

Original: For a start, BAD requires an explicit common knowledge Bayesian belief to be tracked, which not only adds computational burden due to the required sampling steps, but also uses expert knowledge regarding the game dynamics.
Revised: For a start, BAD requires an explicit common knowledge Bayesian belief to be tracked, which not only adds computational burden due to the required sampling steps, but also uses expert knowledge regarding the game dynamics.

Original: Furthermore, BAD, as presented, is trained using actor-critic methods which are sample inefﬁcient and suffer from local optima.
Revised: Furthermore, BAD, as presented, is trained using actor-critic methods which are sample inefﬁcient and suffer from local optima.

Original: In order to get around this, BAD uses population based training, further increasing the number of samples required.
Revised: In order to get around this, BAD uses population based training, further increasing the number of samples required.

Original: Lastly, BAD’s explicit reliance on common knowledge limits the generality of the method.
Revised: Lastly, BAD’s explicit reliance on common knowledge limits the generality of the method.

Original: In this paper we propose the Simpliﬁed Action Decoder (SAD), a method that achieves a similar goal to BAD, but addresses all of the issues mentioned above.
Revised: In this paper we propose the Simpliﬁed Action Decoder (SAD), a method that achieves a similar goal to BAD, but addresses all of the issues mentioned above.

Original: At the core of SAD is a different approach towards resolving the conﬂict between exploration and being interpretable, which, like BAD, relies on the centralized training with decentralized control (CT/DC) regime.
Revised: At the core of SAD is a different approach towards resolving the conﬂict between exploration and being interpretable, which, like BAD, relies on the centralized training with decentralized control (CT/DC) regime.

Original: Under CT/DC information can be exchanged freely amongst all agents during centralized training , as long as the ﬁnal policies are compatible with decentralized execution .
Revised: Under CT/DC information can be exchanged freely amongst all agents during centralized training , as long as the ﬁnal policies are compatible with decentralized execution .

Original: The key insight is that, during training we do not have to chose between being informative, by taking greedy actions, and exploring, by taking random actions.
Revised: The key insight is that during training we do not have to chose between being informative, by taking greedy actions, and exploring, by taking random actions.

Original: To be informative, the greedy actions do not need to be executed by the environment, but only need to be observed by the team mates.
Revised: To be informative, the greedy actions do not need to be executed by the environment, but only need to be observed by the team mates.

Original: Thus in SAD each agent takes two different actions at each time step: One greedy action, which is not presented to the environment but observed by the team mates at the next time step as an additional input, and the “standard” (exploratory) action that gets executed by the environment and is observed by the team mates as part of the environment dynamics.
Revised: Thus in SAD each agent takes two different actions at each time step: One greedy action, which is not presented to the environment but observed by the team mates at the next time step as an additional input, and the “standard” (exploratory) action that gets executed by the environment and is observed by the team mates as part of the environment dynamics.

Original: Importantly, during greedy execution the observed environment action can be used instead of centralized information for the additional input, since now the agent has stopped exploring.
Revised: Importantly, during greedy execution the observed environment action can be used instead of centralized information for the additional input, since now the agent has stopped exploring.

Original: Furthermore, to ensure that these greedy actions and observations get decoded into a meaningful representation, we can optionally train an auxiliary task that predicts key hidden game properties from the action-observation trajectories.
Revised: Furthermore, to ensure that these greedy actions and observations get decoded into a meaningful representation, we can optionally train an auxiliary task that predicts key hidden game properties from the action-observation trajectories.

Original: While we note that this idea is in principle compatible with any kind of model-free deep RL method with minimal modiﬁcations to the core algorithm, we use a distributed version of recurrent DQN in order to improve sample efﬁciency, account for partial observability and reduce the risk of local optima.
Revised: While we note that this idea is in principle compatible with any kind of model-free deep RL method with minimal modiﬁcations to the core algorithm, we use a distributed version of recurrent DQN in order to improve sample efﬁciency, account for partial observability and reduce the risk of local optima.

Original: We also train a joint-action Q-function that consists of the sum of per-agent Q-values to allow for off-policy learning in this multi-agent setting using
Revised: We also train a joint-action Q-function that consists of the sum of per-agent Q-values to allow for off-policy learning in this multi-agent setting using

Original: Value Decomposition Networks (VDN) (Sunehag et al., 2017).
Revised: Value Decomposition Networks (VDN) (Sunehag et al., 2017).

Original: Using SAD we establish a new SOTA for  2-5 players in Hanabi, with a method that not only requires less expert knowledge and compute, but is also more general than previous approaches.
Revised: Using SAD we establish a new SOTA for learning methods for 2-5 players in Hanabi, with a method that not only requires less expert knowledge and compute, but is also more general than previous approaches.

Original: In order to ensure that our results can be easily veriﬁed and extended, we also evaluate our method on a proof-of-principle matrix game andplan to open-source our training code and agents.
Revised: In order to ensure that our results can be easily veriﬁed and extended, we also evaluate our method on a proof-of-principle matrix game and open-source our training code and agents.

Original: Beyond enabling more research into the self-play aspect of Hanabi, we believe these resources will provide a much needed starting point for the ad-hoc teamwork part of the Hanabi challenge.
Revised: Beyond enabling more research into the self-play aspect of Hanabi, we believe these resources will provide a much needed starting point for the ad-hoc teamwork part of the Hanabi challenge.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Our work relates closely to research on emergent communication protocols using deep multi-agent
Revised: Our work relates closely to research on emergent communication protocols using deep multi-agent

Original: RL, as ﬁrst undertaken by Sukhbaatar et al.
Revised: RL, as ﬁrst undertaken by Sukhbaatar et al.

Original: (2016) and Foerster et al.
Revised: (2016) and Foerster et al.

Original: There has been a large number of follow-up papers in this area, so listing all relevant work is beyond the scope and we refer the reader to Nguyen et al.
Revised: There has been a large number of follow-up papers in this area, so listing all relevant work is beyond the scope and we refer the reader to Nguyen et al.

Original: (2018), a recent survey on deep multi-agent RL.
Revised: (2018), a recent survey on deep multi-agent RL.

Original: One major difference to our work is that the environments considered typically contain a cheap-talk channel, which can be modeled as a continuous variable during the course of training.
Revised: One major difference to our work is that the environments considered typically contain a cheap-talk channel, which can be modeled as a continuous variable during the course of training.

Original: This allows agents to, for example, use differentiation across the communication channel in order to learn protocols.
Revised: This allows agents to, for example, use differentiation across the communication channel in order to learn protocols.

Original: In contrast, in our setting agents have to communicate through the observable environment actions themselves, requiring fundamentally different methods.
Revised: In contrast, in our setting agents have to communicate through the observable environment actions themselves, requiring fundamentally different methods.

Original: Furthermore, our work is an example of cooperative multi-agent learning in partially observable settings under centralized training and decentralized control.
Revised: Furthermore, our work is an example of cooperative multi-agent learning in partially observable settings under centralized training and decentralized control.

Original: There have been a large number of papers in this space, with seminal work including MADDPG (Lowe et al., 2017) and COMA (Foerster et al., 2018a), both of which are actor-critic methods that employ a centralized critic with decentralized actors.
Revised: There have been a large number of papers in this space, with seminal work including MADDPG (Lowe et al., 2017) and COMA (Foerster et al., 2018a), both of which are actor-critic methods that employ a centralized critic with decentralized actors.

Original: Again, we refer the reader to Nguyen et al.
Revised: Again, we refer the reader to Nguyen et al.

Original: for a more comprehensive survey.
Revised: for a more comprehensive survey.

Original: Until 2018, work on Hanabi had been focused on hand-coded methods and heuristics.
Revised: Until 2018, work on Hanabi had been focused on hand-coded methods and heuristics.

Original: Some relevant examples include SmartBot (O’Dwyer, 2019) and the so-called “hat-coding” strategies, as implemented by WTFWThat (Wu, 2018).
Revised: Some relevant examples include SmartBot (O’Dwyer, 2019) and the so-called “hat-coding” strategies, as implemented by WTFWThat (Wu, 2018).

Original: These strategies use the information theoretic ideas that allow each hint to reveal information to all other agents at the same time.
Revised: These strategies use the information theoretic ideas that allow each hint to reveal information to all other agents at the same time.

Original: While they do not perform well for 2-player Hanabi due to the smaller action space, they get near perfect scores for 3-5 players.
Revised: While they do not perform well for 2-player Hanabi due to the smaller action space, they get near perfect scores for 3-5 players.

Original: In contrast, so far learning methods have seen limited success on Hanabi.
Revised: In contrast, so far learning methods have seen limited success on Hanabi.

Original: Bard et al.
Revised: Bard et al.

Original: (2019) undertake a systematic evaluation of current Deep RL methods for 2-5 players in two different regimes and open-source the Hanabi-Learning-Environment (HLE) to foster research on the game.
Revised: (2019) undertake a systematic evaluation of current Deep RL methods for 2-5 players in two different regimes and open-source the Hanabi-Learning-Environment (HLE) to foster research on the game.

Original: They evaluate a feed-forward version of DQN trained on 100 million samples and a recurrent actor-critic agent with population based training using 20 billion samples.
Revised: They evaluate a feed-forward version of DQN trained on 100 million samples and a recurrent actor-critic agent with population based training using 20 billion samples.

Original: Notably both agents achieve near 0% win rate for 3-5 players in Hanabi. At a high level their DQN agent is a good starting point for our work.
Revised: Notably, while both agents achieve near 0% win rate for 3-5 players in Hanabim, at a high level their DQN agent is a good starting point for our work.

Original: However, since the authors did not propose any speciﬁc method of accounting for the issues introduced by (cid:15) -greedy exploration in a ToM task, they resorted to setting (cid:15) to zero after a short burn-in phase.
Revised: However, since the authors did not propose any speciﬁc method of accounting for the issues introduced by (cid:15) -greedy exploration in a ToM task, they resorted to setting (cid:15) to zero after a short burn-in phase.

Original: The only state-of-the-art in Hanabi established by an RL agent is from Foerster et al.
Revised: The only state-of-the-art in Hanabi established by an RL agent is from Foerster et al.

Original: (2019) which we refer to in more detail in Section 1 and Section 4.
Revised: (2019) which we refer to in more detail in Section 1 and Section 4.

Original: Recently there have also been attempts to train agents that are robust to different team-mates (Canaan et al., 2019) and even to extend to human-AI collaboration (Liang et al., 2019).
Revised: Recently there have also been attempts to train agents that are robust to different team-mates (Canaan et al., 2019) and even to extend to human-AI collaboration (Liang et al., 2019).

Revised: For a more comprehensive review on previous results on Hanabi we refer the reader to Bard et al.

Original: Poker is another partially observable multi-agent setting, although it is fundamentally different due to the game being zero-sum.
Revised: Poker is another partially observable multi-agent setting, although it is fundamentally different due to the game being zero-sum.

Original: Recent success in Poker has extensively beneﬁted from search (Brown et al.).
Revised: Recent success in Poker has extensively beneﬁted from search (Brown et al.).

Original: Examples of using search in Hanabi include Goodman (2019).
Revised: Examples of using search in Hanabi include Goodman (2019).

Original: For a more comprehensive review on previous results on Hanabi we refer the reader to Bard et al.

Original:  B ACKGROUND 3. S ETTING
Revised:  B ACKGROUND 3. S ETTING

Original: In this paper we assume a Dec-POMDP (Oliehoek, 2012), in which N agents interact in a partially observable environment.
Revised: In this paper we assume a Dec-POMDP (Oliehoek, 2012), in which N agents interact in a partially observable environment.

Original: At each time step agent a ∈ 1 ..N obtains an observation, o at =
Revised: At each time step agent a ∈ 1 ..N obtains an observation, o at =

Original: O ( s t , a ) , where s t ∈ S is the Markov state of the system and O ( s t , a ) is the deterministic observation function.
Revised: O ( s t , a ) , where s t ∈ S is the Markov state of the system and O ( s t , a ) is the deterministic observation function.

Original: Since we are interested in ToM, in our setting the observation function includes the last action of the acting agent, which is observed by all other agents at the next time step.
Revised: Since we are interested in ToM, in our setting the observation function includes the last action of the acting agent, which is observed by all other agents at the next time step.

Original: We note that actions are commonly observable not only in board games but also in some real world multi-agent settings, such as autonomous driving.
Revised: We note that actions are commonly observable not only in board games but also in some real world multi-agent settings, such as autonomous driving.

Original: For simplicity, we restrict ourselves to turn based settings, in which at each time step only the acting agents takes an action, u at , which is sampled from their policy, u a ∼ π aθ ( u a | τ a ) , while all other agents take a no-op action.
Revised: For simplicity, we restrict ourselves to turn based settings, in which at each time step only the acting agents takes an action, u at , which is sampled from their policy, u a ∼ π aθ ( u a | τ a ) , while all other agents take a no-op action.

Original: Here τ a is the action-observation history of agent a , τ a = { o a 0 , u a 0 , r 1 , ..
Revised: Here τ a is the action-observation history of agent a , τ a = { o a 0 , u a 0 , r 1 , ..

Original: r T , o aT } ,
Revised: r T , o aT } ,

Original: T is the length of the episode and θ are the weights of a function approximator that represents the policy, in our case recurrent neural networks, such as LSTMs (Hochreiter & Schmidhuber, 1997).
Revised: T is the length of the episode and θ are the weights of a function approximator that represents the policy, in our case recurrent neural networks, such as LSTMs (Hochreiter & Schmidhuber, 1997).

Original: We further use τ t to describe the state-action sequence, τ = { s 0 , u 0 , r 1 , ..r T , s T } , where u t is the joint action of all agents.
Revised: We further use τ t to describe the state-action sequence, τ = { s 0 , u 0 , r 1 , ..r T , s T } , where u t is the joint action of all agents.

Original: As is typical in cooperative multi-agent RL, the goal of the agents is to maximize the total expected return, J θ =
Revised: As is typical in cooperative multi-agent RL, the goal of the agents is to maximize the total expected return, J θ =

Original: E τ ∼ P ( τ | θ ) R 0 ( τ ) , where R 0 ( τ ) is the return of the trajectory (in general R t ( τ ) =
Revised: E τ ∼ P ( τ | θ ) R 0 ( τ ) , where R 0 ( τ ) is the return of the trajectory (in general R t ( τ ) =

Original: (cid:80) t (cid:48) ≥ t γ t (cid:48) − t r t (cid:48) ) and γ is an optional discount factor.
Revised: (cid:80) t (cid:48) ≥ t γ t (cid:48) − t r t (cid:48) ) and γ is an optional discount factor.

Original: We have also assumed that agents are sharing parameters, θ , as is common in cooperative MARL.
Revised: We have also assumed that agents are sharing parameters, θ , as is common in cooperative MARL.

Original: 3. D ISTRIBUTED R ECURRENT DQN AND A UXILIARY T ASKS
Revised: 3. D ISTRIBUTED R ECURRENT DQN AND A UXILIARY T ASKS

Original: In Q-learning the agent approximates the expected return for a given state action-pair, s, u , assuming that the agent acts greedily with respect to the Q-function for all future time steps, Q ( s, u ) =
Revised: In Q-learning the agent approximates the expected return for a given state action-pair, s, u , assuming that the agent acts greedily with respect to the Q-function for all future time steps, Q ( s, u ) =

Original: E τ ∼ P ( τ | s,u ) R t ( τ ) , where τ = ( s t , u t , r t +1 , . . .
Revised: E τ ∼ P ( τ | s,u ) R t ( τ ) , where τ = { s t , u t , r t +1 , . . .

Original: , s T ) , u t = u and u t (cid:48) =
Revised: , s T } , u t = u and u t (cid:48) =

Original: arg max u (cid:48) Q ( s t (cid:48) , u (cid:48) ) , ∀ t (cid:48) > t .
Revised: arg max u (cid:48) Q ( s t (cid:48) , u (cid:48) ) , ∀ t (cid:48) > t .

Original: A common exploration scheme is (cid:15) -greedy, in which the agent takes a random action with probability (cid:15) and acts greedily otherwise.
Revised: A common exploration scheme is (cid:15) -greedy, in which the agent takes a random action with probability (cid:15) and acts greedily otherwise.

Original: Importantly, the Q-function can be trained efﬁciently using the Bellman equation: Q ( s, u ) =
Revised: Importantly, the Q-function can be trained efﬁciently using the Bellman equation: Q ( s, u ) =

Original: E s (cid:48)
Revised: E s (cid:48)

Original: [ r t +1 + γ max u (cid:48)
Revised: [ r t +1 + γ max u (cid:48)

Original: Q ( s (cid:48) , u (cid:48) )] , where for simplicity we have assumed a deterministic reward.
Revised: Q ( s (cid:48) , u (cid:48) )] , where for simplicity we have assumed a deterministic reward.

Original: In Deep Q-Learning (DQN) (Mnih et al., 2015) the Q-function is parameterized by a deep neural network and trained with transitions sampled from experience replay.
Revised: In Deep Q-Learning (DQN) (Mnih et al., 2015) the Q-function is parameterized by a deep neural network and trained with transitions sampled from experience replay.

Original: In our work we also incorporate other best practice components of the last few years, including doubleDQN (van Hasselt et al., 2015), dueling network architecture (Wang et al., 2015) and prioritized replay (Schaul et al., 2015).
Revised: In our work we also incorporate other best practice components of the last few years, including doubleDQN (van Hasselt et al., 2015), dueling network architecture (Wang et al., 2015) and prioritized replay (Schaul et al., 2015).

Original: We also employ a distributed training architecture similar to the one proposed by Horgan et al.
Revised: We also employ a distributed training architecture similar to the one proposed by Horgan et al.

Original: (2018) where a number of different actors with their own exploration rates collect experiences in parallel and feed them into a central replay buffer.
Revised: (2018) where a number of different actors with their own exploration rates collect experiences in parallel and feed them into a central replay buffer.

Original: Since our setting is partially observable the natural choice for the function approximator is a recurrent neural network.
Revised: Since our setting is partially observable the natural choice for the function approximator is a recurrent neural network.

Original: A combination of these techniques was ﬁrst explored by Kapturowski et al.
Revised: A combination of these techniques was ﬁrst explored by Kapturowski et al.

Original: (2019) in single agent environments such as Atari and DMLab-30.
Revised: (2019) in single agent environments such as Atari and DMLab-30.

Original: Another common best-practice in RL are auxiliary tasks Mirowski et al.
Revised: Another common best-practice in RL are auxiliary tasks Mirowski et al.

Original: ; Jaderberg et al.
Revised: ; Jaderberg et al.

Original: (2016) , in which the agent produces extra output-heads that are trained on supervised task and optimized alongside the RL loss.
Revised: (2016), in which the agent produces extra output-heads that are trained on supervised tasks and optimized alongside the RL loss.

Original: 3. C ENTRALISED T RAINING , D ECENTRALIZED E XECUTION AND J OINT Q- FUNCTIONS
Revised: 3. C ENTRALISED T RAINING , D ECENTRALIZED E XECUTION AND J OINT Q- FUNCTIONS

Original: The most straight forward application of Q-learning to multi-agent settings is Independent Q-Learning (IQL) (Tan, 1993) in which each agent keeps an independent estimate of the expected return, treating all other agents as part of the environment.
Revised: The most straight forward application of Q-learning to multi-agent settings is Independent Q-Learning (IQL) (Tan, 1993) in which each agent keeps an independent estimate of the expected return, treating all other agents as part of the environment.

Original: One challenge with IQL is that the exploratory behavior of other agents is not corrected for via the max operator in the bootstrap.
Revised: One challenge with IQL is that the exploratory behavior of other agents is not corrected for via the max operator in the bootstrap.

Original: Notably, IQL does typically not take any advantage of centralized training with decentralised control (CT/DC), a paradigm under which information can be exchanged freely amongst agents during the training phase as long as the policies rely only on local observations during execution.
Revised: Notably, IQL does typically not take any advantage of centralized training with decentralised control (CT/DC), a paradigm under which information can be exchanged freely amongst agents during the training phase as long as the policies rely only on local observations during execution.

Original: There are various approaches for learning joint-Q-functions in the CT/DC regime.
Revised: There are various approaches for learning joint-Q-functions in the CT/DC regime.

Original: For example,
Revised: For example,

Original: Value-Decomposition-Networks (VDN) (Sunehag et al., 2017) represent the joint-Q-function as a sum of per-agent contributions and QMIX (Rashid et al., 2018) learns a non-linear but monotonic combination of these contributions.
Revised: Value-Decomposition-Networks (VDN) (Sunehag et al., 2017) represent the joint-Q-function as a sum of per-agent contributions and QMIX (Rashid et al., 2018) learns a non-linear but monotonic combination of these contributions.

Original:  M ETHOD 4. T HEORY OF M IND AND B AYESIAN R EASONING
Revised:  M ETHOD 4. T HEORY OF M IND AND B AYESIAN R EASONING

Original: At the very core of interpreting the actions of another agent, and ToM in general, is Bayesian reasoning.
Revised: At the very core of interpreting the actions of another agent, and ToM in general, is Bayesian reasoning.

Original: Fundamentally, asking what a given action by another agent implies about the state of the world requires understanding of why this action was taken.
Revised: Fundamentally, asking what a given action by another agent implies about the state of the world requires understanding of why this action was taken.

Original: To illustrate this, we start out with an agent that has a given belief about the state-action history of the world, τ t , given her own action-observation history τ at : B ( τ t ) =
Revised: To illustrate this, we start out with an agent that has a given belief about the state-action history of the world, τ t , given her own action-observation history τ at : B ( τ t ) =

Original: P ( τ t | τ at ) .
Revised: P ( τ t | τ at ) .

Original: Next the agent observes the action u a (cid:48) t of her team mate, a (cid:48) , and carries out a Bayesian update:
Revised: Next the agent observes the action u a (cid:48) t of her team mate, a (cid:48) , and carries out a Bayesian update:

Original: [Equation]
Revised: [Equation]

Original: where, with a slight abuse of notation, we have used (and will keep using)
Revised: where, with a slight abuse of notation, we have used (and will keep using)

Original: O ( a (cid:48) , τ t ) for the actionobservation history, τ a (cid:48) t , that results from applying the observation function for agent a (cid:48) to τ t at each time step.
Revised: O ( a (cid:48) , τ t ) for the actionobservation history, τ a (cid:48) t , that results from applying the observation function for agent a (cid:48) to τ t at each time step.

Original: Note that for non-deterministic observation functions we would have to marginalize over
Revised: Note that for non-deterministic observation functions we would have to marginalize over

Original: P ( τ a (cid:48) t | τ t ) .
Revised: P ( τ a (cid:48) t | τ t ) .

Original: Clearly, since agents have access to the policy of their teammate during centralised training, we could in principle evaluate this explicit Bayesian belief.
Revised: Clearly, since agents have access to the policy of their teammate during centralised training, we could in principle evaluate this explicit Bayesian belief.

Original: However, beyond the practical difﬁculty of computing this explicit belief, when it is used as an input to the policy it will lead to prohibitively costly higher order beliefs.
Revised: However, beyond the practical difﬁculty of computing this explicit belief, when it is used as an input to the policy it will lead to prohibitively costly higher order beliefs.

Original: The typical workout for this is a public belief over private features which only conditions on common knowledge and can therefore be calculated by all agents individually, we refer to Moravˇcík et al.
Revised: The typical workout for this is a public belief over private features which only conditions on common knowledge and can therefore be calculated by all agents individually, we refer to Morav ˇ cík et al.

Original: (2017); Nayyar et al.
Revised: (2017); Nayyar et al.

Original: ; Foerster et al.
Revised: ; Foerster et al.

Original: (2018b)
Revised: (2018b)

Original: for more details.
Revised: for more details.

Original: Instead, in this work we rely on RNNs to learn implicit representations of the sufﬁcient statistics over the distribution of the Markov state given the action-observation histories, noting that they are unlikely to recover exact beliefs due to the issues mentioned above.
Revised: Instead, in this work we rely on RNNs to learn implicit representations of the sufﬁcient statistics over the distribution of the Markov state given the action-observation histories, noting that they are unlikely to recover exact beliefs due to the issues mentioned above.

Original: 4. E XPLORATION AND B ELIEFS
Revised: 4. E XPLORATION AND B ELIEFS

Original: Next we illustrate the impact of exploration on the beliefs, which we will do in the explicit (exact) case, since it serves as an upper bound on the accuracy of the implicit beliefs.
Revised: Next we illustrate the impact of exploration on the beliefs, which we will do in the explicit (exact) case, since it serves as an upper bound on the accuracy of the implicit beliefs.

Original: Since we are looking at fully-cooperative settings we assume that the optimal policy of the agent is deterministic and any randomness is due to exploration.
Revised: Since we are looking at fully-cooperative settings we assume that the optimal policy of the agent is deterministic and any randomness is due to exploration.

Original: Given that we are focused on value based methods we furthermore assume an (cid:15) -greedy exploration scheme, noting that the same analysis can be extended to other methods.
Revised: Given that we are focused on value based methods we furthermore assume an (cid:15) -greedy exploration scheme, noting that the same analysis can be extended to other methods.

Original: Under this exploration scheme π a (cid:48) ( u a (cid:48) t | O ( a (cid:48) , τ t )) becomes:
Revised: Under this exploration scheme π a (cid:48) ( u a (cid:48) t | O ( a (cid:48) , τ t )) becomes:

Original: [Equation]
Revised: [Equation]

Original:  where we have used u ∗ ( τ t ) to indicate the greedy action of the agent a (cid:48) , u ∗ ( τ t ) =
Revised: ) where we have used u ∗ ( τ t ) to indicate the greedy action of the agent a (cid:48) , u ∗ ( τ t ) =

Original: arg max u Q a (cid:48)
Revised: arg max u Q a (cid:48)

Original: ( u, O ( a (cid:48) , τ t )) and I is the indicator function.
Revised: ( u, O ( a (cid:48) , τ t )) and I is the indicator function.

Original: While the ﬁrst part corresponds to a ﬁltering operator

Original: [Equation]

Original:  in which the indicator function only attributes ﬁnite probability to those histories that are consistent with the action taken under greedy execution [Equation] the exploration term adds a ﬁxed (history independent) probability [Equation]
Revised: While the ﬁrst part corresponds to a ﬁltering operator, in which the indicator function only attributes ﬁnite probability to those histories that are consistent with the action taken under greedy execution, the exploration term adds a ﬁxed (history independent) probability, which effectively ‘blurs’ the posterior:

Original: which effectively ‘blurs’ the posterior:

Original: [Equation]
Revised: [Equation]

Original: We ﬁnd that the posterior includes an additional term of the form B ( τ t ) which carries over an unﬁltered density over the trajectories from the prior.
Revised: We ﬁnd that the posterior includes an additional term of the form B ( τ t ) which carries over an unﬁltered density over the trajectories from the prior.

Original: We further conﬁrm that in the limit of (cid:15) = 1 , the posterior collapses to the prior, P ( τ t | τ at , u a (cid:48) t ) =
Revised: We further conﬁrm that in the limit of (cid:15) = 1 , the posterior collapses to the prior, P ( τ t | τ at , u a (cid:48) t ) =

Original: B ( τ t ) .
Revised: B ( τ t ) .

Original: This can be particularly worrisome in the context of our training setup, whereby different agents run different, and potentially high, (cid:15) throughout the course of training.
Revised: This can be particularly worrisome in the context of our training setup, whereby different agents run different, and potentially high, (cid:15) throughout the course of training.

Original: It fundamentally makes the beliefs obtained less informative.
Revised: It fundamentally makes the beliefs obtained less informative.

Original: While not making the above argument explicitly, the Bayesian Action Decoder (BAD)
Revised: While not making the above argument explicitly, the Bayesian Action Decoder (BAD)

Original: (Foerster et al., 2019), resolves this issue by shifting exploration to the level of deterministic partial policies, rather than action-level, and tracking an approximate Bayesian belief.
Revised: (Foerster et al., 2019), resolves this issue by shifting exploration to the level of deterministic partial policies, rather than action-level, and tracking an approximate Bayesian belief.

Original: As outlined in Section 1 this comes at a huge cost in the complexity of the method, the computation requirements and in the loss of generality of the method.
Revised: As outlined in Section 1 this comes at a huge cost in the complexity of the method, the computation requirements and in the loss of generality of the method.

Original: 4. S IMPLIFIED A CTION D ECODING
Revised: 4. S IMPLIFIED A CTION D ECODING

Original: In this paper we take a drastically simpler and different approach towards the issue.
Revised: In this paper we take a drastically simpler and different approach towards the issue.

Original: We note that the ‘blurring’, which makes decoding of an action challenging, is entirely due to the (cid:15) -greedy exploration term.
Revised: We note that the ‘blurring’, which makes decoding of an action challenging, is entirely due to the (cid:15) -greedy exploration term.

Original: Furthermore, in order for another agent to do an implicit Bayesian update over an action taken, it is not required that this action is executed by the environment.
Revised: Furthermore, in order for another agent to do an implicit Bayesian update over an action taken, it is not required that this action is executed by the environment.

Original: Indeed, if we assume that other agents can observe the greedy action, u ∗ , at every time step and condition their belief updated on this, the terms depending on (cid:15) disappear from the Bayesian update: 
Revised: Indeed, if we assume that other agents can observe the greedy action, u ∗ , at every time step and condition their belief update on this, the terms depending on (cid:15) disappear from the Bayesian update: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Therefore, to have our cake and eat it, in the Simpliﬁed Action Decoder (SAD) the acting agent is allowed to ‘take’ two actions at any given time step.
Revised: Therefore, to have our cake and eat it, in the Simpliﬁed Action Decoder (SAD) the acting agent is allowed to ‘take’ two actions at any given time step during training.

Original: The ﬁrst action, u a , is the standard environment action, which gets executed as usual and is observed by all agents through the observation function at the next time step, as mentioned in Section 3.
Revised: The ﬁrst action, u a , is the standard environment action, which gets executed as usual and is observed by all agents through the observation function at the next time step, as mentioned in Section 3.

Original: The second action, u ∗ , is the greedy action of the active agent.
Revised: The second action, u ∗ , is the greedy action of the active agent.

Original: This action does not get executed by the environment but instead is presented as an additional input to the other agents at the next time step during training, taking advantage of the centralized training regime during which information can be exchanged freely.
Revised: This action does not get executed by the environment but instead is presented as an additional input to the other agents at the next time step, taking advantage of the centralized training regime during which information can be exchanged freely.

Original: Clearly we are not allowed to pass around extra information during decentralized control , but luckily this is not needed.
Revised: Clearly we are not allowed to pass around extra information during decentralized control , but luckily this is not needed.

Original: Since we set (cid:15) to 0 at test time we can simply use the, now greedy, environment action obtained from the observation function as our greedy-action input.
Revised: Since we set (cid:15) to 0 at test time we can simply use the, now greedy, environment action obtained from the observation function as our greedy-action input.

Original: While this is most straight forward in settings where the last action is observed by other agents directly, in principle SAD can also be extended to settings where it is indirectly observed by all agents through the environment dynamics.
Revised: While this is most straight forward in settings where the last action is observed by other agents directly, in principle SAD can also be extended to settings where it is indirectly observed by all agents through the environment dynamics.

Original: In these cases we can replace the greedy-action side-channel with a learned inverse model that recovers the action from the observation history during execution.
Revised: In these cases we can replace the greedy-action side-channel with a learned inverse model that recovers the action from the observation history during execution.

Original: Furthermore, to encourage the agent to meaningfully decode the information contained in the greedy action, we can also add an auxiliary task to the training process.
Revised: Furthermore, to encourage the agent to meaningfully decode the information contained in the greedy action, we can optionally add an auxiliary task to the training process, such as predicting unobserved information from their observation history .

Original: While this idea is compatible with any deep RL algorithm with minimal modiﬁcations, we use a recurrent version of DQN with distributed training, dueling networks and prioritized replay.
Revised: While this idea is compatible with any deep RL algorithm with minimal modiﬁcations, we use a recurrent version of DQN with distributed training, dueling networks and prioritized replay.

Original: We also learn a joint Q-function using VDN in order to address the challenges of multi-agent off-policy learning, please see Section 3 for details on all of these standard methods.
Revised: We also learn a joint Q-function using VDN in order to address the challenges of multi-agent off-policy learning, please see Section 3 for details on all of these standard methods.

Original:  E XPERIMENTS 5. M ATRIX G AME
Revised:  E XPERIMENTS 5. M ATRIX G AME

Original: [Figure]

Original: We ﬁrst verify the effectiveness of SAD in the two step, two player matrix game from Foerster et al.
Revised: We ﬁrst verify the effectiveness of SAD in the two step, two player matrix game from Foerster et al.

Original: (2019), which replicates the communication through action challenge of Hanabi in a highly simpliﬁed setting.
Revised: (2019), which replicates the communication through action challenge of Hanabi in a highly simpliﬁed setting.

Original: In this fully cooperative game each player obtains a privately observed ‘card’, which is drawn iid from two options (1,2).
Revised: In this fully cooperative game each player obtains a privately observed ‘card’, which is drawn iid from two options (1,2).

Original: After observing her card, the ﬁrst player takes one of three possible discrete actions (1, 2, 3).
Revised: After observing her card, the ﬁrst player takes one of three possible discrete actions (1, 2, 3).

Original: Crucially, the second player observes both her own private card and the team mate’s action before acting herself, which establishes the opportunity to communicate.
Revised: Crucially, the second player observes both her own private card and the team mate’s action before acting herself, which establishes the opportunity to communicate.

Original: The payout is a function of both the two private cards and the two actions taken by both agents, as shown in Figure 1.
Revised: The payout is a function of both the two private cards and the two actions taken by both agents, as shown in Figure 1.

Revised: Player 2 (acts second)

Original: Importantly, there are some obvious strategies that do not require any communication.
Revised: P l aye r 1 ( ac t s f i r s t) Card Card C a r d

Revised: C a r d

Revised: Player 2 action

Revised: P l aye r 1 ac t i on “Tiny Hanabi”

Revised: [Figure]

Revised: A

Revised: B

Revised: C

Revised: A

Revised: B

Revised: C

Revised: Importantly, there are some obvious strategies that do not require any communication.

Original: For example, if both player learn to play the 2nd action, the payout is always 8 points, independent of the cards dealt.
Revised: For example, if both player learn to play the 2nd action, the payout is always 8 points, independent of the cards dealt.

Original: However, if the players do learn to communicate it is possible to achieve 10 points for every pair of cards dealt.
Revised: However, if the players do learn to communicate it is possible to achieve 10 points for every pair of cards dealt.

Original: 5. H ANABI
Revised: 5. H ANABI

Original: Hanabi is a fully cooperative card game in which all players work together to complete piles of cards referred to as ﬁreworks .
Revised: Hanabi is a fully cooperative card game in which all players work together to complete piles of cards referred to as ﬁreworks .

Original: Each card has a rank, 1 to 5 , and a color, G / B / W / Y / R .
Revised: Each card has a rank, 1 to 5 , and a color, G / B / W / Y / R .

Original: Each ﬁrework (one per color) starts with a 1 and is ﬁnished once the 5 has been added.
Revised: Each ﬁrework (one per color) starts with aand is ﬁnished once the 5 has been added.

Original: There are three 1 s, one 5 and two of all other ranks for each of the colors, adding up to a total of 50 cards in the deck.
Revised: There are three 1 s, one 5 and two of all other ranks for each of the colors, adding up to a total of 50 cards in the deck.

Original: The twist in
Revised: The twist in

Original: Hanabi is that while players can observe the cards held by their team mates, they cannot observe their own cards and thus need to exchange information with each other in order to understand what cards can be played.
Revised: Hanabi is that while players can observe the cards held by their team mates, they cannot observe their own cards and thus need to exchange information with each other in order to understand what cards can be played.

Original: There are two main means for doing so: First of all, players can take grounded hint actions , in which they reveal the subset of a team mate’s hand that matches a speciﬁc rank or color.
Revised: There are two main means for doing so: First of all, players can take grounded hint actions , in which they reveal the subset of a team mate’s hand that matches a speciﬁc rank or color.

Original: An example hint is “Your third and ﬁfth card are 1 s”.
Revised: An example hint is “Your third and ﬁfth card are 1 s”.

Original: These hint actions cost scarce information tokens , which can be replenished by discarding a card, an action that both removes the card from the game and makes it visible to all players.
Revised: These hint actions cost scarce information tokens , which can be replenished by discarding a card, an action that both removes the card from the game and makes it visible to all players.

Original: Finally players can also choose to play a card.
Revised: Finally players can also choose to play a card.

Original: If this card is the next card for the ﬁrework of the corresponding color, it is added to the ﬁrework and the team scores one point.
Revised: If this card is the next card for the ﬁrework of the corresponding color, it is added to the ﬁrework and the team scores one point.

Original: Otherwise the card is removed from the game, the identity is made public, and the team loses one of the 3 life tokens.
Revised: Otherwise the card is removed from the game, the identity is made public, and the team loses one of the 3 life tokens.

Original: If the team runs out of life tokens before the end of the game, all points collected so far are lost and the game ﬁnishes immediately.
Revised: If the team runs out of life tokens before the end of the game, all points collected so far are lost and the game ﬁnishes immediately.

Original: These rules result in a maximum score of 5 × 5 = 25 points in any game, which corresponds to all ﬁve ﬁreworks being completed with ﬁve cards per ﬁrework.
Revised: These rules result in a maximum score of 5 × 5 = 25 points in any game, which corresponds to all ﬁve ﬁreworks being completed with ﬁve cards per ﬁrework.

Original: To ensure reproducibility and comparability of our results we use the Hanabi Learning Environment (HLE)
Revised: To ensure reproducibility and comparability of our results we use the Hanabi Learning Environment (HLE)

Original: (Bard et al., 2019) for all experimentation.
Revised: (Bard et al., 2019) for all experimentation.

Original: For further details regarding Hanabi and the self-play part of the Hanabi challenge please see Bard et al.
Revised: For further details regarding Hanabi and the self-play part of the Hanabi challenge please see Bard et al.

Original: 5. A RCHITECTURE AND C OMPUTATION R EQUIREMENTS
Revised: 5. A RCHITECTURE AND C OMPUTATION R EQUIREMENTS

Original: We borrow some ideas and insights from prior distributed Q-learning methods while bring extensions to MARL as well as innovations to improve throughput and efﬁciency.
Revised: We borrow some ideas and insights from prior distributed Q-learning methods while bring extensions to MARL as well as innovations to improve throughput and efﬁciency.

Original: Following Horgan et al.
Revised: Following Horgan et al.

Original: (2018) and Kapturowski et al.
Revised: (2018) and Kapturowski et al.

Original: (2019), we use a distributed prioritized replay buffer shared by N asynchronous actors and a centralized trainer that samples mini-batches from the replay buffer to update the model.
Revised: (2019), we use a distributed prioritized replay buffer shared by N asynchronous actors and a centralized trainer that samples mini-batches from the replay buffer to update the model.

Original: In each actor thread, we run K environment sequentially and batch their observations together.
Revised: In each actor thread, we run K environments sequentially and batch their observations together.

Original: The observation batch is then fed into an actor that utilize a GPU to compute a batch of actions.
Revised: The observation batch is then fed into an actor that utilize a GPU to compute a batch of actions.

Original: All asynchronous actors share one GPU and the trainer uses another GPU for gradient computation and model updates.
Revised: All asynchronous actors share one GPU and the trainer uses another GPU for gradient computation and model updates.

Original: This is different from prior works which run single actor and single environment in each thread on a CPU.
Revised: This is different from prior works which run single actor and single environment in each thread on a CPU.

Original: Our method enables us to run a very large number of simulations with moderate computation resources.
Revised: Our method enables us to run a very large number of simulations with moderate computation resources.

Original: In all Hanabi experiments, we run N =actor threads with K = 80 environments in each thread on single machine with 40 CPU cores and  GPUs.
Revised: In all Hanabi experiments, we run N =actor threads with K = 80 environments in each thread on single machine with 40 CPU cores and 2 GPUs.

Original: Without this architectural improvement, it may require at least a few hundred CPU cores to run 6400 Hanabi environments with neural network agents and simulations have to be distributed across multiple machines,which will greatly hinder the reproducibility and accessibility of such research.
Revised: Without this architectural improvement, it may require at least a few hundred CPU cores to run 6400 Hanabi environments, in which case neural network agents and simulations have to be distributed across multiple machines, greatly reducing the reproducibility and accessibility of such research.

Original: Please refer to Appendix A for implementation details and hyper-parameters.
Revised: Please refer to Appendix A for implementation details and hyper-parameters.

Original:  R ESULTS
Revised:  R ESULTS

Original: [Figure]
Revised: [Figure]

Original: As we can see in Figure 2, even in our simple matrix game the greedy action input makes a drastic difference.
Revised: As we can see in Figure 2, even in our simple matrix game the greedy action input makes a drastic difference.

Original: With an average reward of around 9.
Revised: With an average reward of around 9.

Original: points, tabular IQL does well in this task, matching the BAD results from Foerster et al.
Revised: points, tabular IQL does well in this task, matching the BAD results from Foerster et al.

Original: However, just by adding the greedy action as an additional input, we obtain an average performance of 9 . 97 ± 0 .
Revised: However, just by adding the greedy action as an additional input, we obtain an average performance of 9 . 97 ± 0 .

Original: Results are averaged over 100 seeds, and shading is s.e.m.
Revised: Results are averaged over 100 seeds, and shading is s.e.m.

Original: The code is available here: 
Revised: The code is available here: www.

Original: www.bit.ly/2mBJLyk .
Revised: bit ly 2mBJLyk

Original: [Table]

Original: [Table]

Original: 6. H ANABI
Revised: 6. H ANABI

Original: As shown in Table [Table] our ﬁndings from the matrix game are for the most part conﬁrmed on the challenging Hanabi benchmark.
Revised: As shown in Table 1, our ﬁndings from the matrix game are for the most part conﬁrmed on the challenging Hanabi benchmark.

Original: To illustrate the contributions of the different components, we compare average scores and win rates across 13 independent training runs of SAD and three different options: IQL is simply the recurrent DQN agent with parameter sharing,
Revised: To illustrate the contributions of the different components, we compare average scores and win rates across 13 independent training runs of SAD and three different options: IQL is simply the recurrent DQN agent with parameter sharing,

Original: VDN is the same agent but also learns a joint Q-function and ﬁnally SAD & AuxTask is the SAD agent with the auxiliary task.
Revised: VDN is the same agent but also learns a joint Q-function and ﬁnally SAD & AuxTask is the SAD agent with the auxiliary task.

Original: While we ﬁnd that SAD signiﬁcantly outperforms our baselines (IQL and VDN) for 2, 4 and 5 players in terms of average score and/or win rate, there is no signiﬁcant difference for 3 players, where VDN matches the performance of SAD.
Revised: While we ﬁnd that SAD signiﬁcantly outperforms our baselines (IQL and VDN) for 2, 4 and 5 players in terms of average score and/or win rate, there is no signiﬁcant difference for 3 players, where VDN matches the performance of SAD.

Original: Interestingly, the auxiliary task only signiﬁcantly helps the 2-player performance, where it substantially boosts the average score and win rate.
Revised: Interestingly, the auxiliary task only signiﬁcantly helps the 2-player performance, where it substantially boosts the average score and win rate.

Original: In contrast, it drastically hurts performance for 3players, which opens an interesting avenue for future work.
Revised: In contrast, it drastically hurts performance for 3players, which opens an interesting avenue for future work.

Original: For completeness we have included training curves showing average scores and s.e.m.
Revised: For completeness we have included training curves showing average scores and s.e.m.

Original: across all training runs for all numbers of players for our methods and ablations in Appendix B. We ﬁnd that for 5 players the auxiliary task drastically reduces the variance of SAD and intermittently leads to higher performance during training but ultimately results in lower ﬁnal performance.
Revised: across all training runs for all numbers of players for our methods and ablations in Appendix B. We ﬁnd that for 5 players the auxiliary task drastically reduces the variance of SAD and intermittently leads to higher performance during training but ultimately results in lower ﬁnal performance.

Original: We can also clearly see that despite 72 hours of training and billions of samples consumed, the performance has not plateaued for 3-5 players, pointing to an obvious avenue for further improvements.
Revised: We can also clearly see that despite 72 hours of training and billions of samples consumed, the performance has not plateaued for 3-5 players, pointing to an obvious avenue for further improvements.

Original: The original numbers in the Hanabi challenge and BAD used population based training (Jaderberg et al., 2018), effectively reporting maximum performance across a large number of different runs.
Revised: The original numbers in the Hanabi challenge and BAD used population based training (Jaderberg et al., 2018), effectively reporting maximum performance across a large number of different runs.

Original: Therefore, for reproducibility purposes, we report evaluations of the best model from our various training runs for each method in Table 2.
Revised: Therefore, for reproducibility purposes, we report evaluations of the best model from our various training runs for each method in Table 2.

Original: As shown, under this reporting we establish a new SOTA for learning methods on the self-play part of the Hanabi challenge for 2-5 players, with the most drastic improvements being achieved for 3-5 players.
Revised: As shown, under this reporting we establish a new SOTA for learning methods on the self-play part of the Hanabi challenge for 2-5 players, with the most drastic improvements being achieved for 3-5 players.

Original: In particular, we beat both the ACHA agent from Bard et al.
Revised: In particular, we beat both the ACHA agent from Bard et al.

Original: and the BAD agent on average score, even though both of them used population based training and require more compute.
Revised: and the BAD agent on average score, even though both of them used population based training and require more compute.

Original: We note that while we follow the counting convention proposed by the challenge paper,
Revised: We note that while we follow the counting convention proposed by the challenge paper,

Original: BAD was optimized for a different counting scheme, in which agents keep their scores when they run out of lives.
Revised: BAD was optimized for a different counting scheme, in which agents keep their scores when they run out of lives.

Original: This may explain the higher win rate (58.6%) of BAD combined with a relatively low mean score, which is exceeded even by our baseline methods.
Revised: This may explain the higher win rate (58.6%) of BAD combined with a relatively low mean score, which is exceeded even by our baseline methods.

Original: Once again, only the performance for 2-player is signiﬁcantly improved by the auxiliary task and the 3-player setting is an outlier in the sense that SAD does not improve the best performance compared to VDN.
Revised: Once again, only the performance for 2-player is signiﬁcantly improved by the auxiliary task and the 3-player setting is an outlier in the sense that SAD does not improve the best performance compared to VDN.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: In this paper we presented the Simpliﬁed Action Decoder (SAD), a novel deep multi-agent RL algorithm that allows agents to learn communication protocols in settings where no cheap-talk[Table] channel is available.
Revised: In this paper we presented the Simpliﬁed Action Decoder (SAD), a novel deep multi-agent RL algorithm that allows agents to learn communication protocols in settings where no cheap-talk channel is available.

Original: On the challenging benchmark Hanabi our work substantially improves the
Revised: On the challenging benchmark Hanabi our work substantially improves the

Original: SOTA for an RL method for all numbers of players.
Revised: SOTA for an RL method for all numbers of players.

Original: For two players SAD establishes a new high-score  across any method.
Revised: For two players SAD establishes a new high-score [Table] [Table] across any method.

Original: Furthermore we accomplish all of this with a method that is both simpler and requires less compute than previous advances.
Revised: Furthermore we accomplish all of this with a method that is both simpler and requires less compute than previous advances.

Original: While these are encouraging steps, there is clearly more work to do.
Revised: While these are encouraging steps, there is clearly more work to do.

Original: In particular, there remains a large performance gap between the numbers achieved by SAD and the known performance of hat-coding strategies (Wu, 2018) for 3-5 players.
Revised: In particular, there remains a large performance gap between the numbers achieved by SAD and the known performance of hat-coding strategies (Wu, 2018) for 3-5 players.

Original: One possible reason is that SAD does not undertake any explicit exploration in the space of possible conventions.
Revised: One possible reason is that SAD does not undertake any explicit exploration in the space of possible conventions.

Original: Another promising route for future work is to integrate search with RL, since this has produced SOTA results in a number of different domains including Poker, Go and backgammon.
Revised: Another promising route for future work is to integrate search with RL, since this has produced SOTA results in a number of different domains including Poker, Go and backgammon.

Original: A N ETWORK A RCHITECTURE AND H YPER -P AMAMETERS FOR H ANABI
Revised: A N ETWORK A RCHITECTURE AND H YPER -P AMAMETERS FOR H ANABI

Original: Our Hanabi agent uses dueling network architecture (Wang et al., 2015).
Revised: Our Hanabi agent uses dueling network architecture (Wang et al., 2015).

Original: The main body of the network consists of 1 fully connected layer of 512 units and 2 LSTM (Hochreiter & Schmidhuber, 1997) layers of 512 units, followed by two output heads for value and advantages respectively.
Revised: The main body of the network consists of 1 fully connected layer of 512 units and 2 LSTM (Hochreiter & Schmidhuber, 1997) layers of 512 units, followed by two output heads for value and advantages respectively.

Original: The same network conﬁguration is used across all Hanabi experiments.
Revised: The same network conﬁguration is used across all Hanabi experiments.

Original: We take the default featurization of HLE and replace the card knowledge section with the V0-Belief proposed by Foerster et al.
Revised: We take the default featurization of HLE and replace the card knowledge section with the V0-Belief proposed by Foerster et al.

Original: The maximum length of an episode is capped at 80 steps and the entire episode is stored in the replay buffer as one training sample.
Revised: The maximum length of an episode is capped at 80 steps and the entire episode is stored in the replay buffer as one training sample.

Original: This avoids the “slate hidden states” problem as described in Kapturowski et al.
Revised: This avoids the “slate hidden states” problem as described in Kapturowski et al.

Original: (2019) so that we can simply initialize the hidden states of LSTM as zero during training.
Revised: (2019) because we can simply initialize the hidden states of LSTM as zero during training.

Original: For exploration and experience prioritization, we follow the simple strategy as in Horgan et al.
Revised: For exploration and experience prioritization, we follow the simple strategy as in Horgan et al.

Original: (2018) and Kapturowski et al.
Revised: (2018) and Kapturowski et al.

Original: Each actor executes an (cid:15) i -greedy policy where (cid:15) i = (cid:15) 

Revised: Each actor executes an (cid:15) i -greedy policy where (cid:15) i = (cid:15)

Original: + 1 N − 1 α for i ∈ { 0 , ..., N − 1 } but with a smaller (cid:15) = 0 .
Revised: + 1 N − 1 α for i ∈ { 0 , ..., N − 1 } but with a smaller (cid:15) = 0 .

Original: 1 and α = 7 .
Revised: 1 and α = 7 .

Original: For simplicity, all players of a game use the same epsilon.
Revised: For simplicity, all players of a game use the same epsilon.

Original: The per time-step priority δ t is the TD error and per episode priority is computed following δ e = η max t δ i + (1 − η )
Revised: The per time-step priority δ t is the TD error and per episode priority is computed following δ e = η max t δ i + (1 − η )

Original: ˆ δ where η = 0 .
Revised: ˆ δ where η = 0 .

Original: Priority exponent is set to 0 .
Revised: Priority exponent is set to 0 .

Original: 9 and importance sampling exponent is set to 0 .
Revised: 9 and importance sampling exponent is set to 0 .

Original: We use n -step return (Sutton, 1988) and double Q-learning (van Hasselt et al., 2015) for target computation during training.
Revised: We use n -step return (Sutton, 1988) and double Q-learning (van Hasselt et al., 2015) for target computation during training.

Original: The discount factor γ is set to 0.999.
Revised: The discount factor γ is set to 0.999.

Original: The network is updated using Adam optimizer (Kingma & Ba, 2014) with learning rate  = 6 .
Revised: The network is updated using Adam optimizer (Kingma & Ba, 2014) with learning rate lr = 6 .

Original: 25 × 10 − 5 and (cid:15) = 1 . 5 × 10 − 5 .
Revised: 25 × 10 − 5 and (cid:15) = 1 . 5 × 10 − 5 .

Original: Trainer sends its network weights to all actors every 10 updates and target network is synchronized with online network every 2500 updates.
Revised: Trainer sends its network weights to all actors every 10 updates and target network is synchronized with online network every 2500 updates.

Original: These hyper-parameters are ﬁxed across all experiments.
Revised: These hyper-parameters are ﬁxed across all experiments.

Original: In the baseline, we use Independent Q-Learning where each player estimates the Q value and selects action independently at each time-step.
Revised: In the baseline, we use Independent Q-Learning where each player estimates the Q value and selects action independently at each time-step.

Original: Note that all players need to operate on the observations in order to update their recurrent hidden states while only the current player has non-trivial legal moves and other players can only select ‘pass’.
Revised: Note that all players need to operate on the observations in order to update their recurrent hidden states while only the current player has non-trivial legal moves and other players can only select ‘pass’.

Original: Each player then writes its own version of the episode into the prioritized replay buffer and they are sampled independently during training.
Revised: Each player then writes its own version of the episode into the prioritized replay buffer and they are sampled independently during training.

Original: The prioritized replay buffer contains 2 17 (131072) episodes.
Revised: The prioritized replay buffer contains 2 17 (131072) episodes.

Original: We warm up the replay buffer with 10,000 episodes before training starts.
Revised: We warm up the replay buffer with 10,000 episodes before training starts.

Original: Batch size during training is 128 for games of different numbers of players.
Revised: Batch size during training is 128 for games of different numbers of players.

Original: As mentioned in Section 4, the SAD agent is built on top of joint Q-function where the Q value is the sum of the individual Q value of all players given their own actions.
Revised: As mentioned in Section 4, the SAD agent is built on top of joint Q-function where the Q value is the sum of the individual Q value of all players given their own actions.

Original: One episode produces only one training sample with an extra dimension for the number of players.
Revised: One episode produces only one training sample with an extra dimension for the number of players.

Original: The replay buffer size is reduced to 2 16 for 2-player and 3-player games and 2 15 for 4-player and 5-player games.
Revised: The replay buffer size is reduced to 2 16 for 2-player and 3-player games and 2 15 for 4-player and 5-player games.

Original: The batch sizes for 2-,
Revised: The batch sizes for 2-,

Original: 3-, 4-, 5-players are 64, 43, 32, 26 respectively to account for the fact that each sample contains more data.
Revised: 3-, 4-, 5-players are 64, 43, 32, 26 respectively to account for the fact that each sample contains more data.

Original: Auxiliary task can be added to the agent to help it decode the greedy action more effectively.
Revised: Auxiliary task can be added to the agent to help it decode the greedy action more effectively.

Original: In
Revised: In

Original: Hanabi, the natural choice is the predict the card of player’s own hand.
Revised: Hanabi, the natural choice is the predict the card of player’s own hand.

Original: In our experiments, the auxiliary task is to predict the status of a card, which can be playable, discardable, or unknown.
Revised: In our experiments, the auxiliary task is to predict the status of a card, which can be playable, discardable, or unknown.

Original: The loss is the average cross entropy loss per card and is simply added to the TD-error of reinforcement learning during training.
Revised: The loss is the average cross entropy loss per card and is simply added to the TD-error of reinforcement learning during training.

Original: B L EARNING C URVES FOR H ANABI
Revised: B L EARNING C URVES FOR H ANABI

Original: Figure 3 shows learning curves of different algorithms averaged over 13 seeds per algorithm per player setting.
Revised: Figure 3 shows learning curves of different algorithms averaged over 13 seeds per algorithm per player setting.

Original: Shading is error of the mean.
Revised: Shading is error of the mean.

Original: [Figure]
Revised: [Figure]