Paper:
In recent years we have seen fast progress on a number of benchmark problems in AI, with modern methods achieving near or super human performance in Go, Poker and Dota. One common aspect of all of these challenges is that they are by design adversarial or, technically speaking, zero-sum. In contrast to these settings, success in the real world commonly requires humans to collaborate and communicate with others, in settings that are, at least partially, cooperative. In the last year, the card game Hanabi has been established as a new benchmark environment for AI to fill this gap. In particular, Hanabi is interesting to humans since it is entirely focused on theory of mind, i.e., the ability to effectively reason over the intentions, beliefs and point of view of other agents when observing their actions. Learning to be informative when observed by others is an interesting challenge for Reinforcement Learning (RL): Fundamentally, RL requires agents to explore in order to discover good policies. However, when done naively, this randomness will inherently make their actions less informative to others during training. We present a new deep multi-agent RL method, the Simplified Action Decoder (SAD), which resolves this contradiction exploiting the centralized training phase. During training SAD allows other agents to not only observe the (exploratory) action chosen, but agents instead also observe the greedy action of their team mates. By combining this simple intuition with best practices for multi-agent learning, SAD establishes a new SOTA for 2-5 players on the self-play part of the Hanabi challenge. Our ablations show the contributions of SAD compared with the best practice components.

Review:
This paper introduces a novel exploitation of the centralized training for decentralized execution regime for Dec-POMDPs with publicly/commonly known actions.  In particular, the paper augments independent value-based reinforcement learning by allowing each agent to announce the action it would have taken, had it acted greedily. This relaxation is consistent with decentralized execution, at which time agents always act greedily and no counterfactual announcement is required. The paper demonstrates the utility of this trick in Hanabi, where it achieves strong performance when combined with distributed Q-learning and an auxiliary task. The paper is largely well-written.
I think that the trick introduced in this paper is a valuable contribution to the community. As the paper discusses, SAD is simpler and easier to implement than BAD, the algorithm with the most comparable ambitions with respect to scalability. Also, unlike BAD, it is model-free. I think that investigating lightweight alternatives to common/public knowledge based approaches, such as SAD, is an important research direction.
That being said, I have some issues with the presentation of the content in the paper. 
1. Section 4 is problematic. The entirety of the section is based on the assumption that agent a’ observes the Markov state. The paper claims that this assumption is made for simplicity: “where for simplicity we have assumed that agent a’ uses a feed-forward policy and observes the Markov state of the environment.” 
    a. First, this assumption is unmet, with few exceptions. Other agents do 
        NOT in general observe the Markov state. This is a very significant 
        part of why Dec-POMDPs are so difficult. Justifying SAD in a setting 
        that so radically departs from its use case is not informative. 
    b. Second, the claim made by the paper (that the assumption is made 
        for simplicity) is misleading. The assumption is made out of 
        necessity. The cost of neglecting public/common knowledge, as SAD 
        does, is that principled Bayesian reasoning is not possible. It is 
        important that the paper acknowledges this to counterbalance its 
        (valid) criticisms that public/common knowledge based approaches 
        are difficult to scale.
2. That the experiments are expensive is understandable and a fully complete ablation study is not expected. But the results of these experiments should be presented clearly. 
    a. In both Table 1 and Table 2, presenting the “best seed” is 
        not appropriate. The paper should report the mean or the median. 
        Given that only three seeds could be afforded, it is understandable 
        that the results would have high variance. 
    b. In Table 2, (if I am understanding the paper correctly) it is claimed 
        that the s.e.m. is less than or equal to 0.01 over a set of three seeds 
        for all of the experiments. But looking at Figure 4, this is clearly not 
        the case, especially in the four and five player settings. The intended 
        meaning should be clarified.
    c. In Table 1, the paper should report the results for the baseline, VDN, 
        VDN with greedy input, and SAD separately, as is done in Table 2. 
        These are the main experimental results and merit space in the main 
        body of the paper. Moreover, there are a number of results that 
        deserve written attention.
        i. It looks the baseline is very competitive with BAD in two player 
           Hanabi. This in itself is a very interesting finding and is worth 
           discussing.
        ii. Looking at Table 2, it appears that VDN with GreedyInput 
            outperforms SAD in three of the four settings and quite 
            significantly in the five player setting. If this is also the case when 
            results are aggregated over the median or mean, it should be 
            discussed.
3. The greedy input approach is specific to Dec-POMDPs in which actions are publicly/commonly available. This should at least be mentioned somewhere.
Overall, I think the ideas and experimental findings in the paper are very interesting. However, as outlined above, there are a number of issues. At a high level, I think that the paper is too concerned with 1) justifying greedy input with Bayesian reasoning and 2) promoting state of the art results. The ideas and experimental findings are more than sufficient for strong contribution without these things.
In its current state, I feel that this work is a rejection. However, its issues are relatively easily amendable:
1. For each algorithm and each setting, separately report median or mean results over the seeds, with uncertainty.
2. Reallocate space to section 6.2 for a scientific discussion of the experimental results.
3. Remove or qualify the arguments made in section 4.
4. Mention that the greedy input approach is specific to Dec-POMDPs with publicly/commonly available actions.
Were these issues addressed, my opinion would change.