Paper:
Social dilemmas are situations where individuals face a temptation to increase their payoffs at a cost to total welfare. Building artificially intelligent agents that achieve good outcomes in these situations is important because many real world interactions include a tension between selfish interests and the welfare of others. We show how to modify modern reinforcement learning methods to construct agents that act in ways that are simple to understand, nice (begin by cooperating), provokable (try to avoid being exploited), and forgiving (try to return to mutual cooperation). We show both theoretically and experimentally that such agents can maintain cooperation in Markov social dilemmas. Our construction does not require training methods beyond a modification of self-play, thus if an environment is such that good strategies can be constructed in the zero-sum case (eg. Atari) then we can construct agents that solve social dilemmas in this environment.

Review:
This paper addresses multiagent learning problems in which there is a social dilemma: settings where there are no 'cooperative polices' that form an equilibrium. The paper proposes a way of dealing with these problems via amTFT, a variation of the well-known tit-for-that strategy, and presents some empirical results.
My main problem with this paper is clarity and I am afraid that not everything might be technically correct. Let me just list my main concerns in the below.
The definition of social dilemma, is unclear:
"A social dilemma is a game where there are no cooperative policies which form equilibria. In other
words, if one player commits to play a cooperative policy at every state, there is a way for their
partner to exploit them and earn higher rewards at their expense."
does this mean to say "there are no cooperative *Markov* policies" ? It seems to me that the paper precisely intents to show that by resorting to history-dependent policies (such as both using amTFT), there is a cooperative equilibrium. 
I don't understand:
"Note that in a social dilemma there may be policies which achieve the payoffs of cooperative policies because they cooperate on the trajectory of play and prevent exploitation by threatening non-cooperation on states which are never reached by the trajectory. If such policies exist, we call the social dilemma solvable."
is this now talking about non-Markov policies? If not, there seems to be a contradiction?
The work focuses on TFT-like policies, motivated by 
"if one can commit to them, create incentives for a partner to behave cooperatively"
however it seems that, as made clear below definition 4, we can only create such incentives for sufficiently powerful agents, that remember and learn from their failures to cooperate in the past?
Why is the method called "approximate Markov"? As soon as one introduces history dependence, the Markov property stops to hold?
On page 4, I have problems following the text due to inconsistent use of notation: subscripts and superscripts seem random, it is not clear which symbols denote strategy profiles (rather than individual strategies), there seems mix-ups between 'i' and '1' / '2', there is sudden use of \hat{}, and other undefined symbols (Q_CC?).
For all practical purposes, it seems that the made assumptions imply uniqueness of the cooperative joint strategy. I fully appreciate that the coordination question is difficult and important, so if the proposed method is not compatible with dealing with that important question, that strikes me as a large drawback.
I have problems understanding how it is possible to guarantee "If they start in a D phase, they eventually return to a C phase." without making more assumptions on the domain. The clear example being the typical 'heaven or hell' type of problems: what if after one defect, we are trapped in the 'hell' state where no cooperation is even possible? 
"If policies converge with this training then πˆ is a Markov equilibrium (up to function approximation)." There are two problems here:
1) A problem is that very typically things will not converge... E.g., 
Wunder, Michael, Michael L. Littman, and Monica Babes. "Classes of multiagent q-learning dynamics with epsilon-greedy exploration." Proceedings of the 27th International Conference on Machine Learning (ICML-10). 2010.
2) "Up to function approximation" could be arbitrary large?
Another significant problem seems to be with this statement:
"while in the cooperative reward schedule the standard RL convergence guarantees apply. The latter is because cooperative training is equivalent to one super-agent controlling both players and trying to optimize for a single scalar reward." The training of individual learners is quite different from "joint action learners" [Claus & Boutilier 98], and this in turn is different from a 'super-agent' which would also control the exploration. In absence of the super-agent, I believe that the only guarantee is that one will, in the limit, converge to a Nash equilibrum, which might be arbitrary far from the optimal joint policy. And this only holds for the tabular case. See the discussion in 
A concise introduction to multiagent systems and distributed artificial intelligence. N Vlassis. Synthesis Lectures on Artificial Intelligence and Machine Learning 1 (1), 1-71
Also, the approach used in the experiments "Cooperative (self play with both agents receiving sum of rewards) training for both games", would be insufficient for many settings where a cooperative joint policy would be asymmetric.
The entire approach hinges on using rollouts (the commented lines in Algo. 1). However, it is completely not clear to me how this works. The one paragraph is insufficient to get across these crucial parts of the proposed approach.
It is not clear why the tables in Figure 1 are not symmetric; this strikes me as extremely problematic. It is not clear what the colors encode either.
It also seems that "grim" is better against all, except against amTFT, why should we not use that? In general, the explanation of this closely related paper by De Cote & Littman (which was published at UAI'08), is insufficient. It is not quite clear to me what the proposed approach offers over the previous method.