  Policy gradient methods have become one of the most popular classes of
algorithms for multi-agent reinforcement learning. A key challenge, however,
that is not addressed by many of these methods is multi-agent credit
assignment: assessing an agent's contribution to the overall performance, which
is crucial for learning good policies. We propose a novel algorithm called
Dr.Reinforce that explicitly tackles this by combining difference rewards with
policy gradients to allow for learning decentralized policies when the reward
function is known. By differencing the reward function directly, Dr.Reinforce
avoids difficulties associated with learning the Q-function as done by
Counterfactual Multiagent Policy Gradients (COMA), a state-of-the-art
difference rewards method. For applications where the reward function is
unknown, we show the effectiveness of a version of Dr.Reinforce that learns an
additional reward network that is used to estimate the difference rewards.
