  Deep reinforcement learning methods have shown great performance on many
challenging cooperative multi-agent tasks. Two main promising research
directions are multi-agent value function decomposition and multi-agent policy
gradients. In this paper, we propose a new decomposed multi-agent soft
actor-critic (mSAC) method, which incorporates the idea of the multi-agent
value function decomposition and soft policy iteration framework effectively
and is a combination of novel and existing techniques, including decomposed Q
value network architecture, decentralized probabilistic policy, and
counterfactual advantage function (optional). Theoretically, mSAC supports
efficient off-policy learning and addresses credit assignment problem partially
in both discrete and continuous action spaces. Tested on StarCraft II
micromanagement cooperative multiagent benchmark, we empirically investigate
the performance of mSAC against its variants and analyze the effects of the
different components. Experimental results demonstrate that mSAC significantly
outperforms policy-based approach COMA, and achieves competitive results with
SOTA value-based approach Qmix on most tasks in terms of asymptotic perfomance
metric. In addition, mSAC achieves pretty good results on large action space
tasks, such as 2c_vs_64zg and MMM2.
