  In fully cooperative multi-agent reinforcement learning (MARL) settings, the
environments are highly stochastic due to the partial observability of each
agent and the continuously changing policies of the other agents. To address
the above issues, we integrate distributional RL and value function
factorization methods by proposing a Distributional Value Function
Factorization (DFAC) framework to generalize expected value function
factorization methods to their DFAC variants. DFAC extends the individual
utility functions from deterministic variables to random variables, and models
the quantile function of the total return as a quantile mixture. To validate
DFAC, we demonstrate DFAC's ability to factorize a simple two-step matrix game
with stochastic rewards and perform experiments on all Super Hard tasks of
StarCraft Multi-Agent Challenge, showing that DFAC is able to outperform
expected value function factorization baselines.
