Abstract. Multi-agent reinforcement learning (RL) has important implications for the future of human-agent teaming. We show that improved performance with multi-agent RL is not a guarantee of the collaborative behavior thought to be important for solving multi-agent tasks. To address this, we present a novel approach for quantitatively assessing collaboration in continuous spatial tasks with multi-agent RL. Such a metric is useful for measuring collaboration between computational agents and may serve as a training signal for collaboration in future RL paradigms involving humans.  Reinforcement learning (RL) is an attractive option for providing adaptive behavior in computational agents because of its theoretical generalizability to complex problem spaces [1, 2] . In particular, deep RL recently produced striking results [3] . Extending RL to the multi-agent domain has received an increasing amount of attention as the need for human-agent teams has increased, especially with regards to training agents to behave collaboratively [1, [4] [5] [6] [7] [8] . Unfortunately, the nature of multi-agent RL makes guaranteeing collaboration between agents impossible except in limited provable cases, even when these methods yield better task performance [1, 7] . Thus far, evaluating collaboration in multi-agent learning has been accomplished by measuring performance in tasks where coordination is required. While this may be satisfactory for discretized tasks where cooperative policies are provably optimal [1, 6-8] it is not clear that this methodology generalizes well to more complex and continuous tasks (such as those presented in [4, 5] and here). The question at hand is how to assess performance enhancing coordination (or collaboration) between agents in multi-agent RL tasks. Here, we present a method borrowed from the field of ecology, called convergence cross mapping (CCM), and show how it can be used to measure collaboration between agents during a predatorprey pursuit task. Additionally, we show a striking result: a state-of-the-art multi-agent reinforcement learning algorithm does not exhibit coordinated behavior between agents
REF propose causal influence as a way to measure coordination between agents, specifically using Convergence Cross Mapping (CCM) to analyze the degree of dependence between two agents' policies.
Measuring collaborative emergent behavior in multi-agent reinforcement learning
ArXiv
ArXiv
Computer Science