Original: M ULTI -A GENT I MITATION L EARNING WITH C OPULAS
Revised: M ULTI -A GENT I MITATION L EARNING WITH C OPULAS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Multi-agent imitation learning aims to train multiple agents to perform tasks from demonstrations by learning a mapping between observations and actions, which is essential for understanding physical, social, and team-play systems.
Revised: Multi-agent imitation learning aims to train multiple agents to perform tasks from demonstrations by learning a mapping between observations and actions, which is essential for understanding physical, social, and team-play systems.

Original: However, most existing works on modeling multi-agent interactions typically assume that agents make independent decisions based on their observations, ignoring the complex dependence among agents.
Revised: However, most existing works on modeling multi-agent interactions typically assume that agents make independent decisions based on their observations, ignoring the complex dependence among agents.

Original: In this paper, we propose to use copula, a powerful statistical tool for capturing dependence among random variables, to explicitly model the correlation and coordination in multi-agent systems.
Revised: In this paper, we propose to use copula, a powerful statistical tool for capturing dependence among random variables, to explicitly model the correlation and coordination in multi-agent systems.

Original: Our proposed model is able to separately learn marginals that capture the local behavioral patterns of each individual agent, as well as a copula function that solely and fully captures the dependence structure among agents.
Revised: Our proposed model is able to separately learn marginals that capture the local behavioral patterns of each individual agent, as well as a copula function that solely and fully captures the dependence structure among agents.

Original: Extensive experiments on synthetic and real-world datasets show that our model outperforms state-of-the-art baselines across various scenarios in the action prediction task, and is able to generate new trajectories close to expert demonstrations.
Revised: Extensive experiments on synthetic and real-world datasets show that our model outperforms state-of-the-art baselines across various scenarios in the action prediction task, and is able to generate new trajectories close to expert demonstrations.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recent years have witnessed great success of reinforcement learning (RL) for single-agent sequential decision making tasks.
Revised: Recent years have witnessed great success of reinforcement learning (RL) for single-agent sequential decision making tasks.

Original: As many real-world applications (e.g., multi-player games (Silver et al., 2017; Brown & Sandholm, 2019) and trafﬁc light control (Chu et al., 2019)) involve the participation of multiple agents, multi-agent reinforcement learning (MARL) has gained more and more attention.
Revised: As many real-world applications (e.g., multi-player games (Silver et al., 2017; Brown & Sandholm, 2019) and trafﬁc light control (Chu et al., 2019)) involve the participation of multiple agents, multi-agent reinforcement learning (MARL) has gained more and more attention.

Original: However, a key limitation of RL and MARL is the difﬁculty of designing suitable reward functions for complex tasks with implicit goals (e.g., dialogue systems)
Revised: However, a key limitation of RL and MARL is the difﬁculty of designing suitable reward functions for complex tasks with implicit goals (e.g., dialogue systems)

Original: (Russell, 1998; Ng et al., 2000; Fu et al., 2017; Song et al., 2018).
Revised: (Russell, 1998; Ng et al., 2000; Fu et al., 2017; Song et al., 2018).

Original: Indeed, hand-tuning reward functions to induce desired behaviors becomes especially challenging in multi-agent systems, since different agents may have completely different goals and state-action representations (Yu et al., 2019).
Revised: Indeed, hand-tuning reward functions to induce desired behaviors becomes especially challenging in multi-agent systems, since different agents may have completely different goals and state-action representations (Yu et al., 2019).

Original: Imitation learning provides an alternative approach to directly programming agents by taking advantage of expert demonstrations on how a task should be solved.
Revised: Imitation learning provides an alternative approach to directly programming agents by taking advantage of expert demonstrations on how a task should be solved.

Original: Although appealing, most prior works on multi-agent imitation learning typically assume agents make independent decisions after observing a state (i.e., mean-ﬁeld factorization of the joint policy)
Revised: Although appealing, most prior works on multi-agent imitation learning typically assume agents make independent decisions after observing a state (i.e., mean-ﬁeld factorization of the joint policy)

Original: (Zhan et al., 2018; Le et al., 2017;
Revised: (Zhan et al., 2018; Le et al., 2017;

Original: Song et al., 2018; Yu et al., 2019), ignoring the potentially complex dependencies that exist among agents.
Revised: Song et al., 2018; Yu et al., 2019), ignoring the potentially complex dependencies that exist among agents.

Original: Recently, Tian et al.
Revised: Recently, Tian et al.

Original: (2019) and Liu et al.
Revised: (2019) and Liu et al.

Original: (2020) proposed to implement correlated policies with opponent modeling, which incurs unnecessary modeling cost and redundancy, while still lacking coordination during execution.
Revised: (2020) proposed to implement correlated policies with opponent modeling, which incurs unnecessary modeling cost and redundancy, while still lacking coordination during execution.

Original: Compared to the single-agent setting, one major and fundamental challenge in multi-agent learning is how to model the dependence among multiple agents in an effective and scalable way.
Revised: Compared to the single-agent setting, one major and fundamental challenge in multi-agent learning is how to model the dependence among multiple agents in an effective and scalable way.

Original: Inspired by probability theory and statistical dependence modeling, in this work, we propose to use copulas (Sklar, 1959b; Nelsen, 2007; Joe, 2014) to model multi-agent behavioral patterns.
Revised: Inspired by probability theory and statistical dependence modeling, in this work, we propose to use copulas (Sklar, 1959b; Nelsen, 2007; Joe, 2014) to model multi-agent behavioral patterns.

Original: Copulas are powerful statistical tools to describe the dependence among random variables, which have been widely used in quantitative ﬁnance for risk measurement and portfolio optimization (Bouy´e et al., 2000).
Revised: Copulas are powerful statistical tools to describe the dependence among random variables, which have been widely used in quantitative ﬁnance for risk measurement and portfolio optimization (Bouy´e et al., 2000).

Original: Using a copulas-based multi-agent policy enables us to separately learn marginals that capture the local behavioral patterns of each individual agent and a copula function that only and fully captures the dependence structure among the agents.
Revised: Using a copulas-based multi-agent policy enables us to separately learn marginals that capture the local behavioral patterns of each individual agent and a copula function that only and fully captures the dependence structure among the agents.

Original: Such a factorization is capable of modeling arbitrarily complex joint policy and leads to interpretable, efﬁcient and scalable multi-agent imitation learning.
Revised: Such a factorization is capable of modeling arbitrarily complex joint policy and leads to interpretable, efﬁcient and scalable multi-agent imitation learning.

Original: As a motivating example (see Figure 1), suppose there are two agents, each with one-dimensional action space.
Revised: As a motivating example (see Figure 1), suppose there are two agents, each with one-dimensional action space.

Original: In Figure 1a, although two joint policies are quite different, they actually share the same copula (dependence structure) and one marginal.
Revised: In Figure 1a, although two joint policies are quite different, they actually share the same copula (dependence structure) and one marginal.

Original: Our proposed copula-based policy is capable
Revised: Our proposed copula-based policy is capable

Original: [Figure]
Revised: [Figure]

Original: +

Original: [Figure]

Original: of capturing such information and more importantly, we may leverage such information to develop efﬁcient algorithms for such transfer learning scenarios.
Revised: of capturing such information and more importantly, we may leverage such information to develop efﬁcient algorithms for such transfer learning scenarios.

Original: For example, when we want to model teamplay in a soccer game and one player is replaced by his/her substitute while the dependence among different roles are basically the same regardless of players, we can immediately obtain a new joint policy by switching in the new player’s marginal while keeping the copula and other marginals unchanged.
Revised: For example, when we want to model teamplay in a soccer game and one player is replaced by his/her substitute while the dependence among different roles are basically the same regardless of players, we can immediately obtain a new joint policy by switching in the new player’s marginal while keeping the copula and other marginals unchanged.

Original: On the other hand, as shown in Figure 1b, two different joint policies may share the same marginals while having different copulas, which implies that the mean-ﬁeld policy in previous works (only modeling marginal policies and making independent decisions) cannot differentiate these two scenarios to achieve coordination correctly.
Revised: On the other hand, as shown in Figure 1b, two different joint policies may share the same marginals while having different copulas, which implies that the mean-ﬁeld policy in previous works (only modeling marginal policies and making independent decisions) cannot differentiate these two scenarios to achieve coordination correctly.

Original: Towards this end, in this paper, we propose a copula-based multi-agent imitation learning algorithm, which is interpretable, efﬁcient and scalable for modeling complex multi-agent interactions.
Revised: Towards this end, in this paper, we propose a copula-based multi-agent imitation learning algorithm, which is interpretable, efﬁcient and scalable for modeling complex multi agent interactions.

Original: Extensive experimental results on synthetic and real-world datasets show that our proposed method outperforms state-of-the-art multi-agent imitation learning methods in various scenarios and generates multi-agent trajectories close to expert demonstrations.
Revised: Extensive experimental results on synthetic and real-world datasets show that our proposed method outperforms state-of-the-art multi-agent imitation learning methods in various scenarios and generates multi-agent trajectories close to expert demonstrations

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: In this work, we consider the problem of multi-agent imitation learning under the framework of
Revised: In this work, we consider the problem of multi-agent imitation learning under the framework of

Original: Markov games (Littman, 1994), which generalize Markov Decision Processes to multi-agent settings, where N agents are interacting with each other.
Revised: Markov games (Littman, 1994), which generalize Markov Decision Processes to multi-agent settings, where N agents are interacting with each other.

Original: Speciﬁcally, in a Markov game, S is the common state space,
Revised: Speciﬁcally, in a Markov game, S is the common state space,

Original: A i is the action space for agent i ∈ { 1 , . . .
Revised: A i is the action space for agent i ∈ { 1 , . . .

Original: , N } , η ∈ P ( S ) is the initial state distribution and P : S
Revised: , N } , η ∈ P ( S ) is the initial state distribution and P : S

Original: ×
Revised: ×

Original: A 1 × . . .
Revised: A 1 × . . .

Original: ×
Revised: ×

Original: A N → P ( S ) is the state transition distribution of the environment that the agents are interacting with.
Revised: A N → P ( S ) is the state transition distribution of the environment that the agents are interacting with.

Original: Here P ( S ) denotes the set of probability distributions over state space S .
Revised: Here P ( S ) denotes the set of probability distributions over state space S .

Original: Suppose at time t , agents observe s [ t ] ∈ S and take actions a [ t ] :
Revised: Suppose at time t , agents observe s [ t ] ∈ S and take actions a [ t ] :

Original: = ( a 1 [ t ] , . .
Revised: = ( a 1 [ t ] , . .

Original: . , a N [ t ]) ∈
Revised: . , a N [ t ]) ∈

Original: A 1 × . . .
Revised: A 1 × . . .

Original: ×
Revised: ×

Original: A N , the agents will observe state s [ t + 1] ∈ S at time t +with probability P ( s [ t + 1] | s [ t ] , a 1 [ t ] , . . .
Revised: A N , the agents will observe state s [ t + 1] ∈ S at time t +with probability P ( s [ t + 1] | s [ t ] , a 1 [ t ] , . . .

Original: , a N [ t ]) .
Revised: , a N [ t ]) .

Original: In this process, the agents select the joint action a [ t ] by sampling from a stochastic joint policy π : S → P ( A 1 × . . .
Revised: In this process, the agents select the joint action a [ t ] by sampling from a stochastic joint policy π : S → P ( A 1 × . . .

Original: ×
Revised: ×

Original: A N ) .
Revised: A N ) .

Original: In the following, we will use subscript − i to denote all agents except for agent i .
Revised: In the following, we will use subscript − i to denote all agents except for agent i .

Original: For example, ( a i , a − i ) represents the actions of all agents; π i ( a i | s ) and π i ( a i | s, a − i ) represent the marginal and conditional policy of agent i induced by the joint policy π ( a | s ) (through marginalization and Bayes’s rule respectively).
Revised: For example, ( a i , a − i ) represents the actions of all agents; π i ( a i | s ) and π i ( a i | s, a − i ) represent the marginal and conditional policy of agent i induced by the joint policy π ( a | s ) (through marginalization and Bayes’s rule respectively).

Original: We consider the following off-line imitation learning problem.
Revised: We consider the following off-line imitation learning problem.

Original: Suppose we have access to a set of demonstrations D = { τ j } Mj =1 provided by some expert policy π E ( a | s ) , where each expert trajectory τ
Revised: Suppose we have access to a set of demonstrations D = { τ j } Mj =1 provided by some expert policy π E ( a | s ) , where each expert trajectory τ

Original: j
Revised: j

Original: = { ( s j [ t ] , a j [ t ]) }
Revised: = { ( s j [ t ] , a j [ t ]) }

Original: Tt =1 is collected by the following sampling process:
Revised: Tt =1 is collected by the following sampling process:

Original: [Equation]
Revised: s 1 ∼ η ( s ) , a [ t ] ∼ π E ( a | s [ t ]) , s [ t + 1] ∼ P ( s | s [ t ] , a [ t ]) for t ∈ { 1 , . .

Original: [Equation]
Revised: , T } .

Original: The goal is to learn a parametric joint policy π θ to approximate the expert policy π E such that we can do downstream inferences (e.g., action prediction and trajectory generation).
Revised: The goal is to learn a parametric joint policy π θ to approximate the expert policy π E such that we can do downstream inferences (e.g., action prediction and trajectory generation).

Original: The learning problem is off-line as we cannot ask for additional interactions with the expert policy or the environment during training, and the reward is also unknown.
Revised: The learning problem is off-line as we cannot ask for additional interactions with the expert policy or the environment during training, and the reward is also unknown.

Original:  M ODELING M ULTI -A GENT I NTERACTIONS WITH C OPULAS 3. M OTIVATION
Revised:  M ODELING M ULTI -A GENT I NTERACTIONS WITH C OPULAS 3. M OTIVATION

Original: Many modeling methods for multi-agent learning tasks employ a simplifying mean-ﬁeld assumption that the agents make independent action choices after observing a state (Albrecht & Stone, 2018;
Revised: Many modeling methods for multi-agent learning tasks employ a simplifying mean-ﬁeld assumption that the agents make independent action choices after observing a state (Albrecht & Stone, 2018;

Original: Song et al., 2018;
Revised: Song et al., 2018;

Original: Yu et al., 2019), which means the joint policy can be factorized as follows:
Revised: Yu et al., 2019), which means the joint policy can be factorized as follows:

Original: [Equation]
Revised: [Equation]

Original: Such a mean-ﬁeld assumption essentially allows for independent construction of each agent’s policy.
Revised: Such a mean-ﬁeld assumption essentially allows for independent construction of each agent’s policy.

Original: For example, multi-agent behavior cloning by maximum likelihood estimation is now equivalent to performing N single-agent behavior cloning tasks:
Revised: For example, multi-agent behavior cloning by maximum likelihood estimation is now equivalent to performing N single-agent behavior cloning tasks:

Original: [Equation]
Revised: [Equation]

Original: where the occupancy measure ρ π : S ×
Revised: where the occupancy measure ρ π : S ×

Original: A 1 × . . .
Revised: A 1 × . . .

Original: ×
Revised: ×

Original: A N → R denotes the state action distribution encountered when navigating the environment using the joint policy π (Syed et al., 2008; Puterman, 2014) and ρ π ,i is the corresponding marginal occupancy measure.
Revised: A N → R denotes the state action distribution encountered when navigating the environment using the joint policy π (Syed et al., 2008; Puterman, 2014) and ρ π ,i is the corresponding marginal occupancy measure.

Original: However, when the expert agents are making correlated action choices (e.g., due to joint plan and communication in a soccer game), such a simplifying modeling choice is not able to capture the rich dependency structure and coordination among agent actions.
Revised: However, when the expert agents are making correlated action choices (e.g., due to joint plan and communication in a soccer game), such a simplifying modeling choice is not able to capture the rich dependency structure and coordination among agent actions.

Original: To address this issue, recent works (Tian et al., 2019; Liu et al., 2020) propose to use a different factorization of the joint policy such that the dependency among N agents can be preserved:
Revised: To address this issue, recent works (Tian et al., 2019; Liu et al., 2020) propose to use a different factorization of the joint policy such that the dependency among N agents can be preserved:

Original: [Equation]
Revised: [Equation]

Original: Although such a factorization is general and captures the dependency among multi-agent interactions, several issues still remain.
Revised: Although such a factorization is general and captures the dependency among multi-agent interactions, several issues still remain.

Original: First, the modeling cost is increased signiﬁcantly, because now we need to learn N different and complicated opponent policies π − i ( a − i | s ) as well as
Revised: First, the modeling cost is increased signiﬁcantly, because now we need to learn N different and complicated opponent policies π − i ( a − i | s ) as well as

Original: N different marginal conditional policies π i ( a i | s, a − i ) , each with a deep neural network.
Revised: N different marginal conditional policies π i ( a i | s, a − i ) , each with a deep neural network.

Original: It should be noted that there are many redundancies in such a modeling choice.
Revised: It should be noted that there are many redundancies in such a modeling choice.

Original: Speciﬁcally, suppose there are N agents and N > 3 , for agent 1 and N , we need to learn opponent policies π − 1 ( a 2 , . .
Revised: Speciﬁcally, suppose there are N agents and N > 3 , for agent 1 and N , we need to learn opponent policies π − 1 ( a 2 , . .

Original: . , a N | s ) and π − N ( a 1 , . .
Revised: . , a N | s ) and π − N ( a 1 , . .

Original: . , a N − 1 | s ) respectively.
Revised: . , a N − 1 | s ) respectively.

Original: These are potentially high dimensional and might require ﬂexible function approximations.
Revised: These are potentially high dimensional and might require ﬂexible function approximations.

Original: However, the dependency structure among agent 2 to agent N − 1 are modeled in both π − 1 and π − N , which incurs unnecessary modeling cost.
Revised: However, the dependency structure among agent 2 to agent N − 1 are modeled in both π − 1 and π − N , which incurs unnecessary modeling cost.

Original: Second, when executing the policy, each agent i makes decisions through its marginal policy π i ( a i | s ) =
Revised: Second, when executing the policy, each agent i makes decisions through its marginal policy π i ( a i | s ) =

Original: E π − i ( a − i | s ) ( a i | s, a − i ) by ﬁrst sampling a − i from its opponent policy π − i then sampling its action a i from π i ( ·| s, a − i ) .
Revised: E π − i ( a − i | s ) ( a i | s, a − i ) by ﬁrst sampling a − i from its opponent policy π − i then sampling its action a i from π i ( ·| s, a − i ) .

Original: Since each agent is performing such decision process independently, coordination among agents are still impossible due to sampling randomness.
Revised: Since each agent is performing such decision process independently, coordination among agents are still impossible due to sampling randomness.

Original: Moreover, a set of independently learned conditional distributions are not necessarily consistent with each other (i.e., induced by the same joint policy) (Yu et al., 2019).
Revised: Moreover, a set of independently learned conditional distributions are not necessarily consistent with each other (i.e., induced by the same joint policy) (Yu et al., 2019).

Original: In this work, to address above challenges, we draw inspiration from probability theory and propose to use copulas, a statistical tool for describing the dependency structure between random variables, to model the complicated multi-agent interactions in a scalable and efﬁcient way.
Revised: In this work, to address above challenges, we draw inspiration from probability theory and propose to use copulas, a statistical tool for describing the dependency structure between random variables, to model the complicated multi-agent interactions in a scalable and efﬁcient way.

Original: 3. C OPULAS
Revised: 3. C OPULAS

Original: When the components of a multivariate random variable x = ( x 1 , . . .
Revised: When the components of a multivariate random variable x = ( x 1 , . . .

Original: , x N ) are jointly independent, the density of x can be written as:
Revised: , x N ) are jointly independent, the density of x can be written as:

Original: [Equation]
Revised: [Equation]

Original: When the components are not independent, this equality does not hold any more as the dependencies among x 1 , . . .
Revised: When the components are not independent, this equality does not hold any more as the dependencies among x 1 , . . .

Original: , x N can not be captured by the marginals p ( x i ) .
Revised: , x N can not be captured by the marginals p ( x i ) .

Original: However, the differences can be corrected by multiplying the right hand side of Equation (4) with a function that only and fully describes the dependency.
Revised: However, the differences can be corrected by multiplying the right hand side of Equation (4) with a function that only and fully describes the dependency.

Original: Such a function is called a copula (Nelsen, 2007), a multivariate distribution function on the unit hyper-cube with uniform marginals.
Revised: Such a function is called a copula (Nelsen, 2007), a multivariate distribution function on the unit hyper-cube with uniform marginals.

Original: Intuitively, let us consider a random variable x i with continuous cumulative distribution function F i .
Revised: Intuitively, let us consider a random variable x i with continuous cumulative distribution function F i .

Original: Applying probability integral transform gives us a random variable u i
Revised: Applying probability integral transform gives us a random variable u i

Original: = F i ( x i ) , which has standard uniform distribution.
Revised: = F i ( x i ) , which has standard uniform distribution.

Original: Thus one can use this property to separate the information in marginals from the dependency structures among x 1 , . . .
Revised: Thus one can use this property to separate the information in marginals from the dependency structures among x 1 , . . .

Original: , x N by ﬁrst projecting each marginal onto one axis of the hyper-cube and then capture the pure dependency with a distribution on the unit hyper-cube.
Revised: , x N by ﬁrst projecting each marginal onto one axis of the hyper-cube and then capture the pure dependency with a distribution on the unit hyper-cube.

Original: Formally, a copula is the joint distribution of random variables u 1 , . . .
Revised: Formally, a copula is the joint distribution of random variables u 1 , . . .

Original: , u N , each of which is marginally uniformly distributed on the interval [0 , 1] .
Revised: , u N , each of which is marginally uniformly distributed on the interval [0 , 1] .

Original: Furthermore, we introduce the following theorem that provides the theoretical foundations of copulas:
Revised: Furthermore, we introduce the following theorem that provides the theoretical foundations of copulas:

Original: Theorem 1 ((Sklar, 1959a)) .
Revised: Theorem 1 ((Sklar, 1959a)) .

Original: Suppose the multivariate random variable ( x 1 , . . .
Revised: Suppose the multivariate random variable ( x 1 , . . .

Original: , x N ) has marginal cumulative distribution functions F 1 , . .
Revised: , x N ) has marginal cumulative distribution functions F 1 , . .

Original: . , F N and joint cumulative distribution function F , then there exists a unique copula C :
Revised: . , F N and joint cumulative distribution function F , then there exists a unique copula C :

Original: [0 , 1] N → [0 , 1] such that:
Revised: [0 , 1] N → [0 , 1] such that:

Original: [Equation]
Revised: [Equation]

Original: When the multivariate distribution has a joint density f and marginal densities f 1 , . .
Revised: When the multivariate distribution has a joint density f and marginal densities f 1 , . .

Original: . , f N , we have:
Revised: . , f N , we have:

Original: [Equation]
Revised: [Equation]

Original: where c is the probability density function of the copula.
Revised: where c is the probability density function of the copula.

Original: The converse is also true.
Revised: The converse is also true.

Original: Given a copula
Revised: Given a copula

Original: C and marginals F i ( x i ) , then C (cid:0) F 1 ( x 1 ) , . . . , F N ( x N ) (cid:1) =
Revised: C and marginals F i ( x i ) , then C (cid:0) F 1 ( x 1 ) , . . . , F N ( x N ) (cid:1) =

Original: F ( x 1 , . . .
Revised: F ( x 1 , . . .

Original: , x N ) is a N -dimensional cumulative distribution function with marginal distributions F i ( x i ) .
Revised: , x N ) is a N -dimensional cumulative distribution function with marginal distributions F i ( x i ) .

Original: Theorem 1 states that every multivariate cumulative distribution function F ( x 1 , . . .
Revised: Theorem 1 states that every multivariate cumulative distribution function F ( x 1 , . . .

Original: , x N ) can be expressed in terms of its marginals F i ( x i ) and a copula C (cid:0) F 1 ( x 1 ) , . .
Revised: , x N ) can be expressed in terms of its marginals F i ( x i ) and a copula C (cid:0) F 1 ( x 1 ) , . .

Original: , F N ( x N ) (cid:1) .
Revised: , F N ( x N ) (cid:1) .

Original: Comparing Eq.
Revised: Comparing Eq.

Original: (4) with Eq.
Revised: (4) with Eq.

Original: (6), we can see that a copula function encoding correlations between random variables can be used to correct the mean-ﬁeld approximation for arbitrarily complex distribution.
Revised: (6), we can see that a copula function encoding correlations between random variables can be used to correct the mean-ﬁeld approximation for arbitrarily complex distribution.

Original: 3. M ULTI -A GENT I MITATION L EARNING WITH C OPULA - BASED P OLICIES
Revised: 3. M ULTI -A GENT I MITATION L EARNING WITH C OPULA - BASED P OLICIES

Original: A central question in multi-agent imitation learning is how to model the dependency structure among agent decisions properly.
Revised: A central question in multi-agent imitation learning is how to model the dependency structure among agent decisions properly.

Original: As discussed above, the framework of copulas provides a mechanism to decouple the marginal policies (individual behavioral patterns) from the dependency left in the joint policy after removing the information in marginals.
Revised: As discussed above, the framework of copulas provides a mechanism to decouple the marginal policies (individual behavioral patterns) from the dependency left in the joint policy after removing the information in marginals.

Original: In this work, we advocate copula-based policy for multi-agent learning because copulas offer some unique and desirable properties in multi-agent scenarios.
Revised: In this work, we advocate copula-based policy for multi-agent learning because copulas offer some unique and desirable properties in multi-agent scenarios.

Original: For example, suppose we want to model the interactions among players in a sports game.
Revised: For example, suppose we want to model the interactions among players in a sports game.

Original: Using copula-based policy, we will obtain marginal policies for each individual player as well as dependencies among different roles (e.g., forwards and midﬁelders in soccer).
Revised: Using copula-based policy, we will obtain marginal policies for each individual player as well as dependencies among different roles (e.g., forwards and midﬁelders in soccer).

Original: Such a multi-agent learning framework has the following advantages:
Revised: Such a multi-agent learning framework has the following advantages:

Original: • Interpretable .
Revised: • Interpretable .

Original: The learned copula density can be easily visualized to intuitively analyze the correlation among agent actions.
Revised: The learned copula density can be easily visualized to intuitively analyze the correlation among agent actions.

Original: • Scalable .
Revised: • Scalable .

Original: When the marginal policy of agents changes but the dependency among different agents remain the same (e.g., in a soccer game, one player is replaced by his/her substitute, but the dependence among different roles are basically the same regardless of players), we can obtain a new joint policy efﬁciently by switching in the new agent’s marginal while keeping the copula and other marginals unchanged.
Revised: When the marginal policy of agents changes but the dependency among different agents remain the same (e.g., in a soccer game, one player is replaced by his/her substitute, but the dependence among different roles are basically the same regardless of players), we can obtain a new joint policy efﬁciently by switching in the new agent’s marginal while keeping the copula and other marginals unchanged.

Original: • Succinct .
Revised: • Succinct .

Original: The copula-based factorization of the joint policy avoids the redundancy in previous opponent modeling approaches (Tian et al., 2019; Liu et al., 2020) (as discussed in Section 3.1) by separately learning marginals and a copula. 
Revised: The copula-based factorization of the joint policy avoids the redundancy in previous opponent modeling approaches (Tian et al., 2019; Liu et al., 2020) (as discussed in Section 3.1) by separately learning marginals and a copula. [Table]

Original: Learning.
Revised: Learning.

Original: In this section, we discuss how to learn a copula-based policy from a set of expert demonstrations.
Revised: In this section, we discuss how to learn a copula-based policy from a set of expert demonstrations.

Original: Under the framework of Markov games and copulas, we factorize the parametric joint policy as:
Revised: Under the framework of Markov games and copulas, we factorize the parametric joint policy as:

Original: [Equation]
Revised: [Equation]

Original: where π i ( a i | s ; θ i ) is the marginal policy of agent
Revised: where π i ( a i | s ; θ i ) is the marginal policy of agent

Original: i with parameters θ i and F i is the corresponding cumulative distribution function; the function c (parameterized by θ c ) is the density of the copula on the transformed actions u i
Revised: i with parameters θ i and F i is the corresponding cumulative distribution function; the function c (parameterized by θ c ) is the density of the copula on the transformed actions u i

Original: = F i ( a i | s ; θ i ) obtained by processing original actions with probability integral transform.
Revised: = F i ( a i | s ; θ i ) obtained by processing original actions with probability integral transform.

Original: The training algorithm of our proposed method is presented as Algorithm 1 in Appendix A. Given a set of expert demonstrations D , our goal is to learn marginal actions of agents and their copula function.
Revised: The training algorithm of our proposed method is presented as Algorithm 1. Given a set of expert demonstrations D , our goal is to learn marginal actions of agents and their copula function.

Original: Our approach consists of two steps.
Revised: Our approach consists of two steps.

Original: We ﬁrst learn marginal action distributions of each agent given their current state (lines 1-6).
Revised: We ﬁrst learn marginal action distributions of each agent given their current state (lines 1-6).

Original: This is achieved by training MLP marginal that takes as input a state s and output the parameters of marginal action distributions of N agents given the input state (line 3).
Revised: This is achieved by training MLP marginal that takes as input a state s and output the parameters of marginal action distributions of N agents given the input state (line 3).

Original: In our implementation, we use mixture of Gaussians to realize each marginal policy π i ( a i | s ; θ i ) such that we can model complex multi-modal marginals while having a tractable form of the marginal cumulative distribution functions.
Revised: In our implementation, we use mixture of Gaussians to realize each marginal policy π i ( a i | s ; θ i ) such that we can model complex multi-modal marginals while having a tractable form of the marginal cumulative distribution functions.

Original: Therefore, the output of MLP marginal consists of the means, covariance, and weights of components for the N agents’ Gaussian mixtures.
Revised: Therefore, the output of MLP marginal consists of the means, covariance, and weights of components for the N agents’ Gaussian mixtures.

Original: We then calculate the likelihood of each observed action a j based on agent j ’s marginal action distribution (line 5), and maximize the likelihood by optimizing the parameters of MLP marginal (line 6).
Revised: We then calculate the likelihood of each observed action a j based on agent j ’s marginal action distribution (line 5), and maximize the likelihood by optimizing the parameters of MLP marginal (line 6).

Original: After learning marginals, we ﬁx the parameters of marginal MLPs and start learning the copula (lines 7-20).
Revised: After learning marginals, we ﬁx the parameters of marginal MLPs and start learning the copula (lines 7-20).

Original: We ﬁrst process the original demonstrations using probability integral transform and obtain a set of new demonstrations with uniform marginals (lines 8-13).
Revised: We ﬁrst process the original demonstrations using probability integral transform and obtain a set of new demonstrations with uniform marginals (lines 8-13).

Original: Then we learn the density of copula (lines 14-20).
Revised: Then we learn the density of copula (lines 14-20).

Original: Notice that the copula can be implemented as either state-dependent (lines 14-17) or state-independent (lines 18-20): For state-dependent copula, we use MLP copula to take as input the current state s and outputs the parameters of copula density c ( ·| s ) (line 15).
Revised: Notice that the copula can be implemented as either state-dependent (lines 14-17) or state-independent (lines 18-20): For state-dependent copula, we use MLP copula to take as input the current state s and outputs the parameters of copula density c ( ·| s ) (line 15).

Original: Then we calculate the likelihood of copula value u (line 16) and maximize the likelihood by updating MLP copula (line 17).
Revised: Then we calculate the likelihood of copula value u (line 16) and maximize the likelihood by updating MLP copula (line 17).

Original: For state-independent copula, we directly calculate the likelihood of copula value u under c ( · ) (line 19) and learn parameters of c ( · ) by maximizing the likelihood (line 20).
Revised: For state-independent copula, we directly calculate the likelihood of copula value u under c ( · ) (line 19) and learn parameters of c ( · ) by maximizing the likelihood (line 20).

Original: The copula density ( c ( · ) or c ( ·| s ) ) can be implemented using parametric methods such as Gaussian or mixture of Gaussians.
Revised: The copula density ( c ( · ) or c ( ·| s ) ) can be implemented using parametric methods such as Gaussian or mixture of Gaussians.

Original: It is worth noticing that if copula is state-independent, it can also be implemented using non-parametric methods such as kernel density estimation (Parzen, 1962; Davis et al., 2011).
Revised: It is worth noticing that if copula is state-independent, it can also be implemented using non-parametric methods such as kernel density estimation (Parzen, 1962; Davis et al., 2011).

Original: In this way, we no longer learn parameters of copula by maximizing likelihood as in lines 19-20, but simply store all copula values u for density estimation and sampling in inference stage.
Revised: In this way, we no longer learn parameters of copula by maximizing likelihood as in lines 19-20, but simply store all copula values u for density estimation and sampling in inference stage.

Original: We will visualize the learned copula in experiments.
Revised: We will visualize the learned copula in experiments.

Original: Inference and Generation.
Revised: Inference and Generation.

Original: In inference stage, the goal is to predict the joint actions of all agents given their current state s .
Revised: In inference stage, the goal is to predict the joint actions of all agents given their current state s .

Original: The inference algorithm is presented as Algorithm 2 in Appendix A, where we ﬁrst sample a copula value u
Revised: The inference algorithm is presented as Algorithm 2 in Appendix A, where we ﬁrst sample a copula value u

Original: = ( u 1 , · · · , u N ) from the learned copula, either statedependent or state-independent (lines 1-5), then apply inverse probability transform to transform them to the original action space:
Revised: = ( u 1 , · · · , u N ) from the learned copula, either statedependent or state-independent (lines 1-5), then apply inverse probability transform to transform them to the original action space:

Original: ˆ a j = F − 1 j ( u j | s ) (lines 7-10).
Revised: ˆ a j = F − 1 j ( u j | s ) (lines 7-10).

Original: Note that an analytical form of the inverse cumulative distribution function may not always be available.
Revised: Note that an analytical form of the inverse cumulative distribution function may not always be available.

Original: In our implementation, we use binary search to approximately solve this problem since F j is a strictly increasing function, which is shown to be highly efﬁcient in practice.
Revised: In our implementation, we use binary search to approximately solve this problem since F j is a strictly increasing function, which is shown to be highly efﬁcient in practice.

Original: In addition, we can also sample multiple i.i.d.
Revised: In addition, we can also sample multiple i.i.d.

Original: copula values from c ( ·| s ) or c ( · ) (line 3 or 5), transform them into the original action space, and take their average as the predicted action.
Revised: copula values from c ( ·| s ) or c ( · ) (line 3 or 5), transform them into the original action space, and take their average as the predicted action.

Original: This strategy is shown to be able to improve the accuracy of action prediction (in terms of MSE loss), but requires more running time as a trade-off.
Revised: This strategy is shown to be able to improve the accuracy of action prediction (in terms of MSE loss), but requires more running time as a trade-off.

Original: The generation algorithm is presented as Algorithm 3 in Appendix A. To generate new trajectories, we repeatedly predict agent actions given the current state (line 2), then execute the generated action and obtain an updated state from the environment (line 3).
Revised: The generation algorithm is presented as Algorithm 3 in Appendix A. To generate new trajectories, we repeatedly predict agent actions given the current state (line 2), then execute the generated action and obtain an updated state from the environment (line 3).

Revised: The computational complexity of the training and the inference algorithms is analyzed as follows.

Revised: The complexity of each round in Algorithm 1 is O ( MTN ) , where M is the number of trajectories in the training set, T is the length of each trajectory, and N is the number of agents.

Revised: The complexity of Algorithm 2 is O ( N ) .

Revised: The training and the inference algorithms scales linearly with the size of input dataset.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The key problem in multi-agent imitation learning is how to model the dependence structure among multiple interactive agents.
Revised: The key problem in multi-agent imitation learning is how to model the dependence structure among multiple interactive agents.

Original: Le et al.
Revised: Le et al.

Original: (2017) learn a latent coordination model for players in a cooperative game, where different players occupy different roles.
Revised: (2017) learn a latent coordination model for players in a cooperative game, where different players occupy different roles.

Original: However, there are many other multiagent scenarios where agents do not cooperate for a same goal or they do not have speciﬁc roles (e.g., self-driving).
Revised: However, there are many other multiagent scenarios where agents do not cooperate for a same goal or they do not have speciﬁc roles (e.g., self-driving).

Original: Bhattacharyya et al.
Revised: Bhattacharyya et al.

Original: (2018) adopt parameter sharing trick to extend generative adversarial imitation learning to handle multi-agent problems, but it does not model the interaction of agents.
Revised: (2018) adopt parameter sharing trick to extend generative adversarial imitation learning to handle multi-agent problems, but it does not model the interaction of agents.

Original: Interaction Network (Sukhbaatar et al., 2016) learns a physical simulation of objects with binary relations, and CommNet (Sukhbaatar et al., 2016) learns dynamic communication among agents.
Revised: Interaction Network (Battaglia et al., 2016) learns a physical simulation of objects with binary relations, and CommNet (Sukhbaatar et al., 2016) learns dynamic communication among agents.

Original: But they fail to characterize the dependence among agent actions explicitly.
Revised: But they fail to characterize the dependence among agent actions explicitly.

Original: Researchers also propose to infer multi-agent relationship using graph techniques or attention mechanism.
Revised: Researchers also propose to infer multi-agent relationship using graph techniques or attention mechanism.

Original: For example, Kipf et al.
Revised: For example, Kipf et al.

Original: (2018) propose to use graph neural networks (GNN) to infer the type of relationship among agents.
Revised: (2018) propose to use graph neural networks (GNN) to infer the type of relationship among agents.

Original: Hoshen (2017) introduces attention mechanism into multi-agent predictive modeling.
Revised: Hoshen (2017) introduces attention mechanism into multi-agent predictive modeling.

Original: Li et al.
Revised: Li et al.

Original: (2020) combine generative models and attention mechanism to capture behavior generating process of multi-agent systems.
Revised: (2020) combine generative models and attention mechanism to capture behavior generating process of multi-agent systems.

Original: These works address the problem of reasoning relationship among agents rather than capturing their dependence when agents are making decisions.
Revised: These works address the problem of reasoning relationship among agents rather than capturing their dependence when agents are making decisions.

Original: Another line of related work is deep generative models in multi-agent systems.
Revised: Another line of related work is deep generative models in multi-agent systems.

Original: For example, Zhan et al.
Revised: For example, Zhan et al.

Original: (2018) propose a hierarchical framework with programmatically produced weak labels to generate realistic multi-agent trajectories of basketball game.
Revised: (2018) propose a hierarchical framework with programmatically produced weak labels to gen- games

Original: Yeh et al. (2019) use GNN and variational recurrent neural networks (VRNN) to design a permutation equivariant multi-agent trajectory generation model for sports games. Ivanovic et al. (2018) combine conditional variational autoencoder (CVAE) and long-short term memory networks (LSTM) togenerate behavior of basketball players.
Revised: CVAE) and long-short term memory networks (LSTM) to behavior of basketball players.

Original: Most of the existing works focus on agent behavior forecasting but provide limited information regarding the dependence among agent behaviors.
Revised: Most of the existing works focus on agent behavior forecasting but provide limited information regarding the dependence among agent behaviors.

Original:  E XPERIMENTS 5. E XPERIMENTAL S ETUP
Revised:  E XPERIMENTS 5. E XPERIMENTAL S ETUP

Original: Datasets.
Revised: Datasets.

Original: We evaluate our method in three settings.
Revised: We evaluate our method in three settings.

Original: PhySim is a synthetic physical environment where multiple particles are connected by springs.
Revised: PhySim is a synthetic physical environment where 5 particles are connected by springs.

Original: Driving is a synthetic driving environment where one vehicle follows anotherone along a single lane.
Revised: Driving is a synthetic driving environment where one vehicle follows another along a single lane.

Original: RoboCup is collected from an international scientiﬁc robot competition where  teams of multiple robots compete against each other.
Revised: RoboCup is collected from an international scientiﬁc robot competition where two robot teams (including 22 robots) compete against each other.

Original: The detailed dataset description is provided in Appendix B. Experimental environments are shown in Figure 2.
Revised: The detailed dataset description is provided in Appendix B. Experimental environments are shown in Figure 2.

Original: Baselines.
Revised: Baselines.

Original: We compare our method with the following baselines: LR is a logistic regression model that predicts actions of agents using all of their states.
Revised: We compare our method with the following baselines: LR is a logistic regression model that predicts actions of agents using all of their states.

Original: SocialLSTM (Alahi et al., 2016) predicts agent trajectory using RNNs with a social pooling layer in the hidden state of nearby agents.
Revised: SocialLSTM (Alahi et al., 2016) predicts agent trajectory using RNNs with a social pooling layer in the hidden state of nearby agents.

Original: IN (Battaglia et al., 2016) predicts agent states and their interactions using deep neural networks.
Revised: IN (Battaglia et al., 2016) predicts agent states and their interactions using deep neural networks.

Original: CommNet (Sukhbaatar et al., 2016) simulates the inter-agent communication by broadcasting the hidden states of all agents and then predicts their actions.
Revised: CommNet (Sukhbaatar et al., 2016) simulates the inter-agent communication by broadcasting the hidden states of all agents and then predicts their actions.

Original: VAIN (Hoshen, 2017) uses neural networks with attention mechanism for multi-agents modeling.
Revised: VAIN (Hoshen, 2017) uses neural networks with attention mechanism for multi-agents modeling.

Original: NRI (Kipf et al., 2018) designs a graph neural network based model to learn the interaction type among multiple agents.
Revised: NRI (Kipf et al., 2018) designs a graph neural network based model to learn the interaction type among multiple agents.

Original: Since most of the baselines are used for predicting future states given historical state, we change the implementation of their objective functions and use them to predict the current action of agents given historical states.
Revised: Since most of the baselines are used for predicting future states given historical state, we change the implementation of their objective functions and use them to predict the current action of agents given historical states.

Original: Each experiment is repeated 3 times, and we report the mean and standard deviation.
Revised: Each experiment is repeated 3 times, and we report the mean and standard deviation.

Original: 5. R ESULTS
Revised: 5. R ESULTS

Original: [Table]
Revised: [Table]

Original: We compare our method with baselines in the task of action prediction.
Revised: We compare our method with baselines in the task of action prediction.

Original: The results of root mean squared error (RMSE) between predicted actions and groundtruth actions are presented in Table
Revised: The results of root mean squared error (RMSE) between predicted actions and groundtruth actions are presented in Table

Original: [Table]
Revised: [Table]

Original: The number of Gaussian mixture components in our method is set to 2 for all datasets.
Revised: The number of Gaussian mixture components in our method is set to 2 for all datasets.

Original: The results demonstrate that all methods performs the best on PhySim dataset, since agents in PhySim follow simple physical rules and the relationships among them are linear thus easy to infer.
Revised: The results demonstrate that all methods performs the best on PhySim dataset, since agents in PhySim follow simple physical rules and the relationships among them are linear thus easy to infer.

Original: However, the interactions of agents in Driving and
Revised: However, the interactions of agents in Driving and

Original: RoboCup datasets are more complicated, which causes LR and NRI to underperform other baselines.
Revised: RoboCup datasets are more complicated, which causes LR and NRI to underperform other baselines.

Original: The performance of IN, CommNet, and VAIN are similar, which is in accordance with the result reported in Hoshen (2017).
Revised: The performance of IN, CommNet, and VAIN are similar, which is in accordance with the result reported in Hoshen (2017).

Original: Our method is shown to outperform all baselines signiﬁcantly on all three datasets, which demonstrates that explicitly characterizing dependence of agent actions could greatly improve the performance of multi-agent behavior modeling
Revised: Our method is shown to outperform all baselines signiﬁcantly on all three datasets, which demonstrates that explicitly characterizing dependence of agent actions could greatly improve the performance of multi-agent behavior modeling.

Original: [Table]
Revised: [Table]

Original: To investigate the efﬁcacy of copula, we implement three types of copula function:
Revised: To investigate the efﬁcacy of copula, we implement three types of copula function:

Original: Uniform copula means we do not model dependence among agent actions.
Revised: Uniform copula means we do not model dependence among agent actions.

Original: KDE copula uses kernel density estimation to model the copula function, which is stateindependent.
Revised: KDE copula uses kernel density estimation to model the copula function, which is stateindependent.

Original: Gaussian mixtures copula uses Gaussian mixture model to characterize the copula function, of which the parameters are output by an MLP taking as input the current state.
Revised: Gaussian mixtures copula uses Gaussian mixture model to characterize the copula function, of which the parameters are output by an MLP taking as input the current state.

Original: We train the three models on training trajectories, then calculate negative log-likelihood (NLL) of test trajectories using the three trained models.
Revised: We train the three models on training trajectories, then calculate negative log-likelihood (NLL) of test trajectories using the three trained models.

Original: A lower NLL score means that the model assigns high likelihood to given trajectories, showing that it is better at characterizing the dataset.
Revised: A lower NLL score means that the model assigns high likelihood to given trajectories, showing that it is better at characterizing the dataset.

Original: The NLL scores of the three models on the three datasets are reported in Table 2.
Revised: The NLL scores of the three models on the three datasets are reported in Table 2.

Original: The performance ofKDE copula and Gaussian copula both surpasses uniform copula, which demonstrates that modeling dependence among agent actions is essential for improving model expressiveness.
Revised: The performance of copula and Gaussian copula both surpasses uniform copula, which demonstrates that modeling dependence among agent actions is essential for improving model expressiveness.

Original: However, Gaussian copula performs worse than KDE copula, because Gaussian copula is state-dependent thus increases the risk of overﬁtting.
Revised: However, Gaussian copula performs worse than KDE copula, because Gaussian copula is state-dependent thus increases the risk of overﬁtting.

Original: Notice that the performance gap between KDE and Gaussian copula is less on
Revised: Notice that the performance gap between KDE and Gaussian copula is less on

Original: PhySim, since PhySim dataset is much larger so the Gaussian copula can be trained more effectively.
Revised: PhySim, since PhySim dataset is much larger so the Gaussian copula can be trained more effectively.

Original: 5. G ENERALIZATION C APABILITY OF C OPULA
Revised: 5. G ENERALIZATION C APABILITY OF C OPULA

Original: [Table]
Revised: [Table]

Original: One beneﬁt of copulas is that copula captures the pure dependence among agents, regardless of their own marginal action distributions.
Revised: One beneﬁt of copulas is that copula captures the pure dependence among agents, regardless of their own marginal action distributions.

Original: To demonstrate the generalization capabilities of copulas, we design the following experiment.
Revised: To demonstrate the generalization capabilities of copulas, we design the following experiment.

Original: We ﬁrst train our model on the original dataset, and learn marginal action distributions and copula function (which is called original marginals and original copula ).
Revised: We ﬁrst train our model on the original dataset, and learn marginal action distributions and copula function (which is called original marginals and original copula ).

Original: Then we substitute one of the agents with a new agent and use the simulator to generate a new set of trajectories.
Revised: Then we substitute one of the agents with a new agent and use the simulator to generate a new set of trajectories.

Original: Speciﬁcally, this is achieved by doubling the action value of one agent (for example, this can be seen as substituting an existing particle with a lighter one in PhySim).
Revised: Speciﬁcally, this is achieved by doubling the action value of one agent (for example, this can be seen as substituting an existing particle with a lighter one in PhySim).

Original: We retrain our model on new trajectories and learn new marginals and new copula .
Revised: We retrain our model on new trajectories and learn new marginals and new copula .

Original: We evaluate the likelihood of new trajectories based on whether to use the original marginals or original copula, which, accordingly, results in four combinations.
Revised: We evaluate the likelihood of new trajectories based on whether to use the original marginals or original copula, which, accordingly, results in four combinations.

Original: The NLL scores of four combinations are presented in Table 3.
Revised: The NLL scores of four combinations are presented in Table 3.

Original: It is clear, by comparing the ﬁrst and the last column, that “new marginals + new copula” signiﬁcantly outperform “original marginals + original copula”, since new marginals and new copula are trained on new trajectories and therefore characterize the new joint distribution exactly.
Revised: It is clear, by comparing the ﬁrst and the last column, that “new marginals + new copula” signiﬁcantly outperform “original marginals + original copula”, since new marginals and new copula are trained on new trajectories and therefore characterize the new joint distribution exactly.

Original: To see the inﬂuence of marginals and copula more clearly, we further compare the results in column 2 and 3, where we use new copula or new marginals separately.
Revised: To see the inﬂuence of marginals and copula more clearly, we further compare the results in column 2 and 3, where we use new copula or new marginals separately.

Original: It is clear that the model performance does not drop signiﬁcantly if we use the original copula and new marginals (by comparing column 3 and 4), which demonstrates that the copula function basically stays the same even if marginals are changed.
Revised: It is clear that the model performance does not drop signiﬁcantly if we use the original copula and new marginals (by comparing column 3 and 4), which demonstrates that the copula function basically stays the same even if marginals are changed.

Original: The result supports our claim that the learned copula is generalizable in the case where marginal action distributions of agents change but the internal inter-agent relationship stays the same.
Revised: The result supports our claim that the learned copula is generalizable in the case where marginal action distributions of agents change but the internal inter-agent relationship stays the same.

Original: 5. C OPULA V ISUALIZATION
Revised: 5. C OPULA V ISUALIZATION

Original: Another beneﬁt of copulas is that it is able to intuitively demonstrate the correlation among agent actions.
Revised: Another beneﬁt of copulas is that it is able to intuitively demonstrate the correlation among agent actions.

Original: We choose the RoboCup dataset to visualize the learned copula.
Revised: We choose the RoboCup dataset to visualize the learned copula.

Original: As shown in Figure 4a in Appendix C, we ﬁrst randomly select a game (the 6th game) between cyrus2017 and helios and draw trajectories of 10 players in the left team (L2 ∼ L11, except the goalkeeper).
Revised: As shown in Figure 4a in Appendix D, we ﬁrst randomly select a game (the 6th game) between cyrus2017 and helios and draw trajectories of 10 players in the left team (L2 ∼ L11, except the goalkeeper).

Original: It is clear that the 10 players fulﬁll speciﬁc roles: L2 ∼ L4 are defenders, L5 ∼ L8 are midﬁelders, and L9 ∼
Revised: It is clear that the 10 players fulﬁll speciﬁc roles: L2 ∼ L4 are defenders, L5 ∼ L8 are midﬁelders, and L9 ∼

Original: L11 are forwards.
Revised: L11 are forwards.

Original: Then we plot the copula density between the x-axis (the horizontal direction) of
Revised: Then we plot the copula density between the x-axis (the horizontal direction) of

Original: L2 and the x-axis of L3 ∼ L11, respectively, as shown in Figure 4b in Appendix C. These ﬁgures illustrate linear correlation between their moving direction along x-axis, that is, when L2 moves forward other players are also likely to move forward.
Revised: L2 and the x-axis of L3 ∼ L11, respectively, as shown in Figure 4b in Appendix D. These ﬁgures illustrate linear correlation between their moving direction along x-axis, that is, when L2 moves forward other players are also likely to move forward.

Original: However, the correlation strength differs with respect to different players according to the visualized result: L2 exhibits high correlation with L and L4, but low correlation with L9 ∼ L11.
Revised: However, the correlation strength differs with respect to different players according to the visualized result: L2 exhibits high correlation with L and L4, but low correlation with L9 ∼ L11.

Original: This is because L2 ∼ L4 are all defenders so they collaborate more closely with each other, but L9 ∼ L11 are forwards thus far from L2 in the ﬁeld.
Revised: This is because L2 ∼ L4 are all defenders so they collaborate more closely with each other, but L9 ∼ L11 are forwards thus far from L2 in the ﬁeld.

Original: 5. T RAJECTORY G ENERATION
Revised: 5. T RAJECTORY G ENERATION

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: The learned copula can also be used to generate new trajectories.
Revised: The learned copula can also be used to generate new trajectories.

Original: We visualize the result of trajectory generation on RobuCup dataset.
Revised: We visualize the result of trajectory generation on RobuCup dataset.

Original: As shown in Figure 3, the dotted lines denote the ground-truth trajectories of the 10 player in an attack from midﬁeld to the penalty area.
Revised: As shown in Figure 3, the dotted lines denote the ground-truth trajectories of the 10 player in an attack from midﬁeld to the penalty area.

Original: The trajectories generated by our copula model (Figure 3b) are quite similar to the demonstration as they exhibit high consistency.
Revised: The trajectories generated by our copula model (Figure 3b) are quite similar to the demonstration as they exhibit high consistency.

Original: It is clear that midﬁelders and forwards (No. 5 ∼ No.
Revised: It is clear that midﬁelders and forwards (No. 5 ∼ No.

Original: 11) are basically moving to the same direction, and they all make a left turn on their way to penalty area.
Revised: 11) are basically moving to the same direction, and they all make a left turn on their way to penalty area.

Original: However, the generated trajectories by independent modeling show little correlation since the players are all making independent decisions.
Revised: However, the generated trajectories by independent modeling show little correlation since the players are all making independent decisions.

Original: We also present the result of trajectory generation on Driving dataset in Appendix D.
Revised: We also present the result of trajectory generation on Driving dataset in Appendix E.

Original:  C ONCLUSION 
Revised:  C ONCLUSION AND F UTURE W ORK

Original: In this paper, we propose a copula-based multi-agent imitation learning algorithm that is interpretable, efﬁcient and scalable to model complex multi-agent interactions.
Revised: In this paper, we propose a copula-based multi-agent imitation learning algorithm that is interpretable, efﬁcient and scalable to model complex multi-agent interactions.

Original: Sklar’s theorem allows us to separately learn marginal policies that capture the local behavioral patterns of each individual agent and a copula function that only and fully captures the dependence structure among the agents.
Revised: Sklar’s theorem allows us to separately learn marginal policies that capture the local behavioral patterns of each individual agent and a copula function that only and fully captures the dependence structure among the agents.

Original: Compared to previous multi-agent imitation learning methods based on independent policies (meanﬁeld factorization of the joint policy) or opponent modeling, our method is capable of modeling complex dependence among agents and achieving coordination without any modeling redundancy.
Revised: Compared to previous multi-agent imitation learning methods based on independent policies (meanﬁeld factorization of the joint policy) or opponent modeling, our method is capable of modeling complex dependence among agents and achieving coordination without any modeling redundancy.

Original: Experimental results on physical simulation, driving and robot soccer datasets demonstrate the effectiveness of our method compared with state-of-the-art baselines.
Revised: Experimental results on physical simulation, driving and robot soccer datasets demonstrate the effectiveness of our method compared with state-of-the-art baselines.

Original: A P SEUDO C ODE FOR T RAINING , I NFERENCE , AND G ENERATION P ROCEDURE
Revised: We point out two directions of future work. First, the copula function is generalizable only if the dependence structure of agents (i.e., their role assignment) is unchanged. Therefore, it is interesting to study how to efﬁciently apply the learned copula to the scenario with evolving dependence structure. Another practical question is that whether our proposed method can be extended to the setting of decentralized execution, since the step of copula sampling (line 3 or 5 in Algorithm 2) is shared by all agents.

Revised: A straightforward way to solve this problem is to set a ﬁxed sequence of random seeds for all agents in advance, so that the copula sample obtained by all agents is the same at each timestamp.

Revised: Designing a more robust and elegant mechanism for decentralized execution is also a promising direction.

Original: The pseudo code for training, inference, and generation procedure are presented in Algorithm 1, 2, and 3, respectively.
Revised: The pseudo code for inference and generation procedure are presented in Algorithm 2 and 3, respectively.

Revised: Input: Marginal action distribution MLP MLP marginal , state-dependent copula MLP MLP copula or state-independent copula density c ( · ) , current state s

Original:  // Learning marginals copula 
Revised: Output: Predicted action ˆ a // Sample from copula 1 if copula is set as state-dependent then Calculate (parameters of) state-dependent copula density c ( ·| s ) ← MLP copula ( s ) Sample a copula value u

Revised: = ( u 1 , · · · , u N ) from c ( ·| s ) 4 else

Revised: Sample a copula value u

Original: else{+= ( u 1 , · · · , u N ) from c ( · ) // Transform from copula space to action space 6 Calculate (parameters of) the conditional marginal action distributions for all agents: { f j ( ·| s ) } Nj =
 ← MLP marginal ( s ) 7 for agent j = 1 , · · · , N do F j ( ·| s ) ← CDF of f j ( ·| s ) ˆ a j ← F − 1 j ( u j | s ) 10 ˆ a ← (ˆ a 1 , · · · , ˆ a j ) 11 return ˆ a+}
Revised: {+= ( u 1 , · · · , u N ) from c ( · ) // Transform from copula space to action space 6 Calculate (parameters of) the conditional marginal action distributions for all agents: { f j ( ·| s ) } Nj =
 ← MLP marginal ( s ) 7 for agent j = 1 , · · · , N do F j ( ·| s ) ← CDF of f j ( ·| s ) ˆ a j ← F − 1 j ( u j | s ) 10 ˆ a ← (ˆ a 1 , · · · , ˆ a j ) 11 return ˆ a+}

Revised: Input: Inference module (Algorithm 2), initial state s [0] , required length L , environment E Output: Generated trajectory ˆ τ 1 for l = 0 , · · · , L do

Revised: Feed state s [ l ] to the inference module and get the predicted action ˆ a [ l ]

Revised: Execute ˆ a [ l ] in environment E and get a new state s [ l + 1] 4 ˆ τ = { ( s [ l ] , ˆ a [ l ]) } Ll = 5 return ˆ τ

Original: B D ATASET D ETAILS
Revised: B D ATASET D ETAILS

Original: PhySim is collected from a physical simulation environment where ﬁve particles move in a unit 2D box.
Revised: PhySim is collected from a physical simulation environment where 5 particles move in a unit 2D box.

Original: The state is locations of all particles and the action is their acceleration (there is no need to include their velocities in state because accelerations are completely determined by particle locations).
Revised: The state is locations of all particles and the action is their acceleration (there is no need to include their velocities in state because accelerations are completely determined by particle locations).

Original: We add Gaussian noise to the observed values of actions.
Revised: We add Gaussian noise to the observed values of actions.

Original: Particles may be pairwise connected by springs, which can be represented as a binary adjacency matrix A ∈ { 0 , 1 } N × N .
Revised: Particles may be pairwise connected by springs, which can be represented as a binary adjacency matrix A ∈ { 0 , 1 } N × N .

Original: The elasticity between two particles scales linearly with their distance.
Revised: The elasticity between two particles scales linearly with their distance.

Original: At each timestamp, we randomly sample an adjacency matrix from { A 1 , A 2 } to connect all particles, where A 1 and A 2 are set as complimentary (i.e. A 1 + A 

Revised: At each timestamp, we randomly sample an adjacency matrix from { A 1 , A 2 } to connect all particles, where A 1 and A 2 are set as complimentary (i.e. A 1 + A

Original: + I = 1 ) to ensure that they are as different as possible.
Revised: + I = 1 ) to ensure that they are as different as possible.

Original: Therefore, the marginal action distribution of each particle given a system state is Gaussian mixtures with two components.
Revised: Therefore, the marginal action distribution of each particle given a system state is Gaussian mixtures with two components.

Original: Here the coordination signal for particles can be seen as the hidden variable determining which set of springs ( A 1 or A 2 ) is used at current time.
Revised: Here the coordination signal for particles can be seen as the hidden variable determining which set of springs ( A 1 or A 2 ) is used at current time.

Original: We generate 10 , 000 training trajectories, 2 , 000 validation trajectories, and 2 , 000 test trajectories for experiments, where the length of each trajectory is 500 .
Revised: We generate 10 , 000 training trajectories, 2 , 000 validation trajectories, and 2 , 000 test trajectories for experiments, where the length of each trajectory is 500 .

Original: Driving is generated by CARLA 3 (Dosovitskiy et al., 2017), an open-source simulator for autonomous driving research that provides realistic urban environments for training and validation of autonomous driving systems.
Revised: Driving is generated by CARLA 3 (Dosovitskiy et al., 2017), an open-source simulator for autonomous driving research that provides realistic urban environments for training and validation of autonomous driving systems.

Original: To generate the driving data, we design a car following scenario, where a leader car and a follower car drive in the same lane.
Revised: To generate the driving data, we design a car following scenario, where a leader car and a follower car drive in the same lane.

Original: We make the leader car alternatively accelerate to a speed upper bound and slow down to stopping.
Revised: We make the leader car alternatively accelerate to a speed upper bound and slow down to stopping.

Original: The leader car does not care about the follower and drives following its own policy.
Revised: The leader car does not care about the follower and drives following its own policy.

Original: The follower car tries to follow closely the leader car while keeping a safe distance.
Revised: The follower car tries to follow closely the leader car while keeping a safe distance.

Original: Here the state is the locations and velocities of the two cars, and the action is their accelerations.
Revised: Here the state is the locations and velocities of the two cars, and the action is their accelerations.

Original: We generate 1 , 009 trajectories in total, and split the whole data into training, validation, and test set with ratio of 6 : 2 : 2 .
Revised: We generate 1 , 009 trajectories in total, and split the whole data into training, validation, and test set with ratio of 6 : 2 : 2 .

Original: The average length of trajectories is 85 . 5 in
Revised: The average length of trajectories is 85 . 5 in

Original: Driving dataset.
Revised: Driving dataset.

Original: RoboCup (Michael et al., 2017) is collected from an international scientiﬁc robot football competition in which teams of multiple robots compete against each other.
Revised: RoboCup (Michael et al., 2017) is collected from an international scientiﬁc robot football competition in which teams of multiple robots compete against each other.

Original: The original dataset contains all pairings of 10 teams with 25 repetitions of each game ( 1 , 125 games in total).
Revised: The original dataset contains all pairings of 10 teams with 25 repetitions of each game ( 1 , 125 games in total).

Original: The state of a game (locations and velocities of 22 robots) is recorded every 100 ms, resulting in a trajectory of length 6 , 000 for each game (10 min).
Revised: The state of a game (locations and velocities of 22 robots) is recorded every 100 ms, resulting in a trajectory of length 6 , 000 for each game (10 min).

Original: We select the 25 games between two teams, cyrus2017 and helios2017, as the data used in this paper.
Revised: We select the 25 games between two teams, cyrus2017 and helios2017, as the data used in this paper.

Original: The state is locations of 10 robots (except the goalkeeper) in the left team, and the action is their velocities.
Revised: The state is locations of 10 robots (except the goalkeeper) in the left team, and the action is their velocities.

Original: The dataset is split into training, validation, and test set with ratio of 6 : 2 : 2 .
Revised: The dataset is split into training, validation, and test set with ratio of 6 : 2 : 2 .

Original: For the three datasets, to learn the marginal action distribution of each agent (i.e. Gaussian mixtures), we use an MLP with one hidden layer to take as input a state and output the centers of their Gaussian mixtures.
Revised: For the three datasets, to learn the marginal action distribution of each agent (i.e. Gaussian mixtures), we use an MLP with one hidden layer to take as input a state and output the centers of their Gaussian mixtures.

Original: To prevent overﬁtting, the variance of these Gaussian mixtures is parameterized by a free variable for each particle, and the weights of mixtures are set as uniform.
Revised: To prevent overﬁtting, the variance of these Gaussian mixtures is parameterized by a free variable for each particle, and the weights of mixtures are set as uniform.

Original: Each dimension of states and actions in the original datasets are normalized to range [ − 1 , 1] .
Revised: Each dimension of states and actions in the original datasets are normalized to range [ − 1 , 1] .

Original: For PhySim, the number of particles are set to 5 .
Revised: For PhySim, the number of particles are set to 5 .

Original: Learning rate is set to 0 .
Revised: Learning rate is set to 0 .

Original: 01 , and the weight of L2 regularizer is set to 10 − 5 .
Revised: 01 , and the weight of L2 regularizer is set to 10 − 5 .

Original: For Driving, learning rate is 0 .
Revised: For Driving, learning rate is 0 .

Original: 005 and L2 regularizer weight is 
Revised: 005 and L2 regularizer weight is 10 − 5 .

Revised: For RoboCup, learning rate is 0 .

Revised: 001 and L2 regularizer weight is 10 − 6 .

Original: C V ISUALIZED C OPULA ON R OBO C UP D ATASET
Revised: C B ASELINE I MPLEMENTATION D ETAILS

Revised: For LR, we use the default implementation in Python sklearn package.

Revised: For SocialLSTM (Alahi et al., 2016), the dimension of input is set as the dimension of states in each dataset.

Revised: The spatial pooling size is 32 , and we use an 8 × 8 sum pooling window size without overlaps.

Revised: The hidden state dimension in LSTM is 128 .

Revised: The learning rate is 0 .

Revised: For IN (Battaglia et al., 2016), all MLPs are with one hidden layer of 32 units.

Revised: The learning rate is 0 .

Revised: For CommNet (Sukhbaatar et al., 2016), all MLPs are with one hidden layer of 32 units.

Revised: The dimension of hidden states is set to 64 , and the number of communication round is set to 2 .

Revised: The learning rate is 0 .

Revised: For VAIN (Hoshen, 2017), the encoder and decoder functions are implemented as a fully connected neural network with one hidden layer of 32 units.

Revised: The dimension of hidden states is 64 , and the dimension of attention vectors is 10 .

Revised: The learning rate is 0 . 0005 .

Revised: For NRI (Kipf et al., 2018), we use an MLP encoder and an MLP decoder, with one hidden layer of 32 units.

Revised: The learning rate is 0 . 001 .

Revised: D V ISUALIZED C OPULA ON R OBO C UP D ATASET

Original: The trajectories of players in one game as well as the visualized pairwise copula are presented in
Revised: The trajectories of players in one game as well as the visualized pairwise copula are presented in

Original: Figure 4.
Revised: Figure 4.

Original: [Figure]

Original: D G ENERATED T RAJECTORIES ON D RIVING D ATASET
Revised: E G ENERATED T RAJECTORIES ON D RIVING D ATASET

Original: For the Driving dataset [Figure] we randomly select 10 original trajectories and 10 trajectories generated by our method
Revised: For the Driving dataset, we randomly select 10 original trajectories and 10 trajectories generated by our method, and show the visualization results in Figure 5.

Revised: The x-axis is timestamp and y-axis is the location (coordinate) of two cars.

Revised: Our learned policy is shown to be able to maintain the distance between two cars.

Original: [Figure]
Revised: [Figure]

Original: and show the visualization results in Figure 5.

Original: The x

Original: [Figure]
Revised: [Figure]