A SYNCHRONOUS M ULTI -A GENT G ENERATIVE A DVERSARIAL I MITATION L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Imitation learning aims to inversely learn a policy from expert demonstrations, which has been extensively studied in the literature for both single-agent settings with Markov decision processes (MDPs), and multi-agent settings with Markov games (MGs).
However, existing approaches for MGs only work for synchronous scenarios with all agents simultaneously making decisions at each turn, and do not work for general MGs, allowing agents to asynchronously make decisions in different turns.
We propose a novel framework, asynchronous multi-agent generative adversarial imitation learning (AMAGAIL), for general Markov games. The learned expert policies are proven to guarantee subgame perfect equilibrium (SPE), a stronger equilibrium than Nash equilibrium (NE).
The experiment results demonstrate that compared to state-of-the-art baselines, our AMAGAIL model can better infer the policy of each expert agent using their demonstration data collected from asynchronous decision-making scenarios.
I NTRODUCTION
Imitation learning (IL) also known as learning from demonstrations allows agents to imitate expert demonstrations to make optimal decisions without direct interactions with the environment.
Especially, inverse reinforcement learning (IRL) (Ng et al.
(2000)) recovers a reward function of an expert from collected demonstrations, where it assumes that the demonstrator follows an (near-)optimal policy that maximizes the underlying reward.
However, IRL is an ill-posed problem, because a number of reward functions match the demonstrated data (Ziebart et al.
Ho & Ermon (2016); Boularias et al.
(2011)), where various principles, including maximum entropy, maximum causal entropy, and relative entropy principles, are employed to solve this ambiguity (Ziebart et al.
(2008; 2010); Boularias et al.
Ho & Ermon (2016); Zhang et al.
Going beyond imitation learning with single agents discussed above, recent works including Song et al.
(2018),Yu et al.
(2019), have investigated a more general and challenging scenario with demonstration data from multiple interacting agents.
Such interactions are modeled by extending Markov decision processes on individual agents to multi-agent Markov games (MGs) (Littman & Szepesv´ari (1996)).
However, these works only work for synchronous MGs, with all agents making simultaneous decisions in each turn, and do not work for general MGs, allowing agents to make asynchronous decisions in different turns, which is common in many real world scenarios.
For example, in multiplayer games (Knutsson et al.
(2004)), such as Go game, and many card games, players take turns to play, thus inﬂuence each other’s decision.
The order in which agents make decisions has a signiﬁcant impact on the game equilibrium.
In this paper, we propose a novel framework, asynchronous multi-agent generative adversarial imitation learning (AMAGAIL): A group of experts provide demonstration data when playing a Markov game (MG) with an asynchronous decision-making process, and AMAGAIL inversely learns each expert’s decision-making policy.
We introduce a player function governed by the environment to capture the participation order and dependency of agents when making decisions.
The participation order could be deterministic (i.e., agents take turns to act) or stochastic (i.e., agents need to take actions by chance).
A player function of an agent is a probability function: given the perfectly known agent participation history, i.e., at each previous round in the history, we know which agent(s) participated, it provides the probability of the agent participating in the next round.
With the general MG model, our framework generalizes MAGAIL (Song et al.
from the synchronous
Markov games to (asynchronous) Markov games, and the learned expert policies are proven to guarantee subgame perfect equilibrium (SPE) (Fudenberg & Levine (1983)), a stronger equilibrium than the Nash equilibrium (NE) (guaranteed in MAGAIL Song et al.
The experiment results demonstrate that compared to GAIL (Ho & Ermon (2016)) and MAGAIL (Song et al.
our AMAGAIL model can better infer the policy of each expert agent using their demonstration data collected from asynchronous decision-making scenarios.
P RELIMINARIES 2. M ARKOV G AMES
Markov games (MGs) (Littman (1994)) are the cases of N interacting agents, with each agent making a sequence of decisions with strategies only depending on the current state.
A Markov game 1 is denoted as a tuple ( N, S , A , Y, ζ, P, η, r , γ ) with a set of states S and N sets of actions {A i } Ni =1 .
At each time step t with a state s t ∈ S , if the indicator variable I i,t = 1 , an agent i is allowed to take an action; otherwise, I i,t = 0 , the agent i does not take an action.
As a result, the participation vector I t =
[ I 1 ,t , · · · , I N,t ] indicates active vs inactive agents at step t .
The set of all possible participation vectors is denoted as I , namely, I t ∈ I .
Moreover, h t − 1 =
[ I 0 , · · · , I t − 1 ] represent the participation history from step 0 to t − 1 .
The player function Y (governed by the environment) describes the probability of an agent i being allowed to make an action at a step t , given the participation history h t − 1 , namely, Y ( i | h t − 1 ) .
ζ deﬁnes the participation probability of an agent at the initial time step ζ : [ N ] (cid:55)→ [0 , 1] .
Note that, the player function can be naturally extended to a higher-order form when the condition includes both previous participation history and previous state-action history; thus, it can be adapted to non-Markov processes.
The initial states are determined by a distribution η : S (cid:55)→ [0 , 1] .
Let φ denotes no participation, determined by player function Y , the transition process to the next state follows a transition function: P : S ×A 1 ∪{ φ }×· · ·×A N ∪{ φ } (cid:55)→ P ( S ) .
Agent i obtains a (bounded) reward given by a function r i : S×A i
(cid:55)→ R 2 .
Agent i aims to maximize its own total expected return
R i =
(cid:80) ∞ t =
γ t r i,t , where γ ∈ [0 , 1] is the discount factor.
Actions are chosen through a stationary and stochastic policy
π
i : S ×
A i (cid:55)→ [0 , 1] .
In this paper, bold variables without subscript i denote the concatenation of variables for all the agents, e.g., all actions as a , the joint policy deﬁned as π ( a | s ) =
(cid:81)
Ni =
1 π i ( a i | s ) , r as all rewards.
Subscript − i denotes all agents except for i , then ( a i , a − i ) represents the action of all N agents ( a 1 , · · · , a N ) .
We use expectation with respect to a policy π to denote an expectation with respect to the trajectories it generates.
For example, E π ,Y [ r i ( s, a i )]
(cid:44) E s t , a ∼ π , I t ∼ Y
[ (cid:80) ∞ t =
γ t r i ( s t , a i )] , denotes the following sample process as s 0 ∼ η , I 0 ∼ ζ , I t ∼ Y , a ∼ π ( ·| s t ) , s t +1 ∼ P ( s t +1 | s t , a ) , for ∀ i ∈
[ N ] .
Clearly, when the player function Y ( i | h t − 1 )
= 1 for all agents
i ’s at any time step t , a general Markov game boils down to a synchronous Markov game (Littman (1994); Song et al.
(2018)), where all agents take actions at all steps.
To distinguish our work from MAGAIL and be consistent with the literature Chatterjee et al.
(2004) and Hansen et al. , we refer the game setting discussed in MAGAIL as synchronous Markov games (SMGs), and that of our work as Markov games (MGs). 2. S UBGAME P ERFECT E QUILIBRIUM FOR M ARKOV G AMES
In synchronous Markov games (SMGs), all agents make simultaneous decisions at any time step t , with the same goal of maximizing its own total expected return.
Thus, agents’ optimal policies are interrelated and mutually inﬂuenced.
Nash equilibrium (NE) has been employed as a solution concept to resolve the dependency across agents, where no agents can achieve a higher expected reward by unilaterally changing its own policy (Song et al.
However, in Markov games (MGs) allowing asynchronous decisions, there exist situations where agents encounter states (subgames) resulted from other agents’ “trembling-hand” actions.
Since the NE does not consider the “trembling-hand” resulted states and subgames, when trapped in these situations, agents are not able to make optimal decisions based on their polices under NE.
To address this problem, Selten ﬁrstly proposed subgame perfect equilibrium (SPE) (Selten (1965)).
SPE ensures NE for every possible subgame of the original game.
It has been shown that in a ﬁnite or inﬁnite extensive-form game with either discrete or continuous time, best-response strategies all converge to SPE, rather than NE (Selten (1965); Abramsky & Winschel (2017); Xu (2016)).
2. M ULTI -A GENT I MITATION L EARNING IN S YNCHRONOUS M ARKOV G AMES
In synchronous Markov games, MAGAIL (Song et al.
(2018)) was proposed to learn experts’ policies constrained by Nash equilibrium.
Since there may exist multiple Nash equilibrium solutions, a maximum causal entropy regularizer is employed to resolve the ambiguity.
Thus, the optimal policies can be found by solving the following multi-agent reinforcement learning problem.
[Equation]
where H i ( π i ) is the γ -discounted causal entropy of policy π i ∈ Π ,
H i ( π i ) (cid:44)
E π i [ − log π i ( a i | s )]
=
E s t ,a i
∼ π i [ − (cid:80) ∞
t =
γ t log π i ( a i | s t )] , and β is a weight to the entropy regularization term.
In practice, the reward function is unknown.
MAGAIL applies multi-agent IRL (MAIRL) below to recover experts’ reward functions, with ψ as a convex regularizer,
[Equation]
Moreover, MAGAIL solves MARL ◦ MAIRL ψ ( π E ) to inversely learn each expert’s policy via applying generative adversarial imitation learning (Ho & Ermon (2016)) to each expert
i ∈
[ N ] :
[Equation]
D w i is a discriminator for agent i that classiﬁes the experts’ vs policy trajectories.
π θ represent the learned experts’ parameterized policies, which generate trajectories with maximized the scores from
D w i for i ∈ [ N ] .
A SYNCHRONOUS M ULTI -A GENT I MITATION L EARNING
Extending multi-agent imitation learning to general Markov games is challenging, because of the asynchronous decision making and dynamic state (subgame) participating.
In this section, we will tackle this problem using subgame perfect equilibrium (SPE) solution concept.
3. A SYNCHRONOUS M ULTI -A GENT R EINFORCEMENT L EARNING
In a Markov game (MG), the Nash equilibrium needs to be guaranteed at each state s ∈ S 3 , namely, we apply subgame perfect equilibrium (SPE) solution concept instead.
Formally, a set of agent policies { π i } Ni =1 is an SPE if at each state s ∈ S (also considered as a root node of a subgame), no agent can achieve a higher reward by unilaterally changing its policy on the root node or any other descendant nodes of the root node, i.e., ∀ i ∈ [ N ] , ∀
ˆ π
i (cid:54) =
π i , E π
i , π − i ,Y [ r i ] ≥ E
ˆ π
i , π − i ,Y [ r i ] .
Therefore, our constrained optimization problem is (Filar & Vrieze (2012), Theorem 3.7.2)
[Equation] [Equation]
h is q i ( s t , a i | h t − 1 ) =
E π − i ,Y
[ Y ( i | h t − 1 ) r
i ( s t , a i )+ γ (cid:88)
I t ∈I
Pr ( I t | h t − 1 ) (cid:88) s t
+1 ∈S
P ( s t +1 | s t , a s t ) v i ( s t +1 | h t )] , (7) where
Pr ( I t | h t − 1 ) =
(cid:81)
i : I i,t =1 Y ( i | h t − 1 ) (cid:81)
j : I j,t =
(1 − Y ( j | h t − 1 )) is the probability of participation vector I t given history h t − 1 .
The constraints in eq.
(5) guarantee an SPE, i.e., ( v i ( s | h ) − q i ( s, a i | h )) is non-negative for any i ∈ [ N ] .
Consistent with MAGAIL (Song et al.
the objective has a global minimum of zero under SPE, and π forms SPE
if and only if f r ( π , v ) reaches zero while being a feasible solution.
We use AMA-RL ( r ) to denote the set of policies that form a sSPE under reward function r , and can maximize γ -discounted causal entropy of policies:
[Equation]
[Equation]
(9) where q i is deﬁned in eq.
Our objective is to deﬁne a suitable inverse operator AMAIRL in analogy to MAIRL in eq.
The key idea of MAIRL is to choose a reward that creates a margin between a set of experts and every other set of policies.
However, the constraints in SPE optimization eq.
(8) can make this challenging.
To that end, we derive an equivalent Lagrangian formulation of eq.
(8) to deﬁned a margin between the expected rewards of two sets of policies to capture the “difference”.
3. A SYNCHRONOUS M ULTI -A GENT I NVERSE R EINFORCEMENT L EARNING
The SPE constraints in eq.
(9) state that no agent i can obtain a higher expected reward via 1-step temporal (TD) difference learning.
We replace 1-step constraints with (t+1)-step constraints with the solution remaining the same as AMARL.
The general idea is consistent with MAGAIL (Song et al.
The detailed derivation is in Appx A.1.
The updated (t+1)-step constraints are
[Equation]
By implementing the (t+1)-step formulation eq.
(10), we aim to construct the Lagrangian dual of the primal in eq.
Since for any policy π , f r ( π , ˆ v )
=
given ˆ v i deﬁned as in Theorem 1 in
Appx A.1 (proved in Lemma 1 in Appx A.2), we just focus on the constraints in eq.
(10) to get the dual problem
[Equation]
[Equation]
where T ti is the set of all length- t trajectories of the form { s ( j ) , a ( j ) i } tj =0 , with s (0) as initial state, λ is a vector of N · |T ( t ) i | · |H| Lagrange multipliers, and ˆ v i is deﬁned as in Theorem 1 in Appx A.1.
Theorem 2 illustrates that a speciﬁc λ is able to recover the difference of the sum of expected rewards between not all optimal and all optimal policies.
Theorem 2 For any two policies π ∗ and π , let
[Equation]
be the probability of generating the sequence τ i using policy π i and π ∗− i and h t − 1 , where
Pr ( h t − 1 ) =
Pr ( I 0 ) (cid:81) t − 1 k =1 Pr
( I k | h k − 1 ) is the probability of history h t − 1 .
Then
[Equation]
where the dual function is L ( t +1) r ( π ∗ , λ ∗ π ) and each multiplier can be considered as the probability of generating a trajectory of agent i ∈ N , τ i ∈ T ti , and h t − 1 ∈ H .
Theorem 2 (proved in Appx A.3) provides a horizon to establish AMAIRL objective function with regularizer ψ .
[Equation]
where H i ( π i )
=
E π i , π E − i [ − log π i ( a | s )] is the discounted causal entropy for policy
π i when other agents follow π E − i , and β is a hyper-parameter controlling the strength of the entropy regularization term as in GAIL (Ho & Ermon (2016)).
Corollary 2.1.
If I = 1 for all i ∈
[ N ] then AMA-IRL ψ ( π E )
=
MAIRL ψ ( π E ) ; furthermore, if
N = 1 , β = 1 then AMA-IRL ψ ( π E ) =
IRL ψ ( π E ) .
3. A SYNCHRONOUS M ULTI -A GENT O CCUPANCY M EASURE M ATCHING
We ﬁrst deﬁne the asynchronous occupancy measure in Markov games:
Deﬁnition 1 For an agent i ∈ [ N ] with a policy π i ∈ Π , deﬁne its asynchronous occupancy measure
ρ pπ i :
S ×
A i ∪ { φ } (cid:55)→ R as ρ pπ i ( s, a ) =
 
[Equation]
i ( a | [Equation] [Equation] ( s
[Equation]
( i
[Equation]
+ ∞ (cid:88)
[Equation] (cid:88) [Equation] The occupancy measure can be interpreted as the distribution of state-action pairs that an agent i encounters under the participating and nonparticipating situations.
Notably, when ζ ( i ) = 1 ,
Y ( i | h t − 1 ) = 1 for all t ∈ { 1 , ..., ∞} , h t − 1 ∈ H , asynchronous occupancy measure in MG turns to the occupancy measure deﬁned in MAGAIL and GAIL, i.e., ρ pπ i =
ρ π i .
With the additively separable regularization ψ , for each agent i , π E i is the unique optimal response to other experts π E − i .
Therefore we obtain the following theorem (see proof of Theorem 3 in Appendix A.4):
Theorem 3 Assume ψ ( r ) =
(cid:80) Ni =
1 ψ i ( r i ) , ψ i is convex for each i ∈
[ N ] , and that AMA-RL ( r ) has a unique solution 4 for all r ∈ AMA-IRL ψ ( π E ) , then
[Equation]
where π i , E − i denotes π i for agent i , and π E − i for other agents.
In practice, we are only able to calculate ρ p π E and ρ p π .
As following MAGAIL
(Song et al.
, we match the occupancy measure between ρ p π E and ρ p π rather than ρ p π E and ρ pπ i , π E − i .
P RACTICAL A SYNCHRONOUS M ULTI -A GENT I MITATION L EARNING
In this section, we propose practical algorithms for asynchronous multi-agent imitation learning, and introduce three representative scenarios with different player function structures.
4. A SYNCHRONOUS M ULTI -A GENT G ENERATIVE A DVERSARIAL I MITATION L EARNING
The selected ψ i in Proposition 1 (in Appx A.5) contributes to the corresponding generative adversarial model where each agent i has a generator π θ i and a discriminator, D w i .
When the generator is allowed to behave, the produced behavior will receive a score from discriminator.
The generator attempts to train the agent to maximize its score and fool the discriminator.
We optimize the following
[Figure]
[Equation]
In practice, the input of AMAGAIL is Z , the demonstration data from N expert agents in the same environment, where the demonstration data Z = { ( s t , a ) }
Tt =0 are collected by sampling s 0 ∼ η ,
I 0 ∼ ζ, I t ∼ Y, a ∼ π ∗ ( ·| s t ) , s t +1 ∼ P ( s t +1 | s t , a ) .
The assumptions include knowledge of
N, γ, S , A .
Transition P , initial state distribution η , agent distribution ζ , player function Y are all considered as black boxes, and no additional expert interactions with environment during training process are allowed.
In the RL process of ﬁnding each agent’s policy π θ i , we follow MAGAIL
(Song et al.
to apply Multi-agent Actor-Critic with Kronecker-factors (MACK) and use the advantage function with the baseline V ν for variance reduction.
The summarized algorithm is presented in Algorithm 1 in Appx B.
4. P LAYER F UNCTION S TRUCTURES
In MGs, the order in which agents make decisions is determined by the player function Y .
Below, we discuss three representative structures of player function Y , including synchronous participation, deterministic participation, and stochastic participation.
Synchronous participation.
When Y ( i | h t − 1 ) = 1 holds for all agents i ∈
[ N ] at every step t (as shown in Figure 1a), agents make simultaneous actions, and a general Markov game boils down to a simple synchronous Markov game.
Deterministic participation.
When the player function Y ( i | h t − 1 ) is deterministic for all agents
i ∈
[ N ] , it can only output 1 or 0 at each step t .
Many board games, e.g., Go, and Chess, have deterministic player functions, where agents take turns to play.
Figure 1b shows an example of deterministic participation structure.
Stochastic participation.
When the player function is stochastic, namely, Y ( i | h t − 1 ) ∈ [0 , 1] for some agent i ∈
[ N ] at certain time step t , the agent i will make an action by chance.
As illustrated in Figure 1c, three agents all have stochastic player functions at step t , and agent #1 does not take an action at step t , while agent
#2 and #3 happen to take actions.
E XPERIMENTS
We evaluate AMAGAIL with both stochastic and deterministic player function structures under cooperative and competitive games, respectively.
We compared our AMAGAIL with two baselines, including Behavior Cloning (BC) by OpenAI (Dhariwal et al.
and decentralized Multi-agent generative adversarial imitation learning (MAGAIL)
(Song et al.
The results are collected by averaging over 5 random seeds (refer to Appx C for implementation details).
We use the particle environment (Lowe et al.
as a basic setting, and customize it into four games to allow different asynchronous player function structures.
Deterministic Cooperative Navigation : Three agents (agent #1, #2 and #3) need to cooperate to get close to three randomly placed landmarks through physical actions.
They get high rewards if they are close to the landmarks and are penalized for any collision with each other.
Ideally, each agent should cover a single distinct landmark.
In this process, the agents must follow a deterministic participation order to take actions, i.e., in the ﬁrst round all three agents act, in the second round only agent #1 and #2 act, in the third round only agent #1 acts, and repeat these rounds until the game is completed.
Stochastic
Cooperative Navigation : This game is the same with deterministic cooperative navigation except that all three agents have a stochastic player function.
Each agent has 50% chance to act at each round t .
Deterministic Cooperative Reaching : This game has three agents with their goals as cooperatively reaching a single landmark with minimum collision.
In this game, agents follow a deterministic player function, same as that in deterministic cooperative navigation game, to make actions.
Stochastic Predator-Prey :
Three slower cooperating agents (referred to as adversaries) chase a faster agent in an environment of two landmarks; the faster agent acts ﬁrst, then each adversary with a stochastic player function of 50% chance to act with the same goal of catching the faster agent.
The adversaries and the agent need to avoid two randomly placed landmarks.
The adversaries collect rewards when touching the agent, where the agent is penalized.
Note that, an agent that does not participate in a round of a game does not get a reward.
In these four game environments, agents are ﬁrst trained with Multi-agent ACKTR (Wu et al.
Song et al.
, thus the true reward functions are available, which enable us to evaluate the quality of recovered policies.
When generating demonstrations from well-trained expert agents, a “null” (no-participation) as a placeholder action is recorded for each no-participation round in the trajectory.
The quality of a recovered policy is evaluated by calculating agents’ average true reward of a set of generated trajectories.
We compare our AMAGAIL with two baselines - behavior cloning (BC) (Pomerleau (1991)) and decentralized Multi-agent generative adversarial imitation learning (MAGAIL) (Song et al.
Behavior cloning (BC) utilizes the maximum likelihood estimation for each agent independently to approach their policies.
Decentralized multi-agent generative adversarial imitation learning (MAGAIL) treats each agent with a unique discriminator working as the agent’s reward signal and a unique generator as the agent’s policy.
It follows the maximum entropy principle to match agents’ occupancy measures from recovered policies to demonstration data.
5. P ERFORMANCES WITH D ETERMINISTIC AND S TOCHASTIC P LAY F UNCTIONS
We compare AMAGAIL with baselines under three particle environment games, namely, deterministic cooperative navigation , stochastic cooperative navigation , and deterministic cooperative reaching games.
Figure 2 show the normalized rewards, when learning policies with BC, MAGAIL and AMAGAIL, respectively.
When there is only a small amount of expert demonstrations, the normalized rewards of BC and
AMAGAIL increase, especially, when less demonstration data are used, i.e., less than 400 demonstrations.
After a sufﬁcient amount of demonstrations are used, i.e., more than 400, AMAGAIL has higher rewards than BC and MAGAIL.
This makes sense since at certain time steps there exist non-participating agents (based on the player functions), but BC and MAGAIL models consider the no-participation as an action the agent can choose, where in reality it is governed by the environment.
On the other hand, with the introduced player function Y , AMAGAIL characterizes such no participation events correctly, thus more accurately learns the expert policies.
The normalized awards of BC are roughly unchanged in Figure 2(a)&(c), and in Figure 2(b) after 400 demonstrations, which seems contradictory to that of Ross & Bagnell (2010); Song et al.
(2018), and can be explained as follows.
In Figure 2(b) (stochastic cooperative navigation), the performance of BC is low when using less demonstrations, but increases rapidly as more demonstrations are used, and ﬁnally converges to the “best” performance around 0 .
65 with 300 demonstrations.
In Figure 2(a) (resp.
Figure 2(c)), deterministic cooperative navigation (resp.
reaching) is easier to learn compared
[Table]
with the stochastic cooperative navigation game shown in Figure 2(b), since there is no randomness in the player function.
The performance with only 200 demonstrations is already stabilized at 0 .(resp. 0 . 94 ).
In the stochastic cooperative navigation game (Figure 2(b)), AMAGAIL performs consistently better than MAGAIL and BC.
However, in the deterministic cooperative navigation game (Figure 2(b)), with 200 demonstration, AMAGAIL does not perform as well as MAGAIL.
This is due to the game setting, namely, two players actively searching for landmarks are sufﬁcient to gain a high reward in this game.
The last agent, player #3, learned to be “lazy”, without any motivation to promote the total shared reward among all agents.
In this case, it is hard for AMAGAIL to learn a good policy of player #3 with small amount of demonstration data, because player #3’s hasabsence rate, given the pre-deﬁned deterministic participation function.
Hence, AMAGAIL does not have enough state-action pairs to learn player #3.
This gets improved when there are sufﬁcient data, say, more than 400 demonstrations.
When we adjust the game setting from 3 landmarks to 1 landmark, i.e., all agents need to act actively to reach the landmark.
This is captured in the deterministic cooperative reaching game.
In this scenario, an inactive player will lower down the overall reward.
As shown in Figure 2(c), AMAGAIL outperforms BC and MAGAIL consistently, even with a small amount of demonstration data.
5. P ERFORMANCE WITH MIXED GAME MODE
Now, we further evaluate the performance of AMAGAIL under a mixed game mode with both cooperative and adversarial players, i.e., stochastic predator-prey game.
Since there are two competing sides in this game, we cannot directly compare each methods’ performance via expected reward.
Therefore, we use the Song et al.
(2018)’s evaluation paradigm and compare with baselines by letting (agents trained by) BC play against (adversaries trained by) other methods, and vice versa.
From Table 1, AMAGAIL consistently performs better than MAGAIL and BC.
R ELATED W ORK , D ISCUSSION , AND C ONCLUSION
Imitation learning (IL) aims to learn a policy from expert demonstrations, which has been extensively studied in the literature for single agent scenarios (Finn et al.
; Ho & Ermon (2016)).
Behavioral cloning (BC) uses the observed demonstrations to directly learn a policy (Pomerleau (1991);
Torabi et al.
Apprenticeship learning and inverse reinforcement learning (IRL) ((Ng et al.
(2000); Syed & Schapire (2008);
Ziebart et al.
; Boularias et al.
(2011))) seek for recovering the underlying reward based on expert trajectories in order to further learn a good policy via reinforcement learning.
The assumption is that expert trajectories generated by the optimal policy maximize the unknown reward.
Generative adversarial imitation learning (GAIL) and conditional GAIL (cGAIL) incorporate maximum casual entropy IRL (Ziebart et al.
and the generative adversarial networks (Goodfellow et al.
to simultaneously learn non-linear policy and reward functions (Ho & Ermon (2016)
; Zhang et al.
; Baram et al.
A few recent studies on multi-agent imitation learning, such as MAGAIL (Song et al.
(2018) and MAAIRL (Yu et al.
, model the interactions among agents as synchronous Markov games, where all agents make simultaneous actions at each step t .
These works fail to characterize a more general and practical interaction scenario, i.e., Markov games including turn-based games (Chatterjee et al. (2004)), where agents make asynchronous decisions over steps.
In this paper, we make the ﬁrst attempt to propose an asynchronous multi-agent generative adversarial imitation learning (AMAGAIL) framework, which models the asynchronous decision-making process as a Markov game and develops a player function to capture the participation dynamics of agents.
Experimental results demonstrate that our proposed AMAGAIL can accurately learn the experts’ policies from their asynchronous trajectory data, comparing to state-of-the-art baselines.
Beyond capturing the dynamics of participation vs no-participation (as only two participation choices), our proposed player function Y (and
AMAGAIL framework) can also capture a more general case 5 , where Y determines how the agent participates in a particular round, i.e., which action set A mi to follow, with m ∈
[ M ] and M ≥ 1 .
A A PPENDIX A. P ROOFS A. T IME D IFFERENCE L EARNING
Theorem 1.
For a certain policy π and reward r , let ˆ v i ( s ( t ) ; π , r , h t − 1 ) be the unique solution to the Bellman equation:
[Equation]
Denote ˆ q ( t ) i ( { s ( j ) , a ( j ) } t − 1 j =0 , s ( t ) , a ( t ) i ; π , r , h t − 1 ) as the discounted expected return for the i -th agent conditioned on visiting the trajectory { s ( j ) , a ( j ) } t − 1 j =0 , s ( t ) in the ﬁrst t − 1 steps and choosing action a ( t ) i at the t -th step, when other agents using policy π − i : ˆ
[Equation]
( t
[Equation] [Equation]
Then π is subgame perfect equilibrium if and only if:
[Equation]
Theorem 1 illustrates that if we replace the 1-step constraints with ( t + 1) -step constraints, we still get the same solution as AMA-RL( r ) in terms of a subgame perfect equilibrium solution.
A. E XISTENCE AND E QUIVALENCE OF V AND S UBGAME P ERFECT E QUILIBRIUM
Lemma 1 By deﬁnition of ˆ v i ( s ( t ) ; π , r , h t − 1 ) in Theorem 1 and ˆ q i ( s ( t ) , a i ; π , r , h t − 1 ) in eq.
Then for any π , f r ( π , ˆ v )
= 0 .
Furthermore, π is subgame perfect equilibrium under r if and only if ˆ v
i ( s ; π , r , h t − 1 ) ≥
ˆ q i ( s, a i ; π , r , h t − 1 ) for all i ∈ [ N ] , s ∈ S , a i ∈
A i and h t − 1 ∈ H .
Proof We have ˆ v
i ( s ( t ) ; π , r , h t − 1 )
[Equation]
a
[Equation]
+ γ (cid:88)
[Equation]
(cid:88) s
[Equation]
which utilizes the fact that a i and a − i are independent at s .
Therefore, we can easily get f r ( π , ˆ v ) = 0 .
If π is a subgame perfect equilibrium, and existing one or more of the constrains does not hold, so agent i can receive a strictly higher expected reward for rest of the states, which is against the subgame perfect equilibrium assumption.
If the constraints hold, i.e., for all i and ( s, a i ) , ˆ v i ( s ; π , r , h t − 1 ) ≥
ˆ q
i
( s, a i ; π , r , h t − 1 )
then ˆ v i ( s ; π , r , h t − 1 ) ≥
E π i [ˆ q i ( s, a i ; π , r , h t − 1 )]
= ˆ v i ( s ; π , r , h t − 1 ) .
Value iteration, thus, over ˆ v i ( s ; π , r , h t − 1 ) converges.
If one can ﬁnd another policy π (cid:48) so that ˆ v i ( s ; π , r , h t − 1 ) <
E π i [ˆ q i ( s, a i ; π , r , h t − 1 )] , then at least one violation exists in the constraints since π (cid:48)
i is a convex combination over action a i .
Therefore, for any policy π (cid:48) i and action a i for any agent i , E π i [ˆ q i ( s, a i ; π , r , h t − 1 )] ≥ E π (cid:48)
i
[ˆ q i ( s, a i ; π , r , h t − 1 )] always hold, so
π i is the optimal reply to π − i , and π constitutes a subgame perfect equilibrium once it repeats this argument for all agents.
Notably, by assuming f r ( π , v ) = 0 for some v ; if v satisﬁes the assumptions, then v = ˆ v .
(cid:4)
A. P ROOF TO T HEOREM
Proof We use Q ∗ , ˆ q ∗ , ˆ v ∗ to denote the Q, ˆ q and ˆ v quantities deﬁned for policy π ∗ .
For the two terms in L ( t +1) r ( π ∗ , λ ∗ π ) we have:
[Equation]
For agent i , τ i and h t − 1 we have,
[Equation]
, h t − 1 ) .
For any agent i , we note that
[Equation]
which is using π i for agent i for the ﬁrst t steps and using π ∗ i for the remaining steps, whereas other agents follow π ∗− i .
As t −→ ∞ , this converges to E π i E
π ∗− i ,Y [ r i ( s ( j ) , a ( j ) i )] as γ t −→ 0 and ˆ q ∗
i ( s ( t ) , a ( t ) i ; π ∗ , r , h t − 1 ) is bounded.
Moreover, for ˆ v ∗ i ( s (0) ; π ∗ , r , ζ ) we have
[Equation]
Combining the two we have,
[Equation]
which describes the differences in expected rewards.
[Equation]
A. P ROOF TO T HEOREM
Proof For a single agent i where other agents have policy π E − i , we give the following analysis and deﬁnition.
For a policy π i ∈ Π , its occupancy measure deﬁned in Def. 3.3 allows us to write E π i ,Y [ r i ( s, a )]
= (cid:80) s,a ρ pπ i ( s, a ) r i ( s, a ) for any reward function r i .
A basic result is that the set of valid occupancy measures D i (cid:44) {
ρ pπ i : π i ∈ Π } can be written as a feasible set of afﬁne constraints:
D i = { ρ pπ i : ρ pπ i ≥ 0 and (cid:88)
[Equation]
Therefore, the proof of AMA-RL ◦ AMA-IRL can be derived in a similar fashion with GAIL (Ho &
Ermon (2016)) and MAGAIL (Song et al.
[Equation]
A. P ROPOSITION
Proposition 1: If β = 0 and ψ ( r ) = (cid:80) Ni =1 ψ i ( r i ) where ψ i ( r i ) =
E π E ,Y
[ g ( r i )]
if r i > 0; + ∞ otherwise, and
[Equation]
then
[Equation]
Theorem 3 and Proposition 1 discuss the differences from the single agent scenario similar in Song et al.
On the one hand, in Theorem 3 we make the assumption that AMA-RL ( r ) has a unique solution, which is always true in the single agent case due to convexity of the space of the optimal policies.
On the other hand, in Proposition 1 we remove the entropy regularizer because here the causal entropy for π i may depend on the policies of the other agents, so the entropy regularizer on two sides are not the same quantity.
Speciﬁcally, the entropy for the left hand side conditions on π E − i and the entropy for the right hand side conditions on π − i (which would disappear in the single-agent case).
B A PPENDIX B. A LGORITHM
Input: Initial parameters of policies, discriminators and value (baseline) estimators, θ , w , ν ; stateaction pair demonstrations
Z = { ( s j , a ) } Tj = batch size B Markov game as a block box ( N, S , A , P, η, ζ, Y, r , γ ) .
Output: Learned policies π θ
i ’s and reward functions D
w i ’s, for i ∈
[ N ] .
1: for each epoch u = 0 , 1 , 2 , ... do 2:
Generate state-action pairs of batch size B from π u through the process: s 0 ∼ η , I 0 ∼ ζ, I t ∼ Y, a ∼ π u ( ·| s t ) , s t +1 ∼ P ( s t
+
| s t , a ) ; φ is recorded as a placeholder action when an agent does not participate in a round; denote the generated state-action pair set as X .
Sample state-action pairs from Z with batch size B ; denote the demonstrated state-action pair set as X E .
4: for each agent i = 1 , · · · , N do 5:
Filter out state-action pairs ( s, φ ) from X and X E .
Update w i to increase the objective: E X ,Y [log D w i ( s, a i )] + E X E ,Y
[log(1 − D w i ( s, a i ))] .
7: end for 8: for each agent i = 1 , · · · , N do 9:
Compute value estimate V ∗ i and advantage estimate A i for ( s, a i ) ∈ X .
Filter out state-action pairs ( s, φ ) from X and X E .
Update ν i to decrease the objective: E X ,Y [( V ν i ( s ) − V ∗ ( s )) 2 ] .
Update θ i by policy gradient with the setting step sizes: E X ,Y [ (cid:79) θ i
π
θ i ( a i | s i )
A i ( s, a )] .
13: end for 14: end for 15: Return learned policies π θ
i ’s and reward functions D
w i ’s, for i ∈
[ N ] .
C A PPENDIX C. E XPERIMENT D ETAILS C. H YPERPARAMETERS
For the particle environment, we follow the setting of MAGAIL (Song et al.
to use two layer multiple layer perceptrons with 128 cells in each layer for the policy generator network, value
[Table] [Table]
network and the discriminator.
We use a batch size of 1,000.
The policy is trained using Kroneckerfactored Approximate Curvature (K-FAC) optimizer (Martens & Grosse (2015)) with parameters the same in Song et al.
C. D ETAILED R ESULTS
Below we list the exact performance (average over agents and before normalization) in tables 2,and 4.
The means and standard deviations are computed over 1,000 epidodes.
The policies in the cooperative tasks are trained with varying number of expert demonstrations.
The policies in the competitive task are trained on a dataset with 1,000 expert trajectories.
The environment for each episode is drastically different (e.g. location of landmarks are randomly sampled), which leads to the seemingly high standard deviation across episodes.
For each game with different numbers of expert demonstrations, we consider the average reward of a random policy as the minimum reward, and the average reward of an expert policy as the maximum reward.
Then, we obtain normalized reward by running min-max normalization to normalize the rewards from different methods.
The normalized rewards are used in Figure 2 in the paper.
[Table]