Under review as a conference paper at
I MITATION L EARNING FOR M EAN F IELD G AMES WITH C ORRELATED E QUILIBRIA
Anonymous authors
Paper under double-blind review
A BSTRACT
Imitation learning (IL) aims at achieving optimal actions by learning from demonstrated behaviors without knowing the reward function and transition kernels.
Conducting IL with a large population of agents is challenging as agents’ interactions grow exponentially with respect to the population size.
Mean ﬁeld theory provides an efﬁcient tool to study multi-agent problems by aggregating information on the population level.
While the approximation is tractable, it is non-trivial to restore mean ﬁeld Nash equilibria (MFNE) from demonstrations.
Importantly, there are many real-world problems that cannot be explained by the classic MFNE concept; this includes the trafﬁc network equilibrium induced from the public routing recommendations and the pricing equilibrium of goods generated on the
E-commerce platform.
In both examples, correlation devices are introduced to the equilibrium due to the intervention from the platform.
To accommodate this, we propose a novel solution concept named adaptive mean ﬁeld correlated equilibrium (AMFCE) that generalizes MFNE.
On the theory side, we ﬁrst prove the existence of AMFCE, and establish a novel framework based on IL and AMFCE with entropy regularization (MaxEnt-AMFCE) to recover the AMFCE policy from real-world demonstrations.
Signatures from the rough path theory are then applied to characterize the mean-ﬁeld evolution.
A signiﬁcant beneﬁt of MaxEnt-AMFCE is that it can recover both the equilibrium policy and the correlation device from data.
We test our MaxEnt-AMFCE against the state-of-the-art IL algorithms for
MFGs on several tasks (including a real-world trafﬁc ﬂow prediction problem), results justify the effectiveness of our proposed method and show its potential to predicting and explaining large population behavior under correlated signals.
I NTRODUTION
Imitation learning (IL) (Hussein et al., 2017) has been widely adopted to learn the desired behavior through expert demonstrations and led to a series of impressive successes (Silver et al., 2016; Shi et al., 2019; Shang et al., 2019).
Existing imitation learning algorithms cannot handle tasks with a large group of agents due to the curse of dimensionality and the exponential growth of agent interactions when the number of agents increases.
However, many real-world scenarios require the algorithm to handle a large population.
Examples include trafﬁc management and control (Bazzan, 2009), Ad auction (Guo et al., 2019), online business with a large customer base (Ahn et al., 2007) and social behaviors between game bots and humans (Jeong et al., 2015).
For systems with a large population of homogeneous agents, mean ﬁeld theory provides a practically efﬁcient and analytically feasible approach to analyze the otherwise challenging multi-agent games (Guo et al., 2019; Yang et al., 2018b).
In the mean ﬁeld game (MFG) setting, the states of the entire population can be sufﬁciently summarized into an empirical distribution of states thanks to the homogeneity property.
Therefore it sufﬁces to consider a game between a representative agent and an empirical distribution.
Existing (and rather limited) literature on mean-ﬁeld IL assumes that the expert demonstrations are sampled from the classic mean ﬁeld Nash equilibrium (MFNE)
(Yang et al., 2018a; Chen et al., 2022).
The limitation of this framework is not general enough to capture many real-world situations where external and correlated signals inﬂuence the behavior of the entire populations.
Examples include the behavior of drivers on the trafﬁc network with routing recommendations from Google Map or
Apple Map.
Another possible example is the E-commerce platform recommendation for individual
Under review as a sellers on setting up the price for their products.
In these two examples, a mediator or a coordinator recommends decisions but individual agents who seek for greedy decisions could deviate from the recommendation if she/he ﬁnds a better option given the available information.
The existence of the mediator introduces correlations among the behaviors of individual agents.
Therefore, a more general equilibrium concept is needed before we take a step further to learn from expert demonstrations.
Inspired by the concept of correlated equilibrium (CE) for stateless game (Aumann, 1974), there are recent developments on mean ﬁeld correlated equilibrium (MFCE) with state dynamics.
Campi and Fischer assume that a mediator recommends the same stochastic policy to the entire population, resulting in a limited equilibrium set which is the same as the classic MFNE (Campi & Fischer, 2022).
In addition, it is often more practical for the mediator to recommend an action rather than a stochastic policy to individuals (see the trafﬁc routing and e-commerce examples).
Muller et al.
assume that the mediator recommends a time-independent and deterministic policy (sampled from some distribution over the deterministic policy space) to each individual (Muller et al., 2022).
This formulation is also rather limited in terms of describing the behaviors of many real-world applications and enabling sufﬁcient ﬂexibility of the population behavior.
A more general and practical setting is to establish a framework where the mediator could sample a stochastic policy based on some time-dependent signals and recommend action for each individual, which is the exact framework investigated in this paper.
(See Appendix H for a concrete example to show that our equilibrium concept is more general than the one proposed by Muller et al.
(Muller et al., 2022).)
Given the above mentioned limitations of current existing MFCE concepts and mean-ﬁeld IL approaches, we propose a new MFCE framework named adaptive mean ﬁeld correlated equilibrium (AMFCE) with time-dependent correlated signals and an individual agent can adaptively update her belief on the unobserved correlated signal.
We develop a method to recover AMFCE policy based on Maximum Entropy Regularization.
Our framework has the following important and novel ingredients:
• Novel MFCE concept with time-dependent correlated signals and adaptive belief updates from individual agents.
In this paper, we propose a new MFCE framework (called AMFCE) that the mediator recommends an action sampled from a stochastic policy for each agent at every time step.
This is a more general and ﬂexible framework compared to previous works on the MFCE (Muller et al., 2022; Campi & Fischer, 2022).
We prove the existence of AMFCE solution under mild conditions and prove that MFNE is a subclass of AMFCE.
• Entropy Regularization to overcome the equilibrium selection difﬁculty.
Most of the IL algorithms for games face the equilibrium selection issue or identiﬁability issue as there often exist multiple equilibria.
To bypass this difﬁculty, we further propose an entropy regularized AMFCE (MaxEnt-AMFCE) framework which is shown to have a unique solution.
• Using signatures from rough path theory to efﬁciently represent mean-ﬁeld evolution.
Mean ﬁeld information is often inaccessible in practice and it is computationally expensive to approximate the mean ﬁeld information by its empirical distribution.
To overcome this difﬁculty, we adopt signatures from the rough path theory to represent the mean-ﬁeld evolution, which can be easily combined with neural network training architectures and the resulting method is computationally efﬁcient.
With all these ingredients, our correlated mean ﬁeld imitation learning (CMFIL) framework can recover not only the policy but also the correlation device, which is the distribution that the correlated signal is sampled from.
To the best of our knowledge, this paper is the ﬁrst focusing on MFCE with the correlation device providing time-dependent recommendations and allowing adaptive belief updates for individual agents.
In addition, we illustrate the performance of our framework by comparing the state-of-the-art imitation learning algorithms for MFGs on several tasks, including a real-world trafﬁc ﬂow prediction problem.
The experimental results demonstrate that our framework outperforms the baseline in all tasks.
As a by-product, our framework is also suitable for solving MFNE as MFNE is a subclass of AMFCE.
P RELIMINARY : C LASSIC M EAN F IELD N ASH E QUILIBRIUM
This section introduces the classic framework of MFG and the concept of MFNE.
The classic MFG models a game between a representative agent and the state distribution of all the other agents.
Denote P ( X ) as the set of probability distributions over X and denote T = { 0 , 1 , · · · , T } as a set of time indexes.
At time t , after the representative player chooses her action a t according to some measurable policy π t : S → P ( A ) , she will receive a deterministic reward r ( s t , a t , μ t ) and her state will evolve according to current state s t ∈ S and P ( ·| s t , a t , μ t ) , where μ t ∈ P ( S ) represents the population state distribution and S is ﬁnite.
Intuitively, μ t ( s ) =
lim N
[Equation]
can be viewed as the limit of the empirical distribution of an homogeneous N -agent game where s it is the state of agent i at time t and 1 { e } is an indicator function (with value 1 if expression e holds andotherwise).
Here P : S ×
A × P ( S )
→ P ( S ) .
For ﬁxed mean-ﬁeld information μμμ = { μ t } Tt =0 , the objective of the representative agent is to solve the following decision-making problem over all admissible policies πππ = { π t } Tt =
: maximize πππ
[Equation]
subject to
[Equation]
(Classic MFG)
The Mean-ﬁeld Nash Equilibrium (MFNE) is deﬁned as the following.
Deﬁnition 1 (MFNE) .
In (Classic MFG) , a player-population proﬁle ( πππ (cid:3) , μμμ (cid:3) ) is called a MFNE (under initial state μ 0 ) if
(Single player side)
For any policy πππ , any time index t ∈ T , and any initial state s ∈ S , V t ( s,πππ (cid:3) ,μμμ (cid:3) ) ≥ V t ( s,πππ,μμμ (cid:3) ) .
(Population side) { μ ∗ t } Tt =0 [-is the dynamics under the policy πππ (cid:3) starting from s 0 ∼ μ (cid:3) 
 =-]satisﬁes μ ∗ t ( · ) = (cid:3) s ∈S ,a ∈A P ( ·| s, a, μ ∗ t − 1 ) π ∗ t − 1 ( a | s ) μ ∗ t − 1 ( s ) with initial condition μ ∗ 0 = μ 0 .
The single player side condition captures the optimality of πππ (cid:3) , when the population side is ﬁxed.
The population side condition ensures the “consistency” of the solution: it guarantees that the state distribution ﬂow of the single player matches the population state sequence μμμ (cid:3) :
= { μ (cid:3)t } Tt =0 .
P ROBLEM F ORMULATION
This section introduces a novel adaptive mean-ﬁeld correlated equilibrium (AMFCE) framework and establishes the existence of equilibria solutions under mild conditions.
We prove that the solution set of AMFCE is richer than the well-known MFNE.
Furthermore, the maximum entropy principle is adopted to select the solution with maximum entropy among the solution set of the AMFCE.
3. A DAPTIVE M EAN F IELD C ORRELATED E QUILIBRIUM (AMFCE)
To incorporate the correlations introduced by the central platforms in the trafﬁc network example and the E-commerce marketplace example introduced in Section 1, we consider a mediator (or a central agent) who samples a correlated signal z t ∈ Z at each time t , where Z is a ﬁnite signal space.
z t may represent some global conditions such as the weather on day t for the trafﬁc network example and the supply-demand imbalance in month t for the E-commerce marketplace example.
Before discussing the AMFCE, we ﬁrst introduce the concepts of behavioral policy and correlation device.
Deﬁnition 2.
For each time t , the behavioral policy π t : Z × S → Δ( A ) maps from the signal and state spaces to the simplex over the action space.
Given the correlated signal z ∈ Z and an action a ∼ π t ( ·| s, z ) will be independently sampled as a private recommendation for each agent at state s .
Deﬁnition 3.
The per-step correlation device ρ t ∈ Δ( Z ) is a publicly known distribution over the space of correlated signal, from which the mediator will sample the correlated signal at time step t .
Denote ρρρ = { ρ t } Tt =
as correlation device over the entire horizon.
Under
At every time step t , a correlated signal z t is sampled from the per-step correlation device ρ t .
Then a recommendation action a t will be sampled independently from the behavior policy π t ( ·| s t , z t ) , and sent to each agent at state s t .
This recommended action a t is private and only available to the agent.
Mathematically, denote I t = { ρ t , a t , π t ( · , · , · ) , s t , z t − 1 , μ t − 1 } as the information available to the agent at the beginning of step t .
And I
= { ρ 0 , a 0 , π 0 ( · , · , · ) , s 0 } .
Note that the agent only observes the functional form of π t but can not observe the correlated signal z t nor the recommended actions for other agents.
Based on the information I t , the agent will take an action a (cid:6) t (which may be different from the mediator’s recommendation), and then the agent at state s t will transit to the next state according to distribution P ( ·| s t , a t ,
μ t ) ∈ P ( S )
given current mean ﬁeld μ t , which follows:
[Equation]
This implies that, given μ t − 1 and π t − 1 , μ t is fully determined by z t − 1 .
After receiving the recommendation action a t , the agent can predict the correlated signal by
[Equation]
Based on the available information I t at time t , the agent can then update the prediction on the mean ﬁeld distribution of the next time-step for each possible signal z :
[Equation]
The Q function Q πππt ( s, a, μ, z ; πππ (cid:6) ) for individual agent is deﬁned as follows:
[Equation]
where E πππ (cid:2) is the expectation taken with respect to z i ∼ ρ i ( · ) , a i ∼ π i ( ·| s i , z i ) , s i +1 ∼ P ( ·| a i , μ i ) , ∀ i ∈ { t + 1 , t + 2 , · · · , T } .
We can verify that the Q function satisﬁes the following Bellman equation:
[Equation]
where the expectation is taken with respect to z (cid:6) ∼ ρ t +1 ( · ) , s (cid:6) ∼ P ( ·| s [Equation] a [Equation]
μ )
[Equation]
a (cid:6)
[Equation]
π t [Equation] 1 ( ·| s, z (cid:6) ) . Similarly, we deﬁne the optimal Q-function Q ∗ t ( s, a, μ, z ; πππ (cid:6) ) as the Q function associated with the optimal individual policy πππ ∗ given population behavior πππ (cid:6) .
It is easy to show that Q ∗ satisﬁes the following Bellman equation:
[Equation]
where the expectation is taken with respect to z (cid:6) ∼ ρ t +1 ( · ) , s (cid:6) ∼ P ( · | s [Equation] a
[Equation] μ [Equation] It is worth noting that if the policy of population π (cid:6) π
[Equation]
π
[Equation]
is ﬁxed, Q ∗ [Equation] ≥ [Equation]
a
[Equation]
μ
[Equation]
z
[Equation]
πππ [Equation] ) for any πππ . Then by induction, it holds that Q ∗ t ( s, a, μ, z ; πππ (cid:6) ) ≥ Q πππt ( s, a, μ, z ; πππ (cid:6) ) for all t ∈ T .
To introduce the concept of AMFCE, we deﬁne the set of swap function U :=
{ u : A → A} , namely u a function that modiﬁes an action a to an action u ( a ) .
Let Δ t ( s, μ, u ; πππ,ρρρ )
=
E (cid:10) Q πππt ( s, u ( a ) , μ, z ; πππ ) − Q πππt ( s, a, μ, z ; πππ ) (cid:11) , u ∈ U denote the margin of Q function of that agent takes action u ( a ) when a recommendation a is provided by the mediator, where the expectation is taken with respect to z ∼ ρ t ( · ) , a ∼ π t ( ·| s, z ) .
Deﬁnition 4.
The proﬁle ( πππ (cid:3) ,ρρρ ) composed of the time-varying stochastic policy πππ (cid:3) =
{ π (cid:3)t } Tt =0 and the correlation device ρρρ is an adaptive mean ﬁeld correlated equilibrium (AMFCE)
if
• (Single agent side) No agent has an incentive to unilaterally deviate from the recommendation action after predicting the z by (2) ,
i e.
[Equation]
[Table]
[Table]
given the correlated signals { z t } Tt =0 and with initial condition μ [-(cid:3) 
-]∗ 0 = μ 0 .
A toy example named Ocean Ranch is provided below to demonstrate the concept of AMFCE.
Example 1.
Suppose there exists a marine ranch with two sectors.
The regulator of the marine ranch adjusts the size of the ﬁsh entering the two different sectors by giving recommendations for ﬁsh.
The state space of ﬁsh is S
=
{ C, L, R } , and the action space is A =
{ L, R } .
Initial mean ﬁeld μ 0 ( C )
= 1 .
The reward r ( s, a, μ ) = 1 { s = L } μ ( L ) + 1 { s = R } μ ( R ) and T = { 0 , 1 } .
The environment dynamic is deterministic: P ( s t +1 =
R | s t = · , a = R )
= 1 , P ( s t +1 =
L | s t = · , a = R ) = 0 ,
P ( s t +1 =
R | s t = · , a = L ) = 0 , P ( s t +1 =
L | s t = · , a = L ) = 1 .
We prove that in Example 1, the regulator in an AMFCE gives recommendations as follows (see the detailed proof in Appendix C).
First, a random variable z is sampled from the correlated signal space Z = { 0 , 1 } with equal probability
ρ
( z = 0)
=
ρ
( z = 1) = 0 .
5 , and the regulator gives the action recommendation for each ﬁsh according to the policy π
( a = L | z = 0)
= 2 / 3 , π 0 ( a = R | z = 0) = 1 / 3 , π 0 ( a = L | z = 1) = 1 / 3 , π 0 ( a = R | z
= 1)
= 2 / 3 .
Then ﬁsh has no incentive to deviate from the recommendation, so an AMFCE is achieved.
It is worth noting that the above AMFCE solution is not a classic MFNE, because there are only three MFNEs which are shown in Table 1.
3. P ROPERTIES OF AMFCE
This section focuses on the properties of AMFCE, including the conditions to guarantee the existence and its relationship to classic MFNE.
In order to provide the existence of AMFCE solutions, we deﬁne the best response operator
[Equation]
(cid:13) , where the expecation is taken with respect to z t ∼ ρ t ( · ) , s t ∼ P ( ·| s t − 1 , a t − 1 , μ t − 1 ) , a t ∼ π (cid:6) t ( ·| s t , z t ) , μ t =
Φ( μ t − 1 , π t − 1 , z t − 1 ) .
Unless otherwise stated, the expectation E πππ,ρρρ is taken with respect to z t ∼ ρ t ( · ) , s t ∼ P ( ·| s t − 1 , a t − 1 , μ t − 1 ) , a t ∼ π t ( ·| s t , z t ) , μ t =
Φ( μ t − 1 , π t − 1 , z t − 1 ) .
Then the existence of the solution will be derived using Kakutani’s ﬁxed point theorem (Kakutani, 1941) with the operator BR .
We next provide a sufﬁcient condition for the existence of AMFCE.
Theorem 1.
If the functions r ( s, a, μ ) and P ( s (cid:6) | s, a, μ ) are bounded and continuous with respect to μ , there exists an AMFCE solution.
The AMFCE is a more general equilibrium concept than MFNE, which is illustrated in corollary 1.
Corollary 1.
If ( π , μ ) is an MFNE, then it leads to an AMFCE solution ( π ,ρρρ ) with |Z|
= 1 and ρ t ( z ) = 1 for all t ∈ T where z ∈ Z is the single element in the signal space.
The proof is deferred to Appendix D.3.
This proposition implies that the MFNE is a subset of
AMFCE.
The example in Example 1 shows that AMFCE may not be an MFNE.
3. M AXIMUM ENTROPY M EAN F IELD C ORRELATED E QUILIBRIUM
Similar to the classic MFG setting, there may be multiple AMFCEs in our setting.
Consequently,
AMFCE is facing the equilibrium selection issue.
One of the commonly used selection criteria is maximum entropy.
For example, maximum entropy has been introduced to select correlated equilibrium in the normal form game (Ortiz et al., 2007) and
Markov game
(Ziebart et al., 2011).
We integrate the maximum entropy principle into the AMFCE as follows.
Deﬁnition 5.
The maximum entropy AMFCE (MaxEnt-AMFCE) is the one that maximizes the entropy ( π ∗ ,ρρρ ∗ )
= arg max ( π ,ρρρ
) ∈ Π AMFCE H ( πππ,ρρρ ) , with H ( πππ,ρρρ ) = (cid:3) Tt =
E πππ,ρρρ [ − log( π t ( a t | s t , z t ) ρ t ( z t ))] , Π AMFCE the set of all AMFCE solutions.
The MaxEnt-AMFCE can avoid the equilibrium selection problem as it is unique under certain conditions.
Denote Δ( πππ,ρρρ ) = max u,s,t Δ t ( s, μ t , u ; πππ,ρρρ ) , where μ t =
Φ( μ t − 1 , π t − 1 , z t − 1 ) .
Corollary 2.
MaxEnt-AMFCE is a unique equilibrium solution if Δ( πππ,ρρρ ) is convex w.r.t. ( πππ,ρρρ ) .
The proof is deferred to Appendix D.4.
Directly optimizing the entropy is difﬁcult because the policy π t and ρ are coupled.
So we decouple this term by the following proposition.
Proposition 1.
The entropy can be decoupled: H ( πππ,ρρρ )
= (cid:3)
Tt =
[ H ( ρ t ) + E πππ,ρρρ H ( π t
| s t , z t )] .
H ( ρ t ) is the entropy of the correlation device, and
H ( π t | s t , z t ) = − (cid:3) a t ∈A π t ( a t | s t , z t ) log( π t ( a t | s t , z t )) is the entropy of π t ( ·| s t , z t ) .
(See proof in Appendix D.5).
I MITATION L EARNING FOR M EAN F IELD G AME
This section proposes a new framework based on imitation learning to recover AMFCE from collected expert demonstrations.
To avoid the equilibrium selection problem, we choose the MaxEnt-AMFCE solution introduced in Section 3.3.
To emphasize the role of unknown reward function in imitation learning, we use MFRL( r,ρρρ ) to denote the policy of MaxEnt-AMFCE under the reward function r and correlation device ρρρ :
[Equation]
The temperature constant α ≥ 0 is to control the entropy.
The constraint on the AMFCE set makes the optimization problem (6) challenging.
To address this, we provide an equivalent formulation in
Proposition 2 and derive a Lagrangian reformulation of (6).
4. C ORRELATED M EAN F IELD I MITATION L EARNING
We denote J ( πππ,ρρρ ) =
E (cid:12)(cid:3) Tt =
γ t r ( s t , a t , μ t ) (cid:13) , and R ( a 0: T ,πππ,ρρρ ) as the margin of expected return between choosing actions a 0: T := { a t } t ∈T and policy πππ under the correlation device ρρρ : R ( a 0: T ,πππ,ρρρ )
[Equation]
E
[Equation]
− J ( πππ,ρρρ ) , where the expectation is taken with respect to z t ∼ ρ t ( · ) , s t ∼ P ( ·| a t − 1 , s t − 1 , μ t − 1 ) .
And μ t =
Φ( μ t − 1 , π t − 1 , z t − 1 ) .
Then we can get an equivalent constraint of AMFCE.
Proposition 2.
( πππ,ρρρ ) is an AMFCE solution if and only if R ( a 0: T ,πππ,ρρρ ) ≤ 0 , ∀ a 0:
T ∈ A T .
The proof is deferred to Appendix D.6.
Compared to the original formulation (6), it is easier to work with a dual representation without constraints:
[Equation]
s t , a t , μ t ) (cid:5) − J ( πππ,ρρρ ) (cid:15) − α
T (cid:6) t =
E H ( π t | s t , z t ) (7) where D E is a set of action-signal sequence τ k = { a 0 , z 0 , a 1 , z 1 , a 2 , z 2 , · · · , a T , z T } .
We show that (7) captures the difference of expected returns between two policies by selecting λ as follows.
Theorem 2.
For policy πππ and correlation device ρρρ , let λ ∗ πππ ( τ k ) =
(cid:16) Tt =
ρ t ( z t ) π ∗ t ( a t | s t , z t ) be the probability of generating the sequence τ k if the individual policy is πππ ∗ .
Then we have L ( πππ,ρρρ, λ ∗ πππ , r ) =
E [ (cid:3) Tt =
γ t r ( s t , a t , μ t )] − J ( πππ,ρρρ ) − α (cid:3)
Tt =
E H ( π t | s t , z t ) , where the expectation is taken with respect to z t ∼ ρ t ( · ) , s t ∼ P ( ·| s t − 1 , a t − 1 , μ t − 1 ) , a t ∼ π ∗ t ( ·| s t , z t ) , μ t =
Φ( μ t − 1 , π t − 1 , z t − 1 ) .
The proof of Theorem 2 is deferred to Appendix D.7.
In the setting of imitation learning, the reward signal is not accessible.
To construct a suitable reward function rationalizing the expert policy, we need to deﬁne a suitable AMFCE inverse reinforcement learning (AMFCE-IRL) operator which designs a reward to maximize the margin of expected return between expert policy and the other policies:
[Equation]
Data: Expert trajectories D E
= { s 0 , z 0 , a 0 , s 1 , z 1 , a 1 , . . .
s T , z T , a T } Initial mean ﬁeld μ 0 , The weight of gradient penalty β
Result: Policy πππ θ , correlation device πππ φ
Initialization the parameter θ of policy πππ θ and the parameter φ of correlation device ρρρ φ ; for each iteration do
Obtain trajectories from ( π ,ρρρ ) by the process:
[Equation]
Approximate μ t with the signature of ˆ μ t = S ( { z i } ti =0 ) using (11); for i in { 0 , 1 , 2 , . . . } do
Update ω to increase the objective
[Equation]
endfor t in { 0 , 1 , 2 , . . .
} do
Update θ by SAC with small step size:
[Equation]
where the expectation is taken with respect to s 0 ∼ μ 0 , a t ∼ π θ ( ·| s t , z t ) , s t +1 ∼ P ( · | s t , μ [Equation] [Equation] z t ∼ ρ
[Equation]
Update φ with (10); end end ( πππ E ,
ρρρ E ) ∈
Π MaxEnt-AMFCE is the MaxEnt-AMFCE from which expert demonstrations are sampled.
We choose a special regularizer (Ho & Ermon, 2016):
[Equation]
if r
[Equation]
[Equation]
otherwise , where g ( x ) = (cid:20) x − log (1 − e − x ) if x >+ ∞ otherwise
After getting the reward function ˜ r = AMFCR-IRL( πππ E ,ρρρ E ) , we can characterize the AMFCE policy MFRL(˜ r,ρρρ E ) with the learned ˜ r .
Proposition 3.
The policy πππ learned on the reward function recovered by AMFCE-IRL can be characterized as follows:
[Equation]
where the expectation is taken with respect to z t ∼ ρ E t ( · ) , s t ∼ P ( ·| s t − 1 , a t −
[Equation]
, μ
[Equation]
− 1 ) , a t ∼ π t ( ·| s t , z t ) , μ t =
Φ( μ t − 1 , π Et − 1 , z t − 1 ) .
The objective to recover MaxEnt-AMFCE is deﬁned as:
[Equation]
where D ω is the discriminator network parameterized with ω , with input ( s t , a t , μ t ) and output a real number in (0 , 1] .
The ﬁrst expectation is taken with respect to z t ∼ ρ Et ( · ) , s t ∼ P ( ·| s t − 1 , a t − 1 , μ t − 1 ) , a t ∼ π t ( ·| s t , z t ) , μ t =
Φ( μ t − 1 , π Et − 1 , z t − 1 ) .
The proof is deferred to Appendix D.8.
In theory, we assume that neural network D ω has the capacity to approximate reward.
Under this assumption, the AMFCE ( πππ E ,ρρρ E ) could be recovered by optimizing the objective (9).
Simply applying GAIL to solve AMFCE cannot recover ρρρ E , so we derive ρρρ using a gradient descent method (with proof in Appendix D.9): [Table] [Table]
Proposition 4.
If ρρρ φ is parameterized with φ , the gradient to optimize φ given state s is
[Equation]
Now we propose the imitation learning algorithm for AMFCE (Algorithm 1).
This algorithm can recover AMFCE which may not have the maximum entropy if we set α = 0 .
4. R EPRESENTATION OF THE M EAN F IELD I NFORMATION
As the mean ﬁeld appears in the input of discriminator D ω ( s, a, μ ) in (9), it is necessary to ﬁnd an efﬁcient way to represent the mean ﬁeld information.
Since the evolution of mean ﬁeld is deterministic if the sequence of correlated signals and the initial mean ﬁeld μ 0 are given.
Therefore, we use the signature from rough path theory (Kidger & Lyons, 2021; Min & Hu, 2021) to represent the features of the signal sequence z
and hence to characterize the mean-ﬁeld ﬂow μμμ .
It is worth noting that the signature has been recently applied to the ﬁeld of machine learning to extract characteristic features of sequential data in a non-parametric fashion (Min & Ichiba, 2020; Ni et al., 2020).
The use of signatures to encode historical information avoids heavy computational load which often suffered in tasks like training recurrent neural networks.
In addition, the training stability can be signiﬁcantly enhanced since the mapping is invariant.
Deﬁnition 6.
Let x = { x 1 , . . .
, x L } with x i ∈ R d , for all i and L ≥ 2 .
Denote f : [0 , 1] → R d to be the continuous piecewise afﬁne function such that f ( i − 1 L − 1 ) =
x i , ∀ i ∈ { 1 , 2 , . . .
, L } .
[Equation]
where M n = (cid:21) s<
s 1 < ··· <s n <t d f d t ( s 1 ) ⊗· · ·⊗ d f d t ( s n )d t 1 · · · d t n .
The signature of the path x is deﬁned to be S ( f ) 0 , 1 , denoted as S ( x ) .
Signature of sequential data includes inﬁnite terms as shown in the (11), but fortunately, terms M n enjoy factorial decay.
In practice we select the ﬁrst n terms of the signature without losing crucial information of the data (Kidger et al., 2019).
E XPERIMENTS
We evaluate the effectiveness of our algorithm in four environments: Sequential Squeeze, RPS,
Flock, and Trafﬁc Flow Prediction.
We compare our CMFIL framework with MFIRL (Chen et al., 2021), as it is so far the only method to solve MFNE without requiring the knowledge on the reward.
Since MFIRL does not consider correlated signals, we regard the correlated signal as an extension of the global state for their framework.
We also compare CMFIL with MaxEnt ICE, smoothed multinomial distribution over the joint actions and logistic regression (Waugh et al., 2013).
As
[Figure] MaxEnt ICE is designed to recover correlated equilibrium in matrix game, we only compare CMFIL with MaxEnt ICE on tasks that can be reduced to matrix game, such as RPS and Sequential Squeeze with T = { 0 , 1 } .
We use the log loss, E a ∼ π ( ·| s,z ) [ − log(ˆ π ( a | s, z ))] , to mearsure the difference over recovered policy ˆ π and ground truth π .
The Appendix F contains more details.
We evaluate CMFIL on several tasks:
Sequential Squeeze (Squeeze for short), Rock-Paper-Scissors (RPS), Flock and a real-world trafﬁc ﬂow prediction task.
The ﬁrst three experiments are numerical experiments.
The trafﬁc ﬂow prediction task is to predict the trafﬁc ﬂow a complex trafﬁc network based on the real world data.
Details are presented in the Appendix E.
Squeeze: Sequential Squeeze is a game with multi-steps.
The purpose to implement this game is to verify the ability to recover expert policy through demonstrations sampled from a multi-step game.
The learning curve is shown in the Fig.3, and the results are shown in Table 2.
The example of Ocean
Ranch in Example 1 is a special case of Sequential Squeeze, where the horizon equals to 2.
RPS:
This experiment is a traditional mean ﬁeld game task (Chen et al., 2021; Cui & Koeppl, 2021).
The demonstrations are sampled from MFNE, and the cardinality of the correlated signal set is one.
We use RPS to verify that the algorithm proposed can recover the expert demonstrations sampled from MFNE, which also supports the results in Corollary 1.
Flock:
The experiment is based on the movement of ﬁsh (Perrin et al., 2021).
In nature, ﬁsh spontaneously aligns velocity according to the overall movement of the ﬁsh school, so that the ﬁnal ﬁsh school forms a stable movement velocity.
The video provided shows the convergence process ( https://sites.google.com/view/mean-field-imitation-learning/ ).
Trafﬁc Flow Prediction: In the Trafﬁc Flow Prediction task, we use the trafﬁc data of London from
Uber Movement.
The goal of this experiment is to predict the trafﬁc ﬂow of a trafﬁc network (with six locations) in real-world.
Given the large-scale and high-complexity of this task, we compare
CMFIL and MFIRL under this task to test their scalability.
The results for numerical tasks are shown in Table 2.
CMFIL is better than other methods in general.
Supervised learning methods such as logistic regression and smoothed multinomial distribution easily overﬁt.
They may outperform CMFIL in some metrics but suffer from a higher loss than CMFIL in general.
MFIRL shows larger deviations and higher loss than CMFIL in Table 2 and Table 3.
The reason is that MFIRL can not recover AMFCE, and it can not handle correlated signals properly.
Although we have regarded correlated signal as an extension of state.
The reward recovered by
MFIRL is biased because the ground truth reward is independent of the correlated signal.
Furthermore,
CMFIL adds a regularizer ψ for the reward function to avoid overﬁtting, so it also outperforms MFIRL in RPS in which expert demonstrations are sampled from MFNE.
MaxEnt ICE also performs poorly because it has a limited reward function class by assuming a linear reward structure.
Figure 1 shows that CMFIL can recover the correlation device with a fast convergence speed.