D EEP M EAN F IELD G AMES FOR L EARNING O PTIMAL B EHAVIOR P OLICY OF L ARGE P OPULATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
We consider the problem of representing a large population’s behavior policy that drives the evolution of the population distribution over a discrete state space.
A discrete time mean ﬁeld game (MFG) is motivated as an interpretable model founded on game theory for understanding the aggregate effect of individual actions and predicting the temporal evolution of population distributions.
We achieve a synthesis of MFG and Markov decision processes (MDP) by showing that a special MFG is reducible to an MDP.
This enables us to broaden the scope of mean ﬁeld game theory and infer MFG models of large real-world systems via deep inverse reinforcement learning.
Our method learns both the reward function and forward dynamics of an MFG from real data, and we report the ﬁrst empirical test of a mean ﬁeld game model of a real-world social media population.
I NTRODUCTION
Nothing takes place in the world whose meaning is not that of some maximum or minimum.
(Leonhard Euler)
Major global events shaped by large populations in social media, such as the Arab Spring, the Black Lives Matter movement, and the fake news controversy during the 2016 U.S. presidential election, provide signiﬁcant impetus for devising new models that account for macroscopic population behavior resulting from the aggregate decisions and actions taken by all individuals (Howard et al., 2011; Anderson & Hitlin, 2016; Silverman, 2016).
Just as physical systems behave according to the principle of least action, to which Euler’s statement alludes, population behavior consists of individual actions that may be optimal with respect to some objective.
The increasing usage of social media in modern societies lends plausibility to this hypothesis (Perrin, 2015), since the availability of information enables individuals to plan and act based on their observations of the global population state.
For example, a population’s behavior directly affects the ranking of a set of trending topics on social media, represented by the global population distribution over topics, while each user’s observation of this global state inﬂuences their choice of the next topic in which to participate, thereby contributing to future population behavior (Twitter, 2017).
In general, this feedback may be present in any system where the distribution of a large population over a state space is observable (or partially observable) by each individual, whose behavior policy generates actions given such observations.
This motivates multiple criteria for a model of population behavior:
The model captures the dependency between population distribution and their behavior policy.
It represents observed individual behavior as optimal for some implicit reward.
It enables prediction of future distribution over a state space given measurements at previous times, and can be learned from real data.
We present a mean ﬁeld game (MFG) approach to address the modeling and prediction criteria.
Mean ﬁeld games originated as a branch of game theory that provides tractable models of large agent populations, by considering the limit of N -player games as N tends to inﬁnity (Lasry & Lions, 2007).
In this limit, an agent population is represented via their distribution over a state space, and each agent’s optimal strategy is informed by a reward that is a function of the population distribution and their aggregate actions.
The stochastic differential equations that characterize MFG can be specialized to many settings: optimal production rate of exhaustible resources such as oil among many producers (Gu´eant et al., 2011); optimizing between conformity to popular opinion and consistency with one’s initial position in opinion networks (Bauso et al., 2016); and the transition between competing technologies with economy of scale (Lachapelle et al., 2010).
Representing agents as a distribution means that MFG is scalable to arbitrary population sizes, enabling it to simulate real-world phenomenon such as the Mexican wave in stadiums (Gu´eant et al., 2011).
As the model detailed in Section 3 will show, MFG naturally addresses the modeling criteria in our problem context by overcoming limitations of alternative predictive methods.
For example, time series analysis builds predictive models from data, but these models may not provide insight into the motivations that produce a population’s behavior policy, since they do not consider the behavior as the result of optimization of a reward function.
Alternatively, methods that employ the underlying population network structure have assumed that nodes are only inﬂuenced by a local neighborhood, do not include a representation of a global state, and may face difﬁculty in explaining events as the result of uncontrolled implicit optimization (Farajtabar et al., 2015; De et al., 2016).
MFG is unique as a descriptive model whose solution tells us how a system naturally behaves according to its underlying optimal control policy.
This is the essential insight that enables us to draw a connection with the framework of Markov decision processes (MDP) and reinforcement learning (RL) (Sutton & Barto, 1998).
The crucial difference from a traditional MDP viewpoint is that we frame the problem as MFG model inference via MDP policy optimization : we use the MFG model to describe natural system behavior by solving an associated MDP, without imposing any control on the system.
MFG offers a computationally tractable framework for adapting inverse reinforcement learning (IRL) methods (Ng & Russell, 2000; Ziebart et al., 2008; Finn et al., 2016), with ﬂexible neural networks as function approximators, to learn complex reward functions that explain behavior of arbitrarily large populations.
In the other direction, RL enables us to devise a data-driven method for solving an MFG model of a real-world system.
While research on the theory of MFG has progressed rapidly in recent years, with some examples of numerical simulation of synthetic toy problems, there is a conspicuous absence of scalable methods for empirical validation (Lachapelle et al., 2010; Achdou et al., 2012; Bauso et al., 2016).
Therefore, while we show how MFG is well-suited for the speciﬁc problem of modeling population behavior, we also demonstrate a general data-driven approach to MFG inference via a synthesis of MFG and MDP.
Our main contributions are the following.
We propose a data-driven approach to learn an MFG model along with its reward function, showing that research in MFG need not be conﬁned to toy problems with artiﬁcial reward functions.
Speciﬁcally, we derive a discrete time graph-state MFG from general MFG and provide detailed interpretation in a real-world setting (Section 3).
Then we prove that a special case can be reduced to an MDP and show that ﬁnding an optimal policy and reward function in the MDP is equivalent to inference of the MFG model (Section 4).
Using our approach, we empirically validate an MFG model of a population’s activity distribution on social media (Section 5).
The learned MFG model shows signiﬁcantly better predictive performance compared to baselines and offers insights on population behavior.
Our synthesis of MFG with MDP has potential to open new research directions for both ﬁelds.
R ELATED WORK
Mean ﬁeld games originated in the work of Lasry & Lions (2007), and independently as stochastic dynamic games in Huang et al.
(2006), both of which proposed mean ﬁeld problems in the form of differential equations for modeling problems in economics and analyzed the existence and uniqueness of solutions.
Gu´eant et al.
(2011) provided a survey of MFG models and discussed various applications in continuous time and space, such as a model of population distribution that informed the choice of application in our work.
Even though the MFG framework is agnostic towards the choice of cost function (i.e. negative reward), prior work make strong assumptions on the cost in order to attain analytic solutions.
We take a view that the dynamics of any game is heavily impacted by the reward function, and hence we propose methods to learn the MFG reward function from data.
Discretization of MFGs in time and space have been proposed (Gomes et al., 2010; Achdou et al., 2012; Gu´eant, 2015), serving as the starting point for our model of population distribution over discrete topics; while these early work analyze solution properties and lack empirical veriﬁcation, we focus on algorithms for attaining solutions in real-world settings.
Related to our application case, prior work by Bauso et al.
(2016) analyzed the evolution of opinion dynamics in multi-population environments, but they imposed a Gaussian density assumption on the initial population distribution and restrictions on agent actions, both of which limit the generality of the model and are not assumed in our work.
There is a collection of work on numerical ﬁnite-difference methods for solving continuous mean ﬁeld games (Achdou et al., 2012; Lachapelle et al., 2010; Carlini & Silva, 2014).
These methods involve forward-backward or Newton iterations that are sensitive to initialization and have inherent computational challenges for large real-valued state and action spaces, which limit these methods to toy problems and cannot be scaled to real-world problems.
We overcome these limitations by showing how the MFG framework enables adaptation of RL algorithms that have been successful for problems involving unknown reward functions in large real-world domains.
In reinforcement learning, there are numerous value- and policy-based algorithms employing deep neural networks as function approximators for solving MDPs with large state and action spaces (Mnih et al., 2013; Silver et al., 2014; Lillicrap et al., 2015).
Even though there are generalizations to multi-agent settings (Hu et al., 1998; Littman, 2001; Lowe et al., 2017), the MDP and Markov game frameworks do not easily suggest how to represent systems involving thousands of interacting agents whose actions induce an optimal trajectory through time.
In our work, mean ﬁeld game theory is the key to framing the modeling problem such that RL can be applied.
Methods in unknown MDP estimation and inverse reinforcement learning aim to learn an optimal policy while estimating an unknown quantity of the MDP, such as the transition law (Burnetas &
Katehakis, 1997), secondary parameters (Budhiraja et al., 2012), and the reward function (Ng & Russell, 2000). The maximum entropy IRL framework has proved successful at learning reward functions from expert demonstrations (Ziebart et al., 2008; Boularias et al., 2011; Kalakrishnan et al., 2013).
This probabilistic framework can be augmented with deep neural networks for learning complex reward functions from demonstration samples (Wulfmeier et al., 2015; Finn et al., 2016).
Our MFG model enables us to extend the sample-based IRL algorithm in Finn et al.
(2016) to the problem of learning a reward function under which a large population’s behavior is optimal,
and we employ a neural network to process MFG states and actions efﬁciently.
M EAN FIELD GAMES
We begin with an overview of a continuous-time mean ﬁeld games over graphs, and derive a general discrete-time graph-state MFG (Gu ´ eant, 2015).
Then we give a detailed presentation of a discretetime MFG over a complete graph, which will be the focus for the rest of this paper.
3. M EAN FIELD GAMES ON GRAPHS
Let G =
( V , E ) be a directed graph, where the vertex set V = { 1 , . . .
, d } represents d possible states of each agent, and E ⊆ V×V is the edge set consisting of all possible direct transition between states (i.e., a agent can hop from i to j only if ( i, j ) ∈ E ).
For each node i ∈ V , deﬁne V
+ i :=
{ j : ( j, i ) ∈
E} , V − i :=
{ j : ( i, j ) ∈ E } , and ¯ V
+ i :=
V + i ∪ { i } and ¯ V − i :
= V − i ∪ { i } .
Let π i ( t ) be the density (proportion) of agent population in state i at time t , and π ( t ) :
= ( π 1 ( t ) , . . .
, π d ( t )) .
Population dynamics are generated by right stochastic matrices P ( t ) ∈ S ( G ) , where S ( G ) :
= S 1 ( G ) × · · · ×
S d ( G ) and each row P i ( t ) belongs to S
i ( G ) :
= { p ∈ ∆ d − 1 | supp ( p ) ⊂ ¯ V − i } where ∆ d − 1 is the simplex in R d .
Moreover, we have a value function V i ( t ) of state i at time t , and a reward function r i ( π ( t ) , P i ( t )) 1 , quantifying the instantaneous reward for agents in state i taking transitions with probability P i ( t ) when the current distribution is π ( t ) .
We are mainly interested in a discrete time graph state MFG, which is derived from a continuous time MFG by the following proposition.
Appendix A provides a derivation from the continuous time MFG.
Proposition 1.
Under a semi-implicit discretization scheme with unit time step labeled by n , the backward Hamilton-Jacobi-Bellman (HJB) equation and the forward Fokker-Planck equation for each i ∈ { 1 , . . .
, d } and n = 0 , . . .
, N − 1 in a discrete time graph state MFG are given by:
[Equation]
3. D ISCRETE TIME MFG OVER COMPLETE GRAPH
Proposition 1 shows that a discrete time MFG given in Gomes et al.
(2010) can be seen as a special case of a discrete time graph state MFG with a complete graph (such that S ( G ) =
∆ d − 1 ×· · ·× ∆ d −( d of ∆ d − 1 )).
We focus on the complete graph in this paper, as the methodology can be readily applied to general directed graphs.
While Section 4 will show a connection between MFG and MDP, we note here that a “state” in the MFG sense is a node in V and not an MDP state.
2 We now interpret the model using the example of evolution of user activity distribution over topics on social media, to provide intuition and set the context for our real-world experiments in Section 5.
Independent of any particular interpretation, the MFG approach is generally applicable to any problem where population size vastly outnumbers a set of discrete states.
• Population distribution π n ∈ ∆ d − 1 for n = 0 , . .
, N − 1 .
Each π n is a discrete probability distribution over d topics, where π ni is the fraction of people who posted on topic i at time n .
Although a person may participate in more than one topic within a time interval, normalization can be enforced by a small time discretization or by using a notion of “effective population size”, deﬁned as population size multiplied by the max participation count of any person during any time interval.
π 0 is a given initial distribution.
• Transition matrix P n ∈ S ( G ) .
P n ij is the probability of people in topic i switching to topic j at time n , so we refer to P ni as the action of people in topic i .
P n generates the forward equation
[Equation]
• Reward r i ( π n , P ni ) :
= (cid:80) dj
=1 P nij r ij ( π n , P ni ) , for i ∈ { 1 , . . .
, d } .
This is the reward received by people in topic i who choose action P ni at time n , when the distribution is π n .
In contrast to previous work, we learn the reward function from data (Section 4.1).
The only assumption we make is that reward for i depends only on P ni , not on the entire P n .
This is a causality assumption that actions by people in j (cid:54) =
i have no instantaneous effect on the reward for people in topic i .•
Value function V n ∈ R d .
V ni is the expected maximum total reward of being in topic i at time n .
A terminal value V N − 1 is given, which we set to zero to avoid making any assumption on the problem structure beyond what is contained in the learned reward function.
• Average reward e i ( π, P, V ) , for i ∈ { 1 , . . .
, d } and V ∈ R d and P ∈ S ( G ) .
This is the average reward received by agents at topic i when the current distribution is π , action P is chosen, and the subsequent expected maximum total reward is V .
It is deﬁned as:
[Equation]
Intuitively, agents want to act optimally in order to maximize their expected total average reward.
For P ∈ S ( G ) and a vector q ∈ S i ( G ) , deﬁne P ( P, i, q ) to be the matrix equal to P , except with the i -th row replaced by q .
Then a Nash maximizer is deﬁned as follows:
Deﬁnition 1.
A right stochastic matrix P ∈ S ( G ) is a Nash maximizer of e ( π, P, V ) if, given a ﬁxed π ∈ ∆ d − 1 and a ﬁxed V ∈ R d , there is
[Equation]
P
[Equation]
for any i ∈ { 1 , . . .
, d } and any q ∈ S
i ( G ) .
The rows of P form a Nash equilibrium set of actions, since for any topic i , the people in topic i cannot increase their reward by unilaterally switching their action from P i to any q .
Under Deﬁnition 1, the value function of each topic i at each time n satisﬁes the optimality criteria:
[Equation]
A solution of the MFG is a sequence of pairs { ( π n , V n ) } n =0 ,...,N satisfying optimality criteria (6) and forward equation (3).
I NFERENCE OF MFG VIA MDP OPTIMIZATION
A Markov decision process is a well-known framework for optimization problems.
We focus on the discrete time MFG in Section 3.2 and prove a reduction to a single-agent ﬁnite-horizon deterministic
MDP, whose state trajectory under an optimal policy coincides with the forward evolution of the
MFG.
This leads to the essential insight that solving the optimization problem of a single-agent
MDP is equivalent to solving an MFG that describes population behavior.
This connection will enable us to apply efﬁcient inverse RL methods, using measured population trajectories, to learn an MFG model along with its reward function in Section 4.1.
The MDP is constructed as follows:
Deﬁnition 2.
A single-agent ﬁnite-horizon deterministic MDP for a discrete time MFG over a complete graph is deﬁned as:
• States: π n ∈ ∆ d − 1 , the population distribution at time n .
• Actions: P n ∈ S ( G ) , the transition probability matrix at time n .
• Reward: R ( π n , P n ) :
= (cid:80) di
=1 π ni (cid:80) dj
=1 P nij r ij ( π n , P ni ) • Finite-horizon state transition, given by Eq (3): ∀ n ∈ { 0 , . . .
, N − 1 } : π n +1 j
[Equation]
(cid:80) d i =1 P n ij π n i .
Theorem 2.
The value function of a solution to the discrete time MFG over a complete graph deﬁned by optimality criteria (6) and forward equation (3) is a solution to the Bellman optimality equation of the MDP in Deﬁnition 2.
Proof.
Since r ij depends on P n only through row P ni , optimality criteria 6 can be written as
[Equation]
We now deﬁne V ∗ ( π n ) as follows and show that it is the value function of the constructed MDP in
Deﬁnition 2 by verifying that it satisﬁes the Bellman optimality equation:
[Equation]
which is the Bellman optimality equation for the MDP in Deﬁnition 2.
Corollary 1.
Given a start state π 0 , the state trajectory under the optimal policy of the MDP in
Deﬁnition 2 is equivalent to the forward evolution part of the solution to the MFG.
Proof.
Under the optimal policy, equations 11 and 8 are satisﬁed, which means the matrix P generated by the optimal policy at any state π n is the Nash maximizer matrix.
Therefore, the state trajectory { π n } n =0 ,...,N − 1 is the forward part of the MFG solution.
4. R EINFORCEMENT LEARNING SOLUTION FOR MFG
MFG provides a general framework for addressing the problem of modeling population dynamics, while the new connection between MFG and MDP enables us to apply inverse RL algorithms to solve the MDP in Deﬁnition 2 with unknown reward.
In contrast to previous MFG research, most of which impose reward functions that are quadratic in actions and logarithmic in the state distribution
(Gu´eant, 2009; Lachapelle et al., 2010; Bauso et al., 2016), we learn a reward function using demonstration trajectories measured from actual population behavior, to attain a succinct and data-driven representation of the motivation behind population dynamics.
We leverage the MFG forward dynamics (Eq 3) in a sample-based IRL method based on the maximum entropy IRL framework (Ziebart et al., 2008).
From this probabilistic viewpoint, we minimize the relative entropy between a probability distribution p ( τ ) over a space of trajectories T :=
{ τ i } i and a distribution q ( τ ) from which demonstrated expert trajectories are generated (Boularias et al., 2011).
This is related to a path integral IRL formulation, where the likelihood of measured optimal trajectories is evaluated only using trajectories generated from their local neighborhood, rather than uniformly over the whole trajectory space (Kalakrishnan et al., 2013).
Speciﬁcally, making no assumption on the true distribution of optimal demonstration other than matching of reward expectation, we posit that demonstration trajectories τ i = (
[Equation] [Equation]
, π N − 1 , P N − 1 ) i are sampled from the maximum entropy distribution (Jaynes, 1957):
[Equation]
(12) where R W ( τ ) = (cid:80) n R W ( π n , P n ) is the sum of reward of single state-action pairs over a trajectory τ , and W are the parameters of the reward function approximator (derivation in Appendix E).
Intuitively, this means that trajectories with higher reward are exponentially more likely to be sampled.
Given M sample trajectories τ j ∈ D samp from k distributions F 1 ( τ ) , . . .
, F k ( τ ) , an unbiased estimator of the partition function Z = (cid:82) exp(
R W ( τ )) dτ using multiple importance sampling is
ˆ Z := 1 M (cid:80) τ j z j exp( R W ( τ j ))
(Owen & Zhou, 2000), where importance weights are z j := (cid:2) 1 k (cid:80)
k F k ( τ j ) (cid:3) − 1 (derivation in Appendix F).
Each action matrix P is sampled from a stochastic policy F k ( P ; π, θ ) (overloading notation with F ( τ ) ), where π is the current state and θ the policy parameter.
The negative log likelihood of L demonstration trajectories τ i ∈ D demo is:
[Equation]
We build on Guided Cost Learning (GCL) in Finn et al.
(Alg 1) to learn a deep neural network approximation of R W ( π, P ) via stochastic gradient descent on L ( W ) , and learn a policy
F ( P ; π, θ ) using a simple actor-critic algorithm (Sutton & Barto, 1998).
In contrast to GCL, we employ a combination of convolutional neural nets and fully-connected layers to process both the action matrix P and state vector π efﬁciently in a single architecture (Appendix C), analogous to how Lillicrap et al.
(2015) handle image states in Atari games.
Due to our choice of policy parameterization (described below), we also set importance weights to unity for numerical stability.
These implementation choices result in successful learning of a reward representation (Fig 1).
Our forward MDP solver (Alg 2) performs gradient ascent on the policy’s expected start value E
[ v ( π 0 ) | F ( P ; π, θ )] w.r.t. θ , to ﬁnd successively better policies F k ( P ; π, θ ) .
We construct the joint distribution F ( P ; π, θ ) informed by domain knowledge about human population behavior on social media, but this does not reduce the generality of the MFG framework since it is straightforward to employ ﬂexible policy and value networks in a DDPG algorithm when intuition is not available (Silver et al., 2014; Lillicrap et al., 2015).
Our joint distribution is d instances of a d -dimensional
Dirichlet distribution, each parameterized by an α i ∈ R d + .
Each row P i is sampled from
[Equation]
) where B ( · ) is the Beta function and α ij is deﬁned using the softplus function α ij ( π, θ ) :
=
ln(1 + exp { θ ( π j − π i ) } ) , which is a monotonically increasing function of the population density difference π j − π i .
In practice, a constant scaling factor c ∈ R can be applied to α for variance reduction.
Finally, we let F ( P n ; π n , θ ) =
(cid:81) di
=1 f ( P ni ; α i ( π n , θ )) denote the parameterized policy, from which P n is sampled based on π n , and whose logarithmic gradient ∇ θ ln( F ) can be used in a policy gradient algorithm.
We learned an approximate value function ˆ V ( π ; w ) as a baseline for variance reduction, approximated as a linear combination of all polynomial features of π up to second order, with parameter w (Sutton et al., 2000).
0.4 Reward
D e n s i t y
Reward density for training demo and generated transitions Demo (train)
Generated (a) Reward densities on train set 0.
0.4 Reward 0.
D e n s i t y
Reward density for test demo and generated transitions Demo (test)
Generated (
[Figure]
densities on
[Figure]
set  2 Actions
S t a t e s
Reward of state-action pairs 0.
[Figure]
E XPERIMENTS
We demonstrate the effectiveness of our method with two sets of experiments: (i) recovery of an interpretable reward function and (ii) prediction of population trajectory over time.
Our experiment matches the discrete time mean ﬁeld game given in Section 3.2: we use data representing the activity of a Twitter population consisting of 406 users.
We model the evolution of the population distribution over d = 15 topics and N = 16 time steps (9am to midnight) each day for 27 days.
The sequence of state-action pairs { ( π n , P n ) } n =0 ,...,N − 1 measured on each day shall be called a demonstration trajectory .
Although the set of topics differ semantically each day, indexing topics in order of decreasing initial popularity sufﬁces for identifying the topic sets across all days.
As explained earlier, the MFG framework can model populations of arbitrarily large size, and we ﬁnd that our chosen population is sufﬁcient for extracting insights on population behavior.
For evaluating performance on trajectory prediction, we compare MFG with two baselines:
VAR.
Vector autoregression of order 18 trained on 21 demonstration trajectories.
RNN.
Recurrent neural network with a single fully-connected layer and rectiﬁer nonlinearity.
We use Jenson
Shanon Divergence (JSD) as metric to report all our results.
Appendix D provides comprehensive implementation details.
5. I NTERPRETATION OF REWARD FUNCTION
The learned reward is a step towards understanding population behavior from an optimization perspective. We evaluated the reward using four sets of state-action pairs acquired from: 1.
all train demo trajectories; 2. trajectories generated by the learned policy given initial states π 0 of train trajectories; 3.
all test demo trajectories; 4. trajectories generated by the learned policy given initial states π 0 of test trajectories.
We ﬁnd three distinct modes in the density of reward values for both the train group of sets 1 and 2 (Fig 1a) and the test group of sets 3 and 4 (Fig 1b).
Although we do not have access to a ground truth reward function, the low JSD values of 0.13 and 0.017 between reward distributions for demo and generated state-action pairs show generalizability of the learned reward function.
We further investigated the reward landscape with nine state-action pairs (Figure 1c), and ﬁnd that the mode with highest rewards is attained by pairing states that have large mass in topics having high initial popularity (S0) with action matrices that favor transition to topics with higher density (A0).
On the other hand, uniformly distributed state vectors (S2) attain the lowest rewards, while states with a small negative mass gradient from topic 1 to topic d (S1) attain medium rewards.
5. T RAJECTORY PREDICTION
To test the usefulness of the reward and MFG model, the learned policy was used with the forward equation to generate complete trajectories, given initial distributions.
Fig 2a (log scale) shows that MFG has 58% smaller error than VAR when evaluated on the JSD between generated and measured ﬁnal distributions JSD ( π N − 1 generated , π N − 1 measured ) , and 40% smaller error when evaluated on the average JSD over all hours in a day 1 N (cid:80) {+N − 1 n =
+}
[-N − 1 n =
-]
JSD ( π n generated , π n measured ) .
Both measures were averaged over
[Figure] [Figure]
M = 6 held-out test trajectories.
It is worth emphasizing that learning the MFG model required only the initial population distribution of each day in the training set (line 4 in Alg 2), while VAR and RNN used the distributions over all hours of each day.
MFG achieves better prediction performance even with fewer training samples, possibly because it is a more structured approximation of the true mechanism underlying population dynamics, in contrast to VAR and RNN that rely only on regression.
As shown by sample trajectories for topic 0 and 2 in Figures 3, and the average transition matrices in Figure 2b, MFG correctly represents the fact that the real population tends to congregate to topics with higher initial popularity (i.e. lower topic indices), and that the popularity of topicbecomes more dominant across time in each day.
The small real-world dataset size, and the fact that RNN mainly learns state transitions without accounting for actions, could be contributing factors to the lower performance of RNN.
We acknowledge that our design of policy parameterization, although informed by domain knowledge, introduced bias and resulted in noticeable differences between demonstration and generated transition matrices.
This can be addressed using deep policy and value networks, since the MFG framework is agnostic towards choice of policy representation.
5. I NSIGHTS
Under the assumption that measured behavior is optimal for the constructed MDP, the learned reward function favors states with large negative mass gradient in decreasing order of initial topic popularity, and transitions that increase this distribution imbalance. The model’s high prediction accuracy supports two approximate descriptions of population behavior: an equilibrium of individuals optimizing the MFG reward; equivalently, an optimal trajectory for the constructed MDP.
C ONCLUSION
We have motivated and demonstrated a data-driven method to solve a mean ﬁeld game model of population evolution, by proving a connection to Markov decision processes and building on methods in reinforcement learning.
Our method is scalable to arbitrarily large populations, because the MFG framework represents population density rather than individual agents, while the representations are linear in the number of MFG states and quadratic in the transition matrix.
Our experiments on real data show that MFG is a powerful framework for learning a reward and policy that can predict trajectories of a real world population more accurately than alternatives.
Even with a simple policy parameterization designed via some domain knowledge, our method attained superior performance on test data.
It motivates exploration of ﬂexible neural networks for more complex applications.
An interesting extension is to develop an efﬁcient method for solving the discrete time MFG in a more general setting, where the reward at each state i is coupled to the full population transition matrix.
Our work also opens the path to a variety of real-world applications, such as a synthesis of MFG with models of social networks at the level of individual connections to construct a more complete model of social dynamics, and mean ﬁeld models of interdependent systems that may display complex interactions via coupling through global states and reward functions.
A P ROOF OF P ROPOSTION
Given the deﬁnitions in Section 3.1, a mean ﬁeld game is deﬁned by a Hamilton-Jacobi-Bellman (HJB) equation evolving backwards in time and a Fokker-Planck equation evolving forward in time.
The continuous-time Hamilton-Jacobi-Bellman (HJB) equation on G is
[Equation]
where r i ( π, P i ) is the reward function, and V i ( t ) is the value function of state i at time t .
Note that the reward function r i ( π ( t ) , P i ( t )) is often presented as − c i ( π ( t ) , P i ( t )) for some cost function
c i ( π ( t ) , P i ( t )) in the MFG context, and similarly for V i ( t ) .
In addition, we set r i ( π ( t ) , P i ( t )) =
−∞ if P i ( t ) / ∈ S i ( G ) (i.e. P ( t ) must be a valid transition matrix).
For any ﬁxed π ( t ) , let H i ( π ( t ) , · ) be the Legendre transform of c
i ( π ( t ) , · ) deﬁned by
[Equation]
Then the HJB equation (15) is an analogue to the backward equation in mean ﬁeld games
[Equation]
17) where [ V j ( t )
− V i ( t )]
j ∈ ¯ V − i ∈ R | ¯ V − i | is the dual variable of P i .
We can discretize (15) using a semi-implicit scheme with unit time step labeled by n to obtain
[Equation]
Rearranging (18) yields the discrete time HJB equation over a graph (19)
[Equation]
[Equation]
The forward evolving Fokker-Planck equation for the continuous-time graph-state MFG is given by
[Equation]
20) where Q ji ( t ) =
∂
u i H
j
( π ( t ) , [ V k ( t ) − V j ( t )]
k ∈ ¯ V − j ) (21) where
∂ u i H j
( π, u ) is the partial derivative w.r.t.
the coordinate corresponding to the i -th index of the argument u ∈ R | ¯ V − j | .
We can set Q ji ( t ) = 0 for all ( j, i ) / ∈ E , so that Q ( t ) :=
[ Q ji ( t )] can be regarded as the d -by- d inﬁnitesimal generator matrix of states π ( t ) , and hence (20) can be written as π (cid:48)
( t ) =
π ( t ) Q ( t ) , where π ( t ) ∈
R
d is a row vector.
Then an Euler discretization of (20) with unit time step reduces to π n +1 − π n = π n Q n , which can be written as
[Equation]
where P nij :
= Q nij + δ ij .
If the graph G is complete, meaning E
= { ( i, j ) : 1 ≤ i, j ≤ d } , then the summation is taken over j = 1 , . . .
, d .
For ease of presentation, we only consider the complete graph in this paper, as all derivations can be carried out similarly for general directed graphs.
A solution of a mean ﬁeld game deﬁned by (19) and (22) is a collection of V ni and π ni for i = 0 , . . .
, d − 1 and n = 0 , . . . , N − 1 .
B A LGORITHMS
We learn a reward function and policy using an adaptation of GCL (Finn et al., 2016) in Alg 1 and a simple actor-critic Alg 2 (Sutton & Barto, 1998) as a forward RL solver.
1: procedure G UIDED COST LEARNING 2:
Initialize F 0 ( P ; π, θ ) as random policy and reward network weights W3: for iteration 1 to I do 4:
Generate sample trajectories D traj from F k ( P ; π, θ ) 5:
D samp ← D samp ∪ D traj 6: while (cid:12)(cid:12)
Avg π i ,P i ∼D demo ( R W t ( π i , P i ) − R W t − 1 ( π i , P i )) (cid:12)(cid:12) > dR do 7:
Sample demonstration ˆ D demo ⊂
D demo from expert demonstration 8:
Sample ˆ D samp ⊂ D samp 9:
W t +1 ← W t − (cid:15) ∇L ( W t ) using ˆ D demo and ˆ D samp 10: end while 11:
Run Alg 2 on D traj for improved F k + 12: end for 13: return Final reward function R W ( π, P ) and policy F ( P ; π, θ ) 14: end procedure
Input: Generative model F ( P ; π, θ ) , value function ˆ V ( π ; w ) , training data { π 0 } M days
Output: Policy parameter θ , value function parameter w 1: procedure ACTOR - CRITIC -MFG
( F, ˆ V , { π 0 } M days , β, ξ, R W ) 2: initialize θ and w 3: for episodes s = 1 , . .
, S do 4:
Sample initial distribution π 0 from { π 0 } M days 5: for time step n = 0 , . .
, N − 1 do 6:
Sample action P n ∼ F ( P ; π n , θ ) 7:
Generate π n +1 using Eq8:
Receive reward R W ( π n , P n ) 9:
δ ← R + ˆ V ( π n +1 ; w ) − ˆ V ( π n ; w ) 10: w ← w + ξδ ∇ w ˆ V ( π n ; w ) 11: θ ← θ + βδ ∇ θ log( F ( P ; π n , θ )) 12: end for 13: end for 14: end procedure
C R EWARD NETWORK
Our reward network uses two convolutional layers to process the 15 × 15 action matrix P , which is then ﬂattened and concatenated with the state vector π and processed by two fully-connected layers regularized with L1 and L2 penalties and dropout (probability 0.6).
The ﬁrst convolutional layer zero-pads the input into a 19 × 19 matrix and convolves one ﬁlter of kernel size 5 × 5 with stride 1 and applies a rectiﬁer nonlinearity.
The second convolutional layer zero-pads its input into a 17 × 17 matrix and convolves 2 ﬁlters of kernel size 3 × 3 with stride 1 and applies a rectiﬁer nonlinearity.
The fully connected layers have 8 and 4 hidden rectiﬁer units respectively, and the output is a single fully connected tanh unit.
All layers were initialized using the Xavier normal initializer in Tensorﬂow.
D E XPERIMENT DETAILS
By default, Twitter users in a certain geographical region primarily see the trending topics speciﬁc to that region (Twitter, 2017).
This experiment focused on the population and trending topics in the city of Atlanta in the U.S. state of Georgia.
First, a set of 406 active users were collected to form the ﬁxed population.
This was done by collecting a set of high-visibility accounts in Atlanta (e.g. the Atlanta Falcons team), gathering all Twitter users who follow these accounts, ﬁltering for those whose location was set to Atlanta, and ﬁltering for those who responded to least two trending topics within four days.
Data collection proceeded as follows for 27 days: at 9am of each day, a list of the top 14 trending topics on Twitter in Atlanta was recorded; for each hour until midnight, for each topic, the number of users who responded to the topic and the transition counts among topics within the past hour was recorded.
Whether or not a user responded to a topic was determined by checking for posts by the user containing unique words for that topic; the “hashtag” convention of trending topics on Twitter reduces the likelihood of false positives.
The hourly count of people who did not respond to any topic was recorded as the count for a “null topic”.
Although some users may respond to more than one topic within each hour, the data shows that this is negligible, and a shorter time interval can be used to reduce this effect.
The result of data collection is a set of trajectories, one trajectory per day, where each trajectory consists of hourly measurements of the population distribution over d =topics and their transition matrix over N = 16 hours.
The training set consists of trajectories { π 0 ,m , P 0 ,m , . .
, P N − 2 ,m , π N − 1 ,m } m =1 ,...,M over the ﬁrst M = 21 days.
MFG uses the initial distribution π 0 of each day, along with the transition equation of the constructed MDP and the policy F ( P ; π n , θ ) , to produce complete trajectories for training (Alglines 4,6,7). In contrast, VAR and RNN are supervised learning methods and they use all measured distributions.
RNN employs a simple recurrent unit with ReLU as nonlinear activation and weight matrix of dimension d × d .
VAR was implemented using the Statsmodels module in Python, with order 18 selected via random sub-sampling validation with validation set size 5 (Seabold & Perktold, 2010).
For prediction accuracy, all three methods were evaluated against data from 6 held-out test days.
Table 1 shows parameters of Alg 2 and 1.
[Table]
E M AXIMUM ENTROPY DISTRIBUTION
Given a ﬁnite set of trajectories { τ i } i , where each trajectory is a sequence of state-action pairs τ i =
( s i 1 , a i 1 , . . . , ) .
Suppose each trajectory τ i has an unknown probability p i .
The entropy of the probability distribution is H = − (cid:80)
i p i
ln( p i ) .
In the continuous case, we write the differential entropy:
[Equation]
where p ( · ) is the probability density we want to derive.
The constraints are:
[Equation]
The ﬁrst constraint says: the expected reward over all trajectories is equal to an empirical measurement µ r .
We write the Lagrangian L :
[Equation]
For L to be stationary, the Euler-Lagrange equation with integrand denoted by L says
[Equation]
since L does not depend on dpdτ .
Hence
[Equation]
where Z := (cid:82) e − λ 2 r ( τ ) dτ .
Then the constant λ 2 is determined by: µ r =
[Equation]
F M ULTIPLE IMPORTANCE SAMPLING
We show how multiple importance sampling (Owen & Zhou, 2000) can be used to estimate the partition function in the maximum entropy IRL framework.
The problem is to estimate Z := (cid:82) f ( x ) dx .
Let p 1 , . .
. , p m be m proposal distributions, with n j samples from the j -th proposal distribution, so that samples can be denoted X ij for i = 1 , . .
. , n j and j = 1 , . .
, m .
Let w j ( x ) for j = 1 , . . .
, m satisfy
[Equation]
Then deﬁne the estimator
[Equation]
Let S ( p j ) =
{ x | p j ( x ) > 0 } be the support of p j and S ( w j )
=
{ x | w j ( x ) > 0 } be the support of w j , and let them satisfy S ( w j )
⊂ S ( p j ) .
Under these assumptions:
[Equation]
In particular, choose
[Equation]
Then the estimate becomes
[Equation]
where n = (cid:80) mj =1 n j is the total count of samples.
Further assuming that samples are drawn uniformly from all proposal distributions, so that n j = n k = n/m for all j, k ∈ { 1 , . .
, m } , the expression for ˆ Z reduces to the form used in Eq 13:
[Equation]
G A COMPARISON OF MEAN FIELD GAMES AND MULTI - AGENT MDP S
In this section, we discuss the reason that the general MFG is not reducible to a collection of distinct single-agent MDPs, and also not equivalent to a multi-agent MDP.
Assume the case of a general reward function r ij ( π n , P n ) that depends on the full Nash maximizer matrix P n .
G. C OLLECTION OF SINGLE - AGENT MDP S
Consider each topic i as a separate entity associated with a value, rather than subsuming it into an average (as is the case in Section 4).
In order to assign a value to each topic, each tuple ( i, π n ) must be deﬁned as a state, which leads to the problem: since a state requires speciﬁcation of π n , and state transitions depend on the actions for all other topics, the action at each topic is not sufﬁcient for fully specifying the next state.
More formally, consider a value function on the state:
[Equation]
Superﬁcially, this resembles the Bellman optimality equation for the value function in a single-agent stochastic MDP, where s is a state, a is an action, R is an immediate reward, and P ( s (cid:48) | s, a ) is the probability of transition to state s (cid:48) from state s , given action a :
[Equation]
In the second summation, q j can be interpreted as a transition probability, conditioned on the fact that the current topic is i .
The action q selected in the state ( i, π n ) induces a stochastic transition to a next topic j , but the next distribution π n +1 is given by the deterministic forward equation
π n +
=
( P n ) T π n , where P n is the true Nash maximizer matrix.
This means that q j does not completely specify the next state ( j, π n +1 ) , and there is a formal difference between P ( s (cid:48) | s, a ) V ∗ ( s (cid:48) ) and q j V ( j, ( P n ) T π n ) .
Also notice that the Bellman equation sums over all possible next states s (cid:48) , but that is not the case for equation 23, because a full state speciﬁcation requires ( j, π ) .
G. M ULTI - AGENT MDP
The following explanation shows that an exact reduction from the MFG to a multi-agent MDP (i.e.
Markov game) is not possible.
A discrete state space discrete action space multi-agent MDP is deﬁned by d agents moving within a set S of environment states; a collection { A 1 , . . . ,
A d } of sets of actions, one for each agent; a transition function P ( s (cid:48) | s, a 1 , . .
. , a d ) giving the probability of the environment transitioning to state s (cid:48)
given that the current state is s and agents choose actions ¯ a (cid:44) ( a 1 , . .
. , a d ) ; a collection of reward functions { R i ( s, a 1 , . .
. , a d ) } i , one for each agent; and a discount factor γ in the inﬁnite horizon case.
One natural attempt to reduce the MFG to a Markov game is to let the set of π n (with appropriate discretization) be the state space, limit the set of actions to some discretization of the simplex, and consider each topic to be a single agent .
Now, the agent representing topic i is no longer identiﬁed with the set of people who selected topic i : topics have ﬁxed labels for all time, so an agent who accumulates reward for topic i must be associated only with topic i .
Therefore, the value function for agent i is deﬁned only in terms of itself, never depending on the value function of agents j (cid:54) =
i :
[Equation]
where µ := ( µ 1 , . .
. , µ d ) is a set of stationary policies of all agents.
However, recall that the MFG equation for V ni explicitly depends on V n +1 j of all topics j , which would require a different form such as the following:
[Equation]
where the last summation involves the value function V µk for all k .
This difference is exactly the reason that the MFG equations cannot be reduced to the equations of a standard Markov game.