A CTOR -C RITIC P ROVABLY F INDS N ASH E QUILIBRIA OF L INEAR -Q UADRATIC M EAN -F IELD G AMES
A BSTRACT
We study discrete-time mean-ﬁeld Markov games with inﬁnite numbers of agents where each agent aims to minimize its ergodic cost.
We consider the setting where the agents have identical linear state transitions and quadratic cost functions, while the aggregated effect of the agents is captured by the population mean of their states, namely, the mean-ﬁeld state.
For such a game, based on the Nash certainty equivalence principle, we provide sufﬁcient conditions for the existence and uniqueness of its Nash equilibrium.
Moreover, to ﬁnd the Nash equilibrium, we propose a mean-ﬁeld actor-critic algorithm with linear function approximation, which does not require knowing the model of dynamics.
Speciﬁcally, at each iteration of our algorithm, we use the single-agent actor-critic algorithm to approximately obtain the optimal policy of the each agent given the current meanﬁeld state, and then update the mean-ﬁeld state.
In particular, we prove that our algorithm converges to the Nash equilibrium at a linear rate.
To the best of our knowledge, this is the ﬁrst success of applying model-free reinforcement learning with function approximation to discrete-time mean-ﬁeld Markov games with provable non-asymptotic global convergence guarantees.
I NTRODUCTION
In reinforcement learning (RL) (Sutton and Barto, 2018), an agent learns to make decisions that minimize its expected total cost through sequential interactions with the environment.
Multi-agent reinforcement learning (MARL) (Shoham et al., 2003; 2007; Busoniu et al., 2008) aims to extend RL to sequential decision-making problems involving multiple agents.
In a non-cooperative game, we are interested in the Nash equilibrium (Nash, 1951), which is a joint policy of all the agents such that each agent cannot decrease its expected total cost by unilaterally deviating from its Nash policy.
The
Nash equilibrium plays a critical role in understanding the social dynamics of self-interested agents (Ash, 2000; Axtell, 2002) and constructing the optimal policy of a particular agent via ﬁctitious selfplay (Bowling and Veloso, 2000; Ganzfried and Sandholm, 2009).
With the recent development in deep learning (LeCun et al., 2015), MARL with function approximation achieves tremendous empirical successes in applications, including Go (Silver et al., 2016; 2017), Poker (Heinrich and Silver, 2016;
Moravˇc´ık et al., 2017), Star Craft (Vinyals et al., 2019), Dota (OpenAI, 2018), autonomous driving (Shalev-Shwartz et al., 2016), multi-robotic systems (Yang and Gu, 2004), and solving social dilemmas (de Cote et al., 2006; Leibo et al., 2017; Hughes et al., 2018).
However, since the capacity of the joint state and action spaces grows exponentially in the number of agents, such MARL approaches become computationally intractable when the number of agents is large, which is common in real-world applications (Sandholm, 2010; Calderone, 2017; Wang et al., 2017a).
Mean-ﬁeld game is proposed by Huang et al.
; Lasry and Lions (2006a;b; 2007) with the idea of utilizing mean-ﬁeld approximation to model the strategic interactions within a large population.
In a mean-ﬁeld game, each agent has the same cost function and state transition, which depend on the other agents only through their aggregated effect.
As a result, the optimal policy of each agent depends solely on its own state and the aggregated effect of the population, and such an optimal policy is symmetric across all the agents.
Moreover, if the aggregated effect of the population corresponds to the Nash equilibrium, then the optimal policy of each agent jointly constitutes a
Nash equilibrium.
Although such a Nash equilibrium corresponds to an inﬁnite number of agents, it well approximates the Nash equilibrium for a sufﬁciently large number of agents (Bensoussan et al., 2016).
Also, as the aggregated effect of the population abstracts away the strategic interactions between individual agents, it circumvents the computational intractability of the MARL approaches that do not exploit symmetry.
However, most existing work on mean-ﬁeld games focuses on characterizing the existence and uniqueness of the Nash equilibrium rather than designing provably efﬁcient algorithms.
In particular, most existing work considers the continuous-time setting, which requires solving a pair of
Hamilton-Jacobi-Bellman (HJB) and Fokker-Planck (FP) equations, whereas the discrete-time setting is more common in practice, e.g., in the aforementioned applications.
Moreover, most existing approaches, including the ones based on solving the HJB and FP equations, require knowing the model of dynamics (Bardi and Priuli, 2014), or having the access to a simulator, which generates the next state given any state-action pair and aggregated effect of the population (Guo et al., 2019), which is often unavailable in practice.
To address these challenges, we develop an efﬁcient model-free RL approach to mean-ﬁeld game, which provably attains the Nash equilibrium.
In particular, we focus on discrete-time mean-ﬁeld games with linear state transitions and quadratic cost functions, where the aggregated effect of the population is quantiﬁed by the mean-ﬁeld state.
Such games capture the fundamental difﬁculties of general mean-ﬁeld games and well approximates a variety of real-world systems such as power grids (Minciardi and Sacile, 2011), swarm robots (Fang, 2014; Araki et al., 2017; Doerr et al., 2018), and ﬁnancial systems (Zhou and Li, 2000; Huang and Li, 2018).
In detail, based on the
Nash certainty equivalence (NCE) principle (Huang et al., 2006; 2007), we propose a mean-ﬁeld actor-critic algorithm which, at each iteration, given the mean-ﬁeld state µ , approximately attains the optimal policy π ∗ µ of each agent, and then updates the mean-ﬁeld state µ assuming that all the agents follow π ∗ µ .
We parametrize the actor and critic by linear and quadratic functions, respectively, and prove that such a parameterization encompasses the optimal policy of each agent.
Speciﬁcally, we update the actor parameter using policy gradient (Sutton et al., 2000) and natural policy gradient (Kakade, 2002; Peters and Schaal, 2008; Bhatnagar et al., 2009) and update the critic parameter using primal-dual gradient temporal difference (Sutton et al., 2009a;b).
In particular, we prove that given the mean-ﬁeld state µ , the sequence of policies generated by the actor converges linearly to the optimal policy π ∗ µ .
Moreover, when alternatingly update the policy and mean-ﬁeld state, we prove that the sequence of policies and its corresponding sequence of mean-ﬁeld states converge to the unique Nash equilibrium at a linear rate.
Our approach can be interpreted from both “passive” and “active” perspectives: (i) Assuming that each self-interested agent employs the single-agent actorcritic algorithm, the policy of each agent converges to the unique Nash policy, which characterizes the social dynamics of a large population of model-free RL agents.
(ii) For a particular agent, our approach serves as a ﬁctitious self-play method for it to ﬁnd its Nash policy, assuming the other agents give their best responses.
To the best of our knowledge, our work establishes the ﬁrst efﬁcient model-free RL approach with function approximation that provably attains the Nash equilibrium of a discrete-time mean-ﬁeld game.
As a byproduct, we also show that the sequence of policies generated by the single-agent actor-critic algorithm converges at a linear rate to the optimal policy of a linear-quadratic regulator (LQR) problem in the presence of drift, which may be of independent interest.
Related Work.
Mean-ﬁeld game is ﬁrst introduced in Huang et al.
; Lasry and Lions (2006a;b; 2007).
In the last decade, there is growing interest in understanding continuous-time mean-ﬁeld games.
See, e.g., Gu´eant et al.
; Bensoussan et al.
; Gomes et al.
Carmona and Delarue (2013; 2018) and the references therein.
Due to their simple structures, continuous-time linear-quadratic mean-ﬁeld games are extensively studied under various model assumptions.
See Li and Zhang (2008); Bardi (2011); Wang and Zhang (2012); Bardi and Priuli (2014);
Huang et al.
(2016a;b); Bensoussan et al.
(2016; 2017); Caines and Kizilkale (2017); Huang and Huang (2017); Moon and Bas¸ar (2018)
; Huang and Zhou (2019) for examples of this line of work.
Meanwhile, the literature on discrete-time linear-quadratic mean-ﬁeld games remains relatively scarce.
Most of this line of work focuses on characterizing the existence of a Nash equilibrium and the behavior of such a Nash equilibrium when the number of agents goes to inﬁnity (Gomes et al., 2010; Tembine and Huang, 2011; Moon and Bas¸ar, 2014; Biswas, 2015; Saldi et al., 2018a;b; 2019).
See also Yang et al.
(2018a), which applies maximum entropy inverse RL
(Ziebart et al., 2008) to infer the cost function and social dynamics of discrete-time mean-ﬁeld games with ﬁnite state and action spaces.
Our work is most related to Guo et al.
(2019), where they propose a mean-ﬁeld Q-learning algorithm (Watkins and Dayan, 1992) for discrete-time mean-ﬁeld games with ﬁnite state and action spaces.
Such an algorithm requires the access to a simulator, which, given any state-action pair and mean-ﬁeld state, outputs the next state.
In contrast, both our state and action spaces are inﬁnite, and we do not require such a simulator but only observations of trajectories under given mean-ﬁeld state.
Correspondingly, we study the mean-ﬁeld actor-critic algorithm with linear function approximation, whereas their algorithm is tailored to the tabular setting.
Also, our work is closely related to Mguni et al.
(2018), which focuses on a more restrictive setting where the state transition does not involve the mean-ﬁeld state.
In such a setting, mean-ﬁeld games are potential games, which is, however, not true in more general settings (Li et al., 2017; Briani and
Cardaliaguet, 2018).
In comparison, we allow the state transition to depend on the mean-ﬁeld state.
Meanwhile, they propose a ﬁctitious self-play method based on the single-agent actor-critic algorithm and establishes its asymptotic convergence.
However, their proof of convergence relies on the assumption that the single-agent actor-critic algorithm converges to the optimal policy, which is unveriﬁed therein.
In addition, our work is related to Jayakumar and Aditya (2019), where the proposed algorithm is only shown to converge asymptotically to a stationary point of the mean-ﬁeld game.
Our work also extends the line of work on ﬁnding the Nash equilibria of Markov games using
MARL.
Due to the computational intractability introduced by the large number of agents, such a line of work focuses on ﬁnite-agent Markov games (Littman, 1994; 2001; Hu and Wellman, 1998;
Bowling, 2001; Lagoudakis and Parr, 2002; Hu and Wellman, 2003; Conitzer and Sandholm, 2007;
Perolat et al., 2015; P ´ erolat et al., 2016b;a; 2018; Wei et al., 2017; Zhang et al., 2018; Zou et al., 2019; Casgrain et al., 2019).
See also Shoham et al.
(2003; 2007); Busoniu et al.
; Li (2018) for detailed surveys.
Our work is related to Yang et al.
(2018b), where they combine the mean-ﬁeld approximation of actions (rather than states) and Nash Q-learning (Hu and Wellman, 2003)
to study general-sum Markov games with a large number of agents.
However, the Nash Q-learning algorithm is only applicable to ﬁnite state and action spaces, and its convergence is established under rather strong assumptions.
Also, when the number of agents goes to inﬁnity, their approach yields a variant of tabular Q-learning, which is different from our mean-ﬁeld actor-critic algorithm.
For policy optimization, based on the policy gradient theorem, Sutton et al.
; Konda and Tsitsiklis (2000) propose the actor-critic algorithm, which is later generalized to the natural actor-critic algorithm (Peters and Schaal, 2008; Bhatnagar et al., 2009).
Most existing results on the convergence of actor-critic algorithms are based on stochastic approximation using ordinary differential equations (Bhatnagar et al., 2009; Castro and Meir, 2010; Konda and Tsitsiklis, 2000; Maei, 2018), which are asymptotic in nature.
For policy evaluation, the convergence of primal-dual gradient temporal difference is studied in Liu et al.
; Du et al.
; Wang et al.
(2017b)
; Yu (2017);
Wai et al.
However, this line of work assumes that the feature mapping is bounded, which is not the case in our setting.
Thus, the existing convergence results are not applicable to analyzing the critic update in our setting.
To handle the unbounded feature mapping, we utilize a truncation argument, which requires more delicate analysis.
Finally, our work extends the line of work that studies model-free RL for LQR.
For example, Bradtke (1993); Bradtke et al.
(1994) show that policy iteration converges to the optimal policy, Tu and
Recht (2017); Dean et al.
(2017) study the sample complexity of least-squares temporal-difference for policy evaluation.
More recently, Fazel et al.
; Malik et al.
; Tu and Recht (2018) show that the policy gradient algorithm converges at a linear rate to the optimal policy.
See as also
Hardt et al.
; Dean et al.
(2018) for more in this line of work.
Our work is also closely related to Yang et al.
(2019), where they show that the sequence of policies generated by the natural actorcritic algorithm enjoys a linear rate of convergence to the optimal policy.
Compared with this work, when ﬁxing the mean-ﬁeld state, we use the actor-critic algorithm to study LQR in the presence of drift, which introduces signiﬁcant difﬁculties in the analysis.
As we show in § 3, the drift causes the optimal policy to have an additional intercept, which makes the state- and action-value functions more complicated.
Notations.
We denote by (cid:107) M (cid:107) ∗ the spectral norm, ρ ( M ) the spectral radius, σ min ( M ) the minimum singular value, and σ max ( M ) the maximum singular value of a matrix M .
We use (cid:107) α (cid:107) 2 to represent the (cid:96) 2 -norm of a vector α , and ( α ) ji to denote the sub-vector ( α i , α i +1 , . . .
, α j ) (cid:62) , where α k is the k -th entry of the vector α .
For scalars a 1 , . . . , a n , we denote by poly ( a 1 , . . .
, a n ) the polynomial of a 1 , . .
. , a n , and this polynomial may vary from line to line.
We use [ n ] to denote the set { 1 , 2 , . . . , n } for any n ∈ N .
L INEAR -Q UADRATIC M EAN -F IELD G AME
A linear-quadratic mean-ﬁeld N a -player game involves N a ∈ N agents, whose state transitions are given by
[Equation]
Here A ∈ R m × m , B ∈ R m × k , and A ∈ R m × m are matrices, x it ∈ R m and u
it ∈
R k are the state and action vectors of agent i , respectively, the vector d i ∈
R m is a drift term, ω it ∈
R m is an independent random noise term following the Gaussian distribution N (0 , Ψ ω ) , and x t = 1 /N a · (cid:80) N a j =
x
jt is the mean-ﬁeld state.
The agents are coupled through the mean-ﬁeld state x t .
In the linear-quadratic mean-ﬁeld N a -player game, the cost of agent i ∈
[ N a ] at time t ≥ 0 is given by
[Equation]
where Q ∈ R m × m , R ∈ R k × k , and Q ∈ R m × m are matrices, and u it is generated by π i , i.e., the policy of agent i .
To measure the performance of agent i following its policy π i under the inﬂuence of the other agents, we deﬁne the expected total cost of agent i as
[Equation]
We are interested in ﬁnding a Nash equilibrium ( π 1 , π 2 , . .
, π N a ) , which is deﬁned by
[Equation]
That is, agent i cannot further decrease its expected total cost by unilaterally deviating from its Nash policy.
For the simplicity of discussion, we assume that the drift term d i is identical for each agent.
We consider taking the inﬁnite-population limit N a → ∞ , where each agent has an inﬁnitesimal contribution to the dynamics of the system.
Thus, the joint policy of all the agents except agent i can be modeled as a mean-ﬁeld policy π † , and all the agents following such a mean-ﬁeld policy π † generate the mean-ﬁeld state E x † t , where { x † t } t ≥ 0 is generated following the policy π † .
By the symmetry of the agents in terms of their state transitions and cost functions, we focus on a ﬁxed agent and drop the superscript i hereafter.
Before we formally present the formulation of linear-quadratic mean-ﬁeld games, we ﬁrst introduce the following mean-ﬁeld LQR (MF-LQR) problem, which aims to ﬁnd an optimal policy for the ﬁxed agent given the mean-ﬁeld policy π † .
Problem 2.1 (MF-LQR) .
Given the mean-ﬁeld policy π † , we consider the following formulation,
[Equation]
where x t ∈ R m is the state vector, u t ∈ R k is the action vector generated by the policy π , { x † t } t ≥ 0 is the trajectory generated by the policy π † , ω t ∈ R m is an independent random noise term following the Gaussian distribution N (0 , Ψ ω ) , and d ∈ R m is a drift term.
Here the expectation E x † t is taken across all the agents.
We aim to ﬁnd π ∗ such that J ( π ∗ , π † ) =
inf π ∈ Π J ( π, π † ) .
Note that a controllable linear system using linear quadratic optimal control is always stable.
Further combining the fact that our linear closed-loop dynamics in Problem 2.1 is driven by the Gaussian noise term ω t , we know that the Markov chain of states generated by the policy π † admits a stationary distribution and converges to this stationary distribution.
This implies that the mean-ﬁeld state E x † t converges to a constant vector µ † as t → ∞ , which serves as a time-invariant mean-ﬁeld state.
As we consider the ergodic setting, it then sufﬁces to study Problem 2.1 with t sufﬁciently large.
Therefore, the inﬂuence of the mean-ﬁeld policy π † is captured by the mean-ﬁeld state µ † .
By re-formulating
Problem 2.1, with slight abuse of notations, we obtain the following drifted-LQR (D-LQR).
Problem 2.2 (D-LQR) .
Given a mean-ﬁeld state µ ∈ R m , we consider the following formulation,
[Equation]
where x t ∈ R m is the state vector, u t ∈ R k is the action vector generated by the policy π , ω t ∈ R m is an independent random noise term following the Gaussian distribution N (0 , Ψ ω ) , and d ∈ R m is a drift term.
We aim to ﬁnd an optimal policy π ∗ µ such that J µ ( π ∗ µ ) =
inf π ∈ Π J µ ( π ) .
Compared with the most studied LQR problem (Lewis et al., 2012), both the state transition and the cost function in Problem 2.2 have drift terms, which act as the mean-ﬁeld “force” that drives the states away from zero.
Such a mean-ﬁeld “force” introduces additional challenges when solving
Problem 2.2 in the model-free setting (see § 3.3 for details).
On the other hand, the unique optimal policy π ∗ µ of Problem 2.2 admits a linear form π ∗ µ ( x t ) =
− K π ∗ µ x t + b π ∗ µ under certain regularity conditions (Anderson and Moore, 2007), where the matrix K π ∗ µ ∈ R k × m and the vector b π ∗ µ ∈ R k are the parameters of π ∗ µ .
Motivated by such a linear form of the optimal policy, we deﬁne the class of linear-Gaussian policies as
[Equation]
where σ ∈ R and the standard Gaussian noise term η ∈ R k is included to encourage exploration.
To solve Problem 2.2, it sufﬁces to ﬁnd the optimal policy π ∗ µ within Π .
We deﬁne Λ 1 ( µ ) =
π ∗ µ as the optimal policy under the mean-ﬁeld state µ .
Assume that all the agents follow the linear policy π ( x ) = − K π x + b π under the mean-ﬁeld state µ .
By plugging u t = π ( x t ) into the state transition in Problem 2.2, as t → ∞ , we know that these agents generate a new mean-ﬁeld state µ new such that
[Equation]
We deﬁne Λ 2 ( µ, π ) = µ new as such a new mean-ﬁeld state.
Now, we are ready to present the following linear-quadratic mean-ﬁeld game (LQ-MFG).
Problem 2.3 (LQ-MFG) .
We consider the following formulation,
[Equation]
where x t ∈ R m is the state vector, u t ∈ R k is the action vector generated by the policy π , µ ∈ R m is the mean-ﬁeld state, ω t ∈ R m is an independent random noise term following the Gaussian distribution N (0 , Ψ ω ) , and d ∈ R m is a drift term.
We aim to ﬁnd a pair ( µ ∗ , π ∗ ) such that (i)
J ( π ∗ , µ ∗ ) =
inf π ∈ Π J
( π, µ ∗ ) ; (ii) E x ∗ t converges to µ ∗ as t → ∞ , where { x ∗ t } t ≥ 0 is the Markov chain of states generated by the policy π ∗ .
The formulation in Problem 2.3 is studied by Lasry and Lions (2007); Bensoussan et al.
; Saldi et al.
(2018a;b).
We propose a more general formulation in Problem C.2 (see § C of the appendix for details), where an additional interaction term between the state vector x t and the mean-ﬁeld state µ is incorporated into the cost function.
According to our analysis in § C, up to minor modiﬁcation, the results in the following sections also carry over to Problem C.2.
Therefore, for the sake of simplicity, we focus on Problem 2.3 in the sequel.
In Problem 2.3, condition (i) is equivalent to the optimality of the policy π ∗ under the mean-ﬁeld state µ ∗ , namely, Λ 1 ( µ ∗ )
=
π ∗ .
Meanwhile, condition (ii) is equivalent to the invariance of the mean-ﬁeld state µ ∗ given the policy π ∗ , namely, Λ 2 ( µ ∗ , π ∗ ) = µ ∗ .
Such equivalence follows from the NCE principle (Huang et al., 2006; 2007), which also motivates the following deﬁnition of the
Nash equilibrium pair (Saldi et al., 2018a;b).
Deﬁnition 2.4 (Nash Equilibrium Pair) .
The pair ( µ ∗ , π ∗ ) ∈ R m ×
Π constitutes a Nash equilibrium pair of Problem 2.3 if it satisﬁes π ∗ =
Λ 1 ( µ ∗ ) and µ ∗ =
Λ 2 ( µ ∗ , π ∗ ) .
Here µ ∗ is called the Nash mean-ﬁeld state and π ∗ is called the Nash policy.
By Deﬁnition 2.4, Problem 2.3 aims to ﬁnd a Nash equilibrium pair ( µ ∗ , π ∗ ) .
M EAN -F IELD A CTOR -C RITIC
We ﬁrst characterize the existence and uniqueness of the Nash equilibrium pair of Problem 2.3 under mild regularity conditions, and then propose a mean-ﬁeld actor-critic algorithm to obtain such a Nash equilibrium.
As a building block of the mean-ﬁeld actor-critic, we propose the natural actor-critic to solve Problem 2.2.
3. E XISTENCE AND U NIQUENESS OF N ASH E QUILIBRIUM P AIR
We now establish the existence and uniqueness of the Nash equilibrium pair deﬁned in Deﬁnition 2.4.
We impose the following regularity conditions.
Assumption 3.1.
We assume that the following statements hold:
(i)
The algebraic Riccati equation X =
A (cid:62) XA + Q − A (cid:62) XB ( B (cid:62) XB + R )
− 1 B (cid:62)
XA admits a unique symmetric positive deﬁnite solution X ∗ ; (ii) It holds for L
=
L 1 L 3 + L 2 that L 0 < 1 , where
[Equation]
Here K ∗ = − ( B (cid:62) X ∗ B + R ) − 1 B (cid:62) X ∗
A
[Equation]
The ﬁrst assumption is implied by mild regularity conditions on the matrices A , B , Q , and R , which are (1) the positivity of R ; (2) the non-negativity of Q = C (cid:62) C ; (3) the observability of ( A, C ) the stability of A, B ) .
See De Souza et al.
; Lewis et al.
(2012) for more details.
The second assumption is standard in the literature (Bensoussan et al., 2016; Saldi et al., 2018b), which ensures the stability of the LQ-MFG.
In the following proposition, we show that Problem 2.3 admits a unique Nash equilibrium pair.
Proposition 3.2 (Existence and Uniqueness of Nash Equilibrium Pair) .
Under Assumption 3.1, the operator Λ( · ) = Λ 2 ( · , Λ 1 ( · )) is L 0 -Lipschitz, where L 0 is given in Assumption 3.1.
Moreover, there exists a unique Nash equilibrium pair ( µ ∗ , π ∗ ) of Problem 2.3.
Proof.
See § E.1 for a detailed proof.
3. M EAN -F IELD A CTOR -C RITIC FOR LQ-MFG
The NCE principle motivates a ﬁxed-point approach to solve Problem 2.3, which generates a sequence of policies { π s } s ≥ 0 and mean-ﬁeld states { µ s } s ≥ 0 satisfying the following two properties: (i)
Given the mean-ﬁeld state µ s , the policy π s is optimal.
(ii)
The mean-ﬁeld state becomes µ s + as t → ∞ , if all the agents follow π s under the current mean-ﬁeld state µ s .
Here (i) requires solving
Problem 2.2 given the mean-ﬁeld state µ s , while (ii) requires simulating the agents following the [Equation] policy π s given the current mean-ﬁeld µ s .
Based on such properties we propose the mean-ﬁeld actor-critic in Algorithm 1.
Algorithm 1 requires solving Problem 2.2 at each iteration to obtain π s = Λ 1 ( µ s ) and µ s +
=
Λ 2 ( µ s , π s ) .
To this end, we introduce the natural actor-critic in § 3.3 that solves Problem 2.2.
N ATURAL A CTOR -C RITIC FOR D-LQR
Now we focus on solving Problem 2.2 for a ﬁxed mean-ﬁeld state µ , we thus drop the subscript µ hereafter.
With slight abuse of notations, we write π K,b ( x ) =
− Kx
+ b + σ · η to emphasize the dependence on K and b , and J ( K, b ) =
J ( π K,b ) consequently.
Now, we propose the natural actor-critic to solve Problem 2.2.
For any policy π K,b ∈ Π , by the state transition in Problem 2.2, we have
[Equation]
where Ψ (cid:15) = σBB (cid:62) + Ψ ω .
It is known that if ρ ( A − BK ) < 1 , then the Markov chain { x t } t ≥induced by (3.1) has a unique stationary distribution N ( µ K,b , Φ K ) (Anderson and Moore, 2007), where the mean-ﬁeld state µ K,b and the covariance Φ K satisfy that
[Equation]
Meanwhile, the Bellman equation for Problem 2.2 takes the following form
[Equation]
Then by calculation (see Proposition B.2 in § B.1 of the appendix for details), it holds that the expected total cost J ( K, b ) is decomposed as
[Equation]
where J 1 ( K ) and J 2 ( K, b ) are deﬁned as
[Equation]
Here J 1 ( K ) is the expected total cost in the most studied LQR problems (Yang et al., 2019; Fazel et al., 2018), where the state transition does not have drift terms.
Meanwhile, J 2 ( K, b ) corresponds to the expected cost induced by the drift terms.
The following two propositions characterize the properties of J 2 ( K, b ) .
First, we show that J 2 ( K, b ) is strongly convex in b .
Proposition 3.3.
Given any K , the function J 2 ( K, b ) is ν K -strongly convex in b .
Here ν K =
σ min ( Y (cid:62) 1 ,K Y 1 ,K + Y (cid:62) 2 ,K Y 2 ,K ) , where Y 1 ,K = R 1 / 2 K ( I − A + BK ) − 1 B − R 1 / 2 and Y 2 ,K =
Q 1 / 2 ( I − A + BK ) − 1 B .
Also, J 2 ( K, b ) has ι K -Lipschitz continuous gradient in b , where ι K is upper bounded as ι K ≤ [1 − ρ ( A − BK )]
− 2 · ( (cid:107) B (cid:107) 22 · (cid:107) K (cid:107) 22 · (cid:107) R (cid:107) 2 + (cid:107) B (cid:107) 22 · (cid:107) Q (cid:107) 2 ) .
Proof.
See § E.4 for a detailed proof.
Second, we show that min b J 2 ( K, b ) is independent of K .
Proposition 3.4.
We deﬁne b K =
argmin b J 2 ( K, b ) , where J 2 ( K, b ) is deﬁned in (3.6).
It holds that
[Equation]
Moreover, J 2 ( K, b K ) takes the form of
[Equation]
which is independent of K .
Proof.
See § E.2 for a detailed proof.
Since min b J 2 ( K, b ) is independent of K by Proposition 3.4, it holds that the optimal K ∗ is the same as argmin K J 1 ( K ) .
This motivates us to minimize J ( K, b ) by ﬁrst updating K following the gradient direction ∇ K J 1 ( K ) to the optimal K ∗ , then updating b following the gradient direction ∇ b J 2 ( K ∗ , b ) .
We now design our algorithm based on this idea.
We deﬁne Υ K , p K,b , and q K,b as
[Equation]
where f K,b
= ( I − A + BK ) −(cid:62)
[( A − BK ) (cid:62) P K ( Bb + Aµ + d ) − K (cid:62) Rb ] .
By calculation (see
Proposition B.3 in § B.1 of the appendix for details), the gradients of J 1 ( K ) and J 2 ( K, b ) take the forms of
[Equation]
Our algorithm follows the natural actor-critic method (Bhatnagar et al., 2009) and actor-critic method (Konda and Tsitsiklis, 2000).
Speciﬁcally, (i) To obtain the optimal K ∗ , in the critic update step, we estimate the matrix Υ K by (cid:98) Υ K via a policy evaluation algorithm, e.g., Algorithm 3 or
Algorithm 4 (see § B.2 and § B.3 of the appendix for details); in the actor update step, we update K via K ← K − γ ·
( (cid:98) Υ 22 K K − (cid:98)
Υ 21 K ) , where the term (cid:98) Υ
22 K K − (cid:98)
Υ 21 K is the estimated natural gradient.
(ii)
To obtain the optimal b ∗ given K ∗ , in the critic update step, we estimate Υ K ∗ , q K ∗ ,b , and µ K ∗ ,b by (cid:98) Υ K ∗ , (cid:98) q K ∗ ,b , and (cid:98) µ K ∗ ,b via a policy evaluation algorithm;
In the actor update step, we update b via b ← b − γ · (cid:98) ∇ b J 2 ( K ∗ , b ) , where (cid:98) ∇ b J 2 ( K ∗ , b )
= (cid:98) Υ 22 K ∗ ( − K ∗ (cid:98) µ K ∗ ,b + b ) + (cid:98) Υ
21 K ∗ (cid:98) µ K ∗ ,b + (cid:98) q K ∗ ,b is the estimated gradient.
Combining the above procedure, we obtain the natural actor-critic for Problem 2.2, which is stated in Algorithm 2.
One may want to apply gradient method to J ( K, b ) directly in the joint space of K and b .
However, the gradient dominance property of J 1 ( K ) in the most studied LQR problem (Yang et al., 2019) no longer holds for J ( K, b ) .
Therefore, the convergence of the gradient method to J ( K, b ) is not guaranteed in our problem.
G LOBAL C ONVERGENCE R ESULTS
The following theorem establishes the rate of convergence of Algorithm 1 to the Nash equilibrium pair ( µ ∗ , π ∗ ) of Problem 2.3.
• Mean-ﬁeld state µ and initial policy π K 0 ,b 0 .
• Numbers of iterations N , H , { (cid:101) T n , T n } n ∈ [ N ] , { (cid:101) T bh , T bh } h ∈
[ H ] .
• Stepsizes γ , γ b , { γ n,t } n ∈ [ N ] ,t ∈ [ T n ] , { γ bh,t } h ∈
[ H ] ,t ∈
[ T bh ]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Equation] Theorem 4.1 (Convergence of Algorithm 1) .
For a sufﬁciently small tolerance ε > 0 , we set the number of iterations S in Algorithm 1 such that
[Equation]
For any s ∈ [ S ] [Equation] we deﬁne [Equation]
where ν K ∗ is deﬁned in Proposition [Equation] 3 and
[Equation]
In the s -th policy update step in Line 3 of Algorithm 1, we set the inputs via Theorem B.4 such that
J µ s ( π s +1 )
− J µ s ( π ∗ µ s ) <
ε s , where the expected total cost J µ s ( · ) is deﬁned in Problem 2.2, and π ∗ µ s = Λ 1 ( µ s ) is the optimal policy under the mean-ﬁeld state µ s .
Then it holds with probability at least 1 − ε 5 that
[Equation]
Here µ ∗ is the Nash mean-ﬁeld state, K S and b S are parameters of the policy π S , and K ∗ and b ∗ are parameters of the Nash policy π ∗ .
Proof Sketch.
The proof of the theorem is based on the convergence of the natural actor-critic algorithm 2 and a contraction argument.
First, we prove in Theorem B.4 that Algorithm 2 converges linearly to the optimal policy of Problem 2.2.
By this, in each iteration of Algorithm 1, we control the error between µ s +1 and µ ∗ s +1 to be ε s > 0 with high probability, where µ ∗ s +1 is the mean-ﬁeld state generated by the optimal policy Λ 1 ( µ s ) ; in other words, µ ∗ s +1 = Λ( µ s ) .
Combining the fact from Proposition 3.2 that Λ( · ) is a contraction, we deduce that
[Equation]
with high probability, where (cid:101) ε s > 0 is some error term and is speciﬁed in the detailed proof.
Moreover, by telescoping sum, and note that the sum (cid:80) Ss =1 (cid:101) ε s is upper bounded by the desired error ε , we conclude the theorem.
See § D.1 for a detailed proof.
We highlight that if the inputs of Algorithm 1 satisfy the conditions stated in Theorem B.4, it holds that J µ s ( π s +1 ) − J µ s ( π ∗ µ s ) <
ε s for any s ∈ [ S ] .
See Theorem B.4 in § B.1 of the appendix for details.
By Theorem 4.1, Algorithm 1 converges linearly to the unique Nash equilibrium pair ( µ ∗ , π ∗ ) of Problem 2.3.
To the best of our knowledge, this theorem is the ﬁrst successful attempt to establish that reinforcement learning with function approximation ﬁnds the Nash equilibrium pairs in mean-ﬁeld games with theoretical guarantee, which lays the theoretical foundations for applying modern reinforcement learning techniques to general mean-ﬁeld games.
C ONCLUSION
For the discrete-time linear-quadratic mean-ﬁeld games, we provide sufﬁcient conditions for the existence and uniqueness of the Nash equilibrium pair.
Moreover, we propose the mean-ﬁeld actorcritic algorithm with linear function approximation that is shown converges to the Nash equilibrium pair with linear rate of convergence.
Our algorithm can be modiﬁed to use other parametrized function classes, including deep neural networks, for solving mean-ﬁeld games.
For future research, we aim to extend our algorithm to other variations of mean-ﬁeld games including risk-sensitive mean-ﬁeld games (Saldi et al., 2018a; Tembine et al., 2014), robust mean-ﬁeld games (Bauso et al., 2016), and partially observed mean-ﬁeld games (Saldi et al., 2019).
A N OTATIONS IN THE A PPENDIX
In the proof, for convenience, for any invertible matrix M , we denote by M −(cid:62) =
( M − 1 ) (cid:62) =
( M (cid:62) ) − 1 and (cid:107) M (cid:107) F the Frobenius norm.
We also denote by svec ( M ) the symmetric vectorization of the symmetric matrix M , which is the vectorization of the upper triangular matrix of the symmetric matrix M , with off-diagonal entries scaled by √ 2 .
We denote by smat ( · ) the inverse operation.
For any matrices G and H , we denote by G ⊗ H the Kronecker product, and G ⊗ s H the symmetric Kronecker product, which is deﬁned as a mapping on a vector svec ( M ) such that [Equation]
For notational simplicity, we write E π ( · ) to emphasize that the expectation is taken following the policy π .
B A UXILIARY A LGORITHMS AND A NALYSIS B. R ESULTS IN D-LQR
In this section, we provide auxiliary results in analyzing Problem 2.2.
First, we introduce the value functions of the Markov decision process (MDP) induced by Problem 2.2.
We deﬁne the state- and action-value functions V K,b ( x ) and Q K,b ( x, u ) as follows
[Equation]
where x t follows the state transition, and u t follows the policy π K,b given x t .
In other words, we have u t =
− Kx t + b + ση t , where η t ∼ N (0 , I ) .
The following proposition establishes the close forms of these value functions.
Proposition B.1.
The state-value function V K,b ( x ) takes the form of
[Equation]
and the action-value function Q K,b ( x, u ) takes the form of
Q K
[Equation]
b
[Equation]
where f K,b
= ( I − A + BK ) −(cid:62)
[( A − BK ) (cid:62) P K ( Bb + Aµ + d ) − K (cid:62) Rb ] , and Υ K , p K,b , and q K,b are deﬁned in (3.7).
Proof.
See § E.6 for a detailed proof.
By Proposition B.1, we know that V K,b ( x ) is quadratic in x , while Q K,b ( x, u ) is quadratic in ( x (cid:62) , u (cid:62) ) (cid:62) .
Now, we show that (3.5) holds.
Proposition B.2.
The expected total cost J ( K, b ) deﬁned in Problem 2.2 takes the form of
[Equation]
, where
[Equation]
Here µ K,b is deﬁned in (3.2), Φ K is deﬁned in (3.3), and P K is deﬁned in (3.4).
Proof.
See § E.3 for a detailed proof.
The following proposition establishes the gradients of J 1 ( K ) and J 2 ( K, b ) , respectively.
Proposition B.3.
The gradient of J 1 ( K ) and the gradient of J 2 ( K, b ) with respect to b take the forms of
[Equation]
where Υ K and q K,b are deﬁned in (3.7).
Proof.
See § E.5 for a detailed proof.
The following theorem establishes the convergence of Algorithm 2.
Theorem B.4 (Convergence of Algorithm 2) .
Assume that ρ ( A − BK 0 ) < 1 .
Let ε > 0 be a sufﬁciently small tolerance. We set
[Equation]
[Table]
where C , C
[Table]
and γ 0 are positive absolute constants, { K n } n ∈ [ N ] and { b h } h ∈
[ H ] are the sequences generated by Algorithm 2, λ K n is speciﬁed in Proposition B.6, and ν K N is speciﬁed in Proposition 3.3.
Then it holds with probability at least 1 − ε 10 that
[Equation]
where M b ( µ ) is deﬁned in (4.3).
Proof.
See § D.2 for a detailed proof.
By Theorem B.4, given any mean-ﬁeld state µ , Algorithm 2 converges linearly to the optimal policy π ∗ µ of Problem 2.2.
B. P RIMAL -D UAL P OLICY E VALUATION A LGORITHM
Note that the critic update steps in Algorithm 2 are built upon the estimators of the matrix Υ K and the vector q K,b .
We now derive a policy evaluation algorithm to establish the estimators of Υ K and q K,b , which is based on gradient temporal difference algorithm (Sutton et al., 2009a).
We deﬁne the feature vector as
[Equation]
(B.5) where ϕ
[Equation]
[Equation]
Recall svec ( M ) gives the symmetric vectorization of the symmetric matrix M .
We also deﬁne
[Equation]
where Υ K , p K,b , and q K,b are deﬁned in (3.7).
To estimate Υ K and q K,b , it sufﬁces to estimate α K,b .
Meanwhile, we deﬁne
[Equation]
where ( x (cid:48) , u (cid:48) ) is the state-action pair after ( x, u ) following the policy π K,b and the state transition.
The following proposition characterizes the connection between Θ K,b and α K,b .
Proposition B.5.
It holds that
[Equation]
where ψ ( x, u ) is deﬁned in (B.5), α K,b is deﬁned in (B.6), and Θ K,b is deﬁned in (B.7).
Proof.
See § E.7 for a detailed proof.
By Proposition B.5, to obtain α K,b , it sufﬁces to solve the following linear system in ζ =
( ζ 1 , [Equation] [Equation]
[Equation]
where for notational convenience, we deﬁne
[Equation]
The following proposition shows that Θ K,b is invertible.
Proposition B.6.
If ρ ( A − BK ) < 1 , then the matrix Θ K,b is invertible, and (cid:107) Θ K,b (cid:107) 2 ≤ 4(1 + (cid:107) K (cid:107) 2 F ) 2 · (cid:107) Φ K (cid:107) 22 .
Also, σ min ( (cid:101)
Θ K,b ) ≥ λ K , where λ K only depends on (cid:107) K (cid:107) 2 and ρ ( A − BK ) .
Proof.
See § E.8 for a detailed proof.
By Proposition B.6, Θ K,b is invertible.
Therefore, (B.8) admits the unique solution ζ K, b = ( J ( K, b ) , α (cid:62) K,b ) (cid:62) .
Now, we present the primal-dual gradient temporal difference algorithm.
Primal-Dual Gradient Method.
Instead of solving (B.8) directly, we minimize the following loss function with respect to ζ =
(( ζ 1 ) (cid:62) , ( ζ
[Equation]
By Fenchel’s duality
[Equation]
the minimization of (B.10) is equivalent to the following primal-dual min-max problem,
[Equation]
where we restrict the primal variable ζ in a compact set V ζ and the dual variable ξ in a compact set V ξ , which are speciﬁed in Deﬁnition B.7.
It holds that
[Equation]
The primal-dual gradient method updates ζ and ξ via
[Equation]
Estimation of Mean-Field State µ K,b .
To utilize the primal-dual gradient method in (B.13), it remains to evaluate the feature vector ψ ( x, u ) . Note that by (B.5), the evaluation of the feature vector ψ ( x, u ) requires the mean-ﬁeld state µ K,b .
In what follows, we establish the estimator (cid:98) µ K,b of the mean-ﬁeld state µ K,b by simulating the MDP following the policy π K,b for (cid:101) T steps, and calculate the estimated feature vector (cid:98) ψ ( x, u ) by [Equation]
[Equation]
) where (cid:98) ϕ
( x, u ) takes the form of
[Equation]
We now deﬁne the sets V ζ and V ξ in (B.11).
Deﬁnition B.7.
Given K 0 and b 0 such that ρ ( A − BK 0 ) < 1 and J ( K 0 , b 0 ) < ∞ , we deﬁne the sets V ζ and V ξ as
[Equation]
Here M ζ, 1 , M ζ, [Equation] and M ξ are constants independent of K and b
[Equation]
which take the forms of
M ζ, 1 =
[Equation]
where C is a positive absolute constant, and κ Q and κ R are condition numbers of Q and R , respectively.
We summarize the primal-dual gradient temporal difference algorithm in Algorithm 3.
Hereafter, for notational convenience, we denote by (cid:98) ψ t the estimated feature vector (cid:98) ψ ( x t , u t ) .
We now characterize the rate of convergence of Algorithm 3.
Theorem B.8 (Convergence of Algorithm 3) .
Given K 0 , b 0 , K , and b such that ρ ( A − BK 0 ) <and J ( K, b ) ≤ J ( K 0 , b 0 ) , we deﬁne the sets V ζ and V ξ through Deﬁnition B.7.
Let γ t = γ 0 t − 1 / 2 , where γ 0 is a positive absolute constant. Let ρ [-( A − BK ) < ρ < 1 , when the number of iterations (cid:101) T and T are sufﬁciently large such that (cid:101) T ≤ poly 
-]∈
( ρ ( A − BK ) , 1) . For (cid:101) T ≥ poly 0 ( (cid:107) K (cid:107) F , (cid:107) b (cid:107) 2 , (cid:107) µ (cid:107) 2 , J ( K 0 , b 0 )) · (1 − ρ ) − 6 and a sufﬁciently large T , it holds with probability at least 1 − T − 4 − (cid:101) T − 6 that (cid:107) (cid:98) α K,b − α K,b (cid:107) 22 ≤ λ − 2 K · poly 1 (cid:0) (cid:107) K (cid:107) F , (cid:107) b (cid:107) 2 , (cid:107) µ (cid:107) 2 , J ( K 0 , b 0 ) (cid:1) · (cid:20) log
[Equation]
(cid:101)
[Equation]
(cid:21) , Set δ t ← ζ 1 t + ( (cid:98) ψ t − (cid:98) ψ t +1 ) (cid:62) ζ 2 t − c t via ζ 1 t
[Equation]
1 ← ζ
[Equation]
− γ [Equation] · ( ξ
[Equation] + (cid:98) ψ [Equation]
t ξ [Equation]
t ) , ζ [Equation] 1 ← ζ
[Equation]
− γ
[Equation]
ψ
[Equation]
ψ
[Equation]
ξ
[Equation]
[Table]
[Table]
γ t +1 ) · ξ 1 t + γ t +1 · ( ζ 1 t − c t ) , ξ 2 t +1 ← (1 − γ t +1 ) · ξ 2 t + γ t +1 · δ t +1 · (cid:98) ψ t .
Project ζ t +1 and ξ t +1 to V ζ and V ξ , respectively.
15: end for 16: Set (cid:98) α K,b ← ( (cid:80) Tt =1 γ t ) − 1 · ( (cid:80) Tt =1 γ t · ζ 2 t ) , and (cid:98) Υ K ← smat ( (cid:98) α K,b, 1 ) , (cid:18)(cid:98) p K,b (cid:98) q K,b (cid:19) ← (cid:98) α K,b, 2 − (cid:98)
Υ K (cid:18) (cid:98) µ K,b − K
(cid:98) µ K,b + b (cid:19) ,
where (cid:98) α K,b, 1 =
( (cid:98) α K,b ) ( k + d +1)
( k + d ) / 2and (cid:98) α K,b, 2 = ( (cid:98) α K,b ) ( k + d +3)
( k + d ) / 2 ( k + d +1)
( k + d ) / 2+1 .
17: Output: Estimators (cid:98) µ K,b , (cid:98) Υ K , and (cid:98) q K,b . where λ K is deﬁned in Proposition B.6. Same bounds for (cid:107) (cid:98) Υ K − Υ K (cid:107) 2 F , (cid:107) (cid:98) p K,b − p K,b (cid:107) 22 , and (cid:107) (cid:98) q K,b − q K,b (cid:107) 22 hold. Meanwhile, it holds with probability at least 1 − (cid:101) T − 6 that (cid:107) (cid:98) µ K,b − µ
[Table]
Proof. See § D.3 for a detailed proof. B. T EMPORAL D IFFERENCE P OLICY E VALUATION A LGORITHM Besides the primal-dual gradient temporal difference algorithm, we can also evaluate α K,b by TD(0) method (Sutton and Barto, 2018) in practice, which is presented in Algorithm 4.
Note that in related literature (Bhandari et al., 2018; Korda and La, 2015), non-asymptotic convergence analysis of TD(0) method with linear function approximation is only applied to discounted
MDP. As for our ergodic setting, the convergence of TD(0) method is only shown asymptotically (Borkar and Meyn, 2000; Kushner and Yin, 2003) using ordinary differential equation method.
Therefore, in the convergence theorem proposed in § 3, we only focus on the primal-dual gradient temporal difference method (Algorithm 3) to establish non-asymptotic convergence result.
C G ENERAL F ORMULATION
Compared with Problem 2.3, a more general formulation includes an additional term x (cid:62) t
Pµ in the cost function.
For the completeness of this paper, we deﬁne this general formulation here.
Following from a same argument as in § 2, it sufﬁces to study the setting where t is sufﬁciently large.
First, we propose the following general drifted LQR (general D-LQR) problem, which is parallel to Problem 2.2.
Given the mean-ﬁeld state µ , take action u t following π K,b , observe the cost c t , and generate the next state x t +1 .
Set δ t +1 ← ζ
+ ( (cid:98) ψ t − (cid:98) ψ
1 ) (cid:62) − c Update parameters via ζ 1 t +1 ← (1 − γ + γ
+1 · c ζ
← ζ
− γ
+ δ t +1 · (cid:98) ψ t .
Project ζ t to V (cid:48) ζ , where V (cid:48) ζ is a compact set.
α K,b ← (
Tt =1 γ t ) − 1 · ( (cid:80) Tt =1 γ t · ζ
, and (cid:98) Υ
← smat ( α
, (cid:18)(cid:98) p b (cid:98) q b
← (cid:98) α K,b, 2 − (cid:98) Υ K (cid:18) (cid:98) µ K,b − K (cid:98) µ K,b + b (cid:19) , where (cid:98) α K,b, 1 = ( (cid:98) α K,b ) ( k + d +1)( k + d ) / 2and (cid:98) α K,b, 2 = ( (cid:98) α K,b ) ( k + d +3)( k + d ) / 2 ( k + d +1)( k + d ) / 2+1 .
15: Output: Estimators (cid:98) µ K,b , (cid:98) Υ K , and (cid:98) q K,b .
Problem C.1 (General D-LQR) .
For any given mean-ﬁeld state µ ∈ R m , consider the following formulation
[Equation]
where x t ∈ R m is the state vector, u t ∈ R k is the action vector generated by the policy π , ω t ∈ R m is an independent random noise term following the Gaussian distribution N (0 , Ψ ω ) , and d ∈ R m is a drift term.
We aim to ﬁnd an optimal policy π ∗ µ such that (cid:101) J µ ( π ∗ µ ) =
inf π ∈ Π (cid:101) J µ ( π ) .
In Problem C.1, the unique optimal policy π ∗ µ ( · ) still admits a linear form π ∗ µ ( x t ) =
− K π ∗ µ x t + b π ∗ µ (Anderson and Moore, 2007), where the matrix K π ∗ µ ∈ R k × m and the vector b π ∗ µ ∈ R k are the parameters of the policy π .
It then sufﬁces to ﬁnd the optimal policy in the class Π deﬁned in (2.1).
Parallel to Problem 2.3, we deﬁne the general LQ-MFG problem as follows.
Problem C.2 (General LQ-MFG) .
We consider the following formulation
[Equation]
where x t ∈ R m is the state vector, u t ∈ R k is the action vector generated by the policy π , µ ∈ R m is the mean-ﬁeld state, ω t ∈ R m is an independent random noise term following the Gaussian distribution N (0 , Ψ ω ) , and d ∈ R m is a drift term.
We aim to ﬁnd a pair ( µ ∗ , π ∗ ) such that (i) (cid:101) J
( π ∗ , µ ∗ ) =
inf π ∈ Π (cid:101) J ( π, µ ∗ ) ; (ii) E x ∗ t converges to µ ∗ as t → ∞ , where { x ∗ t } t ≥ 0 is the Markov chain of states generated by the policy π ∗ .
One can see that Problem C.2 aims to ﬁnd a Nash equilibrium pair ( µ ∗ , π ∗ ) .
Similar to the discussions in § 3.2, to solve Problem C.2, one can design an algorithm similar to
Algorithm 1, which solves Problem C.1 and obtain the new mean-ﬁeld state at each iteration.
We omit the detailed algorithm here.
Now, we focus on solving Problem C.1 in the sequel.
Similar to § 3.3, we drop the subscript µ when we focus on Problem C.1 for a ﬁxed µ .
We write π K,b ( x ) =
− Kx
+ b + σ · η to emphasize the dependence on K and b , and (cid:101) J ( K, b )
= (cid:101) J ( π K,b ) consequently.
We derive an explicit form of the expected total cost (cid:101) J ( K, b ) in the following proposition.
Proposition C.3.
The expected total cost (cid:101) J ( K, b ) in Problem C.1 is decomposed as
[Equation]
where (cid:101) J 1 ( K ) and (cid:101) J 2 ( K, b ) take the forms of
[Equation]
Here µ K,b is given in (3.2), Φ K is given in (3.3), and P K is given in (3.4).
Proof.
The proof is similar to the one of Proposition B.2.
Thus we omit it here.
Compared with the form of J ( K, b ) given in (3.5), we see that the only difference is that (cid:101) J ( K, b ) contains an extra term 2 µ (cid:62)
Pµ K,b in (cid:101)
J 2 ( K, b ) , which is only a linear term in b (recall that µ K,b is linear in b by (3.2)).
Thus, (cid:101) J 2 ( K, b ) is still strongly convex in b , as shown in the proposition below.
Proposition C.4.
Given any K , the function (cid:101) J 2 ( K, b ) is ν K -strongly convex in b , here ν K =
σ min ( Y (cid:62) 1 ,K Y 1 ,K + Y (cid:62) 2 ,K Y 2 ,K ) , where Y 1 ,K = R 1 / 2 K ( I − A + BK ) − 1 B − R 1 / 2 and Y 2 ,K =
Q 1 / 2 ( I − A + BK ) − 1 B .
Also, (cid:101) J 2 ( K, b ) has ι K -Lipschitz continuous gradient in b , where ι K is upper bounded such that ι K
[Equation]
Proof.
The proof is similar to the one of Proposition 3.3.
Thus we omit it here.
Parallel to Proposition 3.4, we derive a similar proposition in the sequel.
Proposition C.5.
Denote by (cid:101) b K = argmin b (cid:101) J 2 ( K, b ) , then (cid:101) J 2 ( K, (cid:101) b K ) takes the form (cid:101)
J 2 ( K, (cid:101) b K )
=
(cid:18)
Aµ + d
P (cid:62)
[Equation]
µ (cid:19) , which is independent of K .
Here S =
[( I − A ) Q − 1 ( I − A ) (cid:62) + BR − 1 B (cid:62) ] − 1 .
And (cid:101)
b K takes the form
[Equation]
Proof.
The proof is similar to the one of Proposition 3.4.
Thus we omit it here.
Similar to Problem 2.2, we deﬁne the state- and action-value functions as
[Equation]
where the x (cid:48) is the state generated by the state transition after the state-action pair ( x, u ) .
A slight modiﬁcation of Proposition B.1 gives the proposition below.
Proposition C.6.
For Problem C.1, the state-value function [Equation] V K,b [Equation]
takes the form
[Equation]
and the action-value function (cid:101) Q K,b ( x, u ) takes the form [Equation] =
[Equation]
Here the matrix Υ K is given in (3.7), and the vectors (cid:101) p K,b , (cid:101) q K,b are given as
[Equation]
where the vector (cid:101) f K,b = ( I − A + BK ) −(cid:62)
[( A − BK ) (cid:62) P K ( Bb + Aµ + d ) − K (cid:62)
Rb + Pµ ] .
Proof.
The proof is similar to the one of Proposition B.1.
Thus we omit it here.
Now we establish the gradients of (cid:101) J ( K, b ) for Problem C.1.
Proposition C.7.
The gradient of (cid:101) J 1 ( K ) and the gradient of (cid:101) J 2 ( K, b ) w.r.t.
b takes the form ∇
[Equation]
where the matrix Υ K is given in (3.7), and the vector (cid:101) q K,b is given in (C.1).
Proof.
The proof is similar to the one of Proposition B.3.
Thus we omit it here.
Equipped with above results, parallel to the analysis in § 3, it is clear that by slight modiﬁcation of Algorithms 1, 2, and 3, we derive similar actor-critic algorithms to solve both Problem C.2 and
Problem C.1, where all the non-asymptotic convergence results hold.
We omit the algorithms and the convergence results here.
D P ROOFS OF T HEOREMS D. P ROOF OF T HEOREM 4.
We deﬁne µ ∗ s +1 = Λ( µ s ) , which is the mean-ﬁeld state generated by the optimal policy π K ∗ ( µ s ) ,b ∗ ( µ s )
=
Λ 1 ( µ s ) under the current mean-ﬁeld state µ s . By Proposition 3.4, the optimal K ∗ ( µ ) is independent of the mean-ﬁeld state µ . Therefore, we write K ∗ =
K ∗ ( µ ) hereafter for notational convenience.
By (3.2), we know that
[Equation]
We deﬁne
[Equation]
which is the mean-ﬁeld state generated by the policy π s under the current mean-ﬁeld state µ s , where
K s and b s are the parameters of the policy π s .
By triangle inequality, we have
[Equation]
where µ s +1 is generated by Algorithm 1.
We upper bound E 1 , E 2 , and E 3 in the sequel.
Upper Bound of E 1 . By Theorem B.4, it holds with probability at least 1 − ε 10 that
[Equation]
Upper Bound of E 2 . By the triangle inequality, we have [Equation] By Taylor’s expansion, we have
[Equation]
Meanwhile, by Taylor’s expansion, it holds with probability at least 1 −
[Equation] 10 that
[Equation]
where the last inequality comes from Theorem B.4.
By plugging (D.4) and (D.5) in (D.3), it holds with probability at least 1 − ε 10 that
[Equation]
By Proposition 3.4, it holds that
[Equation]
where the scalar L 1 is deﬁned in Assumption
[Equation]
Meanwhile, by Theorem B
[Equation]
4, it holds with probability at least 1 − ε 10 that
[Equation]
where M b ( µ s ) is deﬁned in (4.3).
Combining (D.6), (D.7), (D.8), and the choice of ε s in (4.2), it holds with probability at least 1 − ε 10 that [Equation]
Upper Bound of E 3 .
By Proposition 3.2, we have
[Equation]
where L [-
-]0 =
L 1 L 3 + L 2 by Assumption 3.1.
By plugging (D.2), (D.9), and (D.10) in (D.1), we know that
[Equation]
which holds with probability at least 1 − ε 10 .
Following from (D.11) and a union bound argument with S = O (log(1 /ε )) , it holds with probability at least 1 − ε 5 that
[Equation]
where we use the fact that L 0 < 1 by Assumption 3.1. By the choice of S in (4.1), it further holds with probability at least 1 − ε 6 that
[Equation]
By Theorem B.4 and the choice of ε s in (4.2), it holds with probability at least 1 − ε 5 that
[Equation]
Meanwhile, by the triangle inequality and the choice of ε s in (4.2), it holds with probability at least 1 − ε 5 that [Equation] where the second inequality comes from Theorem B.4 and Proposition 3.4, and the last inequality comes from (D.12). By (D.12), (D.13), and (D.14), we conclude the proof of the theorem.
D. P ROOF OF T HEOREM B.
Proof.
We ﬁrst show that J 1 ( K N ) − J 1 ( K ∗ ) <
ε/ 2 with a high probability, then show that
J 2 ( K N , b H ) − J 2 ( K ∗ , b ∗ ) <
ε/ 2 with a high probability.
Then we have
[Equation]
with a high probability, which proves Theorem B.4.
Part 1.
We show that J 1 ( K N ) − J 1 ( K ∗ ) <
ε/ 2 with a high probability.
We ﬁrst bound J 1 ( K 1 ) − J 1 ( K 2 ) for any K 1 and K 2 .
By Proposition B.2, J 1 ( K ) takes the form of
[Equation]
D [Equation] [Equation] The following lemma calculates y (cid:62) P K 1 y − y
(cid:62) P K 2 y for any K 1 and K 2 .
Lemma D.1.
Assume that ρ ( A − BK 1 ) < 1 and ρ ( A − BK 2 ) < 1 .
For any state vector y , we denote by { y t } t ≥
the sequence generated by the state transition y t +1 =
( A − BK 2 ) y t with initial state y
= y .
It holds that
[Equation]
where
[Equation]
Here Υ K is deﬁned in (3.7).
Proof.
See § F.1 for a detailed proof.
The following lemma shows that J 1 ( K ) is gradient dominant.
Lemma D.2.
Let K ∗ be the optimal parameter and K be a parameter such that J 1 ( K ) < ∞ , then it holds that
[Equation]
Proof.
See § F.2 for a detailed proof.
Recall that from Algorithm 2, the parameter K is updated via
[Equation]
where (cid:98) Υ K n is the output of Algorithm 3.
We upper bound | J 1 ( K n +1 ) − J 1 ( K ∗ ) | in the sequel.
First, we show that if J 1 ( K n ) − J 1 ( K ∗ ) ≥ ε/ 2 holds for any n ≤ N , we obtain that
[Equation]
which holds with probability at least 1 − ε 13 .
We prove (D.19) by mathematical induction.
Suppose that [Equation] which holds for n = 0 . In what follows, we deﬁne (cid:101) K n +1 as
[Equation] [Equation] where Υ K n is given in (3.7). By (D.21), we have
[Equation]
where the ﬁrst equality comes from (D.15), the second equality comes from Lemma D.
[Equation]
and the last inequality comes from the trace inequality. By the deﬁnition of Υ K in (3.7), we obtain that
[Equation]
where the second inequality comes from Proposition B.2. By plugging (D.23) and the choice of stepsize γ ≤ [ (cid:107) R (cid:107) 2 + (cid:107) B [Equation] 22 · J
[Equation]
K 0 ) · σ − 1min (Ψ (cid:15) )] − 1 into (D.22)
[Equation]
we obtain that
[Equation]
where the last inequality comes from Lemma D.2.
The following lemma upper bounds | J 1 ( (cid:101) K n +1 ) − J 1 ( K n +1 ) | .
Lemma D.3.
Assume that J 1 ( K n ) ≤ J 1 ( K 0 ) .
It holds with probability at least 1 − ε 15 that
[Equation]
where K n +1 and (cid:101) K n +1 are deﬁned in (D.18) and (D.21), respectively.
Proof.
See § F.3 for a detailed proof.
Combining (D.24) and Lemma D.3, if J 1 ( K n ) − J 1 ( K ∗ ) ≥ ε/ 2 , it holds with probability at least 1 − ε 15 that
[Equation]
(D.25) Combining (D.20) and (D.25), it holds with probability at least 1 − ε 15 that
[Equation]
Finally, following from a union bound argument and the choice of N in Theorem B.4, if J 1 ( K n ) − J 1 ( K ∗ ) ≥ ε/ 2 holds for any n ≤ N , we have
[Equation]
which holds with probability at least 1 − ε 13 . Thus, we complete the proof of (D.19). Combining (D.24) and (D.25), for J 1 ( K n ) − J 1 ( K ∗ ) ≥ ε/ 2 , we have
[Equation]
which holds with probability at least 1 − ε {+
+}
Meanwhile, following from a union bound argument and the choice of N in Theorem B.4, it holds with probability at least 1 − ε 11 that
[Equation] The following lemma upper bounds (cid:107) K N − K ∗ (cid:107) F .
Lemma D.4.
For any K , we have
[Equation]
Proof.
See § F.4 for a detailed proof.
Combining (D.26) and Lemma D.4, we have
[Equation]
which holds with probability 1 − ε 11 .
Part 2.
We show that J 2 ( K N , b H ) − J 2 ( K ∗ , b ∗ ) <
ε/ 2 with high probability.
Following from
Proposition 3.4, it holds that J 2 ( K ∗ , b ∗ ) =
J 2 ( K N , b K N ) .
Therefore, it sufﬁces to show that J 2 ( K N , b H ) − J 2 ( K N , b K N ) <
ε/ 2 .
First, we show that if J 2 ( K N , b h ) − J 2 ( K N , b K N ) ≥ ε/ 2 for any h ≤ H , we obtain that
[Equation]
which holds with probability at least 1 − ε 13 . We prove (D.28) by mathematical induction. Suppose that [Equation]
Recall that by Algorithm 2, the parameter b is updated via
[Equation]
Here
[Equation]
D
[Equation]
31) where (cid:98) Υ K N and (cid:98) q K N ,b h are the outputs of Algorithm 3.
We deﬁne (cid:101)
b h +1 as
[Equation]
Here ∇
[Equation]
[Equation]
where Υ K N and q K N ,b h are deﬁned in (3.7).
We upper bound J 2 ( K N , b h +1 ) − J 2 ( K N , b K N ) in the sequel. Following from (D.32) and Proposition 3.3, we have
[Equation]
where ν K N is speciﬁed in Proposition 3.3.
The following lemma upper bounds | J 2 ( K N , b h +1 ) −
J 2 ( K N , (cid:101)
b h +1 ) | .
Lemma D.5.
Assume that J 2 ( K N , b h ) ≤ J 2 ( K N [Equation]
b 0 ) . It holds with probability at least 1 − ε 15 that
[Equation]
where b h +1 and (cid:101) b h +1 are deﬁned in (D.30) and (D.32), respectively.
Proof.
See § F.5 for a detailed proof.
Combining (D.34) and Lemma D.5, we know that if J 2 ( K N , b h ) − J 2 ( K N , b K N ) ≥ ε , it holds with probability at least 1 − ε 15 that
[Equation]
Combining (D.29) and (D.35), it holds with probability at least 1 − ε 15 that
[Equation]
Following from a union bound argument and the choice of H in Theorem B.4, if J 2 ( K N , b h ) − J 2 ( K N , b K N ) ≥ ε holds for any h ≤ H , we have [Equation]
which holds with probability at least 1 − ε {+
+}
Thus, we ﬁnish the proof of (D.28).
Combining (D.34) and Lemma D.5, for J 2 ( K N , b h ) − J 2 ( K N , b K N ) ≥ ε/ 2 , we have
[Equation]
{+which holds with probability at least 1 − ε 
 Meanwhile, following from a union bound argument and+} the choice of H in Theorem B.4, it holds with probability at least 1 − ε 11 that
[Equation]
By Proposition 3.3 and (D.36), it holds with probability at least 1 − ε 11 that [Equation] Following from Proposition 3.4, we know that [Equation]
Combining (D.27), (D.37), and (D.38), it holds with probability 1 − ε 10 that [Equation] where [Equation]
We ﬁnish the proof of the theorem.
D. P ROOF OF T HEOREM B.
Proof.
We follow the proof of Theorem 4.2 in Yang et al.
(2019), where they only consider LQR without drift terms.
Since our proof requires much more delicate analysis, we present it here.
Part 1.
We denote by (cid:98) ζ and (cid:98)
ξ the primal and dual variables generated by Algorithm 3.
We deﬁne the primal-dual gap of (B.11) as [Equation] D. [Equation]
In the sequel, we upper bound (cid:107) (cid:98) α K,b − α K,b (cid:107) 2 using (D.39).
We deﬁne ζ K,b and ξ ( ζ ) as
[Equation]
Following from (B.12), we know that
[Equation] The following lemma shows that ζ K,b ∈ V ζ and ξ ( ζ ) ∈ V ξ for any ζ ∈ V ζ .
Lemma D.6.
Under the assumptions in Theorem B.8, it holds that ζ K,b
= ( J ( K, b ) , α (cid:62) K,b ) (cid:62) ∈ V ζ .
Also, for any ζ ∈ V ζ , the vector ξ ( ζ ) deﬁned in (D.40) satisﬁes that ξ ( ζ ) ∈ V ξ .
Proof.
See § F.6 for a detailed proof.
By (B.12), we know that ∇ ζ F ( ζ K,b , 0)
=
and ∇ ξ F ( ζ K,b , 0) = 0 . Combining Lemma D.6, it holds that ( ζ K,b , 0) is a saddle point of the function F ( ζ, ξ ) deﬁned in (B.11).
Following from (D.39), it holds that
[Equation]
where the ﬁrst equality comes from (D.41), and the second equality comes from the fact that ξ ( (cid:98) ζ ) = argmax ξ ∈V ξ F ( (cid:98) ζ, ξ ) by (D.40) and Lemma D.6.
We upper bound the RHS of (D.42) and lower bound the LHS of (D.42) in the sequel.
As for the RHS of (D.42), it holds for any ξ ∈ V ξ that
[Equation]
=
[Equation]
where the ﬁrst equality comes from the fact that ξ ( ζ ) = argmax ξ ∈V ξ F (
[Equation]
, ξ ) by (D.40) and Lemma D.6, the second equality comes from (D.41), and the last equality holds by taking ζ = ζ K,b ∈ V ζ .
Meanwhile, we lower bound the LHS of (D.42) as
[Equation]
where the ﬁrst equality comes from the deﬁnition of (cid:101) Θ K,b in (B.9), and the ﬁrst inequality comes from Proposition B.6. Here λ K is deﬁned in Proposition B.6. Combining (D.42), (D.43), and (D.44), it holds that
[Equation]
which ﬁnishes the proof of this part.
Part 2.
We now upper bound gap ( (cid:98) ζ, (cid:98) ξ ) .
We denote by (cid:101) z t = (
[Equation]
(cid:62) t ) (cid:62) for t ∈ [ (cid:101) T ] , where (cid:101) x t and (cid:101)
u t are generated in Line 6 of Algorithm 3.
Following from the state transition in Problem 2.3 and the form of the linear policy, { (cid:101) z t } t ∈
[ (cid:101) T ] follows the following transition,
[Equation]
where
[Equation]
Note that we have
[Equation]
Then by the property of spectral radius
[Equation]
it holds that
[Equation]
Thus, the Markov chain generated by (D.46) admits a unique stationary distribution N ( µ z , Σ z ) , where
[Equation]
The following lemma characterizes the average
[Equation]
Lemma D.7.
It holds that [Equation]
where (cid:107) µ (cid:101) {+T (cid:107) 
+}
≤ M µ · (1 − ρ ) − 2 · (cid:107) µ z (cid:107) 2 and (cid:107) (cid:101) Σ (cid:101)
[Equation]
M
[Equation]
[Equation]
Here M µ and M Σ are positive absolute constants.
Moreover, it holds with probability at least 1 − (cid:101) T − 6 that
[Equation]
Proof.
See § F.7 for a detailed proof.
Lemma D.7 gives that [Equation] which holds with probability at least 1 − (cid:101) T − 6 . We now apply a truncation argument to show that gap ( (cid:98) ζ, (cid:98) ξ ) is upper bounded.
We deﬁne the event E in the sequel. Following from Lemma D.7, it holds for any z ∼ N ( µ z , Σ z ) that
[Equation]
By Lemma G.3, there exists a positive absolute constant C 0 such that
[Equation]
49) where we write (cid:101)
Σ z = Σ
z + 1 / (cid:101) T · (cid:101) Σ (cid:101) T for notational convenience.
By taking τ = C 1 · log T · (cid:107) (cid:101) Σ z (cid:107) F in (D.49) for a sufﬁciently large positive absolute constant C 1 , it holds that
[Equation]
We deﬁne the event E t, 1 for any t ∈
[ T ] as
[Equation]
Then by (D.50), it holds for any t ∈ [ T ] that [Equation]
Also, we deﬁne
[Equation]
Following from a union bound argument and (D.51), it holds that
[Equation]
Also, conditioning on E 1 , it holds for sufﬁciently large (cid:101) T that
[Equation]
where (cid:101) C 1 , C 2 , and C 3 are positive absolute constants.
Here, the ﬁrst inequality comes from the deﬁnition of E 1 in (D.52), the second inequality comes from Lemma D.7, and the third inequality comes from (D.47).
Also, we deﬁne the following event
[Equation]
Then by Lemma D.7, we know that [Equation] for (cid:101) T sufﬁciently large.
We deﬁne the event E as [Equation] Then following from (D.53), (D.56), and a union bound argument, we know that [Equation]
Now, we deﬁne the truncated feature vector (cid:101) ψ ( x, u ) as (cid:101) ψ
( x, u ) = (cid:98) ψ ( x, u ) 1 E , the truncated cost function (cid:101) c ( x, u ) as (cid:101) c
( x, u ) = c ( x, u ) 1 E , and also the truncated objective function (cid:101) F ( ζ, ξ ) as (cid:101)
[Equation]
where we write (cid:101) ψ
= (cid:101) ψ
( x, u ) and (cid:101) c
= (cid:101) c
( x, u ) for notational convenience.
Here the expectation is taken following the policy π K,b and the state transition.
The following lemma establishes the upper bound of | F ( ζ, ξ ) − (cid:101) F ( ζ, ξ ) | , where F ( ζ, ξ ) and (cid:101) F ( ζ, ξ ) are deﬁned in (B.11) and (D.57), respectively.
Lemma D.8.
It holds with probability at least 1 − (cid:101) T − 6 that
[Equation]
Proof.
See § F.8 for a detailed proof.
Following from (D.39) and Lemma D.8, it holds with probability at least 1 − (cid:101) T − 6 that
[Equation]
where we deﬁne (cid:102) gap ( (cid:98) ζ, (cid:98) ξ ) as
[Equation]
Therefore, to upper bound of gap ( ζ, ξ ) , we only need to upper bound (cid:102) gap ( ζ, ξ ) .
Part 3.
We upper bound (cid:102) gap ( ζ, ξ ) in the sequel.
We ﬁrst show that the trajectory generated by the policy π K,b and the state transition in Problem 2.2 is β -mixing.
Lemma D.9.
Consider a linear system y t +1 =
Dy t + ϑ + υ t , where { y t } t ≥ [-
-]0 ⊂ R m , the matrix
D ∈ R m × m satisfying ρ ( D ) < 1 , the vector ϑ ∈ R m , and υ t ∼ N (0 , Σ) is the Gaussians.
We denote by (cid:36) t the marginal distribution of y t for any t ≥ 0 .
Meanwhile, assume that the stationary distribution of { y t } t ≥ 0 is a Gaussian distribution
N (( I − D ) − 1 ϑ, Σ ∞ ) , where Σ ∞ is the covariance matrix.
We deﬁne the β -mixing coefﬁcients for any n ≥ 1 as follows
[Equation]
Then, for any ρ ∈ ( ρ ( D ) , 1) , the β -mixing coefﬁcients satisfy that
[Equation]
where C ρ,D,ϑ is a constant, which only depends on ρ , D , and ϑ .
We say that the sequence { y t } t ≥is β -mixing with parameter ρ .
Proof.
See Proposition 3.1 in Tu and Recht (2017) for details.
Recall that by (3.1), the sequence { x t } t ≥ 0 follows [Equation] where the matrix A − BK satisﬁes that ρ ( A − BK ) < 1 .
Therefore, by Lemma D.9, the sequence { z t } t ≥ 0 is β -mixing with parameter ρ ∈ ( ρ ( A − BK ) , 1) , where z t = ( x (cid:62) t , u (cid:62) t ) (cid:62) .
The following lemma upper bounds the primal-dual gap for a convex-concave problem.
Lemma D.10.
Let X
and Y be two compact and convex sets such that (cid:107) x − x (cid:48) (cid:107) 2 ≤ M and (cid:107) y − y (cid:48)
(cid:107) 2 ≤ M for any x, x (cid:48)
∈ X
and y, y (cid:
) ∈ Y .
We consider solving the following minimax problem
[Equation]
where the objective function H ( x, y ) is convex in x and concave in y .
In addition, we assume that the distribution (cid:36) (cid:15) is β -mixing with β ( n ) ≤ C (cid:15) · ρ n , where C (cid:15) is a constant.
Meanwhile, we assume that it holds almost surely that G ( x, y ; (cid:15) ) is (cid:101)
L 0 -Lipschitz in both x and y , the gradient ∇ x G ( x, y ; (cid:15) ) is (cid:101)
L 1 -Lipschitz in x for any y ∈ Y , the gradient ∇ y G ( x, y ; (cid:15) ) is (cid:101)
L 1 -Lipschitz in y for any x ∈ X , where C (cid:15) , (cid:101) L 0 , (cid:101)
L 1 > 1 .
Each step of our gradient-based method takes the following forms,
[Equation]
where the operators Γ X and Γ Y projects the variables back to X and Y , respectively, and the stepsizes take the form γ t = γ 0 · t − 1 / 2 for a constant γ 0 > 0 .
Moreover, let (cid:98) x
= ( (cid:80) Tt =1 γ t ) − 1 ( (cid:80)
Tt =1 γ t x t ) and (cid:98)
y
=
( (cid:80) Tt =1 γ t ) − 1 ( (cid:80)
Tt =1 γ t y t ) be the ﬁnal output of the gradient method after T iterations, then there exists a positive absolute constant C , such that for any δ ∈ (0 , 1) , the primal-dual gap to the minimax problem is upper bounded as
[Equation]
which holds with probability at least 1 − δ .
Proof.
See Theorem 5.4 in Yang et al.
for details.
To use Lemma D.10, we deﬁne the function G ( ζ, ξ ; (cid:101) ψ, (cid:101) ψ [Equation] ) as
[Equation]
, where (cid:101) ψ
= (cid:101) ψ
( x, u ) and (cid:101) ψ (cid:48) =
(cid:101) ψ ( x (cid:48) , u (cid:48) ) .
Note that the gradients of G ( ζ, ξ ; (cid:101) ψ, (cid:101) ψ (cid:48) ) take the form ∇ ζ G ( ζ, ξ ; (cid:101) ψ, (cid:101) ψ (cid:48) )
=
[Equation]
By Deﬁnition B.7 and Lemma D.6, we know that
[Equation]
This gives the Lipschitz constant (cid:101) L 0 in Lemma D.10 for G ( ζ, ξ ; (cid:101) ψ,
[Equation]
ψ (cid:48) ) . Also, the Hessians of G ( ζ, ξ ; (cid:101) ψ, (cid:101) ψ
[Equation]
) take the forms of
[Equation]
which follows that
[Equation]
This gives the Lipschitz constant (cid:101)
L
[Equation]
[Equation]
) and ∇ ξ G ( ζ, ξ ; (cid:101) ψ, (cid:101) ψ (cid:48) ) .
Moreover, note that (D.54) provides an upper bound of M , combining (D.59), (D.60) and Lemma
D.10, it holds with probability at least 1 − T − 5 that
[Equation]
gap [Equation] (D.61)
Combining (D.45), (D.58), and (D.61), we know that [Equation] Same bounds for (cid:107) (cid:98) Υ K − Υ K (cid:107)
[Equation]
F , (cid:107) (cid:98) p K,b − p K,b (cid:107) 22 , and (cid:107) (cid:98) q K,b − q K,b (cid:107) 22 hold.
We ﬁnish the proof of the theorem.
E P ROOFS OF P ROPOSITIONS E. P ROOF OF P ROPOSITION 3.
Proof.
We follow a similar proof as in the one of Theorem 1.1 in Sznitman (1991) and Theorem 3. in Bensoussan et al.
Note that for any policy π K,b ∈ Π , the parameters K and b uniquely determine the policy.
We deﬁne the following metric on Π .
Deﬁnition E.1.
For any π K 1 ,b 1 , π K 2 ,b 2 ∈ Π , we deﬁne the following metric,
[Equation]
where c 1 and c 2 are positive constants.
One can verify that Deﬁnition E.1 satisﬁes the requirement of being a metric.
We ﬁrst evaluate the forms of the operators Λ 1 ( · ) and Λ 2 ( · , · ) . Forms of the operators Λ 1 ( · ) and Λ 2 ( · , · ) . By the deﬁnition of Λ 1 ( µ ) , which gives the optimal policy under the mean-ﬁeld state µ , it holds that [Equation]
where π ∗ µ solves Problem 2.2.
This gives the form of Λ 1 ( · ) .
We now turn to Λ 2 ( µ, π ) , which gives the mean-ﬁeld state µ new generated by the policy π under the current mean-ﬁeld state µ .
In Problem
2.2, the sequence of states { x t } t ≥ 0 constitutes a Markov chain, which admits a unique stationary distribution.
Thus, by the state transition in Problem 2.2 and the form of the linear-Gaussian policy, we have
[Equation]
where K π and b π are parameters of the policy π .
By solving (E.1) for µ new , it holds that
[Equation]
This gives the form of Λ 2 ( · , · ) .
Next, we compute the Lipschitz constants for Λ 1 ( · ) and Λ 2 ( · , · ) .
Lipschitz constant for Λ 1 ( · ) .
By Proposition 3.4, for any µ 1 , µ 2 ∈ R m , the optimal K ∗ is ﬁxed for Problem 2.2.
Therefore, by the form of the optimal b K given in Proposition 3.4, it holds that
[Equation]
where L 1 is deﬁned in Assumption 3.1.
Lipschitz constants for Λ 2 ( · , · ) .
By Proposition 3.4, for any µ 1 , µ 2 ∈ R m , the optimal K ∗ is ﬁxed for Problem 2.2.
Thus, for any π ∈ Π such that π is an optimal policy under some µ ∈ R m , it holds that
[Equation]
where L 2 is deﬁned in Assumption 3.1, and K π = K ∗ is the parameter of the policy π .
Meanwhile, for any mean-ﬁeld state µ ∈ R m , and any poicies π 1 , π 2 ∈ Π that are optimal under some mean-ﬁeld states µ 1 , µ 2 , respectively, we have
[Equation]
where in the last equality, we use the fact that K π 1 = K π 2 = K ∗ by Proposition 3.4.
Here L 3 is deﬁned in Assumption 3.1, and b π 1 and b π 2 are the parameters of the policies π 1 and π 2 .
Now we show that the operator Λ( · ) is a contraction.
For any µ 1 , µ 2 ∈ R m , it holds that
[Equation]
where the ﬁrst inequality comes from triangle inequality, the second inequality comes from (E.3) and (E.4), and the last inequality comes from (E.2).
By Assumption 3.1, we know that L 0 =
L 1 L 3 + L 2 < 1 , which shows that the operator Λ( · ) is a contraction.
Moreover, by Banach ﬁxedpoint theorem, we obtain that Λ( · ) has a unique ﬁxed point, which gives the unique equilibrium pair of Problem 2.3. We ﬁnish the proof of the proposition.
E. P ROOF OF P ROPOSITION 3.
Proof.
By the deﬁnition of J 2 ( K, b ) in (3.6) and the deﬁnition of µ K,b in (3.2), the problem [Equation]
is equivalent to the following constrained problem,
[Equation]
Following from the KKT conditions of (E.5) [Equation] it holds that
[Equation]
where
[Equation]
By solving (E.6)
[Equation]
the minimizer of (E.5) takes the form of
[Equation]
By substituting (E.7) into the deﬁnition of J 2 ( K, b ) in (3.6)
[Equation]
we have
[Equation]
Meanwhile, by calculation, we have
[Equation]
Therefore, the term N (cid:62) K M − 1 K N K in (E.8) takes the form of
[Equation]
By plugging (E.9) into (E.8), we have
[Equation]
Also, by plugging (E.9) into (E.7), we have
[Equation]
We ﬁnish the proof of the proposition
[Equation]
E. P ROOF OF P ROPOSITION B.
Proof.
By the deﬁnition of the cost function c ( x, u ) in Problem 2.2 (recall that we drop the subscript µ when we focus on Problem 2.2)
[Equation]
we have
[Equation]
10) where we write c t = c ( x t , u t ) for notational convenience. Here in the second line we use u t = π K,b ( x t ) = − Kx t + b + ση t .
Therefore, combining (E.10) and the deﬁnition of J ( K, b ) in Problem 2.
[Equation]
we have
[Equation]
Now, by iteratively applying (3.3) and (3.4), we have
[Equation]
where P K is given in (3.4).
Combining (E.11) and (E.12), we know that
[Equation]
where
[Equation]
We ﬁnish the proof of the proposition [Equation] E. P ROOF OF P ROPOSITION 3.
Proof. By calculating the Hessian matrix of J 2 ( K, b ) , we have
[Equation]
which is a positive deﬁnite matrix independent of b . We denote by its minimum singular value as ν K . Also, note that (cid:107)∇ 2 bb J 2 ( K, b ) (cid:107) 2 is upper bounded as
[Equation]
Therefore
[Equation]
it holds that [Equation] where ι K is the maximum singular value of ∇ 2 bb J 2 ( K, b ) .
We ﬁnish the proof of the proposition
[Equation]
E. P ROOF OF P ROPOSITION B.
Proof.
Following from Proposition B.2, it holds that [Equation] where f K ( y ) = y (cid:62) P K y .
By the deﬁnition of P K in (3.4), we obtain that
[Equation]
(E
[Equation]
Also, we have ∇
[Equation]
By plugging (E.15) into (E.14), we have [Equation] [Equation] By iteratively applying (E.16), it holds that
[Equation]
[Equation] where y t +1 = ( A − BK )
{+y t with y 
 = y . Now, combining (E.13) and (E.17), it holds that+}
[Equation]
where Υ K is deﬁned in (3.7). Meanwhile, combining the form of µ K,b in (3.2), it holds by calculation that [Equation] where q K,b is deﬁned in (3.7).
We ﬁnish the proof of the proposition.
E. P ROOF OF P ROPOSITION B.
Proof.
From the deﬁnition of V K,b ( x ) in (B.1) and the deﬁnition of the cost function c ( x, u ) in
Problem 2.2, it holds that
[Equation]
Combining (3.1)
[Equation]
we know that V K,b ( x ) is a quadratic function taking the form of V K,b ( x ) =
x (cid:62) Gx + r (cid:62) x
+ h , where G , r , and h are functions of K and b .
Note that V K,b ( x ) satisﬁes that
[Equation]
by substituting the form of c ( x
[Equation]
− Kx + b ) in Problem 2.2 and J ( K, b ) in (3.5) into (E.18)
[Equation]
we obtain that
[Equation]
By comparing the quadratic terms and linear terms on both the LHS and RHS in (E.19), we obtain that
[Equation]
where f K,b = ( I − A + BK ) −(cid:62) [( A − BK ) (cid:62) P K ( Bb + Aµ + d ) − K (cid:62) Rb ] . Also, by the deﬁnition of V K,b ( x ) in (B.1), we know that E [ V K,b ( x )] = 0 , where the expectation is taken following the stationary distribution generated by the policy π K,b and the state transition.
Therefore, we have
[Equation]
which shows that [Equation]
For the action-value function Q K,b ( x, u ) , by plugging (E.20) into (B.2), we obtain that [Equation]
We ﬁnish the proof of the proposition.
E. P ROOF OF P ROPOSITION B.
Proof.
By Proposition B.1, it holds that Q K,b takes the following linear form
[Equation]
where β K,b is a scalar independent of x and u .
Note that Q K,b ( x, u ) satisﬁes that
[Equation]
) where ( x (cid:48) , u (cid:48) ) is the state-action pair after ( x, u ) following the policy π K,b and the state transition.
Combining (E.21) and (E.22), we obtain that
[Equation]
By left multiplying ψ ( x, u ) to both sides of (E.23), and taking the expectation, we have
[Equation]
Combining the deﬁnition of the matrix Θ K,b in (B.7), we have
[Equation]
which concludes the proof of the proposition.
E. P ROOF OF P ROPOSITION B.
Proof.
Invertibility and Upper Bound.
We denote by z t =
( x (cid:62) t , u (cid:62) t ) (cid:62) for any t ≥ 0 .
Then following from the state transition and the policy π K,b , the transition of { z t } t ≥ 0 takes the form of
[Equation]
24) where L , ν and δ are deﬁned as
[Equation]
Note that L also takes the form of
[Equation]
(cid:18)
I
[Equation]
(cid:19) ( A B ) .
Combining the fact that ρ ( UV ) =
ρ ( V U ) for any matrices
U and V , we know that ρ ( L ) =
ρ ( A −
BK ) < 1 , which veriﬁes the stability of (E.24).
Following from the stability of (E.24), we know that the Markov chain generated by (E.24) admits a unique stationary distribution N ( µ z , Σ z ) , where µ z and Σ z satisfy that [Equation]
ν
[Equation]
where
[Equation]
Also, we know that Σ z takes the form of Σ
[Equation]
= Cov (cid:34)(cid:18)
[Equation]
where Φ K is deﬁned in (3.3).
The following lemma establishes the form of Θ K,b .
Lemma E.2.
The matrix Θ K,b in (B.7) takes the form of
[Equation]
Proof.
See § F.9 for a detailed proof.
Note that since ρ ( L ) < 1 , both I − L ⊗ s L and I − L are positive deﬁnite.
Therefore, by Lemma E.2, the matrix Θ K,b is invertible.
This ﬁnishes the proof of the invertibility of Θ K,b .
Moreover, by (E.25) and Lemma E.2, we upper bound the spectral norm of Θ K,b as
[Equation]
which proves the upper bound of (cid:107) Θ K,b (cid:107) 2 .
Minimum singular value.
To lower bound σ min ( (cid:101) Θ K,b ) , we only need to upper bound σ max ( (cid:101)
Θ − 1 K,b ) = (cid:107) (cid:101)
Θ − 1 K,b (cid:107) 2 .
We ﬁrst calculate (cid:101)
Θ − 1 K,b .
Recall that the matrix (cid:101) Θ K,b in (B.8) takes the form of
[Equation]
By the deﬁnition of the feature vector ψ ( x, u ) in (B.5), the vector (cid:101) σ z [Equation] K,b [ ψ ( x, u )] takes the form of
[Equation]
where 0 k + m denotes the all-zero column vector with dimension k + m .
Also, since Θ K,b is invertible, the matrix (cid:101)
Θ K,b is also invertible, whose inverse takes the form of
[Equation]
The following lemma upper bounds the spectral norm of (cid:101)
Θ − 1 K,b .
Lemma E.3.
The spectral norm of the matrix (cid:101) Θ − 1 K,b is upper bounded by a positive constant (cid:101) λ K , where (cid:101) λ K only depends on (cid:107) K (cid:107) 2 and ρ ( A − BK ) .
Proof.
See § F.10 for a detailed proof.
By Lemma E.3, we know that σ min ( (cid:101) Θ K,b ) is lower bounded by a positive constant λ K = 1 / (cid:101) λ K , which only depends on (cid:107) K (cid:107) 2 and ρ ( A − BK ) .
This concludes the proof of the proposition.
F P ROOFS OF L EMMAS F. P ROOF OF L EMMA D.
Proof.
Following from (3.4) [Equation] it holds that
[Equation]
Meanwhile, by the state transition y t +
[Equation]
( A − BK 2 ) y t , we know that
[Equation]
By plugging (F.2) into (F.1), it holds that
[Equation]
Also
[Equation]
it holds that
[Equation]
Combining (F.3) and (F.4), we have [Equation] y (cid:62) t P K 1 y t ) .
(F.5)
Also, by the state transition y t +1 =
( A − BK 2 ) y t , it holds for any t ≥ 0 that y
[Equation]
t + y
[Equation]
t K
[Equation]
RK 2 y t + y
[Equation]
t +1 P K 1 y t
+1 − y
[Equation]
t P K 1 y t = y
[Equation]
t
[Equation]
[Equation]
[Equation]
t
[Equation]
y
[Equation]
(F
[Equation]
where the matrix Υ K 1 is deﬁned in (3.7).
By plugging (F.6) into (F.5), we have
[Equation]
where D K 1 ,K 2 ( y )
= 2 y
[Equation]
( K 2 − K 1 )(Υ 22 K 1 K 1 − Υ 21 K 1 ) y
+ y (cid:62) ( K 2 − K 1 ) (cid:62) Υ 22 K 1 ( K 2 − K 1 ) y .
We ﬁnish the proof of the lemma.
F. P ROOF OF L EMMA D.
Proof.
We prove (D.16) and (D.17) separately in the sequel. Proof of (D.16) .
From the deﬁnition of J 1 ( K ) in (3.6), we have
[Equation]
where in the last equality, we apply Lemma D.1 and the expectation is taken following the transition y t
+1 =
( A − BK ∗ )
y t with initial state y 0 ∼ N (0 , Ψ (cid:15) ) . Here we denote by D K,K ∗ ( y ) as [Equation]
Also, we write D K,K ∗ ( y ) as
[Equation]
Note that the ﬁrst term on the RHS of (F.8) is positive, due to the fact that it is a quadratic form of a positive deﬁnite matrix, we lower bound D K,K ∗ ( y ) as
[Equation]
Combining (F.7) and (F.9), it holds that
[Equation]
where the last line comes from the fact that Υ 22 K
= R + B (cid:62) P K B (cid:23) R .
This complete the proof of (D.16).
Proof of (D.17) .
Note that for any (cid:101) K , it holds by the optimality of K ∗ that
[Equation]
where the expectation is taken following the transition y t
+1 =
( A − B (cid:101) K )
y t with initial state y 0 ∼
N (0 , Ψ (cid:15) ) . By taking (cid:101) K = K − (Υ 22 K ) − 1 (Υ 22 K K − Υ 21 K ) and following from a similar calculation as in (F.8), the function D K, (cid:101) K ( y ) takes the form of
[Equation]
− Υ 21 K ) y. (F.11) Combining (F.10) and (F.11), it holds that
[Equation]
where we use the fact that Φ (cid:101) K = [Equation] A − B
[Equation]
B
[Equation] + Ψ [Equation] (cid:23) Ψ
[Equation]
in the last line.
This ﬁnishes the proof of (D.17).
F. P ROOF OF L EMMA D.
Proof.
By Proposition B.2, we have
[Equation]
The following lemma upper bounds the term (cid:107) P (cid:101) K n +1 − P K n +1 (cid:107) 2 .
Lemma F.1.
Suppose that the parameters K and (cid:101) K satisfy that
[Equation]
then it holds that
[Equation]
Proof.
See Lemma 5.7 in Yang et al.
for a detailed proof.
To use Lemma F
[Equation]
[Equation]
it sufﬁces to verify that (cid:101) K n +1 and K n +1 satisfy (F
[Equation]
Note that from the deﬁnitions of K n +1 and (cid:101) K n +1 in (D.18) and (D
[Equation]
21), respectively, we have
[Equation]
F
[Equation]
Now, we upper bound the RHS of (F.15).
For the term (cid:107) A − B (cid:101) K n +1 (cid:107) 2 , it holds by the deﬁnition of (cid:101) K n +1 in (D.21)
that
[Equation]
By the deﬁnition of Υ K n in (3.7)
[Equation]
we upper bound (cid:107) Υ K n (cid:107) 2 as
[Equation]
where the last line comes from the fact that
[Equation]
As for the term (cid:107) Φ (cid:101) K n +1 (cid:107) 2 in (F.15), from the fact that
[Equation]
it holds that
[Equation]
Therefore, combining (F.15), (F.16), (F.17), and (F.18), we know that
[Equation]
From Theorem B.8, it holds with probability at least 1 − T − 4 n −
[Equation]
T
[Equation]
n that
[Equation]
which holds for any ρ ∈ ( ρ ( A − BK n ) , 1)
Note that from the choice of T n and (cid:101) T n in the statement of Theorem B.4 that
[Equation]
it holds that
[Equation]
[Table]
[Equation]
[Table]
Combining (F.19), (F.20), and (F.21), we know that (F.13) holds with probability at least 1 − εfor sufﬁciently small ε > 0 .
Meanwhile, by (F.16), (F.17), and (F.18), the RHS of (F.14) is upper bounded as
[Equation]
Now, by Lemma F.1, it holds with probability at least 1 − ε 15 that
[Equation]
where the second inequality comes from (F.22), and the last inequality comes from (F.20) and (F.21).
Combining (F.12) and (F.23), it holds with probability at least 1 − ε 15 that
[Equation]
which concludes the proof of the lemma.
Proof. Note that
Υ 22 K ∗ K ∗ − Υ
21 K ∗ is the natural gradient of J 1 at the minimizer K ∗ , which implies that
[Equation]
By Lemma D.
[Equation]
it holds that [Equation] where we use (F.24) in the last line. Here the expectations are taken following the transition y t
+1 =
( A − BK )
y t with initial state y 0 ∼ N (0 , Ψ (cid:15) ) . Also, we have [Equation]
where we use the fact that Φ K = ( A − BK ) Φ K ( A − BK )
+Ψ
[Equation] Ψ [Equation] and Υ 22 K ∗ =
R + B
[Equation]
P K ∗ B
[Equation]
R in the last line. Combining (F.25) and (F.26), we have [Equation] We conclude the proof of the lemma.
F. P ROOF OF L EMMA D.
Proof.
Following from Proposition 3.3, we have
[Equation]
where ν K N and ι K N are deﬁned in Proposition 3.3.
Also, following from Proposition B
[Equation]
[Equation] it holds that
[Equation]
Combining (F.27), (F.28), and the fact that ν
[Equation]
we know that
[Equation]
Note that from the deﬁnition of (cid:98) ∇ b J 2 ( K N , b h ) and ∇ b J 2 ( K N , b h ) in (D.31) and (D.33), respectively, it holds by triangle inequality that
[Equation]
22 K N − Υ 22 K N (cid:107)
[Equation]
· (cid:107) b
[Equation]
(cid:107)
[Equation]
(cid:107) (cid:98)
Υ
21 K N − Υ 21 K N (cid:107) 2 · (cid:107) (cid:98) µ K N
[Equation]
b h (cid:107)
[Equation]
+ (cid:107) (cid:98) q K N [Equation] b h − q K N
[Equation]
b h (cid:107)+ (cid:107) Υ 21 K N (cid:107) 2 · (cid:107) (cid:98) µ K N [Equation] b h − µ K N
[Equation]
b h (cid:107) 2 .
By Theorem B
[Equation]
8, combining the fact that J 2 ( K N , b h ) ≤ J 2 ( K N , b 0 ) and the fact that (cid:107) µ K N ,b (cid:107) 2 ≤ J ( K N , b 0 ) /σ min ( Q ) , we know that with probability at least 1 − ( T b n ) − 4 − ( (cid:101) T b n ) − 6 , it holds for any ρ ∈ ( ρ ( A − BK N ) , 1) that [Equation]
[Equation]
Following from the choices of γ b , T bn , and (cid:101) T bn in the statement of Theorem B.4, it holds that [Table]
[Equation]
[Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table]
[Equation]
[Table]
Further combining (F.29) and (F.30)
[Table]
it holds with probability at least 1 − ε 15 that
[Equation]
We then ﬁnish the proof of the lemma.
F. P ROOF OF L EMMA D.
Proof.
We show that ζ K,b ∈ V ζ and ξ ( ζ ) ∈ V ξ for any ζ ∈ V ζ separately.
Part 1.
First we show that ζ K,b ∈ V ζ .
Note that from Deﬁnition B.7, we know that ζ 1 K,b
= J ( K, b ) satisﬁes that 0 ≤ ζ 1 K,b ≤ J ( K 0 , b 0 ) .
It remains to show that ζ 2 K,b
= α K,b satisﬁes that (cid:107) ζ 2 K,b (cid:107) 2 ≤ M ζ .
By the deﬁnition of α K,b in (B.6), we know that
[Equation]
where f K,b = ( I − A + BK ) −(cid:62) [( A − BK ) (cid:62) P K ( Bb + Aµ + d ) − K (cid:62) Rb ] and for notational simplicity, we denote by µ uK,b = − Kµ K,b + b .
We only need to bound Υ K , µ K,b , µ uK,b , P K , and f K,b .
Note that by Proposition B.2, the expected total cost J ( K, b ) takes the form of
[Equation]
Thus, we have
[Equation]
which imply that
[Equation]
For Υ K , it holds that
[Equation]
which gives
[Equation]
{+· (cid:107) P K (cid:107) F , (cid:107) Υ K (cid:107) 2 ≤ ( (cid:107) Q (cid:107) 
+}
+ (cid:107) R (cid:107) 2 )
{++ (cid:0) (cid:107) A (cid:107) 
+}
+ (cid:107) B (cid:107) 2 (cid:1) 2 · (cid:107) P K (cid:107) 2 .
Combining (F.32) and the fact that (cid:107) P K (cid:107) F ≤ √ m · (cid:107) P K (cid:107) 2 , we know that (cid:107) Υ K [Equation] ≤
[Equation]
(cid:107) [Equation] F + [Equation]
F [Equation] + (cid:0) (cid:107) A (cid:107) 2 F + (cid:107) B (cid:107) 2 F (cid:1) · √ m · J ( K 0 , b 0 ) /σ min (Ψ ω ) , (cid:107) Υ K (cid:107) 2 ≤ (cid:0) (cid:107) Q (cid:107) 2 + (cid:107) R (cid:107) 2 (cid:1) + (cid:0) (cid:107) A (cid:107) 2 + (cid:107) B (cid:107) 2 (cid:1) 2 · J ( K 0 , b 0 ) /σ min
(Ψ ω ) .
(F.33)
Now, we upper bound the vector f K,b .
Note that by algebra, the vector f K,b takes the form of
[Equation]
Therefore, we upper bound f K,b as
[Equation]
Combining (F.31), (F.32), (F.33)
[Equation]
and (F.34), it holds that
[Equation]
Therefore, it holds that ζ K,b ∈ V ζ .
Part 2.
Now we show that for any ζ ∈ V ζ , we have ξ ( ζ ) ∈ V ξ .
Recall that from (D.41), it holds that ξ
[Equation]
( K,
[Equation]
[Equation]
Then we have
[Equation]
where we use the fact that since ζ ∈ V ζ , we have 0 ≤ ζ 1 ≤ J ( K 0 , b 0 ) by Deﬁnition B.7.
Also, by (F.35), we have
[Equation]
Note that we upper bound B 1 as
[Equation]
Following from the deﬁnition of ψ ( x, u ) in (B.5), we know that
[Equation]
where Σ z is deﬁned as Σ z = Cov (cid:34)(cid:18) xu (cid:19)(cid:35) [Equation] Combining (F.38) and (F.39), we have
[Equation]
By Proposition B.
[Equation]
we upper bound B 2 as
[Equation]
where we use the fact that ζ ∈ V ζ and Deﬁnition B
[Equation]
As for the term B 3 in (F.37)
[Equation]
we utilize the following lemma to provide an upper bound.
Lemma F.2.
The vector E π K,b [ c ( x, u ) ψ ( x, u )] takes the following form,
[Equation]
Here the matrix Σ z takes the form of
[Equation]
Proof.
See § F.11 for a detailed proof.
From Lemma F.2 and (F.32), it holds that
[Equation]
Moreover, by the deﬁnition of Σ z in (E.25), combining the triangle inequality
[Equation]
we have the following bounds for (cid:107) Σ z (cid:107) F and (cid:107) Σ z (cid:107) 2 ,
[Equation]
Also, we have
[Equation]
which gives the upper bound for Φ K as follows,
[Equation]
Therefore, combining (F.37), (F.40), (F.41), (F.42), (F.43)
[Equation]
and (F.44), we know that
[Equation]
By (F.36) and (F.45), we know that ξ ( ζ ) ∈ V ξ for any ζ ∈ V ζ .
We conclude the proof of the lemma.
F. P ROOF OF L EMMA D.
Proof.
Assume that (cid:101)
z
∼ N ( µ † , Σ † ) .
Following from the fact that
[Equation]
it holds that
[Equation]
F [Equation] [Equation] where Ψ δ = (cid:18)
[Equation]
Ψ ω
[Equation] (cid:62)
[Equation]
(cid:19) .
From (D.47), we know that µ z takes the form of µ
[Equation]
(F.47) Therefore, combining (F.46) and (F.47), we have
[Equation]
[Equation]
We denote by
[Equation]
[Table]
L
[Table]
L
[Table]
ν.
Meanwhile, it holds that
[Equation]
where M µ is a positive absolute constant.
For the covariance, note that for any random variables X ∼ N ( µ 1 , Σ 1 ) and Y ∼ N ( µ 2 , Σ 2 ) , we know that
Z
= X + Y ∼ N ( µ 1 + µ 2 , Σ) , where (cid:107) Σ (cid:107) F ≤ 2 (cid:107) Σ 1 (cid:107) F + 2 (cid:107) Σ 2 (cid:107) F . Combining (F.46), we know that (cid:98) µ z ∼ N ( E (cid:98) µ z , (cid:101) Σ (cid:101) T / (cid:101) T ) , where (cid:101) Σ (cid:101) T satisﬁes that
[Equation]
T
[Equation]
T
[Equation]
F ≤
[Equation]
which implies that
[Equation]
where M Σ is a positive absolute constant.
Combining (F.48), (F.49), and (F.50), we obtain that [Equation] , where (cid:107) µ (cid:101) T (cid:107) 2 ≤ M µ · (1 − ρ ) − 2 · (cid:107) µ z (cid:107) 2 and (cid:107) (cid:101) Σ (cid:101) T (cid:107) F ≤ M Σ · (1 − ρ ) − 1 · (cid:107) Σ z (cid:107) F .
Moreover, by the Gaussian tail inequality, it holds with probability at least 1 − (cid:101) T − 6 that [Equation]
Then we ﬁnish the proof of the lemma.
F. P ROOF OF L EMMA D.
Proof.
We continue using the notations given in § D.3.
We deﬁne
[Equation]
where (cid:98) ψ = (cid:98) ψ ( x, u ) is the estimated feature vector.
Here the expectation is only taken over the trajectory generated by the state transition and the policy π K,b , conditioning on the randomness induced when calculating the estimated feature vectors.
Thus, the function (cid:98) F ( ζ, ξ ) is still random, where the randomness comes from the estimated feature vectors.
Note that | F ( ζ, ξ ) − (cid:101) F ( ζ, ξ ) | ≤ | F ( ζ, ξ ) − (cid:98) F ( ζ, ξ ) | + | (cid:98) F ( ζ, ξ ) − (cid:101) F ( ζ, ξ ) | .
Thus, we only need to upper bound | F ( ζ, ξ ) − (cid:98) F ( ζ, ξ ) | and | (cid:98) F ( ζ, ξ ) − (cid:101) F ( ζ, ξ ) | .
Part 1.
First we upper bound | F ( ζ, ξ ) − (cid:98) F ( ζ, ξ ) | .
Note that by algebra, we have
[Equation]
(F.
[Equation]
where the expectation is only taken over the trajectory generated by the state transition and the policy π K,b .
From Lemma D.7, it holds that[-with probability at least 1 − (cid:101) T − 6 , the term E ( (cid:107) ψ − ψ (cid:48) (cid:107) 
 + 2 (cid:107) (cid:98) ψ (cid:107) 2 ) is upper bounded by a polynomial poly ( (cid:107) Φ K (cid:107) ∗ , (cid:107) K-]
[Equation]
Therefore, combining (F.52), it holds with probability at least 1 − (cid:101) T − 6 that [Equation]
where the expectation is conditioned on the randomness induced when calculating the estimated feature vectors. Also, we know that [Equation]
Therefore, combining (F.51), (F.53), (F.54), and Deﬁnition B.7, it holds with probability at least 1 − (cid:101) T − 6 that [Equation]
[Equation]
Following from the deﬁnitions of ψ ( x, u ) in (B.5) and (cid:98) ψ ( x, u ) in (B.14), we upper bound (cid:107) ψ ( x, u ) − (cid:98) ψ ( x, u ) (cid:107) 2 for any x and u as [Equation] ψ
[Equation]
u
[Equation]
−
[Equation] 22 = [Equation] 56) where µ z is deﬁned in (D.47), (cid:98) µ z is deﬁned in (D.48), and z = ( x (cid:62) , u (cid:62) ) (cid:62) .
Also, by Lemma D.7, we know that
[Equation]
which holds with probability at least 1 − (cid:101) T − 6 .
Combining (F.55), (F.56), and (F.57), it holds with probability at least 1 − (cid:101) T − 6 that
[Equation]
Part 2.
We now upper bound | (cid:98) F ( ζ, ξ ) − (cid:101) F ( ζ, ξ ) | in the sequel.
By deﬁnitions, we have
[Equation]
where we deﬁne the event E (cid:48) as
[Equation]
where E 2 is deﬁned in (D.55).
Combining the fact that P ( E 2 ) ≥ 1 − (cid:101) T − 6 and Lemma G.3, it holds that P ( E (cid:48) ) ≥
[Equation]
T
[Equation]
5 −
[Equation] T [Equation] Following a similar argument as in Part 1 , it holds from (F.59) that
[Equation]
for sufﬁciently large T and (cid:101) T .
Now, combining (F.58) and (F.60), by triangle inequality, it holds with probability at least 1 − (cid:101) T −that
[Equation]
We ﬁnish the proof of the lemma.
F. P ROOF OF L EMMA E.
Proof.
Recall that the feature vector ψ ( x, u ) takes the following form
[Equation]
We then have
[Equation]
where we denote by y =
z − µ z
[Equation]
and ( x (cid:48) , u (cid:48) ) is the state-action pair after ( x, u ) following the state transition and the policy π K,b .
Therefore, for any symmetric matrices M
[Equation]
N and any vectors m
[Equation]
n , it holds from (B.7) and (F.61) that
[Equation]
where the expectations are taken over y ∼ N (0 , Σ z ) and δ ∼ N (0 , Ψ δ ) .
We evaluate the terms A 1 ,
A 2 , A 3 , and A 4 in the sequel.
For the terms A 2 and A 3 in (F.62), by the fact that y =
z − µ z ∼ N (0 , Σ z ) , we know that these two terms vanish.
For A 4 , it holds that
[Equation]
For A 1 , by algebra
[Equation]
we have
[Equation]
Now, by applying Lemma G.1 to the ﬁrst term on the RHS of (F.64)
[Equation]
we know that
[Equation]
where we use the fact that Σ z = L Σ z L (cid:62) + Ψ δ in the last equality.
By using the property of the operator svec ( · ) and the deﬁnition of the symmetric Kronecker product
[Equation]
we obtain that
[Equation]
(F.65)
Combining (F.62), (F.63), and (F.65), we obtain that
[Equation]
Thus, the matrix Θ K,b takes the following form,
[Equation]
which concludes the proof of the lemma.
F. P ROOF OF L EMMA E.
Proof.
From the deﬁnition of (cid:101) Θ K,b in (B.9), it holds that [Equation] where (cid:101) σ z is deﬁned as
[Equation]
We bound the RHS of (F.66) in the sequel.
For the term Θ − 1 K,b (cid:101) σ z , combining Lemma E.2, we have
[Equation]
where we use the property of the symmetric Kronecker product in the second and last line
[Equation] By taking the spectral norm on both sides of (F.67)
[Equation]
it holds that
[Equation]
where in the third line we use Lemma G.2 to the matrix L ⊗ s L .
Similarly, we upper bound (cid:107) Θ − 1 K,b (cid:107)in the sequel
[Equation]
Thus, combining (F.66)
[Equation]
(F.68)
[Equation]
and (F.69)
[Equation]
we obtain that
[Equation]
Now it remains to characterize σ min (Σ z ) .
For any vectors s ∈ R m and r ∈ R k , we have
[Equation]
The ﬁrst term on the RHS of (F.71) is lower bounded as
[Equation]
where the last inequality comes from the fact that σ min (Φ K ) ≥ σ
[Equation]
(Ψ
[Equation]
) by (3.3).
The second term on the RHS of (F.71) takes the form of
[Equation]
Therefore, combining (F.71), (F.72), and (F.73), we have
[Equation]
From this, we know that
[Equation]
F
[Equation]
Thus, combining (F.70) and (F.74), we know that (cid:107) (cid:101) Θ − 1 K,b (cid:107) 2 is upper bounded by a constant (cid:101) λ K , where (cid:101)
λ K only depends on (cid:107) K (cid:107) 2 and ρ ( L )
=
ρ ( A − BK ) .
This ﬁnishes the proof of the lemma.
F.
P ROOF OF L EMMA F.
Proof.
First, note that the cost function c ( x, u ) takes the following form,
[Equation]
For any matrix V and vectors v x , v u , it holds that
[Equation]
In the sequel
[Equation]
we calculate D 1 and D 2 respectively.
Calculation of D 1 .
Note that by the deﬁnition of ψ ( x, u ) in (B.5), it holds that
[Equation]
Here z =
( x (cid:62) , u (cid:62) )
(cid:62) and µ z = E π K,b ( z ) .
For the ﬁrst term on the RHS of (F.76), note that z − µ z ∼ N (0 , Σ z ) . Therefore, by Lemma G.1, we obtain that [Equation] F.
[Equation]
Meanwhile, the second term on the RHS of (F.76) takes the form of E [Equation] b
[Equation]
Combining (F.76), (F.77)
[Equation] and (F.78) [Equation] we obtain that
[Equation]
Calculation of D 2 . By the deﬁnition of the feature vector ψ ( x, u ) in (B.5), we know that
[Equation]
k
[Equation]
[Equation]
F.80) Now, combining (F.75), (F.79) [Equation] and (F.80), it holds that
[Equation]
which concludes the proof of the lemma
[Equation]
G A UXILIARY R ESULTS
Lemma G.1.
Assume that the random variable w ∼ N (0 , I ) [Equation] and let U and V be two symmetric matrices, then it holds that
[Equation]
Proof.
See Magnus et al.
(1978) and Magnus (1979) for a detailed proof.
Lemma G.2.
Let M , N be commuting symmetric matrices, and let α 1 , . .
. , α n , β 1 , . .
, β n denote their eigenvalues with v 1 , . .
. , v n a common basis of orthogonal eigenvectors.
Then the n ( n + 1) /eigenvalues of M ⊗ s N are given by (
α i β j + α j β i ) / 2 , where 1 ≤ i ≤ j ≤ n .
Proof.
See Lemma 2 in Alizadeh et al.
(1998) for a detailed proof.
Lemma G.3.
For any integer m > 0 , let A ∈ R m × m and η ∼ N (0 , I m ) .
Then, there exists some absolute constant C > 0 such that for any t ≥ 0 , we have
[Equation]
Proof.
See Rudelson et al.
for a detailed proof.