A CTOR -C RITIC P ROVABLY F INDS N ASH E QUILIBRIA OF L INEAR -Q UADRATIC M EAN -F IELD G AMES
Anonymous authors
Paper under double-blind review
A BSTRACT
We study discrete-time mean-ﬁeld Markov games with inﬁnite numbers of agents where each agent aims to minimize its ergodic cost.
We consider the setting where the agents have identical linear state transitions and quadratic cost functions, while the aggregated effect of the agents is captured by the population mean of their states, namely, the mean-ﬁeld state.
For such a game, based on the Nash certainty equivalence principle, we provide sufﬁcient conditions for the existence and uniqueness of its Nash equilibrium.
Moreover, to ﬁnd the Nash equilibrium, we propose a mean-ﬁeld actor-critic algorithm with linear function approximation, which does not require knowing the model of dynamics.
Speciﬁcally, at each iteration of our algorithm, we use the single-agent actor-critic algorithm to approximately obtain the optimal policy of the each agent given the current meanﬁeld state, and then update the mean-ﬁeld state.
In particular, we prove that our algorithm converges to the Nash equilibrium at a linear rate.
To the best of our knowledge, this is the ﬁrst success of applying model-free reinforcement learning with function approximation to discrete-time mean-ﬁeld Markov games with provable non-asymptotic global convergence guarantees.
I NTRODUCTION
In reinforcement learning (RL) (Sutton and Barto, 2018), an agent learns to make decisions that minimize its expected total cost through sequential interactions with the environment.
Multi-agent reinforcement learning (MARL) (Shoham et al., 2003; 2007; Busoniu et al., 2008) aims to extend RL to sequential decision-making problems involving multiple agents.
In a non-cooperative game, we are interested in the Nash equilibrium (Nash, 1951), which is a joint policy of all the agents such that each agent cannot decrease its expected total cost by unilaterally deviating from its Nash policy.
The
Nash equilibrium plays a critical role in understanding the social dynamics of self-interested agents (Ash, 2000; Axtell, 2002) and constructing the optimal policy of a particular agent via ﬁctitious selfplay (Bowling and Veloso, 2000; Ganzfried and Sandholm, 2009).
With the recent development in deep learning (LeCun et al., 2015), MARL with function approximation achieves tremendous empirical successes in applications, including Go (Silver et al., 2016; 2017), Poker (Heinrich and Silver, 2016;
Moravˇc´ık et al., 2017), Star Craft (Vinyals et al., 2019), Dota (OpenAI, 2018), autonomous driving (Shalev-Shwartz et al., 2016), multi-robotic systems (Yang and Gu, 2004), and solving social dilemmas (de Cote et al., 2006; Leibo et al., 2017; Hughes et al., 2018).
However, since the capacity of the joint state and action spaces grows exponentially in the number of agents, such MARL approaches become computationally intractable when the number of agents is large, which is common in real-world applications (Sandholm, 2010; Calderone, 2017; Wang et al., 2017a).
Mean-ﬁeld game is proposed by Huang et al.
; Lasry and Lions (2006a;b; 2007) with the idea of utilizing mean-ﬁeld approximation to model the strategic interactions within a large population.
In a mean-ﬁeld game, each agent has the same cost function and state transition, which depend on the other agents only through their aggregated effect.
As a result, the optimal policy of each agent depends solely on its own state and the aggregated effect of the population, and such an optimal policy is symmetric across all the agents.
Moreover, if the aggregated effect of the population corresponds to the Nash equilibrium, then the optimal policy of each agent jointly constitutes a
Nash equilibrium.
Although such a Nash equilibrium corresponds to an inﬁnite number of agents, it well approximates the Nash equilibrium for a sufﬁciently large number of agents (Bensoussan et al.,
Also, as the aggregated effect of the population abstracts away the strategic interactions between individual agents, it circumvents the computational intractability of the MARL approaches that do not exploit symmetry.
However, most existing work on mean-ﬁeld games focuses on characterizing the existence and uniqueness of the Nash equilibrium rather than designing provably efﬁcient algorithms.
In particular, most existing work considers the continuous-time setting, which requires solving a pair of
Hamilton-Jacobi-Bellman (HJB) and Fokker-Planck (FP) equations, whereas the discrete-time setting is more common in practice, e.g., in the aforementioned applications.
Moreover, most existing approaches, including the ones based on solving the HJB and FP equations, require knowing the model of dynamics (Bardi and Priuli, 2014), or having the access to a simulator, which generates the next state given any state-action pair and aggregated effect of the population (Guo et al., 2019), which is often unavailable in practice.
To address these challenges, we develop an efﬁcient model-free RL approach to mean-ﬁeld game, which provably attains the Nash equilibrium.
In particular, we focus on discrete-time mean-ﬁeld games with linear state transitions and quadratic cost functions, where the aggregated effect of the population is quantiﬁed by the mean-ﬁeld state.
Such games capture the fundamental difﬁculties of general mean-ﬁeld games and well approximates a variety of real-world systems such as power grids (Minciardi and Sacile, 2011), swarm robots (Fang, 2014; Araki et al., 2017; Doerr et al., 2018), and ﬁnancial systems (Zhou and Li, 2000; Huang and Li, 2018).
In detail, based on the
Nash certainty equivalence (NCE) principle (Huang et al., 2006; 2007), we propose a mean-ﬁeld actor-critic algorithm which, at each iteration, given the mean-ﬁeld state µ , approximately attains the optimal policy π ∗ µ of each agent, and then updates the mean-ﬁeld state µ assuming that all the agents follow π ∗ µ .
We parametrize the actor and critic by linear and quadratic functions, respectively, and prove that such a parameterization encompasses the optimal policy of each agent.
Speciﬁcally, we update the actor parameter using policy gradient (Sutton et al., 2000) and natural policy gradient (Kakade, 2002; Peters and Schaal, 2008; Bhatnagar et al., 2009) and update the critic parameter using primal-dual gradient temporal difference (Sutton et al., 2009a;b).
In particular, we prove that given the mean-ﬁeld state µ , the sequence of policies generated by the actor converges linearly to the optimal policy π ∗ µ .
Moreover, when alternatingly update the policy and mean-ﬁeld state, we prove that the sequence of policies and its corresponding sequence of mean-ﬁeld states converge to the unique Nash equilibrium at a linear rate.
Our approach can be interpreted from both “passive” and “active” perspectives: (i) Assuming that each self-interested agent employs the single-agent actorcritic algorithm, the policy of each agent converges to the unique Nash policy, which characterizes the social dynamics of a large population of model-free RL agents.
(ii) For a particular agent, our approach serves as a ﬁctitious self-play method for it to ﬁnd its Nash policy, assuming the other agents give their best responses.
To the best of our knowledge, our work establishes the ﬁrst efﬁcient model-free RL approach with function approximation that provably attains the Nash equilibrium of a discrete-time mean-ﬁeld game.
As a byproduct, we also show that the sequence of policies generated by the single-agent actor-critic algorithm converges at a linear rate to the optimal policy of a linear-quadratic regulator (LQR) problem in the presence of drift, which may be of independent interest.
Related Work.
Mean-ﬁeld game is ﬁrst introduced in Huang et al.
; Lasry and Lions (2006a;b; 2007).
In the last decade, there is growing interest in understanding continuous-time mean-ﬁeld games.
See, e.g., Gu´eant et al.
; Bensoussan et al.
; Gomes et al.
Carmona and Delarue (2013; 2018) and the references therein.
Due to their simple structures, continuous-time linear-quadratic mean-ﬁeld games are extensively studied under various model assumptions.
See Li and Zhang (2008); Bardi (2011); Wang and Zhang (2012); Bardi and Priuli (2014);
Huang et al.
(2016a;b); Bensoussan et al.
(2016; 2017); Caines and Kizilkale (2017); Huang and Huang (2017); Moon and Bas¸ar (2018)
; Huang and Zhou (2019) for examples of this line of work.
Meanwhile, the literature on discrete-time linear-quadratic mean-ﬁeld games remains relatively scarce.
Most of this line of work focuses on characterizing the existence of a Nash equilibrium and the behavior of such a Nash equilibrium when the number of agents goes to inﬁnity (Gomes et al., 2010; Tembine and Huang, 2011; Moon and Bas¸ar, 2014; Biswas, 2015; Saldi et al., 2018a;b; 2019).
See also Yang et al.
(2018a), which applies maximum entropy inverse RL
(Ziebart et al., 2008) to infer the cost function and social dynamics of discrete-time mean-ﬁeld games with ﬁnite state and action spaces.
Our work is most related to Guo et al.
(2019), where they propose a mean-ﬁeld Q-learning algorithm (Watkins and Dayan, 1992) for discrete-time mean-ﬁeld games with ﬁnite state and action spaces.
Such an algorithm requires the access to a simulator, which, given any state-action pair and mean-ﬁeld state, outputs the next state.
In contrast, both our state and action spaces are inﬁnite, and we do not require such a simulator but only observations of trajectories under given mean-ﬁeld state.
Correspondingly, we study the mean-ﬁeld actor-critic algorithm with linear function approximation, whereas their algorithm is tailored to the tabular setting.
Also, our work is closely related to Mguni et al.
(2018), which focuses on a more restrictive setting where the state transition does not involve the mean-ﬁeld state.
In such a setting, mean-ﬁeld games are potential games, which is, however, not true in more general settings (Li et al., 2017; Briani and Cardaliaguet, 2018).
In comparison, we allow the state transition to depend on the mean-ﬁeld state.
Meanwhile, they propose a ﬁctitious self-play method based on the single-agent actor-critic algorithm and establishes its asymptotic convergence.
However, their proof of convergence relies on the assumption that the single-agent actor-critic algorithm converges to the optimal policy, which is unveriﬁed therein.
In addition, our work is related to Jayakumar and Aditya (2019), where the proposed algorithm is only shown to converge asymptotically to a stationary point of the mean-ﬁeld game.
Our work also extends the line of work on ﬁnding the Nash equilibria of Markov games using
MARL.
Due to the computational intractability introduced by the large number of agents, such a line of work focuses on ﬁnite-agent Markov games (Littman, 1994; 2001; Hu and Wellman, 1998;
Bowling, 2001; Lagoudakis and Parr, 2002; Hu and Wellman, 2003; Conitzer and Sandholm, 2007;
Perolat et al., 2015; P ´ erolat et al., 2016b;a; 2018; Wei et al., 2017; Zhang et al., 2018; Zou et al., 2019; Casgrain et al., 2019).
See also Shoham et al.
(2003; 2007); Busoniu et al.
; Li (2018) for detailed surveys.
Our work is related to Yang et al.
(2018b), where they combine the mean-ﬁeld approximation of actions (rather than states) and Nash Q-learning (Hu and Wellman, 2003)
to study general-sum Markov games with a large number of agents.
However, the Nash Q-learning algorithm is only applicable to ﬁnite state and action spaces, and its convergence is established under rather strong assumptions.
Also, when the number of agents goes to inﬁnity, their approach yields a variant of tabular Q-learning, which is different from our mean-ﬁeld actor-critic algorithm.
For policy optimization, based on the policy gradient theorem, Sutton et al.
; Konda and Tsitsiklis (2000) propose the actor-critic algorithm, which is later generalized to the natural actor-critic algorithm (Peters and Schaal, 2008; Bhatnagar et al., 2009).
Most existing results on the convergence of actor-critic algorithms are based on stochastic approximation using ordinary differential equations (Bhatnagar et al., 2009; Castro and Meir, 2010; Konda and Tsitsiklis, 2000; Maei, 2018), which are asymptotic in nature.
For policy evaluation, the convergence of primal-dual gradient temporal difference is studied in Liu et al.
; Du et al.
; Wang et al.
(2017b)
; Yu (2017);
Wai et al.
However, this line of work assumes that the feature mapping is bounded, which is not the case in our setting.
Thus, the existing convergence results are not applicable to analyzing the critic update in our setting.
To handle the unbounded feature mapping, we utilize a truncation argument, which requires more delicate analysis.
Finally, our work extends the line of work that studies model-free RL for LQR.
For example, Bradtke (1993); Bradtke et al.
(1994) show that policy iteration converges to the optimal policy, Tu and
Recht (2017); Dean et al.
(2017) study the sample complexity of least-squares temporal-difference for policy evaluation.
More recently, Fazel et al.
; Malik et al.
; Tu and Recht (2018) show that the policy gradient algorithm converges at a linear rate to the optimal policy.
See as also
Hardt et al.
; Dean et al.
(2018) for more in this line of work.
Our work is also closely related to Yang et al.
(2019), where they show that the sequence of policies generated by the natural actorcritic algorithm enjoys a linear rate of convergence to the optimal policy.
Compared with this work, when ﬁxing the mean-ﬁeld state, we use the actor-critic algorithm to study LQR in the presence of drift, which introduces signiﬁcant difﬁculties in the analysis.
As we show in § 3, the drift causes the optimal policy to have an additional intercept, which makes the state- and action-value functions more complicated.
Notations.
We denote by (cid:107) M (cid:107) ∗ the spectral norm, ρ ( M ) the spectral radius, σ min ( M ) the minimum singular value, and σ max ( M ) the maximum singular value of a matrix M .
We use (cid:107) α (cid:107) 2 to represent the (cid:96) 2 -norm of a vector α , and ( α ) ji to denote the sub-vector ( α i , α i +1 , . . .
, α j ) (cid:62) , where α k is the k -th entry of the vector α .
For scalars a 1 , . . . , a n , we denote by poly ( a 1 , . . .
, a n ) the polynomial of a 1 , . .
. , a n , and this polynomial may vary from line to line.
We use [ n ] to denote the set { 1 , 2 , . . . , n } for any n ∈ N .
L INEAR -Q UADRATIC M EAN -F IELD G AME
A linear-quadratic mean-ﬁeld N a -player game involves N a ∈ N agents, whose state transitions are given by
[Equation]
Here A ∈ R m × m , B ∈ R m × k , and A ∈ R m × m are matrices, x it ∈ R m and u
it ∈
R k are the state and action vectors of agent i , respectively, the vector d i ∈
R m is a drift term, ω it ∈
R m is an independent random noise term following the Gaussian distribution N (0 , Ψ ω ) {+, and x t = 1 /N a · (cid:80) N a j =
 x+}
jt is the mean-ﬁeld state. The agents are coupled through the mean-ﬁeld state x t .
In the linear-quadratic mean-ﬁeld N a -player game, the cost of agent i ∈
[ N a ] at time t ≥ 0 is given by
[Equation]
where Q ∈ R m × m , R ∈ R k × k , and Q ∈ R m × m are matrices, and u it is generated by π i , i.e., the policy of agent i .
To measure the performance of agent i following its policy π i under the inﬂuence of the other agents, we deﬁne the expected total cost of agent i as
[Equation]
We are interested in ﬁnding a Nash equilibrium ( π 1 , π 2 , . .
, π N a ) , which is deﬁned by
J i ( π
[Equation]
, π i
[Equation]
, π
[Equation]
, π
[Equation] [Equation]
π
i − 1 , (cid:101) π i , π i +1 , . . .
, π N a ) , ∀ (cid:101)
π i , i ∈
[ N a ] .
That is, agent i cannot further decrease its expected total cost by unilaterally deviating from its Nash policy.
For the simplicity of discussion, we assume that the drift term d i is identical for each agent.
We consider taking the inﬁnite-population limit N a → ∞ , where each agent has an inﬁnitesimal contribution to the dynamics of the system.
Thus, the joint policy of all the agents except agent i can be modeled as a mean-ﬁeld policy π † , and all the agents following such a mean-ﬁeld policy π † generate the mean-ﬁeld state E x † t , where { x † t } t ≥ 0 is generated following the policy π † .
By the symmetry of the agents in terms of their state transitions and cost functions, we focus on a ﬁxed agent and drop the superscript i hereafter.
Before we formally present the formulation of linear-quadratic mean-ﬁeld games, we ﬁrst introduce the following mean-ﬁeld LQR (MF-LQR) problem, which aims to ﬁnd an optimal policy for the ﬁxed agent given the mean-ﬁeld policy π † .
Problem 2.1 (MF-LQR) .
Given the mean-ﬁeld policy π † , we consider the following formulation,
[Equation]
where x t ∈ R m is the state vector, u t ∈ R k is the action vector generated by the policy π , { x † t } t ≥ 0 is the trajectory generated by the policy π † , ω t ∈ R m is an independent random noise term following the Gaussian distribution N (0 , Ψ ω ) , and d ∈ R m is a drift term.
Here the expectation E x † t is taken across all the agents.
We aim to ﬁnd π ∗ such that J ( π ∗ , π † ) =
inf π ∈ Π J ( π, π † ) .
Note that a controllable linear system using linear quadratic optimal control is always stable. Further combining the fact that our linear closed-loop dynamics in Problem 2.1 is driven by the Gaussian noise term ω t , we know that the Markov chain of states generated by the policy π † admits a stationary distribution and converges to this stationary distribution.
This implies that the mean-ﬁeld state E x † t converges to a constant vector µ † as t → ∞ , which serves as a time-invariant mean-ﬁeld state.
As we consider the ergodic setting, it then sufﬁces to study Problem 2.1 with t sufﬁciently large.
Therefore, the inﬂuence of the mean-ﬁeld policy π † is captured by the mean-ﬁeld state µ † .
By re-formulating Problem 2.1, with slight abuse of notations, we obtain the following drifted-LQR (D-LQR).
Problem 2.2 (D-LQR) .
Given a mean-ﬁeld state µ ∈ R m , we consider the following formulation,
[Equation]
where x t ∈ R m is the state vector, u t ∈ R k is the action vector generated by the policy π , ω t ∈ R m is an independent random noise term following the Gaussian distribution N (0 , Ψ ω ) , and d ∈ R m is a drift term.
We aim to ﬁnd an optimal policy π ∗ µ such that J µ ( π ∗ µ ) =
inf π ∈ Π J µ ( π ) .
Compared with the most studied LQR problem (Lewis et al., 2012), both the state transition and the cost function in Problem 2.2 have drift terms, which act as the mean-ﬁeld “force” that drives the states away from zero.
Such a mean-ﬁeld “force” introduces additional challenges when solving Problem 2.2 in the model-free setting (see § 3.3 for details).
On the other hand, the unique optimal policy π ∗ µ of Problem 2.2 admits a linear form π ∗ µ ( x t ) =
− K π ∗ µ x t + b π ∗ µ under certain regularity conditions (Anderson and Moore, 2007), where the matrix K π ∗ µ ∈ R k × m and the vector b π ∗ µ ∈ R k are the parameters of π ∗ µ .
Motivated by such a linear form of the optimal policy, we deﬁne the class of linear-Gaussian policies as
[Equation]
where σ ∈ R and the standard Gaussian noise term η ∈ R k is included to encourage exploration.
To solve Problem 2.2, it sufﬁces to ﬁnd the optimal policy π ∗ µ within Π .
We deﬁne Λ 1 ( µ ) =
π ∗ µ as the optimal policy under the mean-ﬁeld state µ .
Assume that all the agents follow the linear policy π ( x ) = − K π x + b π under the mean-ﬁeld state µ .
By plugging u t = π ( x t ) into the state transition in Problem 2.2, as t → ∞ , we know that these agents generate a new mean-ﬁeld state µ new such that
[Equation] We deﬁne Λ 2 ( µ, π ) = µ new as such a new mean-ﬁeld state. Now, we are ready to present the following linear-quadratic mean-ﬁeld game (LQ-MFG). Problem 2.3 (LQ-MFG) .
We consider the following formulation,
[Equation]
where x t ∈ R m is the state vector, u t ∈ R k is the action vector generated by the policy π , µ ∈ R m is the mean-ﬁeld state, ω t ∈ R m is an independent random noise term following the Gaussian distribution N (0 , Ψ ω ) , and d ∈ R m is a drift term.
We aim to ﬁnd a pair ( µ ∗ , π ∗ ) such that (i)
J ( π ∗ , µ ∗ ) =
inf π ∈ Π J
( π, µ ∗ ) ; (ii) E x ∗ t converges to µ ∗ as t → ∞ , where { x ∗ t } t ≥ 0 is the Markov chain of states generated by the policy π ∗ .
The formulation in Problem 2.3 is studied by Lasry and Lions (2007); Bensoussan et al.
; Saldi et al.
(2018a;b).
We propose a more general formulation in Problem C.2 (see § C of the appendix for details), where an additional interaction term between the state vector x t and the mean-ﬁeld state µ is incorporated into the cost function. According to our analysis in § C, up to minor modiﬁcation, the results in the following sections also carry over to Problem C.2. Therefore, for the sake of simplicity, we focus on Problem 2.3 in the sequel. In Problem 2.3, condition (i) is equivalent to the optimality of the policy π ∗ under the mean-ﬁeld state µ ∗ , namely, Λ 1 ( µ ∗ )
=
π ∗ . Meanwhile, condition (ii) is equivalent to the invariance of the mean-ﬁeld state µ ∗ given the policy π ∗ , namely, Λ 2 ( µ ∗ , π ∗ ) = µ ∗ . Such equivalence follows from the NCE principle (Huang et al., 2006; 2007), which also motivates the following deﬁnition of the Nash equilibrium pair (Saldi et al., 2018a;b).
Deﬁnition 2.4 (Nash Equilibrium Pair) .
The pair ( µ ∗ , π ∗ ) ∈ R m ×
Π constitutes a Nash equilibrium pair of Problem 2.3 if it satisﬁes π ∗ =
Λ 1 ( µ ∗ ) and µ ∗ =
Λ 2 ( µ ∗ , π ∗ ) .
Here µ ∗ is called the Nash mean-ﬁeld state and π ∗ is called the Nash policy.
By Deﬁnition 2.4, Problem 2.3 aims to ﬁnd a Nash equilibrium pair ( µ ∗ , π ∗ ) .
M EAN -F IELD A CTOR -C RITIC
We ﬁrst characterize the existence and uniqueness of the Nash equilibrium pair of Problem 2.3 under mild regularity conditions, and then propose a mean-ﬁeld actor-critic algorithm to obtain such a Nash equilibrium.
As a building block of the mean-ﬁeld actor-critic, we propose the natural actor-critic to solve Problem 2.2.
3. E XISTENCE AND U NIQUENESS OF N ASH E QUILIBRIUM P AIR
We now establish the existence and uniqueness of the Nash equilibrium pair deﬁned in Deﬁnition 2.4.
We impose the following regularity conditions.
Assumption 3.1.
We assume that the following statements hold:
(i)
The algebraic Riccati equation X =
A (cid:62) XA + Q − A (cid:62) XB ( B (cid:62) XB + R )
− 1 B (cid:62)
XA admits a unique symmetric positive deﬁnite solution X ∗ ; (ii) It holds for L {+
+}
=
L 1 L 3 + L 2 that L 0 < 1 , where
[Equation]
L [Equation] Here K ∗ = − ( B (cid:62) X ∗ B + R ) − 1 B (cid:62) X ∗ A .
The ﬁrst assumption is implied by mild regularity conditions on the matrices A , B , Q , and R , which are (1) the positivity of R ; (2) the non-negativity of Q = C (cid:62) C ; (3) the observability of ( A, C ) ; (4) the stability of ( A, B ) .
See De Souza et al. ; Lewis et al.
(2012) for more details.
The second assumption is standard in the literature (Bensoussan et al., 2016; Saldi et al., 2018b), which ensures the stability of the LQ-MFG.
In the following proposition, we show that Problem 2.3 admits a unique Nash equilibrium pair.
Proposition 3.2 (Existence and Uniqueness of Nash Equilibrium Pair) .
Under Assumption 3.1, the operator Λ( · ) = Λ 2 ( · , Λ 1 ( · )) is L 0 -Lipschitz, where L 0 is given in Assumption 3.1.
Moreover, there exists a unique Nash equilibrium pair ( µ ∗ , π ∗ ) of Problem 2.3.
Proof.
See § E.1 for a detailed proof.
3. M EAN -F IELD A CTOR -C RITIC FOR LQ-MFG
The NCE principle motivates a ﬁxed-point approach to solve Problem 2.3, which generates a sequence of policies { π s } s ≥ 0 and mean-ﬁeld states { µ s } s ≥ 0 satisfying the following two properties: (i)
Given the mean-ﬁeld state µ s , the policy π s is optimal.
(ii)
The mean-ﬁeld state becomes µ s + as t → ∞ , if all the agents follow π s under the current mean-ﬁeld state µ s .
Here (i) requires solving
Problem 2.2 given the mean-ﬁeld state µ s , while (ii) requires simulating the agents following the policy π s given the current mean-ﬁeld µ s .
Based on such properties, we propose the mean-ﬁeld actor-critic in Algorithm 1.
Algorithm 1 requires solving Problem 2.2 at each iteration to obtain π s = Λ 1 ( µ s ) and µ s {++
+}
=
Λ 2 ( µ s , π s ) .
[-Algorithm 1 requires solving Problem 2.2 at each iteration to obtain π s ≈ Λ 1 ( µ s ) and µ s +
-]
To this end, we introduce the natural actor-critic in § 3.3 that solves Problem 2.2.
N ATURAL A CTOR -C RITIC FOR D-LQR
Now we focus on solving Problem 2.2 for a ﬁxed mean-ﬁeld state µ , we thus drop the subscript µ hereafter.
With slight abuse of notations, we write π K,b ( x ) =
− Kx + b + σ · η to emphasize
[Equation] the dependence on K and b , and J ( K, b ) =
J ( π K,b ) consequently.
Now, we propose the natural actor-critic to solve Problem 2.2.
For any policy π K,b ∈ Π , by the state transition in Problem 2.2, we have
[Equation]
where Ψ (cid:15) = σBB (cid:62) + Ψ ω .
It is known that if ρ ( A − BK ) < 1 , then the Markov chain { x t } t ≥induced by (3.1) has a unique stationary distribution N ( µ K,b , Φ K ) (Anderson and Moore, 2007), where the mean-ﬁeld state µ K,b and the covariance Φ K satisfy that
[Equation]
Meanwhile, the Bellman equation for Problem 2.2 takes the following form
[Equation]
Then by calculation (see Proposition B.2 in § B.1 of the appendix for details), it holds that the expected total cost J ( K, b ) is decomposed as
[Equation]
where J 1 ( K ) and J 2 ( K, b ) are deﬁned as
[Equation]
Here J 1 ( K ) is the expected total cost in the most studied LQR problems (Yang et al., 2019; Fazel et al., 2018), where the state transition does not have drift terms.
Meanwhile, J 2 ( K, b ) corresponds to the expected cost induced by the drift terms.
The following two propositions characterize the properties of J 2 ( K, b ) .
First, we show that J 2 ( K, b ) is strongly convex in b .
Proposition 3.3.
Given any K , the function J 2 ( K, b ) is ν K -strongly convex in b .
Here ν K =
σ min ( Y (cid:62) 1 ,K Y 1 ,K + Y (cid:62) 2 ,K Y 2 ,K ) , where Y 1 ,K = R 1 / 2 K ( I − A + BK ) − 1 B − R 1 / 2 and Y 2 ,K =
Q 1 / 2 ( I − A + BK ) − 1 B .
Also, J 2 ( K, b ) has ι K -Lipschitz continuous gradient in b , where ι K is upper bounded as ι K ≤ [1 − ρ ( A − BK )]
− 2 · ( (cid:107) B (cid:107) 22 · (cid:107) K (cid:107) 22 · (cid:107) R (cid:107) 2 + (cid:107) B (cid:107) 22 · (cid:107) Q (cid:107) 2 ) .
Proof.
See § E.4 for a detailed proof.
Second, we show that min b J 2 ( K, b ) is independent of K .
Proposition 3.4.
We deﬁne b K =
argmin b J 2 ( K, b ) , where J 2 ( K, b ) is deﬁned in (3.6).
It holds that
[Equation]
Moreover, J 2 ( K, b K ) takes the form of
[Equation]
which is independent of K .
Proof.
See § E.2 for a detailed proof.
Since min b J 2 ( K, b ) is independent of K by Proposition 3.4, it holds that the optimal K ∗ is the same as argmin K J 1 ( K ) .
This motivates us to minimize J ( K, b ) by ﬁrst updating K following the gradient direction ∇ K J 1 ( K ) to the optimal K ∗ , then updating b following the gradient direction ∇ b J 2 ( K ∗ , b ) .
We now design our algorithm based on this idea.
We deﬁne Υ K , p K,b , and q K,b as
[Equation]
where f K,b
= ( I − A + BK ) −(cid:62)
[( A − BK ) (cid:62) P K ( Bb + Aµ + d ) − K (cid:62) Rb ] .
By calculation (see
Proposition B.3 in § B.1 of the appendix for details), the gradients of J 1 ( K ) and J 2 ( K, b ) take the forms of
[-= 2(Υ 
 K K − Υ 21 K )Φ K and ∇ b J 2 ( K, b ) = Υ 22 K ( − Kµ K,b + b )+Υ 21 K µ K,b + q K,b .-][Equation]
Our algorithm follows the natural actor-critic method (Bhatnagar et al., 2009) and actor-critic method (Konda and Tsitsiklis, 2000).
Speciﬁcally, (i) To obtain the optimal K ∗ , in the critic update step, we estimate the matrix Υ K by (cid:98) Υ K via a policy evaluation algorithm, e.g., Algorithm 3 or Algorithm 4 (see § B.2 and § B.3 of the appendix for details); in the actor update step, we update K via K ← K − γ ·
( (cid:98) Υ 22 K K − (cid:98)
Υ 21 K ) , where the term (cid:98) Υ
22 K K − (cid:98)
Υ 21 K is the estimated natural gradient.
(ii) To obtain the optimal b ∗ given K ∗ , in the critic update step, we estimate Υ K ∗ , q K ∗ ,b , and µ K ∗ ,b by (cid:98) Υ K ∗ , (cid:98) q K ∗ ,b , and (cid:98) µ K ∗ ,b via a policy evaluation algorithm;
In the actor update step, we update b via b ← b − γ · (cid:98) ∇ b J 2 ( K ∗ , b ) , where (cid:98) ∇ b J 2 ( K ∗ , b [Equation] = [Equation]
[Equation]
∗ (cid:98) µ K ∗ ,b + [Equation] ) + (cid:98) Υ 21 K ∗ (cid:98) µ K ∗ ,b + (cid:98) q K ∗ ,b is the estimated gradient. Combining the above procedure, we obtain the natural actor-critic for Problem 2.2, which is stated in Algorithm 2.
One may want to apply gradient method to J ( K, b ) directly in the joint space of K and b . However, the gradient dominance property of J 1 ( K ) in the most studied LQR problem (Yang et al., 2019) no longer holds for J ( K, b ) . Therefore, the convergence of the gradient method to J ( K, b ) is not guaranteed in our problem.  G LOBAL C ONVERGENCE R ESULTS
The following theorem establishes the rate of convergence of Algorithm 1 to the Nash equilibrium pair ( µ ∗ , π ∗ ) of Problem 2.3.
Theorem 4.1 (Convergence of Algorithm 1) .
For a sufﬁciently small tolerance ε > 0 , we set the number of iterations S in Algorithm 1 such that
[Equation]
For any s ∈ [ S ] , we deﬁne ε s = min (cid:110)(cid:2) 1 − ρ ( A − BK ∗ ) (cid:3) 4 (cid:0) (cid:107) B (cid:107) 2 + (cid:107) A (cid:107) 2 (cid:1) − 4 (cid:0) (cid:107) µ s (cid:107) − 2 2 + (cid:107) d (cid:107) − 2 2 (cid:1) · σ min (Ψ (cid:15) ) · σ min ( R ) · ε 2 , ν K ∗ · (cid:2) 1 − ρ ( A − BK ∗ ) (cid:3) 4 · (cid:107) B (cid:107) − 2 2 · M b ( µ s ) · ε 2 , ε (cid:111) · 2 − s − 10 , (4.2)
• Mean-ﬁeld state µ and initial policy π K 0 ,b 0 .
• Numbers of iterations N , H , { (cid:101) T n , T n } n ∈ [ N ] , { (cid:101) T bh , T bh } h ∈
[ H ] .
• Stepsizes γ , γ b , { γ n,t } n ∈ [ N ] ,t ∈ [ T n ] , { γ bh,t } h ∈
[ H ] ,t ∈
[ T bh ] .
[Table]
[Table]
[Table]
[Table]
state (cid:98) µ K,b = (cid:98) µ K N ,b H .
where ν K ∗ is deﬁned in Proposition 3.3 and
[Equation]
In the s -th policy update step in Line 3 of Algorithm 1, we set the inputs via Theorem B.4 such that
J µ s ( π s +1 )
− J µ s ( π ∗ µ s ) <
ε s , where the expected total cost J µ s ( · ) is deﬁned in Problem 2.2, and π ∗ µ s = Λ 1 ( µ s ) is the optimal policy under the mean-ﬁeld state µ s . Then it holds with probability at least 1 − ε 5 that
[Equation] Here µ ∗ is the Nash mean-ﬁeld state, K S and b S are parameters of the policy π S , and K ∗ and b ∗ are parameters of the Nash policy π ∗ . Proof Sketch.
The proof of the theorem is based on the convergence of the natural actor-critic algorithm 2 and a contraction argument. First, we prove in Theorem B.4 that Algorithm 2 converges linearly to the optimal policy of Problem 2.2. By this, in each iteration of Algorithm 1, we control the error between µ s +1 and µ ∗ s +1 to be ε s > 0 with high probability, where µ ∗ s +1 is the mean-ﬁeld state generated by the optimal policy Λ 1 ( µ s ) ; in other words, µ ∗ s +1 = Λ( µ s ) .
Combining the fact from Proposition 3.2 that Λ( · ) is a contraction, we deduce that [Equation] with high probability, where (cid:101) ε s > 0 is some error term and is speciﬁed in the detailed proof. Moreover, by telescoping sum, and note that the sum (cid:80) Ss =1 (cid:101) ε s is upper bounded by the desired error ε , we conclude the theorem.
See § D.1 for a detailed proof.
We highlight that if the inputs of Algorithm 1 satisfy the conditions stated in Theorem B.4, it holds that J µ s ( π s +1 ) − J µ s ( π ∗ µ s ) <
ε s for any s ∈ [ S ] .
See Theorem B.4 in § B.1 of the appendix for details.
By Theorem 4.1, Algorithm 1 converges linearly to the unique Nash equilibrium pair ( µ ∗ , π ∗ ) of Problem 2.3.
To the best of our knowledge, this theorem is the ﬁrst successful attempt to establish that reinforcement learning with function approximation ﬁnds the Nash equilibrium pairs in mean-ﬁeld games with theoretical guarantee, which lays the theoretical foundations for applying modern reinforcement learning techniques to general mean-ﬁeld games.
C ONCLUSION
For the discrete-time linear-quadratic mean-ﬁeld games, we provide sufﬁcient conditions for the existence and uniqueness of the Nash equilibrium pair.
Moreover, we propose the mean-ﬁeld actorcritic algorithm with linear function approximation that is shown converges to the Nash equilibrium pair with linear rate of convergence.
Our algorithm can be modiﬁed to use other parametrized function classes, including deep neural networks, for solving mean-ﬁeld games.
For future research, we aim to extend our algorithm to other variations of mean-ﬁeld games including risk-sensitive mean-ﬁeld games (Saldi et al., 2018a;
Tembine et al., 2014), robust mean-ﬁeld games (Bauso et al., 2016) and partially observed mean-ﬁeld games (Saldi et al., 2019)
A N OTATIONS IN THE P ROOFS
In the proof, for convenience, for any invertible matrix M , we denote by M −(cid:62) =
( M − 1 ) (cid:62) =
( M (cid:62) ) − 1 and (cid:107) M (cid:107) F the Frobenius norm.
We also denote by svec ( M ) the symmetric vectorization of the symmetric matrix M , which is the vectorization of the upper triangular matrix of the symmetric matrix M , with off-diagonal entries scaled by √ 2 .
Also, we denote by smat ( · ) the inverse operation.
For notational simplicity, we write z =
[ x (cid:62) , u (cid:62) ] (cid:62) as the combination of the state vector x and the action vector u .
Also, we write E π [ · ] to emphasize that the expectation is taken following the policy π .
B A UXILIARY A LGORITHMS AND A NALYSIS B. R ESULTS IN D-LQR
In this section, we provide auxiliary results in analyzing Problem 2.2.
First, we introduce the value functions of the Markov decision process (MDP) induced by Problem 2.2.
We deﬁne the state-value function V K,b ( x ) as follows
[Equation]
and the action-value function Q K,b ( x, u ) as follows
[Equation]
where the x (cid:48) is the state generated by the state transition after the state-action pair ( x, u ) .
We establish the close forms of these value functions in the following proposition.
Proposition B.1.
The state-value function V K,b ( x ) takes the form
[Equation]
and the action-value function Q K,b ( x, u ) takes the form
Q K
[Equation]
b
[Equation]
x, u
[Equation]
where the matrix Υ K and the vectors p K,b , q K,b are given in (3.7).
Proof.
See § E.6 for a detailed proof.
By Proposition B.1, we know that V K,b ( x ) is quadratic in x , while Q K,b ( x, u ) is quadratic in x and u .
Now, we show that J ( K, b ) is decomposed as (3.5).
Proposition B.2.
The expected total cost J ( K, b ) deﬁned in Problem 2.2 is decomposed as
[Equation]
where J 1 ( K ) and J 2 ( K, b ) take the following forms
[Equation]
Here µ K,b is given in (3.2), Φ K is given in (3.3), and P K is given in (3.4).
Proof.
See § E.3 for a detailed proof.
We establish the gradients of J ( K, b ) in the following proposition.
Proposition B.3.
The gradient of J 1 ( K ) and the gradient of J 2 ( K, b ) with respect to b take the forms
[Equation]
where the matrix Υ K and the vector q K,b are given in (3.7).
Proof.
See § E.5 for a detailed proof.
Equipped with above results
[Equation]
the following theorem establishes the convergence of Algorithm 2.
Theorem B.4 (Convergence of Algorithm 2) .
Let the initial policy π K 0 ,b 0 be stable.
In Algorithm 2, for a sufﬁciently small tolerance ε > 0 , we set
[Equation]
where C is a positive absolute constant.
Let { K n } n ∈
[ N ] and { b h } h ∈
[ H ] be the sequences of parameters generated by Algorithm 2.
In the n -th critic update step in Line 3 of Algorithm 2, we set
[Equation]
where γ 0 is some positive constant
[Equation]
and λ K n is speciﬁed in Proposition B
[Equation]
Also, after obtaining
K N from Algorithm
[Equation]
we set
[Equation]
where C 0 is some positive absolute constant and the number ν K N is speciﬁed in Proposition 3.3.
In the h -th critic update step in Line 7 of Algorithm 2, we set
[Equation]
2 , (cid:107) µ (cid:107) 2 , J ( K n , b 0 ) (cid:1) · λ − 4 K N · ν − 2 K N · (cid:2) 1 − ρ ( A − BK N ) (cid:3) − 16 · ε − 8 , γ bh,t = γ 0 · t − 1 / 2 , where ν K N is speciﬁed in Proposition 3.3.
Then with probability at least 1 − ε 10 , it holds that
J ( K N , b H ) − J
( K ∗ , b ∗ ) <
ε , (cid:107) K N − K ∗ (cid:107) F ≤ [ σ − 1min (Ψ (cid:15) ) · σ − 1min ( R ) · ε ] 1 / 2 , and (cid:107) b
H − b ∗ (cid:107) 2 ≤ (2 ε/ν K ∗ ) 1 / 2 .
Proof.
See § D.2 for a detailed proof.
By this theorem, given any mean-ﬁeld state µ , we show that Algorithm 2 converges linearly to the optimal policy π ∗ µ of Problem 2.2.
In the proof of Theorem B.4, we use the convergence results of
Algorithm 3, which is provided in § B.2 of the appendix.
B. P RIMAL -D UAL P OLICY E VALUATION A LGORITHM
Note that critic update steps in Algorithm 2 need estimators of the matrix Υ K and the vector q K,b .
We now derive a policy evaluation algorithm to estimate these quantities, based on gradient temporal difference algorithm (Sutton et al., 2009a).
For notational convenience, we denote by the feature vector
[Equation]
(B.5) where
[Equation]
Recall svec ( M ) gives the symmetric vectorization of the symmetric matrix M .
We also denote by the vector
[Equation]
where the matrix Υ K and vectors p K,b and q K,b are given in (3.7).
Note that to estimate Υ K and q K,b , we only need to estimate α K,b .
We also deﬁne the following matrix
[Equation]
where ( x (cid:48) , u (cid:48) ) is the state-action pair after ( x, u ) following the policy π K,b and the state transition.
The matrix Θ K,b deﬁned in (B.7) connects with the vector α K,b by the following proposition.
Proposition B.5.
The following equation holds:
[Equation]
where the vector α K,b is given in (B.6), the matrix Θ K,b is given in (B.7), and the feature vector ψ ( x, u ) is deﬁned in (B.5).
Proof.
See § E.7 for a detailed proof.
Motivated by Proposition B.5, to obtain the vector α K,b , we only need to solve the following linear system in ζ =
( ζ 1 , ζ (cid:62)
[Equation]
where the matrix (cid:101) Θ K,b takes the form
[Equation]
Note that in the above linear system (B.8), if the matrix Θ K,b is invertible, then the whole linear system admits the unique solution ζ K,b
= ( J ( K, b ) , α (cid:62) K,b ) (cid:62) .
Under mild condition, the following proposition veriﬁes the invertibility of Θ K,b .
Proposition B.6.
If ρ ( A − BK ) < 1 , then the matrix Θ K,b is invertible, and its spectral norm is upper bounded by 4(1 + (cid:107) K (cid:107) 2 F ) 2 · (cid:107) Φ K (cid:107) 2 ∗ .
Moreover, σ min ( (cid:101)
Θ K,b ) is lower bounded by a positive constant λ K , where λ K only depends on σ , σ min (Ψ ω ) and ρ ( A − BK ) .
Proof.
See § E.8 for a detailed proof.
Hereafter, for notational convenience, we denote by (cid:98) ψ t the estimated feature vector (cid:98) ψ ( x t , u t ) .
Now, we present the primal-dual gradient temporal difference algorithm in Algorithm 3.
Primal-Dual Gradient Method.
In Algorithm 3, instead of solving the linear system (B.8) directly, we minimize the following loss function
[Equation]
state µ u π K,b
Set δ t +1 ← ζ 1 t + ( (cid:98) ψ t − (cid:98) ψ t +1 ) (cid:62) ζ 2 t − c t .
via ζ 1 t +1 ← ζ 1 t − γ t +1 · ( ξ 1 t + (cid:98) ψ (cid:62) t ξ 2 t ) , ζ 2 t +1 ← ζ 2 t − γ t +1 · (cid:98) ψ t ( (cid:98) ψ t − (cid:98) ψ t +1 ) (cid:62) ξ 2 t , ξ 1 t +1 ← (1 − γ t +1 ) · ξ 1 t + γ t +1 · ( ζ 1 t − c t ) , ξ 2 t +1 ← (1 − γ t +1 ) · ξ 2 t + γ t +1 · δ t +1 · (cid:98) ψ t .
Project ζ t +1 and ξ t +1 to V ζ and V ξ , respectively.
15: end for 16: Set (cid:98) α K,b ← ( (cid:80) Tt =1 γ t ) − 1 · ( (cid:80) Tt =1 γ t · ζ 2 t ) , and (cid:98) Υ K ← smat ( (cid:98) α K,b, 1 ) , (cid:18)(cid:98)
p K,b (cid:98) q K,b (cid:19) ← (cid:98) α K,b, 2 − (cid:98)
Υ K (cid:18) (cid:98) µ K,b − K
(cid:98) µ K,b + b (cid:19) ,
where (cid:98) α K,b, 1 =
( (cid:98) α K,b ) ( k + d +1)
( k + d ) / 2and (cid:98) α K,b, 2 = ( (cid:98) α K,b ) ( k + d +3)
( k + d ) / 2 ( k + d +1)
( k + d ) / 2+1 .
17: Output: Estimators (cid:98) µ K,b , (cid:98) Υ K , and (cid:98) q K,b .
Further, by Fenchel’s duality, we convert the above problem to the following primal-dual min-max problem
[Equation]
where we restrict the primal variable ζ in some compact set V ζ and the dual variable ξ in some compact set V ξ , which are detailedly speciﬁed in Condition B.7.
By taking the gradient of the objective in (B.9) with respect to ζ and ξ
[Equation]
we obtain the following gradients
[Equation]
This gives a primal-dual gradient method to solve the min-max problem (B.9).
Estimate of Mean State µ K,b .
Note that in the deﬁnition of the feature vector in (B.5), we need the mean µ K,b of the state to evaluate the feature vector ψ ( x, u ) .
Therefore, prior to the main body of Algorithm 3, we obtain the estimator (cid:98) µ K,b by simulating the MDP under the policy π K,b for (cid:101) T iterations, and compute the estimated feature vector (cid:98) ψ via
[Equation]
where (cid:98) ϕ
( x, u ) takes the following form
[Equation]
Before stating the convergence result of Algorithm
[Equation]
we specify the sets V ζ and V ξ .
Condition B.7 .
Given K 0 and b 0 such that ρ ( A − BK 0 ) < 1 .
We deﬁne the sets V ζ and V ξ in (B.9) as
[Equation]
Here M ζ, 1 , M ζ, 2 and M ξ are constants independent of K and b , which take the following forms
[Equation]
(cid:107) Q (cid:107) F + (cid:107) R (cid:107) F (cid:1)
+ (cid:0) (cid:107) A (cid:107) 2 F + (cid:107) B (cid:107) 2 F (cid:1) · √ d · J ( K 0 , b 0 ) /σ
min (Ψ ω ) (cid:105) + (cid:104)(cid:0) (cid:107) Q (cid:107) ∗ + (cid:107) R (cid:107) ∗ (cid:1) + (cid:0) (cid:107) A (cid:107) ∗ + (cid:107) B (cid:107) ∗ (cid:1) 2 · J ( K 0 , b 0 ) /σ min
(Ψ ω ) (cid:105) · (cid:2)
J ( K 0 , b 0 )
/σ min ( Q )
+ J ( K 0 , b 0 ) /σ min
( R ) (cid:3) + (cid:0) (cid:107) A (cid:107) ∗ + (cid:107) B (cid:107) ∗ (cid:1) · J ( K 0 , b 0 ) 2 · σ − 1min (
[Equation] [Equation]
[Equation] [Equation]
[Equation]
where C is a positive absolute constant, and κ Q and κ R are condition numbers of the matrices Q and R , respectively.
We now characterize the convergence of Algorithm 3.
Theorem B.8 (Convergence of Algorithm 3) .
Given K 0 , b 0 , K and b such that ρ ( A − BK 0 ) < 1 and
J ( K, b ) ≤ J ( K 0 , b 0 ) , we deﬁne the sets V ζ and V ξ through Condition B.7.
Given the stepsize γ t = γ 0 · t − 1 / 2 for some constant γ 0 > 0 , then for any ρ such that ρ ( A − BK ) < ρ < 1 , when the number of iterations (cid:101) T and T are sufﬁciently large such that (cid:101) T ≤ poly
( (cid:107) K (cid:107) F , (cid:107) b (cid:107) 2 , (cid:107) µ (cid:107) 2 , J ( K 0 , b 0 ))
· (1 − ρ ) − 6 , with probability at least 1 − T − 4 − (cid:101) T − 6 , the output (cid:98) α K,b satisﬁes that (cid:107) (cid:98) α K,b − α K,b (cid:107) 22 ≤ λ − 2 K · poly (cid:0) (cid:107) K (cid:107) F , (cid:107) b (cid:107) 2 , (cid:107) µ (cid:107) 2 , J ( K
[Equation]
b 0 ) (cid:1) · (cid:20) log 6 T
[Equation]
log (cid:101) T (cid:101)
[Equation]
[Equation] ρ )(cid:21) , where λ K is speciﬁed in Proposition B.6.
Same bounds for (cid:107) (cid:98)
Υ K − Υ K (cid:107) F , (cid:107) (cid:98) p K,b − p K,b (cid:107) 2 and (cid:107) (cid:98) q K,b − q K,b (cid:107) 2 hold.
Also, with probability at least 1 − (cid:101) T − 6 , it holds that
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Proof.
See § D.3 for a detailed proof.
The policy evaluation algorithm is used in the critic update steps in Algorithm 2.
Therefore, the above theorem is crucial in the proof of the corresponding convergence result stated in Theorem
B.4.
B. T EMPORAL D IFFERENCE P OLICY E VALUATION A LGORITHM
We also apply TD(0) method (Sutton and Barto, 2018) to evaluate the policy, which is presented in
Algorithm 4.
(cid:80) (cid:101)
Tt =1 (cid:101) x t .
7: Sample x 0 from the the stationary distribution N ( µ K,b , Φ K ) .
8: for t = 0 , . . .
, T do 9:
Given the mean-ﬁeld state µ , take action (cid:101) u t following π K,b and generate the next state (cid:101) x t +1 .
Set δ t +1 ← ζ 1 t + ( (cid:98) ψ t − (cid:98) ψ t +1 ) (cid:62) ζ 2 t − c t .
Update parameters via ζ 1 t +1 ← (1 − γ t +1 ) · ζ 1 t + γ t +1 · c t and ζ 2 t +1 ← ζ 2 t − γ t +1 · δ t +1 · (cid:98) ψ t .
Project ζ t to some compact set V (cid:48) ζ .
13: end for 14: Set (cid:98) α K,b ← ( (cid:80) Tt =1 γ t ) − 1 · ( (cid:80) Tt =1 γ t · ζ 2 t ) , and (cid:98)
Υ K ← smat ( (cid:98) α K,b, 1 ) , (cid:18)(cid:98) p K,b (cid:98) q K,b (cid:19) ← (cid:98) α K,b, 2 − (cid:98)
Υ K (cid:18) (cid:98) µ K,b − K
(cid:98) µ K,b + b (cid:19) ,
where (cid:98) α K,b, 1 =
( (cid:98) α K,b ) ( k + d +1)
( k + d ) / 2and (cid:98) α K,b, 2 = ( (cid:98) α K,b ) ( k + d +3)
( k + d ) / 2 ( k + d +1)
( k + d ) / 2+1 .
15: Output: Estimators (cid:98) µ K,b , (cid:98)
Υ K , and (cid:98) q K,b .
Note that in related literature (Bhandari et al., 2018; Korda and La, 2015), non-asymptotic convergence analysis of TD(0) method with linear function approximation is only applied to discounted
MDP; as for our ergodic setting, the convergence of TD(0) method is only shown asymptotically (Borkar and Meyn, 2000; Kushner and Yin, 2003) using ordinary differential equation method.
Therefore, in the convergence theorem proposed in § 3, we only focus on the primal-dual gradient temporal difference method (Algorithm 3) to establish non-asymptotic convergence result.
However, in practice, we still use TD(0) algorithm.
C G ENERAL F ORMULATION
Compared with Problem 2.3, a more general formulation includes an additional term x (cid:62) t Pµ in the cost function.
For the completeness of this paper, we deﬁne this general formulation here.
Following from a same argument as in § 2, it sufﬁces to study the setting where t is sufﬁciently large.
First, we propose the following general drifted LQR (general D-LQR) problem, which is parallel to Problem 2.2.
Problem C.1 (General D-LQR) .
For any given mean-ﬁeld state µ ∈ R m , consider the following formulation
[Equation]
where x t ∈ R m is the state vector, u t ∈ R k is the action vector generated by the policy π , ω t ∈ R m is an independent random noise term following the Gaussian distribution N (0 , Ψ ω ) , and d ∈ R m is a drift term.
We aim to ﬁnd an optimal policy π ∗ µ such that (cid:101) J µ ( π ∗ µ ) =
inf π ∈ Π (cid:101) J µ ( π ) .
In Problem C.1, the unique optimal policy π ∗ µ ( · ) still admits a linear form π ∗ µ ( x t ) =
− K π ∗ µ x t + b π ∗ µ (Anderson and Moore, 2007), where the matrix K π ∗ µ ∈ R k × m and the vector b π ∗ µ ∈ R k are the parameters of the policy π .
It then sufﬁces to ﬁnd the optimal policy in the class Π deﬁned in (2.1).
Parallel to Problem 2.3, we deﬁne the general LQ-MFG problem as follows. Problem C.2 (General LQ-MFG) . We consider the following formulation [Equation]
where x t ∈ R m is the state vector, u t ∈ R k is the action vector generated by the policy π , µ ∈ R m is the mean-ﬁeld state, ω t ∈ R m is an independent random noise term following the Gaussian distribution N (0 , Ψ ω ) , and d ∈ R m is a drift term.
We aim to ﬁnd a pair ( µ ∗ , π ∗ ) such that (i) (cid:101) J
( π ∗ , µ ∗ ) =
inf π ∈ Π (cid:101) J ( π, µ ∗ ) ; (ii) E x ∗ t converges to µ ∗ as t → ∞ , where { x ∗ t } t ≥ 0 is the Markov chain of states generated by the policy π ∗ .
One can see that Problem C.2 aims to ﬁnd a Nash equilibrium pair ( µ ∗ , π ∗ ) .
Similar to the discussions in § 3.2, to solve Problem C.2, one can design an algorithm similar to
Algorithm 1, which solves Problem C.1 and obtain the new mean-ﬁeld state at each iteration.
We omit the detailed algorithm here.
Now, we focus on solving Problem C.1 in the sequel.
Similar to § 3.3, we drop the subscript µ when we focus on Problem C.1 for a ﬁxed µ .
We write π K,b ( x ) =
− Kx + b + σ · η to emphasize the dependence on K and b , and (cid:101) J ( K, b ) = (cid:101) J ( π K,b ) consequently.
We derive an explicit form of the expected total cost (cid:101) J ( K, b ) in the following proposition.
Proposition C.3.
The expected total cost (cid:101) J ( K, b ) in Problem C.1 is decomposed as [Equation] b ) + σ [Equation]
· tr
[Equation]
R ) + µ [Equation] Qµ, where (cid:101) J 1 ( K ) and (cid:101) J 2 ( K, b ) take the forms of
[Equation]
Here µ K,b is given in (3.2), Φ K is given in (3.3), and P K is given in (3.4).
Proof.
The proof is similar to the one of Proposition B.2.
Thus we omit it here.
Compared with the form of J ( K, b ) given in (3.5), we see that the only difference is that (cid:101) J ( K, b ) contains an extra term 2 µ (cid:62)
Pµ K,b in (cid:101)
J 2 ( K, b ) , which is only a linear term in b (recall that µ K,b is linear in b by (3.2)).
Thus, (cid:101) J 2 ( K, b ) is still strongly convex in b , as shown in the proposition below.
Proposition C.4.
Given any K , the function (cid:101) J 2 ( K, b ) is ν K -strongly convex in b , here ν K =
σ min ( Y (cid:62) 1 ,K Y 1 ,K + Y (cid:62) 2 ,K Y 2 ,K ) , where Y 1 ,K = R 1 / 2 K ( I − A + BK ) − 1 B − R 1 / 2 and Y 2 ,K =
Q 1 / 2 ( I − A + BK ) − 1 B .
Also, (cid:101) J 2 ( K, b ) has ι K -Lipschitz continuous gradient in b , where ι K is upper bounded such that ι [Equation] ≤ [
[Equation]
Proof.
The proof is similar to the one of Proposition 3.3.
Thus we omit it here.
Parallel to Proposition 3.4, we derive a similar proposition in the sequel.
Proposition C.5.
Denote by (cid:101) b K = argmin b (cid:101) J 2 ( K, b ) , then (cid:101) J 2 ( K, (cid:101) b K ) takes the form (cid:101) [Equation] [Equation]
[Equation]
b
[Equation]
=
[Equation]
which is independent of K .
Here S =
[( I − A ) Q − 1 ( I − A ) (cid:62) + BR
[Equation]
1 B [Equation] [Equation]
And (cid:101)
b K takes the form
[Equation]
Proof.
The proof is similar to the one of Proposition 3.4.
Thus we omit it here.
Similar to Problem 2.2, we deﬁne the state- and action-value functions as
[Equation]
where the x (cid:48) is the state generated by the state transition after the state-action pair ( x, u ) .
A slight modiﬁcation of Proposition B.1 gives the proposition below.
Proposition C.6.
For Problem C.1, the state-value function (cid:101) V K,b ( x ) takes the form
[Equation]
and the action-value function (cid:101) Q K,b ( x, u ) takes the form (cid:101)
[Equation]
Here the matrix Υ K is given in (3.7), and the vectors (cid:101) p K,b , (cid:101) q K,b are given as
[Equation]
where the vector (cid:101) f K,b = ( I − A + BK ) −(cid:62)
[( A − BK ) (cid:62) P K ( Bb + Aµ + d ) − K (cid:62)
Rb + Pµ ] .
Proof.
The proof is similar to the one of Proposition B.1.
Thus we omit it here.
Now we establish the gradients of (cid:101) J ( K, b ) for Problem C.1.
Proposition C.7.
The gradient of (cid:101) J 1 ( K ) and the gradient of (cid:101) J 2 ( K, b ) w.r.t.
b takes the form ∇
[Equation]
where the matrix Υ K is given in (3.7), and the vector (cid:101) q K,b is given in (C.1).
Proof.
The proof is similar to the one of Proposition B.3.
Thus we omit it here.
Equipped with above results, parallel to the analysis in § 3, it is clear that by slight modiﬁcation of Algorithms 1, 2, and 3, we derive similar actor-critic algorithms to solve both Problem C.2 and
Problem C.1, where all the non-asymptotic convergence results hold.
We omit the algorithms and the convergence results here.
D P ROOFS OF T HEOREMS D. P ROOF OF T HEOREM 4.
We denote by µ ∗ s +1 = Λ( µ s ) , which is the mean-ﬁeld state generated by the optimal policy π K ∗ ,b ∗ ( µ s )
=
Λ 1 ( µ s ) under the current mean-ﬁeld state µ s (note that for any mean-ﬁeld state µ , the optimal K ∗ ( µ ) is independent of µ by Proposition 3.4; therefore, we write K ∗ here for convenience).
By (3.2), we know that
[Equation]
Also, we denote by
[Equation]
which is the mean-ﬁeld state generated by the policy π K s ,b s under the current mean-ﬁeld state µ s .
According to Algorithm 1, it is obvious that the output µ s +1 is an estimator of (cid:101) µ s +1 .
By triangular inequality, we have
[Equation]
We bound E 1 , E 2 and E 3 in the sequel.
Bound on E 1 : Note that from Theorem B.8 and the choice of the inputs in Algorithm 3 stated in
Theorem B.4, it holds that
[Equation]
with probability at least 1 − ε 10 , where ε s is given in (4.2).
Bound on E 2 : Note that by deﬁnition, combining with triangular inequality, we have
[Equation]
For the term (cid:107) Bb ∗ ( µ s )
+ Aµ s + d (cid:107) 2 , combining Proposition
[Equation] [Equation]
it holds that
[Equation]
where the scalar L 1 is given in Assumption 3.1.
Moreover, from Theorem B.
[Equation]
by the choice of N s , it holds with probability at least 1 − ε 10 that
[Equation]
(D.5) Combining (D.3), (D.4), (D.5) and the choice of ε s in (4.2), we deduce that
E 2 ≤ ε/ 8 · 2 − s (D.6) holds with probability at least 1 − ε 10 .
Bound on E 3 : We have
[Equation]
where we use the fact that the operator Λ( · ) has Lipschitz constant L
=
L 1 L 3 + L 2 according to
Proposition 3.2.
From (D.2), (D.6) and (D.7), combining (D.1), we know that
[Equation]
By telescoping (D.8), we obtain that
[Equation]
Moreover, by the choice of S in (4.1) and the deﬁnition of L 0 in Assumption 3.1, we know that (cid:107) µ S − µ ∗ (cid:107) < ε .
This concludes the theorem.
D. P ROOF OF T HEOREM B.
Proof.
We ﬁrst show that J 1 ( K N ) − J 1 ( K ∗ ) <
ε/ 2 with a high probability, then show that
J 2 ( K N , b H ) − J 2 ( K ∗ , b ∗ ) <
ε/ 2 with a high probability.
Then we have
J
[Equation]
K N
[Equation]
b
[Equation]
b
[Equation]
<
[Equation]
with a high probability, which proves the theorem.
Part 1.
We show that J 1 ( K N ) − J 1 ( K ∗ ) <
ε/ 2 with a high probability.
We proceed our proof by ﬁrst showing that J 1 ( K ) is gradient dominant, and then by gradient dominance, we prove the convergence of Algorithm 2 combining the convergence result of Algorithm 3 in Theorem B.8.
To show that J 1 ( K ) is gradient dominant, we ﬁrst bound J 1 ( K 1 ) − J 1 ( K 2 ) for any K 1 and K 2 .
By
Proposition B.2, J 1 ( K ) takes the form
[Equation]
We use the following cost difference lemma to bound y (cid:62) P K 1 y − y
(cid:62) P K 2 y .
Lemma D.1 (Cost Difference) .
Denote by K 1 and K 2 two parameters, which satisfy that ρ ( A −
BK 1 ) < 1 and ρ ( A − BK 2 ) < 1 .
Moreover, for any state vector y , we denote by { y t } t ≥
the sequence generated by the transition y t
+1 =
( A − BK 2 ) y t with initial state y
= y .
Then it satisﬁes that
[Equation]
Here we denote by the function
[Equation]
where the matrix Υ K is given in (3.7).
Proof.
See § F.1 for a detailed proof.
Now, based on Lemma D.1, the following lemma shows that J 1 ( K ) is gradient dominant.
Lemma D.2 (Gradient Dominance) .
Let K ∗ be the optimal parameter and K be a parameter such that J 1 ( K ) is ﬁnite, then we have the following lower bound for J 1 ( K ) − J 1 ( K ∗ )
[Equation]
and the following upper bound
[Equation]
[Table]
Proof.
See § F.2 for a detailed proof.
We now use the above results to show the convergence of Algorithm 2.
Recall that according to
Algorithm 2, the parameter K is updated via
[Equation]
where (cid:98) Υ K n is the output of Algorithm 3.
We also deﬁne (cid:101) K n +1 as the exact update (cid:101)
[Equation]
) where Υ K n is given in (3.7)
[Equation]
We proceed to bound | J 1 ( K n +1 ) − J 1 ( K ∗ ) | in the sequel.
First, we claim that J 1 ( K N ) ≤
J 1 ( K N − 1 ) ≤ · · · ≤ J 1 ( K 0 ) with high probability.
We prove the claim by mathematical induction.
Suppose that J 1 ( K n ) ≤ J 1 ( K n − 1 ) ≤ · · · ≤ J 1 ( K 0 ) (this holds for n = 0 ).
Recall the deﬁnition of (cid:101) K n +1 in (D.11) and the deﬁnition of J 1 ( K ) given in (3.6), we have
[Equation]
) where the ﬁrst line comes from (D.9) and the second line comes from Lemma D.1.
Note that by the deﬁnition of Υ K in (3.7), the term (cid:107) Υ 22 K n (cid:107) ∗ is upper bounded as
[Equation]
Combining (D.12) and the choice of stepsize γ ≤ [ (cid:107) R (cid:107) ∗ + (cid:107) B (cid:107) 2 ∗ · J 1 ( K 0 ) · σ − 1min (Ψ (cid:15) )] − 1 , it holds that
[Equation]
≤
[Equation] [Equation]
where the last line comes from Lemma D.2.
This also implies that J 1 ( (cid:101) K n +1 ) ≤ J 1 ( K n ) .
Now, we use the following lemma to establish an upper bound of | J 1 ( (cid:101) K n +1 ) − J 1 ( K n +1 ) | .
Lemma D.3.
Suppose that J 1 ( K n ) ≤ J 1 ( K 0 ) .
Under the conditions stated in Theorem B.4, with probability at least 1 − ε 15 , it holds that
[Equation]
where K n +1 is given in (D.10) and (cid:101) K n +1 is given in (D.11).
Proof.
See § F.10 for a detailed proof.
Now, by applying Lemma D.3, we know that if J 1 ( K n ) − J 1 ( K ∗ ) ≥ ε/ 2 , combining (D.13) and (D.14), it holds with probability at least 1 − ε 15 that
[Equation]
This shows that with probability at least 1 − ε 15 , it holds that J 1 ( K n +1 ) ≤ J 1 ( K n ) ≤ · · · ≤ J 1 ( K 0 ) .
Finally, note that the total number of iterations N goes only with log(1 /ε ) , we know that J 1 ( K N ) ≤
J 1 ( K N − 1 ) ≤ · · · ≤ J 1 ( K 0 ) with probability at least 1 − ε 13 , as long as J 1 ( K n ) − J 1 ( K ∗ ) ≥ ε/for any n ≤ N .
This ﬁnishes the proof of the claim.
Also, when J 1 ( K n ) − J 1 ( K ∗ ) ≥ ε/ 2 , by (D.13) and (D.15), with probability at least 1 − ε 13 , it holds that
[Equation]
Again, combining the fact that N goes only with log(1 /ε ) , we know that
[Equation]
with probability at least 1 − ε 11 .
Now we only need to give an error bound of (cid:107) K n − K ∗ (cid:107) F .
We upper bound (cid:107) K − K ∗ (cid:107) F using
J 1 ( K ) − J 1 ( K ∗ ) in the following lemma.
Lemma D.4.
For any K , we have
[Equation]
Proof.
See § F.3 for a detailed proof.
Combining Lemma D.4, we show the error bound (cid:107) K N − K ∗ (cid:107) F
[Equation]
Part 2.
We show that J 2 ( K N , b H ) − J 2 ( K ∗ , b ∗ ) <
ε/ 2 with a high probability.
Following from
Proposition 3.4, it holds that J 2 ( K ∗ , b ∗ ) =
J 2 ( K N , b K N ) .
Therefore, to show that J 2 ( K N , b H ) −
J 2 ( K ∗ , b ∗ ) < ε/ 2 , we only need to prove that J 2 ( K N , b H ) − J 2 ( K ∗ , b ∗ ) < ε/ 2 .
First, we use mathematical induction to show that J 2 ( K N , b H ) ≤ J 2 ( K N , b H − 1 ) ≤ · · · ≤
J 2 ( K N , b 1 ) ≤ J 2 ( K N , b 0 ) , as long as J 2 ( K N , b h ) − J 2 ( K N , b K N ) ≥ ε/ 2 for any h ≤ H .
Suppose that J 2 ( K N , b h ) ≤ J 2 ( K N , b h − 1 ) ≤ · · · ≤ J 2 ( K N , b 0 ) (this holds for h = 0 ).
Recall that according to Algorithm 2, the parameter b is updated by
[Equation]
(D
[Equation]
16) where (cid:98) ∇ b J 2 ( K N , b h )
= (cid:98) Υ 22 K N ( − K N (cid:98) µ K N ,b h + b h )
+ (cid:98) Υ 21 K N (cid:98) µ K N ,b h + (cid:98) q K N ,b h is the estimated gradient at b h , and the matrix (cid:98) Υ K N and the vector (cid:98) q K N ,b h are the outputs of Algorithm 3.
We deﬁne (cid:101)
b h +1 as an exact update (cid:101)
[Equation]
(D.17) where ∇ b J 2 ( K N , b h ) =
Υ 22 K N ( − K N µ K N ,b h + b h ) + Υ 21 K N µ K N ,b h + q K N ,b h is the exact gradient at b h , and Υ K N and q K N ,b h are given in (3.7).
We proceed to bound J 2 ( K N , b h +1 ) − J 2 ( K N , b K N ) in the sequel, where b K N =
argmin b
J 2 ( K N , b ) .
Note that by the convexity and smoothness of J 2 ( K N , b ) in Proposition 3.3, we have
J
[Equation]
K N ,
[Equation]
where in the inequalities, we use the fact that J 2 ( K N , b ) has ι K N -Lipschitz continuous gradient and is ι K N -strongly convex by Proposition 3.3.
Note that (D.18) also implies that J 2 ( K N , (cid:101)
b h +1 ) <
J 2 ( K N , b h ) .
Now, we only need to establish the error bound |
J 2 ( (cid:101) K N , b h +1 ) − J 2 ( K N , b h +1 ) | .
The following lemma quantiﬁes (cid:12)(cid:12)
[Equation]
K N
[Equation]
b
[Equation]
−
[Equation]
[Equation]
N
[Equation]
b
[Equation]
+
[Equation]
(cid:12)(cid:12) .
Lemma D.5.
Assume that J 2 ( K N , b h ) ≤ J 2 ( K N , b 0 ) .
Under the conditions stated in Theorem B.4, with probability at least 1 − ε 15 , we have
[Equation]
where (cid:98) ∇ b J 2 ( K N , b h ) is the approximate gradient given in (D.16), while ∇ b J 2 ( K N , b h ) is the exact gradient given in (D.17).
Proof.
See § F.4 for a detailed proof.
Now, by applying Lemma D.5, we know that if J 2 ( K N , b h ) − J 2 ( K N , b K N ) ≥ ε , combining (D.18), it holds with probability at least 1 − ε 15 that
[Equation]
This shows that with probability at least 1 − ε 15 , it holds that J 2 ( K N ,
b h +1 ) ≤ J 2 ( K N , b h ) ≤ · · · ≤ J 2 ( K N , b 0 ) .
Note that the total number of iterations H only goes by log(1 /ε ) .
Therefore, we know that as long as J 2 ( K N , b h ) − J 2 ( K N , b K N ) ≥ ε for any h ≤ H , it holds that J 2 ( K N , b H ) ≤
J 2 ( K N , b H − 1 ) ≤ · · · ≤ J 2 ( K N , b 0 ) with probability at least 1 − ε 13 .
By this, we ﬁnish the claim.
Then, if J 2 ( K N , b h ) − J 2 ( K N , b K N ) ≥ ε/ 2 , by (D.18) and (D.19), with probability at least 1 − ε 13 , it holds that
[Equation]
Again, combining the fact that H goes only with log(1 /ε ) , we know that
[Equation]
with probability at least 1 − ε 11 .
Moreover, combining the fact in Proposition 3.3 that J 2 ( K N , b ) is strongly convex in b , we concludes the theorem.
D. P ROOF OF T HEOREM B.
Proof.
We follow the proof of Theorem 4.2 in Yang et al.
(2019), where the authors only consider
LQR without drift terms.
Since our proof requires much more delicate analysis, we present it here.
Part 1.
We proceed to show that ( ζ K,b , 0) is a saddle point of the problem (B.9).
We ﬁrst show that ζ K,b ∈ V ζ and ξ ( ζ ) ∈ V ξ for any ζ ∈ V ζ , where ξ ( ζ ) is deﬁned as ξ ( ζ ) = argmax ξ F ( ζ, ξ ) , by the following lemma.
Lemma D.6.
The vector ζ K,b =
( J ( K, b ) , α (cid:62) K,b ) (cid:62) ∈ V ζ .
Moreover, for any ζ ∈ V ζ , the vector ξ ( ζ ) deﬁned above satisﬁes that ξ ( ζ ) ∈ V ξ .
Proof.
See § F.5 for detailed proof.
Note that ∇ ζ F ( ζ K,b , 0)
=
and ∇ ξ F ( ζ K,b , 0) = 0 , combining the above Lemma D.6, we know that ( ζ K,b , 0) is a saddle point of the function F ( ζ, ξ ) deﬁned in the problem (B.9).
This ﬁnishes the proof of our Part 1 .
Part 2.
We deﬁne the primal-dual gap of (B.9) as
[Equation]
D
[Equation]
which indeed captures the performance of ( (cid:98) ζ, (cid:98) ξ ) .
In the sequel, we proceed to relate (D.20) with the estimation error (cid:107) (cid:98) α K,b − α K,b (cid:107) 2 .
First, note that for the estimator (cid:98) ζ returned by Algorithm 3, we have
[Equation]
21) where the second line comes from (F.14) and the deﬁnition of ξ ( ζ ) in Part 1 .
Moreover, note that for any ξ ∈ V ξ
[Equation]
the follows hold:
[Equation]
D [Equation] where the ﬁrst line comes from the deﬁnition of ξ ( ζ ) in Part 1 , the second line comes from (F.14), while the last equality holds by taking ζ = ζ K,b ∈ V ζ .
Further, we establish a lower for the LHS of (D.21) as follows
[Equation]
where the matrix (cid:101) Θ K,b is given in (B.8) and the scalar λ K is speciﬁed in Proposition B.6.
Combining (D.21), (D.22) and (D.23), we obtain the relation between gap ( (cid:98) ζ, (cid:98) ξ ) and (cid:107) (cid:98) α K,b − α K,b (cid:107) 2 :
[Equation]
(cid:98) α K,b − α K,b (cid:107) 22 ≤ λ − 2 K · gap
[Equation]
(cid:98)
[Equation]
[Equation] [Equation]
Part 3.
We now proceed to upper bound the primal-dual gap gap ( (cid:98) ζ, (cid:98) ξ ) , then by (D.24) in Part 2 , we establish the upper bound of the estimation error (cid:107) (cid:98) α K,b − α K,b (cid:107) 2 .
Note that since the state x and action u follow Gaussian distributions, therefore, they are unbounded.
First we utilize Hansen-Wright inequality (Lemma G.3) to pick up an event where these two random variables are bounded.
Before pick up such an event, the following Lemma D.7 characterize the distribution of (cid:98) µ z = 1 / (cid:101) T · (cid:80) (cid:101)
Tt =1 (cid:101) z t , where (cid:101) z t
=
[ (cid:101) x
(cid:62) t , (cid:101) u (cid:62) t ] (cid:62) is the concatenation of the state (cid:101) x t and the action (cid:101) u t .
Then following transition holds
[Equation]
where ν
= (cid:18)
Aµ + d − K ( Aµ + d ) + b (cid:19)
[Equation]
δ t
[Equation]
(cid:19) ,
[Equation]
(cid:18)
[Equation]
B
[Equation]
(cid:19) .
Note that we have
L = (cid:18)
[Equation]
B
[Equation]
KB (cid:19) =
(cid:18)
I
[Equation]
K (cid:19) ( A B ) .
Then by the property of spectral radius, it holds that
[Equation]
Thus, the Markov chain { (cid:101) z t } t ≥ 0 generated by the transition admits a unique stationary distribution
N ( µ z , Σ z ) .
The following lemma characterizes the average (cid:98) µ z = 1 / (cid:101) T · (cid:80) (cid:101)
Tt =1 (cid:101) z t .
Lemma D.7.
The average (cid:98) µ z = 1 / (cid:101) T · (cid:80) (cid:101)
Tt =1 (cid:101) z t , where (cid:101) z t
=
[ (cid:101) x
(cid:62) t , (cid:101) u (cid:62) t ] (cid:62) , satisﬁes that
[Equation]
z
[Equation]
N
[Equation]
where µ (cid:101) T and (cid:101) Σ (cid:101) T are bounded such that there exist positive absolute constants M µ and M Σ such that (cid:107) µ (cid:101) T (cid:107) 2 ≤ M µ · (1 − ρ ) − 2 · (cid:107) µ z (cid:107) 2 and (cid:107) (cid:101) Σ (cid:101) T (cid:107) F ≤ M Σ · (1 − ρ ) − 1 · (cid:107) Σ z (cid:107) F .
This gives that with probability at least 1 − (cid:101) T − 6 , it holds that
[Equation]
Proof.
See § F.6 for a detailed proof.
The above lemma also gives the error bound (cid:107) (cid:98) µ K,b − µ K,b (cid:107) 2 .
Now, note that for any z ∼ N ( µ z , Σ z ) , the random variable z − (cid:98) µ z + 1 / (cid:101) T · µ (cid:101) T follows N (0 ,
Σ z + 1 / (cid:101) T · (cid:101) Σ (cid:101) T ) .
By Lemma G.3, there exists an absolute constant C 0 > 0 such that
[Equation]
where, for notational convenience, we write the matrix (cid:101)
Σ z = Σ
z + 1 / (cid:101) T · (cid:101) Σ (cid:101) T .
By taking τ =
C 1 · log T · (cid:107) (cid:101)
Σ z (cid:107) F for some sufﬁciently large positive absolute constant C 1 , the above inequality takes the form
[Equation]
We deﬁne the following event for any t ∈
[ T ] :
[Equation]
Then by (D.25), we know that P ( E t, 1 ) ≥ 1 − T − 6 for any t ∈ [ T ] .
Also, we deﬁne E 1 = ∩ t ∈
[ T ] E t, 1 , then we know that P ( E 1 ) ≥ 1 − T − 5 by union bound.
Also, by the deﬁnition of E 1 , conditioning on the event E 1 , it holds that
[Equation]
[Equation]
for sufﬁciently large (cid:101) T .
Here C 2 and C 3 are positive absolute constants.
Moreover, we deﬁne the following event
[Equation]
Then by Lemma D.7, we know that P ( E 2 ) ≥ 1 − (cid:101) T − 6 for sufﬁciently large (cid:101) T .
We deﬁne the event E as the intersection of the event E 1 and E 2 , then by union bound, we know that P ( E ) ≥ 1 − T − 5 − (cid:101) T − 6 .
Now, we deﬁne the truncated feature vector (cid:101) ψ ( x, u ) as (cid:101) ψ
( x, u ) = (cid:98) ψ ( x, u ) 1 E , and also the truncated objective function as
[Equation]
) where we write (cid:101)
ψ
= (cid:101) ψ
( x, u ) to simplify the notations, the function (cid:101) c
= (cid:101) c
( x, u ) is the truncated cost, which is deﬁned as (cid:101)
c ( x, u ) =
c ( x, u ) 1 E , and the expectation is taken over the trajectory generated by the policy π K,b .
The following lemma establishes the upper bound of | F ( ζ, ξ ) − (cid:101) F ( ζ, ξ ) | , where
F ( ζ, ξ ) and (cid:101) F ( ζ, ξ ) are given in (B.9) and (D.27) respectively.
Lemma D.8.
Given F ( ζ, ξ ) and (cid:101) F ( ζ, ξ ) in (B.9) and (D.27), it holds that
[Equation]
with probability at least 1 − (cid:101) T − 6 .
Proof.
See § F.7 for a detailed proof.
By Lemma D.8, we know that with probability at least 1 − (cid:101) T − 6 , it holds that
[Equation]
Therefore, to obtain the bound of gap ( ζ, ξ ) , we only need to bound the term max ξ ∈V ξ (cid:101) F
[Equation]
ζ
[Equation]
ξ ) − min ζ ∈V ζ (cid:101) F ( ζ, (cid:98) ξ ) in (D.28).
We ﬁrst use the following lemma to characterize the dependency of the trajectories generated by the policy π K,b and the state transition in Problem 2.2.
Lemma D.9.
Consider a linear system x t +1 =
Dx t + d + (cid:15) t , where { x t } t ≥
⊂ R m , the matrix
D ∈ R m × m satisfying ρ ( D ) < 1 , the vector d ∈ R m , and (cid:15) t ∼ N (0 , Σ) is the Gaussians.
We denote by (cid:36) t the marginal distribution of x t for each t ≥ 0 .
Besides, it is easy to verify that the stationary distribution of this Markov chain is a Gaussian distribution N
(( I − D ) − 1 d, Σ ∞ ) , where Σ ∞ is the covariance matrix.
We deﬁne the β -mixing coefﬁcients for any n ≥ 1 as follows
[Equation]
Then, for any ρ ∈ ( ρ ( D ) , 1) , the mixing coefﬁcients satisfy that
[Equation]
where the scalar C ρ,D,
d is a constant which only depends on ρ , D and d .
We say that the sequence { x t } t ≥ 0 is β -mixing with parameter ρ .
Proof.
See Proposition 3.1 in Tu and Recht (2017) for a detailed proof.
Note that under the state transition in Problem 2.2, the sequence { x t } t ≥ 0 follows (3.1), where the matrix A − BK satisﬁes that ρ ( A − BK ) < 1 .
Therefore, according to Lemma D.9, we know that the sequence { z t = ( x (cid:62) t , u (cid:62) t ) (cid:62) } t ≥ 0 is β -mixing with parameter ρ ∈ ( ρ ( A − BK ) , 1) .
Now the following lemma helps us to establish the primal-dual gap for a convex-concave problem.
Lemma D.10.
Let X
and Y be two compact and convex sets such that (cid:107) x − x (cid:48) (cid:107) 2 ≤ M and (cid:107) y − y (cid:48)
(cid:107) 2 ≤ M for any x, x (cid:48)
∈ X
and y, y (cid:
) ∈ Y .
We consider solving the following problem
[Equation]
where the objective function F ( x, y ) is convex in x and concave in y .
In addition, we assume that the distribution (cid:36) (cid:15) is the stationary distribution induced by a Markov chain { (cid:15) t } t ≥ 0 , which is βmixing with β ( n ) ≤ C (cid:15) · ρ n , where C (cid:15) is a constant.
Moreover, we assume that it holds almost surely that G ( x, y ; (cid:15) ) is (cid:101)
L 0 -Lipschitz in both x and y , the gradient ∇ x G ( x, y ; (cid:15) ) is (cid:101)
L 1 -Lipschitz in x for any y ∈ Y , the gradient ∇ y G ( x, y ; (cid:15) ) is (cid:101)
L 1 -Lipschitz in y for any x ∈ X , where for simplicity we assume that C (cid:15) , (cid:101) L 0 , (cid:101)
L 1 > 1 .
Each step of our gradient-based method takes the following form:
[Equation]
) (cid:3) , where the operators Γ X and Γ Y projects the variables back to X and Y , respectively, and the stepsizes take the form γ t = γ 0 · t − 1 / 2 for some constant γ 0 > 0 .
Moreover, let (cid:98) x
= ( (cid:80) Tt =1 γ t ) − 1 ( (cid:80)
Tt =1 γ t x t ) and (cid:98)
y
=
( (cid:80) Tt =1 γ t ) − 1 ( (cid:80)
Tt =1 γ t y t ) be the ﬁnal output of the gradient method after T iterations, then there exists an absolute constant C > 0 , such that for any δ ∈ (0 , 1) , the primal-dual gap to the minimax optimization problem satisﬁes that
[Equation]
with probability at least 1 − δ .
Proof.
See Theorem 5.4 in Yang et al.
for a detailed proof.
To use Lemma D.10, we deﬁne the function G ( ζ, ξ ; (cid:101) ψ, (cid:101) ψ (cid:48) ) as
[Equation]
where recall that (cid:101) ψ
= (cid:101) ψ
( x, u ) and (cid:101) ψ (cid:48) =
(cid:101) ψ ( x (cid:48) , u (cid:48) ) .
Note that its gradients take the form ∇ ζ G ( ζ, ξ ; (cid:101) ψ, [Equation] ψ
[Equation]
) =
[Equation]
By Condition B.7 and Lemma D.6, we know that
[Equation]
This gives the Lipschitz constant (cid:101) L 0 in Lemma D.10 for G ( ζ, ξ ; (cid:101) ψ, [Equation] ψ [Equation]
Also, the Hessians take the forms
[Equation]
which follows that
[Equation]
This gives the Lipschitz constant (cid:101) L 1 in Lemma D [Equation] 10 for
[Equation]
ψ, (cid:101) ψ
[Equation]
) and ∇ ξ G ( ζ, ξ ; (cid:101) ψ, (cid:101) ψ (cid:48) ) .
Moreover, note that (D.26) provides an upper bound of M , combining (D.29), (D.30) and Lemma
D.10, we know that
[Equation]
holds with probability at least 1 − T − 5 .
Combining (D.24) and (D.28), we conclude the theorem.
E P ROOFS OF P ROPOSITIONS E. P ROOF OF P ROPOSITION 3.
Proof.
We follow a similar argument as shown in the proof of Theorem 1.1 in Sznitman (1991) and
Theorem 3.2 in Bensoussan et al.
Note that for any policy π K,b ∈ Π , the matrix K and the vector b uniquely determine the policy.
This motivates us to deﬁne the metric on Π as follows.
Deﬁnition E.1.
For any π K 1 ,b 1 , π K 2 ,b 2 ∈ Π , we deﬁne the metric on the set of policies Π as
[Equation]
where c 1 and c 2 are positive constants.
One can easily verify that the above deﬁnition satisﬁes the requirement of being a metric.
We compute the Lipschitz constants for both operators Λ 1 ( · ) and Λ 2 ( · , · ) in the sequel.
Lipschitz constant for Λ 1 ( · ) .
Note that by Proposition 3.4, for any µ 1 , µ 2 ∈ R m , the optimal parameter policy K ∗ is ﬁxed for Problem 2.2.
Therefore, by the form of the optimal b K given in
Proposition 3.4, it holds that
[Equation]
Lipschitz constants for Λ 2 ( · , · ) .
Note that by Proposition 3.4, for any µ 1 , µ 2 ∈ R m , the induced optimal parameter policy K ∗ is ﬁxed for Problem 2.2.
We thus have for any π ∈ Π such that π is an optimal policy under some µ ∈ R m
[Equation]
it holds that [Equation] [Equation] Moreover, for any π 1 , π 2 ∈ Π such that there exists µ 1 , µ 2 ∈ R m such that π 1 , π 2 are optimal under µ 1 , µ 2 , and for any population mean µ ∈ R m , we have
[Equation]
Now we proceed to show that the operator is a contraction.
For any µ 1 , µ 2 ∈ R m , the following inequality holds:
[Equation]
where in the second line, we use triangular inequality; in the third line, we use (E.2) and (E.3); in the last line, we use (E.1).
By Assumption 3.1, we know that L 0 =
L 1 L 3 + L 2 < 1 , which shows that the operator Λ( · ) is a contraction.
Therefore, by Banach ﬁxed-point theorem, we conclude that Λ( · ) has a unique ﬁxed point, which gives the equilibrium pair of Problem 2.3.
E. P ROOF OF P ROPOSITION 3.
Proof.
Note that by the deﬁnition of J 2 ( K, b ) in (3.6) and the deﬁnition of µ K,b in (3.2), the problem min
b J 2 ( K, b ) is a constrained optimization program
[Equation]
Consider the KKT conditions of (E.4), the minimizer satisﬁes that
[Equation]
where we that
[Equation]
By solving (E.5)
[Equation]
we obtain the minimizer to the program as follows
[Equation]
By substituting (E.6) into the deﬁnition of J 2 ( K, b ) in (3.6)
[Equation]
we have
[Equation]
Moreover, by algebra, we have
[Equation]
Therefore
[Equation]
the term N (cid:62) K M − 1 K N K in (E.7) takes the following form
[Equation]
Combining (E.7), we have
[Equation]
Also, combining (E.6), we have
[Equation]
Then we ﬁnish the proof of the proposition
[Equation]
E. P ROOF OF P ROPOSITION B.
Proof.
By the deﬁnition of the cost function c ( x, u ) in Problem 2. [Equation] we have
[Equation]
where in the second line we use the form of the linear policy.
Therefore, combining (E.8) and the deﬁnition of J ( K, b ) in Problem 2.
[Equation]
we have
[Equation]
Now, by applying (3.3) and (3.4) repeatedly, we have
[Equation]
where P K is given in (3.4).
This ﬁnishes the proof of the proposition.
E. P ROOF OF P ROPOSITION 3.
Proof.
By calculating the Hessian matrix of J 2 ( K, b ) directly
[Equation]
we have
[Equation]
which is a positive constant matrix, whose minimum singular value is denoted by ν K > 0 .
Moreover, note that the spectral norm of ∇ 2 bb J 2 ( K, b ) is upper bounded as
[Equation]
∗ ·
[Equation]
∗ (cid:1) .
Therefore
[Equation]
we know that the maximum singular value ι K of ∇ 2 bb J 2 ( K, b ) is upper bounded
[Equation]
This ﬁnishes the proof.
E. P ROOF OF P ROPOSITION B.
Proof.
It holds that
[Equation]
also, we have
[Equation]
Combining the above equations
[Equation]
we obtain the form of ∇ K J 1 ( K ) .
Moreover, the form of ∇ b J 2 ( K, b ) is obtained simply by taking gradient w.r
[Equation]
t. b , combining the deﬁnition of µ K,b in (3.2) and the Lyapunov equation (3.3).
This concludes the theorem.
E. P ROOF OF P ROPOSITION B.
Proof.
From the deﬁnition of V K,b ( x ) in (B.1) and the deﬁnition of the cost function c ( x, u ) in
Problem 2.2, it holds that
[Equation]
Combining (3.1)
[Equation]
we see that V K,b ( x ) is indeed a quadratic function taking the form V K,b ( x ) = x (cid:62) Gx
+ r (cid:62) x
+ h , where G , r and h are functions of K and b .
Note that V K,b ( x ) satisﬁes the
Bellman equation,
i
[Equation]
e.,
[Equation]
then by substituting the form of c ( x [Equation] − Kx + b ) in Problem 2.2 and J ( K [Equation]
b ) in (3.5)
[Equation]
we obtain that x
[Equation]
By comparing the quadratic terms and linear terms in (E.9), we obtain that
[Equation]
Also, by the deﬁnition of V K,b ( x ) in (B.1), we know that E [ V K,b ( x )] = 0 .
Therefore, we have
[Equation]
This ﬁnishes the proof of (B.3).
For the action-value function Q K,b ( x, u ) , by the deﬁnition in (B.2) and (B.3), we obtain (B.4).
This ﬁnishes the proof of the proposition.
E. P ROOF OF P ROPOSITION B.
Proof.
By Proposition B.1, it holds that Q K,b takes the following linear form
[Equation]
where β K,b is a scalar independent of x and u .
Given the ergodic cost, recall that the Bellman equation takes the form
[Equation]
where ( x (cid:48) , u (cid:48) ) is the state-action pair after ( x, u ) following the policy π K,b and the state transition.
Combining (E.10) and (E.11), we obtain that
[Equation]
By left multiplying ψ ( x, u ) to both sides of (E.12), and taking the total expectation, we have E [Equation] K
[Equation]
Combining the deﬁnition of the matrix Θ K,b in (B.7), we conclude the proposition.
E. P ROOF OF P ROPOSITION B.
Proof.
Invertibility and Upper Bound.
We ﬁrst introduce some notations.
We deﬁne the vector z =
( x (cid:62) , u (cid:62) )
(cid:62) .
Then following the state transition and the policy π K,b , the transition of z takes the form
[Equation]
where the matrix L and vectors ν , δ are given as
[Equation]
Note that the matrix L also takes the form
[Equation]
Combining the fact that ρ ( UV ) =
ρ ( V U ) for any matrices
U and V , we know that ρ ( L ) =
ρ ( A −
BK ) < 1 , which gives the stability of (E.13).
Note that by (E.13), we see that the mean µ z and the covariance
Σ
z satisfy the following equations
[Equation]
where the matrix Ψ
δ is the covariance of the vector δ , which takes the form
[Equation]
Also, the covariance matrix Σ z takes the following form
[Equation]
+ (cid:18)
[Equation]
− K (cid:19)
Φ K (cid:18)
I − K (cid:19) (cid:62) .
(E.15)
Now, we establish the following lemma.
Lemma E.2.
The matrix Θ K,b in (B.7) takes the following form
[Equation]
Proof.
See § F.8 for a detailed proof.
Note that since ρ ( L ) < 1 , both the matrices I − L ⊗ s L and I − L are positive deﬁnite.
Therefore, by Lemma E.2, the matrix Θ K,b is invertible.
This ﬁnishes the proof of the invertibility of the matrix Θ K,b .
Moreover, from (E.15) and Lemma E.2, we upper bound the spectral norm of Θ K,b as
[Equation]
This proves the upper bound of the spectral norm.
Minimum singular value.
To lower bound σ min ( (cid:101) Θ K,b ) , we only need to upper bound σ max ( (cid:101)
Θ − 1 K,b ) .
We ﬁrst proceed to calculate (cid:101)
Θ − 1 K,b .
Recall that the matrix (cid:101) Θ K,b in the linear system (B.8) takes the following form
[Equation]
By the deﬁnition of the feature vector ψ ( x, u ) , we know that the vector (cid:101) σ z = E π K,b [ ψ ( x, u )] takes the form
[Equation]
where 0 k + m denotes the all-zero column vector with dimension k + m .
Also, since Θ K,b is invertible, the matrix (cid:101)
Θ K,b is also invertible, whose inverse is given as
[Equation]
The following lemma characterizes the spectral norm of the matrix (cid:101)
Θ − 1 K,b .
Lemma E.3.
The spectral norm of the matrix (cid:101) Θ − 1 K,b is upper bounded by some positive constant (cid:101) λ K , where (cid:101) λ K only depends on ρ ( A − BK ) , σ and σ min (Ψ ω ) .
Proof.
See § F.9 for a detailed proof.
By Lemma E.3, we know that minimum singular value of the matrix (cid:101) Θ K,b is lower bounded by a positive constant λ K = 1 / (cid:101) λ K , which only depends on ρ ( A − BK ) , σ and σ min (Ψ ω ) .
This concludes the proposition.
F P ROOFS OF L EMMAS F. P ROOF OF L EMMA D.
Proof.
By the fact that P K 2 satisﬁes the Bellman equation in (3.4), we have
[Equation]
By the transition y t +1 =
( A − BK 2 ) y t , we know that y t
=
( A − BK 2 ) t y .
Therefore, it holds that
[Equation]
Then by telescoping
[Equation]
we have
[Equation]
Also, for any t ≥ 0 , we have
[Equation]
Combining (F.1) and (F.2), we ﬁnish the proof.
F. P ROOF OF L EMMA D.
Proof.
Upper Bound.
From the deﬁnition of J 1 ( K ) in (3.6), we have
[Equation]
where in the last equality, we apply Lemma D.1 and { y t } t ≥ 0 follows the transition y t
+1 =
( A −
BK ∗ ) y t .
Also, we write D K,K ∗ ( y ) as
[Equation]
Note that the ﬁrst term on the RHS of (F.4) is positive, due to the fact that it is a quadratic form of a positive deﬁnite matrix, we therefore lower bound D K,K ∗ ( y ) as
[Equation]
Therefore, combining (F.3) and (F.5), it holds that
[Equation]
where the second line comes from direct computation, while the last line comes from the fact that Υ 22 K
= R + B
[Equation]
KP B (cid:22) R .
This complete the proof of the upper bound.
Lower Bound.
Note that for any (cid:101) K , it holds that
[Equation]
where { y t } t ≥ 0 follows the transition y t
+1 =
( A − B (cid:101) K ) y t .
Therefore, by choosing (cid:101) K = K − (Υ 22 K ) − 1 (Υ 22 K K − Υ 21 K ) , by a similar calculation in (F.4), the function D K, (cid:101) K ( y ) takes the form
[Equation]
F [Equation]
Combining (F.6) and (F.7), we obtain a lower bound as
[Equation]
Here in the last line, we use the fact that Φ (cid:101) K (cid:23) Ψ (cid:15) .
This ﬁnishes the proof of the lower bound.
F. P ROOF OF L EMMA D.
Proof.
Note that from Lemma D.1, it holds that
[Equation]
where { y t } t ≥ 0 follows the transition y t
+1 =
( A − BK )
y t .
Here in the second line, we use the fact
that Υ 22 K ∗ K ∗ − Υ 21 K ∗ is the natural gradient of J 1 ( · ) evaluated at K ∗ , which implies that Υ 22 K ∗ K ∗ − Υ 21 K ∗ = 0 ; while in the last line, we use the fact that (cid:107) Φ K (cid:107) ∗ ≥ σ min (Ψ (cid:15) ) and (cid:107) Υ 21 K ∗ (cid:107) ∗ ≥ σ min ( R ) .
This concludes the lemma.
F. P ROOF OF L EMMA D.
Proof.
Note that by Proposition 3.3, we have
[Equation]
Also, we upper bound (cid:107)∇ b J
[Equation]
K
[Equation]
b
[Equation]
as (cid:13)(cid:13) ∇ b J
[Equation]
(cid:13)(cid:13) 2 ≤ poly
[Equation]
A − BK N )
[Equation] [Equation]
F.9)
Combining (F.8), (F.9) and the fact that ν K N ≤ ι K N ≤ [1 − ρ ( A − BK N )] − 2 · poly 2 (
[Equation]
K N (cid:107)
[Equation]
) , we know that
[Equation]
≤
[Equation]
Note that from the deﬁnition of ∇ b J 2 ( K N , b h ) and (cid:98) ∇ b J 2 ( K N , b h ) , we have (cid:13)(cid:13)
[Equation]
[Table] [Table]
[Equation]
[Table]
≤
[Table]
[Equation]
From Theorem B.8, combining the fact that J 2 ( K N , b h ) ≤ J 2 ( K N , b 0 ) and the fact that (cid:107) µ K N ,b (cid:107) 2 ≤ J ( K N , b 0 ) /σ min ( Q ) , we know that with probability at least 1 − ( T b ) − 4 − ( (cid:101) T b ) − 6 , it holds that
[Equation]
bn ) 1 / 8 · (1 − ρ ) (cid:21) .
To show that (D.19) holds, we only need to pick γ b , T bn and (cid:101) T bn such that γ
[Equation]
N
[Equation] [Equation]
≤ ν K N · γ b ·
[Equation] [Equation]
in other words, we pick γ
[Equation]
A
[Equation]
[Table] [Table]
then (D.19) holds with probability at least 1 − ε 15 .
This corresponds to the choices of parameters in the statement of Theorem B
[Table]
We then ﬁnish the proof
[Table]
Proof.
Part 1.
First we proceed to prove that ζ K,b ∈ V ζ .
Note that from Condition B.7, we know that ζ 1 K,b
= J ( K, b ) satisﬁes that 0 ≤ ζ 1 K,b ≤ J ( K 0 , b 0 ) .
It remains to show that ζ 2 K,b
= α K,b satisﬁes that (cid:107) ζ 2 K,b (cid:107) 2 ≤ M ζ .
By the deﬁnition of α K,b in (B.6), we know that
[Equation]
where for notational simplicity, we denote by µ uK,b the mean-ﬁeld action, i.e., µ uK,b = − Kµ K,b + b .
We only need to bound Υ K , µ K,b , µ uK,b , P K , and f K,b .
Note that the expected total cost J ( K, b ) takes the form
[Equation]
Thus, we have
[Equation]
which imply that
[Equation]
For Υ K
[Equation]
note that we decompose the matrix in the following way:
[Equation]
which gives
[Equation]
Combining (F.11) and the fact that (cid:107) U (cid:107) F ≤ √ m · (cid:107) U (cid:107) ∗ for any U ∈ R m × m , we know that
[Equation]
Now we focus on the bound of the vector f K,b .
Note that we write f K,b in the following way
[Equation]
we therefore upper bound f K,b as
[Equation]
BK )
[Equation]
κ
[Equation]
K
[Equation]
F ) (F.13)
Combining (F.10), (F.11), (F.12) and (F.13), we know that (cid:107) ζ
[Equation]
b
[Equation]
1 + M ζ, 2 · (1 + (cid:107) K (cid:107) F ) · [1 − ρ ( A − BK )] − 1 .
Therefore, we know that ζ K,b K ∈ V ζ .
Part 2.
Now we show that for any ζ ∈ V ζ , we have ξ ( ζ ) ∈ V ξ .
Note that ξ ( ζ ) is given by
[Equation] [Equation]
Then we have
[Equation] [Equation]
and also
[Equation]
Note that we upper bound B 1 as
[Equation]
where Σ z is given in (E.15).
Also, by Proposition B.
[Equation]
we bound B 2 as
[Equation]
As for the term B 3 in (F.16)
[Equation]
we utilize the following lemma to provide an upper bound.
Lemma F.1.
The vector E π K,bK [ c ( x, u ) ψ ( x, u )] has the following form
[Equation]
Here the matrix Σ z takes the form
[Equation]
Proof.
See § F.11 for detailed proof.
From Lemma F.1 and (F.11)
[Equation]
we obtain the upper bound for B
[Equation] [Equation]
Moreover, by the deﬁnition of Σ z in (E.15)
[Equation]
combining the triangular inequality
[Equation]
we have the following bounds for the Frobenius norm and spectral norm of Σ z , respectively:
[Equation]
Also, by similar techniques we used in deriving (F.11)
[Equation]
we have
[Equation]
which gives the upper bound for Φ K as follows
[Equation]
Therefore, combining (F.16), (F.17), (F.18), (F.19)
[Equation]
(F.20) and (F.21)
[Equation]
we know that
[Equation]
By (F.15) and (F.22)
[Equation]
we know that ξ ( ζ ) ∈ V ξ for any ζ ∈ V ζ .
From this, we conclude the lemma.
F. P ROOF OF L EMMA D.
Proof.
Assume that (cid:101)
z
∼ N ( µ † , Σ † ) .
Recall that the following transition holds
[Equation]
We know that
[Equation]
1 (cid:88) i =
( L (cid:62) )
i Ψ δ L
i (cid:33) , (F.24) where Ψ δ is the covariance matrix of δ t , which takes the form
[Equation]
(cid:19) .
From (F.23), we know that µ z takes the form µ z
=
( I − L )
[Equation]
1 ν = ∞ (cid:88) j =
L j ν.
Therefore, combining (F.24), we have
[Equation]
(cid:88)
[Equation]
(F.25)
It holds that (cid:13) (cid:13)(cid:13)(cid:13)(cid:13)(cid:101) T (cid:101) T (cid:88) t =
[Equation]
µ
[Equation]
(cid:101) T (cid:101) T (cid:88) t = ∞ (cid:88)
[Equation]
=
t
[Equation]
ν (cid:13) (cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:101)
[Equation]
(cid:101) T (cid:88) t
= ρ ( L ) t
[Equation]
(cid:107) µ † (cid:107) 2 +
[Equation]
T (cid:88)
[Equation]
= ∞
[Equation]
=
t
[Equation]
( L ) i · (cid:107) ν
(cid:107)≤ (cid:2) 1 − ρ ( L ) (cid:3) −(cid:101) T · (cid:107) µ † (cid:107) 2 + (cid:2) 1 − ρ ( L ) (cid:3) −
[Equation]
where M µ is a positive absolute constant.
For the covariance, note that for any random variables
X ∼ N ( µ 1 , Σ 1 ) and Y ∼ N ( µ 2 , Σ 2 ) , we know that Z
= X + Y ∼ N ( µ 1 + µ 2 , Σ) , where (cid:107) Σ (cid:107) F ≤ 2 (cid:107) Σ 1 (cid:107) F + 2 (cid:107) Σ 2 (cid:107) F .
Combining (F.24), we know that (cid:98) µ z ∼ N ( E (cid:98) µ z , (cid:101) Σ (cid:101) T / (cid:101) T ) , where (cid:101) Σ (cid:101) T satisﬁes that
[Equation]
which implies that
[Equation]
where M Σ is a positive absolute constant.
Combining (F.25), (F.26) and (F.27), we conclude that (cid:98) µ z follows the distribution.
The inequality follows by a Gaussian tail bounds.
Then we ﬁnish the proof of the lemma.
F. P ROOF OF L EMMA D.
Proof.
We continue using the notations given in § D.3.
We deﬁne
[Equation]
where (cid:98) ψ = (cid:98) ψ ( x, u ) is the estimated feature vector.
Here the expectation is only taken over the trajectory generated by the state transition and the policy π K,b , instead of the randomness when calculating the estimated feature vectors.
Thus, the function (cid:98) F ( ζ, ξ ) is still random, where the randomness comes from the estimated feature vectors.
Note that | F ( ζ, ξ ) − (cid:101) F ( ζ, ξ ) | ≤ | F ( ζ, ξ ) − (cid:98) F ( ζ, ξ ) | + | (cid:98) F ( ζ, ξ ) − (cid:101) F ( ζ, ξ ) | .
Thus, we only need to characterize | F ( ζ, ξ ) − (cid:98) F ( ζ, ξ ) | and | (cid:98) F ( ζ, ξ ) − (cid:101)
F ( ζ, ξ ) | .
Part 1.
First we characterize | F ( ζ, ξ ) − (cid:98) F ( ζ, ξ ) | .
Note that by algebra, we have
[Equation]
F
[Equation]
28) where the expectation is only taken over the trajectory generated by the state transition and the policy π K,b .
From P ( (cid:107) (cid:98) µ z − µ z
+ 1 / (cid:101) T · µ (cid:101) T (cid:107) 2 ≤ C 1 ) ≥ 1 − (cid:101) T − 6 , we know that with probability at least 1 − (cid:101) T − 6 , the term E ( (cid:107) ψ − ψ (cid:48) (cid:107)
+ 2 (cid:107) (cid:98) ψ (cid:107) 2 ) is upper bounded by a polynomial poly ( (cid:107) Φ K (cid:107) ∗ , (cid:107) K
[Equation]
F , (cid:107) b (cid:107) 2 , (cid:107) µ (cid:107) 2 , J ( K 0 , b 0 )) .
Also, the term E ( c ) is also upper bounded by a polynomial poly ( (cid:107) Φ K (cid:107) ∗ , (cid:107) K (cid:107) F , (cid:107) b (cid:107) 2 , (cid:107) µ (cid:107) 2 , J ( K 0 , b 0 )) by the deﬁnition of the cost function c ( x, u ) .
Therefore, combining Condition B.7 and (F.28), with probability at least 1 − (cid:101) T − 6 , we obtain that
[Equation]
We upper bound the term (cid:107) ψ ( x, u ) − (cid:98) ψ ( x, u ) (cid:107) 2 for any x and u as
[Equation]
, u
[Equation]
where z = [ x (cid:62) , u (cid:62) ] (cid:62) .
Also, by Lemma D.7, we know that
[Equation]
(F.31) holds with probability at least 1 − (cid:101) T − 6 .
Combining (F.29), (F.30) and (F.31), it holds that
[Equation]
32) with probability at least 1 − (cid:101) T − 6 .
Part 2.
We now characterize | (cid:98) F ( ζ, ξ ) − (cid:101) F ( ζ, ξ ) | in the sequel.
By deﬁnitions, we have
[Equation]
(F.
[Equation]
where E (cid:48) is an event deﬁned as
[Equation]
which, by Lemma G. [Equation] satisﬁes that P ( E (cid:48) ) ≥ 1 − T − 5 − (cid:101) T − 6 .
By a similar argument as in Part 1 , we upper bound the two absolute values on the RHS of (F.33) as
[Equation]
for sufﬁciently large T and (cid:101) T .
Therefore, combining (F.32) and (F.34), by triangular inequality, we ﬁnish the proof of the lemma
[Equation]
F. P ROOF OF L EMMA E.
Proof.
Recall that the feature vector ψ ( x, u ) takes the following form
[Equation]
We then have
[Equation]
where we denote by y =
z − µ z , and x (cid:48) and u (cid:48) are the state and action following the state transition and the policy π K
[Equation]
b .
Therefore, for any symmetric matrices M
[Equation]
N and any vectors m
[Equation]
n , the following equation holds
[Equation]
where the expectations are taken over the distribution y ∼ N (0 , Σ z ) and δ ∼ N (0 , Ψ δ ) .
For the terms A 2 and A 3 in (F.35), by the fact that y =
z − µ z ∼ N (0 , Σ z ) , we know that these two terms vanish.
For A 4 , it holds that
[Equation]
For A 1 , by algebra, we have
[Equation]
Now, by applying Lemma G
[Equation]
1 to the ﬁrst term on the RHS of (F.37)
[Equation]
we know that
[Equation]
where we use (E.14) in the last equality.
By using the property of the operator svec ( · ) and the deﬁnition of the symmetric Kronecker product
[Equation]
we obtain that
[Equation]
[Equation]
Combining (F.35), (F.36) and (F.38), we obtain that
[Equation]
Thus, the matrix Θ K,b takes the following form
[Equation]
This concludes the lemma.
F. P ROOF OF L EMMA E.
Proof.
By the property of the spectral norm, we upper bound (cid:107) (cid:101)
Θ − 1 K,b (cid:107) ∗ as
[Equation]
We proceed to bound the RHS of (F.39).
For the term Θ − 1 K,b (cid:101) σ z , combining Lemma E.2 and (E.16), we have
[Equation]
where we use the property of the symmetric Kronecker product in the second and last line.
Therefore
[Equation]
we have
[Equation]
where in the third line we use Lemma G.2 to the matrix L ⊗ s L .
Similarly, we upper bound (cid:107) Θ − 1 K
[Equation]
in the sequel
[Equation]
(F.41)
Thus, combining (F.39)
[Equation]
(F.40) and (F.41)
[Equation]
we obtain that
[Equation]
Now it remains to characterize σ min (Σ z ) .
For any vectors s ∈ R m and r ∈ R k
[Equation]
we have
[Equation]
F.
[Equation]
The ﬁrst term on the RHS of (F.43) is lower bounded as
[Equation]
) where the last inequality comes from the fact that σ min (Φ K ) ≥ σ min (Ψ ω ) by (3.3).
The second term on the RHS of (F.43) takes the form E η ∼N (0 ,I ) [( σr (cid:62) η ) 2 ]
= σ 2 (cid:107) r (cid:107) 22 .
Therefore, combining (F.43) and (F.44), we have
[Equation]
From this, we see that σ min (Σ z ) is lower bounded by min { σ min (Ψ ω ) , σ 2 − (cid:107) K (cid:107) 2 ∗ · σ min (Ψ ω ) } , which only depends on σ min (Ψ ω ) and σ .
Thus, combining (F.42), we know that (cid:107) (cid:101) Θ − 1 K,b (cid:107) ∗ is upper bounded by some constant (cid:101) λ K , where (cid:101)
λ K only depends on ρ ( A − BK ) , σ and σ min (Ψ ω ) .
This ﬁnishes the proof of the lemma.
F. P ROOF OF L EMMA D.
Proof.
By direct calculation and Proposition B.2, we have
[Equation]
The following lemma helps establish the upper bound of the term (cid:107) P (cid:101) K n +1 − P K n +1 (cid:107) ∗ .
Lemma F.2.
Suppose that the parameters K and (cid:101) K satisfy that
[Equation]
then it holds that
[Equation]
Proof.
See Lemma 5.7 in Yang et al.
for a detailed proof.
To use Lemma F.2, we only need to verify that (cid:101) K n +1 and K n +1 satisfy (F.46).
Note that from (D.10) and (D.11), we have
[Equation]
Φ
[Equation]
∗ .
(F.48)
Now we proceed to upper bound the RHS of (F.48).
For the term (cid:107) A − B (cid:101) K n +1 (cid:107) ∗ , we have
[Equation]
BK
[Equation]
K
[Equation]
≤
[Equation]
1 +
[Equation]
K n
[Equation]
(F.49)
And by the deﬁnition of Υ K n in (3.7), we upper bound (cid:107) Υ K n (cid:107) ∗ as (cid:107) Υ K n
[Equation]
∗ ≤
[Equation]
Q
[Equation]
∗ +
[Equation]
∗
[Equation]
A
[Equation]
F
[Equation]
F
[Equation]
2 · (cid:107) P K n (cid:107) ∗
[Equation]
, (F
[Equation]
50) where the last line comes from the fact that (cid:107) P K n (cid:107) ∗ ≤ J 1 ( K n ) · σ − 1min (Ψ (cid:15) ) ≤ J 1 ( K 0 ) · σ − 1min (Ψ (cid:15) ) .
As for the term (cid:107) Φ (cid:101) K n +1 (cid:107) ∗ in (F.48), we upper bound it as (cid:107)
[Equation]
K n +1 (cid:107) ∗ ≤ J 1 ( (cid:101) K n +1 ) · σ − 1 min ( Q ) ≤ J 1 ( K 0 ) · σ − 1 min ( Q ) .
(F.51)
Therefore, combining (F.48), (F.49), (F.50) and (F.51), we know that
[Equation]
Also, by the above results, the RHS of (F.47) is upper bounded as
[Equation]
Note that from the policy evaluation theorem B.8, we know that with probability at least 1 − T − 4 − (cid:101) T − 6 , it holds that
[Equation]
[Table]
+
[Table]
n
[Table]
µ
[Table]
λ
[Table]
· log
[Table] [Table]
[Table]
By choosing T n and (cid:101) T n such that
[Table]
[Equation]
in other words
[Equation]
we pick
[Equation]
then we know that (F.46) holds with probability at least 1 − ε 10 for sufﬁciently small ε > 0 .
This corresponds to the parameters that we choose in the statement of Theorem B
[Equation]
By applying Lemma
F.2, combining (F.45), we derive that
[Equation]
holds with probability at least 1 − ε 15 .
By this, we ﬁnish the proof of the lemma.
F.
P ROOF OF L EMMA F.
Proof.
First, note that the cost function c ( x, u ) is written in the following way
[Equation]
uK [Equation] For any test matrix V and vectors v x , v u , we proceed by the following calculations
[Equation]
In the sequel
[Equation]
we calculate D 1 and D 2 respectively.
Calculation of D 1 .
Note that from the deﬁnition of ψ ( x, u ) in (B.5), we calculate D 1 as
[Equation]
(F.53)
Here z =
( x (cid:62) , u (cid:62) )
(cid:62) and µ z = E π K,bK ( z ) .
For the ﬁrst term on the RHS of (F.53), note that z − µ z is a centralized Gaussian, whose covariance matrix is Σ z ; therefore, by Lemma G.1, we obtain that
[Equation]
Moreover, the second term on the RHS of (F.53) is calculated as
[Equation]
Combining the above two equations and (F.53), we obtain that
[Equation]
Calculation of D 2 : By the deﬁnition of the feature vector ψ ( x, u ) in (B.5), we know that
[Equation]
Now, combining (F.52), (F.54) and (F.55), we conclude the lemma.
G A UXILIARY R ESULTS
Lemma G.1.
Assume that the random variable w ∼ N (0 , I ) , and let U and V be two symmetric matrices, then it holds that
[Equation]
Proof.
See Magnus et al.
Magnus (1979) for a detailed proof.
Lemma G.2.
Let M , N be commuting symmetric matrices, and let α 1 , . .
. , α n , β 1 , . .
, β n denote their eigenvalues with v 1 , . .
. , v n a common basis of orthogonal eigenvectors.
Then the n ( n + 1) /eigenvalues of M ⊗ s N are given by (
α i β j + α j β i ) / 2 , where 1 ≤ i ≤ j ≤ n .
Proof.
See Lemma 2 in Alizadeh et al.
(1998) for a detailed proof.
Lemma G.3.
For any integer m > 0 , let A ∈ R m × m and η ∼ N (0 , I m ) .
Then, there exists some absolute constant C > 0 such that for any t ≥ 0 , we have
[Equation]
Proof.
See Rudelson et al.
for a detailed proof.