A VERAGE - CASE A CCELERATION FOR B ILINEAR G AMES AND N ORMAL M ATRICES
A BSTRACT
Advances in generative modeling and adversarial learning have given rise to renewed interest in smooth games.
However, the absence of symmetry in the matrix of second derivatives poses challenges that are not present in the classical minimization framework.
While a rich theory of average-case analysis has been developed for minimization problems, little is known in the context of smooth games.
In this work we take a ﬁrst step towards closing this gap by developing average-case optimal ﬁrst-order methods for a subset of smooth games.
We make the following three main contributions.
First, we show that for zero-sum bilinear games the average-case optimal method is the optimal method for the minimization of the Hamiltonian.
Second, we provide an explicit expression for the optimal method corresponding to normal matrices, potentially non-symmetric.
Finally, we specialize it to matrices with eigenvalues located in a disk and show a provable speed-up compared to worst-case optimal algorithms.
We illustrate our ﬁndings through numerical simulations with a varying degree of mismatch with our assumptions.
I NTRODUCTION
The traditional analysis of optimization algorithms is a worst-case analysis (Nemirovski, 1995; Nesterov, 2004).
This type of analysis provides a complexity bound for any input from a function class, no matter how unlikely.
However, since hard-to-solve inputs might rarely occur in practice, the worst-case complexity bounds might not be representative of the observed running time.
A more representative analysis is given by the average-case complexity, averaging the algorithm’s complexity over all possible inputs.
This analysis is standard for analyzing, e g., sorting (Knuth, 1997) and cryptography algorithms (Katz & Lindell, 2014).
Recently, a line of work (Berthier et al., 2020; Pedregosa & Scieur, 2020; Lacotte & Pilanci, 2020; Paquette et al., 2020) focused on optimal methods for the optimization of quadratics, speciﬁed by a symmetric matrix.
While worst-case analysis uses bounds on the matrix eigenvalues to yield upper and lower bounds on convergence, average-case analysis relies on the expected distribution of eigenvalues and provides algorithms with sharp optimal convergence rates.
While the algorithms developed in this context have been shown to be efﬁcient for minimization problems, these have not been extended to smooth games.
A different line of work considers algorithms for smooth games but studies worst-case optimal methods (Azizian et al., 2020).
In this work, we combine average-case analysis with smooth games, and develop novel average-case optimal algorithms for ﬁnding the root of a linear system determined by a (potentially non-symmetric) normal matrix.
We make the following main contributions :
Inspired by the problem of ﬁnding equilibria in smooth games, we develop average-case optimal algorithms for ﬁnding the root of a non-symmetric afﬁne operator, both under a normality assumption (Thm. 4.1), and under the extra assumption that eigenvalues of the operator are supported in a disk (Thm. 4.2).
The proposed method shows a polynomial speedup compared to the worst-case optimal method, veriﬁed by numerical simulations.
We make a novel connection between average-case optimal methods for optimization, and average-case optimal methods for bilinear games.
In particular, we show that solving the Hamiltonian using an average-case optimal method is optimal (Theorem 3.1) for bilinear games.
This result complements (Azizian et al., 2020), who proved that Polyak Heavy Ball algorithm on the
Hamiltonian is asymptotically worst-case optimal for bilinear games.
A VERAGE - CASE ANALYSIS FOR NORMAL MATRICES
In this paper we consider the following class of problems.
Deﬁnition 1.
Let A ∈ R d × d be a real matrix and
x (cid:63) ∈ R d a vector.
The non-symmetric (afﬁne) operator ( NSO ) problem is deﬁned as:
[Equation]
This problem generalizes that of minimization of a convex quadratic function f , since we can cast the latter in this framework by setting the operator F = ∇ f .
The set of solutions is an afﬁne subspace that we will denote X (cid:63) .
We will ﬁnd convenient to consider the distance to this set, deﬁned as
[Equation] [Equation]
In this paper we will develop average-case optimal methods.
For this, we consider A and x (cid:63) to be random vectors, and a random initialization x 0 .
This induces a probability distribution over
NSO problems, and we seek to ﬁnd methods that have an optimal expected suboptimality w.r.t.
this distribution.
Denoting E
( A , x (cid:63) , x 0 ) the expectation over these random problems, we have that average-case optimal methods they verify the following property at each iteration t
[Equation]
The last condition on x t stems from restricting the class of algorithms to ﬁrst-order methods.
The class of ﬁrst-order methods encompasses many known schemes such as gradient descent with momentum, or full-matrix AdaGrad.
However, methods such as Adam (Kingma & Ba, 2015) or diagonal AdaGrad (Duchi et al., 2011) are not in this class, as the diagonal re-scaling creates iterates x t outside the span of previous gradients.
Although we will focus on the distance to the solution, the results can be extended to other convergence criteria such as (cid:107) F ( x t ) (cid:107) 2 .
Finally, note that the expectations in this paper are on the problem instance and not on the randomness of the algorithm.
2. O RTHOGONAL RESIDUAL POLYNOMIALS AND FIRST - ORDER METHODS
The analysis of ﬁrst-order methods simpliﬁes through the use of polynomials.
This section provides the tools required to leverage this connection.
Deﬁnition 2.
A residual polynomial is a polynomial P that satisﬁes P (0)
= 1 .
Proposition 2.1.
(Hestenes et al., 1952)
If the sequence ( x t ) t ∈ Z + is generated by a ﬁrst-order method, then there exist residual polynomials P t , each one of degree at most t , verifying
[Equation]
As we will see, optimal average-case method are strongly related to orthogonal polynomials.
We ﬁrst deﬁne the inner product between polynomials, where we use z ∗ for the complex conjugate of z ∈ C .
Deﬁnition 3.
For P, Q ∈ R [ X ] , we deﬁne the inner product (cid:104)· , ·(cid:105) µ for a measure µ over C as
[Equation]
Deﬁnition 4.
A sequence of polynomials { P i } is orthogonal (resp.
orthonormal ) w.r.t.
(cid:104)· , ·(cid:105) µ if (cid:104) P
i , P i (cid:105) µ > 0 (resp.
= 1 ) ; (cid:104) P
i , P j (cid:105) µ =
if i (cid:54) =
j.
2. E XPECTED S PECTRAL D ISTRIBUTION
Following (Pedregosa & Scieur, 2020), we make the following assumption on the problem family.
Assumption 1. x 0 − x (cid:63) is independent of A , and E ( x 0 , x (cid:63) )
[( x 0 − x (cid:63) )
( x 0 − x (cid:63) ) (cid:62) ]
= R 2 d I d .
We will also require the following deﬁnitions to characterize difﬁculty of a problem class.
Let { λ 1 , . . .
, λ d } be the eigenvalues of a matrix A ∈ R d × d .
We deﬁne the empirical spectral distribution of A as the probability measure
[Equation]
(5) where δ λ i is the Dirac delta, a distribution equal to zero everywhere except at λ i and whose integral over the entire real line is equal to one.
Note that with this deﬁnition, (cid:82) D dˆ µ A ( λ ) corresponds to the proportion of eigenvalues in D .
When A is a matrix-valued random variable, µ A is a measure-valued random variable.
As such, we can deﬁne its expected spectral distribution
[Equation]
which by the Riesz representation theorem is the measure that veriﬁes (cid:82)
f
d µ = E A [ (cid:82) f d µ A ] for all measureable f .
Surprisingly, the expected spectral distribution is the only required characteristic to design optimal algorithms in the average-case.
2. E XPECTED ERROR OF FIRST - ORDER METHODS
In this section we provide an expression for the expected convergence in terms of the residual polynomial and the expected spectral distribution introduced in the previous section.
To go further in the analysis, we have to assume that A is a normal matrix.
Assumption 2.
The (real) random matrix A is normal, that is, it veriﬁes AA (cid:62) =
A (cid:62) A .
Normality is equivalent to A having the spectral decomposition A
= U Λ U ∗ , where U is unitary, i.e., U ∗ U = UU ∗
=
I .
We now have everything to write the expected error of a ﬁrst-order algorithm applied to (NSO).
Theorem 2.1.
Consider the application of a ﬁrst-order method associated to the sequence of polynomials { P t } (Proposition 2.1) on the problem (NSO).
Let µ be the expected spectral distribution of A .
Under Assumptions 1 and 2, we have
[Equation]
Before designing optimal algorithms for certain speciﬁc distributions, we compare our setting with the average-case accelerating for minimization problems of Pedregosa & Scieur (2020), who proposed optimal optimization algorithms in the average-case.
2. D IFFICULTIES OF F IRST -O RDER M ETHODS ON G AMES AND R ELATED W ORK
This section compares our contribution with the existing framework of average-case optimal methods for quadratic minimization problems.
Deﬁnition 5.
Let H ∈ R
d × d be a random symmetric positive-deﬁnite matrix and x (cid:63) ∈ R d a random vector.
These elements determine the following random quadratic minimization problem
[Equation]
As in our paper, Pedregosa & Scieur (2020) ﬁnd deterministic optimal ﬁrst-order algorithms in expectation w.r.t.
the matrix H , the solution x (cid:63) , and the initialization x 0 .
Since they work with problem (OPT), their problem is equivalent to (NSO) with the matrix
A = H .
However, they have the stronger assumption that the matrix is symmetric , which implies being normal.
The normality assumption is restrictive in the case of game theory, as they do not always naturally ﬁt such applications.
However, this set is expressive enough to consider interesting cases, such as bilinear games, and our experiments show that our ﬁndings are also consistent with non-normal matrices.
Using orthogonal residual polynomials and spectral distributions, they derive the explicit formula of the expected error.
Their result is similar to Theorem 2.1, but the major difference is the domain of the integral, a real positive line in convex optimization, but a shape in the complex plane in our case.
This shape plays a crucial role in the rate of converge of ﬁrst-order algorithms, as depicted in the work of Azizian et al.
; Bollapragada et al.
In the case of optimization methods, they show that optimal schemes in the average-case follow a simple three-term recurrence arising from the three-term recurrence for residual orthogonal polynomials for the measure λµ ( λ ) .
Indeed, by Theorem 2.1 the optimal method corresponds to the residual polynomials minimizing (cid:104) P, P (cid:105) µ , and the following result holds:
Theorem 2.2.
(Fischer, 1996, § 2.4) When µ is supported in the real line, the residual polynomial of degree t minimizing (cid:104) P, P (cid:105) µ is given by the degree t residual orthogonal polynomial w.r.t.
λµ ( λ ) .
However, the analogous result does not hold for general measures in C , and hence our arguments will make use of the following Theorem 2.3 instead, which links the residual polynomial of degree at most t that minimizes (cid:104) P, P (cid:105) µ to the sequence of orthonormal polynomials for µ .
Theorem 2.3.
[Theorem 1.4 of Assche (1997)] Let µ be a positive Borel measure in the complex plane.
The minimum of the integral (cid:82) C | P ( λ ) | 2 d µ ( λ ) over residual polynomials P of degree lower or equal than t is uniquely attained by the polynomial
[Equation]
with optimal value (cid:90)
[Equation]
where ( φ k ) k is the orthonormal sequence of polynomials with respect to the inner product (cid:104)· , ·(cid:105) µ .
In the next sections we consider cases where the optimal scheme is identiﬁable.
A VERAGE - CASE O PTIMAL M ETHODS FOR B ILINEAR G AMES
We consider the problem of ﬁnding a Nash equilibrium of the zero-sum minimax game given by
[Equation]
Let θ 1 , θ (cid:63) 1 ∈ R d 1 , θ 2 , θ (cid:63) 2 ∈ R d 2 , M ∈ R d 1 × d 2 and d def = d
1 + d 2 .
The vector ﬁeld of the game (Balduzzi et al., 2018) is deﬁned as F ( x )
=
A ( x − x (cid:63) ) , where
[Equation]
As before, X (cid:63) denotes the set of points x such that F ( x ) = 0 , which is equivalent to the set of
Nash equilibrium.
If M is sampled independently from x 0 , x (cid:63) and x 0 − x (cid:63) has covariance R 2 d I d ,
Assumption 1 is fulﬁlled.
Since A is skew-symmetric, it is in particular normal and Assumptionis also satisﬁed.
We now show that the optimal average-case algorithm to solve bilinear problems is Hamiltonian gradient descent with momentum, described below in its general form.
Contrary to the methods in
Azizian et al.
(2020), the method we propose is anytime (and not only asymptotically) average-case optimal.
Optimal average-case algorithm for bilinear games.
Initialization.
x − 1 = x 0 = (cid:0) θ 1 , 0 , θ 2 , 0 (cid:1) , sequence { h t , m t } given by Theorem 3.1.
Main loop.
For t ≥ 0 ,
[Equation]
The quantity 12 (cid:107) F ( x ) (cid:107) 2 is commonly known as the Hamiltonian of the game (Balduzzi et al., 2018), hence the name Hamiltonian gradient descent .
Indeed, g t = ∇ (cid:0) 12 (cid:107) F ( x ) (cid:107) 2 (cid:1) when F is afﬁne:
[Equation]
The following theorem shows that (11) is indeeed the optimal average-case method associated to the minimization problem
min x (cid:0) 12 (cid:107) F ( x ) (cid:107) 2 (cid:1) , as the following theorem shows.
Theorem 3.1.
Suppose that Assumption 1 holds and that the expected spectral distribution of
MM (cid:62) is absolutely continuous with respect to the Lebesgue measure.
Then, the method (11) is average-case optimal for bilinear games when h t , m t are chosen to be the coefﬁcients of the average-case optimal minimization of 12 (cid:107) F ( x ) (cid:107) 2 .
How to ﬁnd optimal coefﬁcients?
Since 12 (cid:107) F ( x ) (cid:107) 2 is a quadratic problem, the coefﬁcients { h t , m t } can be found using the average-case framework for quadratic minimization problems of (Pedregosa & Scieur, 2020, Theorem 3.1).
Proof sketch.
When computing the optimal polynomial x t = P t ( A )( x 0 − x (cid:63) ) , we have that the residual orthogonal polynomial P t behaves differently if t is even or odd.
• Case 1: t is even.
In this case, we observe that the polynomial P t ( A ) can be expressed as Q t/ 2 ( − A 2 ) , where ( Q t ) t ≥ 0 is the sequence of orthogonal polynomials w.r.t.
the expected spectral density of − A 2 , whose eigenvalues are real and positive.
This gives the recursion in (11).
• Case 2: t is odd.
There is no residual orthogonal polynomial of degree t for t odd.
Instead, odd iterations do correspond to the intermediate computation of g t in (11), but not to an actual iterate.
3. P ARTICULAR CASE : M WITH I . I . D . COMPONENTS
We now show the optimal method when the entries of M are i.i.d.
sampled.
For simplicity, we order the players such that d 1 ≤ d 2 .
Assumption 3.
Assume that each component of M is sampled iid from a distribution of mean 0 and variance σ 2 , and we take d 1 , d 2 → ∞ with d 1 d 2 → r < 1 .
In such case, the spectral distribution of 1 d 2 MM (cid:62) tends to the Marchenko-Pastur law, supported in [ (cid:96), L ] and with density:
[Equation]
where L
[Equation]
Proposition 3.1.
When M satisﬁes Assumption 3, the optimal parameter of scheme (11) are
[Equation]
where ρ = 1+ r √ r , δ t = ( −
[Equation]
− δ t − 1 ) − 1 , δ 0 =
[Equation]
Proof.
By Theorem 3.1, the problem reduces to ﬁnding the optimal average-case algorithm for the problem min x 12 (cid:107) F ( x ) (cid:107) 2 .
Since the expected spectral distribution of 1 d 2 MM (cid:62) is the MarchenkoPastur law, we can use the optimal algorithm from (Pedregosa & Scieur, 2020, Section 5).
G ENERAL AVERAGE - CASE OPTIMAL METHOD FOR NORMAL OPERATORS
In this section we derive general average-case optimal ﬁrst-order methods for normal operators.
First, we need to assume the existence of a three-term recurrence for residual orthogonal polynomials (Assumption 4).
As mentioned in subsection 2.4, for general measures in the complex plane, the existence of a three-term recurrence of orthogonal polynomials is not ensured.
In Proposition B.3 in
Appendix B we give a sufﬁcient condition for its existence, and in the next subsection we will show speciﬁc examples where the residual orthogonal polynomials satisfy the three-term recurrence.
Assumption 4 (Simplifying assumption) .
The sequence of residual polynomials { ψ t } t ≥ 0 orthogonal w.r.t.
the measure µ , deﬁned on the complex plane, admits the three-term recurrence
[Equation] [Equation] Under Assumption 4, Theorem 4.1 shows that the optimal algorithm can also be written as an average of iterates following a simple three-terms recurrence. Theorem 4.1.
{+Under Assumption 4 and the assumptions of Theorem 2.1, the following algorithm is optimal in the average case, with y − 1 = y 
+}
= x 0 : y t = a t y t − 1 + (1 − a t ) y t − 2 + b t F ( y t − 1 ) x t =
B t
B t + β t x t − 1 + β t
B t + β t y t , β t = φ 2 t (0) ,
B t =
B t − 1 + β t − 1 ,
B 0 = 0 .
(16) where ( φ k (0))
k ≥ 0 can be computed using the three-term recurrence (upon normalization).
Moreover, E ( A , x (cid:63) , x 0 ) dist( x t , X (cid:63) ) converges to zero at rate 1 /B t .
Remark.
Notice that it is not immediate that (16) fulﬁlls the deﬁnition of ﬁrst-order algorithms stated in (2), as y t is clearly a ﬁrst-order method but x t is an average of the iterates y t .
Using that
F is an afﬁne function we see that x t indeed fulﬁlls (2).
Remark. Assumption 4 is needed for the sequence ( y t ) t ≥ 0 to be computable using a three-term recurrence. However, for some distribution, the associated sequence of orthogonal polynomials may admit another recurrence that may not satisfy Assumption 4. 4. C IRCULAR SPECTRAL DISTRIBUTIONS
In random matrix theory, the circular law states that if A is an n × n matrix with i.i.d. entries of mean
C and variance R 2 /n , as n → ∞ the spectral distribution of A tends to the uniform distribution on D C,R .
In this subsection we apply Theorem 4.1 to a class of spectral distributions speciﬁed by
Assumption 5, which includes the uniform distribution on D C,R . Even though the random matrices with i.i.d entries are not normal, we will see in section 6 that the empirical results for such matrices are consistent with our theoretical results under the normality assumption. Assumption 5. Assume that the expected spectral distribution µ A is supported in the complex plane on the disk D C,R of center C ∈ R , C > 0 and radius R < C .
Moreover, assume that the spectral density is circularly symmetric, i.e. there exists a probability measure µ R supported on [0 , R ] such for all f measurable and r ∈ [0 , R ] , d µ A ( C + re iθ ) =
12 π d θ d µ R ( r ) . Proposition 4.1. If µ satisﬁes Assumption 5, the sequence of orthonormal polynomials is ( φ t ) t ≥ 0 , [Equation]
, where
[Equation] Example.
The uniform distribution in D C,R is to d µ R = 2 rR 2 d r , and K t,R = R t / √ t + 1 . From Proposition 4.1, the sequence of residual polynomials is given by φ t ( λ ) /φ t (0) = (cid:0) 1 − λC (cid:1) t , which implies that Assumption 4 is fulﬁlled with a t = 1 , b t = − 1 C . Thus, by Theorem 4.1 we have
Theorem 4.2.
{+Given an initialization x 0 ( y 
+}
= x 0 ) , if Assumption 5 is fulﬁlled with R < C and the assumptions of Theorem 2.1 hold, then the average-case optimal ﬁrst-order method is
[Equation]
Moreover, E ( A , x (cid:63) , x 0 ) dist( x t , X (cid:63) ) converges to zero at rate 1 /B t .
We now compare Theorem 4.2 with worst-case methods studied in Azizian et al.
They give a worst-case convergence lower bound of ( R/C ) 2 t on the quantity dist( z t , X (cid:63) ) for ﬁrst-order methods ( z t ) t ≥ 0 on matrices with eigenvalues in the disk D C,R .
By the classical analysis of ﬁrst-order methods, this rate is achievable by gradient descent with stepsize 1 /C , i.e. the iterates y t deﬁned in (18).
However, by equation (79) in Proposition D.3 we have that under slight additional assumptions (those of Proposition 5.2), lim t →∞ E [dist( x t , X
[Equation]
[Equation]
[dist
[Equation]
y
[Equation]
[Equation]
C 2 holds.
That is, the average-case optimal algorithm outperforms gradient descent by a constant factor depending on the conditioning R/C .
A SYMPTOTIC BEHAVIOR
The recurrence coefﬁcients of the average-case optimal method typically converges to limiting values when t → ∞ , which gives an ”average-case asymptotically optimal ﬁrst-order method” with constant coefﬁcients.
For the case of symmetric operators with spectrum in [ (cid:96), L ] , Scieur & Pedregosa (2020) show that under mild conditions, the asymptotically optimal algorithm is the Polyak momentum method with coefﬁcients depending only on (cid:96) and L .
For bilinear games, since the average-case optimal algorithm is the average-case optimal algorithm of an optimization algorithm, we can make use of their framework to obtain the asymptotic algorithm (see Theorem 3 of Scieur &
Pedregosa (2020)).
Proposition 5.1. Assume that the expected spectral density µ MM (cid:62) of MM (cid:62) is supported in [ (cid:96), L ] for 0 < (cid:96) < L , and strictly positive in this interval. Then, the asymptotically optimal algorithm for bilinear games is the following version of Polyak momentum:
[Equation]
Notice that the algorithm in (19) is the worst-case optimal algorithm from Proposition 4 of Azizian et al.
For the case of circularly symmetric spectral densities with support on disks, we can also compute the asymptotically optimal algorithm.
Proposition 5.2.
Suppose that the assumptions of Theorem 4.2 hold with µ R ∈ P ([0 , R ]) fulﬁlling µ R ([ r, R ])
=
Ω(( R − r ) κ ) for r in [ r 0 , R ] for some r 0 ∈ [0 , R ) and for some κ ∈ Z .
{+Then, the average-case asymptotically optimal algorithm is, with y 
+}
= x 0 :
[Equation]
Moreover, the convergence rate for this algorithm is asymptotically the same one as for the optimal algorithm in Theorem 4.2.
Namely, lim t →∞ E [dist( x t , X (cid:63) )]
B t = 1 .
The condition on µ R simply rules out cases in which the spectral density has exponentially small mass around 1.
It is remarkable that in algorithm (20) the averaging coefﬁcients can be expressed so simply in terms of the quantity R/C .
Notice also that while the convergence rate of the algorithm is slower than the convergence rate for the optimal algorithm by deﬁnition, both rates match in the limit, meaning that the asymptotically optimal algorithm also outperforms gradient descent by a constant factor 1 − R 2 C 2 in the limit t → ∞ .
E XPERIMENTS
We compare some of the proposed methods on settings with varying degrees of mismatch with our assumptions.
Bilinear Games.
We consider min-max bilinear problems of the form (10), where the entries of
M are generated i.i.d.
from a standard Gaussian distribution.
We vary the ratio r = d/n parameter for d = 1000 and compare the average-case optimal method of Theorems 3.1 and 5.1, the asymptotic worst-case optimal method of (Azizian et al., 2020) and extragradient (Korpelevich, 1976).
In all cases, we use the convergence-rate optimal step-size assuming knowledge of the edges of the spectral distribution.
The spectral density for these problems is displayed in the ﬁrst row of Figure 1 and the benchmark results on the second row.
Average-case optimal methods always outperform other methods, and the largest gain is in the ill-conditioned regime ( r ≈ 1) .
Circular Distribution.
For our second experiment we choose A as a matrix with iid Gaussian random entries, therefore the support of the distribution of its eigenvalue is a disk.
Note that A does not satisfy the normality assumption of Assumption 2.
Figure 1 (third row) compares the average-case optimal methods from Theorems 4.2 and 5.2 on two datasets with different levels of conditioning.
Note that the methods converge despite the violation of Assumption 2, suggesting a broader applicability than the one proven in this paper.
We leave this investigation for future work.
D ISCUSSION AND F UTURE R ESEARCH D IRECTIONS
In this paper, we presented a general framework for the design of optimal algorithms in the averagecase for afﬁne operators F , whose underlying matrix is possibly non-symmetric.
However, our approach presents some limitations, the major one being the restriction to normal matrices.
Fortunately, the numerical experiments above suggests that this assumption can be relaxed.
Developing a theory without that assumption is left for future work.
Another avenue for future work is to analyze the nonlinear-case in which the non-symmetric operator A is non-linear, as well as the case in which it is accessed through a stochastic estimator (as done by (Loizou et al., 2020) for the worst-case analysis).
A CKNOWLEDGEMENTS
C. Domingo-Enrich has been partially funded by “la Caixa” Foundation (ID 100010434), under agreement LCF/BQ/AA18/11680094, and partially funded by the NYU Computer Science Department.
[Figure]
3) and eigenvalue distribution of a design matrix generated with iid entries for two different degrees of conditioning.
Depite the normality assumption not being satisﬁed, we still observe an improvement of average-case optimal methods vs worst-case optimal ones.
A P ROOF OF T HEOREM 2. A. P RELIMINARIES
Before proving Theorem 2.1, we quickly analyze the distance function (1), recalled below,
[Equation]
The deﬁnition of the distance function is not practical for the theoretical analysis.
Fortunately, it is possible to ﬁnd a simple expression that uses the orthogonal projection matrix Π to the kernel
Ker ( A ) .
Since Π is an orthogonal projection matrix to the kernel of a linear transformation, it satisﬁes
[Equation]
The normality assumption on A implies also that
[Equation]
Indeed, the spectral decomposition of A is
[Equation]
and then Π =
U 2 U ∗ 2 .
The next proposition uses Π to derive the explicit solution of the (1).
Proposition A.1.
We have that
[Equation]
Proof.
We ﬁrst parametrize the set of solution X (cid:63) .
By deﬁnition we have
[Equation]
Which can be written in terms of the kernel of A as
[Equation]
From this, we can rewrite the distance function (1) as
[Equation]
The minimum can be attained at different points, but in particular at w = − ( y − x (cid:63) ) , which proves the statement.
We now simpliﬁes further the result of the previous proposition in the case where x t is generated by a ﬁrst order method.
Proposition A.2.
For every iterate x t of a ﬁrst-order methods, i.e., x t satisﬁes
[Equation]
we have that
[Equation]
Proof.
We start with the result of Proposition A.1,
[Equation]
The norm can be split into
[Equation]
Since x t is generated by a ﬁrst order method, we have
[Equation]
Since P (0) = 1 , the polynomial can be factorized as P ( A )
=
I + AQ t − 1 ( A ) , Q t − 1 being a polynomial of degree t − 1 .
Therefore, (cid:107) Π( x t − x (cid:63) )
(cid:107) 2 reads
[Equation]
which prove the statement.
A. P ROOF OF THE THEOREM
We are now ready to prove the main result.
Theorem 2.1.
Consider the application of a ﬁrst-order method associated to the sequence of polynomials { P t } (Proposition 2.1) on the problem (NSO).
Let µ be the expected spectral distribution of
A .
Under Assumptions 1 and 2, we have
[Equation]
Proof.
We start with the result of Proposition A.2,
[Equation]
We now write the expectation of the distance function,
E
[Equation]
eigenvalues
[Equation]
However, | P (0) | 2 = 1 and tr Π corresponds to the number of zero eigenvalues of A , therefore, E [ dist ( x t , X (cid:63) )] = R E (cid:34)(cid:90) C \{ 0 } | P ( λ ) |
[Equation] δ λ i [Equation]
λ
[Equation]
λ [Equation] B P ROOFS OF T HEOREM 3.1 AND P ROPOSITION 3.  Proposition B.1.
[Block determinant formula]
If A, B, C, D are (not necessarily square) matrices,
[Equation]
if D is invertible.
Deﬁnition 6 (Pushforward of a measure) .
Recall that the pushforward f ∗ µ of a measure µ by a function f is deﬁned as the measure such that for all measurable g ,
[Equation]
Equivalently, if X is a random variable with distribution µ , then f ( X ) has distribution f ∗ µ .
Proposition B.2.
Assume that the dimensions of M ∈ R d x × d y fulﬁll d x ≤ d y and let r =
d x /d y .
Let µ MM (cid:62) be the expected spectral distribution of the random matrix MM (cid:62)
∈ R
d x × d x , and assume that it is absolutely continuous with respect to the Lebesgue measure.
The expected spectral distribution of A is contained in the imaginary line and is given by
[Equation]
for λ ∈ R .
If d x ≥ d y , then (25) holds with µ M (cid:62) M in place of µ MM (cid:62) and 1 /r in place of r .
Proof.
By the block determinant formula, we have that for s (cid:54) = 0 ,
[Equation]
[Equation]
Thus, for every eigenvalue − λ ≤ 0 of − MM (cid:62) , both i √ λ and − i √ λ are eigenvalues of A .
Since rank ( MM (cid:62) )
= rank ( M ) , we have rank ( A ) = 2 rank ( M ) .
Thus, the rest of the eigenvalues of A are 0 and there is a total of d − 2 d 1 =
d 2 − d 1 of them.
Notice that
[Equation]
Let f + ( λ )
=
i √ λ, f − ( λ ) = − i √ λ , and let ( f + ) ∗ µ MM (cid:62) (resp., ( f − ) ∗ µ MM (cid:62) ) be the pushforward measure of µ MM (cid:62) by the function f + (resp., f − ).
Thus, by the deﬁnition of the pushforward measure (Deﬁnition 6),
[Equation]
We compute the pushforwards ( f + ) ∗ µ MM (cid:62) , ( f − ) ∗ µ MM (cid:62) performing the change of variables y = ±
i √ λ under the assumption that µ MM (cid:62) ( λ
[Equation]
=
[Equation]
[Equation]
[Equation]
which means that the density of ( f + ) ∗ µ MM (cid:62) at y ∈
i R ≥ 0 is 2 | y |
ρ MM (cid:62) ( | y | 2 ) and the density of ( f − ) ∗ µ MM (cid:62) at y ∈ − i R ≥ 0 is also 2 | y |
ρ MM (cid:62) ( | y | 2 ) .
Proposition B.3.
The condition
[Equation]
is sufﬁcient for any sequence ( P k ) k ≥ 0 of orthogonal polynomials of increasing degrees to satisfy a three-term recurrence of the form
[Equation]
31) where
[Equation]
Proof.
Since λP k − 1 ( λ ) is a polynomial of degree k , and ( P j ) 0 ≤ j ≤ k is a basis of the polynomials of degree up to k , we can write
[Equation]
Now, remark that for all j < k − 2 , (cid:104) P k − 1 , λP j (cid:105) = 0 because the inner product of P k − 1 with a polynomial of degree at most k − 2 .
If we make use of the condition (30), this implies that (cid:104) λP
{+k − 1 , P j (cid:105) = 
+}
for all j < k − 2 .
Plugging this into (33), we obtain (31).
Proposition B.4.
Let Π R t be the set of polynomials with real coefﬁcients and degree at most t .
For t ≥ 0 even, the minimum of the problem
[Equation]
is attained by an even polynomial with real coefﬁcients.
Proof.
Since d µ ( iλ ) def = | λ | ρ MM (cid:62) ( | λ | 2 ) d | λ | is supported in the imaginary axis and is symmetric with respect to 0 , for all polynomials P, Q ,
[Equation]
Hence, (cid:104) P ( λ ) , λQ ( λ ) (cid:105) = 0 implies (cid:104) λP
( λ ) , Q ( λ ) (cid:105) = 0 .
By Proposition B.3, a three-term recurrence (31) and (32) for the orthonormal sequence ( φ t ) t ≥ 0 of polynomials holds.
By Proposition B.5, the orthonormal polynomials ( φ t ) t ≥ 0 of even (resp. odd) degree are even (resp. odd) and have real coefﬁcients.
Hence, for all t ≥ 0 even
[Equation]
36) is an even polynomial with real coefﬁcients.
By Theorem 2.3, this polynomial attains the minimum of the problem
[Equation]
) and, a fortiori, the minimum of the problem in (34), in which the minimization is restricted polynomials with real coefﬁcients instead of complex coefﬁcients.
Proposition B.5.
The polynomials ( φ t ) t ≥ 0 of the orthonormal sequence corresponding to the measure µ ( iλ ) = | λ | ρ MM (cid:62) ( | λ | 2 ) d | λ | have real coefﬁcients and are even (resp. odd) for even (resp. odd) k .
Proof.
The proof is by induction.
The base case follows from the choice φ 0 = 1 .
Assuming that φ k − 1 ∈ R [ X ] by the induction hypothesis, we show that α k = 0 (where α k is the coefﬁcient from (31) and (32)):
[Equation]
The last equality follows from | φ k − 1 ( iλ ) | 2 = | φ k − 1 ( − iλ ) | 2 , which holds because φ k − 1 ( iλ ) ∗ =
φ k − 1 ( − iλ ) , and in turn this is true because φ k − 1 ∈ R
[ X ] by the induction hypothesis.
Once we have seen that α k = 0 , it is straightforward to apply the induction hypothesis once again to show that φ k also satisﬁes the even/odd property.
Namely, for k even (resp. odd), γ k P k = λP k − 1 − β k P k − 2 , and the two polynomials in the right-hand side have even (resp. odd) degrees.
Finally, φ k must have real coefﬁcients because φ k − 1 and φ k − 2 have real coefﬁcients by the induction hypothesis, and the recurrence coefﬁcient β k is real, as
[Equation]
Proposition B.6.
Let t ≥ 0 even.
Assume that on R > 0 , the expected spectral density µ MM (cid:62) has
Radon-Nikodym derivative ρ MM (cid:62) with respect to the Lebesgue measure.
If
[Equation]
and
[Equation]
then P (cid:63)t ( λ ) =
Q (cid:63)t/ 2 ( − λ 2 ) .
Proof.
First, remark that the equalities in (40) and (41) are well deﬁned because the arg min are unique by Theorem 2.3.
Without loss of generality, assume that d x ≤ d y (otherwise switch the players), and let r def =
d x /d y < 1 .
Since,
[Equation]
each eigenvalue of MM (cid:62) ∈ R d x × d x is an eigenvalue of − A 2 with doubled duplicity, and the rest of eigenvalues are zero.
Hence, we have µ − A 2 = (cid:0) 1 − 2 /
[Equation]
1 + 1 r ) (cid:1)
[Equation]
[Equation]
2 µ MM (cid:62) / (1 + 1 r ) .
Thus, for all t ≥ 0 ,
[Equation]
By Proposition B.4, for an even t ≥ 0 the minimum in (41) is attained by an even polynomial with real coefﬁcients. Hence, [Equation] Moreover, for any polynomial Q t/ 2 that attains the minimum on the right-most term, the polynomial
P t ( λ ) = Q t/ 2 ( − λ 2 ) attains the minimum on the left-most term. In particular, using (43), P (cid:63)t ( λ ) def =
Q (cid:63) t/ 2 ( − λ 2 ) attains the minimum on the left-most term. Theorem 3.1. Suppose that Assumption 1 holds and that the expected spectral distribution of MM (cid:62) is absolutely continuous with respect to the Lebesgue measure.
Then, the method (11) is average-case optimal for bilinear games when h t , m t are chosen to be the coefﬁcients of the average-case optimal minimization of 12 (cid:107) F ( x ) (cid:107) 2 .
Proof.
Making use of Theorem 2.1 and Proposition B.2, we obtain that for any ﬁrst-order method using the vector ﬁeld F , [Equation] Let Q (cid:63)t/ 2 , P (cid:63)t be as deﬁned in (41) and (40).
For t ≥ 0 even the iteration t of the average-case optimal method for the bilinear game must satisfy
[Equation] On the other hand, the ﬁrst-order methods for the minimization of the function 1 2 (cid:107) F ( x ) (cid:107) 2 make use of the vector ﬁeld ∇ (cid:0) 12 (cid:107) F ( x ) (cid:107) 2 (cid:1) =
A (cid:62) (
Ax + b ) = − A 2 ( x − x (cid:63) ) .
Let µ − A 2 be the spectral density of − A 2 .
By Theorem 2.1, the average-case optimal ﬁrst-order method for the minimization problem is the one for which the residual polynomial P t (Proposition 2.1) minimizes the functional (cid:82)
R P 2 t d µ − A 2 .
That is, the residual polynomial is Q (cid:63)t .
From (46), we see that the t -th iterate of the average-case optimal method for F is equal to the t/ 2 -th iterator of the average-case optimal method for ∇ (cid:0) 12 (cid:107) F ( x ) (cid:107) 2 (cid:1) . C P ROOFS OF T HEOREM 4.1 AND T HEOREM 4.
Theorem 4.1.
Under Assumption 4 and the assumptions of Theorem 2.1, the following algorithm is optimal in the average case, with y − 1 = y
= x 0 :
[Equation]
[Equation] where ( φ k (0))
k ≥ 0 can be computed using the three-term recurrence (upon normalization).
Moreover, E ( A , x (cid:63) , x 0 ) dist( x t , X (cid:63) ) converges to zero at rate 1 /B t . Proof. We prove by induction that
[Equation]
The base step t = 0 holds trivially because φ 0 = 1 . Assume that (47) holds for t − 1 . Subtracting x (cid:63) from (16), we have
[Equation] If
[Equation] 49) by the induction hypothesis for t − 1 and (48), we have
[Equation]
which concludes the proof of (47). The only thing left is to show (49), again by induction. The base case follows readily from y {+
 = x+} 0 in (16).
Dividing by φ t (0) 2 , we rewrite (49) as
[Equation]
where ψ t is the t -th orthogonal residual polynomial of sequence. By Assumption 4, ψ t must satisfy the recurrence in (15).
If we subtract x ∗ from the second line of (16), we apply the induction hypothesis and then the recurrence in (15), we obtain [Equation]
thus concluding the proof of (49).
Proposition C.1. Suppose that Assumption 5 holds with C = 0 , that is, the circular support of µ is centered at 0 .
Then, the basis of orthonormal polynomials for the scalar product
[Equation]
where [Equation] Proof.
First, we will show that if µ satisﬁes Assumption 5 with C = 0 , then (cid:104) λ i , {+λ j (cid:105) = 
 if j, k ≥with j (cid:54)+} = k (without loss of generality, suppose that j > k ). [Equation]
[Equation] And for all k ≥ 0 ,
[Equation] Proposition 4.1.
If µ satisﬁes Assumption 5, the sequence of orthonormal polynomials is ( φ t ) t ≥ 0 , [Equation] K
[Equation]
, where K t,R = (cid:113)(cid:82) R
[Equation] [Equation]
d
[Equation] Proof.
The result follows from Proposition C.1 using the change of variables z → z + C . To compute the measure µ R for the uniform measure on D C,R , we perform a change of variables to circular coordinates:
[Equation]
And
[Equation] Theorem 4.2.
Given an initialization x 0 ( y
= x 0 ) , if Assumption 5 is fulﬁlled with R < C and the assumptions of Theorem 2.1 hold, then the average-case optimal ﬁrst-order method is
[Equation]
Moreover, E ( A , x (cid:63) , x 0 ) dist( x t , X (cid:63) ) converges to zero at rate 1 /B t . Proof. By Proposition 4.1, the sequence of residual orthogonal polynomials is given by ψ t ( λ ) = φ t ( λ ) /φ t (0) =
(cid:0) 1 − λC (cid:1) t .
Hence, Assumption 4 is fulﬁlled with a t = 1 , b t = − 1 C , as ψ t ( λ ) = ψ t − 1 ( λ ) − λC ψ t − 1 ( λ ) . We apply Theorem 4.1 and make use of the fact that φ k (0) 2 = C 2 k K 2 t,R . See
Proposition D.3 for the rate on dist( x t , X (cid:63) ) . D P ROOF OF P ROPOSITION 5.  Proposition D.1. Suppose that the assumptions of Theorem 4.2 hold with the probability measure µ R fulﬁlling µ R ([ r, R ])
=
Ω(( R − r ) κ ) for r in [ r 0 , R ] for some r 0 ∈ [0 , R ) and for some κ ∈ Z .
Then, [Equation] Proof. Given (cid:15) > 0 , let c (cid:15)
∈
Z ≥ 0 be the minimum such that [Equation] Deﬁne Q t,R def = R 2 t K 2 t,R .
Then,
[Equation]
Now, on one hand, using that Q t,R is an increasing sequence on t ,
Q t,R
[Equation]
On the other hand, for t ≥ c
[Equation] [Equation]
Thus, we want to upper-bound (cid:82) t k dds Q s,R d s .
First, notice that
[Equation] [Equation]
By concavity of the logarithm function we obtain log( Rr ) ≤
Rr 0 − 1 for r ∈ [ r 0 , R ] . Choose r 0 close enough to R so that Rr 0 − 1 ≤
[Equation]
/c
[Equation]
We obtain that
[Equation] Thus, [Equation]
d [Equation] Using that log x ≤ x [Equation]
for k ∈ [ t − c (cid:15) , t ] we can bound the ﬁrst term of (65) as (cid:90) t
[Equation]
In the last inequality we use that by Proposition D.2, for t large enough, Q t,R = R 2 t K 2 t,R ≤ (2 t + 1) k /c 1 .
For k ∈ [ t − c (cid:15) , t ] , the second term of (65) can be bounded as
[Equation]
t [Equation] From (65), (66) and (67), we obtain that for t large enough, for k ∈ [ t − c (cid:15) , t ] , (cid:90) t [Equation]
d [Equation] Hence, we can bound the right-hand side of (62): Q t,R
[Equation]
The last inequality follows from the deﬁnition of c (cid:15) in (59).
Since (cid:15) is arbitrary, by the sandwich theorem applied on (60), (61) and (69),
[Equation] Proposition D.2.
Under the assumptions of Theorem 4.2, we have that there exists c 1 > 0 such that for t large enough, [Equation]
Proof.
By the assumption on µ R , there exist r 0 , c 1 , κ > 0 such that [Equation] ) where the beta function B ( x, y ) is deﬁned as
[Equation]
Using the link between the beta function and the gamma function B ( x, y ) = Γ( x )Γ( y ) / Γ( x + y ) , and Stirling’s approximation, we obtain that for ﬁxed y and large x ,
B ( x, y ) ∼ Γ( y ) x − y . Hence, for t large enough, B (2 t + 1 , κ ) ∼ Γ( κ )(2 t + 1) − κ = ( κ − 1)!(2 t + 1) − κ . Hence, from (72) we obtain that there exist c (cid:48) 1 depending only on κ and r 0 such that for t large enough
[Equation] Proposition 5.2.
Suppose that the assumptions of Theorem 4.2 hold with µ R ∈ P ([0 , R ]) fulﬁlling µ R ([ r, R ])
=
Ω(( R − r ) κ ) for r in [ r 0 , R ] for some r 0 ∈ [0 , R ) and for some κ ∈ Z .
Then, the average-case asymptotically optimal algorithm is, with y
= x 0 :
[Equation]
Moreover, the convergence rate for this algorithm is asymptotically the same one as for the optimal algorithm in Theorem 4.2.
Namely, lim t →∞ E [dist( x t , X (cid:63) )]
B t = 1 . Proof.
The proof follows directly from Theorem 4.2 and Proposition D.1.
See (77) and (79) in Proposition D.3 for the statement regarding the convergence rate.
Proposition D.3. For the average-case optimal algorithm (18) , [Equation] For the average-case asymptotically optimal algorithm (20) ,
E [Equation] C [Equation]
+ (cid:18)
[Equation] C (cid:19) [Equation] For the iterates y t in (18) , i.e. gradient descent with stepsize 1 /C , we have
[Equation]
Moreover, for all t ≥ 0 , we have ξ opt ( t ) ≤ ξ asymp ( t ) , and under the assumptions of (5.1) , [Equation]
Proof.
To show (76), (77), (78), we use the expression x t − x (cid:63) = P t ( A )( x 0 − x (cid:63) ) Proposition 2.1) and then evaluate (cid:107) P t (cid:107) 2 µ = (cid:82) C \{ 0 } | P t | 2 d µ (Theorem 2.1). For (76), the value of (cid:107) P t (cid:107) 2 µ follows directly from Theorem 2.3, which states that the value for the optimal residual polynomial P t is
[Equation]
A simple proof by induction shows that for the asymptotically optimal algorithm (20), the following expression holds for all t ≥ 0 :
[Equation] Thus, [Equation]
which concludes the proof of (77), as
[Equation]
By equation (52), [Equation] Thus, for the y t iterates, (cid:107) P t (cid:107) 2 µ = K 2 t,R C 2 t , and (78) follows. Now, ξ opt ( t ) ≤ ξ asymp ( t ) , ∀ t ≥ 0 is a consequence of ξ opt ( t ) being the rate of the optimal algorithm.
And [Equation] follows from Proposition D.1. To show lim t →∞ ξ opt ( t ) ξ GD ( t
[Equation] = 1 − R 2 C 2 , which concludes the proof, we rewrite [Equation]
86) using that by deﬁnition, Q k,R = R 2 k /K 2 k,R . Now, let c (cid:15)
∈ Z ≥ 0 such that ∞ (cid:88) [Equation]
(cid:15) [Equation] (cid:15)
[Equation]
Using the same argument as in Proposition D.1 (see (68)), for t large enough and k ∈ [ t − c (cid:15) , t ] , (cid:90) t [Equation] Hence, for t large enough,
[Equation] k [Equation]
[Equation] [Equation] (cid:33)
[Equation]
89) which can be made arbitrarily close to (cid:16) 1 − (cid:0) RC (cid:1) 2 (cid:17) 1 Q t,R by taking (cid:15) > 0 small enough. Plugging this into (86), we obtain that we can make ξ asymp ( t ) arbitrarily close to (cid:16) 1 − (cid:0) RC (cid:1) 2 (cid:17) (cid:0) RC (cid:1) 2 t 1 Q t,R = (cid:16) 1 − (cid:0) RC (cid:1) 2 (cid:17) ξ GD ( t ) by taking t large enough.