A VERAGE - CASE A CCELERATION FOR B ILINEAR G AMES AND N ORMAL M ATRICES
Anonymous authors
Paper under double-blind review
A BSTRACT
Advances in generative modeling and adversarial learning have given rise to renewed interest in smooth games.
However, the absence of symmetry in the matrix of second derivatives poses challenges that are not present in the classical minimization framework.
While a rich theory of average-case analysis has been developed for minimization problems, little is known in the context of smooth games.
In this work we take a ﬁrst step towards closing this gap by developing average case optimal ﬁrst order methods for a subset of smooth games.
We make the following three main contributions.
First, we show that for zero-sum bilinear games the average-case optimal method is the optimal method for the minimization of the Hamiltonian.
Second, we provide an explicit expression for the optimal method corresponding to normal matrices, potentially non-symmetric.
Finally, we specialize it to matrices with eigenvalues located in a disk and show a provable speed-up compared to worst-case optimal algorithms.
We illustrate our ﬁndings through benchmarks with a varying degree of mismatch with our assumptions.
I NTRODUCTION
The traditional analysis of optimization algorithms is a worst-case analysis (Nemirovski, 1995; Nesterov, 2004).
This type of analysis provides a complexity bound for any input from a function class, no matter how unlikely.
However, since hard-to-solve inputs might rarely occur in practice, the worst-case complexity bounds might not be representative of the observed running time.
A more representative analysis is given by the average-case complexity, averaging the algorithm’s complexity over all possible inputs.
This analysis is standard for analyzing, e.g., sorting (Knuth, 1997) and cryptography algorithms (Katz & Lindell, 2014).
Recently, a line of work (Berthier et al., 2020; Pedregosa & Scieur, 2020; Lacotte & Pilanci, 2020; Paquette et al., 2020) focused on optimal methods for the optimization of quadratics, speciﬁed by a symmetric matrix.
While worst-case analysis uses bounds on the matrix eigenvalues to yield upper and lower bounds on convergence, average-case analysis relies on the expected distribution of eigenvalues and provides algorithms with sharp optimal convergence rates.
While the algorithms developed in this context have been shown to be efﬁcient for minimization problems, these have not been extended to smooth games.
A different line of work considers smooth games but studies worst-case optimal methods (Azizian et al., 2020).
In this work, we combine the two previous trends and develop novel average-case optimal algorithms for ﬁnding the root of a linear system determined by a (potentially non-symmetric) normal matrix.
We make the following main contributions :
• Inspired by the problem of ﬁnding equilibria in smooth games, we develop average-case optimal algorithms for ﬁnding the root of a non-symmetric afﬁne operator, both under a normality assumption (Theorem 4.1), and under the extra assumption that eigenvalues of the operator are supported in a disk (Theorem 4.2).
The proposed method, and its asymptotic variant, show a polynomial speedup compared to worst-case optimal method, veriﬁed by numerical simulations.
• We make a novel connection between average-case optimal methods for optimization, and average-case optimal methods for bilinear games.
In particular, we show that solving the Hamiltonian using an average-case optimal method is optimal (Theorem 3.1).
This result complements (Azizian et al., 2020), who proved that the Polyak Heavy Ball algorithm on the Hamiltonian is asymptotically worst-case optimal (section 3).
A VERAGE - CASE ANALYSIS FOR NORMAL MATRICES
In this paper we consider the following class of problems.
Deﬁnition 1.
Let A ∈ R d × d be a real matrix and
x (cid:63) ∈ R d a vector.
The non-symmetric (afﬁne) operator ( NSO ) problem is deﬁned as:
[Equation]
This problem generalizes that of minimization of a convex quadratic function f , since we can cast the latter in this framework by setting the operator F = ∇ f .
The set of solutions is an afﬁne subspace that we will denote X (cid:63) .
We will ﬁnd convenient to consider the distance to this set, deﬁned as
[Equation]
In this paper we will develop average-case optimal methods.
For this, we consider A and x (cid:63) to be random vectors, and a random initialization x 0 .
This induces a probability distribution over
NSO problems, and we seek to ﬁnd methods that have an optimal expected suboptimality w.r.t.
this distribution.
More precisely, average-case optimal methods solve the following at each iteration t :
[Equation]
The last condition on x t stems from restricting the class of algorithms to ﬁrst-order methods.
This class encompasses many known schemes such as gradient descent with momentum, or full-matrix
AdaGrad.
However, methods such as Adam (Kingma & Ba, 2015) or diagonal AdaGrad (Duchi et al., 2011) are not in this class, as the diagonal re-scaling creates iterates x t outside the span of previous gradients.
Although we will focus on the distance to the solution, the results can be extended to other convergence criteria such as (cid:107) F ( x t ) (cid:107) 2 .
Finally, note that the expectations in this paper are on the problem instance and not on the randomness of the algorithm.
2. O RTHOGONAL RESIDUAL POLYNOMIALS AND FIRST - ORDER METHODS
The analysis of ﬁrst-order methods simpliﬁes through the use of polynomials.
This section provides the tools required to leverage this connection.
Deﬁnition 2.
A residual polynomial is a polynomial P that satisﬁes P (0)
= 1 .
Proposition 2.1.
(Hestenes et al., 1952)
If the sequence ( x t ) t ∈ Z + is generated by a ﬁrst-order method, then there exist residual polynomials P t , each one of degree at most t , verifying
[Equation]
As we will see, optimal average-case method are strongly related to orthogonal polynomials.
We ﬁrst deﬁne the inner product between polynomials.
Deﬁnition 3.
For P, Q ∈ R [ X ] , we deﬁne the inner product (cid:104)· , ·(cid:105) µ for a measure µ over C as
[Equation]
Deﬁnition 4.
A sequence of polynomials { P i } is orthogonal (resp.
orthonormal ) w.r.t.
(cid:104)· , ·(cid:105) µ if (cid:104) P
i , P i (cid:105) µ > 0 (resp.
= 1 ) ; (cid:104) P
i , P j (cid:105) µ =
if i (cid:54) =
j.
2. E XPECTED S PECTRAL D ISTRIBUTION
Following (Pedregosa & Scieur, 2020), we make the following assumption on the problem family.
Assumption 1. x 0 − x (cid:63) is independent of A , and E x 0 , x (cid:63) [( x 0 − x (cid:63) )
( x 0 − x (cid:63) ) (cid:62) ]
= R 2 d I d .
We will also require the following deﬁnitions to characterize difﬁculty of a problem class.
Let { λ 1 , . . .
, λ d } be the eigenvalues of a matrix A ∈ R d × d .
We deﬁne the empirical spectral distribution of A as the probability measure
[Equation]
where δ λ i is the Dirac delta, a distribution equal to zero everywhere except at λ i and whose integral over the entire real line is equal to one.
Note that with this deﬁnition, (cid:82) D d µ A ( λ ) corresponds to the proportion of eigenvalues in D .
When A is a matrix-valued random variable, µ A is a measure-valued random variable.
As such, we can deﬁne its expected spectral distribution
[Equation]
which by the Riesz representation theorem is the measure that veriﬁes (cid:82) f d
[Equation]
E A
[Equation]
f [Equation] µ
[Equation]
] for all measureable f .
Surprisingly, the expected spectral distribution is the only required characteristic to design optimal algorithms in the average-case.
2. E XPECTED ERROR OF FIRST - ORDER METHODS
In this section we provide an expression for the expected convergence in terms of the residual polynomial and the expected spectral distribution introduced in the previous section.
To go further in the analysis, we have to assume that A is a normal matrix.
Assumption 2.
The (real) random matrix A is normal, that is, it veriﬁes AA (cid:62) =
A (cid:62) A .
Normality is equivalent to A having the spectral decomposition A
= U Λ U ∗ , where U is unitary, i.e., U ∗ U = UU ∗
=
I .
We now have everything to write the expected error of a ﬁrst-order algorithm applied to (NSO).
Theorem 2.1.
Consider the application of a ﬁrst-order method associated to the sequence of polynomials { P t } (Proposition 2.1) on the problem (NSO).
Let µ being the spectral distribution of A .
Under Assumptions 1 and 2, we have
[Equation]
Before designing optimal algorithms for certain speciﬁc distributions, we compare our setting with the average-case accelerating for minimization problems of Pedregosa & Scieur (2020), who proposed optimal optimization algorithms in the average-case.
2. D IFFICULTIES OF F IRST -O RDER M ETHODS ON G AMES AND R ELATED W ORK
This section compares our contribution with the existing framework of average-case optimal methods for quadratic minimization problems.
Averave-case optimal minimization methods.
In the minimization framework, Pedregosa & Scieur (2020) derived ﬁrst-order methods that are average-case optimal for problems of the form
[Equation]
where H ∈ R d × d is a random symmetric positive-deﬁnite matrix and x (cid:63) ∈ R d a random vector. This problem is equivalent to solving H ( x − x (cid:63) ) for x , and so can be seen as a speciﬁc instance of the non-symmetric operator problem NSO with A = H .
Our framework is signiﬁcantly more general, as it relaxes the symmetric positive deﬁnite assumption on the matrix H to normal matrices.
This allows to include important problems that do not ﬁt in the minimization framework such as bilinear games, whose underlying matrix is skew-symmetric . Difﬁculties of an extension to non-symmetric matrices. The absence of symmety in the operator matrix A creates two key difﬁculties that are not present int he minimization framework: complex eigenvalues and absence of three-term recurrence.
Complex eigenvalues. Theorem 2.1 derives an expression for the average-case error in terms of the residual polynomial and the expected spectral density.
Contrary to existing results in the minimization framework (Pedregosa & Scieur, 2020, Theorem 2.1), the absence of symmetry in the matrix A implies that eigenvalues are complex-valued and so the domain of integral is a shape in the complex plane.
This shape plays a crucial role in the rate of converge of ﬁrst-order algorithms, as depicted in the work of Azizian et al.
; Bollapragada et al.
Absence of thee-term recurrence .
(Pedregosa & Scieur, 2020) relied on the optimal residual polynomial being an orthogonal polynomial w.r.t.
a weight function deﬁned on the real line.
In such case, the associated optimal method in the average-case can be express in a simple momentum-like form. Unfortunately, this result is no longer true in our case, and we have to restore to weaker results like the following, that express the optimal residual polynomial as a linear combination of orthonormal polynomials.
Theorem 2.2.
[Theorem 1.4 of Assche (1997)] Let µ be a positive Borel measure in the complex plane.
The minimum of the integral (cid:82) C | P ( λ ) | 2 d µ ( λ ) over residual polynomials P of degree lower or equal than t is uniquely attained by the polynomial P
[Equation]
, with optimal value
[Equation]
where ( φ k ) k is the orthonormal sequence of polynomials with respect to the inner product (cid:104)· , ·(cid:105) µ .
Because of this, the optimal methods in this and the minimization framework are signiﬁcantly different, an potentially much more complicated than a simple three-term recurrence.
In the next sections we consider some cases where the optimal scheme follows a simple structure.
A VERAGE - CASE O PTIMAL M ETHODS FOR B ILINEAR G AMES
We consider the following problem of ﬁnding a Nash equilibrium of the zero-sum game
[Equation]
to which we will refer as the bilinear game , where θ 1 , θ (cid:63) 1 ∈ R d 1 , θ 2 , θ (cid:63) 2 ∈ R d 2 , M ∈ R d 1 × d 2 and d def = d 1 + d 2 .
The vector ﬁeld of the game (Balduzzi et al., 2018) is deﬁned as
[Equation]
As before, X (cid:63) denotes the set of points solutions F ( x ) = 0 , which is equivalent to the set of
Nash equilibrium.
If M is sampled independently from x 0 , x (cid:63) and x 0 − x (cid:63) has covariance R 2 d I d ,
Assumption 1 is fulﬁlled.
Since A is skew-symmetric, it is in particular normal and Assumptionis also satisﬁed.
We now show that the optimal average-case algorithm to solve bilinear problems is Hamiltonian gradient descent with momentum, described below in its general form.
Contrary to the methods in
Azizian et al.
(2020), the method we propose is anytime (and not only asymptotically) average-case optimal.
Optimal average-case algorithm for bilinear games.
Initialization.
x − 1 = x 0 = (cid:0) θ 1 , 0 , θ 2 , 0 (cid:1) , sequence { h t , m t } given by Theorem 3.1.
Main loop.
For t ≥ 0 ,
[Equation]
The quantity 12 (cid:107) F ( x ) (cid:107) 2 is commonly known as the Hamiltonian of the game (Balduzzi et al., 2018), hence the name Hamiltonian gradient descent .
Indeed, g t = ∇ (cid:0) 12 (cid:107) F ( x ) (cid:107) 2 (cid:1) when F is afﬁne:
[Equation]
The following theorem shows that (11) is indeeed the optimal average-case method associated to the minimization problem
min x (cid:0) 12 (cid:107) F ( x ) (cid:107) 2 (cid:1) , as the following theorem shows.
Theorem 3.1.
Suppose that Assumption 1 holds and that the spectral distribution of MM (cid:62) is absolutely continuous with respect to the Lebesgue measure.
Then, the method (11) is averagecase optimal for zero-sum bilinear games when h t , m t are chosen to be the coefﬁcients of the average-case optimal minimization of 12 (cid:107) F ( x ) (cid:107) 2 .
How to ﬁnd optimal coefﬁcients?
Since 12 (cid:107) F ( x ) (cid:107) 2 is a quadratic problem, the coefﬁcients { h t , m t } can be found using the average-case framework for quadratic minimization problems of (Pedregosa & Scieur, 2020, Theorem 3.1).
Proof sketch.
When computing the optimal polynomial x t =
P t ( A )( x 0 − x (cid:63) ) , we have that the residual orthogonal polynomial P t behaves differently if t is even or odd.
• Case 1: t is even.
In this case, we observe that the polynomial P t ( A ) can be expressed as Q t/ 2 ( − A 2 ) , where ( Q t ) t ≥ 0 is the sequence of orthogonal polynomials w.r.t.
the expected spectral density of − A 2 , whose eigenvalues are real and positive.
This gives the recursion in (11).
• Case 2: t is odd.
There is no residual orthogonal polynomial of degree t for t odd.
Instead, odd iterations do correspond to the intermediate computation of g t in (11), but not to an actual iterate.
Comparison with the worst-case results of Azizian et al. A consequence of Theorem 3.1 is that average-case optimal method for bilinear games can be interpreted as the minimization averagecase optimal method of (Pedregosa & Scieur, 2020) applied to the Hamiltonian. Azizian et al. (2020, Propositions 4 and 5) provide a similar result in the worst-case analysis framework.
They show that the optimal worst-case rate can be achieved with Polyak’s Heavy Ball method (optimal in the worst-case analysis) also applied to the Hamiltonian.
Hence, in both cases the optimal method consists in minimizing the Hamiltonian with an optimal (worst-case or average-case, depending on the framework) method.
Furthermore, in section 5 we show that as the number of iterations goes to inﬁnity the coefﬁcients of the average-case optimal method converge to the aforementioned worstcase optimal method.
3. P ARTICULAR CASE : M WITH I . I . D . COMPONENTS
We now show the optimal method when the entries of M are i.i.d.
sampled.
For simplicity, we order the players such that d 1 ≤ d 2 .
Assumption 3.
Assume that each component of M is sampled iid from a distribution of mean 0 and variance σ 2 , and we take d 1 , d 2 → ∞ with d 1 d 2 → r < 1 .
In such case, the spectral distribution of 1 d 2 MM (cid:62) tends to the Marchenko-Pastur law, supported in [ (cid:96), L ] and with density:
[Equation]
where L def
[Equation]
[Equation]
Proposition 3.1.
When M satisﬁes Assumption 3, the optimal parameters of scheme (11) are
[Equation]
where
[Equation]
Proof.
By Theorem 3.1, the problem reduces to ﬁnding the optimal average-case algorithm for the problem min x 12 (cid:107) F ( x ) (cid:107) 2 .
Since the expected spectral distribution of 1 d 2 MM (cid:62) is the MarchenkoPastur law, we can use the optimal algorithm from (Pedregosa & Scieur, 2020, Section 5).
G ENERAL AVERAGE - CASE OPTIMAL METHOD FOR NORMAL OPERATORS
In this section we derive general average-case optimal ﬁrst-order methods for normal operators.
First, we need to assume the existence of a three-term recurrence for residual orthogonal polynomials (Assumption 4).
As mentioned in subsection 2.4, for general measures in the complex plane, the existence of a three-term recurrence of orthogonal polynomials is not ensured.
In Proposition B.3 in
Appendix B we give a sufﬁcient condition for its existence, and in the next subsection we will show speciﬁc examples where the residual orthogonal polynomials satisfy the three-term recurrence.
Assumption 4 (Simplifying assumption) .
The sequence of residual polynomials { ψ t } t ≥ 0 orthogonal w.r.t.
the measure µ , deﬁned on the complex plane, admits the three-term recurrence
[Equation]
Under Assumption 4, Theorem 4.1 shows that the optimal algorithm can also be written as an average of iterates following a simple three-terms recurrence.
Theorem 4.1.
Under Assumptions 1, 2 and 4, the following algorithm is optimal in the average case, with y − 1 = y
= x 0 :
[Equation]
where ( φ k (0))
k ≥ 0 can be computed using the three-term recurrence (upon normalization).
Moreover, E ( A , x (cid:63) , x 0 ) dist( x t , X (cid:63) ) converges to zero at rate 1 /B t .
Remark.
Using that F is an afﬁne function, we can show that the above recurrence fulﬁlls the deﬁnition of a ﬁrst-order algorithm (2).
Remark.
Assumption 4 is needed for the sequence ( y t ) t ≥ 0 to be computable using a three-term recurrence.
However, for some distribution, the associated sequence of orthogonal polynomials may admit another recurrence that may not satisfy Assumption 4.
[Figure]
4. C IRCULAR SPECTRAL DISTRIBUTIONS
In random matrix theory, the circular law states that if A is an n × n matrix with i.i.d.
entries of mean C and variance R 2 /n , as n → ∞ the spectral distribution of A tends to the uniform distribution on D C,R , which we deﬁne to be the disk of center C and radius R > 0 in the complex plane.
In this subsection we apply Theorem 4. to a class of spectral distributions speciﬁed by Assumption 5, which includes the uniform distribution on D C,R .
Even though the random matrices with i.i.d entries are not normal, in section 6 we see that the empirical results for such matrices are consistent with our theoretical results under the normality assumption.
Assumption 5.
Assume that the spectral distribution µ A is supported in the complex plane on the disk D C,R of center C ∈ R , C > 0 and radius R < C .
Moreover, assume that the spectral density is circularly symmetric, i.e. there exists a probability measure µ R supported on [0 , R ] such for all f measurable and r ∈ [0 , R ] , d µ A ( C + re iθ ) =
12 π d θ d µ R ( r ) .
Proposition 4.
If Assumption 5, sequence polynomials is ( φ t ) t ≥
[Equation]
where K t,R = (cid:113)(cid:82) R 0 r 2 t d [Equation]
[Equation] [Equation] Example The uniform distribution in the circle D C,R satisﬁes Assumption 5.
More formally, this distribution corresponds to d µ R = 2 rR 2 d r , and K t,R = R t / √ t + 1 .
From Proposition 4.1, the sequence of residual polynomials is given by φ t ( λ ) /φ t (0) =
(cid:0) 1 − λC (cid:1) t , which implies that Assumption 4 is fulﬁlled with a t = 1 , b t = − 1 C .
Thus, by Theorem 4.1 we have
Theorem 4.2.
Given an initialization x 0 ( y
= x 0 ) , if Assumption 5 is fulﬁlled with R < C and Assumptions 1 and 2 hold, then the average-case optimal ﬁrst-order method is
[Equation]
Moreover, E ( A , x (cid:63) , x 0 ) dist( x t , X (cid:63) ) converges to zero at rate 1 /B t .
Comparison with the worst-case results of Azizian et al.
They give a worst-case convergence lower bound of ( R/C ) 2 t on the quantity dist( z t , X (cid:63) ) for ﬁrst-order methods ( z t ) t ≥ 0 on matrices with eigenvalues in the disk D C,R .
By the classical analysis of ﬁrst-order methods, this rate is achievable by gradient descent with stepsize 1 /C , i.e. the iterates y t deﬁned in (18).
However, by equation (79) in Proposition D.3 we have that under slight additional assumptions (those of Proposition 5.2), lim t →∞ E [dist( x t , X (cid:63) )] / E
[dist( y t , X (cid:63) )] = 1 − R 2 C 2 holds.
That is, the average-case optimal algorithm outperforms gradient descent by a constant factor depending on the conditioning R/C , showcasing that average-case analysis is subtler than worst-case analysis.
This effect can be observed in the last row of Figure 2.
A SYMPTOTIC BEHAVIOR
The recurrence coefﬁcients of the average-case optimal method typically converges to limiting values when t → ∞ , which gives an “average-case asymptotically optimal ﬁrst-order method” with constant coefﬁcients.
For the case of symmetric operators with spectrum in [ (cid:96), L ] , Scieur & Pedregosa (2020) show that under mild conditions, the asymptotically optimal algorithm is the Polyak momentum method with coefﬁcients depending only on (cid:96) and L .
Since the average-case optimal algorithm to ﬁnd solutions of bilinear games is the average-case optimal algorithm for the optimization of the Hamiltonian, we can make use of their framework to obtain the asymptotic algorithm (see Theorem 3 of Scieur & Pedregosa (2020)).
Proposition 5.1.
In the setting of Theorem 3.1, assume that the spectral density µ MM (cid:62) of MM (cid:62) is supported in [ (cid:96), L ] for 0 < (cid:96) <
L , and strictly positive in this interval.
Then, the asymptotically optimal algorithm for bilinear games is the following version of Polyak momentum:
[Equation]
As anticipated at the end of section 3, notice that the algorithm in (19) is the worst-case optimal algorithm from Proposition 4 of Azizian et al.
For the case of circularly symmetric spectral densities with support on disks, we can also compute the asymptotically optimal algorithm.
Proposition 5.2.
Suppose that the assumptions of Theorem 4.2 hold with µ R ∈ P ([0 , R ]) fulﬁlling µ R ([ r, R ])
=
Ω(( R − r ) κ ) for r in [ r 0 , R ] for some r 0 ∈ [0 , R ) and for some κ ∈ Z .
Then, the average-case asymptotically optimal algorithm is, with y
= x 0 :
[Equation]
Moreover, the convergence rate for this algorithm is asymptotically the same one as for the optimal algorithm in Theorem 4.2.
Namely, lim t →∞ E [dist( x t , X (cid:63) )]
B t = 1 , where B t is deﬁned in (18) .
The condition on µ R simply rules out cases in which the spectral density has exponentially small mass around 1.
It is remarkable that in algorithm (20) the averaging coefﬁcients can be expressed so simply in terms of the quantity R/C .
Notice also that while the convergence rate of the algorithm is slower than the convergence rate for the optimal algorithm by deﬁnition, both rates match in the limit, meaning that the asymptotically optimal algorithm also outperforms gradient descent by a constant factor 1 − R 2 C 2 in the limit t → ∞ .
E XPERIMENTS
We compare some of the proposed methods on settings with varying degrees of mismatch with our assumptions.
Bilinear Games.
We consider min-max bilinear problems of the form (9), where the entries of M are generated i.i.d.
from a standard Gaussian distribution.
We vary the ratio r = d/n parameter for d = 1000 and compare the average-case optimal method of Theorems 3.1 and 5.1, the asymptotic worst-case optimal method of (Azizian et al., 2020) and extragradient (Korpelevich, 1976).
In all cases, we use the convergence-rate optimal step-size assuming knowledge of the edges of the spectral distribution.
The spectral density for these problems is displayed in the ﬁrst row of Figure 2 and the benchmark results on the second row.
Average-case optimal methods always outperform other methods, and the largest gain is in the ill-conditioned regime ( r ≈ 1) .
Circular Distribution.
For our second experiment we choose A as a matrix with iid Gaussian random entries, therefore the support of the distribution of its eigenvalue is a disk.
Note that A does not satisfy the normality assumption of Assumption 2.
Figure 2 (third row) compares the average-case optimal methods from Theorems 4.2 and 5.2 on two datasets with different levels of conditioning.
Note that the methods converge despite the violation of Assumption 2, suggesting a broader applicability than the one proven in this paper.
We leave this investigation for future work.
D ISCUSSION AND F UTURE R ESEARCH D IRECTIONS
In this paper, we presented a general framework for the design of optimal algorithms in the averagecase for afﬁne operators F , whose underlying matrix is possibly non-symmetric.
Ultimately, the goal is that our ﬁndings inform the choice of training algorithms for minimax-based machine learning techniques such as GANs or adversarial examples.
The main limitation of the proposed approach is the restriction to normal matrices.
However, numerical experiments suggest this assumption is either unnecessary or can be relaxed.
A direction of future research could be to relax this normality assumption and to allow a wider class of spectral densities.
Other extensions of practical interest include the extension to problems with non-linear operators and stochastic algorithms.
In the context of stochastic algorithms, (Loizou et al., 2020) has recently derived worst-case convergence guarantees for the stochastic Hamiltonian method, although the average-case analysis is still unknown. [Figure] matrix generated with iid entries for two different degrees of conditioning. Depite the normality assumption not being satisﬁed, we still observe an improvement of averagecase optimal methods vs worst-case optimal ones.
A P ROOF OF T HEOREM 2. A. P RELIMINARIES
Before proving Theorem 2.1, we quickly analyze the distance function (1), recalled below,
[Equation]
The deﬁnition of the distance function is not practical for the theoretical analysis.
Fortunately, it is possible to ﬁnd a simple expression that uses the orthogonal projection matrix Π to the kernel
Ker ( A ) .
Since Π is an orthogonal projection matrix to the kernel of a linear transformation, it satisﬁes
[Equation]
The normality assumption on A implies also that
[Equation]
Indeed, the spectral decomposition of A is
[Equation]
and then Π =
U 2 U ∗ 2 .
The next proposition uses Π to derive the explicit solution of the (1).
Proposition A.1.
We have that
[Equation]
Proof.
We ﬁrst parametrize the set of solution X (cid:63) .
By deﬁnition we have
[Equation]
Which can be written in terms of the kernel of A as
[Equation]
From this, we can rewrite the distance function (1) as
[Equation]
The minimum can be attained at different points, but in particular at w = − ( y − x (cid:63) ) , which proves the statement.
We now simpliﬁes further the result of the previous proposition in the case where x t is generated by a ﬁrst order method.
Proposition A.2.
For every iterate x t of a ﬁrst-order methods, i.e., x t satisﬁes
[Equation]
we have that
[Equation]
Proof.
We start with the result of Proposition A.1,
[Equation]
The norm can be split into
[Equation]
Since x t is generated by a ﬁrst order method, we have
[Equation]
Since P (0) = 1 , the polynomial can be factorized as P ( A )
=
I + AQ t − 1 ( A ) , Q t − 1 being a polynomial of degree t − 1 .
Therefore, (cid:107) Π( x t − x (cid:63) )
(cid:107) 2 reads
[Equation]
which prove the statement.
A. P ROOF OF THE THEOREM
We are now ready to prove the main result.
Theorem 2.1.
Consider the application of a ﬁrst-order method associated to the sequence of polynomials { P t } (Proposition 2.1) on the problem (NSO).
Let µ being the spectral distribution of A .
Under Assumptions 1 and 2, we have
[Equation]
Proof.
We start with the result of Proposition A.2,
[Equation]
We now write the expectation of the distance function,
E
[Equation]
eigenvalues
[Equation]
However, | P (0) | 2 = 1 and tr Π corresponds to the number of zero eigenvalues of A , therefore,
E [ dist ( x t , X (cid:63) )]
= R E (cid:34)(cid:90)
C \{ 0 } | P ( λ ) |
[Equation]
δ λ
i
[Equation]
λ
[Equation]
λ
[Equation]
B P ROOFS OF T HEOREM 3.1 AND P ROPOSITION 3.
Proposition B.1.
[Block determinant formula]
If A, B, C, D are (not necessarily square) matrices,
[Equation]
if D is invertible.
Deﬁnition 5 (Pushforward of a measure) .
Recall that the pushforward f ∗ µ of a measure µ by a function f is deﬁned as the measure such that for all measurable g ,
[Equation]
Equivalently, if X is a random variable with distribution µ , then f ( X ) has distribution f ∗ µ .
Proposition B.2.
Assume that the dimensions of M ∈ R d x × d y fulﬁll d x ≤ d y and let r =
d x /d y .
Let µ MM (cid:62) be the spectral distribution of the random matrix MM (cid:62)
∈ R
d x × d x , and assume that it is absolutely continuous with respect to the Lebesgue measure.
The spectral distribution of A is contained in the imaginary line and is given by
[Equation]
for λ ∈ R .
If d x ≥ d y , then (25) holds with µ M (cid:62) M in place of µ MM (cid:62) and 1 /r in place of r .
Proof.
By the block determinant formula, we have that for s (cid:54) = 0 ,
[Equation]
[Equation]
Thus, for every eigenvalue − λ ≤ 0 of − MM (cid:62) , both i √ λ and − i √ λ are eigenvalues of A .
Since rank ( MM (cid:62) )
= rank ( M ) , we have rank ( A ) = 2 rank ( M ) .
Thus, the rest of the eigenvalues of A are 0 and there is a total of d − 2 d 1 =
d 2 − d 1 of them.
Notice that
[Equation]
Let f + ( λ )
=
i √ λ, f − ( λ ) = − i √ λ , and let ( f + ) ∗ µ MM (cid:62) (resp., ( f − ) ∗ µ MM (cid:62) ) be the pushforward measure of µ MM (cid:62) by the function f + (resp., f − ).
Thus, by the deﬁnition of the pushforward measure (Deﬁnition 5),
[Equation]
We compute the pushforwards ( f + ) ∗ µ MM (cid:62) , ( f − ) ∗ µ MM (cid:62) performing the change of variables y = ±
i √ λ under the assumption that µ MM (cid:62) ( λ [Equation] =
[Equation] MM [Equation] ( λ
[Equation]
dλ :
[Equation]
which means that the density of ( f + ) ∗ µ MM (cid:62) at y ∈
i R ≥ 0 is 2 | y |
ρ MM (cid:62) ( | y | 2 ) and the density of ( f − ) ∗ µ MM (cid:62) at y ∈ − i R ≥ 0 is also 2 | y |
ρ MM (cid:62) ( | y | 2 ) .
Proposition B.3.
The condition
[Equation]
is sufﬁcient for any sequence ( P k ) k ≥ 0 of orthogonal polynomials of increasing degrees to satisfy a three-term recurrence of the form
[Equation]
) where
[Equation]
Proof.
Since λP k − 1 ( λ ) is a polynomial of degree k , and ( P j ) 0 ≤ j ≤ k is a basis of the polynomials of degree up to k , we can write
[Equation]
Now, remark that for all j < k − 2 , (cid:104) P k − 1 , λP j (cid:105) = 0 because the inner product of P k − 1 with a polynomial of degree at most k − 2 .
If we make use of the condition (30), this implies that (cid:104) λP
k − 1 , P j (cid:105) =
for all j < k − 2 .
Plugging this into (33), we obtain (31).
Proposition B.4.
Let Π R t be the set of polynomials with real coefﬁcients and degree at most t .
For t ≥ 0 even, the minimum of the problem
[Equation]
is attained by an even polynomial with real coefﬁcients.
Proof.
Since d µ ( iλ ) def = | λ | ρ MM (cid:62) ( | λ | 2 ) d | λ | is supported in the imaginary axis and is symmetric with respect to 0 , for all polynomials P, Q ,
[Equation]
Hence, (cid:104) P ( λ ) , λQ ( λ ) (cid:105) = 0 implies (cid:104) λP
( λ ) , Q ( λ ) (cid:105) = 0 .
By Proposition B.3, a three-term recurrence (31) and (32) for the orthonormal sequence ( φ t ) t ≥ 0 of polynomials holds.
By Proposition B.5, the orthonormal polynomials ( φ t ) t ≥ 0 of even (resp. odd) degree are even (resp. odd) and have real coefﬁcients.
Hence, for all t ≥ 0 even
[Equation]
36) is an even polynomial with real coefﬁcients.
By Theorem 2.2, this polynomial attains the minimum of the problem
[Equation]
) and, a fortiori, the minimum of the problem in (34), in which the minimization is restricted polynomials with real coefﬁcients instead of complex coefﬁcients.
Proposition B.5.
The polynomials ( φ t ) t ≥ 0 of the orthonormal sequence corresponding to the measure µ ( iλ ) = | λ | ρ MM (cid:62) ( | λ | 2 ) d | λ | have real coefﬁcients and are even (resp. odd) for even (resp. odd) k .
Proof.
The proof is by induction.
The base case follows from the choice φ 0 = 1 .
Assuming that φ k − 1 ∈ R [ X ] by the induction hypothesis, we show that α k = 0 (where α k is the coefﬁcient from (31) and (32)):
[Equation]
The last equality follows from | φ k − 1 ( iλ ) | 2 = | φ k − 1 ( − iλ ) | 2 , which holds because φ k − 1 ( iλ ) ∗ =
φ k − 1 ( − iλ ) , and in turn this is true because φ k − 1 ∈ R
[ X ] by the induction hypothesis.
Once we have seen that α k = 0 , it is straightforward to apply the induction hypothesis once again to show that φ k also satisﬁes the even/odd property.
Namely, for k even (resp. odd), γ k P k = λP k − 1 − β k P k − 2 , and the two polynomials in the right-hand side have even (resp. odd) degrees.
Finally, φ k must have real coefﬁcients because φ k − 1 and φ k − 2 have real coefﬁcients by the induction hypothesis, and the recurrence coefﬁcient β k is real, as
[Equation]
Proposition B.6.
Let t ≥ 0 even.
Assume that on R > 0 , the spectral density µ MM (cid:62) has RadonNikodym derivative ρ MM (cid:62) with respect to the Lebesgue measure.
If
[Equation]
and
[Equation]
then P (cid:63)t ( λ ) =
Q (cid:63)t/ 2 ( − λ 2 ) .
Proof.
First, remark that the equalities in (40) and (41) are well deﬁned because the arg min are unique by Theorem 2.2.
Without loss of generality, assume that d x ≤ d y (otherwise switch the players), and let r def =
d x /d y < 1 .
Since,
[Equation]
each eigenvalue of MM (cid:62) ∈ R d x × d x is an eigenvalue of − A 2 with doubled duplicity, and the rest of eigenvalues are zero.
Hence, we have µ − A 2 = (cid:0) 1 − 2 /
[Equation]
1 + 1 r ) (cid:1)
[Equation]
[Equation] 2 µ MM (cid:62) / (1 + 1 r ) .
Thus, for all t ≥ 0 ,
[Equation]
By Proposition B.4, for an even t ≥ 0 the minimum in (41) is attained by an even polynomial with real coefﬁcients.
Hence,
[Equation]
Moreover, for any polynomial Q t/ 2 that attains the minimum on the right-most term, the polynomial
P t ( λ ) =
Q t/ 2 ( − λ 2 ) attains the minimum on the left-most term.
In particular, using (43), P (cid:63)t ( λ ) def =
Q (cid:63) t/ 2 ( − λ 2 ) attains the minimum on the left-most term.
Theorem 3.1.
Suppose that Assumption 1 holds and that the spectral distribution of MM (cid:62) is absolutely continuous with respect to the Lebesgue measure.
Then, the method (11) is average-case optimal for zero-sum bilinear games when h t , m t are chosen to be the coefﬁcients of the averagecase optimal minimization of 12 (cid:107) F ( x ) (cid:107) 2 .
Proof.
Making use of Theorem 2.1 and Proposition B.2, we obtain that for any ﬁrst-order method using the vector ﬁeld F ,
[Equation]
Let Q (cid:63)t/ 2 , P (cid:63)t be as deﬁned in (41) and (40).
For t ≥ 0 even the iteration t of the average-case optimal method for the bilinear game must satisfy
[Equation]
[Equation] On the other hand, the ﬁrst-order methods for the minimization of the function 1 2 (cid:107) F ( x ) (cid:107) 2 make use of the vector ﬁeld ∇ (cid:0) 12 (cid:107) F ( x ) (cid:107) 2 (cid:1) =
A (cid:62) (
Ax + b )
= − A 2 ( x − x (cid:63) ) .
Let µ − A 2 be the spectral density of − A 2 .
By Theorem 2.1, the average-case optimal ﬁrst-order method for the minimization problem is the one for which the residual polynomial P t (Proposition 2.1) minimizes the functional (cid:82)
R P 2 t d µ − A 2 .
That is, the residual polynomial is Q (cid:63)t .
From (46), we see that the t -th iterate of the average-case optimal method for F is equal to the t/ 2 -th iterator of the average-case optimal method for ∇ (cid:0) 12 (cid:107) F ( x ) (cid:107) 2 (cid:1) .
C P ROOFS OF T HEOREM 4.1 AND T HEOREM 4.
Theorem 4.1.
Under Assumptions 1, 2 and 4, the following algorithm is optimal in the average case, with y − 1 = y
= x 0 :
[Equation]
[Equation]
where ( φ k (0))
k ≥ 0 can be computed using the three-term recurrence (upon normalization).
Moreover, E ( A , x (cid:63) , x 0 ) dist( x t , X (cid:63) ) converges to zero at rate 1 /B t .
Proof.
We prove by induction that
[Equation]
The base step t = 0 holds trivially because φ 0 = 1 .
Assume that (47) holds for t − 1 .
Subtracting x (cid:63) from (16), we have
[Equation] [Equation]
If
[Equation]
49) by the induction hypothesis for t − 1 and (48), we have
[Equation]
which concludes the proof of (47).
The only thing left is to show (49), again by induction.
The base case follows readily from y
= x 0 in (16).
Dividing by φ t (0) 2 , we rewrite (49) as
[Equation]
where ψ t is the t -th orthogonal residual polynomial of sequence.
By Assumption 4, ψ t must satisfy the recurrence in (15).
If we subtract x ∗ from the second line of (16), we apply the induction hypothesis and then the recurrence in (15), we obtain
[Equation]
thus concluding the proof of (49)
[Equation]
Proposition C.1.
Suppose that Assumption 5 holds with C = 0 , that is, the circular support of µ is centered at 0 .
Then, the basis of orthonormal polynomials for the scalar product
[Equation]
where K k,R = (cid:113) [Equation] (cid:82) [Equation]
Proof.
First, we will show that if µ satisﬁes Assumption 5 with C = 0 , then (cid:104) λ i , λ j (cid:105) =
if j, k ≥with j (cid:54) =
k (without loss of generality, suppose that j > k ).
[Equation]
And for all k ≥ 0 ,
[Equation]
Proposition 4.1.
If µ satisﬁes Assumption 5, the sequence of orthonormal polynomials is ( φ t ) t ≥ 0 ,
[Equation]
where K
[Equation]
,R =
[Equation]
Proof.
The result follows from Proposition C.1 using the change of variables z → z + C .
To compute the measure µ R for the uniform measure on D C,R , we perform a change of variables to circular coordinates:
[Equation]
And
[Equation]
Theorem 4.2.
Given an initialization x 0 ( y
= x 0 ) , if Assumption 5 is fulﬁlled with R < C and Assumptions 1 and 2 hold, then the average-case optimal ﬁrst-order method is
[Equation]
Moreover, E ( A , x (cid:63) , x 0 ) dist( x t , X (cid:63) ) converges to zero at rate 1 /B t .
Proof.
By Proposition 4.1, the sequence of residual orthogonal polynomials is given by ψ t ( λ ) =
φ t ( λ ) /φ t (0) =
(cid:0) 1 − λC (cid:1) t .
Hence, Assumption 4 is fulﬁlled with a t = 1 , b t = − 1 C , as ψ t ( λ ) =
ψ t − 1 ( λ ) − λC ψ t − 1 ( λ ) .
We apply Theorem 4.1 and make use of the fact that φ k (0) 2 = C 2 k K 2 t,R .
See
Proposition D.3 for the rate on dist( x t , X (cid:63) ) .
D P ROOF OF P ROPOSITION 5.
Proposition D.1.
Suppose that the assumptions of Theorem 4.2 hold with the probability measure µ R fulﬁlling µ R ([ r, R ])
=
Ω(( R − r ) κ ) for r in [ r 0 , R ] for some r 0 ∈ [0 , R ) and for some κ ∈ Z .
Then,
[Equation]
Proof.
Given (cid:15) > 0 , let c (cid:15)
∈
Z ≥ 0 be the minimum such that
[Equation]
Deﬁne Q t,R def =
R 2 t K 2 t,R .
Then,
[Equation]
Now, on one hand [Equation] using that Q t
[Equation]
R is an increasing sequence on t , Q t,
[Equation]
On the other hand, for t ≥ c (cid:15) ,
[Equation]
Thus, we want to upper-bound (cid:82) t k dds Q s,R d s .
First, notice that
[Equation]
[Equation]
By concavity of the logarithm function we obtain log( Rr ) ≤
Rr 0 − 1 for r ∈
[ r 0 , R ] .
Choose r 0 close enough to R so that
Rr 0 − 1 ≤
[Equation]
/c (cid:15) .
We obtain that
[Equation]
Thus,
[Equation]
d
[Equation]
Using that log x ≤ x
[Equation]
for k ∈ [ t − c (cid:15) , t ] we can bound the ﬁrst term of (65) as (cid:90) t
[Equation]
In the last inequality we use that by Proposition D.2, for t large enough, Q t,R = R 2 t K 2 t,R ≤ (2 t + 1) k /c 1 .
For k ∈ [ t − c (cid:15) , t ] , the second term of (65) can be bounded as
[Equation]
t
[Equation]
From (65), (66) and (67), we obtain that for t large enough, for k ∈ [ t − c (cid:15) , t ] , (cid:90) t
[Equation]
Hence, we can bound the right-hand side of (62):
Q t,R
[Equation]
The last inequality follows from the deﬁnition of c (cid:15) in (59).
Since (cid:15) is arbitrary, by the sandwich theorem applied on (60), (61) and (69),
[Equation]
Proposition D.2.
Under the assumptions of Theorem 4.2, we have that there exists c 1 > 0 such that for t large enough,
[Equation]
Proof.
By the assumption on µ R , there exist r 0 , c 1 , κ > 0 such that
[Equation]
) where the beta function B ( x, y ) is deﬁned as
[Equation]
Using the link between the beta function and the gamma function B ( x, y )
=
Γ( x )Γ( y ) /
Γ( x + y ) , and Stirling’s approximation, we obtain that for ﬁxed y and large x ,
B ( x, y ) ∼ Γ( y ) x − y .
Hence, for t large enough, B (2 t + 1 , κ ) ∼ Γ( κ )(2 t + 1) − κ = ( κ − 1)!(2 t + 1) − κ .
Hence, from (72) we obtain that there exist c (cid:48)
1 depending only on κ and r 0 such that for t large enough
[Equation]
Proposition 5.2.
Suppose that the assumptions of Theorem 4.2 hold with µ R ∈ P ([0 , R ]) fulﬁlling µ R ([ r, R ])
=
Ω(( R − r ) κ ) for r in [ r 0 , R ] for some r 0 ∈ [0 , R ) and for some κ ∈ Z .
Then, the average-case asymptotically optimal algorithm is, with y
= x 0 :
[Equation]
Moreover, the convergence rate for this algorithm is asymptotically the same one as for the optimal algorithm in Theorem 4.2.
Namely, lim t →∞ E [dist( x t , X (cid:63) )]
B t = 1 , where B t is deﬁned in (18) .
Proof.
The proof follows directly from Theorem 4.2 and Proposition D.1.
See (77) and (79) in
Proposition D.3 for the statement regarding the convergence rate.
Proposition D.3.
For the average-case optimal algorithm (18) ,
[Equation]
For the average-case asymptotically optimal algorithm (20) ,
[Equation]
C
[Equation]
C (cid:19)
[Equation]
For the iterates y t in (18) , i.e. gradient descent with stepsize 1 /C , we have
[Equation]
Moreover, for all t ≥ 0 , we have ξ opt ( t ) ≤ ξ asymp ( t ) , and under the assumptions of (5.1) ,
[Equation]
Proof.
To show (76), (77), (78), we use the expression x t − x (cid:63) =
P t ( A )( x 0 − x (cid:63) )
Proposition 2.1) and then evaluate (cid:107) P t (cid:107) 2 µ = (cid:82) C \{ 0 } | P t | 2 d µ (Theorem 2.1).
For (76), the value of (cid:107) P t (cid:107) 2 µ follows directly from Theorem 2.2, which states that the value for the optimal residual polynomial P t is
[Equation]
A simple proof by induction shows that for the asymptotically optimal algorithm (20), the following expression holds for all t ≥ 0 :
[Equation]
Thus,
[Equation]
which concludes the proof of (77), as
[Equation]
By equation (52),
[Equation]
Thus, for the y t iterates, (cid:107) P t (cid:107) 2 µ = K 2 t,R C 2 t , and (78) follows.
Now, ξ opt ( t ) ≤ ξ asymp ( t ) , ∀ t ≥ 0 is a consequence of ξ opt ( t ) being the rate of the optimal algorithm.
And
[Equation]
) follows from Proposition D.1.
To show lim t →∞ ξ opt ( t ) ξ GD ( t ) = 1 − R 2 C 2 , which concludes the proof, we rewrite ξ asymp
[Equation]
using that by deﬁnition, Q k,R = R 2 k /K 2 k,R .
Now, let c (cid:15)
∈
Z ≥ 0 such that ∞ (cid:88)
[Equation]
Using the same argument as in Proposition D.1 (see (68)), for t large enough and k ∈ [ t − c (cid:15) , t ] , (cid:90) t
[Equation]
Hence, for t large enough,
[Equation]
(cid:18)
[Equation]
C
[Equation]
( t [Equation] k ) +
t
[Equation]
c (cid:15) (cid:88) k
=
Q k,R (cid:18) R
C (cid:19) 2( t − k ) (cid:33) + (cid:18) R
C (cid:19) 2 t ≤
[Equation]
[Equation]
C
[Equation]
[Equation]
t
[Equation]
k
[Equation]
c (cid:15) (cid:18) [Equation] ( t [Equation]
k )
+
t
[Equation]
c (cid:15)
[Equation]
k =
[Equation]
≤ [Equation] 1 −
[Equation]
[Equation]
(cid:15) )
[Equation]
(cid:32)
[Equation]
(cid:18) R
C (cid:19)
[Equation]
(cid:33) (cid:15)
[Equation]
89) which can be made arbitrarily close to (cid:16) 1 − (cid:0) RC (cid:1) 2 (cid:17) 1 Q t,R by taking (cid:15) > 0 small enough. Plugging this into (86), we obtain that we can make ξ asymp ( t ) arbitrarily close to (cid:16) 1 − (cid:0) RC (cid:1) 2 (cid:17) (cid:0) RC (cid:1) 2 t 1 Q t,R = (cid:16) 1 − (cid:0) RC (cid:1) 2 (cid:17) ξ GD ( t ) by taking t large enough.