E IGEN G AME U NLOADED W HEN PLAYING GAMES IS BETTER THAN OPTIMIZING
A BSTRACT
We build on the recently proposed EigenGame that views eigendecomposition as a competitive game.
EigenGame’s updates are biased if computed using minibatches of data, which hinders convergence and more sophisticated parallelism in the stochastic setting.
In this work, we propose an unbiased stochastic update that is asymptotically equivalent to EigenGame, enjoys greater parallelism allowing computation on datasets of larger sample sizes and outperforms EigenGame in experiments.
We present applications to ﬁnding the principal components of massive datasets and performing spectral clustering of graphs.
We analyze and discuss our proposed update in the context of EigenGame and the shift in perspective from optimization to games
I NTRODUCTION
Large, high-dimensional datasets containing billions of samples are commonplace.
Dimensionality reduction to extract the most informative features is an important step in the data processing pipeline which enables faster learning of classiﬁers and regressors (Dhillon et al. , 2013), clustering (Kannan and Vempala, 2009), and interpretable visualizations.
Many dimensionality reduction and clustering techniques rely on eigendecomposition at their core including principal component analysis (Jolliffe, 2002), locally linear embedding (Roweis and Saul, 2000), multidimensional scaling (Mead, 1992),
Isomap (Tenenbaum et al. , 2000), and graph spectral clustering (Von Luxburg, 2007).
Numerical solutions to the eigenvalue problem have been approached from a variety of angles for centuries: Jacobi’s method, Rayleigh quotient, power (von Mises) iteration (Golub and Van der
Vorst, 2000).
For large datasets that do not ﬁt in memory, approaches that access only subsets—or minibatches —of the data at a time have been proposed.
Recently, EigenGame (Gemp et al. , 2021) was introduced with the novel perspective of viewing the set of eigenvectors as the Nash strategy of a suitably deﬁned game.
While this work demonstrated an algorithm that was empirically competitive given access to only subsets of the data, its performance degraded with smaller minibatch sizes, which are required to ﬁt high dimensional data onto devices.
One path towards circumventing EigenGame’s need for large minibatch sizes is parallelization.
In a data parallel approach, updates are computed in parallel on partitions of the data and then combined such that the aggregate update is equivalent to a single large-batch update.
The technical obstacle preventing such an approach for EigenGame lies in the bias of its updates, i.e., the divide-and-conquer
EigenGame update is not equivalent to the large-batch update.
Biased updates are not just a theoretical nuisance; they can slow and even prevent convergence to the solution (made obvious in Figure 4).
In this work we introduce a formulation of EigenGame which admits unbiased updates which we term µ -EigenGame.
We will refer to the original formulation of EigenGame as α -EigenGame.µ
-EigenGame
and α -EigenGame are contrasted in Figure 1.
Unbiased updates allow us to increase the effective batch size using data parallelism.
Lower variance updates mean that µ -EigenGame should converge faster and to more accurate solutions than α
-EigenGame regardless of batch size.
In Figure 1a (top), the density of the shaded region shows the distribution of steps taken by the
[Figure]
stochastic variant of each algorithm after 100 burn-in steps.
Although the expected path of α -EG is slightly more direct, its stochastic variant has much larger variance.
Figure 1a (bottom) shows that with increasing iterations, the µ -EG trajectory approaches its expected value whereas α -EG exhibits larger bias.
Figure 1b further supports µ -EigenGame’s reduced bias with details in Sections 3 and 4.
Our contributions : In the rest of the paper, we present our new formulation of EigenGame, analyze its bias and propose a novel unbiased parallel variant, µ -EigenGame with stochastic convergence guarantees.
µ -EigenGame’s utilities are distinct from α -EigenGame and offer an alternative perspective.
We demonstrate its performance with extensive experiments including dimensionality reduction of massive data sets and clustering a large social network graph.
We conclude with discussions of the algorithm’s design and context within optimization, game theory, and neuroscience.
P RELIMINARIES AND RELATED WORK
In this work, we aim to compute the top- k right singular vectors of data X , which is either represented as a matrix, X ∈ R n × d , of n d -dimensional samples, or as a d -dimensional random variable.
In either case, we assume we can repeatedly sample a minibatch X t from the data of size n (cid:48) < n , X t ∈ R n (cid:48)
× d .
The top- k right singular vectors of the dataset are then given by the top- k eigenvectors of the (sample) covariance matrix, C = E
[ 1 n (cid:48) X (cid:62) t X t ] =
E [ C t ] .
For small datasets, SVD is appropriate.
However, the time, O (min { nd 2 , n 2 d } ) , and space, O ( nd ) , complexity of SVD prohibit its use for larger datasets (Shamir, 2015) including when X is a random variable.
For larger datasets, stochastic, randomized, or sketching algorithms are better suited.
Stochastic algorithms such as Oja’s algorithm (Oja, 1982; Allen-Zhu and Li, 2017) perform power iteration (Rutishauser, 1971) to iteratively improve an approximation, maintaining orthogonality of the eigenvectors typically through repeated QR decompositions.
Alternatively, randomized algorithms (Halko et al. , 2011; Sarlos, 2006; Cohen et al. , 2017)
ﬁrst compute a random projection of the data onto a ( k + p ) -subspace approximately containing the top- k subspace.
This is done using techniques similar to Krylov subspace iteration methods (Musco and Musco, 2015).
After projecting, a call to SVD is then made on this reduced-dimensionality data matrix.
Sketching algorithms (Feldman et al. , 2020) such as Frequent Directions (Ghashami et al. , 2016) also target learning the top- k subspace by maintaining an overcomplete sketch matrix of size ( k + p ) ×
d and maintaining a span of the top subspace with repeated calls to SVD.
In both the randomized and sketching approaches, a ﬁnal SVD of the n × ( k + p ) dataset is required to recover the desired singular vectors.
Although the
SVD scales linearly in n , some datasets are too large to ﬁt in memory; in this case, an out-of-memory
SVD may sufﬁce (Haidar et al. , 2017).
For this reason, the direct approach of stochastic algorithms, which avoid an SVD call altogether, is appealing when processing very large datasets.
1: Given: data stream X t ∈ R n (cid:48) × d , vectors ˆ v 0 i ∈
S d − 1 , step sequence η t , and iterations T .
2: ˆ v i ← ˆ v
i for all i 3: for t = 1 : T do 4: parfor i = 1 : k do 5: rewards ←
[Table]
6: penalties ←
[Table] [Table]
: ˜ ∇ µi ← rewards − penalties 8: ˜ ∇ µ,Ri ← ˜ ∇ µi − (cid:104)
˜ ∇ µi , ˆ v i (cid:105) ˆ v i 9: ˆ v (cid:48)
i ← ˆ v i + η t ˜ ∇ µ,Ri 10: ˆ v
i ← ˆ v (cid:48)
i ||
ˆ v (cid:48)
i || 11: end parfor 12: end for 13: return all ˆ v
i
A large literature on distributed approaches to
PCA exists (Liang et al. , 2014;
Garber et al. , 2017; Fan et al. , 2019).
These typically follow the pattern of computing solutions locally and then aggregating them in a single round (or minimal rounds) of communication.
The modern distributed machine learning setting which has evolved to meet the needs of deep learning is fundamentally different.
Many accelerators joined with fast interconnects means the cost of communication is low compared to the cost of a single update step, however existing approaches to distributed PCA cannot take full advantage of this.
Notation : We follow the same notation as Gemp et al.
Variables returned by an approximation algorithm are distinguished from the true solutions with hats, e.g., the column-wise matrix of eigenvectors ˆ V approximates V .
We order the columns of V such that the i th column, v i , is the eigenvector with the i th largest eigenvalue λ i .
The set of all eigenvectors { v j } with λ j larger than λ i , namely v i ’s parents , will be denoted by v j<i .
Similarly, sums over subsets of indices may be abbreviated as (cid:80) j<i
= (cid:80) i − 1 j =1 .
The set of all parents and children of v i are denoted by v − i .
Let the i th
eigengap g i
=
λ
i − λ
i +1 .
We assume the standard Euclidean inner product (cid:104) u, v (cid:105) =
u (cid:62) v and denote the unit-sphere and simplex in ambient space R d
S d and ∆ d
1 respectively.
α -EigenGame.
We build on the algorithm introduced by Gemp et al.
(2021), which we refer to here as α -EigenGame.
This algorithm is derived by formulating the eigendecomposition of a symmetric positive deﬁnite matrix as the Nash equilibrium of a game among k players, each player i owning the approximate eigenvector ˆ v i ∈ S
d − 1 .
Each player is also assigned a utility function, u αi (ˆ v i | ˆ v j<i ) , that they must maximize:
[Equation]
These utilities balance two terms, one that rewards a ˆ v i that captures more variance in the data and a second term that penalizes ˆ v i for failing to be orthogonal to each of its parents ˆ v j<i (these terms are indicated with Var and Align -penalty in equation (1)).
In α -EigenGame, each player simultaneously updates ˆ v i with gradient ascent, and it is shown that this process converges to the
Nash equilibrium.
We are interested in extending this approach to the data parallel setting where each player i may distribute its update computation over multiple devices.
A SCALABLE UNBIASED ALGORITHM
We present our novel modiﬁcation to α -EigenGame called µ -EigenGame along with intuition, theory, and empirical support for critical lemmas.
We begin with identifying and systematically removing the bias that exists in the α -EigenGame updates.
We then explain how removing bias allows us to exploit modern compute architectures culminating in the development of a highly parallelizable algorithm.
3. α -E IGEN G AME ’ S BIASED UPDATES
Consider partitioning the sample covariance matrix C t into a sum of m matrices as C t = 1 n (cid:48)
X (cid:62) t X t =m (cid:80) m mn (cid:48)
X (cid:62) tm X tm = 1 m (cid:80) m C tm .
For sake of exposition, we drop the additional subscript t on
C in what follows.
We would like α -EigenGame to parallelize over these partitions.
However, the gradient of u αi with respect to ˆ v i does not decompose cleanly over the data partitions:
[Equation]
We include the superscript α on the EigenGame gradient to differentiate it from the µ -EigenGame direction later.
The nonlinear appearance of C in the penalty terms makes obtaining an unbiased gradient difﬁcult.
The quadratic term in the numerator of equation (2) could be made unbiased by using two sample estimates of C , one for each term.
But the appearance of the term in the denominator does not have an easy solution.
C m is likely singular for small n (cid:48)
( n (cid:48) < d ) which increases the likelihood of a small denominator, i.e., a large penalty coefﬁcient (boxed), if we were to estimate the denominator with samples.
The result is an update that emphasizes penalizing orthogonality over capturing data variance.
Techniques exist to reduce the bias of samples of ratios of random variables, but to our knowledge, techniques to obtain unbiased estimates are not available.
This was conjectured by Gemp et al.
as the reason for why α -EigenGame performed worse with small minibatches.
3. R EMOVING α -E IGEN G AME ’ S BIAS
It is helpful to rearrange equation (2) to shift perspective from estimating a penalty coefﬁcient (in red) to estimating a penalty direction (in blue):
[Equation]
The penalty direction in equation (3) is still difﬁcult to estimate.
However, consider the case where ˆ v j is any eigenvector of C with associated (unknown) eigenvalue λ (cid:48) .
In this case, C ˆ v j = λ (cid:48)
ˆ v j and the penalty direction (in blue) simpliﬁes to ˆ v j because || ˆ v j || = 1 .
While this assumption is certainly not met at initialization, α -EigenGame leads each ˆ v j towards v j , so we can expect this assumption to be met asymptotically.
This intuition motivates the following µ -EigenGame update direction for ˆ v i with inexact parents ˆ v j (compare orange in equation (4) to blue in equation (3)):
[Equation]
We use ∆ instead of ∇ because the direction is not a gradient (discussed later).
Notice how the strictly linear appearance of C in µ -EigenGame allows the update to easily decompose over the data partitions in equation (4).
The µ -EigenGame update satisﬁes two important properties.
Lemma 1 (Asymptotic equivalence) .
The µ -EigenGame direction, ∆ µi , with exact parents ( ˆ v j = v j ∀ j < i ) is equivalent to α -EigenGame.
Proof.
We start with α -EigenGame and add a superscript e to its gradient to emphasize this is the gradient computed with exact parents ( ˆ v j = v j ).
Then simplifying, we ﬁnd ∇ α,ei
[Equation]
Therefore, once the ﬁrst ( i − 1) eigenvectors are learned, learning the i th eigenvector with µEigenGame is equivalent to learning with α -EigenGame.
Lemma 2 (Zero bias) .
Unbiased estimates of ∆ µi can be obtained with samples from p ( X ) .
Proof.
Let X ∼ p
( X ) where X ∈ R d and p ( X ) is the uniform distribution over the dataset.
Then
[Equation]
where all expectations are with respect to p ( X ) .
These two lemmas provide the foundation for a performant algorithm.
The ﬁrst enables convergence to the desired solution, while the second facilitates scaling to larger datasets.
Algorithm 1 presents pseudocode for µ -EigenGame where computation is parallelized over the k players.
3. M ODEL AND DATA PARALLELISM
[Figure] [Figure]
In our setting we have a number of connected devices.
Speciﬁcally we consider the parallel framework speciﬁed by TPUv3 available in Google Cloud, however our setup is applicable to any multi-host, multi-device system.
The α -EigenGame formulation (Gemp et al. , 2021) considers an extreme form of model parallelism (Figure 2a) where each device has its own unique set of eigenvectors.
In this work we further consider a different form of model and data parallelism which is directly enabled by having unbiased updates (Figure 2b).
This enables µ -EigenGame to deal with both high-dimensional problems as well as massive sample sizes.
Here each set of eigenvectors is copied on M devices.
Update directions are computed on each device individually using a different data stream and then combined by summing or averaging.
Updates are applied to a single copy and this is duplicated across the M − 1 remaining devices.
In this way, updates are computed using an M × larger effective batch size while still allowing device-wise model parallelism.
This setting is particularly useful when the number of samples is very large.
This form of parallelism is not possible using the original EigenGame formulation since it relies on combining unbiased updates.
In this sense, the parallelism discussed in this work generalizes that introduced by Gemp et al.
Note that we also allow for within-device parallelism.
That is, each v i in Figure 2 is a contiguous collection of eigenvectors which are updated independently, in parallel, on a given device (for example using vmap in Jax).
We provide pseudocode in Algorithm 2 in the appendix which simply augments Algorithm 1 with an additional parallelized for-loop and aggregation step over available devices.
We also provide detailed Jax pseudo-code for parallel µ -EigenGame in Appendix F.
We compare the empirical scaling performance of µ -EigenGame against α -EigenGame on a 14 billion sample dataset in section 5.
SVD AS THE SOLUTION TO A NEW E IGEN G AME
We theoretically examine the µ -EigenGame algorithm and 1) prove that, using only minibatches of data, µ -EigenGame converges globally to the true eigenvectors, which 2) comprise the Nash equilibrium of a novel game formulation we recover through deriving pseudo -utility functions from update rules.
Beyond proving speciﬁc theoretical properties of µ -EigenGame, we believe these proof techniques may be of wider interest to the community.
4. C ONVERGENCE TO SVD
The asymptotic equivalence of µ -EigenGame to α -EigenGame ensures µ -EigenGame is globally, asymptotically convergent and its unbiased updates ensure it is scalable.
Proof in appendix C.
Theorem 1 (Global convergence) .
Given a positive deﬁnite covariance matrix C with the top- k eigengaps positive and a square-summable, not summable step size sequence η t (e.g., 1 /t ), Algorithmconverges to the top- k eigenvectors asymptotically ( lim T →∞ ) with probability 1 .
This stochastic asymptotic convergence result is complimentary to the deterministic (full-batch) ﬁnite-sample result in Gemp et al.
(2021) where each ˆ v i is learned in sequence.
In contrast, the proof above applies when learning all ˆ v i in parallel.
We leave ﬁnite-sample convergence to future work (Durmus et al. , 2020).
4. SVD IS N ASH OF µ -E IGEN G AME
We arrived at µ -EigenGame by analyzing and improving properties of the α -EigenGame update.
However, the µ -EigenGame update direction is linear in each ˆ v i .
This suggests we may be able to design a pseudo -utility function for it.
Rearranging the update direction from equation (4) as
[Equation]
reveals that we can reverse-engineer the following utility function
[Equation]
where is the stop gradient operator commonly used in deep learning packages.
As the name implies, stops gradients from ﬂowing through its argument so that equation (8) appears linear in ˆ v i instead of quadratic when differentiating the expression.
In light of this, we have renamed ∆ µi to ˜ ∇ µi to emphasize that it is a pseudo -gradient of u µi .
Note that without the stop gradient, the true gradient of u µi would be [ A + A (cid:62) ]
ˆ v i rather than A ˆ v i
where A
=
[ I − (cid:80)
j<
i ˆ v (cid:62)
j ˆ v j ] C .
We analyze this alternative in Appendix H.1 and ﬁnd it, interestingly, to perform worse than µ -EigenGame empirically.
The utility function u µi has an intuitive meaning.
It is the Rayleigh quotient for the matrix C i
=
[ I − (cid:80)
j<
i ˆ v (cid:62) ˆ v j ] C , which gives the covariance after the subspace spanned by ˆ v j<i has been removed.
In other words, player i is directed to ﬁnd the largest eigenvalue in the orthogonal complement of the approximate top-
( i − 1) subspace.
This approach is known as “deﬂating" the matrix C .
Figure 1b illustrates µ -EigenGame’s reduced bias when estimating the new utility function (and resulting optimum) from an average over minibatches.
Deﬁnition 1 ( µ -Eigen Game ) .
Let µ -EigenGame be the game with players i ∈ { 1 , . .
. , k } , their respective strategy spaces ˆ v i ∈ S d − 1 , and their corresponding utilities u µi as deﬁned in equation (8).
Theorem 2.
Top-
k SVD is the unique Nash of µ -EigenGame given symmetric C with the top- k eigengaps positive.
Proof.
We will show by induction that each v i is the unique best response to v − i , which implies they constitute the unique Nash equilibrium.
First, consider player 1 ’s utility.
It is the Rayleigh quotient of C because ˆ v 1 is constrained to the unit-sphere, i.e., u µ
= ˆ v (cid:62) 1 C ˆ v 1 =
ˆ v (cid:62) 1 C ˆ v 1 ˆ v (cid:62) 1 ˆ v 1 .
Therefore, we know v 1 maximizes u µ
and the maximizer is unique because its eigengap g 1 > 0 .
In game theory parlance, v 1 is a best response to v − 1 .
The proof continues by induction.
The utility of player i is u µ i
= ˆ v (cid:62) i
[ I − (cid:80)
j<i v j v (cid:62) j ] C ˆ v i , which is the Rayleigh quotient with the subspace spanned by the top ( i − 1) eigenvectors removed.
Therefore, the maximizer of u µi is the largest eigenvector in the remaining subspace, i.e., v i .
As before, g i > 0 , so this maximizer is unique.
This shows that each v i is the unique best response to v − i , therefore, the set of v i forms the unique Nash.
Notice how the induction proof of Theorem 2 relies on a) the hierarchy of vectors ( v 1 does not depend on v − 1 ) and b) the fact that u µ i need only be a sensible utility when all player
i ’s parents
[Figure]
The Nash property is important because it enables the use of any black-box procedure for computing best responses.
Like prior work, we develop a gradient method for optimizing each utility, however, that is not a requirement.
Any approach sufﬁces if it can efﬁciently compute a best response.
E XPERIMENTS
As in EigenGame, we omit the projection of gradients onto the tangent space of the sphere; speciﬁcally, we omit line 8 in Algorithm 1.
As discussed in Gemp et al.
(2021), this has the effect of intelligently adapting the step size to use smaller learning rates near the ﬁxed point.
To ease comparison with previous work, we count the longest correct eigenvector streak as introduced by Gemp et al.
(2021), which measures the number of eigenvectors that have been learned, in order, to within an angular threshold (e.g., π/ 8 ) of the true eigenvectors.
We also measure how well the set of ˆ v
i captures the top- k subspace with a normalized subspace distance: 1 − 1 /k ·
Tr( U ∗ P ) ∈ [0 , 1] where U ∗ =
V V † and P =
ˆ V ˆ V † (Tang, 2019).
We provide additional experiments in Appendix A.
MNIST.
We compare µ -EigenGame against α -EigenGame, GHA (Sanger, 1989), Matrix Krasulina (Tang, 2019), and Oja’s algorithm (Allen-Zhu and Li, 2017) on the M NIST dataset.
We ﬂatten each image in the training set to obtain a 60 , 000 × 784 dimensional matrix X .
Figure 3 demonstrates µ -EigenGame’s robustness to minibatch size.
It performs best in the longest streak metric and better than α -EigenGame in subspace distance.
We attribute this improvement to its unbiased updates and additional acceleration effects which we discuss in detail in section H.2.
Meena conversational model.
This dataset consists a subset of the 40 billion words used to train the transformer-based Meena language model (Adiwardana et al. , 2020).
The subset was preprocessed to remove duplicates and then embedded using the trained model.
The dataset consists of n ≈ 14 billion embeddings each with dimensionality d = 2560 ; its total size is 131 TB.
Due to its moderate dimensionality we can exactly compute the ground truth solution by iteratively accumulating the covariance matrix of the data and computing its eigendecomposition.
On a single machine this takes 1.5 days (but is embarrassingly parallelizable with MapReduce).
We use minibatches of size 4,096 in each TPU.
We do model parallelism across 4 TPUs so we see 16,384 samples per iteration.
We test two additional degrees of data parallelism with 4 × (16 TPUs, 65,536 samples) and 8 × (32 TPUs, 131,072 samples) the amount of data per iteration respectively.
We compute and apply updates using SGD with a learning rate of 5 × 10 − 5 and Nesterov momentum with a factor of 0.9.
Figure 4 compares the mean performance of µ -EigenGame against α -EigenGame as a function of the degree of parallelism in computing the top k = 256 eigenvectors (standard errors computed over 5 random seeds).
Each TPU is tasked with learning 32 contiguous eigenvectors.
We see that increasing the degree of parallelism has no effect on the performance of α -EigenGame.
As expected, it is unable to take advantage of the higher data throughput since its updates are biased and cannot be meaningfully linearly combined across copies.
In contrast, the performance of µ -EigenGame scales with the effective batch size achieved through parallelism.
µ -EigenGame ( 8 × ) is able to recovereigenvectors in less than 40,000 iterations in 2 hours 45 minutes (approximately 0.5 epochs).
Spectral clustering on graphs.
We conducted an experiment on learning the eigenvectors of the graph Laplacian of a social network graph (Leskovec and McAuley, 2012) for the purpose of spectral clustering.
The eigenvalues of the graph Laplacian reveal several interesting properties as well such as the number of connected components, an approximation to the sparsest cut, and the diameter of a connected graph (Chung et al. , 1994).
Given a graph with a set of nodes V and set of edges E , the graph Laplacian can be written as
L = X
(cid:62) X where each row of the incidence matrix X ∈ R |E|×|V| represents a distinct edge;
X e =( i,j ) ∈E is a vector containing only 2 nonzero entries, a 1 at index i and a − 1 at index j
(Horaud, 2009).
In this setting, the eigenvectors of primary interest are the bottom- k ( λ |V| , λ |V|− 1 , . . . )
rather than the top- k ( λ 1 , λ 2 , . . . )
, however, a simple algebraic manipulation allows us to reuse a topk solver.
By deﬁning the matrix L − = λ ∗ I − L with λ ∗ > λ 1 , we ensure L − (cid:31)
and the top- k eigenvectors of L − are the bottom- k of L .
The update in equation (4) is transformed into ˜ ∇ µi =
( λ ∗ I − L )
ˆ v i − (cid:80)
j<i (cid:0) ˆ v (cid:62) i
( λ ∗ I − L )ˆ v j (cid:1) ˆ v j .
We provide efﬁcient pseudo-code in Appendix G.
The Facebook graph consists of 134 , 833 nodes, 1 , 380 , 293 edges, and 8 connected components, each formed by a set of Facebook pages belonging to a distinct category, e.g., Government, TV shows, etc.
(Leskovec and Krevl, 2014; Rozemberczki et al. , 2019).
We add a single edge between every pair of components to create a connected graph.
By projecting this graph onto the bottomeigenvectors of the graph Laplacian using µ -EG ( M = 1 , n (cid:48)
=
η t = |E| 1000 ) and then running k -means clustering (Pedregosa et al. , 2011), we are able to recover the ground truth clusters (see Figure 5a) with 99 .
92% accuracy.
The experiment was run on a single CPU.
D ISCUSSION 6. U TILITIES TO UPDATES AND BACK
Figure 5b summarizes the relationships advising the designs of the various EigenGame algorithms.
Starting from the α -EigenGame utility, its update is arrived at by simply following the standard gradient ascent paradigm.
In noticing that stochastic estimates of the gradient are biased, we arrive at the µ -EigenGame update by considering how to remove this bias in a principled manner.
Sacriﬁcing the exact steepest decent direction for a direction that allows unbiased estimates is a tradeoff that in this case has beneﬁts.
Also, while ˜ ∇ µi is not a gradient (except with exact parents), the new penalties have properties (above) that make them intuitively more desirable than the originals; they are adaptive to the state of the system (discussed further in section H.2).
[Figure]
u µi u αi ˜ ∇ µi ∇ αi
Var & Align remove bias
[Figure]
We derive pseudo-utilities with desired theoretical properties by integrating the new updates with help from the stop gradient operator.
However, it is unlikely that this utility would be developed independently of these steps to solve the problem at hand (see Appendix H for more details).
This suggests an alternative approach to algorithm design complementary to the optimization perspective: directly designing updates themselves which converge to the desired solution, reminiscent of previous paradigms that drove neuro-inspired learning rules.
6. B RIDGING H EBBIAN AND OPTIMIZATION APPROACHES
The Generalized Hebbian Algorithm (GHA) (Sanger, 1989; Gang et al. , 2019; Chen et al. , 2019) update direction for ˆ v i with inexact parents ˆ v j is similar to µ -EigenGame:
[Equation]
C appears linearly in this update so GHA can also be parallelized.
In contrast to µ -EigenGame, GHA additionally penalizes the alignment of ˆ v i to itself and removes the unit norm constraint on ˆ v i (not shown).
Without any constraints, GHA overﬂows in experiments.
We take the approach of Gemp et al.
and constrain ˆ v i to the unit-ball ( || ˆ v
i || ≤ 1 ) rather than the unit-sphere ( || ˆ
v i ||
= 1 ).
The connection between GHA and µ -EigenGame is interesting because unlike µ -EigenGame, GHA is a Hebbian learning algorithm inspired by neuroscience and its update rule is not motivated from the perspective of maximizing of a utility function.
Game formulations of classical machine learning problems may provide a bridge between statistical and biologically inspired viewpoints.
C ONCLUSION
We introduced µ -EigenGame, an unbiased, globally convergent, parallelizable algorithm that recovers the top- k eigenvectors of a symmetric positive deﬁnite matrix.
We demonstrated the performance of µ -EigenGame on large scale dimension reduction and clustering problems.
We discussed technical details of µ -EigenGame within the context of game theory, machine learning and neuroscience.
Like its predecessor, µ -EigenGame is a k -player, general-sum game allowing model parallelism over players; our unbiased reformulation allows even greater parallelism over data.
Furthermore, the hierarchy and Nash property enable the exploration of more sophisticated best responses.
µ -EigenGame’s improved robustness to smaller minibatches makes it more amenable to being used as part of deep learning, optimization (Krummenacher et al. , 2016), and regularization (Miyato et al. , 2018) techniques which leverage spectral information of gradient covariances or Hessians.
Graph spectral methods have also recently shown to be related to state-of-the-art representation learning algorithms (HaoChen et al. , 2021) further cementing the importance of efﬁcient SVD algorithms in modern machine learning.
Acknowledgements.
We would like to thank Trevor Cai, Rosalia Schneider, Dimitrios Vytiniotis for invaluable help with optimizing algorithm performance on TPU.
We also thank Maribeth Rauh,
Zonglin Li, Daniel Adiwardana and the Meena team for providing us with data and assistance.
And ﬁnally, we thank Alexander Novikov for helpful feedback on the manuscript
A E XPERIMENTS ON SYNTHETIC DATA
[Figure]
i
[Figure]
We validate µ -EigenGame in a full-batch setting on two synthetic datasets: one with exponentially decaying spectrum; the other with a linearly decaying spectrum.
Figure 6 shows µ -EigenGame outperforms α -EigenGame on the former and matches its performance on the latter.
We discuss possible reasons for this gap in the discussion in Section 6.
B P ARALLELIZED A LGORITHM
Riemannian Manifolds.
Before introducing an algorithm for µ -EigenGame, we ﬁrst brieﬂy review necessary terminology for learning on Riemannian manifolds Absil et al.
(2009), speciﬁcally for the sphere.
The notation T ˆ v i S
d − 1 denotes the set of vectors tangent to the sphere at a point ˆ v i (i.e., any vector orthogonal to ˆ v i ).
R ˆ v i ( z )
=
ˆ v i
+
z
|| ˆ
v i
+ z
|| is the commonly used restriction of the retraction on
S d − 1 to the tangent bundle at ˆ v
i (i.e., step in tangent direction z and then unit-normalize the result).
The operator Π ˆ v i ( y ) =
( I − ˆ v (cid:62) i ˆ v i )
y projects the direction y onto T ˆ v i S
d − 1 .
Combining these tools together results in a movement along the Riemannian manifold: ˆ v ( t +1)
i ← R ˆ v i (cid:0) Π
ˆ v
i ( y ) (cid:1) .
We present pseudocode for µ -EigenGame below where computation is parallelized both over the k players and over M machines per player.
Algorithm 2 µ -EigenGame R 1: Given: data stream X t ∈ R n (cid:48) × d , number of parallel machines M per player (minibatch size per machine n (cid:48)(cid:48) =
n (cid:48) M ), initial vectors ˆ v 0 i ∈ S
d − 1 , step size sequence η t , and number of iterations T .
2: ˆ v i ← ˆ v
i for all i 3: for t = 1 : T do 4: parfor i = 1 : k do 5: parfor m = 1 : M do 6: rewards ← X (cid:62) tm X tm ˆ
v
i 7: penalties ← (cid:80) j<i (cid:104)
X tm ˆ v i , X tm ˆ v j (cid:105)
ˆ v j 8: ˜ ∇ µim ← ( rewards − penalties ) /n
(cid:48)(cid:48)
˜ ∇ µ,Rim ← ˜ ∇ µim − (cid:104) ˜ ∇ µim , ˆ v i (cid:105) ˆ v i 10: end parfor 11: ˜ ∇ µ,Ri ← 1 M (cid:80) m
[ ˜ ∇ µ,Rim ] 12: ˆ v (cid:48)
i ← ˆ v i + η t ˜ ∇ µ,R i 13: ˆ v i ← ˆ v (cid:48)
i ||
ˆ v (cid:48)
i || 14: end parfor 15: end for 16: return all ˆ v
i
C G LOBAL S TOCHASTIC C ONVERGENCE
Theorem 1 (Global Convergence) .
Given a positive deﬁnite covariance matrix C with the top- k eigengaps positive and a square-summable, not summable step size sequence η t (e.g., 1 /t ), Algorithmconverges to the top- k eigenvectors asymptotically ( lim T →∞ ) with probability 1 .
Proof.
Assume none of the ˆ v i are initialized to an angle exactly 90 ◦ away from the true eigenvector: (cid:104)
ˆ v i , v i (cid:105) (cid:54)
= 0 .
The set of vectors { ˆ v i : (cid:104) ˆ v i , v i (cid:105) = 0 } has Lebesgue measure 0 , therefore, the above assumption holds w.p.1.
The update direction for the top eigenvector ˆ v 1 is exactly equal to that of α -EigenGame ( ˜ ∇ µ 1 = ∇ α 1 ), therefore, they have the same limit points for ˆ v 1 .
The proof then proceeds by induction.
As ˆ v j<i approach their limit points, the update for the i th eigenvector ˆ v i approaches that of α -EigenGame ( ˜ ∇ µi = ∇ αi ) and, by Lemma 3, the stable region of µ -EigenGame also shrinks to a point around the top- k eigenvectors.
Denote the “update ﬁeld” H ( v ) to match the work of Shah (2019).
H ( v ) is simply the concatenation of all players’ Riemannian update rules, i.e., all players updating in parallel using their Riemannian updates:
[Equation]
where ∆ µi is deﬁned in equation (7) and ( I − p 1 p (cid:62) 1 )∆ µi projects ∆ µi onto the tangent space of player
i
’s unit sphere
The result is then obtained by applying Theorem 7 of Shah (2019) with the following information:
a) the unit-sphere is a compact manifold with an injectivity radius of π , b) the update ﬁeld is a polynomial in { v i } and therefore smooth (analytic), and c) by Lemma 4 (see Appendix E) the update noise constitutes a bounded martingale difference sequence.
While the convergence proof for α -EG provides ﬁnite-sample rates, it only applies to the algorithm applied sequentially (not parallelized over eigenvectors) and in the deterministic setting (minibatch contains the entire data set).
The experiments in Gemp et al.
(2021) apply the algorithm in parallel and with mini batch sizes, meaning the α -EG theorem does not actually apply to their experimental setting.
That is to say, the α -EG paper proposes updating eigenvectors in parallel in practice despite the lack of convergence guarantee.
In contrast, our convergence theorem applies to µ -EG when applied in parallel (over the eigenvectors) and in the stochastic setting (with mini batch sizes), which is what we examine empirically in our experiments.
The downside is that we do not provide ﬁnite-sample convergence rates.
Although we do not provide convergence rates, Lemma 1 proves that the µ -EG update converges to the α -EG update for each eigenvector, so intuitively, we expect the convergence rates to be relatively similar given that the algorithms are equivalent in the limit.
Note that in the full batch setting where stochasticity does not conﬂate the differences between the two algorithms, Figurein Appendix A empirically supports the similarity of the convergence rates for the two algorithms.
Figure 3 (minibatch of 1024 ) which looks at a large (but not full) minibatch size, also shows a small difference between convergence for the two algorithms.
In summary, the α -EG convergence theorem is impractical—it provides convergence rates for a (non-parallel) algorithm in the (non-stochastic) setting, which is a combination that α -EG paper does not suggest be applied in practice.
In contrast, our µ -EG convergence theorem is practical—it provides asymptotic convergence for a parallel algorithm in the stochastic setting.
Difﬁculties Obtaining Finite Sample Rates In consideration of a ﬁnite sample convergence result, we consulted Durmus et al.
The primary obstacle to applying their convergence theorem is the construction of a suitable Lyapunov function to satisfy their Assumption A.2 stated on page 4.
Constructing Lyapunov functions is typically a tedious, unpredictable process.
The work in Durmus et al.
(2020) is very recent and ﬁnite sample convergence of Riemannian stochastic approximation (i.e., update directions are not gradients of any function) schemes is cutting edge, highly technical research.
This is in contrast to Riemannian optimization (i.e., update directions are the gradient of a function), which is much more mature.
We hope theory advances in the near future to a point where we can more easily provide convergence rates for algorithms like α -EigenGame.
D E RROR P ROPAGATION / S ENSITIVITY A NALYSIS
Lemma 3.
An O ( (cid:15) ) angular error of parent ˆ v j<i implies an O ( (cid:15) ) angular error in the location of the solution for ˆ v i .
Proof.
The proof proceeds in three steps:
O ( (cid:15) ) angular error of parent =
⇒ O
( (cid:15) )
Euclidean error of parent 2.
O ( (cid:15) )
Euclidean error of parent =
⇒ O
( (cid:15) )
Euclidean error of norm of child gradient 3.
O ( (cid:15) )
Euclidean error of norm child gradient + instability of minimum at ± π 2 = ⇒ O
( (cid:15) ) angular error of child’s solution.
Angular error in the parent can be converted to Euclidean error by considering the chord length between the mis-speciﬁed parent and the true parent direction.
The two vectors plus the chord form an isoceles triangle with the relation that chord length l = 2 sin( (cid:15) 2 ) is O ( (cid:15) ) for (cid:15) (cid:28) 1 .
Next, write the mis-speciﬁed parents as ˆ v j = v j + w j where || w j || is O ( (cid:15) ) as we have just shown.
Let b equal the difference between the Riemannian update direction ˜ ∇ µi with approximate parents and that with exact parents.
All directions we consider here are the Riemannian directions, i.e., they have been projected onto the tangent space of the sphere.
Then
[Equation] [Equation]
and the norm of the difference is
[Equation] [Equation]
We can further bound the summands with
[Equation]
(cid:15) ) .
This upper bound on the norm of the difference between the two directions translates to a lower bound on the inner product of the two directions wherever ||
˜ ∇ µ,ei || > (cid:15) , speciﬁcally
(cid:104) ˜
∇ µ,ei , ˜ ∇ µi (cid:105) >
(see Figure 7a).
And recall that the direction with exact parents is equivalent to the gradient of α -EigenGame with exact parents, ∇ α,ei .
Therefore, by a Lyapunov argument, the ˜ ∇ µi direction is an ascent direction on the α -EigenGame utility where it forms an acute angle (positive inner product) with ∇ α,ei .
Furthermore, ∇ α,ei is the gradient of a utility function that is sinusoidal along the sphere manifold; speciﬁcally, it is a cosine with period π and positive amplitude dependent on the spectrum of C (c.f. equation (8) of Gemp
[Figure]
et al.
We can derive an upper bound on the size of the angular region for which ˜ ∇ µi is not necessarily an ascent direction (the “?" marks in Figure 7).
This region is deﬁned as the set of angles for which the norm of the utility’s derivative is small, i.e., ||∇ α,ei || ≤ (cid:15) .
The derivative of cosine is sine, which depends linearly on its argument (angle) for small values, therefore, | θ | ≤ O ( (cid:15) ) or | π 2 − θ | ≤ O ( (cid:15) ) .
As long as ˆ v i does not lie within the | π 2 − O ( (cid:15) ) | region, µ -EigenGame will ascend the utility landscape to within O ( (cid:15) ) angular error of the true eigenvector v i .
In the limit as (cid:15) → 0 , the size of the | π 2 − O ( (cid:15) )
| region vanishes to a point, v ⊥ i .
To understand the stability of this point, we can again appeal to the analysis from Gemp et al.
(2021)—see equation (8) on page 7 of that work.
The
Jacobian of ˜ ∇ µi and the Hessian of u αi are equal with exact parents, and we know that its Riemannian
Hessian is positive deﬁnite if the i th eigengap is positive: H R ˆ v i [ u αi ] (cid:23) ( λ i − λ i +1 )
I .
This means that the point v ⊥ i is a repeller for α -EigenGame.
Similarly to before, we can show more formally that an
O ( (cid:15) ) perturbation to parents results in an O ( (cid:15) ) perturbation to the Jacobian of ˜ ∇ µi from H [ u αi ] :
[Equation]
where W is some matrix with O (1) entries (w.r.t.
(cid:15) ).
For the sphere, the Riemannian Jacobian is a linear function of the Jacobian ( J R ˆ v i
=
( I − ˆ v i ˆ v (cid:62) i ) J − (ˆ v (cid:62) i J ˆ v i )
I =
H R
ˆ v i [ u αi ] − O ( (cid:15) ) ) and therefore the error remains O ( (cid:15) ) .
The set of (non)symmetric, positive semideﬁnite matrices ( A is p.s.d.
iff y (cid:62)
Ay ≥ 0 ∀ y ) forms a closed convex cone, the interior of which contains positive deﬁnite matrices Wang et al.
Therefore, J R ˆ v i remains in this set after a small enough O ( (cid:15) ) perturbation.
Therefore, in the limit (cid:15) → 0 , the spectrum of the Jacobian will also be positive deﬁnite indicating the point v ⊥ i is a repeller.
This is indicated by the blue arrows in Figure 7b.
Figure 7b summarizes the results that the stable region for α -EigenGame consists of an O ( (cid:15) ) ball around the true optimum for parents with O ( (cid:15) ) angular error.
E N OISE IS M ARTINGALE D IFFERENCE S EQUENCE
Let ∆ µ,ti =
[Equation]
) i be the µ -EigenGame update direction computed using the full expected covariance matrix.
Let ˆ∆ µ,ti
[Equation]
be the update direction computed using a minibatch estimate of the covariance matrix where minibatches are unbiased because they are formed from data sampled uniformly at random from the dataset.
Deﬁne M V ( t ) i,t +1 = ˆ∆ µ,ti − ∆ µ,ti and let M V ( t ) t +1 =
[ M V ( t ) 1 ,t +1 , . . .
, M V ( t ) k,t +1 ] (cid:62) where V ( t )
=
{ v ( t ) i ∈
[ k ] } .
Lemma 4.
{ M V ( t ) t +1 } is a bounded martingale difference sequence with respect to the increasing σ -ﬁelds
[Equation]
Proof.
Given the ﬁltration F t , we ﬁnd
[Equation]
29) where the ﬁrst equality holds because each C t is formed from a minibatch sampled i.i.d.
from the dataset and therefore independent of the ﬁltration.
This result holds for all i ∈ [ k ] , therefore
E [ M V
[Equation]
t )
[Equation]
1 |F t ] = 0 .
[Equation] [Equation]
Furthermore,
[Equation]
36) where λ i ≤ max { 1 , ( i − 2) 2 } is the max singular value of P 1 and ξ 2 is the maximum eigenvalue of E
[( C t − C ) (cid:62) ( C t − C )] over all t .
Summing over i we ﬁnd
[Equation]
This is a worst case bound.
As the parent eigenvectors converge, the max singular value of P converges to 1 so that the variance of the magnitude of the martingale difference is upper bounded by kξ 2 .
Note : For a ﬁnite dataset of n samples or for a distribution with bounded moments like the Gaussian distribution, ξ will be ﬁnite.
However, for other distributions like the Cauchy distribution, ξ may be unbounded, so care should be taken when running µ -EigenGame in different stochastic settings.
F J AX PSEUDOCODE
For the sake of reproducibility we have included pseudocode in Jax.
We use the Optax 2 optimization library Hessel et al.
(2020) and the Jaxline training framework 3 .
Our graph algorithm is a straightforward modiﬁcation of the provided pseudo-code.
See section G for details.
1 """ 2 Copyright
The EigenGame Unloaded Authors.
35 Licensed under the Apache License, Version 2.0 (the "License"); 6 you may not use this file except in compliance with the License.
7 You may obtain a copy of the License at 8 9 https://www.apache.org/licenses/LICENSE-2.
10 11 Unless required by applicable law or agreed to in writing, software 12 distributed under the License is distributed on an "AS IS" BASIS, 13 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14 See the License for the specific language governing permissions and 15 limitations under the License.
jax 19 import
20 import jax.numpy as jnp 21 22 def eg_grads vi
.ndarray, weights jnp.ndarray, jnp ndarray, jnp ndarray ndarray: "
Args: vi
weights: penalty coefficients, shape (k, d), i.e., vectors on rows data: shape (N, d), minibatch X_t
Returns: grads: shape (d,), gradient for vi """ weights_ij = (jnp.sign(weights + 0.5) - 1.) / 2.
# maps -1 to -1 else to data_vi =
jnp.dot(data, vi) data_eigs = jnp.transpose(jnp.dot(data, jnp.transpose(eigs)))
# Xvj on row j vi_m_vj
=
jnp.dot(data_eigs, data_vi) penalty_grads =
vi_m_vj * jnp.transpose(eigs) penalty_grads =
jnp.dot(penalty_grads, weights_ij) grads = jnp.dot(jnp.transpose(data), data_vi)
+ penalty_grads return grads 4446 def utility(vi, weights, eigs, data): 2 https://github.com/deepmind/optaxhttps://
.com/
Eigengame utilities.
""" data_vi = jnp dot data, vi) data_eigs = jnp transpose(jnp dot data jnp transpose(eigs)))
# on row j vi_m_vj2 = jnp.dot
data_eigs, data_vi)**2.
vj_m_vj =
jnp.sum(
_eigs *
_eigs,
=1) r_ij = vi_m_vj2 / vj_m_vj util =
jnp dot jnp array ij
) return state
return keys =
[Table]
G µ -E IGEN G AME ON G RAPHS
Algorithm 3 receives a stream of edges represented as a matrix with edges on the rows and outgoing node id ( out ) and incoming node id ( in ) as nonegative integers on the columns.
The method zeros_like ( z ) returns an array of zeros with the same dimensions as z .
The method index_add ( z, idx, val ) adds the values in array val to z at the corresponding indices in array idx with threadsafe locking so that indices in idx may be duplicated.
Both methods are available in J AX .
The largest eigenvector ˆ v 1 is learned to estimate λ 1 and may be discarded.
The bottom- k eigenvectors are returned by the algorithm in increasing order.
Algorithm 3 expects k + 1 random unit vectors as input rather than k
in order to additionally estimate the top eigenvector necessary for the computation; otherwise, the inputs are the same as Algorithm 1.
H A LGORITHM D ESIGN P ROCESS
In section 4.1, we presented u µi as the Rayleigh quotient of a deﬂated matrix (repeated in equation (8) for convencience):
[Equation] [Equation]
Alternatively, we can consider u µi as equation (42) in light of the derivation for u αi by Gemp et al.
In that case, utilities are constructed from entries in the matrix
3: λ 1 ←
V
* 4: for t = 1 : T do 5: parfor i = 1 : k + 1 do 6: parfor m = 1 : M do 7:
[ Xv ] i
=
ˆ v i ( out tm ) − ˆ v i ( in tm ) 8:
[ X (cid:62) Xv ] i ← zeros_like
(ˆ v i ) 9:
[ X (cid:62) Xv ] i ← index_add ([ X (cid:62) Xv ] , out tm , [ Xv ] i ) 10:
[ X (cid:62) Xv ] i ← index_add ([ X (cid:62) Xv ] , in tm , − [ Xv ] i ) 11: if i = 1 then 12: λ 1 ← ||
[ Xv ] i ||13: ˜ ∇ µit (cid:48)
← [ X (cid:62) Xv ] i 14: else 15: ˜ ∇ µim ← λ 1 [ˆ v i − (cid:80) 1 <j<i (ˆ v (cid:62) i ˆ v j )ˆ v j ] 16:
[ Xv ] j = ˆ v j ( out tm ) − ˆ v j ( in tm ) for all j 17: ˜ ∇ µit (cid:48) − =
[ X (cid:62) Xv ] i − (cid:80) 1 <j<i ([ Xv ] (cid:62) i
[ Xv ] j )ˆ v j 18: end if 19: end parfor 20:
[Equation]
[Equation] [Equation]
23: end parfor 24: end for 25: return { ˆ v i | i ∈ { 2 , . . .
, k + 1 }}
[Equation] [Equation]
It is argued
that if ˆ V diagonalizes M and captures maximum variance, then the diagonal (cid:104) ˆ v i , C ˆ v i (cid:105) terms must be maximized and the off-diagonal (cid:104)
ˆ v i , C ˆ v j (cid:105) terms must be zero.
As the latter mixed terms may be negative, the authors square the mixed terms to form “minimizable utilities” and divide them by (cid:104)
ˆ v j , C ˆ v j (cid:105) so that they have similar “units" to the terms (cid:104)
ˆ v i , C
ˆ v i (cid:105) of the ﬁrst type.
In contrast, the u µi utilities could be arrived at by instead multiplying the mixed terms by (cid:104)
ˆ v i , ˆ v j (cid:105) .
While this ensures the mixed terms are positive with exact parents (because (cid:104) ˆ v i , Cv j (cid:105)
=
λ j (cid:104)
ˆ v i , v j (cid:105) )
, it does not ensure they are always positive in general 4 .
In other words, u µi is deﬁned in way such that the ⊥ -penalties actually encourage vectors to align at times when they should in fact do the opposite!
We therefore consider it unlikely that anyone would pose equation (42) as a utility if coming from the perspective of α -EigenGame.
We could have extended the diagram in Figure 5b to include this dead end link.
We have also included the true gradient of u µi as a logical endpoint.
We present these extensions in Figure 8.
[Equation]
Var &
[Equation]
bias remove
[Equation]
H. G RADIENT A SCENT ON u µi
If we remove the stop gradient from equation (41), we are left with equation (45):
[Equation]
If we then differentiate this utility
[Figure]
we ﬁnd its gradient is
[Equation]
We also reran experiments with this update direction, ∇ µi on the synthetic and MNIST domains.
The update is unbiased, so it would be expected to scale well, however, it (in orange) appears to scale more poorly than µ -EigenGame with smaller minibatches.
[Figure]
In Figure 10, ˜ µ -EG appears to converge in terms of subspace error but slows in terms of longest eigenvector streak.
˜ µ -EG updates are also unbiased so we would expect it is convergent globally, but it underperforms relative to µ -EG.
In contrast, α -EG stalls in terms of subspace error likely due to bias.
Note that with exact parents, mu-EG and mu-tilde-EG have the same update (plug Cvj =
λ j vj into equation (46)), so the difference must come from when the parents are still inaccurate.
In Figure 11, we have plotted the norm of the difference between subsequent values of the eigenvectors over training, i.e., how “far” v i moves after every update.
Note all algorithms were run with the same ﬁxed step size of 10 − 3 , which was optimal for each algorithm in this setting.
Clearly, the gradient version of α -EigenGame ( ˜ µ -EG) shown in Figure 11c exhibits higher norms overall.
We believe this is due to the higher variance penalty terms (all methods maximize the same Rayleigh quotient term).
Note that both α -EG and µ -EG construct their pentalty directions by a weighted sum of terms.
These terms are computed differently, but both compute weights with inner products between v i and v j after projecting onto the samples in the minibatch X t .
For example, µ -EG
[Figure]
computes (cid:104)
X t v i , X t v j (cid:105)
= v (cid:62) i
C t v j .
Without loss of generality, assume C is a diagonal matrix (with the eigenvalues on its diagonal).
Then (cid:104) v (cid:62) i
Cv j (cid:105) =
(cid:80) k
λ k v ik v jk .
The eigenvectors v i = e i in this case, and so the inner product measures alignment between v i and v j in the dimensions that v i and v j are trained to be orthogonal.
Due to noise in the minibatches X t , v i and v j may “drift” in the remaining dimensions.
Projecting essentially ignores these though because they are weighted by small eigenvalues.
In contrast, ˜ µ -EG computes weights as raw inner products between v i and v j .
Therefore, any drift of v i and v j due to noise in the minibatch samples contributes to the inner product: (cid:104) v i , v j (cid:105) =
(cid:80) k v ik v jk .
We suspect this is the reason ˜ µ -EG exhibits higher drift distance.
In summary, α -EG updates are computed as a weighted sum of terms where the weights are computed using inner products between v i and its parents after projecting them to a lower dimensional space.
Computing the inner product in this particular space results in lower variance for each inner product.
Unfortunately, α -EG updates are biased, so while they exhibit relatively low “norm of drift”, they converge to the incorrect solution (parents never converge to precise solution which prohibits children from learning accurately solutions).
˜ µ -EG updates are unbiased, so they should converge to the correct solution in the limit, but they exhibit higher variance due to their penalty weights being computed in the original high dimensional space (i.e., they pick up every little bit of noise).
Finally, µ -EG updates are unbiased and compute their penalty weights in a lower dimensional space, suppressing the bulk of the noise that appears from drift caused by randomness in the minibatches.
They exhibit the lowest levels of “norm of drift”.
[Figure]
(a-c) correspond to µ -EigenGame, α -EigenGame, and the gradient version of α -EigenGame discussed above respectively.
Each experiment is conducted with a minibatch size of 32 over ﬁve epochs of training and averaged over 10 trials.
Each curve shows the update distance after each iteration for one of the top- 16 eigenvectors.
Cyan curves indicate eigenvectors higher in the hierarchy (e.g., v 1 ) and magenta curves indicate eigenvectors lower in the hierarchy.
H. A CCELERATION
We conjecture that µ -EigenGame converges more quickly than α -EigenGame because of the following two claims.
Claim 1 The penalty terms of ˜ ∇ µi are all within 90 ◦ of those of ∇ αi because (cid:68) C ˆ v j ˆ v (cid:62) j C ˆ v j , ˆ v j (cid:69) = 1 > 0 .
Claim 2 The penalty terms of ˜ ∇ µi are all smaller in magnitude than those of ∇ αi : || ˆ v j || ≤ (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
C ˆ v j ˆ v (cid:62) j C ˆ v j (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) .
Indeed, consider the direction C
ˆ v j .
By properties of the vector rejection, we know the rejection of this direction onto the tangent space of the unit sphere has magnitude less than or equal to that of the original vector,
||
C ˆ v j || .
The projection is ( I − ˆ v j ˆ v (cid:62) j )( C ˆ v j ) .
Therefore, the rejection is ˆ v j ˆ v (cid:62) j
( C ˆ v j ) and, by the preceding argument, we know its magnitude | ˆ v (cid:62) j C ˆ v j ||| ˆ v j || is less than or equal to ||
C ˆ v j || .
Rearranging the inequality completes the proof.
By Claim 1 , the penalty directions of µ -EG and α -EG approximately agree.
And by Claim 2 , α -EG’s penalty direction is shorter.
Consider a scenario where a parent of ˆ v
i has not converged and transiently occupies space along ˆ v i ’s geodesic to its true endpoint
ˆ v i , a strong penalty term will force ˆ v i to take a roundabout trajectory, thereby slowing its convergence.
A weaker penalty term allows ˆ v i to pass through regions occupied by its parent as long as its parent is not an eigenvector.
Recall from
Section 3 that the two utilities are equivalent when the parents are eigenvectors.