Original: E IGEN G AME U NLOADED W HEN PLAYING GAMES IS BETTER THAN OPTIMIZINGAnonymous authors
Revised: E IGEN G AME U NLOADED W HEN PLAYING GAMES IS BETTER THAN OPTIMIZING

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We build on the recently proposed EigenGame that views eigendecomposition as a competitive game.
Revised: We build on the recently proposed EigenGame that views eigendecomposition as a competitive game.

Original: EigenGame’s updates are biased if computed using minibatches of data, which hinders convergence and more sophisticated parallelism in the stochastic setting.
Revised: EigenGame’s updates are biased if computed using minibatches of data, which hinders convergence and more sophisticated parallelism in the stochastic setting.

Original: In this work, we propose an unbiased stochastic update that is asymptotically equivalent to EigenGame, enjoys greater parallelism allowing computation on datasets of larger sample sizes, and outperforms EigenGame in experiments.
Revised: In this work, we propose an unbiased stochastic update that is asymptotically equivalent to EigenGame, enjoys greater parallelism allowing computation on datasets of larger sample sizes, and outperforms EigenGame in experiments.

Original: We present applications to ﬁnding the principal components of massive datasets and performing spectral clustering of graphs.
Revised: We present applications to ﬁnding the principal components of massive datasets and performing spectral clustering of graphs.

Original: We analyze and discuss our proposed update in the context of EigenGame and the shift in perspective from optimization to games
Revised: We analyze and discuss our proposed update in the context of EigenGame and the shift in perspective from optimization to games.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Large, high-dimensional datasets containing billions of samples are commonplace.
Revised: Large, high-dimensional datasets containing billions of samples are commonplace.

Original: Dimensionality reduction to extract the most informative features is an important step in the data processing pipeline which enables faster learning of classiﬁers and regressors (Dhillon et al. , 2013), clustering (Kannan and Vempala, 2009), and interpretable visualizations.
Revised: Dimensionality reduction to extract the most informative features is an important step in the data processing pipeline which enables faster learning of classiﬁers and regressors (Dhillon et al. , 2013), clustering (Kannan and Vempala, 2009), and interpretable visualizations.

Original: Many dimensionality reduction and clustering techniques rely on eigendecomposition at their core including principal component analysis (Jolliffe, 2002), locally linear embedding (Roweis and Saul, 2000), multidimensional scaling (Mead, 1992),
Revised: Many dimensionality reduction and clustering techniques rely on eigendecomposition at their core including principal component analysis (Jolliffe, 2002), locally linear embedding (Roweis and Saul, 2000), multidimensional scaling (Mead, 1992),

Original: Isomap (Tenenbaum et al. , 2000), and graph spectral clustering (Von Luxburg, 2007).
Revised: Isomap (Tenenbaum et al. , 2000), and graph spectral clustering (Von Luxburg, 2007).

Original: Numerical solutions to the eigenvalue problem have been approached from a variety of angles for centuries: Jacobi’s method, Rayleigh quotient, power (von Mises) iteration (Golub and Van der
Revised: Numerical solutions to the eigenvalue problem have been approached from a variety of angles for centuries: Jacobi’s method, Rayleigh quotient, power (von Mises) iteration (Golub and Van der

Original: Vorst, 2000).
Revised: Vorst, 2000).

Original: For large datasets that do not ﬁt in memory, approaches that access only subsets—or minibatches —of the data at a time have been proposed.
Revised: For large datasets that do not ﬁt in memory, approaches that access only subsets—or minibatches —of the data at a time have been proposed.

Original: Recently, EigenGame (Gemp et al. , 2021) was introduced with the novel perspective of viewing the set of eigenvectors as the Nash strategy of a suitably deﬁned game.
Revised: Recently, EigenGame (Gemp et al. , 2021) was introduced with the novel perspective of viewing the set of eigenvectors as the Nash strategy of a suitably deﬁned game.

Original: While this work demonstrated an algorithm that was empirically competitive given access to only subsets of the data, its performance degraded with smaller minibatch sizes, which are required to ﬁt high dimensional data onto devices.
Revised: While this work demonstrated an algorithm that was empirically competitive given access to only subsets of the data, its performance degraded with smaller minibatch sizes, which are required to ﬁt high dimensional data onto devices.

Original: One path towards circumventing EigenGame’s need for large minibatch sizes is parallelization.
Revised: One path towards circumventing EigenGame’s need for large minibatch sizes is parallelization.

Original: In a data parallel approach, updates are computed in parallel on partitions of the data and then combined such that the aggregate update is equivalent to a single large-batch update.
Revised: In a data parallel approach, updates are computed in parallel on partitions of the data and then combined such that the aggregate update is equivalent to a single large-batch update.

Original: The technical obstacle preventing such an approach for EigenGame lies in the bias of its updates, i.e., the divide-and-conquer
Revised: The technical obstacle preventing such an approach for EigenGame lies in the bias of its updates, i.e., the divide-and-conquer

Original: EigenGame update is not equivalent to the large-batch update.
Revised: EigenGame update is not equivalent to the large-batch update.

Original: Biased updates are not just a theoretical nuisance; they can slow and even prevent convergence to the solution (made obvious in Figure 4a).
Revised: Biased updates are not just a theoretical nuisance; they can slow and even prevent convergence to the solution (made obvious in Figure 4a).

Original: In this work we introduce a formulation of EigenGame which admits unbiased updates which we term µ -EigenGame.
Revised: In this work we introduce a formulation of EigenGame which admits unbiased updates which we term µ -EigenGame.

Original: We will refer to the original formulation of EigenGame as α -EigenGame.µ
Revised: We will refer to the original formulation of EigenGame as α -EigenGame.µ

Original: -EigenGame
Revised: -EigenGame

Original: and α -EigenGame are contrasted in Figure 1.
Revised: and α -EigenGame are contrasted in Figure 1.

Original: Unbiased updates allow us to increase the effective batch size using data parallelism.
Revised: Unbiased updates allow us to increase the effective batch size using data parallelism.

Original: Lower variance updates mean that µ -EigenGame should converge faster and to more accurate solutions than α
Revised: Lower variance updates mean that µ -EigenGame should converge faster and to more accurate solutions than α

Original: -EigenGame regardless of batch size.
Revised: -EigenGame regardless of batch size.

Original: In Figure 1a (top), the density of the shaded region shows the distribution of steps taken by thestochastic variant of each algorithm after 100 burn-in steps. Although the expected path of α -EG is
Revised: In Figure 1a (top), the density of the shaded region shows the distribution of steps taken by the

Original: [Figure]
Revised: [Figure]

Revised: stochastic variant of each algorithm after 100 burn-in steps.

Original:  slightly more direct, its stochastic variant has much larger variance.
Revised: Although the expected path of α -EG is slightly more direct, its stochastic variant has much larger variance.

Original: Figure 1a (bottom) shows that with increasing iterations, the µ -EG trajectory approaches its expected value whereas α -EG exhibits larger bias.
Revised: Figure 1a (bottom) shows that with increasing iterations, the µ -EG trajectory approaches its expected value whereas α -EG exhibits larger bias.

Original: Figure 1b further supports µ -EigenGame’s reduced bias with details in Sections 3 and 4.
Revised: Figure 1b further supports µ -EigenGame’s reduced bias with details in Sections 3 and 4.

Original: Our contributions : In the rest of the paper, we present our new formulation of EigenGame, analyze its bias and propose a novel unbiased parallel variant, µ -EigenGame with stochastic convergence guarantees.
Revised: Our contributions : In the rest of the paper, we present our new formulation of EigenGame, analyze its bias and propose a novel unbiased parallel variant, µ -EigenGame with stochastic convergence guarantees.

Original: µ -EigenGame’s utilities are distinct from α -EigenGame and offer an alternative perspective.
Revised: µ -EigenGame’s utilities are distinct from α -EigenGame and offer an alternative perspective.

Original: We demonstrate its performance with extensive experiments including dimensionality reduction of massive data sets and clustering a large social network graph.
Revised: We demonstrate its performance with extensive experiments including dimensionality reduction of massive data sets and clustering a large social network graph.

Original: We conclude with discussions of the algorithm’s design and context within optimization, game theory, and neuroscience.
Revised: We conclude with discussions of the algorithm’s design and context within optimization, game theory, and neuroscience.

Original:  P RELIMINARIES AND RELATED WORK
Revised:  P RELIMINARIES AND RELATED WORK

Original: In this work, we aim to compute the top- k right singular vectors of data X , which is either represented as a matrix, X ∈ R n × d , of n d -dimensional samples, or as a d -dimensional random variable.
Revised: In this work, we aim to compute the top- k right singular vectors of data X , which is either represented as a matrix, X ∈ R n × d , of n d -dimensional samples, or as a d -dimensional random variable.

Original: In either case, we assume we can repeatedly sample a minibatch X t from the data of size n (cid:48) < n , X t ∈ R n (cid:48)
Revised: In either case, we assume we can repeatedly sample a minibatch X t from the data of size n (cid:48) < n , X t ∈ R n (cid:48)

Original: × d .
Revised: × d .

Original: The top- k right singular vectors of the dataset are then given by the top- k eigenvectors of the (sample) covariance matrix, C = E
Revised: The top- k right singular vectors of the dataset are then given by the top- k eigenvectors of the (sample) covariance matrix, C = E

Original: [ 1 n (cid:48) X (cid:62) t X t ] =
Revised: [ 1 n (cid:48) X (cid:62) t X t ] =

Original: E [ C t ] .
Revised: E [ C t ] .

Original: For small datasets, SVD is appropriate.
Revised: For small datasets, SVD is appropriate.

Original: However, the time, O (min { nd 2 , n 2 d } ) , and space, O ( nd ) , complexity of SVD prohibit its use for larger datasets (Shamir, 2015) including when X is a random variable.
Revised: However, the time, O (min { nd 2 , n 2 d } ) , and space, O ( nd ) , complexity of SVD prohibit its use for larger datasets (Shamir, 2015) including when X is a random variable.

Original: For larger datasets, stochastic, randomized, or sketching algorithms are better suited.
Revised: For larger datasets, stochastic, randomized, or sketching algorithms are better suited.

Original: Stochastic algorithms such as Oja’s algorithm (Oja, 1982; Allen-Zhu and Li, 2017) perform power iteration (Rutishauser, 1971) to iteratively improve an approximation, maintaining orthogonality of the eigenvectors typically through repeated QR decompositions.
Revised: Stochastic algorithms such as Oja’s algorithm (Oja, 1982; Allen-Zhu and Li, 2017) perform power iteration (Rutishauser, 1971) to iteratively improve an approximation, maintaining orthogonality of the eigenvectors typically through repeated QR decompositions.

Original: Alternatively, randomized algorithms (Halko et al. , 2011; Sarlos, 2006; Cohen et al. , 2017)
Revised: Alternatively, randomized algorithms (Halko et al. , 2011; Sarlos, 2006; Cohen et al. , 2017)

Original: ﬁrst compute a random projection of the data onto a ( k + p ) -subspace approximately containing the top- k subspace.
Revised: ﬁrst compute a random projection of the data onto a ( k + p ) -subspace approximately containing the top- k subspace.

Original: This is done using techniques similar to Krylov subspace iteration methods (Musco and Musco, 2015).
Revised: This is done using techniques similar to Krylov subspace iteration methods (Musco and Musco, 2015).

Original: After projecting, a call to SVD is then made on this reduced-dimensionality data matrix.
Revised: After projecting, a call to SVD is then made on this reduced-dimensionality data matrix.

Original: Sketching algorithms (Feldman et al. , 2020) such as Frequent Directions (Ghashami et al. , 2016) also target learning the top- k subspace by maintaining an overcomplete sketch matrix of size ( k + p ) ×
Revised: Sketching algorithms (Feldman et al. , 2020) such as Frequent Directions (Ghashami et al. , 2016) also target learning the top- k subspace by maintaining an overcomplete sketch matrix of size ( k + p ) ×

Original: d and maintaining a span of the top subspace with repeated calls to SVD.
Revised: d and maintaining a span of the top subspace with repeated calls to SVD.

Original: In both the randomized and sketching approaches, a ﬁnal SVD of the n × ( k + p ) dataset is required to recover the desired singular vectors.
Revised: In both the randomized and sketching approaches, a ﬁnal SVD of the n × ( k + p ) dataset is required to recover the desired singular vectors.

Original: Although the
Revised: Although the

Original: SVD scales linearly in n , some datasets are too large to ﬁt in memory; in this case, an out-of-memory
Revised: SVD scales linearly in n , some datasets are too large to ﬁt in memory; in this case, an out-of-memory

Original: SVD may sufﬁce (Haidar et al. , 2017).
Revised: SVD may sufﬁce (Haidar et al. , 2017).

Original: For this reason, the direct approach of stochastic algorithms, which avoid an SVD call altogether, is appealing when processing very large datasets.
Revised: For this reason, the direct approach of stochastic algorithms, which avoid an SVD call altogether, is appealing when processing very large datasets.

Original: [-2: ˆ v i ← ˆ v 
-]
Revised: [-2: ˆ v i ← ˆ v 
-][Table]

Original: i for all i 3: for t = 1 : T do 4: parfor i = 1 : k do 5: rewards ← 1 n (cid:48) X (cid:62) t X t ˆ v i 6: penalties ←n (cid:48) (cid:80)
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: <i (cid:104) X t
Revised: ˆ v i ←

Original: [Table]
Revised: [Table]

Original: vi , X
Revised: v

Original: [Table]
Revised: [Table]

Original: v
Revised: for all i 3: for t = 1 : T do 4: parfor i = [Table] : k do 5: rewards ←

Original: [Table]
Revised: [Table]

Original: (cid:105)
Revised: (cid:48)

Original: [Table]
Revised: [Table]

Original:  j 7: ˜ ∇ µi ← rewards − penalties 8: ˜ ∇ µ,Ri ← ˜ ∇ µi − (cid:104)
Revised: v i 6: penalties ←n (cid:48) (cid:80) j<i (cid:104) X t ˆ v i , X t ˆ v j (cid:105) ˆ v j 7: ˜ ∇ µi ← rewards − penalties 8: ˜ ∇ µ,Ri ← ˜ ∇ µi − (cid:104)

Original: ˜ ∇ µi , ˆ v i (cid:105) ˆ v i 9: ˆ v (cid:48)
Revised: ˜ ∇ µi , ˆ v i (cid:105) ˆ v i 9: ˆ v (cid:48)

Original: i ← ˆ v i + η t ˜ ∇ µ,Ri 10: ˆ v
Revised: i ← ˆ v i + η t ˜ ∇ µ,Ri 10: ˆ v

Original: i ← ˆ v (cid:48)
Revised: i ← ˆ v (cid:48)

Original: i ||
Revised: i ||

Original: ˆ v (cid:48)
Revised: ˆ v (cid:48)

Original: i || 11: end parfor 12: end for 13: return all ˆ v
Revised: i || 11: end parfor 12: end for 13: return all ˆ v

Original: i
Revised: i

Original: A large literature on distributed approaches to
Revised: A large literature on distributed approaches to

Original: PCA exists (Liang et al. , 2014;
Revised: PCA exists (Liang et al. , 2014;

Original: Garber et al. , 2017; Fan et al. , 2019).
Revised: Garber et al. , 2017; Fan et al. , 2019).

Original: These typically follow the pattern of computing solutions locally and then aggregating them in a single round (or minimal rounds) of communication.
Revised: These typically follow the pattern of computing solutions locally and then aggregating them in a single round (or minimal rounds) of communication.

Original: The modern distributed machine learning setting which has evolved to meet the needs of deep learning is fundamentally different.
Revised: The modern distributed machine learning setting which has evolved to meet the needs of deep learning is fundamentally different.

Original: Many accelerators joined with fast interconnects means the cost of communication is low compared to the cost of a single update step, however existing approaches to distributed PCA cannot take full advantage of this.
Revised: Many accelerators joined with fast interconnects means the cost of communication is low compared to the cost of a single update step, however existing approaches to distributed PCA cannot take full advantage of this.

Original: Notation : We follow the same notation as Gemp et al.
Revised: Notation : We follow the same notation as Gemp et al.

Original: Variables returned by an approximation algorithm are distinguished from the true solutions with hats, e.g., the column-wise matrix of eigenvectors ˆ V approximates V .
Revised: Variables returned by an approximation algorithm are distinguished from the true solutions with hats, e.g., the column-wise matrix of eigenvectors ˆ V approximates V .

Original: We order the columns of V such that the i th column, v i , is the eigenvector with the i th largest eigenvalue λ i .
Revised: We order the columns of V such that the i th column, v i , is the eigenvector with the i th largest eigenvalue λ i .

Original: The set of all eigenvectors { v j } with λ j larger than λ i , namely v i ’s parents , will be denoted by v j<i .
Revised: The set of all eigenvectors { v j } with λ j larger than λ i , namely v i ’s parents , will be denoted by v j<i .

Original: Similarly, sums over subsets of indices may be abbreviated as (cid:80) j<i
Revised: Similarly, sums over subsets of indices may be abbreviated as (cid:80) j<i

Original: = (cid:80) i − 1 j =1 .
Revised: = (cid:80) i − 1 j =1 .

Original: The set of all parents and children of v i are denoted by v − i .
Revised: The set of all parents and children of v i are denoted by v − i .

Original: Let the i th
Revised: Let the i th

Original: eigengap g i
Revised: eigengap g i

Original: =
Revised: =

Original: λ
Revised: λ

Original: i − λ
Revised: i − λ

Original: i +1 .
Revised: i +1 .

Original: We assume the standard Euclidean inner product (cid:104) u, v (cid:105) =
Revised: We assume the standard Euclidean inner product (cid:104) u, v (cid:105) =

Original: u (cid:62) v and denote the unit-sphere andsimplex in ambientspace R d with S d − 1 and ∆ d − 1 respectively. α -EigenGame.
Revised: u (cid:62) v and denote the unit-sphere and ambient α -EigenGame.

Original: We build on the algorithm introduced by Gemp et al.
Revised: We build on the algorithm introduced by Gemp et al.

Original: (2021), which we refer to here as α -EigenGame.
Revised: (2021), which we refer to here as α -EigenGame.

Original: This algorithm is derived by formulating the eigendecomposition of a symmetric positive deﬁnite matrix as the Nash equilibrium of a game among k players, each player i owning the approximate eigenvector ˆ v i ∈ S
Revised: This algorithm is derived by formulating the eigendecomposition of a symmetric positive deﬁnite matrix as the Nash equilibrium of a game among k players, each player i owning the approximate eigenvector ˆ v i ∈ S

Original: d − 1 .
Revised: d − 1 .

Original: Each player is also assigned a utility function, u αi (ˆ v i | ˆ v j<i ) , that they must maximize:
Revised: Each player is also assigned a utility function, u αi (ˆ v i | ˆ v j<i ) , that they must maximize:

Original: [Equation]
Revised: [Equation]

Original: These utilities balance two terms, one that rewards a ˆ v i that captures more variance in the data and a second term that penalizes ˆ v i for failing to be orthogonal to each of its parents ˆ v j<i (these terms are indicated with Var and Align -penalty in equation (1)).
Revised: These utilities balance two terms, one that rewards a ˆ v i that captures more variance in the data and a second term that penalizes ˆ v i for failing to be orthogonal to each of its parents ˆ v j<i (these terms are indicated with Var and Align -penalty in equation (1)).

Original: In α -EigenGame, each player simultaneously updates ˆ v i with gradient ascent, and it is shown that this process converges to the
Revised: In α -EigenGame, each player simultaneously updates ˆ v i with gradient ascent, and it is shown that this process converges to the

Original: Nash equilibrium.
Revised: Nash equilibrium.

Original: We are interested in extending this approach to the data parallel setting where each player i may distribute its update computation over multiple devices.
Revised: We are interested in extending this approach to the data parallel setting where each player i may distribute its update computation over multiple devices.

Original:  A SCALABLE UNBIASED ALGORITHM
Revised:  A SCALABLE UNBIASED ALGORITHM

Original: We present our novel modiﬁcation to α -EigenGame called µ -EigenGame along with intuition, theory, and empirical support for critical lemmas.
Revised: We present our novel modiﬁcation to α -EigenGame called µ -EigenGame along with intuition, theory, and empirical support for critical lemmas.

Original: We begin with identifying and systematically removing the bias that exists in the α -EigenGame updates.
Revised: We begin with identifying and systematically removing the bias that exists in the α -EigenGame updates.

Original: We then explain how removing bias allows us to exploit modern compute architectures culminating in the development of a highly parallelizable algorithm.
Revised: We then explain how removing bias allows us to exploit modern compute architectures culminating in the development of a highly parallelizable algorithm.

Original: 3. α -E IGEN G AME ’ S BIASED UPDATES
Revised: 3. α -E IGEN G AME ’ S BIASED UPDATES

Original: Consider partitioning the sample covariance matrix C t into a sum of m matrices as C t = 1 n (cid:48)
Revised: Consider partitioning the sample covariance matrix C t into a sum of m matrices as C t = 1 n (cid:48)

Original: X (cid:62) t X t =m (cid:80) m mn (cid:48)
Revised: X (cid:62) t X t =m (cid:80) m mn (cid:48)

Original: X (cid:62) tm X tm = 1 m (cid:80) m C tm .
Revised: X (cid:62) tm X tm = 1 m (cid:80) m C tm .

Original: For sake of exposition, we drop the additional subscript t on
Revised: For sake of exposition, we drop the additional subscript t on

Original: C in what follows.
Revised: C in what follows.

Original: We would like α -EigenGame to parallelize over these partitions.
Revised: We would like α -EigenGame to parallelize over these partitions.

Original: However, the gradient of u αi with respect to ˆ v i does not decompose cleanly over the data partitions:
Revised: However, the gradient of u αi with respect to ˆ v i does not decompose cleanly over the data partitions:

Original: [Equation]
Revised: [Equation]

Original: We include the superscript α on the EigenGame gradient to differentiate it from the µ -EigenGame direction later.
Revised: We include the superscript α on the EigenGame gradient to differentiate it from the µ -EigenGame direction later.

Original: The nonlinear appearance of C in the penalty terms makes obtaining an unbiased gradient difﬁcult.
Revised: The nonlinear appearance of C in the penalty terms makes obtaining an unbiased gradient difﬁcult.

Original: The quadratic term in the numerator of equation (2) could be made unbiased by using two sample estimates of C , one for each term.
Revised: The quadratic term in the numerator of equation (2) could be made unbiased by using two sample estimates of C , one for each term.

Original: But the appearance of the term in the denominator does not have an easy solution.
Revised: But the appearance of the term in the denominator does not have an easy solution.

Original: C m is likely singular for small n (cid:48)
Revised: C m is likely singular for small n (cid:48)

Original: ( n (cid:48) < d ) which increases the likelihood of a small denominator, i.e., a large penalty coefﬁcient (boxed), if we were to estimate the denominator with samples.
Revised: ( n (cid:48) < d ) which increases the likelihood of a small denominator, i.e., a large penalty coefﬁcient (boxed), if we were to estimate the denominator with samples.

Original: The result is an update that emphasizes penalizing orthogonality over capturing data variance.
Revised: The result is an update that emphasizes penalizing orthogonality over capturing data variance.

Original: Techniques exist to reduce the bias of samples of ratios of random variables, but to our knowledge, techniques to obtain unbiased estimates are not available.
Revised: Techniques exist to reduce the bias of samples of ratios of random variables, but to our knowledge, techniques to obtain unbiased estimates are not available.

Original: This was conjectured by Gemp et al.
Revised: This was conjectured by Gemp et al.

Original: as the reason for why α -EigenGame performed worse with small minibatches.
Revised: as the reason for why α -EigenGame performed worse with small minibatches.

Original: 3. R EMOVING α -E IGEN G AME ’ S BIAS
Revised: 3. R EMOVING α -E IGEN G AME ’ S BIAS

Original: It is helpful to rearrange equation (2) to shift perspective from estimating a penalty coefﬁcient (in red) to estimating a penalty direction (in blue):
Revised: It is helpful to rearrange equation (2) to shift perspective from estimating a penalty coefﬁcient (in red) to estimating a penalty direction (in blue):

Original: [Equation]
Revised: [Equation]

Original: The penalty direction in equation (3) is still difﬁcult to estimate.
Revised: The penalty direction in equation (3) is still difﬁcult to estimate.

Original: However, consider the case where ˆ v j is any eigenvector of C with associated (unknown) eigenvalue λ (cid:48) .
Revised: However, consider the case where ˆ v j is any eigenvector of C with associated (unknown) eigenvalue λ (cid:48) .

Original: In this case, C ˆ v j = λ (cid:48)
Revised: In this case, C ˆ v j = λ (cid:48)

Original: ˆ v j and the penalty direction (in blue) simpliﬁes to ˆ v j because || ˆ v j || = 1 .
Revised: ˆ v j and the penalty direction (in blue) simpliﬁes to ˆ v j because || ˆ v j || = 1 .

Original: While this assumption is certainly not met at initialization, α -EigenGame leads each ˆ v j towards v j , so we can expect this assumption to be met asymptotically.
Revised: While this assumption is certainly not met at initialization, α -EigenGame leads each ˆ v j towards v j , so we can expect this assumption to be met asymptotically.

Original: This intuition motivates the following µ -EigenGame update direction for ˆ v i with inexact parents ˆ v j (compare orange in equation (4) to blue in equation (3)):∆
Revised: This intuition motivates the following µ -EigenGame update direction for ˆ v i with inexact parents ˆ v j (compare orange in equation (4) to blue in equation (3)):

Original: [Equation]
Revised: [Equation]

Original: We use ∆ instead of ∇ because the direction is not a gradient (discussed later).
Revised: We use ∆ instead of ∇ because the direction is not a gradient (discussed later).

Original: Notice how the strictly linear appearance of C in µ -EigenGame allows the update to easily decompose over the data partitions in equation (4).
Revised: Notice how the strictly linear appearance of C in µ -EigenGame allows the update to easily decompose over the data partitions in equation (4).

Original: The µ -EigenGame update satisﬁes two important properties.
Revised: The µ -EigenGame update satisﬁes two important properties.

Original: Lemma 1 (Asymptotic equivalence) .
Revised: Lemma 1 (Asymptotic equivalence) .

Original: The µ -EigenGame direction, ∆ µi , with exact parents ( ˆ v j = v j ∀ j < i ) is equivalent to α -EigenGame.
Revised: The µ -EigenGame direction, ∆ µi , with exact parents ( ˆ v j = v j ∀ j < i ) is equivalent to α -EigenGame.

Original: Proof.
Revised: Proof.

Original: We start with α -EigenGame and add a superscript e to its gradient to emphasize this is the gradient computed with exact parents ( ˆ v j = v j ).
Revised: We start with α -EigenGame and add a superscript e to its gradient to emphasize this is the gradient computed with exact parents ( ˆ v j = v j ).

Original: Then simplifying, we ﬁnd ∇ α,ei
Revised: Then simplifying, we ﬁnd ∇ α,ei

Original: [Equation]
Revised: [Equation]

Original: Therefore, once the ﬁrst ( i − 1) eigenvectors are learned, learning the i th eigenvector with µEigenGame is equivalent to learning with α -EigenGame.
Revised: Therefore, once the ﬁrst ( i − 1) eigenvectors are learned, learning the i th eigenvector with µEigenGame is equivalent to learning with α -EigenGame.

Original: Lemma 2 (Zero bias) .
Revised: Lemma 2 (Zero bias) .

Original: Unbiased estimates of ∆ µi can be obtained with samples from p ( X ) .
Revised: Unbiased estimates of ∆ µi can be obtained with samples from p ( X ) .

Original: Proof.
Revised: Proof.

Original: Let X ∼ p
Revised: Let X ∼ p

Original: ( X ) where X ∈ R d and p ( X ) is the uniform distribution over the dataset.
Revised: ( X ) where X ∈ R d and p ( X ) is the uniform distribution over the dataset.

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: where all expectations are with respect to p ( X ) .
Revised: where all expectations are with respect to p ( X ) .

Original: These two lemmas provide the foundation for a performant algorithm.
Revised: These two lemmas provide the foundation for a performant algorithm.

Original: The ﬁrst enables convergence to the desired solution, while the second facilitates scaling to larger datasets.
Revised: The ﬁrst enables convergence to the desired solution, while the second facilitates scaling to larger datasets.

Original: Algorithm 1 presents pseudocode for µ -EigenGame where computation is parallelized over the k players.
Revised: Algorithm 1 presents pseudocode for µ -EigenGame where computation is parallelized over the k players.

Original: 3. M ODEL AND DATA PARALLELISM
Revised: 3. M ODEL AND DATA PARALLELISM

Original: [Figure]
Revised: [Figure]

Original: In our setting we have a number of connected devices.
Revised: In our setting we have a number of connected devices.

Original: Speciﬁcally we consider the parallel framework speciﬁed by TPUv3 available in Google Cloud, however our setup is applicable to any multi-host, multi-device system.
Revised: Speciﬁcally we consider the parallel framework speciﬁed by TPUv3 available in Google Cloud, however our setup is applicable to any multi-host, multi-device system.

Original: The α -EigenGame formulation (Gemp et al. , 2021) considers an extreme form of model parallelism (Figure 2a) where each device has its own unique set of eigenvectors.
Revised: The α -EigenGame formulation (Gemp et al. , 2021) considers an extreme form of model parallelism (Figure 2a) where each device has its own unique set of eigenvectors.

Original: In this work we further consider a different form of model and data parallelism which is directly enabled by having unbiased updates (Figure 2b).
Revised: In this work we further consider a different form of model and data parallelism which is directly enabled by having unbiased updates (Figure 2b).

Original: This enables µ -EigenGame to deal with both high-dimensional problems as well as massive sample sizes.
Revised: This enables µ -EigenGame to deal with both high-dimensional problems as well as massive sample sizes.

Original: Here each set of eigenvectors is copied on M devices.
Revised: Here each set of eigenvectors is copied on M devices.

Original: Update directions are computed on each device individually using a different data stream and then combined by summing or averaging.
Revised: Update directions are computed on each device individually using a different data stream and then combined by summing or averaging.

Original: Updates are applied to a single copy and this is duplicated across the M − 1 remaining devices.
Revised: Updates are applied to a single copy and this is duplicated across the M − 1 remaining devices.

Original: In this way, updates are computed using an M × larger effective batch size while still allowing device-wise model parallelism.
Revised: In this way, updates are computed using an M × larger effective batch size while still allowing device-wise model parallelism.

Original: This setting is particularly useful when the number of samples is very large.
Revised: This setting is particularly useful when the number of samples is very large.

Original: This form of parallelism is not possible using the original EigenGame formulation since it relies on combining unbiased updates.
Revised: This form of parallelism is not possible using the original EigenGame formulation since it relies on combining unbiased updates.

Original: In this sense, the parallelism discussed in this work generalizes that introduced by Gemp et al.
Revised: In this sense, the parallelism discussed in this work generalizes that introduced by Gemp et al.

Original: Note that we also allow for within-device parallelism.
Revised: Note that we also allow for within-device parallelism.

Original: That is, each v i in Figure 2 is a contiguous collection of eigenvectors which are updated independently, in parallel, on a given device (for example using vmap in Jax).
Revised: That is, each v i in Figure 2 is a contiguous collection of eigenvectors which are updated independently, in parallel, on a given device (for example using vmap in Jax).

Original: We provide pseudocode in Algorithm 2 in the appendix which simply augments Algorithm 1 with an additional parallelized for-loop and aggregation step over available devices.
Revised: We provide pseudocode in Algorithm 2 in the appendix which simply augments Algorithm 1 with an additional parallelized for-loop and aggregation step over available devices.

Original: We also provide detailed Jax pseudo-code for parallel µ -EigenGame in Appendix F.
Revised: We also provide detailed Jax pseudo-code for parallel µ -EigenGame in Appendix F.

Original: We compare the empirical scaling performance of µ -EigenGame against α -EigenGame on a 14 billion sample dataset in section 5.
Revised: We compare the empirical scaling performance of µ -EigenGame against α -EigenGame on a 14 billion sample dataset in section 5.

Original:  SVD AS THE SOLUTION TO A NEW E IGEN G AME
Revised:  SVD AS THE SOLUTION TO A NEW E IGEN G AME

Original: We theoretically examine the µ -EigenGame algorithm and 1) prove that, using only minibatches of data, µ -EigenGame converges globally to the true eigenvectors, which 2) comprise the Nash equilibrium of a novel game formulation we recover through deriving pseudo -utility functions from update rules.
Revised: We theoretically examine the µ -EigenGame algorithm and 1) prove that, using only minibatches of data, µ -EigenGame converges globally to the true eigenvectors, which 2) comprise the Nash equilibrium of a novel game formulation we recover through deriving pseudo -utility functions from update rules.

Original: Beyond proving speciﬁc theoretical properties of µ -EigenGame, we believe these proof techniques may be of wider interest to the community.
Revised: Beyond proving speciﬁc theoretical properties of µ -EigenGame, we believe these proof techniques may be of wider interest to the community.

Original: 4. C ONVERGENCE TO SVD
Revised: 4. C ONVERGENCE TO SVD

Original: The asymptotic equivalence of µ -EigenGame to α -EigenGame ensures µ -EigenGame is globally, asymptotically convergent and its unbiased updates ensure it is scalable.
Revised: The asymptotic equivalence of µ -EigenGame to α -EigenGame ensures µ -EigenGame is globally, asymptotically convergent and its unbiased updates ensure it is scalable.

Original: Proof in appendix C.
Revised: Proof in appendix C.

Original: Theorem 1 (Global convergence) .
Revised: Theorem 1 (Global convergence) .

Original: Given a positive deﬁnite covariance matrix C with the top- k eigengaps positive and a square-summable, not summable step size sequence η t (e.g., 1 /t ), Algorithmconverges to the top- k eigenvectors asymptotically ( lim T →∞ ) with probability 1 .
Revised: Given a positive deﬁnite covariance matrix C with the top- k eigengaps positive and a square-summable, not summable step size sequence η t (e.g., 1 /t ), Algorithmconverges to the top- k eigenvectors asymptotically ( lim T →∞ ) with probability 1 .

Original: This stochastic asymptotic convergence result is complimentary to the deterministic (full-batch) ﬁnite-sample result in Gemp et al.
Revised: This stochastic asymptotic convergence result is complimentary to the deterministic (full-batch) ﬁnite-sample result in Gemp et al.

Original: (2021) where each ˆ v i is learned in sequence.
Revised: (2021) where each ˆ v i is learned in sequence.

Original: In contrast, the proof above applies when learning all ˆ v i in parallel.
Revised: In contrast, the proof above applies when learning all ˆ v i in parallel.

Original: We leave ﬁnite-sample convergence to future work (Durmus et al. , 2020).
Revised: We leave ﬁnite-sample convergence to future work (Durmus et al. , 2020).

Original: 4. SVD IS N ASH OF µ -E IGEN G AME
Revised: 4. SVD IS N ASH OF µ -E IGEN G AME

Original: We arrived at µ -EigenGame by analyzing and improving properties of the α -EigenGame update.
Revised: We arrived at µ -EigenGame by analyzing and improving properties of the α -EigenGame update.

Original: However, the µ -EigenGame update direction is linear in each ˆ v i .
Revised: However, the µ -EigenGame update direction is linear in each ˆ v i .

Original: This suggests we may be able to design a pseudo -utility function for it.
Revised: This suggests we may be able to design a pseudo -utility function for it.

Original: Rearranging the update direction from equation (4) as
Revised: Rearranging the update direction from equation (4) as

Original: [Equation]
Revised: [Equation]

Original: reveals that we can reverse-engineer the following utility function
Revised: reveals that we can reverse-engineer the following utility function

Original: [Equation]
Revised: [Equation]

Original: where is the stop gradient operator commonly used in deep learning packages.
Revised: where is the stop gradient operator commonly used in deep learning packages.

Original: As the name implies, stops gradients from ﬂowing through its argument so that equation (8) appears linear in ˆ v i instead of quadratic when differentiating the expression.
Revised: As the name implies, stops gradients from ﬂowing through its argument so that equation (8) appears linear in ˆ v i instead of quadratic when differentiating the expression.

Original: In light of this, we have renamed ∆ µi to ˜ ∇ µi to emphasize that it is a pseudo -gradient of u µi .
Revised: In light of this, we have renamed ∆ µi to ˜ ∇ µi to emphasize that it is a pseudo -gradient of u µi .

Original: Note that without the stop gradient, the true gradient of u µi would be [ A + A (cid:62) ]
Revised: Note that without the stop gradient, the true gradient of u µi would be [ A + A (cid:62) ]

Original: ˆ v i rather than A ˆ v i
Revised: ˆ v i rather than A ˆ v i

Original: where A
Revised: where A

Original: =
Revised: =

Original: [ I − (cid:80)
Revised: [ I − (cid:80)

Original: j<
Revised: j<

Original: i ˆ v (cid:62)
Revised: i ˆ v (cid:62)

Original: j ˆ v j ] C .
Revised: j ˆ v j ] C .

Original: We analyze this alternative in Appendix I.1 and ﬁnd it, interestingly, to perform worse than µ -EigenGame empirically.
Revised: We analyze this alternative in Appendix I.1 and ﬁnd it, interestingly, to perform worse than µ -EigenGame empirically.

Original: The utility function u µi has an intuitive meaning.
Revised: The utility function u µi has an intuitive meaning.

Original: It is the Rayleigh quotient for the matrix C i
Revised: It is the Rayleigh quotient for the matrix C i

Original: =
Revised: =

Original: [ I − (cid:80)
Revised: [ I − (cid:80)

Original: j<
Revised: j<

Original: i ˆ v (cid:62) ˆ v j ] C , which gives the covariance after the subspace spanned by ˆ v j<i has been removed.
Revised: i ˆ v (cid:62) ˆ v j ] C , which gives the covariance after the subspace spanned by ˆ v j<i has been removed.

Original: In other words, player i is directed to ﬁnd the largest eigenvalue in the orthogonal complement of the approximate top-
Revised: In other words, player i is directed to ﬁnd the largest eigenvalue in the orthogonal complement of the approximate top-

Original: ( i − 1) subspace.
Revised: ( i − 1) subspace.

Original: This approach is known as “deﬂating" the matrix C .
Revised: This approach is known as “deﬂating" the matrix C .

Original: Figure 1b illustrates µ -EigenGame’s reduced bias when estimating the new utility function (and resulting optimum) from an average over minibatches.
Revised: Figure 1b illustrates µ -EigenGame’s reduced bias when estimating the new utility function (and resulting optimum) from an average over minibatches.

Original: Deﬁnition 1 ( µ -Eigen Game ) .
Revised: Deﬁnition 1 ( µ -Eigen Game ) .

Original: Let µ -EigenGame be the game with players i ∈ { 1 , . .
Revised: Let µ -EigenGame be the game with players i ∈ { 1 , . .

Original: . , k } , their respective strategy spaces ˆ v i ∈ S d − 1 , and their corresponding utilities u µi as deﬁned in equation (8).
Revised: . , k } , their respective strategy spaces ˆ v i ∈ S d − 1 , and their corresponding utilities u µi as deﬁned in equation (8).

Original: Theorem 2.
Revised: Theorem 2.

Original: Top-
Revised: Top-

Original: k SVD is the unique Nash of µ -EigenGame given symmetric C with the top- k eigengaps positive.
Revised: k SVD is the unique Nash of µ -EigenGame given symmetric C with the top- k eigengaps positive.

Original: Proof.
Revised: Proof.

Original: We will show by induction that each v i is the unique best response to v − i , which implies they constitute the unique Nash equilibrium.
Revised: We will show by induction that each v i is the unique best response to v − i , which implies they constitute the unique Nash equilibrium.

Original: First, consider player 1 ’s utility.
Revised: First, consider player 1 ’s utility.

Original: It is the Rayleigh quotient of C because ˆ v 1 is constrained to the unit-sphere, i.e., u µ 

Revised: It is the Rayleigh quotient of C because ˆ v 1 is constrained to the unit-sphere, i.e., u µ

Original: = ˆ v (cid:62) 1 C ˆ v 1 =
Revised: = ˆ v (cid:62) 1 C ˆ v 1 =

Original: ˆ v (cid:62) 1 C ˆ v 1 ˆ v (cid:62) 1 ˆ v 1 .
Revised: ˆ v (cid:62) 1 C ˆ v 1 ˆ v (cid:62) 1 ˆ v 1 .

Original: Therefore, we know v 1 maximizes u µ 

Revised: Therefore, we know v 1 maximizes u µ

Original: and the maximizer is unique because its eigengap g 1 > 0 .
Revised: and the maximizer is unique because its eigengap g 1 > 0 .

Original: In game theory parlance, v 1 is a best response to v − 1 .
Revised: In game theory parlance, v 1 is a best response to v − 1 .

Original: The proof continues by induction.
Revised: The proof continues by induction.

Original: The utility of player i is u µ i
Revised: The utility of player i is u µ i

Original: = ˆ v (cid:62) i
Revised: = ˆ v (cid:62) i

Original: [ I − (cid:80)
Revised: [ I − (cid:80)

Original: j<i v j v (cid:62) j ] C ˆ v i , which is the Rayleigh quotient with the subspace spanned by the top ( i − 1) eigenvectors removed.
Revised: j<i v j v (cid:62) j ] C ˆ v i , which is the Rayleigh quotient with the subspace spanned by the top ( i − 1) eigenvectors removed.

Original: Therefore, the maximizer of u µi is the largest eigenvector in the remaining subspace, i.e., v i .
Revised: Therefore, the maximizer of u µi is the largest eigenvector in the remaining subspace, i.e., v i .

Original: As before, g i > 0 , so this maximizer is unique.
Revised: As before, g i > 0 , so this maximizer is unique.

Original: This shows that each v i is the unique best response to v − i , therefore, the set of v i forms the unique Nash.
Revised: This shows that each v i is the unique best response to v − i , therefore, the set of v i forms the unique Nash.

Original: Notice how the induction proof of Theorem 2 relies on a) the hierarchy of vectors ( v 1 does not depend on v − 1 ) and b) the fact that u µ i need only be a sensible utility when all player
Revised: Notice how the induction proof of Theorem 2 relies on a) the hierarchy of vectors ( v 1 does not depend on v − 1 ) and b) the fact that u µ i need only be a sensible utility when all player

Original: i ’s parents
Revised: i ’s parents

Original: [Figure]
Revised: [Figure]

Original: The Nash property is important because it enables the use of any black-box procedure for computing best responses.
Revised: The Nash property is important because it enables the use of any black-box procedure for computing best responses.

Original: Like prior work, we develop a gradient method for optimizing each utility, however, that is not a requirement.
Revised: Like prior work, we develop a gradient method for optimizing each utility, however, that is not a requirement.

Original: Any approach sufﬁces if it can efﬁciently compute a best response.
Revised: Any approach sufﬁces if it can efﬁciently compute a best response.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: As in EigenGame, we omit the projection of gradients onto the tangent space of the sphere; speciﬁcally, we omit line 8 in Algorithm 1.
Revised: As in EigenGame, we omit the projection of gradients onto the tangent space of the sphere; speciﬁcally, we omit line 8 in Algorithm 1.

Original: As discussed in Gemp et al.
Revised: As discussed in Gemp et al.

Original: (2021), this has the effect of intelligently adapting the step size to use smaller learning rates near the ﬁxed point.
Revised: (2021), this has the effect of intelligently adapting the step size to use smaller learning rates near the ﬁxed point.

Original: To ease comparison with previous work, we count the longest correct eigenvector streak as introduced by Gemp et al.
Revised: To ease comparison with previous work, we count the longest correct eigenvector streak as introduced by Gemp et al.

Original: (2021), which measures the number of eigenvectors that have been learned, in order, to within an angular threshold (e.g., π/ 8 ) of the true eigenvectors.
Revised: (2021), which measures the number of eigenvectors that have been learned, in order, to within an angular threshold (e.g., π/ 8 ) of the true eigenvectors.

Original: We also measure how well the set of ˆ v
Revised: We also measure how well the set of ˆ v

Original: i captures the top- k subspace with a normalized subspace distance: 1 − 1 /k ·
Revised: i captures the top- k subspace with a normalized subspace distance: 1 − 1 /k ·

Original: Tr( U ∗ P ) ∈ [0 , 1] where U ∗ =
Revised: Tr( U ∗ P ) ∈ [0 , 1] where U ∗ =

Original: V V † and P =
Revised: V V † and P =

Original: ˆ V ˆ V † (Tang, 2019).
Revised: ˆ V ˆ V † (Tang, 2019).

Original: We provide additional experiments in Appendix A.
Revised: We provide additional experiments in Appendix A.

Original: MNIST.
Revised: MNIST.

Original: We compare µ -EigenGame against α -EigenGame, GHA (Sanger, 1989), Matrix Krasulina (Tang, 2019), and Oja’s algorithm (Allen-Zhu and Li, 2017) on the M NIST dataset.
Revised: We compare µ -EigenGame against α -EigenGame, GHA (Sanger, 1989), Matrix Krasulina (Tang, 2019), and Oja’s algorithm (Allen-Zhu and Li, 2017) on the M NIST dataset.

Original: We ﬂatten each image in the training set to obtain a 60 , 000 × 784 dimensional matrix X .
Revised: We ﬂatten each image in the training set to obtain a 60 , 000 × 784 dimensional matrix X .

Original: Figure 3 demonstrates µ -EigenGame’s robustness to minibatch size.
Revised: Figure 3 demonstrates µ -EigenGame’s robustness to minibatch size.

Original: It performs best in the longest streak metric and better than α -EigenGame in subspace distance.
Revised: It performs best in the longest streak metric and better than α -EigenGame in subspace distance.

Original: We attribute this improvement to its unbiased updates and additional acceleration effects which we discuss in detail in section I.2.
Revised: We attribute this improvement to its unbiased updates and additional acceleration effects which we discuss in detail in section I.2.

Original: Meena conversational model.
Revised: Meena conversational model.

Original: This dataset consists a subset of the 40 billion words used to train the transformer-based Meena language model (Adiwardana et al. , 2020).
Revised: This dataset consists a subset of the 40 billion words used to train the transformer-based Meena language model (Adiwardana et al. , 2020).

Original: The subset was preprocessed to remove duplicates and then embedded using the trained model.
Revised: The subset was preprocessed to remove duplicates and then embedded using the trained model.

Original: The dataset consists of n ≈ 14 billion embeddings each with dimensionality d = 2560 ; its total size is 131 TB.
Revised: The dataset consists of n ≈ 14 billion embeddings each with dimensionality d = 2560 ; its total size is 131 TB.

Original: Due to its moderate dimensionality we can exactly compute the ground truth solution by iteratively accumulating the covariance matrix of the data and computing its eigendecomposition.
Revised: Due to its moderate dimensionality we can exactly compute the ground truth solution by iteratively accumulating the covariance matrix of the data and computing its eigendecomposition.

Original: On a single machine this takes 1.5 days (but is embarrassingly parallelizable with MapReduce).
Revised: On a single machine this takes 1.5 days (but is embarrassingly parallelizable with MapReduce).

Original: We use minibatches of size 4,096 in each TPU.
Revised: We use minibatches of size 4,096 in each TPU.

Original: We do model parallelism across 4 TPUs so we see 16,384 samples per iteration.
Revised: We do model parallelism across 4 TPUs so we see 16,384 samples per iteration.

Original: We test two additional degrees of data parallelism with 4 × (16 TPUs, 65,536 samples) and 8 × (32 TPUs, 131,072 samples) the amount of data per iteration respectively.
Revised: We test two additional degrees of data parallelism with 4 × (16 TPUs, 65,536 samples) and 8 × (32 TPUs, 131,072 samples) the amount of data per iteration respectively.

Original: We compute and apply updates using SGD with a learning rate of 5 × 10 − 5 and Nesterov momentum with a factor of 0.9.
Revised: We compute and apply updates using SGD with a learning rate of 5 × 10 − 5 and Nesterov momentum with a factor of 0.9.

Original: [Figure]
Revised: [Figure]

Original: Figure 4a compares the mean performance of µ -EigenGame against α -EigenGame as a function of the degree of parallelism in computing the top k = 256 eigenvectors (standard errors computed over 5 random seeds).
Revised: Figure 4a compares the mean performance of µ -EigenGame against α -EigenGame as a function of the degree of parallelism in computing the top k = 256 eigenvectors (standard errors computed over 5 random seeds).

Original: Each TPU is tasked with learning 32 contiguous eigenvectors.
Revised: Each TPU is tasked with learning 32 contiguous eigenvectors.

Original: We see that increasing the degree of parallelism has no effect on the performance of α -EigenGame.
Revised: We see that increasing the degree of parallelism has no effect on the performance of α -EigenGame.

Original: As expected, it is unable to take advantage of the higher data throughput since its updates are biased and cannot be meaningfully linearly combined across copies.
Revised: As expected, it is unable to take advantage of the higher data throughput since its updates are biased and cannot be meaningfully linearly combined across copies.

Original: In contrast, the performance of µ -EigenGame scales with the effective batch size achieved through parallelism.
Revised: In contrast, the performance of µ -EigenGame scales with the effective batch size achieved through parallelism.

Original: µ -EigenGame ( 8 × ) is able to recovereigenvectors in less than 40,000 iterations in 2 hours 45 minutes (approximately 0.5 epochs).
Revised: µ -EigenGame ( 8 × ) is able to recovereigenvectors in less than 40,000 iterations in 2 hours 45 minutes (approximately 0.5 epochs).

Original: Spectral clustering on graphs.
Revised: Spectral clustering on graphs.

Original: We conducted an experiment on learning the eigenvectors of the graph Laplacian of a social network graph (Leskovec and McAuley, 2012) for the purpose of spectral clustering.
Revised: We conducted an experiment on learning the eigenvectors of the graph Laplacian of a social network graph (Leskovec and McAuley, 2012) for the purpose of spectral clustering.

Original: The eigenvalues of the graph Laplacian reveal several interesting properties as well such as the number of connected components, an approximation to the sparsest cut, and the diameter of a connected graph (Chung et al. , 1994).
Revised: The eigenvalues of the graph Laplacian reveal several interesting properties as well such as the number of connected components, an approximation to the sparsest cut, and the diameter of a connected graph (Chung et al. , 1994).

Original: Given a graph with a set of nodes V and set of edges E , the graph Laplacian can be written as
Revised: Given a graph with a set of nodes V and set of edges E , the graph Laplacian can be written as

Original: L = X
Revised: L = X

Original: (cid:62) X where each row of the incidence matrix X ∈ R |E|×|V| represents a distinct edge;
Revised: (cid:62) X where each row of the incidence matrix X ∈ R |E|×|V| represents a distinct edge;

Original: X e =( i,j ) ∈E is a vector containing only 2 nonzero entries, a 1 at index i and a − 1 at index j
Revised: X e =( i,j ) ∈E is a vector containing only 2 nonzero entries, a 1 at index i and a − 1 at index j

Original: (Horaud, 2009).
Revised: (Horaud, 2009).

Original: In this setting, the eigenvectors of primary interest are the bottom- k ( λ |V| , λ |V|− 1 , . . . )
Revised: In this setting, the eigenvectors of primary interest are the bottom- k ( λ |V| , λ |V|− 1 , . . . )

Original: rather than the top- k ( λ 1 , λ 2 , . . . )
Revised: rather than the top- k ( λ 1 , λ 2 , . . . )

Original: , however, a simple algebraic manipulation allows us to reuse a topk solver.
Revised: , however, a simple algebraic manipulation allows us to reuse a topk solver.

Original: By deﬁning the matrix L − = λ ∗ I − L with λ ∗ > λ 1 , we ensure L − (cid:31) 

Revised: By deﬁning the matrix L − = λ ∗ I − L with λ ∗ > λ 1 , we ensure L − (cid:31)

Original: and the top- k eigenvectors of L − are the bottom- k of L .
Revised: and the top- k eigenvectors of L − are the bottom- k of L .

Original: The update in equation (4) is transformed into ˜ ∇ µi =
Revised: The update in equation (4) is transformed into ˜ ∇ µi =

Original: ( λ ∗ I − L )
Revised: ( λ ∗ I − L )

Original: ˆ v i − (cid:80)
Revised: ˆ v i − (cid:80)

Original: j<i (cid:0) ˆ v (cid:62) i
Revised: j<i (cid:0) ˆ v (cid:62) i

Original: ( λ ∗ I − L )ˆ v j (cid:1) ˆ v j .
Revised: ( λ ∗ I − L )ˆ v j (cid:1) ˆ v j .

Original: We provide efﬁcient pseudo-code in Appendix H.
Revised: We provide efﬁcient pseudo-code in Appendix H.

Original: The Facebook graph consists of 134 , 833 nodes, 1 , 380 , 293 edges, and 8 connected components, each formed by a set of Facebook pages belonging to a distinct category, e.g., Government, TV shows, etc.
Revised: The Facebook graph consists of 134 , 833 nodes, 1 , 380 , 293 edges, and 8 connected components, each formed by a set of Facebook pages belonging to a distinct category, e.g., Government, TV shows, etc.

Original: (Leskovec and Krevl, 2014; Rozemberczki et al. , 2019).
Revised: (Leskovec and Krevl, 2014; Rozemberczki et al. , 2019).

Original: We add a single edge between every pair of components to create a connected graph.
Revised: We add a single edge between every pair of components to create a connected graph.

Original: By projecting this graph onto the bottomeigenvectors of the graph Laplacian using µ -EG ( M = 1 , n (cid:48)
Revised: By projecting this graph onto the bottomeigenvectors of the graph Laplacian using µ -EG ( M = 1 , n (cid:48)

Original: =
Revised: =

Original: η t = |E| 1000 ) and then running k -means clustering (Pedregosa et al. , 2011), we are able to recover the ground truth clusters (see Figure 4b) with 99 .
Revised: η t = |E| 1000 ) and then running k -means clustering (Pedregosa et al. , 2011), we are able to recover the ground truth clusters (see Figure 4b) with 99 .

Original: 92% accuracy.
Revised: 92% accuracy.

Original: The experiment was run on a single CPU.
Revised: The experiment was run on a single CPU.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We introduced µ -EigenGame, an unbiased, globally convergent, parallelizable algorithm that recovers the top- k eigenvectors of a symmetric positive deﬁnite matrix.
Revised: We introduced µ -EigenGame, an unbiased, globally convergent, parallelizable algorithm that recovers the top- k eigenvectors of a symmetric positive deﬁnite matrix.

Original: We demonstrated the performance of µ -EigenGame on large scale dimension reduction and clustering problems.
Revised: We demonstrated the performance of µ -EigenGame on large scale dimension reduction and clustering problems.

Original: Like its predecessor, µ -EigenGame is a k -player, general-sum game allowing model parallelism over players; our unbiased reformulation allows even greater parallelism over data.
Revised: Like its predecessor, µ -EigenGame is a k -player, general-sum game allowing model parallelism over players; our unbiased reformulation allows even greater parallelism over data.

Original: A E XPERIMENTS ON SYNTHETIC DATA
Revised: A E XPERIMENTS ON SYNTHETIC DATA

Original: [Figure]
Revised: [Figure]

Original: We validate µ -EigenGame in a full-batch setting on two synthetic datasets: one with exponentially decaying spectrum; the other with a linearly decaying spectrum.
Revised: We validate µ -EigenGame in a full-batch setting on two synthetic datasets: one with exponentially decaying spectrum; the other with a linearly decaying spectrum.

Original: Figure 5 shows µ -EigenGame outperforms α -EigenGame on the former and matches its performance on the latter.
Revised: Figure 5 shows µ -EigenGame outperforms α -EigenGame on the former and matches its performance on the latter.

Original: We discuss possible reasons for this gap in the discussion in Section G.
Revised: We discuss possible reasons for this gap in the discussion in Section G.

Original: B P ARALLELIZED A LGORITHM
Revised: B P ARALLELIZED A LGORITHM

Original: Riemannian Manifolds.
Revised: Riemannian Manifolds.

Original: Before introducing an algorithm for µ -EigenGame, we ﬁrst brieﬂy review necessary terminology for learning on Riemannian manifolds Absil et al.
Revised: Before introducing an algorithm for µ -EigenGame, we ﬁrst brieﬂy review necessary terminology for learning on Riemannian manifolds Absil et al.

Original: (2009), speciﬁcally for the sphere.
Revised: (2009), speciﬁcally for the sphere.

Original: The notation T ˆ v i S
Revised: The notation T ˆ v i S

Original: d − 1 denotes the set of vectors tangent to the sphere at a point ˆ v i (i.e., any vector orthogonal to ˆ v i ).
Revised: d − 1 denotes the set of vectors tangent to the sphere at a point ˆ v i (i.e., any vector orthogonal to ˆ v i ).

Original: R ˆ v i ( z )
Revised: R ˆ v i ( z )

Original: =
Revised: =

Original: ˆ v i
Revised: ˆ v i

Original: +
Revised: +

Original: z
Revised: z

Original: || ˆ
Revised: || ˆ

Original: v i
Revised: v i

Original: + z
Revised: + z

Original: || is the commonly used restriction of the retraction on
Revised: || is the commonly used restriction of the retraction on

Original: S d − 1 to the tangent bundle at ˆ v
Revised: S d − 1 to the tangent bundle at ˆ v

Original: i (i.e., step in tangent direction z and then unit-normalize the result).
Revised: i (i.e., step in tangent direction z and then unit-normalize the result).

Original: The operator Π ˆ v i ( y ) =
Revised: The operator Π ˆ v i ( y ) =

Original: ( I − ˆ v (cid:62) i ˆ v i )
Revised: ( I − ˆ v (cid:62) i ˆ v i )

Original: y projects the direction y onto T ˆ v i S
Revised: y projects the direction y onto T ˆ v i S

Original: d − 1 .
Revised: d − 1 .

Original: Combining these tools together results in a movement along the Riemannian manifold: ˆ v ( t +1)
Revised: Combining these tools together results in a movement along the Riemannian manifold: ˆ v ( t +1)

Original: i ← R ˆ v i (cid:0) Π
Revised: i ← R ˆ v i (cid:0) Π

Original: ˆ v
Revised: ˆ v

Original: i ( y ) (cid:1) .
Revised: i ( y ) (cid:1) .

Original: We present pseudocode for µ -EigenGame below where computation is parallelized both over the k players and over M machines per player.
Revised: We present pseudocode for µ -EigenGame below where computation is parallelized both over the k players and over M machines per player.

Original: Algorithm
Revised: Algorithm

Original: 1: Given: data stream X t ∈ R n (cid:48)
Revised: 1: Given: data stream X t ∈ R n (cid:48)

Original: × d , number of parallel machines M per player (minibatch size per machine n (cid:48)(cid:48) =
Revised: × d , number of parallel machines M per player (minibatch size per machine n (cid:48)(cid:48) =

Original: n (cid:48) M ), initial vectors ˆ v 0 i ∈ S
Revised: n (cid:48) M ), initial vectors ˆ v 0 i ∈ S

Original: d − 1 , step size sequence η t , and number of iterations T .
Revised: d − 1 , step size sequence η t , and number of iterations T .

Original: 2: ˆ v i ← ˆ v 

Revised: 2: ˆ v i ← ˆ v

Original: i for all i 3: for t = 1 : T do 4: parfor i = 1 : k do 5: parfor m = 1 : M do 6: rewards ← X (cid:62) tm X tm ˆ
Revised: i for all i 3: for t = 1 : T do 4: parfor i = 1 : k do 5: parfor m = 1 : M do 6: rewards ← X (cid:62) tm X tm ˆ

Original: v
Revised: v

Original: i 7: penalties ← (cid:80) j<i (cid:104)
Revised: i 7: penalties ← (cid:80) j<i (cid:104)

Original: X tm ˆ v i , X tm ˆ v j (cid:105)
Revised: X tm ˆ v i , X tm ˆ v j (cid:105)

Original: ˆ v j 8: ˜ ∇ µim ← ( rewards − penalties ) /n
Revised: ˆ v j 8: ˜ ∇ µim ← ( rewards − penalties ) /n

Original: (cid:48)(cid:48) 

Revised: (cid:48)(cid:48)

Original: ˜ ∇ µ,Rim ← ˜ ∇ µim − (cid:104) ˜ ∇ µim , ˆ v i (cid:105) ˆ v i 10: end parfor 11: ˜ ∇ µ,Ri ← 1 M (cid:80) m
Revised: ˜ ∇ µ,Rim ← ˜ ∇ µim − (cid:104) ˜ ∇ µim , ˆ v i (cid:105) ˆ v i 10: end parfor 11: ˜ ∇ µ,Ri ← 1 M (cid:80) m

Original: [ ˜ ∇ µ,Rim ] 12: ˆ v (cid:48)
Revised: [ ˜ ∇ µ,Rim ] 12: ˆ v (cid:48)

Original: i ← ˆ v i + η t ˜ ∇ µ,R i 13: ˆ v i ← ˆ v (cid:48)
Revised: i ← ˆ v i + η t ˜ ∇ µ,R i 13: ˆ v i ← ˆ v (cid:48)

Original: i ||
Revised: i ||

Original: ˆ v (cid:48)
Revised: ˆ v (cid:48)

Original: i || 14: end parfor 15: end for 16: return all ˆ v
Revised: i || 14: end parfor 15: end for 16: return all ˆ v

Original: i
Revised: i

Original: C G LOBAL S TOCHASTIC C ONVERGENCE
Revised: C G LOBAL S TOCHASTIC C ONVERGENCE

Original: Theorem 1 (Global Convergence) .
Revised: Theorem 1 (Global Convergence) .

Original: Given a positive deﬁnite covariance matrix C with the top- k eigengaps positive and a square-summable, not summable step size sequence η t (e.g., 1 /t ), Algorithmconverges to the top- k eigenvectors asymptotically ( lim T →∞ ) with probability 1 .
Revised: Given a positive deﬁnite covariance matrix C with the top- k eigengaps positive and a square-summable, not summable step size sequence η t (e.g., 1 /t ), Algorithmconverges to the top- k eigenvectors asymptotically ( lim T →∞ ) with probability 1 .

Original: Proof.
Revised: Proof.

Original: Assume none of the ˆ v i are initialized to an angle exactly 90 ◦ away from the true eigenvector: (cid:104)
Revised: Assume none of the ˆ v i are initialized to an angle exactly 90 ◦ away from the true eigenvector: (cid:104)

Original: ˆ v i , v i (cid:105) (cid:54)
Revised: ˆ v i , v i (cid:105) (cid:54)

Original: = 0 .
Revised: = 0 .

Original: The set of vectors { ˆ v i : (cid:104) ˆ v i , v i (cid:105) = 0 } has Lebesgue measure 0 , therefore, the above assumption holds w.p.1.
Revised: The set of vectors { ˆ v i : (cid:104) ˆ v i , v i (cid:105) = 0 } has Lebesgue measure 0 , therefore, the above assumption holds w.p.1.

Original: The update direction for the top eigenvector ˆ v 1 is exactly equal to that of α -EigenGame ( ˜ ∇ µ 1 = ∇ α 1 ), therefore, they have the same limit points for ˆ v 1 .
Revised: The update direction for the top eigenvector ˆ v 1 is exactly equal to that of α -EigenGame ( ˜ ∇ µ 1 = ∇ α 1 ), therefore, they have the same limit points for ˆ v 1 .

Original: The proof then proceeds by induction.
Revised: The proof then proceeds by induction.

Original: As ˆ v j<i approach their limit points, the update for the i th eigenvector ˆ v i approaches that of α -EigenGame ( ˜ ∇ µi = ∇ αi ) and, by Lemma 3, the stable region of µ -EigenGame also shrinks to a point around the top- k eigenvectors.
Revised: As ˆ v j<i approach their limit points, the update for the i th eigenvector ˆ v i approaches that of α -EigenGame ( ˜ ∇ µi = ∇ αi ) and, by Lemma 3, the stable region of µ -EigenGame also shrinks to a point around the top- k eigenvectors.

Original: Denote the “update ﬁeld” H ( v ) to match the work of Shah (2019).
Revised: Denote the “update ﬁeld” H ( v ) to match the work of Shah (2019).

Original: H ( v ) is simply the concatenation of all players’ Riemannian update rules, i.e., all players updating in parallel using their Riemannian updates:
Revised: H ( v ) is simply the concatenation of all players’ Riemannian update rules, i.e., all players updating in parallel using their Riemannian updates:

Original: [Equation]
Revised: [Equation]

Original: where ∆ µi is deﬁned in equation (7) and ( I − p 1 p (cid:62) 1 )∆ µi projects ∆ µi onto the tangent space of player i ’s unit sphere.
Revised: where ∆ µi is deﬁned in equation (7) and ( I − p 1 p (cid:62) 1 )∆ µi projects ∆ µi onto the tangent space of player i ’s unit sphere.

Original: The result is then obtained by applying Theorem 7 of Shah (2019) with the following information: a) the unit-sphere is a compact manifold with an injectivity radius of π , b) the update ﬁeld is a polynomial in { v i } and therefore smooth (analytic), and c) by Lemma 4 (see Appendix E) the update noise constitutes a bounded martingale difference sequence.
Revised: The result is then obtained by applying Theorem 7 of Shah (2019) with the following information: a) the unit-sphere is a compact manifold with an injectivity radius of π , b) the update ﬁeld is a polynomial in { v i } and therefore smooth (analytic), and c) by Lemma 4 (see Appendix E) the update noise constitutes a bounded martingale difference sequence.

Original: While the convergence proof for α -EG provides ﬁnite-sample rates, it only applies to the algorithm applied sequentially (not parallelized over eigenvectors) and in the deterministic setting (minibatch contains the entire data set).
Revised: While the convergence proof for α -EG provides ﬁnite-sample rates, it only applies to the algorithm applied sequentially (not parallelized over eigenvectors) and in the deterministic setting (minibatch contains the entire data set).

Original: The experiments in Gemp et al.
Revised: The experiments in Gemp et al.

Original: (2021) apply the algorithm in parallel and with mini batch sizes, meaning the α -EG theorem does not actually apply to their experimental setting.
Revised: (2021) apply the algorithm in parallel and with mini batch sizes, meaning the α -EG theorem does not actually apply to their experimental setting.

Original: That is to say, the α -EG paper proposes updating eigenvectors in parallel in practice despite the lack of convergence guarantee.
Revised: That is to say, the α -EG paper proposes updating eigenvectors in parallel in practice despite the lack of convergence guarantee.

Original: In contrast, our convergence theorem applies to µ -EG when applied in parallel (over the eigenvectors) and in the stochastic setting (with mini batch sizes), which is what we examine empirically in our experiments.
Revised: In contrast, our convergence theorem applies to µ -EG when applied in parallel (over the eigenvectors) and in the stochastic setting (with mini batch sizes), which is what we examine empirically in our experiments.

Original: The downside is that we do not provide ﬁnite-sample convergence rates.
Revised: The downside is that we do not provide ﬁnite-sample convergence rates.

Original: Although we do not provide convergence rates, Lemma 1 proves that the µ -EG update converges to the α -EG update for each eigenvector, so intuitively, we expect the convergence rates to be relatively similar given that the algorithms are equivalent in the limit.
Revised: Although we do not provide convergence rates, Lemma 1 proves that the µ -EG update converges to the α -EG update for each eigenvector, so intuitively, we expect the convergence rates to be relatively similar given that the algorithms are equivalent in the limit.

Original: Note that in the full batch setting where stochasticity does not conﬂate the differences between the two algorithms, Figurein Appendix A empirically supports the similarity of the convergence rates for the two algorithms.
Revised: Note that in the full batch setting where stochasticity does not conﬂate the differences between the two algorithms, Figurein Appendix A empirically supports the similarity of the convergence rates for the two algorithms.

Original: Figure 3 (minibatch of 1024 ) which looks at a large (but not full) minibatch size, also shows a small difference between convergence for the two algorithms.
Revised: Figure 3 (minibatch of 1024 ) which looks at a large (but not full) minibatch size, also shows a small difference between convergence for the two algorithms.

Original: In summary, the α -EG convergence theorem is impractical—it provides convergence rates for a (non-parallel) algorithm in the (non-stochastic) setting, which is a combination that α -EG paper does not suggest be applied in practice.
Revised: In summary, the α -EG convergence theorem is impractical—it provides convergence rates for a (non-parallel) algorithm in the (non-stochastic) setting, which is a combination that α -EG paper does not suggest be applied in practice.

Original: In contrast, our µ -EG convergence theorem is practical—it provides asymptotic convergence for a parallel algorithm in the stochastic setting.
Revised: In contrast, our µ -EG convergence theorem is practical—it provides asymptotic convergence for a parallel algorithm in the stochastic setting.

Original: Difﬁculties Obtaining Finite Sample Rates In consideration of a ﬁnite sample convergence result, we consulted Durmus et al.
Revised: Difﬁculties Obtaining Finite Sample Rates In consideration of a ﬁnite sample convergence result, we consulted Durmus et al.

Original: The primary obstacle to applying their convergence theorem is the construction of a suitable Lyapunov function to satisfy their Assumption A.2 stated on page 4.
Revised: The primary obstacle to applying their convergence theorem is the construction of a suitable Lyapunov function to satisfy their Assumption A.2 stated on page 4.

Original: Constructing Lyapunov functions is typically a tedious, unpredictable process.
Revised: Constructing Lyapunov functions is typically a tedious, unpredictable process.

Original: The work in Durmus et al.
Revised: The work in Durmus et al.

Original: (2020) is very recent and ﬁnite sample convergence of Riemannian stochastic approximation (i.e., update directions are not gradients of any function) schemes is cutting edge, highly technical research.
Revised: (2020) is very recent and ﬁnite sample convergence of Riemannian stochastic approximation (i.e., update directions are not gradients of any function) schemes is cutting edge, highly technical research.

Original: This is in contrast to Riemannian optimization (i.e., update directions are the gradient of a function), which is much more mature.
Revised: This is in contrast to Riemannian optimization (i.e., update directions are the gradient of a function), which is much more mature.

Original: We hope theory advances in the near future to a point where we can more easily provide convergence rates for algorithms like α -EigenGame.
Revised: We hope theory advances in the near future to a point where we can more easily provide convergence rates for algorithms like α -EigenGame.

Original: D E RROR P ROPAGATION / S ENSITIVITY A NALYSIS
Revised: D E RROR P ROPAGATION / S ENSITIVITY A NALYSIS

Original: Lemma 3.
Revised: Lemma 3.

Original: An O ( (cid:15) ) angular error of parent ˆ v j<i implies an O ( (cid:15) ) angular error in the location of the solution for ˆ v i .
Revised: An O ( (cid:15) ) angular error of parent ˆ v j<i implies an O ( (cid:15) ) angular error in the location of the solution for ˆ v i .

Original: Proof.
Revised: Proof.

Original: The proof proceeds in three steps:
Revised: The proof proceeds in three steps:

Original: O ( (cid:15) ) angular error of parent =
Revised: O ( (cid:15) ) angular error of parent =

Original: ⇒ O
Revised: ⇒ O

Original: ( (cid:15) )
Revised: ( (cid:15) )

Original: Euclidean error of parent 2.
Revised: Euclidean error of parent 2.

Original: O ( (cid:15) )
Revised: O ( (cid:15) )

Original: Euclidean error of parent =
Revised: Euclidean error of parent =

Original: ⇒ O
Revised: ⇒ O

Original: ( (cid:15) )
Revised: ( (cid:15) )

Original: Euclidean error of norm of child gradient 3.
Revised: Euclidean error of norm of child gradient 3.

Original: O ( (cid:15) )
Revised: O ( (cid:15) )

Original: Euclidean error of norm child gradient + instability of minimum at ± π 2 = ⇒ O
Revised: Euclidean error of norm child gradient + instability of minimum at ± π 2 = ⇒ O

Original: ( (cid:15) ) angular error of child’s solution.
Revised: ( (cid:15) ) angular error of child’s solution.

Original: Angular error in the parent can be converted to Euclidean error by considering the chord length between the mis-speciﬁed parent and the true parent direction.
Revised: Angular error in the parent can be converted to Euclidean error by considering the chord length between the mis-speciﬁed parent and the true parent direction.

Original: The two vectors plus the chord form an isoceles triangle with the relation that chord length l = 2 sin( (cid:15) 2 ) is O ( (cid:15) ) for (cid:15) (cid:28) 1 .
Revised: The two vectors plus the chord form an isoceles triangle with the relation that chord length l = 2 sin( (cid:15) 2 ) is O ( (cid:15) ) for (cid:15) (cid:28) 1 .

Original: Next, write the mis-speciﬁed parents as ˆ v j = v j + w j where || w j || is O ( (cid:15) ) as we have just shown.
Revised: Next, write the mis-speciﬁed parents as ˆ v j = v j + w j where || w j || is O ( (cid:15) ) as we have just shown.

Original: Let b equal the difference between the Riemannian update direction ˜ ∇ µi with approximate parents and that with exact parents.
Revised: Let b equal the difference between the Riemannian update direction ˜ ∇ µi with approximate parents and that with exact parents.

Original: All directions we consider here are the Riemannian directions, i.e., they have been projected onto the tangent space of the sphere.
Revised: All directions we consider here are the Riemannian directions, i.e., they have been projected onto the tangent space of the sphere.

Original: Then
Revised: Then

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and the norm of the difference is
Revised: and the norm of the difference is

Original: [Equation]
Revised: [Equation]

Original: We can further bound the summands with
Revised: We can further bound the summands with

Original: [Equation]
Revised: [Equation]

Original: This upper bound on the norm of the difference between the two directions translates to a lower bound on the inner product of the two directions wherever ||
Revised: This upper bound on the norm of the difference between the two directions translates to a lower bound on the inner product of the two directions wherever ||

Original: ˜ ∇ µ,ei || > (cid:15) , speciﬁcally
Revised: ˜ ∇ µ,ei || > (cid:15) , speciﬁcally

Original: (cid:104) ˜
Revised: (cid:104) ˜

Original: ∇ µ,ei , ˜ ∇ µi (cid:105) >
Revised: ∇ µ,ei , ˜ ∇ µi (cid:105) >

Original: (see Figure 6a).
Revised: (see Figure 6a).

Original: And recall that the direction with exact parents is equivalent to the gradient of α -EigenGame with exact parents, ∇ α,ei .
Revised: And recall that the direction with exact parents is equivalent to the gradient of α -EigenGame with exact parents, ∇ α,ei .

Original: Therefore, by a Lyapunov argument, the ˜ ∇ µi direction is an ascent direction on the α -EigenGame utility where it forms an acute angle (positive inner product) with ∇ α,ei .
Revised: Therefore, by a Lyapunov argument, the ˜ ∇ µi direction is an ascent direction on the α -EigenGame utility where it forms an acute angle (positive inner product) with ∇ α,ei .

Original: Furthermore, ∇ α,ei is the gradient of a utility function that is sinusoidal along the sphere manifold; speciﬁcally, it is a cosine with period π and positive amplitude dependent on the spectrum of C (c.f. equation (8) of Gemp
Revised: Furthermore, ∇ α,ei is the gradient of a utility function that is sinusoidal along the sphere manifold; speciﬁcally, it is a cosine with period π and positive amplitude dependent on the spectrum of C (c.f. equation (8) of Gemp

Original: [Figure]
Revised: [Figure]

Original: et al.
Revised: et al.

Original: We can derive an upper bound on the size of the angular region for which ˜ ∇ µi is not necessarily an ascent direction (the “?" marks in Figure 6).
Revised: We can derive an upper bound on the size of the angular region for which ˜ ∇ µi is not necessarily an ascent direction (the “?" marks in Figure 6).

Original: This region is deﬁned as the set of angles for which the norm of the utility’s derivative is small, i.e., ||∇ α,ei || ≤ (cid:15) .
Revised: This region is deﬁned as the set of angles for which the norm of the utility’s derivative is small, i.e., ||∇ α,ei || ≤ (cid:15) .

Original: The derivative of cosine is sine, which depends linearly on its argument (angle) for small values, therefore, | θ | ≤ O ( (cid:15) ) or | π 2 − θ | ≤ O ( (cid:15) ) .
Revised: The derivative of cosine is sine, which depends linearly on its argument (angle) for small values, therefore, | θ | ≤ O ( (cid:15) ) or | π 2 − θ | ≤ O ( (cid:15) ) .

Original: As long as ˆ v i does not lie within the | π 2 − O ( (cid:15) ) | region, µ -EigenGame will ascend the utility landscape to within O ( (cid:15) ) angular error of the true eigenvector v i .
Revised: As long as ˆ v i does not lie within the | π 2 − O ( (cid:15) ) | region, µ -EigenGame will ascend the utility landscape to within O ( (cid:15) ) angular error of the true eigenvector v i .

Original: In the limit as (cid:15) → 0 , the size of the | π 2 − O ( (cid:15) )
Revised: In the limit as (cid:15) → 0 , the size of the | π 2 − O ( (cid:15) )

Original: | region vanishes to a point, v ⊥ i .
Revised: | region vanishes to a point, v ⊥ i .

Original: To understand the stability of this point, we can again appeal to the analysis from Gemp et al.
Revised: To understand the stability of this point, we can again appeal to the analysis from Gemp et al.

Original: (2021)—see equation (8) on page 7 of that work.
Revised: (2021)—see equation (8) on page 7 of that work.

Original: The
Revised: The

Original: Jacobian of ˜ ∇ µi and the Hessian of u αi are equal with exact parents, and we know that its Riemannian
Revised: Jacobian of ˜ ∇ µi and the Hessian of u αi are equal with exact parents, and we know that its Riemannian

Original: Hessian is positive deﬁnite if the i th eigengap is positive: H R ˆ v i [ u αi ] (cid:23) ( λ i − λ i +1 )
Revised: Hessian is positive deﬁnite if the i th eigengap is positive: H R ˆ v i [ u αi ] (cid:23) ( λ i − λ i +1 )

Original: I .
Revised: I .

Original: This means that the point v ⊥ i is a repeller for α -EigenGame.
Revised: This means that the point v ⊥ i is a repeller for α -EigenGame.

Original: Similarly to before, we can show more formally that an
Revised: Similarly to before, we can show more formally that an

Original: O ( (cid:15) ) perturbation to parents results in an O ( (cid:15) ) perturbation to the Jacobian of ˜ ∇ µi from H [ u αi ] :
Revised: O ( (cid:15) ) perturbation to parents results in an O ( (cid:15) ) perturbation to the Jacobian of ˜ ∇ µi from H [ u αi ] :

Original: [Equation]
Revised: [Equation]

Original: where W is some matrix with O (1) entries (w.r.t.
Revised: where W is some matrix with O (1) entries (w.r.t.

Original: (cid:15) ).
Revised: (cid:15) ).

Original: For the sphere, the Riemannian Jacobian is a linear function of the Jacobian ( J R ˆ v i
Revised: For the sphere, the Riemannian Jacobian is a linear function of the Jacobian ( J R ˆ v i

Original: =
Revised: =

Original: ( I − ˆ v i ˆ v (cid:62) i ) J − (ˆ v (cid:62) i J ˆ v i )
Revised: ( I − ˆ v i ˆ v (cid:62) i ) J − (ˆ v (cid:62) i J ˆ v i )

Original: I =
Revised: I =

Original: H R
Revised: H R

Original: ˆ v i [ u αi ] − O ( (cid:15) ) ) and therefore the error remains O ( (cid:15) ) .
Revised: ˆ v i [ u αi ] − O ( (cid:15) ) ) and therefore the error remains O ( (cid:15) ) .

Original: The set of (non)symmetric, positive semideﬁnite matrices ( A is p.s.d.
Revised: The set of (non)symmetric, positive semideﬁnite matrices ( A is p.s.d.

Original: iff y (cid:62)
Revised: iff y (cid:62)

Original: Ay ≥ 0 ∀ y ) forms a closed convex cone, the interior of which contains positive deﬁnite matrices Wang et al.
Revised: Ay ≥ 0 ∀ y ) forms a closed convex cone, the interior of which contains positive deﬁnite matrices Wang et al.

Original: Therefore, J R ˆ v i remains in this set after a small enough O ( (cid:15) ) perturbation.
Revised: Therefore, J R ˆ v i remains in this set after a small enough O ( (cid:15) ) perturbation.

Original: Therefore, in the limit (cid:15) → 0 , the spectrum of the Jacobian will also be positive deﬁnite indicating the point v ⊥ i is a repeller.
Revised: Therefore, in the limit (cid:15) → 0 , the spectrum of the Jacobian will also be positive deﬁnite indicating the point v ⊥ i is a repeller.

Original: This is indicated by the blue arrows in Figure 6b.
Revised: This is indicated by the blue arrows in Figure 6b.

Original: Figure 6b summarizes the results that the stable region for α -EigenGame consists of an O ( (cid:15) ) ball around the true optimum for parents with O ( (cid:15) ) angular error.
Revised: Figure 6b summarizes the results that the stable region for α -EigenGame consists of an O ( (cid:15) ) ball around the true optimum for parents with O ( (cid:15) ) angular error.

Original: E N OISE IS M ARTINGALE D IFFERENCE S EQUENCE
Revised: E N OISE IS M ARTINGALE D IFFERENCE S EQUENCE

Original: Let ∆ µ,ti=
Revised: Let ∆ µ,ti

Original: [Equation]
Revised: [Equation]

Original: (cid:105) C ˆ v
Revised: (cid:105) [Equation] ˆ v

Original: [Equation]
Revised: [Equation]

Original: t ) i be the µ -EigenGame update direction computed using the full expected covariance matrix.
Revised: t ) i be the µ -EigenGame update direction computed using the full expected covariance matrix.

Original: Let ˆ∆ µ,ti
Revised: Let ˆ∆ µ,ti

Original: [Equation]
Revised: [Equation]

Original: be the update direction computed using a minibatch estimate of the covariance matrix where minibatches are unbiased because they are formed from data sampled uniformly at random from the dataset.
Revised: be the update direction computed using a minibatch estimate of the covariance matrix where minibatches are unbiased because they are formed from data sampled uniformly at random from the dataset.

Original: Deﬁne M V ( t ) i,t +1 = ˆ∆ µ,ti − ∆ µ,ti and let M V ( t ) t +1 =
Revised: Deﬁne M V ( t ) i,t +1 = ˆ∆ µ,ti − ∆ µ,ti and let M V ( t ) t +1 =

Original: [ M V ( t ) 1 ,t +1 , . . .
Revised: [ M V ( t ) 1 ,t +1 , . . .

Original: , M V ( t ) k,t +1 ] (cid:62) where V ( t )
Revised: , M V ( t ) k,t +1 ] (cid:62) where V ( t )

Original: =
Revised: =

Original: { v ( t ) i ∈
Revised: { v ( t ) i ∈

Original: [ k ] } .
Revised: [ k ] } .

Original: Lemma 4.
Revised: Lemma 4.

Original: { M V ( t ) t +1 } is a bounded martingale difference sequence with respect to the increasing σ -ﬁelds
Revised: { M V ( t ) t +1 } is a bounded martingale difference sequence with respect to the increasing σ -ﬁelds

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Given the ﬁltration F t , we ﬁnd
Revised: Given the ﬁltration F t , we ﬁnd

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst equality holds because each C t is formed from a minibatch sampled i.i.d.
Revised: where the ﬁrst equality holds because each C t is formed from a minibatch sampled i.i.d.

Original: from the dataset and therefore independent of the ﬁltration.
Revised: from the dataset and therefore independent of the ﬁltration.

Original: This result holds for all i ∈ [ k ] , therefore
Revised: This result holds for all i ∈ [ k ] , therefore

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Furthermore,
Revised: Furthermore,

Original: [Equation]
Revised: [Equation]

Original: 35) where λ i ≤ max { 1 , ( i − 2) 2 } is the max singular value of P 1 and ξ 2 is the maximum eigenvalue of E
Revised: 35) where λ i ≤ max { 1 , ( i − 2) 2 } is the max singular value of P 1 and ξ 2 is the maximum eigenvalue of E

Original: [( C t − C ) (cid:62) ( C t − C )] over all t .
Revised: [( C t − C ) (cid:62) ( C t − C )] over all t .

Original: Summing over i we ﬁnd
Revised: Summing over i we ﬁnd

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: This is a worst case bound.
Revised: This is a worst case bound.

Original: As the parent eigenvectors converge, the max singular value of P converges to 1 so that the variance of the magnitude of the martingale difference is upper bounded by kξ 2 .
Revised: As the parent eigenvectors converge, the max singular value of P converges to 1 so that the variance of the magnitude of the martingale difference is upper bounded by kξ 2 .

Original: Note : For a ﬁnite dataset of n samples or for a distribution with bounded moments like the Gaussian distribution, ξ will be ﬁnite.
Revised: Note : For a ﬁnite dataset of n samples or for a distribution with bounded moments like the Gaussian distribution, ξ will be ﬁnite.

Original: However, for other distributions like the Cauchy distribution, ξ may be unbounded, so care should be taken when running µ -EigenGame in different stochastic settings.
Revised: However, for other distributions like the Cauchy distribution, ξ may be unbounded, so care should be taken when running µ -EigenGame in different stochastic settings.

Original: F J AX PSEUDOCODE
Revised: F J AX PSEUDOCODE

Original: For the sake of reproducibility we have included pseudocode in Jax.
Revised: For the sake of reproducibility we have included pseudocode in Jax.

Original: We use the Optax 2 optimization library Hessel et al.
Revised: We use the Optax 2 optimization library Hessel et al.

Original: (2020) and the Jaxline training framework 3 .
Revised: (2020) and the Jaxline training framework 3 .

Original: Our graph algorithm is a straightforward modiﬁcation of the provided pseudo-code.
Revised: Our graph algorithm is a straightforward modiﬁcation of the provided pseudo-code.

Original: See section H for details.
Revised: See section H for details.

Original: 1 """ 2 Copyright 

Revised: 1 """ 2 Copyright

Original: The EigenGame Unloaded Authors.
Revised: The EigenGame Unloaded Authors.

Original: 35 Licensed under the Apache License, Version 2.0 (the "License"); 6 you may not use this file except in compliance with the License.
Revised: 35 Licensed under the Apache License, Version 2.0 (the "License"); 6 you may not use this file except in compliance with the License.

Original: 7 You may obtain a copy of the License at 8 9 https://www.apache.org/licenses/LICENSE-2.
Revised: 7 You may obtain a copy of the License at 8 9 https://www.apache.org/licenses/LICENSE-2.

Original: 10 11 Unless required by applicable law or agreed to in writing, software 12 distributed under the License is distributed on an "AS IS" BASIS, 13 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
Revised: 10 11 Unless required by applicable law or agreed to in writing, software 12 distributed under the License is distributed on an "AS IS" BASIS, 13 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

Original: 14 See the License for the specific language governing permissions and 15 limitations under the License.
Revised: 14 See the License for the specific language governing permissions and 15 limitations under the License.

Original: 16 """ 17 18 import jax 19 import optax 20 import jax.numpy as jnp 21 22 def eg_grads(vi: jnp.ndarray, weights: jnp.ndarray, eigs: jnp.ndarray, data: jnp.ndarray) -> jnp.ndarray: """
Revised: 16 """ 17 18 import jax 19 import optax 20 import jax.numpy as jnp 21 22 def eg_grads(vi: jnp.ndarray, weights: jnp.ndarray, eigs: jnp.ndarray, data: jnp.ndarray) -> jnp.ndarray: """

Original: Args: vi: d,), eigenvector to be updated weights: shape k,), mask for penalty coefficients, eigs: shape (k, d), i.e., vectors on rows data: shape (N, d), minibatch X_t
Revised: Args: vi: shape (d,), eigenvector to be updated weights: shape (k,), mask for penalty coefficients, eigs: shape (k, d), i.e., vectors on rows data: shape (N, d), minibatch X_t

Original: Returns: grads: shape (d,), gradient for vi """ weights_ij = (jnp.sign(weights + 0.5) - 1.) / 2.
Revised: Returns: grads: shape (d,), gradient for vi """ weights_ij = (jnp.sign(weights + 0.5) - 1.) / 2.

Original: # maps -1 to -1 else to data_vi =
Revised: # maps -1 to -1 else to data_vi =

Original: jnp.dot(data, vi) data_eigs = jnp.transpose(jnp.dot(data, jnp.transpose(eigs)))
Revised: jnp.dot(data, vi) data_eigs = jnp.transpose(jnp.dot(data, jnp.transpose(eigs)))

Original: # Xvj on row j vi_m_vj
Revised: # Xvj on row j vi_m_vj

Original: =
Revised: =

Original: jnp.dot(data_eigs, data_vi) penalty_grads =
Revised: jnp.dot(data_eigs, data_vi) penalty_grads =

Original: vi_m_vj * jnp.transpose(eigs) penalty_grads =
Revised: vi_m_vj * jnp.transpose(eigs) penalty_grads =

Original: jnp.dot(penalty_grads, weights_ij) grads = jnp.dot(jnp.transpose(data), data_vi)
Revised: jnp.dot(penalty_grads, weights_ij) grads = jnp.dot(jnp.transpose(data), data_vi)

Original: + penalty_grads return grads 4446 def utility(vi, weights, eigs, data): 2 https://github.com/deepmind/ https://
Revised: + penalty_grads return grads 4446 def utility(vi, weights, eigs, data): 2 https: https

Original: """Compute Eigengame utilities.
Revised: """Compute Eigengame utilities.

Original: util vi """ data_vi = jnp.dot(data, vi) data_eigs = jnp.transpose(jnp.dot(data, jnp.transpose(eigs)))
Revised: util vi """ data_vi = jnp.dot(data, vi) data_eigs = jnp.transpose(jnp.dot(data, jnp.transpose(eigs)))

Original: # Xvj on row j vi_m_vj2 =
Revised: # Xvj on row j vi_m_vj2 =

Original: jnp.dot(data_eigs, data_vi)**2.
Revised: jnp.dot(data_eigs, data_vi)**2.

Original: vj_m_vj =
Revised: vj_m_vj =

Original: jnp.sum(data_eigs * data_eigs, axis=1) r_ij = vi_m_vj2 / vj_m_vj util =
Revised: jnp.sum(data_eigs * data_eigs, axis=1) r_ij = vi_m_vj2 / vj_m_vj util =

Original: jnp jnp array r ij ) return util return utilities vi and update directions
Revised: jnp.dot(jnp.array(r_ij), weights) return util state

Original: Wrap in jax.vmap k dimension." "" utilities = utility(vi, weights, V inputs) grads = eg grads V
Revised: " utilities = utility

Original: ) returngrads, utilities 27 28 def Wrap in jax """ updates return vi_new, opt_state{+
+}
Revised: return utilities {+
+}

Original: Listing 2:EigenGame Update functions.
Revised: Listing 2: Update def self local

Original: G D ISCUSSION G. U TILITIES TO UPDATES AND BACK
Revised: G D ISCUSSION G. U TILITIES TO UPDATES AND BACK

Original: [Equation]
Revised: [Equation]

Original: remove the direction is unlikely.
Revised: remove bias direction is unlikely.

Original: Figure 7 summarizes the relationships advising the designs of the various EigenGame algorithms.
Revised: Figure 7 summarizes the relationships advising the designs of the various EigenGame algorithms.

Original: Starting from the α -EigenGame utility, its update is arrived at by simply following the standard gradient ascent paradigm.
Revised: Starting from the α -EigenGame utility, its update is arrived at by simply following the standard gradient ascent paradigm.

Original: In noticing that stochastic estimates of the gradient are biased, we arrive at the µ -EigenGame update by considering how to remove this bias in a principled manner.
Revised: In noticing that stochastic estimates of the gradient are biased, we arrive at the µ -EigenGame update by considering how to remove this bias in a principled manner.

Original: Sacriﬁcing the exact steepest decent direction for a direction that allows unbiased estimates is a tradeoff that in this case has beneﬁts.
Revised: Sacriﬁcing the exact steepest decent direction for a direction that allows unbiased estimates is a tradeoff that in this case has beneﬁts.

Original: Also, while ˜ ∇ µi is not a gradient (except with exact parents), the new penalties have properties (above) that make them intuitively more desirable than the originals; they are adaptive to the state of the system (discussed further in section I.2).
Revised: Also, while ˜ ∇ µi is not a gradient (except with exact parents), the new penalties have properties (above) that make them intuitively more desirable than the originals; they are adaptive to the state of the system (discussed further in section I.2).

Original: We derive pseudo-utilities with desired theoretical properties by integrating the new updates with help from the stop gradient operator.
Revised: We derive pseudo-utilities with desired theoretical properties by integrating the new updates with help from the stop gradient operator.

Original: However, it is unlikely that this utility would be developed independently of these steps to solve the problem at hand (see Appendix I for more details).
Revised: However, it is unlikely that this utility would be developed independently of these steps to solve the problem at hand (see Appendix I for more details).

Original: This suggests an alternative approach to algorithm design complementary to the optimization perspective: directly designing updates themselves which converge to the desired solution, reminiscent of previous paradigms that drove neuro-inspired learning rules.
Revised: This suggests an alternative approach to algorithm design complementary to the optimization perspective: directly designing updates themselves which converge to the desired solution, reminiscent of previous paradigms that drove neuro-inspired learning rules.

Original: G. B RIDGING H EBBIAN AND OPTIMIZATION APPROACHES
Revised: G. B RIDGING H EBBIAN AND OPTIMIZATION APPROACHES

Original: The Generalized Hebbian Algorithm (GHA) (Sanger, 1989; Gang et al. , 2019; Chen et al. , 2019) update direction for ˆ v i with inexact parents ˆ v j is similar to µ -EigenGame:
Revised: The Generalized Hebbian Algorithm (GHA) (Sanger, 1989; Gang et al. , 2019; Chen et al. , 2019) update direction for ˆ v i with inexact parents ˆ v j is similar to µ -EigenGame:

Original: [Equation]
Revised: [Equation]

Original: C appears linearly in this update so GHA can also be parallelized.
Revised: C appears linearly in this update so GHA can also be parallelized.

Original: In contrast to µ -EigenGame, GHA additionally penalizes the alignment of ˆ v i to itself and removes the unit norm constraint on ˆ v i (not shown).
Revised: In contrast to µ -EigenGame, GHA additionally penalizes the alignment of ˆ v i to itself and removes the unit norm constraint on ˆ v i (not shown).

Original: Without any constraints, GHA overﬂows in experiments.
Revised: Without any constraints, GHA overﬂows in experiments.

Original: We take the approach of Gemp et al.
Revised: We take the approach of Gemp et al.

Original: and constrain ˆ v i to the unit-ball ( || ˆ v
Revised: and constrain ˆ v i to the unit-ball ( || ˆ v

Original: i || ≤ 1 ) rather than the unit-sphere ( || ˆ
Revised: i || ≤ 1 ) rather than the unit-sphere ( || ˆ

Original: v i ||
Revised: v i ||

Original: = 1 ).
Revised: = 1 ).

Original: The connection between GHA and µ -EigenGame is interesting because unlike µ -EigenGame, GHA is a Hebbian learning algorithm inspired by neuroscience and its update rule is not motivated from the perspective of maximizing of a utility function.
Revised: The connection between GHA and µ -EigenGame is interesting because unlike µ -EigenGame, GHA is a Hebbian learning algorithm inspired by neuroscience and its update rule is not motivated from the perspective of maximizing of a utility function.

Original: Game formulations of classical machine learning problems may provide a bridge between statistical and biologically inspired viewpoints.
Revised: Game formulations of classical machine learning problems may provide a bridge between statistical and biologically inspired viewpoints.

Original: H µ -E IGEN G AME ON G RAPHS
Revised: H µ -E IGEN G AME ON G RAPHS

Original: Algorithm 3 receives a stream of edges represented as a matrix with edges on the rows and outgoing node id ( out ) and incoming node id ( in ) as nonegative integers on the columns.
Revised: Algorithm 3 receives a stream of edges represented as a matrix with edges on the rows and outgoing node id ( out ) and incoming node id ( in ) as nonegative integers on the columns.

Original: The method zeros_like ( z ) returns an array of zeros with the same dimensions as z .
Revised: The method zeros_like ( z ) returns an array of zeros with the same dimensions as z .

Original: The method index_add ( z, idx, val ) adds the values in array val to z at the corresponding indices in array idx with threadsafe locking so that indices in idx may be duplicated.
Revised: The method index_add ( z, idx, val ) adds the values in array val to z at the corresponding indices in array idx with threadsafe locking so that indices in idx may be duplicated.

Original: Both methods are available in J AX .
Revised: Both methods are available in J AX .

Original: The largest eigenvector ˆ v 1 is learned to estimate λ 1 and may be discarded.
Revised: The largest eigenvector ˆ v 1 is learned to estimate λ 1 and may be discarded.

Original: The bottom- k eigenvectors are returned by the algorithm in increasing order.
Revised: The bottom- k eigenvectors are returned by the algorithm in increasing order.

Original: Algorithm 3 expects k + 1 random unit vectors as input rather than k
Revised: Algorithm 3 expects k + 1 random unit vectors as input rather than k

Original: in order to additionally estimate the top eigenvector necessary for the computation; otherwise, the inputs are the same as Algorithm 1.
Revised: in order to additionally estimate the top eigenvector necessary for the computation; otherwise, the inputs are the same as Algorithm 1.

Original: I A LGORITHM D ESIGN P ROCESS
Revised: I A LGORITHM D ESIGN P ROCESS

Original: In section 4. [Equation] we presented u µi as the Rayleigh quotient of a deﬂated matrix (repeated in equation (8) for convencience):
Revised: In section 4.1, we presented u µi as the Rayleigh quotient of a deﬂated matrix (repeated in equation (8) for convencience):

Original: [Equation]
Revised: [Equation]

Original: Alternatively
Revised: Alternatively

Original: [Equation]
Revised: [Equation]

Original: we can consider u µi as equation (42) in light of the derivation for u αi by Gemp et al.
Revised: we can consider u µi as equation (42) in light of the derivation for u αi by Gemp et al.

Original: In that case, utilities are constructed from entries in the matrix
Revised: In that case, utilities are constructed from entries in the matrix

Original: [Equation]
Revised: [Equation]

Original: It is argued
Revised: It is argued

Original: that if ˆ V diagonalizes M and captures maximum variance, then the diagonal (cid:104) ˆ v i , C ˆ v i (cid:105) terms must be maximized and the off-diagonal (cid:104)
Revised: that if ˆ V diagonalizes M and captures maximum variance, then the diagonal (cid:104) ˆ v i , C ˆ v i (cid:105) terms must be maximized and the off-diagonal (cid:104)

Original: ˆ v i , C ˆ v j (cid:105) terms must be zero.
Revised: ˆ v i , C ˆ v j (cid:105) terms must be zero.

Original: As the latter mixed terms may be negative, the authors square the mixed terms to form “minimizable utilities” and divide 
Revised: As the latter mixed terms may be negative, the authors square the mixed terms to form “minimizable utilities” and divide do do 7: [

Original: 4: for do i = 1 : k + 1 do 6: parfor m = 1 : M do 7:

Original: [ Xv ] i

Original: =
Revised: =

Original: ˆ v i ( out tm ) − ˆ v i in tm ) 8: [ X(cid:62) Xv ] i ← zeros_like 
Revised: 8: [ X i ← zeros_like 9: [ X i ← index add

Original: (ˆ v i ) 9: [ X (cid:62) Xv ] i ← index_add ([ X(cid:62) Xv ] , out tm , [ Xv ] i ) 10: [ X (cid:62) Xv ] i ← index_add ([ X (cid:62) Xv ] , in tm , − i ) 11: if
Revised: X

Original: 1 then 12: λ 1 ← ||
Revised: X i ← index_add then 12: λ 1 ← ||

Original: [ Xv ] i ||13: ˜ ∇ µit (cid:48)
Revised: ] i ||13: ˜ ∇ µit (cid:48)

Original: ← [ X (cid:62)Xv ] i 14: else 15: ˜ ∇ µim ← λ 1 [ˆ v i − (cid:80) 1 <j<i (ˆ v (cid:62) i ˆ v j )ˆ v j ] 16:
Revised: ← [ X (cid:62) ] i 14: else 15: ˜ ∇ µim ← λ 1 [ˆ v i − (cid:80) 1 <j<i ( v (cid:62) i ˆ v j )ˆ v j ] 16:

Original: [ Xv ] j = ˆ v j ( out tm ) − ˆ vj ( in tm ) for all j 17: ˜ ∇ µit (cid:48) − =
Revised: [ Xv ] j = ˆ v j ( out tm ) − ˆ v ) for all j 17: ˜ ∇ µit (cid:48) − =

Original: [ X (cid:62) Xv ] i − (cid:80) 1 <j<i ([ Xv ] (cid:62) i
Revised: [ X (cid:62) Xv ] i − (cid:80) 1 <j<i ([ Xv ] (cid:62) i

Original: [ Xv ] j )ˆ v j 18: end if 19: end parfor 20: ˜ ∇ µi ← 1 n (cid:48) (cid:80) t (cid:48)
Revised: [ Xv ] j )ˆ v j 18: end if 19: end parfor 20: ˜ ∇ µi ← 1 n (cid:48) (cid:80) t (cid:48)

Original: [ ˜ ∇ µit (cid:48) ]
Revised: [ ˜ ∇ µit (cid:48) ]

Original: 21: ˆ v (cid:48)
Revised: 21: ˆ v (cid:48)

Original: i ← ˆ v i + η t ˜ ∇ µi 22:
Revised: i ← ˆ v i + η t ˜ ∇ µi 22:

Original: ˆ v i ← ˆ v (cid:48)
Revised: ˆ v i ← ˆ v (cid:48)

Original: i ||
Revised: i ||

Original: ˆ v (cid:48)
Revised: ˆ v (cid:48)

Original: i || 23: end parfor 24: end for 25: return { ˆ v i | i ∈ { 2 , . . .
Revised: i || 23: end parfor 24: end for 25: return { ˆ v i | i ∈ { 2 , . . .

Original: , k + 1 }} them by (cid:104)
Revised: , k + 1 }} them by (cid:104)

Original: ˆ v j , C ˆ v j (cid:105) so that they have similar “units" to the terms (cid:104)
Revised: ˆ v j , C ˆ v j (cid:105) so that they have similar “units" to the terms (cid:104)

Original: ˆ v i , C
Revised: ˆ v i , C

Original: ˆ v i (cid:105) of the ﬁrst type.
Revised: ˆ v i (cid:105) of the ﬁrst type.

Original: In contrast, the u µi utilities could be arrived at by instead multiplying the mixed terms by (cid:104)
Revised: In contrast, the u µi utilities could be arrived at by instead multiplying the mixed terms by (cid:104)

Original: ˆ v i , ˆ v j (cid:105) .
Revised: ˆ v i , ˆ v j (cid:105) .

Original: While this ensures the mixed terms are positive with exact parents (because (cid:104) ˆ v i , Cv j (cid:105)
Revised: While this ensures the mixed terms are positive with exact parents (because (cid:104) ˆ v i , Cv j (cid:105)

Original: =
Revised: =

Original: λ j (cid:104)
Revised: λ j (cid:104)

Original: ˆ v i , v j (cid:105) )
Revised: ˆ v i , v j (cid:105) )

Original: , it does not ensure they are always positive in general 4 .
Revised: , it does not ensure they are always positive in general 4 .

Original: In other words, u µi is deﬁned in way such that the ⊥ -penalties actually encourage vectors to align at times when they should in fact do the opposite!
Revised: In other words, u µi is deﬁned in way such that the ⊥ -penalties actually encourage vectors to align at times when they should in fact do the opposite!

Original: We therefore consider it unlikely that anyone would pose equation (42) as a utility if coming from the perspective of α -EigenGame.
Revised: We therefore consider it unlikely that anyone would pose equation (42) as a utility if coming from the perspective of α -EigenGame.

Original: We could have extended the diagram in Figure 7 to include this dead end link.
Revised: We could have extended the diagram in Figure 7 to include this dead end link.

Original: We have also included the true gradient of u µ i as a logical endpoint.
Revised: We have also included the true gradient of u µ i as a logical endpoint.

Original: We present these extensions in Figure 8.
Revised: We present these extensions in Figure 8.

Original: [Equation]
Revised: [Equation]

Original: bias remove
Revised: bias remove

Original: [Equation]
Revised: [Equation]

Revised: 4 e.g., let

Original:  (cid:20)
Revised: = (cid:20)

Original: I. G RADIENT A SCENT ON u 
Revised: I. G RADIENT A SCENT ON u µi

Original: µi

Original: If we remove the stop gradient from equation (41), we are left with equation (45):
Revised: If we remove the stop gradient from equation (41), we are left with equation (45):

Original: [Equation]
Revised: [Equation]

Original: If we then differentiate this utility
Revised: If we then differentiate this utility

Original: [Figure]
Revised: [Figure]

Original: we ﬁnd its gradient is
Revised: we ﬁnd its gradient is

Original: [Equation]
Revised: [Equation]

Original: We also reran experiments with this update direction, ∇ µi on the synthetic and MNIST domains.
Revised: We also reran experiments with this update direction, ∇ µi on the synthetic and MNIST domains.

Original: The update is unbiased, so it would be expected to scale well, however, it (in orange) appears to scale more poorly than µ -EigenGame with smaller minibatches
Revised: The update is unbiased, so it would be expected to scale well, however, it (in orange) appears to scale more poorly than µ -EigenGame with smaller minibatches

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: In Figure
Revised: In Figure

Original: [Figure]
Revised: [Figure]

Original: ˜ µ -EG appears to converge in terms of subspace error but slows in terms of longest eigenvector streak.
Revised: ˜ µ -EG appears to converge in terms of subspace error but slows in terms of longest eigenvector streak.

Original: ˜ µ -EG updates are also unbiased so we would expect it is convergent globally
Revised: ˜ µ -EG updates are also unbiased so we would expect it is convergent globally

Original: [Figure]
Revised: [Figure]

Original: but it underperforms relative to µ -EG.
Revised: but it underperforms relative to µ -EG.

Original: In contrast, α -EG stalls in terms of subspace error likely due to bias.
Revised: In contrast, α -EG stalls in terms of subspace error likely due to bias.

Original: Note that with exact parents, mu-EG and mu-tilde-EG have the same update (plug Cvj =
Revised: Note that with exact parents, mu-EG and mu-tilde-EG have the same update (plug Cvj =

Original: λ j vj into equation (46)), so the difference must come from when the parents are still inaccurate.
Revised: λ j vj into equation (46)), so the difference must come from when the parents are still inaccurate.

Original: In Figure 11, we have plotted the norm of the difference between subsequent values of the eigenvectors over training, i.e., how “far” v i moves after every update.
Revised: In Figure 11, we have plotted the norm of the difference between subsequent values of the eigenvectors over training, i.e., how “far” v i moves after every update.

Original: Note all algorithms were run with the same ﬁxed step size of 10 − 3 , which was optimal for each algorithm in this setting.
Revised: Note all algorithms were run with the same ﬁxed step size of 10 − 3 , which was optimal for each algorithm in this setting.

Original: Clearly, the gradient version of α -EigenGame ( ˜ µ -EG) shown in Figure 11c exhibits higher norms overall.
Revised: Clearly, the gradient version of α -EigenGame ( ˜ µ -EG) shown in Figure 11c exhibits higher norms overall.

Original: We believe this is due to the higher variance penalty terms (all methods maximize the same Rayleigh quotient term).
Revised: We believe this is due to the higher variance penalty terms (all methods maximize the same Rayleigh quotient term).

Original: Note that both α -EG and µ -EG construct their pentalty directions by a weighted sum of terms.
Revised: Note that both α -EG and µ -EG construct their pentalty directions by a weighted sum of terms.

Original: These terms are computed differently, but both compute weights with inner products between v i and v j after projecting onto the samples in the minibatch X t .
Revised: These terms are computed differently, but both compute weights with inner products between v i and v j after projecting onto the samples in the minibatch X t .

Original: For example, µ -EG computes (cid:104)
Revised: For example, µ -EG computes (cid:104)

Original: X t v i , X t v j (cid:105)
Revised: X t v i , X t v j (cid:105)

Original: = v (cid:62) i
Revised: = v (cid:62) i

Original: C t v j .
Revised: C t v j .

Original: Without loss of generality, assume C is a diagonal matrix (with the eigenvalues on its diagonal).
Revised: Without loss of generality, assume C is a diagonal matrix (with the eigenvalues on its diagonal).

Original: Then (cid:104) v (cid:62) i
Revised: Then (cid:104) v (cid:62) i

Original: Cv j (cid:105) =
Revised: Cv j (cid:105) =

Original: (cid:80) k
Revised: (cid:80) k

Original: λ k v ik v jk .
Revised: λ k v ik v jk .

Original: The eigenvectors v i = e i in this case, and so the inner product measures alignment between v i and v j in the dimensions that v i and v j are trained to be orthogonal.
Revised: The eigenvectors v i = e i in this case, and so the inner product measures alignment between v i and v j in the dimensions that v i and v j are trained to be orthogonal.

Original: Due to noise in the minibatches X t , v i and v j may “drift” in the remaining dimensions.
Revised: Due to noise in the minibatches X t , v i and v j may “drift” in the remaining dimensions.

Original: Projecting essentially ignores these though because they are weighted by small eigenvalues.
Revised: Projecting essentially ignores these though because they are weighted by small eigenvalues.

Original: In contrast, ˜ µ -EG computes weights as raw inner products between v i and v j .
Revised: In contrast, ˜ µ -EG computes weights as raw inner products between v i and v j .

Original: Therefore, any drift of v i and v j due to noise in the minibatch samples contributes to the inner product: (cid:104) v i , v j (cid:105) =
Revised: Therefore, any drift of v i and v j due to noise in the minibatch samples contributes to the inner product: (cid:104) v i , v j (cid:105) =

Original: (cid:80) k v ik v jk .
Revised: (cid:80) k v ik v jk .

Original: We suspect this is the reason ˜ µ -EG exhibits higher drift distance.
Revised: We suspect this is the reason ˜ µ -EG exhibits higher drift distance.

Original: In summary, α -EG updates are computed as a weighted sum of terms where the weights are computed using inner products between v i and its parents after projecting them to a lower dimensional space.
Revised: In summary, α -EG updates are computed as a weighted sum of terms where the weights are computed using inner products between v i and its parents after projecting them to a lower dimensional space.

Original: Computing the inner product in this particular space results in lower variance for each inner product.
Revised: Computing the inner product in this particular space results in lower variance for each inner product.

Original: Unfortunately, α -EG updates are biased, so while they exhibit relatively low “norm of drift”, they converge to the incorrect solution (parents never converge to precise solution which prohibits children from learning accurately solutions).
Revised: Unfortunately, α -EG updates are biased, so while they exhibit relatively low “norm of drift”, they converge to the incorrect solution (parents never converge to precise solution which prohibits children from learning accurately solutions).

Original: ˜ µ -EG updates are unbiased, so they should converge to the correct solution in the limit, but they exhibit higher variance due to their penalty weights being computed in the original high dimensional space (i.e., they pick up every little bit of noise).
Revised: ˜ µ -EG updates are unbiased, so they should converge to the correct solution in the limit, but they exhibit higher variance due to their penalty weights being computed in the original high dimensional space (i.e., they pick up every little bit of noise).

Original: Finally, µ -EG updates are unbiased and compute their penalty weights in a lower dimensional space, suppressing the bulk of the noise that appears from drift caused by randomness in the minibatches.
Revised: Finally, µ -EG updates are unbiased and compute their penalty weights in a lower dimensional space, suppressing the bulk of the noise that appears from drift caused by randomness in the minibatches.

Original: They exhibit the lowest levels of “norm of drift”.
Revised: They exhibit the lowest levels of “norm of drift”.

Original: [Figure]
Revised: [Figure]

Original: I. A CCELERATION
Revised: I. A CCELERATION

Original: We conjecture that µ -EigenGame converges more quickly than α -EigenGame because of the following two claims.
Revised: We conjecture that µ -EigenGame converges more quickly than α -EigenGame because of the following two claims.

Original: Claim 1 The penalty terms of ˜ ∇ µi are all within 90 ◦ of those of ∇ αi because (cid:68) C ˆ v j ˆ v (cid:62) j C ˆ v j , ˆ v j (cid:69) = 1 > 0 .
Revised: Claim 1 The penalty terms of ˜ ∇ µi are all within 90 ◦ of those of ∇ αi because (cid:68) C ˆ v j ˆ v (cid:62) j C ˆ v j , ˆ v j (cid:69) = 1 > 0 .

Original: Claim 2 The penalty terms of ˜ ∇ µi are all smaller in magnitude than those of ∇ αi : || ˆ v j || ≤ (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
Revised: Claim 2 The penalty terms of ˜ ∇ µi are all smaller in magnitude than those of ∇ αi : || ˆ v j || ≤ (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)

Original: C ˆ v j ˆ v (cid:62) j C ˆ v j (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
Revised: C ˆ v j ˆ v (cid:62) j C ˆ v j (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)

Original: Indeed, consider the direction C
Revised: Indeed, consider the direction C

Original: ˆ v j .
Revised: ˆ v j .

Original: By properties of the vector rejection, we know the rejection of this direction onto the tangent space of the unit sphere has magnitude less than or equal to that of the original vector,
Revised: By properties of the vector rejection, we know the rejection of this direction onto the tangent space of the unit sphere has magnitude less than or equal to that of the original vector,

Original: ||
Revised: ||

Original: C ˆ v j || .
Revised: C ˆ v j || .

Original: The projection is ( I − ˆ v j ˆ v (cid:62) j )( C ˆ v j ) .
Revised: The projection is ( I − ˆ v j ˆ v (cid:62) j )( C ˆ v j ) .

Original: Therefore, the rejection is ˆ v j ˆ v (cid:62) j
Revised: Therefore, the rejection is ˆ v j ˆ v (cid:62) j

Original: ( C ˆ v j ) and, by the preceding argument, we know its magnitude | ˆ v (cid:62) j C ˆ v j ||| ˆ v j || is less than or equal to ||
Revised: ( C ˆ v j ) and, by the preceding argument, we know its magnitude | ˆ v (cid:62) j C ˆ v j ||| ˆ v j || is less than or equal to ||

Original: C ˆ v j || .
Revised: C ˆ v j || .

Original: Rearranging the inequality completes the proof.
Revised: Rearranging the inequality completes the proof.

Original: By Claim 1 , the penalty directions of µ -EG and α -EG approximately agree.
Revised: By Claim 1 , the penalty directions of µ -EG and α -EG approximately agree.

Original: And by Claim 2 , α -EG’s penalty direction is shorter.
Revised: And by Claim 2 , α -EG’s penalty direction is shorter.

Original: Consider a scenario where a parent of ˆ v
Revised: Consider a scenario where a parent of ˆ v

Original: i has not converged and transiently occupies space along ˆ v i ’s geodesic to its true endpoint
Revised: i has not converged and transiently occupies space along ˆ v i ’s geodesic to its true endpoint

Original: ˆ v i , a strong penalty term will force ˆ v i to take a roundabout trajectory, thereby slowing its convergence.
Revised: ˆ v i , a strong penalty term will force ˆ v i to take a roundabout trajectory, thereby slowing its convergence.

Original: A weaker penalty term allows ˆ v i to pass through regions occupied by its parent as long as its parent is not an eigenvector.
Revised: A weaker penalty term allows ˆ v i to pass through regions occupied by its parent as long as its parent is not an eigenvector.

Original: Recall from
Revised: Recall from

Original: Section 3 that the two utilities are equivalent when the parents are eigenvectors.
Revised: Section 3 that the two utilities are equivalent when the parents are eigenvectors.