Original: E FFICIENT R IEMANNIAN O PTIMIZATION ON THE S TIEFEL M ANIFOLD VIA THE C AYLEY T RANSFORM
Revised: E FFICIENT R IEMANNIAN O PTIMIZATION ON THE S TIEFEL M ANIFOLD VIA THE C AYLEY T RANSFORM

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Strictly enforcing orthonormality constraints on parameter matrices has been shown advantageous in deep learning.
Revised: Strictly enforcing orthonormality constraints on parameter matrices has been shown advantageous in deep learning.

Original: This amounts to Riemannian optimization on the Stiefel manifold, which, however, is computationally expensive.
Revised: This amounts to Riemannian optimization on the Stiefel manifold, which, however, is computationally expensive.

Original: To address this challenge, we present two main contributions: (1) A new efﬁcient retraction map based on an iterative Cayley transform for optimization updates, and (2) An implicit vector transport mechanism based on the combination of a projection of the momentum and the Cayley transform on the Stiefel manifold
Revised: To address this challenge, we present two main contributions: (1) A new efﬁcient retraction map based on an iterative Cayley transform for optimization updates, and (2) An implicit vector transport mechanism based on the combination of a projection of the momentum and the Cayley transform on the Stiefel manifold.

Original: We specify two new optimization algorithms Cayley SGD with momentum and Cayley ADAM on the Stiefel manifold
Revised: We specify two new optimization algorithms: Cayley SGD with momentum, and Cayley ADAM on the Stiefel manifold.

Original: Convergence ofthe Cayley SGD is theoretically analyzed
Revised: Convergence of Cayley SGD is theoretically analyzed.

Original: Our experiments for CNN training demonstrate that both algorithms : (a) Use less running time per iteration relative to existing approaches which also enforce orthonormality of CNN parameters
Revised: Our experiments for CNN training demonstrate that both algorithms: (a) Use less running time per iteration relative to existing approaches that enforce orthonormality of CNN parameters; and (b) Achieve faster convergence rates than the baseline SGD and ADAM algorithms without compromising the performance of the CNN.

Original: ; and

Original: (b

Original: Achieve faster convergence rates than the baseline SGD and ADAM algorithms without compromising the CNN’s performance.

Original: The Cayley SGD and Cayley ADAM are also shown to reduce the training time for optimizing the unitary transition matrices in RNNs.
Revised: Cayley SGD and Cayley ADAM are also shown to reduce the training time for optimizing the unitary transition matrices in RNNs.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Orthonormality has recently gained much interest, as there are signiﬁcant advantages of enforcing orthonormality on parameter matrices of deep neural networks.
Revised: Orthonormality has recently gained much interest, as there are signiﬁcant advantages of enforcing orthonormality on parameter matrices of deep neural networks.

Original: For CNNs, Bansal et al.
Revised: For CNNs, Bansal et al.

Original: (2018) show that this improves accuracy and gives a faster empirical convergence rate, Huang et al.
Revised: (2018) show that orthonormality constraints improve accuracy and gives a faster empirical convergence rate, Huang et al.

Original: (2018a) show that orthonormality stabilizes the distribution of neural activations in training, and Cogswell et al.
Revised: (2018a) show that orthonormality stabilizes the distribution of neural activations in training and Cogswell et al.

Original: (2015) show that orthonormality reduces overﬁtting and improves generalization.
Revised: (2015) show that orthonormality reduces overﬁtting and improves generalization.

Original: For RNNs, Arjovsky et al.
Revised: For RNNs, Arjovsky et al.

Original: ; Zhou et al.
Revised: ; Zhou et al.

Original: (2006) show that the orthogonal hidden-to-hidden matrix alleviates the vanishing and exploding-gradient problems.
Revised: (2006) show that the orthogonal hidden-to-hidden matrix alleviates the vanishing and exploding-gradient problems.

Original: Riemannian optimization on the Stiefel manifold, which represents a set of  orthonormal matrices of the same size, is an elegant framework for optimization under orthonormality constraints.
Revised: Riemannian optimization on the Stiefel manifold, which represents the set of all orthonormal matrices of the same size, is an elegant framework for optimization under orthonormality constraints.

Original: But its high computational cost is limiting in many applications, including  deep learning.
Revised: But its high computational cost is limiting its applications, including in deep learning.

Original: Recent efﬁcient approaches incorporate orthogonality in deep learning only for square parameter matrices (Arjovsky et al., 2016; Dorobantu et al., 2016), or indirectly through regularization (Bansal et al., 2018).
Revised: Recent efﬁcient approaches incorporate orthogonality in deep learning only for square parameter matrices (Arjovsky et al., 2016; Dorobantu et al., 2016), or indirectly through regularization (Bansal et al., 2018), which however does not guarantee the exact orthonormality of parameter matrices.

Original: To address the aforementioned limitations, our ﬁrst main contribution is an efﬁcient estimation of the retraction mapping based on the Cayley transform for updating large non-square matrices of parameters on the Stiefel manifold.
Revised: To address the aforementioned limitations, our ﬁrst main contribution is an efﬁcient estimation of the retraction mapping based on the Cayley transform for updating large non-square matrices of parameters on the Stiefel manifold.

Original: We specify an efﬁcient iterative algorithm for estimating the
Revised: We specify an efﬁcient iterative algorithm for estimating the

Original: Cayley transform that consists of only a few matrix multiplications, while the closed-form Cayley transform requires costly matrix inverse operations (Vorontsov et al., 2017; Wisdom et al., 2016).
Revised: Cayley transform that consists of only a few matrix multiplications, while the closed-form Cayley transform requires costly matrix inverse operations (Vorontsov et al., 2017; Wisdom et al., 2016).

Revised: The efﬁciency of the retraction mapping is both theoretically proved and empirically veriﬁed in the paper.

Original: Our second main contribution is aimed at improving convergence rates of training by taking into account the momentum in our optimization on the Stiefel manifold.
Revised: Our second main contribution is aimed at improving convergence rates of training by taking into account the momentum in our optimization on the Stiefel manifold.

Original: We derive a new approach to move the tangent vector between tangent spaces of the manifold, instead of using the standard parallel transport.
Revised: We derive a new approach to move the tangent vector between tangent spaces of the manifold, instead of using the standard parallel transport.

Original: Speciﬁcally, we regard the Stiefel manifold as a submanifold of a Euclidean space.
Revised: Speciﬁcally, we regard the Stiefel manifold as a submanifold of a Euclidean space.

Original: This allows for representing the vector transport (Absil et al., 2009) as a projection onto the tangent space.
Revised: This allows for representing the vector transport (Absil et al., 2009) as a projection onto the tangent space.

Original: As we show, since the Cayley transform implicitly projects gradients onto the tangent space, the momentum updates result in an implicit vector transport.
Revised: As we show, since the Cayley transform implicitly projects gradients onto the tangent space, the momentum updates result in an implicit vector transport.

Original: Thus, we ﬁrst compute a linear combination of the momentum and  gradient in the Euclidean space, and then update the network parameters using the Cayley transform, without explicitly performing the vector transport.
Revised: Thus, we ﬁrst compute a linear combination of the momentum and the gradient in the Euclidean space, and then update the network parameters using the Cayley transform, without explicitly performing the vector transport.

Original: We apply the above two contributions to generalize the standard SGD with momentum and ADAM (Kingma & Ba, 2014) to the Stiefel manifold, resulting in our two new optimization algorithms called Cayley SGD with momentum and Cayley ADAM.
Revised: We apply the above two contributions to generalize the standard SGD with momentum and ADAM (Kingma & Ba, 2014) to the Stiefel manifold, resulting in our two new optimization algorithms called Cayley SGD with momentum and Cayley ADAM.

Original: A theoretical analysis of  convergence ofthe Cayley SGD is presented.
Revised: A theoretical analysis of the convergence of Cayley SGD is presented.

Original: Similar analysis forthe Cayley ADAM is omitted, since it is very similar to the analysis presented in (Becigneul & Ganea, 2019).
Revised: Similar analysis for Cayley ADAM is omitted, since it is very similar to the analysis presented in (Becigneul & Ganea, 2019).

Original: The Cayley SGD and Cayley ADAM are empirically evaluated on image classiﬁcation using VGG and Wide Resnet on the CIFAR-10 and CIFAR-100 datasets (Krizhevsky & Hinton, 2009).
Revised: Cayley SGD and Cayley ADAM are empirically evaluated on image classiﬁcation using VGG and Wide ResNet on the CIFAR-10 and CIFAR-100 datasets (Krizhevsky & Hinton, 2009).

Original: The experiments show thatthe Cayley SGD and Cayley ADAM achieve better classiﬁcation performance and faster convergence rate than the baseline SGD with momentum and ADAM.
Revised: The experiments show that Cayley SGD and Cayley ADAM achieve better classiﬁcation performance and faster convergence rate than the baseline SGD with momentum and ADAM.

Original: While the baselines take less time per epoch since they do not enforce the orthonormality constraint, they take more epochs until convergence thanthe Cayley SGD and Cayley ADAM.
Revised: While the baselines take less time per epoch since they do not enforce the orthonormality constraint, they take more epochs until convergence than Cayley SGD and Cayley ADAM.

Original: In comparison with existing optimization methods that also account for orthonormality – e.g., Polar decomposition, QR decomposition, or closed-form Cayley transform – our Cayley SGD and Cayley ADAM run much faster and yield equally good or better performance in image classiﬁcation.
Revised: In comparison with existing optimization methods that also account for orthonormality – e.g., Polar decomposition, QR decomposition, or closed-form Cayley transform – our Cayley SGD and Cayley ADAM run much faster and yield equally good or better performance in image classiﬁcation.

Original: Finally, we apply the aforementioned two contributions to training of the unitary RNNs.
Revised: Finally, we apply the aforementioned two contributions to training of the unitary RNNs.

Original: Wisdom et al.
Revised: Wisdom et al.

Original: (2016) proposes the full capacity unitary RNN that updates the hidden-to-hidden transition matrix with the closed-form Cayley transform.
Revised: (2016) proposes the full capacity unitary RNN that updates the hidden-to-hidden transition matrix with the closed-form Cayley transform.

Original: In contrast, for our RNN training, we use the more efﬁcient Cayley SGD with momentum and Cayley ADAM.
Revised: In contrast, for our RNN training, we use the more efﬁcient Cayley SGD with momentum and Cayley ADAM.

Original: The results show that our RNN training takes less running time per iteration without compromising performance.
Revised: The results show that our RNN training takes less running time per iteration without compromising performance.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: This section reviews closely related work on using orthonormal constraints in neural-network training, which can be broadly divided in two groups:  regularization and Riemannian optimization.
Revised: There is a host of literature on using orthonormality constraints in neural-network training. This section reviews closely related work, which can be broadly divided in two groups: orthonormality regularization and Riemannian optimization.

Original: Regularization approaches can be divided as hard, which strictly enforce orthonormality, and soft.
Revised: Regularization approaches can be divided as hard, which strictly enforce orthonormality, and soft.

Original: Hard regularizations are computationally expensive.
Revised: Hard regularizations are computationally expensive.

Original: For example, Huang et al.
Revised: For example, Huang et al.

Original: (2018b) extend 
Revised: (2018b) extend Batch

Original: Batch Normalization (Ioffe & Szegedy, 2015) withthe ZCA, and hence require costly eigenvalue decomposition.
Revised: Normalization (Ioffe & Szegedy, 2015) with ZCA, and hence require costly eigenvalue decomposition.

Original: Huang et al.
Revised: Huang et al.

Original: (2018a) derive a closed-form solution that also requires eigenvalue decomposition.
Revised: (2018a) derive a closed-form solution that also requires eigenvalue decomposition.

Original: Bansal et al.
Revised: Bansal et al.

Original: Cisse et al.
Revised: Cisse et al.

Original: ; Xiong et al.
Revised: ; Xiong et al.

Original: (2016) introduce mutual coherence regularization and spectral restricted isometry regularization; however, these regularizations are soft.
Revised: (2016) introduce mutual coherence regularization and spectral restricted isometry regularization; however, these regularizations are soft in that they can not guarantee orthonormality.

Original: Riemannian optimization guarantees that the solution respects orthonormality constraints.
Revised: Riemannian optimization guarantees that the solution respects orthonormality constraints.

Original: For example, Cho & Lee (2017) replace Batch Normalization layers in a CNN with Riemannian optimization on the Grassmann manifold G ( n, 1) , where the parameters are normalized but not orthonormalized.
Revised: For example, Cho & Lee (2017) replace Batch Normalization layers in a CNN with Riemannian optimization on the Grassmann manifold G ( n, 1) , where the parameters are normalized but not orthonormalized.

Original: Also, Vorontsov et al.
Revised: Also, Vorontsov et al.

Original: ; Wisdom et al.
Revised: ; Wisdom et al.

Original: ; Lezcano-Casado & Mart ´ ınez-Rubio (2019);
Revised: (2016); Lezcano-Casado & Mart´ınez-Rubio (2019);

Original: Helfrich et al.
Revised: Helfrich et al.

Original: (2017) perform Riemannian optimization on theunitary group  matrices to stablize RNNs training, but their technique cannot be applied to non-square parameter matrices.
Revised: (2017) perform Riemannian optimization on the group of unitary matrices to stablize RNNs training, but their technique cannot be applied to non-square parameter matrices.

Original: Becigneul & Ganea (2019) introduce a more general Riemannian optimization, but do not show how to efﬁciently perform this optimization on the Stiefel manifold.
Revised: Becigneul & Ganea (2019) introduce a more general Riemannian optimization, but do not show how to efﬁciently perform this optimization on the Stiefel manifold.

Original: The key challenge of Riemannian optimization is that exponential mapping — the standard step for estimating the next update point — is computationally expensive on the Stiefel manifold.
Revised: The key challenge of Riemannian optimization is that exponential mapping — the standard step for estimating the next update point — is computationally expensive on the Stiefel manifold.

Original: Some methods use an efﬁcient pseudo-geodesic retraction instead of the exponential mapping.
Revised: Some methods use an efﬁcient pseudo-geodesic retraction instead of the exponential mapping.

Original: For example, Absil & Malick (2012); Gawlik & Leok (2018); Manton (2002) use a projection-based method to map the gradient back to the Stiefel manifold that rely on computational expensive SVD.
Revised: For example, Absil & Malick (2012); Gawlik & Leok (2018); Manton (2002) use a projection-based method to map the gradient back to the Stiefel manifold that rely on computational expensive SVD.

Original: Other approaches are based on the closed-form Cayley transform (Fiori et al., 2012; Jiang & Dai, 2015;
Revised: Other approaches are based on the closed-form Cayley transform (Fiori et al., 2012; Jiang & Dai, 2015;

Original: Nishimori & Akaho, 2005; Zhu, 2017), but require the costly matrix inversion.
Revised: Nishimori & Akaho, 2005; Zhu, 2017), but require the costly matrix inversion.

Original: Also, Wen & Yin (2013) reduce the cost of the Cayley transform by making the restrictive assumption that the matrix size n × p is such that 2 p (cid:28) n .
Revised: Also, Wen & Yin (2013) reduce the cost of the Cayley transform by making the restrictive assumption that the matrix size n × p is such that 2 p (cid:28) n .

Original: Unfortunately, this algorithm is not efﬁcient when 2 p ≥ n .
Revised: Unfortunately, this algorithm is not efﬁcient when 2 p ≥ n .

Original:  P RELIMINARY
Revised:  P RELIMINARY

Original: This section brieﬂy reviews some well-known properties of the Riemannian and Stiefel manifolds.
Revised: This section brieﬂy reviews some well-known properties of the Riemannian and Stiefel manifolds.

Original: The interested reader is referred to Boothby (1986);
Revised: The interested reader is referred to Boothby (1986);

Original: Edelman et al.
Revised: Edelman et al.

Original: and references therein.
Revised: and references therein.

Original: 3. R IEMANNIAN M ANIFOLD
Revised: 3. R IEMANNIAN M ANIFOLD

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: Riemannian Manifold: A Riemannian manifold ( M , ρ ) is a smooth manifold M equipped with a Riemannian metric ρ deﬁned as the inner product on the tangent space T x M for each point x , ρ x ( · , · ) :
Revised: Riemannian Manifold: A Riemannian manifold ( M , ρ ) is a smooth manifold M equipped with a Riemannian metric ρ deﬁned as the inner product on the tangent space T x M for each point x , ρ x ( · , · ) :

Original: T x M × T x M → R .
Revised: T x M × T x M → R .

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: Geodesic, Exponential map and Retraction map: A geodesic is a locally shortest curve on a manifold.
Revised: Geodesic, Exponential map and Retraction map: A geodesic is a locally shortest curve on a manifold.

Original: An exponential map, Exp x ( · ) , maps a tangent vector, v ∈ T x , to a manifold,
Revised: An exponential map, Exp x ( · ) , maps a tangent vector, v ∈ T x , to a manifold,

Original: M .
Revised: M .

Original: Exp x ( tv ) represents a geodesic γ ( t ) : t ∈ [0 , 1] on a manifold, s.t. γ (0)
Revised: Exp x ( tv ) represents a geodesic γ ( t ) : t ∈ [0 , 1] on a manifold, s.t. γ (0)

Original: = x, ˙ γ (0) = v .
Revised: = x, ˙ γ (0) = v .

Original: A retraction map is deﬁned as a smooth mapping R x : T x M → M on a manifold iff R x (0) =
Revised: A retraction map is deﬁned as a smooth mapping R x : T x M → M on a manifold iff R x (0) =

Original: x and DR x (0)
Revised: x and DR x (0)

Original: =
Revised: =

Original: id T x M , where DR x denotes the derivative of R x , id T x M denotes an identity map deﬁned on T x M .
Revised: id T x M , where DR x denotes the derivative of R x , id T x M denotes an identity map deﬁned on T x M .

Original: It is easy to show that any exponential map is a retraction map.
Revised: It is easy to show that any exponential map is a retraction map.

Original: As computing an exponential map is usually expensive, a retraction map is often used as an efﬁcient alternative.
Revised: As computing an exponential map is usually expensive, a retraction map is often used as an efﬁcient alternative.

Original: Deﬁnition 3.
Revised: Deﬁnition 3.

Original: Parallel transport and vector transport: Parallel transport is a method to translate a vector along a geodesic on a manifold while preserving norm.
Revised: Parallel transport and vector transport: Parallel transport is a method to translate a vector along a geodesic on a manifold while preserving norm.

Original: A vector transport τ is a smooth map deﬁned on a retraction R of a manifold M , τ : T x M × T x M → T R ( η x ) , ( η x , ξ x ) (cid:55)→ τ η x ( ξ x ) .
Revised: A vector transport τ is a smooth map deﬁned on a retraction R of a manifold M , τ : T x M × T x M → T R ( η x ) , ( η x , ξ x ) (cid:55)→ τ η x ( ξ x ) .

Original: τ satisﬁes the following properties: (1) Underlying retraction: τ η x ( ξ x ) ∈ T R ( η x ) ; (2) Consistency: τ 0 x ξ x = ξ x , ∀ ξ x ∈ T x M ; (3) Linearity:
Revised: τ satisﬁes the following properties: (1) Underlying retraction: τ η x ( ξ x ) ∈ T [Equation] ( η x ) ; (2) Consistency: τ 0 x ξ x = ξ x , ∀ ξ x ∈ T x M ; (3) Linearity:

Original: τ η x ( aξ x + bζ x ) =
Revised: τ η x ( aξ x + bζ x ) =

Original: aτ η x ( ξ x ) + bτ η x ( ζ x ) .
Revised: aτ η x ( ξ x ) + bτ η x ( ζ x ) .

Original: Usually, a vector transport is a computationally efﬁcient alternative to a parallel transport.
Revised: Usually, a vector transport is a computationally efﬁcient alternative to a parallel transport.

Original: 3. S TIEFEL M ANIFOLD
Revised: 3. S TIEFEL M ANIFOLD

Original: The Stiefel manifold St ( n, p ) , n ≥ p , is a Riemannian manifold that is composed of all n × p orthonormal matrices { X ∈ R n × p : X T X = I } .
Revised: The Stiefel manifold St ( n, p ) , n ≥ p , is a Riemannian manifold that is composed of all n × p orthonormal matrices { X ∈ R n × p : X T X = I } .

Original: In the rest of the paper, we will use notation M =
Revised: In the rest of the paper, we will use notation M =

Original: St ( n, p ) to denote the Stiefel manifold.
Revised: St ( n, p ) to denote the Stiefel manifold.

Original: We regard M as an embeded submanifold of a Euclidean space.
Revised: We regard M as an embeded submanifold of a Euclidean space.

Original: Hence, the Riemannian metric ρ is the Euclidean metric as: ρ X ( Z 1 , Z 2 )
Revised: Hence, the Riemannian metric ρ is the Euclidean metric as: ρ X ( Z 1 , Z 2 )

Original: =
Revised: =

Original: tr ( Z (cid:62) 1 Z 2 ) , where
Revised: tr ( Z (cid:62) 1 Z 2 ) , where

Original: Z 1 , Z 2 are tangent vectors in T X M .
Revised: Z 1 , Z 2 are tangent vectors in T X M .

Original: The tangent space of M at X is deﬁned as
Revised: The tangent space of M at X is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: The projection of a matrix Z ∈ R n × p to T X M can be computed as
Revised: The projection of a matrix Z ∈ R n × p to T X M can be computed as

Original: [Equation]
Revised: [Equation]

Original: where:
Revised: where:

Original: W = ˆ W − ˆ W (cid:62)

Original: [Equation]
Revised: [Equation]

Original: Given a derivative of the objective function ∇ f ( X ) at X in the Euclidean space, we can compute the Riemannian gradient ∇ M f ( X ) on the Stiefel manifold as a projection onto T X M using π T X ( ∇ f ( X )) given by Eq. 2.
Revised: Given a derivative of the objective function ∇ f ( X ) at X in the Euclidean space, we can compute the Riemannian gradient ∇ M f ( X ) on the Stiefel manifold as a projection onto T X M using π T X ( ∇ f ( X )) given by Eq.(2).

Original: It follows that optimization of f on the Riemannian manifold can be computed as follows.
Revised: It follows that optimization of f on the Riemannian manifold can be computed as follows.

Original: First, compute ∇ M f ( X t ) in T X M .
Revised: First, compute ∇ M f ( X t ) in T X M .

Original: Second, transport the momentum M t to the current tangent space T X M and combine it linearly with the current Riemannian gradient ∇ M f ( X t ) to update the momentum M t +1 .
Revised: Second, transport the momentum M t to the current tangent space T X M and combine it linearly with the current Riemannian gradient ∇ M f ( X t ) to update the momentum M t +1 .

Original: Finally, third, update the new parameter X t +1 along a curve on the manifold with the initial direction as M t +1 .
Revised: Finally, third, update the new parameter X t +1 along a curve on the manifold with the initial direction as M t +1 .

Original: While the exponential map and parallel transport can be used to update parameters and momentums in optimization on the Riemannian manifold, they are computationally infeasible on the Stiefel manifold.
Revised: While the exponential map and parallel transport can be used to update parameters and momentums in optimization on the Riemannian manifold, they are computationally infeasible on the Stiefel manifold.

Original: In the following section, we specify our computationally efﬁcient alternatives.
Revised: In the following section, we specify our computationally efﬁcient alternatives.

Original: 3.2. P ARAMETER U PDATES BY I TERATIVE C AYLEY T RANSFORM
Revised: 3.2. P ARAMETER U PDATES BY I TERATIVE C AYLEY T RANSFORM

Original: The Cayley transform computes a parametric curve on the Stiefel manifold using a skew-symmetric matrix (Nishimori & Akaho, 2005).
Revised: The Cayley transform computes a parametric curve on the Stiefel manifold using a skew-symmetric matrix (Nishimori & Akaho, 2005).

Original: The closed-form of the Cayley transform is given by:
Revised: The closed-form of the Cayley transform is given by:

Original: [Equation]
Revised: [Equation]

Original: where W is a skew-symmetric matrix,  X is on the Stiefel manifold, and α ≥ 0 is a parameter that represents  length on the curve.
Revised: where W is a skew-symmetric matrix, i.e. W (cid:62) = − W , X is on the Stiefel manifold, and α ≥ 0 is a parameter that represents the length on the curve.

Original: It is straightforward to verify that
Revised: It is straightforward to verify that

Original: [Equation]
Revised: [Equation]

Original: From Deﬁnition 2 and the deﬁnition of the tangent space of the Stiefel manifold given by Eq. 1, the
Revised: From Deﬁnition 2 and the deﬁnition of the tangent space of the Stiefel manifold given by Eq.(1), the

Original: Cayley transform is a valid retraction map on the Stiefel manifold.
Revised: Cayley transform is a valid retraction map on the Stiefel manifold.

Original: By choosing W =
Revised: By choosing W =

Original: ˆ W − ˆ W (cid:62) ,[Equation] where ˆ W = ∇ f ( X ) X (cid:62) − 12 X
Revised: ˆ W − ˆ W (cid:62) , where ˆ W = ∇ f ( X ) X (cid:62) − 12 X

Original: ( X (cid:62) ∇ f ( X ) X (cid:62) ) as in Eq. 2, we see that the Cayley transform implicitly projects gradient on the tangent space as its initial direction.
Revised: ( X (cid:62) ∇ f ( X ) X (cid:62) ) as in Eq.(2), we see that the Cayley transform implicitly projects gradient on the tangent space as its initial direction.

Original: Therefore, the Cayley transform can represent an update for the parameter matrices on the Stiefel manifold.
Revised: Therefore, the Cayley transform can represent an update for the parameter matrices on the Stiefel manifold.

Original: However, the closed-form Cayley transform in Eq. 3 involves computing the expensive matrix inversion, which cannot be efﬁciently performed in large deep neural networks.
Revised: However, the closed-form Cayley transform in Eq.(3) involves computing the expensive matrix inversion, which cannot be efﬁciently performed in large deep neural networks.

Original: Our ﬁrst contribution is an iterative estimation of the Cayley transform that efﬁciently uses only matrix multiplications, and thus is more efﬁcient than the closed form in Eq.
Revised: Our ﬁrst contribution is an iterative estimation of the Cayley transform that efﬁciently uses only matrix multiplications, and thus is more efﬁcient than the closed form in Eq.(3).

Original: We represent the
Revised: We represent the

Original: Cayley transform with the following recurrence relationship:
Revised: Cayley transform with the following ﬁxed-point iteration:

Original: [Equation]
Revised: [Equation]

Original:  which can be proved by moving Y ( α ) on the right-hand side to the left-hand side in Eq.
Revised: (5) which can be proved by moving Y ( α ) on the right-hand side to the left-hand side in Eq.(3).

Original: The expression in Eq. 5 is an efﬁcient approximation of Eq. 3.
Revised: The expression in Eq.(5) is an efﬁcient approximation of Eq.(3).

Original: In Sec. 5, we will analyze its convergence rate to the closed-form Eq. 3.
Revised: In Sec. 5, we will analyze its convergence rate to the closed-form Eq.(3).

Original: 3.2. M OMENTUM U PDATES BY THE I MPLICIT V ECTOR T RANSPORT
Revised: 3.2. M OMENTUM U PDATES BY THE I MPLICIT V ECTOR T RANSPORT

Original: Our second contribution is an efﬁcient way to perform momentum updates on the Stiefel manifold.
Revised: Our second contribution is an efﬁcient way to perform momentum updates on the Stiefel manifold.

Original: We specify an implicit vector transport by combining the Cayley transform and momentum update in an elegant way without explicitly computing the vector transport.
Revised: We specify an implicit vector transport by combining the Cayley transform and momentum updates in an elegant way without explicitly computing the vector transport.

Original: As the Stiefel manifold can be viewed as a submanifold of the Euclidean space R , we have a natural inclusion of the tangent space T X M ⊂ R .
Revised: As the Stiefel manifold can be viewed as a submanifold of the Euclidean space R , we have a natural inclusion of the tangent space T X M ⊂ R .

Original: Consequently, the vector transport on the Stiefel manifold is the projection on the tangent space.
Revised: Consequently, the vector transport on the Stiefel manifold is the projection on the tangent space.

Original: Formally, for two tangent vectors ξ X , η X ∈ T X M , the vector transport of ξ X along a retraction map r ( η X ) , denoted as τ η X ( ξ X ) , can be computed as:
Revised: Formally, for two tangent vectors ξ X , η X ∈ T X M , the vector transport of ξ X along a retraction map r ( η X ) , denoted as τ η X ( ξ X ) , can be computed as:

Original: [Equation]
Revised: [Equation]

Original: We specify the retraction map r ( · ) as the Cayley transform Y in Eq. 3.
Revised: We specify the retraction map r ( · ) as the Cayley transform Y in Eq.(3).

Original: At optimization step k , in
Revised: At optimization step k , in

Original: Eq. 6, we choose ξ X = η X = M k , where M k is the momentum in step k − 1 .
Revised: Eq.(6) , we choose ξ X = η X = M k , where M k is the momentum in step k − 1 .

Original: Then, we compute the vector transport of M k as τ M k ( M k ) =
Revised: Then, we compute the vector transport of M k as τ M k ( M k ) =

Original: π T Xk ( M k ) .
Revised: π T X k ( M k ) .

Original: As the projection onto a tangent space is a linear map, then ατ M k ( M k ) + [Equation] M 
Revised: As the projection onto a tangent space is a linear map, then ατ M k ( M k ) + β ∇ M f ( X k )

Original: [Equation] X [Equation]
Revised: =

Original: ) = [Equation] M 
Revised: απ T Xk ( M k )

Original: [Equation] + βπ T Xk [Equation] ∇ f ( X k ))
Revised: + βπ T Xk ( ∇ f ( X k ))

Original: =
Revised: =

Original: π T Xk ( αM k + β ∇ f ( X k )) .
Revised: π T Xk ( αM k + β ∇ f ( X k )) .

Original: Thus we ﬁrst compute a linear combination of the Euclidean gradient ∇ f ( X ) and the momentum M k , as in the Euclidean space, and then use the iterative Cayley transform to update the parameters, without explicitly estimating the vector transport, since the Cayley transform implicitly project  vector onto the tangent space.
Revised: Thus we ﬁrst compute a linear combination of the Euclidean gradient ∇ f ( X ) and the momentum M k , as in the Euclidean space, and then use the iterative Cayley transform to update the parameters, without explicitly estimating the vector transport, since the Cayley transform implicitly project a vector onto the tangent space.

Original:  A LGORITHMS
Revised:  A LGORITHMS

Original: This section speciﬁes our Cayley SGD and Cayley ADAM algorithms.
Revised: This section speciﬁes our Cayley SGD and Cayley ADAM algorithms.

Original: Both represent our efﬁcient
Revised: Both represent our efﬁcient

Original: Riemannian optimization on the Stiefel manifold that consists of two main steps.
Revised: Riemannian optimization on the Stiefel manifold that consists of two main steps.

Original: As the Cayley transform implicitly projects gradient and momentum vectors onto the tangent space, we ﬁrst linearly combine the momentum of the previous iteration with the stochastic gradient of the objective function f at the current point X , denoted as G ( X ) .
Revised: As the Cayley transform implicitly projects gradient and momentum vectors onto the tangent space, we ﬁrst linearly combine the momentum of the previous iteration with the stochastic gradient of the objective function f at the current point X , denoted as G ( X ) .

Original: Then, we use the iterative Cayley transform to estimate the next optimization point based on the updated momentum.
Revised: Then, we use the iterative Cayley transform to estimate the next optimization point based on the updated momentum.

Original: This is used to generalize the conventional SGD with momentum and ADAM algorithms to our two new Riemannian optimizations on the Stiefel manifold, as described in Section 4.1 and Section 4.2.
Revised: This is used to generalize the conventional SGD with momentum and ADAM algorithms to our two new Riemannian optimizations on the Stiefel manifold, as described in Section 4.1 and Section 4.2.

Original: We generalize the heavy ball (HB) momentum update (Ghadimi et al., 2015; Zavriev & Kostyuk, 1993) in the k th optimization step 1 to the Stiefel manifold. Theoretically, it can be represented as:
Revised: [Table]

Original: [Equation]
Revised: [Table]

Original: where Y ( α, X k , W k ) is a curve that starts at X k with length α on the Stiefel manifold, speciﬁed by the Cayley transform in Eq. In practice, we efﬁciently perform the updates in Eq. 7 by the
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: 1 − β k 2 ) (cid:46) Update biased second raw moment estimate 7: r ← (1 − β k 1 ) (cid:112) ˆ v k +

Revised: (cid:46) Update biased second raw moment estimate 7: r ← (1 − β k 1 ) (cid:112) ˆ v k +

Original: + (cid:15) (cid:46) Estimate biased-corrected ratio 8: ˆ W k ← M k +1 X (cid:62)
Revised: + (cid:15) (cid:46) Estimate biased-corrected ratio 8: ˆ W k ← M k +1 X (cid:62)

Original: k − 12 X k
Revised: k − 12 X k

Original: ( X (cid:62) k M k
Revised: ( X (cid:62) k M k

Original: +1 X (cid:62) k )
Revised: +1 X (cid:62) k )

Original: (cid:46) Compute the auxiliary skew-symmetric matrix 9:
Revised: (cid:46) Compute the auxiliary skew-symmetric matrix 9:

Original: W k ← ( ˆ W k − ˆ W (cid:62) k ) /r 10:
Revised: W k ← ( ˆ W k − ˆ W (cid:62) k ) /r 10:

Original: M k +1 ← rW k X k (cid:46) Project momentum onto the tangent space 11:
Revised: M k +1 ← rW k X k (cid:46) Project momentum onto the tangent space 11:

Original: α ← min { lr, 2 q/ ( (cid:107) W k (cid:107) + (cid:15) ) } (cid:46) Select adaptive learning rate for contraction mapping 12:
Revised: α ← min { l, 2 q/ ( (cid:107) W k (cid:107) + (cid:15) ) } (cid:46) Select adaptive learning rate for contraction mapping 12:

Original: Initialize Y 

Revised: Initialize Y

Original: ← X k − αM k
Revised: ← X k − αM k

Original: + (cid:46)
Revised: + (cid:46)

Original: Iterative estimation of the Cayley Transform 13: for i = 1 to s do 14:
Revised: Iterative estimation of the Cayley Transform 13: for i = 1 to s do 14:

Original: Y i ←
Revised: Y i ←

Original: X k − α 2 W
Revised: X k − α 2 W

Original: ( X k + Y i − 1 ) 15:
Revised: ( X k + Y i − 1 ) 15:

Original: Update X k +1 ← Y s proposed iterative Cayley transform and implicit vector transport on the Stiefel manifold. 
Revised: Update X k +1 ← Y s 4. C AYLEY SGD WITH M OMENTUM We generalize the heavy ball (HB) momentum update (Ghadimi et al., 2015; Zavriev & Kostyuk, 1993) in the k th optimization step 1 to the Stiefel manifold. Theoretically, it can be represented as:

Revised: [Equation]

Revised: where Y ( α, X k , W k ) is a curve that starts at X k with length α on the Stiefel manifold, speciﬁed by the Cayley transform in Eq.(3).

Revised: In practice, we efﬁciently perform the updates in Eq.(7) by the proposed iterative Cayley transform and implicit vector transport on the Stiefel manifold.

Original: Specially, we ﬁrst update the momentum as if it were in the Euclidean space.
Revised: Specially, we ﬁrst update the momentum as if it were in the Euclidean space.

Original: Then, we update the new parameters by iterative Cayley transform.
Revised: Then, we update the new parameters by iterative Cayley transform.

Original: Finally, we correct the momentum M k +1 by projecting it to T X k M .
Revised: Finally, we correct the momentum M k +1 by projecting it to T X k M .

Original: The details of our Cayley SGD algorithm are shown in Alg. 1.
Revised: The details of our Cayley SGD algorithm are shown in Alg. 1.

Original: 4.T HE ADAM ON THE S TIEFEL M ANIFOLD
Revised: 4. ADAM ON THE S TIEFEL M ANIFOLD

Original: ADAM is a recent ﬁrst-order optimization method for stochastic objective functions.
Revised: ADAM is a recent ﬁrst-order optimization method for stochastic objective functions.

Original: It estimates adaptive lower-order moments and uses adaptive learning rate.
Revised: It estimates adaptive lower-order moments and uses adaptive learning rate.

Original: The algorithm is designed to combine the advantages of both AdaGrad, which performs well in sparse-gradient cases, and RMSProp, which performs well in non-stationary cases.
Revised: The algorithm is designed to combine the advantages of both AdaGrad, which performs well in sparse-gradient cases, and RMSProp, which performs well in non-stationary cases.

Original: We generalize ADAM to the Stiefel manifold by making three modiﬁcations to the vanilla ADAM.
Revised: We generalize ADAM to the Stiefel manifold by making three modiﬁcations to the vanilla ADAM.

Original: First, we replace the standard gradient and momentum with the corresponding ones on the Stiefel manifold, as described in Section 4.1.
Revised: First, we replace the standard gradient and momentum with the corresponding ones on the Stiefel manifold, as described in Section 4.1.

Original: Second, we use a manifold-wise adaptive learning rate that assign a same learning rate for all entries in a parameter matrix as in (Absil et al., 2009).
Revised: Second, we use a manifold-wise adaptive learning rate that assign a same learning rate for all entries in a parameter matrix as in (Absil et al., 2009).

Original: Third, we use the Cayley transform to update the parameters.
Revised: Third, we use the Cayley transform to update the parameters.

Original: The Cayley ADAM is summarized in Alg 2.
Revised: Cayley ADAM is summarized in Alg 2.

Original:  C ONVERGENCE A NALYSIS
Revised:  C ONVERGENCE A NALYSIS

Original: In this section, we analyze convergence of the iterative Cayley transform and Cayley SGD with momentum.
Revised: In this section, we analyze convergence of the iterative Cayley transform and Cayley SGD with momentum.

Original: To facilitate our analysis, we make the following common assumption.
Revised: To facilitate our analysis, we make the following common assumption.

Original: Assumption 1.
Revised: Assumption 1.

Original: The gradient ∇ f of the objective function f is Lipschitz continuous
Revised: The gradient ∇ f of the objective function f is Lipschitz continuous

Original: [Equation]
Revised: [Equation]

Original: where L > 0 is a constant.
Revised: L [Equation] 0 is a constant

Original: [Equation]
Revised: [Equation]

Original: Lipschitz continuous is widely applicable to deep learning architectures.
Revised: Lipschitz continity is widely applicable to deep learning architectures.

Original: For some models using
Revised: For some models using

Original: ReLU, the derivative of ReLU is Lipschitz continuous almost everywhere with an appropriate Lipschitz constant L in Assumption 1 , except for a small neighbourhood around 0, whose measure tends to 0 .
Revised: ReLU, the derivative of ReLU is Lipschitz continuous almost everywhere with an appropriate Lipschitz constant L in Assumption 1 , except for a small neighbourhood around 0, whose measure tends to 0 .

Original: Such cases do not affect either analysis in theory or training in practice.
Revised: Such cases do not affect either analysis in theory or training in practice.

Original: The above assumption allows proving the following contraction mapping theorem.
Revised: The above assumption allows proving the following contraction mapping theorem.

Original: Theorem 1.
Revised: Theorem 1.

Original: For α ∈ (0 , min { 1 , 2 (cid:107) W (cid:107) } ) , the iteration Y
Revised: For α ∈ (0 , min { 1 , 2 (cid:107) W (cid:107) } ) , the iteration Y

Original: i +

Revised: i +

Original: =
Revised: =

Original: X + α 2 W (cid:0) X
Revised: X + α 2 W (cid:0) X

Original: + Y
Revised: + Y

Original: i (cid:1) is a contraction mapping and converges to the closed-form Cayley transform Y ( α ) given by Eq. 3.
Revised: i (cid:1) is a contraction mapping and converges to the closed-form Cayley transform Y ( α ) given by Eq.(3).

Original: Speciﬁcally, at iteration i , ||
Revised: Speciﬁcally, at iteration i , ||

Original: Y
Revised: Y

Original: i − Y ( α ) ||
Revised: i − Y ( α ) ||

Original: =
Revised: =

Original: o ( α 2+ i ) .
Revised: o ( α 2+ i ) .

Original: Theorem 1 shows the iterative Cayley transform will converge.
Revised: Theorem 1 shows the iterative Cayley transform will converge.

Original: Specially, it converges faster than other approximation algorithms, such as, e.g., the Newton iterative and Neumann Series which achieves error bound o
Revised: Specially, it converges faster than other approximation algorithms, such as, e.g., the Newton iterative and Neumann Series which achieves error bound o

Original: ( α i ) at the i th iteration.
Revised: ( α i ) at the i th iteration.

Original: We further prove the following result on convergence:
Revised: We further prove the following result on convergence:

Original: Theorem 2.
Revised: Theorem 2.

Original: Given an objective function f ( X ) that satisﬁes Assumption 1, letthe Cayley SGD with momentum run for t iterations with G ( X k ) .
Revised: Given an objective function f ( X ) that satisﬁes Assumption 1, let Cayley SGD with momentum run for t iterations with G ( X k ) .

Original: For α = min { 1 − βL , A √ t +1 } , where A is a positive constant, we have: min k =0 , ··· ,t E [ (cid:107)∇ M f ( X k ) (cid:107) 2 ]
Revised: For α = min { 1 − βL , A √ t +1 } , where A is a positive constant, we have: min k =0 , ··· ,t E [ (cid:107)∇ M f ( X k ) (cid:107) 2 ]

Original: = o ( 1 √ t +1 )
Revised: = o ( 1 √ t +1 )

Original: → 0 , as t → ∞ .
Revised: → 0 , as t → ∞ .

Original: The proofs of Theorem 1 and Theorem 2 are presented in the appendix.
Revised: The proofs of Theorem 1 and Theorem 2 are presented in the appendix.

Original:  E XPERIMENTS 6. ORTHONORMALITY IN CNN S
Revised:  E XPERIMENTS 6. ORTHONORMALITY IN CNN S

Original: In CNNs, for a convolutional layer with kernel ˆ K ∈ R c out × c in × h × w , we ﬁrst reshape ˆ K into a matrix K of size p × n , where p =
Revised: In CNNs, for a convolutional layer with kernel ˆ K ∈ R c out × c in × h × w , we ﬁrst reshape ˆ K into a matrix K of size p × n , where p =

Original: c out , n = c in × h × w .
Revised: c out , n = c in × h × w .

Original: In most cases, we have p ≤ n .
Revised: In most cases, we have p ≤ n .

Original: Then, we restrict the matrix K on the Stiefel manifold usingthe Cayley SGD or Cayley ADAM, while other parameters are optimized withthe SGD and ADAM.
Revised: Then, we restrict the matrix K on the Stiefel manifold using Cayley SGD or Cayley ADAM, while other parameters are optimized with SGD and ADAM.

Original: Datasets: We evaluatethe Cayley SGD or Cayley ADAM in image classiﬁcation on the CIFAR and CIFAR100 datasets (Krizhevsky & Hinton, 2009).
Revised: Datasets: We evaluate Cayley SGD or Cayley ADAM in image classiﬁcation on the CIFAR10 and CIFAR100 datasets (Krizhevsky & Hinton, 2009).

Original: CIFAR10 and CIFAR100 consist of of 50, training images and 10,000 test images, and have 10 and 100 mutually exclusive classes.
Revised: CIFAR10 and CIFAR100 consist of of 50, training images and 10,000 test images, and have 10 and 100 mutually exclusive classes.

Original: Models: We use two networks — Vgg (Simonyan & Zisserman, 2014) and Wide Resnet (Zagoruyko & Komodakis, 2016) — that give the state-of-the-art performance on CIFAR10 and CIFAR100.
Revised: Models: We use two networks — VGG (Simonyan & Zisserman, 2014) and Wide ResNet (Zagoruyko & Komodakis, 2016) — that obtain state of the art performance on CIFAR10 and CIFAR100.

Original: For Vgg, every convolutional layer is followed by a batch normalization layer and a ReLU.
Revised: For VGG, every convolutional layer is followed by a batch normalization layer and a ReLU.

Original: For Wide Resnet, we use basic blocks, where two consecutive 3-by-3 convolutional layers are followed by the batch normalization and ReLU activation, respectively.
Revised: For Wide ResNet, we use basic blocks, where two consecutive 3-by-3 convolutional layers are followed by the batch normalization and ReLU activation, respectively.

Original: Training Strategies: We use different learning rates lr e and lr st for weights on the Euclidean space and the Stiefel manifold, respectively.
Revised: Training Strategies: We use different learning rates l e and l st for weights on the Euclidean space and the Stiefel manifold, respectively.

Original: We set the weight decay as 0.0005, momentum as 0.9, and minibatch size as 128.
Revised: We set the weight decay as 0.0005, momentum as 0.9, and minibatch size as 128.

Original: The initial learning rates are set as lr e = 0 .
Revised: The initial learning rates are set as l e = 0 .

Original: 01 and lr st = 0 .
Revised: 01 and l st = 0 .

Original: 2 forthe Cayley SGD and lr e = 0 . 01 and lr st = 0 .
Revised: 2 for Cayley SGD and l e = 0 . 01 and l st = 0 .

Original: 4 forthe Cayley ADAM.
Revised: 4 for Cayley ADAM.

Original: During training, we reduce the learning rates by a factor of 0 .
Revised: During training, we reduce the learning rates by a factor of 0 .

Original: 2 at 60, 120,  160 epochs.
Revised: 2 at 60, 120, and 160 epochs.

Original: The total number of epochs in training is 200.
Revised: The total number of epochs in training is 200.

Original: In training, the data samples are normalized using the mean and variance of the training set, and augmented by randomly ﬂipping training images.
Revised: In training, the data samples are normalized using the mean and variance of the training set, and augmented by randomly ﬂipping training images.

Original: Our baselines includethe SGD with momentum and ADAM.
Revised: Our baselines include SGD with momentum and ADAM.

Original: We follow the same training strategies as mentioned above, except for the initial learning rates set to 0.1 and 0.001, respectively. 
Revised: We follow the same training strategies as mentioned above, except for the initial learning rates set to 0.1 and 0.001, respectively. [Table] [Table]

Original: Performance:
Revised: Performance:

Original: Table 1 and Table 2 show classiﬁcation errors on CIFAR10 and CIFAR100 respectively using different optimization algorithms.
Revised: Table 1 and Table 2 show classiﬁcation errors on CIFAR10 and CIFAR100 respectively using different optimization algorithms.

Original: As shown in the tables, the proposed two algorithms achieve competitive performance, and for certain deep architectures, the best performance.
Revised: As shown in the tables, the proposed two algorithms achieve competitive performance [Table] and for certain deep architectures, the best performance.

Original: Speciﬁcally, the network WRN-28-10 trained withthe Cayley ADAM achieves the best error rate of 3 .[Table]
Revised: Specifically, the network WRN-28-10 trained with Cayley ADAM achieves the best error rate of 3 .

Original: [Table] and 18 .
Revised: 57% and 18 .

Original: 10% on CIFAR10 and CIFAR100 respectively.
Revised: 10% on CIFAR10 and CIFAR100 respectively.

Original: Fig.
Revised: Fig.

Original: 1 compares training curves of our algorithms and baselines, and shows that boththe Cayley SGD and Cayley ADAM converge faster than the baselines.
Revised: 1 compares training curves of our algorithms and baselines in terms of epochs [Table] and shows that both Cayley SGD and Cayley ADAM converge faster than the baselines.

Original: In particular, the training curves of the baselines tend to get stuck in a plateau before the learning rate drops, which is not the case with our algorithms.
Revised: In particular, the training curves of the baselines tend to get stuck in a plateau before the learning rate drops, which is not the case with our algorithms.

Original: This might be because the baselinesto do not enforce orthonormality of network parameters.
Revised: This might be because the baselines do not enforce orthonormality of network parameters.

Original: In training, the backpropagation of orthonormal weight vectors, in general, does not affect each other, and thus has greater chances to explore new parameter regions. 
Revised: In training, the backpropagation of orthonormal weight vectors, in general, does not affect each other, and thus has greater chances to explore new parameter regions. Fig.

Revised: 2 also compares the training loss curve in terms of time.

Revised: Our Cayley SGD and Cayley ADAM converge the fastest among methods that also address orthonormality.

Revised: Although the baselines SGD and ADAM converge faster at the beginning due to their training efﬁciency, our Cayley SGD and Cayley ADAM can catch up with the baseline after 12000 seconds, which corresponds to the 120th epoch of SGD and ADAM, and the 60th epoch of Cayley SGD and Cayley ADAM.

Original: [Figure]
Revised: [Figure]

Original: Comparison with State of the Art.
Revised: Comparison with State of the Art.

Original: We compare the proposed algorithms with two sets of state of the art.
Revised: We compare the proposed algorithms with two sets of state of the art.

Original: The ﬁrst set of approaches are soft orthonormality regularization approaches (Bansal et al., 2018).
Revised: The ﬁrst set of approaches are soft orthonormality regularization approaches (Bansal et al., 2018).

Original: Specially, for a weight matrix K ∈ R n × p , SO penalizes
Revised: Specially, for a weight matrix K ∈ R n × p , SO penalizes

Original: ||
Revised: ||

Original: KK (cid:62) −
Revised: KK (cid:62) −

Original: I ||
Revised: I ||

Original: 2 F , DSO penalizes ( || KK (cid:62) −
Revised: 2 F , DSO penalizes ( || KK (cid:62) −

Original: I ||
Revised: I ||

Original: 2 F + || K (cid:62) K −
Revised: 2 F + || K (cid:62) K −

Original: I || 2 F ), the SRIP penalizes the spectral norm of ( KK (cid:62) − I ) .
Revised: I || 2 F ), the SRIP penalizes the spectral norm of ( KK (cid:62) − I ) .

Original: The second set of approaches includes the following hard orthonormality methods: Polar decomposition(Absil et al., 2009), QR decomposition(Absil et al., 2009), closed-form Cayley transform, Wen&Yin
Revised: The second set of approaches includes the following hard orthonormality methods: Polar decomposition(Absil et al., 2009), QR decomposition(Absil et al., 2009), closed-form Cayley transform, Wen&Yin

Original: (Wen & Yin, 2013), OMDSM(Huang et al., 2018a), DBN(Huang et al., 2018b).
Revised: (Wen & Yin, 2013), OMDSM(Huang et al., 2018a), DBN(Huang et al., 2018b).

Original: Note that we do not include momentum in Polar decomposition and QR decomposition as previous work does not specify
Revised: Note that we do not in-

Original: [Table]
Revised: [Figure]

Original: [Table] the momentum.
Revised: clude momentum in Polar decomposition and QR decomposition as previous work does not specify the momentum.

Original: Also, we use the closed-form Cayley transform without momentum as an ablation study of the momentum effect.
Revised: Also, we use the closed-form Cayley transform without momentum as an ablation study of the momentum effect.

Original: All experiments are evaluated on the benchmark network WRN-2810.
Revised: All experiments are evaluated on the benchmark network WRN28-10.

Original: Table 3 shows that our algorithms achieve comparable error rates withthe state of art work.
Revised: Table 3 shows that our algorithms achieve comparable error rates with state of the art.

Original: Moreover, we run all algorithms  on one TITAN Xp GPU, andcompare their average training time  per epoch.
Revised: All algorithms are run on one TITAN Xp GPU, and their average training time are compared per epoch.

Original: Table 3 shows that our algorithms run much faster than existing algorithms, except for the baseline SCD and ADAM which do not impose orthonormality constraints.
Revised: Table 3 shows that our algorithms run much faster than existing algorithms, except for the baseline SGD and ADAM which do not impose orthonormality constraints.

Original: 6. ORTHONORMALITY IN RNN S
Revised: 6. ORTHONORMALITY IN RNN S

Original: In RNNs, the hidden-to-hidden transition matrix K can be modeled as  unitary matrix (Arjovsky et al., 2016).
Revised: In RNNs, the hidden-to-hidden transition matrix K can be modeled as a unitary matrix (Arjovsky et al., 2016).

Original: Wisdom et al.
Revised: Wisdom et al.

Original: (2016) model the hidden-to-hidden transition matrix as a full-capacity unitary matrix on the complex Stiefel manifold: St ( C N ) =
Revised: (2016) model the hidden-to-hidden transition matrix as a full-capacity unitary matrix on the complex Stiefel manifold: St ( C N ) =

Original: { K ∈ C N × N : K H K
Revised: { K ∈ C N × N : K H K

Original: = I } .
Revised: = I } .

Original: Pixel-by-pixel MNIST:
Revised: Pixel-by-pixel MNIST:

Original: We evaluate the proposed algorithms on the challenging pixel-by-pixel
Revised: We evaluate the proposed algorithms on the challenging pixel-by-pixel

Original: MNIST task for long-term memory.
Revised: MNIST task for long-term memory.

Original: The task was used to test the capacity of uRNNs (Arjovsky et al., 2016;
Revised: The task was used to test the capacity of uRNNs (Arjovsky et al., 2016;

Original: Wisdom et al., 2016).
Revised: Wisdom et al., 2016).

Original: Following the same setting as in Wisdom et al.
Revised: Following the same setting as in Wisdom et al.

Original: , we reshape MNIST images of 28 × 28 pixels to a T = 784 pixel-by-pixel sequences, and select 5000 out of the 60000 training examples for the early stopping validation.
Revised: , we reshape MNIST images of 28 × 28 pixels to a T = 784 pixel-by-pixel sequences, and select 5,000 out of the 60,000 training examples for the early stopping validation.

Original: Training: Wisdom et al.
Revised: Training: Wisdom et al.

Original: (2016) restricted the transition unitary matrix on the Stiefel manifold via  closed-form Cayley transform.
Revised: (2016) restricted the transition unitary matrix on the Stiefel manifold via a closed-form Cayley transform.

Original: On the contrary, we usethe Cayley SGD with momentum and Cayley
Revised: On the contrary, we use Cayley SGD with momentum and Cayley

Original: ADAM to reduce the training time.
Revised: ADAM to reduce the training time.

Original: Table 4 shows that the proposed algorithms reduces the training  time by about 35% for all settings of the network, while maintaining the same level of accuracy.
Revised: Table 4 shows that the proposed algorithms reduce the training [Table] [Table] time by about 35% for all settings of the network, while maintaining the same level of accuracy.

Original: All experiments are performed on one TITAN
Revised: All experiments are performed on one TITAN

Original: Xp GPU.
Revised: Xp GPU.

Original: Checking Unitariness: To show that the proposed algorithms are valid optimization algorithms on the Stiefel manifold, we check the unitariness of
Revised: Checking Unitariness: To show that the proposed algorithms are valid optimization algorithms on the Stiefel manifold, we check the unitariness of

Original: the hidden-to-hidden matrix K by computing the error term
Revised: the hidden-to-hidden matrix K by computing the error term

Original: ||
Revised: ||

Original: K H K −
Revised: K H K −

Original: I || F during training.
Revised: I || F during training.

Original: Table 5 compares average errors for varying numbers of iterations s .
Revised: Table 5 compares average errors for varying numbers of iterations s .

Original: As can be seen, the iterative Cayley transform can approximate the unitary matrix when s = 2 .
Revised: As can be seen, the iterative Cayley transform can approximate the unitary matrix when s = 2 .

Original: The iterative Cayley transform performs even better than the closed form Cayley transform, which might be an effect of the rounding error of the matrix inversion as in Eq. 3.
Revised: The iterative Cayley transform performs even better than the closed form Cayley transform, which might be an effect of the rounding error of the matrix inversion as in Eq.(3).

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We speciﬁed an efﬁcient way to enforce the exact orthonormal constraints on parameters by optimization on the Stiefel manifold.
Revised: We proposed an efﬁcient way to enforce the exact orthonormal constraints on parameters by optimization on the Stiefel manifold.

Original: The iterative Cayley transform was applied to the conventional
Revised: The iterative Cayley transform was applied to the conventional

Original: SGD and ADAM for specifying two new algorithms: Cayley SGD with momentum and Cayley
Revised: SGD and ADAM for specifying two new algorithms: Cayley SGD with momentum and Cayley

Original: ADAM, and the theoretical analysis of convergence of the former.
Revised: ADAM, and the theoretical analysis of convergence of the former.

Original: The experiments show that both algorithms achieve comparable performance and faster convergence over the baseline SGD and ADAM in training of the standard Vgg and ResNet on CIFAR10 and CIFAR100, as well as RNNs on the pixel-by-pixel MNIST task.
Revised: Experiments show that both algorithms achieve comparable performance and faster convergence over the baseline SGD and ADAM in training of the standard VGG and ResNet on CIFAR10 and CIFAR100, as well as RNNs on the pixel-by-pixel MNIST task.

Original: Both Cayley SGD with momentum and Cayley ADAM take less runtime per iteration than all existing hard orthonormal methods and soft orthonormal methods, and can be applied to non-square parameter matrices.
Revised: Both Cayley SGD with momentum and Cayley ADAM take less runtime per iteration than all existing hard orthonormal methods and soft orthonormal methods, and can be applied to non-square parameter matrices.

Original: A C ONVERGENCE R ATE C OMPARISON IN TIME
Revised: A CKNOWLEDGEMENTS

Original: In this section, we analyse the time convergence rate of the proposed methods. Fig.

Original: 2 shows the training loss curve of Cayley SGD and Cayley Adam with other optimization methods for CIFAR and CIFAR100. Our Cayley SGD and Cayley Adam converge the fastest in terms of time among methods that also address orthonormality. Although the baselines SGD and Adam converge faster at the beginning due to their training efﬁciency, our Cayley SGD and Cayley Adam can catch up with the baseline after 12000 seconds, and their classiﬁcation accuracy are lower as shown in Table 1 and 2, because of the lack of orthogonality constraints. [Figure]
Revised: This work was supported in part by NSF grant IIS-1911232, DARPA XAI Award N66001-17-2and AFRL STTR AF18B-T002.

Original: B P RELIMINARY
Revised: A P RELIMINARY

Original: In this section, we derive some properties of the Stiefel manifold in this section to facilitate the proofs of Theorem 1 and Theorem 2 in the main paper.
Revised: In this section, we derive some properties of the Stiefel manifold in this section to facilitate the proofs of Theorem 1 and Theorem 2 in the main paper.

Original: Considering the Stiefel manifold M is a bounded set and Lipschitz Assumption 1 on the gradient in Sec. 5, one straightforward conclusion is that both ∇ f ( X ) and its Stiefel manifold gradient ∇ M f ( X ) that is a projection onto the tangent space are bounded.
Revised: Considering the Stiefel manifold M is a bounded set and Lipschitz Assumption 1 on the gradient in Sec. 5, one straightforward conclusion is that both ∇ f ( X ) and its Stiefel manifold gradient ∇ M f ( X ) that is a projection onto the tangent space are bounded.

Original: Formally, there exists a positive constant G , such that
Revised: Formally, there exists a positive constant G , such that

Original: [Equation]
Revised: [Equation]

Original: As stochastic gradient G ( X ) =
Revised: As stochastic gradient G ( X ) =

Original: G ( X ; ξ ) is the gradient of a sub-dataset, where ξ is a stochastic variable for data samples and we are working are a ﬁnite dataset, it’s straightforward to show that
Revised: G ( X ; ξ ) is the gradient of a sub-dataset, where ξ is a stochastic variable for data samples and we are working are a ﬁnite dataset, it’s straightforward to show that

Original: G ( X ) and its Riemannian stochastic gradient G M ( X ) are also bounded.
Revised: G ( X ) and its Riemannian stochastic gradient G M ( X ) are also bounded.

Original: For brevity, we still use the same upper bound G , such that:
Revised: For brevity, we still use the same upper bound G , such that:

Original: [Equation]
Revised: [Equation]

Original: Recall the recursion in Eq. 7, we show that the momentum is also bounded:
Revised: Recall the recursion in Eq.(7), we show that the momentum is also bounded:

Original: [Equation]
Revised: [Equation]

Original: Therefore, we know that W k in Eq. 7 is bounded.
Revised: Therefore, we know that W k in Eq.(7) is bounded.

Original: C P ROOF OF T HEOREM
Revised: B P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: By subtracting the iterative relationship Eq. 5 by its i th iteration Y i +

Revised: By subtracting the iterative relationship Eq.(5) by its i th iteration Y i +

Original: =
Revised: =

Original: X + α 2 W (cid:0) X
Revised: X + α 2 W (cid:0) X

Original: + Y
Revised: + Y

Original: i (cid:1) , we have:
Revised: i (cid:1) , we have:

Original: [Equation]
Revised: [Equation]

Original: Therefore, since W is bounded, for α <

Original: [Equation]
Revised: [Equation]

Original:  W 
Revised: Therefore, since W is bounded, for α < 2 (cid:107) W (cid:107) , such that α (cid:107) W (cid:107)< 1 , the iteration in Eq.(5) is a contraction mapping, and it will converge to the closed-from solution Y ( α ) .

Original: [Equation] , such that α
Revised: By differentiate Eq.(5), we have:

Original: [Equation]
Revised: [Equation]

Original: W

Original: [Equation]
Revised: [Equation]

Original: < 1 , the iteration in Eq. 5 is a contraction mapping, and it will converge to the closed-from solution Y ( α ) . By differentiate Eq. 5, we have:
Revised: therefore, dY ( α ) dα and d 2 Y ( α ) dαare bounded, i.e. there exist a positive constant C , such that:

Original: [Equation]
Revised: [Equation]

Original: 13) therefore, dY ( α ) dα and d 2 Y ( α ) dαare bounded, i.e. there exist a positive constant C , such that:

Original: [Equation]
Revised: [Equation]

Original: Using the Taylor expansion of Y in Eq. 3, we have:
Revised: Using the Taylor expansion of Y in Eq.(3), we have:

Original: [Equation]
Revised: [Equation]

Original: 15) where γ k ∈ (0 , 1) .
Revised: where γ k ∈ (0 , 1) .

Original: Given Y 

Revised: Given Y

Original: =
Revised: =

Original: X k
Revised: X k

Original: + αM
Revised: + αM

Original: k
Revised: k

Original: +1 , we have:
Revised: +1 , we have:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since W is bounded and α || W ||< 1 , then,
Revised: Since W is bounded and α || W ||< 1 , then,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: D P ROOF OF T HEOREM
Revised: C P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: Use Taylor expansion of Y ( α ) , the process of Cayley SGD with momentum Eq. 7 can be written as:
Revised: Use Taylor expansion of Y ( α ) , the process of Cayley SGD with momentum Eq.(7) can be written as:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where γ k ∈ (0 , 1) .
Revised: where γ k ∈ (0 , 1) .

Original: Using the fact that
Revised: Using the fact that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where π N X ( M ) is the projection onto the normal space, and
Revised: where π N X ( M ) is the projection onto the normal space [Equation] and

Original: [Equation]
Revised: [Equation]

Original: Then, the projection of momentum can be represented as:
Revised: Then, the projection of momentum can be represented as:

Original: [Equation]
Revised: [Equation]

Original: Then the momentum update in Eq. 7 is equivalent to:
Revised: Then the momentum update in Eq.(7) is equivalent to:

Original: [Equation]
Revised: [Equation]

Original: Therefore

Original: [Equation]
Revised: [Equation]

Original:  the paramter update in Eq. 7 can be represented as:
Revised: Therefore, the paramter update in Eq.(7) can be represented as:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Since || M || 
Revised: Since || M || ,

Original: [Equation]

Original: || X
Revised: || X

Original: || [Equation] and (cid:107) d 2 Y dα 2 (cid:107) are bounded, there is a positive constant D , such that
Revised: || , and (cid:107) d 2 Y dα 2 (cid:107) are bounded, there is a positive constant D , such that

Original: [Equation]
Revised: [Equation]

Original: To facilitate the analysis of Cayle SGD with momentum

Original: [Equation]
Revised: [Equation]

Original:  we introduce auxiliary variables { P k } , such that:
Revised: To facilitate the analysis of Cayle SGD with momentum, we introduce auxiliary variables { P k } , such that:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Since f ( X ) is a smooth function according to Assumption
Revised: Since f ( X ) is a smooth function according to Assumption

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: Then, we have
Revised: Then [Equation] we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: By taking expectation over the both sides [Equation] we have:
Revised: By taking expectation over the both sides, we have:

Original: [Equation]
Revised: [Equation]

Original: By noticing that:
Revised: By noticing that:

Original: [Equation]
Revised: [Equation]

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: According to the Lipschitz continuous property in Assumption
Revised: According to the Lipschitz continuous property in Assumption

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: ) where
Revised: 34) where

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since α ≤ 1 − βL , then
Revised: Since α ≤ 1 − βL , then

Original: [Equation]
Revised: [Equation]

Original: By summing Eq. over all k [Equation] we have
Revised: By summing Eq.(34) over all k , we have

Original: [Equation]
Revised: [Equation]

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Use the fact that α = min { 1 − βL , A √ t +1 } , and notice that Z 

Revised: Use the fact that α = min { 1 − βL , A √ t +1 } [Equation] and notice that Z

Original: = X 0 , therefore,
Revised: = X 0 , therefore,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Therefore, min k =0 , ··· ,t E [ ||∇ M f
Revised: Therefore, min k =0 , ··· ,t E [ ||∇ M f

Original: ( X k ) || 2 ]
Revised: ( X k ) || 2 ]

Original: = o ( 1 √ t +1 )
Revised: = o ( 1 √ t +1 )

Original: → 0 , as t → ∞ .
Revised: → 0 , as t → ∞ .