L EARNING A DVERSARIAL E XAMPLES WITH R IEMANNIAN G EOMETRY
Anonymous authors
Paper under double-blind review
A BSTRACT
Adversarial examples, referred to as augmented data points generated by imperceptible perturbation of input samples, have recently drawn much attention.
Wellcrafted adversarial examples may even mislead state-of-the-art deep models to make wrong predictions easily.
To alleviate this problem, many studies focus on investigating how adversarial examples can be generated and/or resisted.
All the existing work handles this problem in the Euclidean space, which may however be unable to describe data geometry.
In this paper, we propose a generalized framework that addresses the learning problem of adversarial examples with Riemannian geometry.
Speciﬁcally, we deﬁne the local coordinate systems on Riemannian manifold, develop a novel model called Adversarial Training with Riemannian
Manifold, and design a series of theory that manages to learn the adversarial examples in the Riemannian space feasibly and efﬁciently.
The proposed work is important in that (1) it is a generalized learning methodology since Riemmanian manifold space would be degraded to the Euclidean space in a special case; (2) it is the ﬁrst work to tackle the adversarial example problem tractably through the perspective of geometry; (3) from the perspective of geometry, our method leads to the steepest direction of the loss function.
We also provide a series of theory showing that our proposed method can truly ﬁnd the decent direction for the loss function with a comparable computational time against traditional adversarial methods.
Finally, the proposed framework demonstrates superior performance to the traditional counterpart methods on benchmark data including MNIST, CIFAR10 and SVHN.
I NTRODUCTION
Recently Deep Neural Networks (DNN) achieve a big success on a wide range of challengeable tasks, such as image classiﬁcation, speech recognition, and object detection (LeCun et al.
, 2015)(He et al., 2017).
However, recent studies have found that DNNs can be easily fooled by some special input called adversarial examples which are referred to as augmented data points generated by imperceptible perturbation of input samples (Szegedy et al., 2013)(Biggio et al., 2013)(Nguyen et al., 2015).
Although such perturbation is visually imperceptible, it can lead DNNs to make incorrect predictions with high conﬁdence.
There have been a lot of proposals studying how to generate more powerful adversarial examples, and how to build up robust networks to defend them (Goodfellow et al., 2014)(Szegedy et al., 2013).
This interesting problem was ﬁrst studied in (Liu & Nocedal, 1989) where the authors proposed a simple way to produce adversarial examples with L-BFGS optimization.
A more powerful approach
Fast Gradient Sign Method (FGSM) is later proposed in (Goodfellow et al., 2014) and further extended to a more general case with l p constraint for perturbation (Lyu et al., 2015)(Shaham et al., 2015).
The multi-step variant FGSM k was proposed in (Kurakin et al., 2016) which is essentially projected gradient decent (PGD) on the negative loss.
Aside from studying how to generate adversarial examples, some researchers developed methods to defend them.
The adversarial training was proposed by (Goodfellow et al., 2014)
(Lyu et al., 2015) which augmented the training set with adversarial examples.
This method not only increases the model robustness for adversarial examples but also improves the generalization for benign samples.
Some feature squeezing (Xu et al., 2017) and defensive distillation (Papernot et al., 2016) were also exploited to resist adversarial attacking.
Semisupervised version of adversarial training was developed in (Miyato et al., 2015)
(Miyato et al.,
(Miyato et al., 2016) called Virtual Adversarial Training (VAT), where the output distribution was smoothed by penalizing the KL-divergence between outputs of original and adversarial examples.
Moreover, some researchers provided the theory and principles for adversarial examples (Fawzi et al., 2016).
Furthermore, (Ma et al., 2018) have demonstrated that the adversarial examples are a dense region of pixel space instead of isolated points.
All these existing adversarial training methods simply consider the adversarial example problem in the Euclidean space with the orthogonal coordinate system.
Speciﬁcally, this traditional adversarial training methods aim to solve a robust optimization problem (Lyu et al., 2015):
[Equation]
where L denotes a loss function and the pair of input and label ( x, y ) is assumed to be drawn from the data distribution D .
The robust optimization problem is deﬁned as a min-max problem with respect to the worst perturbation ϵ and the best model parameters θ .
The adversarial perturbation ϵ is restricted within l p -ball around benign example x .
The FGSM can be seen as a special case with p = ∞ .
Such restriction is deﬁned in Euclidean space and the similarity between two points is measured by Minkowski distance.
[Figure]
However, data points may be in practice attached on a geometric manifold which cannot be appropriately described with Euclidean coordinate system.
In this case, the Euclidean metric would be not rational.
Moreover, existing adversarial training methods usually search the worst perturbation through the gradient of loss function with respect to x , since the gradient is considered as the steepest direction.
However, in a geometric manifold, particularly in Riemannian space, the gradient of a loss function unnecessarily presents the steepest direction.
Figure 1 illustrates the difference between a
Riemannian space and the Euclidean space, where the detailed mathematic notation can be seen in
Section 2.
Clearly, In this ﬁgure, assuming that the data are attached in the manifold as deﬁned in Figure 1(a), the Euclidean distance may not appropriately reﬂect the true distance between two points (e.g., ξ (0) and ξ ( a ) ).
In this paper, we extend the traditional adversarial problem to Riemannian space and propose a novel adversarial method called Adversarial Training with Riemannian Manifold (ATRM).
ATRM is regarded as a generalized framework in that Riemannian space contains the Euclidean space as a special case.
In more details, we start with deﬁning the local coordinate system and Riemannian metric tensor to evaluate the similarity between two points in Riemannian space.
We then propose to restrict the adversarial perturbation within l p -ball around natural examples x on Riemannian manifold, and develop a framework to search the worst perturbation through the generalized trust region method.
Our proposed method is to solve the adversarial problem from the perspective of geometry which is similar to Natural Gradient methods (Amari, 1998), however, our method is implemented on input space instead of parameter space of DNNs.
We list the main contributions of this paper as follows: 1) To our best knowledge, this is the ﬁrst work to tackle the adversarial example problem through the perspective of geometry.
We study the adversarial example in the more generalized Riemannian space of which Euclidean space is a special case.
Our method considers the curvature information of the loss function which can be viewed as the second order method, enabling a more accurate direction of adversarial perturbation.
Importantly, from the perspective of geometry, our method leads to the steepest direction of loss function in Riemannian space.
We also provide a series of theory showing that our proposed method can truly ﬁnd the decent direction for the loss function with a comparable computational time against traditional adversarial method (one more backward backward prorogation).
M AIN M ETHODOLOGY
In this section, we ﬁrst study how to generate adversarial examples within l 2 -ball in Riemannian
Manifold.
We then deﬁne the local metric tensor on the manifold and generate the adversarial examples using our proposed ATRM with l 2 -ball restriction.
Next, we extend our proposed method to l p -ball restriction on Riemannian manifold.
Finally, we propose the adversarial training method called ATRM and also provide theories showing that ATRM can ﬁnd the decent direction of the loss.
2. R IEMANNIAN G EOMETRY
In this subsection, we ﬁrst introduce the mathematical notations and some background of geometry.
The Einstein notation is used in this subsection.
The Riemannian Manifold is deﬁned as Deﬁnition 2.1:
Deﬁnition 2.1.
(Riemannian Manifold (Walschap, 2012))
In differential geometry, a Riemannian manifold ( M, g ) is a real smooth manifold M equipped with inner product in tangent space T p M at each point p varying smoothly on M , deﬁned by positive deﬁnite metric tensor g p .
Theorem 2.1.
(Geodesic (Amari, 2016))
A curve that connects two points by a minimal distance is a geodesic under the Levi-Civita connection, where the length of a curve c = ξ ( t ) connecting ξ (0) and ξ ( a ) is given by
[Equation]
(2) where g ij ( t ) denotes the metric tensor at point ξ ( t ) .
The geodesic can be calculated with Theorem 2.1, which measures the distance of two points on
Riemannian manifold.
It can be seen as an extension of the Euclidian distance in Euclidian space.
Lemma 2.2.
Let ξ (0) and ξ ( a ) be two close points on Riemannian manifold, where ξ ( t ) is a shortest curve connecting these two points.
Then, the distance between these two points can be computed by:
[Equation]
where dθ i = ˙
ξ ( t ) i dt is the i th component of small vector dθ and g ij ( t ) is the metric tensor (proof can be seen in Appendix D ).
We can also rewrite ds 2 in form of inner production:
[Equation]
where basis vectors { e i } is the set of tangent vectors along the coordinate curves.
Combining these two equations, we have:
[Equation]
Therefore, the metric tensor G =
( g ij ) is the inner product of basis vectors.
In the case of Euclidean space (orthonormal coordinate system), the metric tensor is:
[Equation]
where δ ij represents the Kronecker delta.
The Euclidean space can be viewed as a special case of
Riemannian space with the identity metric tensor.
For traditional adversarial training, the data are assumed to be in Euclidean space with the identity metric tensor.
In this paper, we consider a more general case that the data are attached on the Riemannian manifold with positive deﬁnite metric tensor varying smoothly on the manifold.
Figure 1 illustrates the difference between the two spaces.
2. A DVERSARIAL P ERTURBATION WITHIN l 2 - BALL ON M ANIFOLD
In this subsection, we introduce in details how we exploit a generalized trust region method to search the adversarial perturbation on the Riemannian manifold.
To search the adversarial examples on the Riemannian manifold feasibly, we propose to solve the inner optimization problem of (1) with the generalized trust region method:
[Equation]
where ϵ is a small perturbation and σ is a small value.
T k {L , x, ϵ } denotes the k th order Taylor expansion of function L ( . )
around x evaluated at x + ϵ .
Here k can be set to 1 or 2 .
d ( x, x + ϵ ) represents the distance between two points x and x + ϵ .
In this paper, we follow the setting of previous research (Lyu et al., 2015) (Shaham et al., 2015) and provide the 1 th Taylor expansion for the loss function L ( . ) .
We deﬁne the d ( . ) geodesic distance between two points on Riemannian manifold.
Since the perturbation ϵ is very small, by applying Lemma 2.2, (7) can be reformulated as:
[Equation]
where G =
( g ij ( x )) denotes the metric tensor at the point x on data manifold.
In (7), since geodesic d ( x, x + ϵ ) ≤ σ , the geodesic can be evaluated by (3).
It can be proved that we can solve the problem ( 8) with Lagrangian multiplier method and the worst perturbation can be given as:
[Equation]
[Equation]
The detailed proof of (9) can be seen in Appendix A .
2. D EFINING M ETRIC T ENSOR
In the previous subsection, we have derived that the direction of the worst perturbation is relevant to metric tensor and the ﬁrst derivative of L with respect to x .
It is easy to evaluate ∇ x L in DNNs through back propagation.
It is then very crucial to deﬁne the Riemannian manifold for input and the metric tensor G .
Since, the adversarial examples are closely related to loss function and classiﬁcation boundary, we can deﬁne the Riemannian manifold and metric tensor from loss function.
Speciﬁcally,
We ﬁrst take the n th order Taylor expansion of L around x evaluated at x + ϵ ( n is an extremely large value):
[Equation]
where L ( n ) x ( x ) denotes the n th order derivative of L with respect to x and
[Equation]
The loss function L ( x + ϵ ) can be written as summation of its 1 th order Taylor expansion and the other components of Taylor series m ( x ) .
The 1 th order Taylor expansion is more reliable when | m ( x ) | is small enough.
Naturally, the restriction can be deﬁned as:
[Equation]
However, it is difﬁcult to deal with the constraint (12) in the optimization problem.
We can simplify it with its upper bound by using Lemma 2.3.
Lemma 2.3.
Assume H be a symmetric square matrix in R n × n and r ∈ R n be a vector.
Then we have | r T Hr | ≤ r T | H | r
and | H | represents the matrix with taking the absolute value of each eigenvalue of H (proof can be seen in Appendix E ).
Based on Lemma 2.3, we can have
[Equation]
(13) where H is the Hessian matrix of L with respect to x and σ is a small value.
When the loss function is locally convex with respect to x , the Hessian matrix H is positive semideﬁnite matrix and the absolute Hessian matrix is the same as the Hessian matrix.
When the loss function is not locally convex, the non-negative eigenvalues of absolute Hessian matrix keep the same with the Hessian matrix while the negative eigenvalues are changed to positive ones.
The curvature information is partially kept. (13) can be viewed as the distance between two close points x and x + ϵ on the
Riemannian manifold with metric tensor | H | .
Comparing (13) with constraint of (8), we can deﬁne the metric tensor as | H | .
Substituting | H | in (8), we can easily get the worst perturbation:
[Equation]
In contrast to the traditional adversarial training methods, the metric tensor | H | of our method involves the curvature information of the loss function which can be seen as the second order method.
Through the perspective of geometry, the direction of gradient is not guaranteed to be steepest in
Riemannian space, however, the metric tensor adjusts it to the steepest one as illustrated in Figure 2.
Note that, for DNNs, it is easy to evaluate the Hessian matrix of L with respect to input x by back propagation.
and green lines present the contour lines. The steepest direction is given by the gradient of the loss function, which is orthogonal to contour lines only when an orthonormal coordinate system is used in Euclidean space. In the Riemannian space, the steepest direction is not guaranteed to be orthogonal to contour lines. However, adjusted with the metric tensor | H | − 1 , the direction of gradient can be approximate to the steepest direction.
The graph is better viewed in color.
2. A DVERSARIAL P ERTURBATION WITHIN l p - BALL ON M ANIFOLD
In the previous subsections, we have calculated the worst perturbation within l 2 -ball on Riemannian manifold.
Similar to traditional adversarial training methods, we can also extend our method to l p -ball on manifold.
First, we introduce Lemma 2.4 before we proceed to the case of l p -ball.
Lemma 2.4.
Let A a real symmetric positive deﬁnite matrix in R n × R n .
Then we have a unique positive deﬁnite matrix S in R n × R n so that A = S 2 (proof can be seen in Appendix F ).
Using Lemma 2.4, we reformulate the constraint of (13)
[Equation]
where S is a positive deﬁnite matrix which we call a transformation matrix.
Then we can easily extend (15) to l p -ball on manifold:
[Equation]
Substituting (16) with the constraint of (8), we can easily evaluate the corresponding worst perturbation (details are provided in Appendix B ):
[Equation]
where p ∗ is the dual of p , i.e., 1 p (cid:3) + 1 p = 1 .
Clearly, when p = 2 , the worst perturbation is reduced to (14) which is the case of the perturbation within l 2 -ball on manifold.
When p = ∞ , our method reduces to the generalized FGSM:
[Equation]
Though we can evaluate the adversarial perturbation for any l p -ball on Riemannan manifold through (17).
In this paper, we focus on the constraint of l 2 -ball and will develop the corresponding adversarial training method to improve the performance of DNNs in the next subsection.
2. A DVERSARIAL T RAINING M ETHOD WITH R IEMANNIAN M ANIFOLD
After studying the adversarial examples on manifold, we now consider to design an optimization method to improve the DNNs using the theory in the previous subsections.
We ﬁrst deﬁne the overall optimization as the robust optimization problem in a way similar to the traditional adversarial training:
[Equation]
In this paper, we focus on the l 2 -ball constraint, while it is easily extended to the l p -ball constraint.
In the case of l 2 , we can reduce the constraint in (19) to ϵ T | H | ϵ ≤ σ .
To optimize this problem, we can ﬁrst solve the inner optimization problem then followed by the outer one.
We then repeat this process until it converges.
The whole process is shown as Algorithm 1.
On the other hand, Algorithmdemonstrates the function of approximating | H | − 1 ∇ x L
( y i , x i , θ ) .
Hessian matrix may require a large amount of computation.
In Algorithm 1, we approximate | H | − 1 ∇ x L
( y i , x i , θ ) with the ﬁrst derivative of loss function with respect to input.
Speciﬁcally, we simply modify the Limited-memory
BroydenCFletcherCGoldfarbCShanno (L-BFGS) method.
More details can be seen in Appendix C .
In Algorithm 1, | H | − 1 ∇ x L
( y i , x i , θ ) represents the normalization of | H | − 1 ∇ x L ( y i , x i , θ ) .
We now provide the theoretical analysis showing that our proposed method truly offers the decent direction for the optimization problem as (Madry et al., 2017).
In order to do this, we ﬁrst present
Theorem 2.5.
1: function A PPRO HD( y i , x i , θ, ζ ) 2:
Give ζ very small value 3: g 0 = ∇ x L ( y i , x i , θ ) 4: g 1 = ∇ x L ( y i , x i + ζg 0 , θ ) 5: y = g 1 − g6: s = ζg7: ρ = 1 y T s 8: α
{+= ρs T g9: q = g 1 − αy 10: r 0 = q11: β = ρy T r12: r 1 = r 
+}
+ ( α − β ) s 13: return r14: end function
Theorem 2.5.
(Danskin).
Let S be nonempty compact topological space and g : R × S
→ R such that g ( ., δ ) is differentiable for every δ ∈ S and ∇ θ g ( θ, δ ) is continuous on R n × S .
[-Give ζ very small value 3: g 0 = ∇ x L ( y i , x i , θ ) 4: g 1 = ∇ x L ( y i , x i + ζg 0 , θ ) 5: y = g 1 − g6: s = ζg7: ρ = 1 y T s 8: α = ρs T g9: q = g 1 − αy 10: r 0 = q11: β = ρy T r12: r 1 = r 
 + ( α − β ) s 13: return r14: end function-]Also assume δ ∗ ( θ ) = { δ ∈ arg max δ ∈ S g ( θ, δ ) } .
Then the max-function ϕ ( θ ) = max δ ∈ S g
( θ, δ ) is locally
Lipschitz continuous, directionally differentiable, and its directional derivatives satisfy: ϕ ′ ( θ ) =
sup δ ∈ δ (cid:3) ( θ ) h T ∇ g ( θ, δ )
In particular, if for some θ ∈ R n the set δ ∗ ( θ ) =
{ δ ∗ θ } is a singleton, the max-function is differentiable at θ and ∇ ϕ ( θ )
=
∇ θ g ( θ, δ ∗ θ ) .
This theorem states that the gradients of ϕ ( θ ) are local objects and the gradients are locally the same as that of g ( θ, δ ∗ θ ) .
With Theorem 2.5, we describe the theory showing that our proposed optimization method truly offers the decent direction:
Lemma 2.6.
Let ˆ δ ∈ S be a maximizer of max δ L ( θ, x + θ, y ) .
Then, we have that −∇ theta L ( θ, x
+ ˆ δ, y ) is a decent direction for ϕ ( θ ) =
max δ ∈ S L ( θ, x + δ, y ) .
Proof.
We apply Theorem 2.5 that g ( θ, δ ) :
= L ( θ, x + δ, y ) and S = B p ( σ ) , which is deﬁned as the l p -ball with radius σ on Riemannian manifold.
The directional derivative in the direction of h = ∇ θ L ( θ, x + ˆ δ, y ) satisﬁes:
[Equation]
2. C OMPUTATIONAL A NALYSIS
Compared with traditional adversarial training methods, our proposed ATRM need compute the
Hessian matrix of the loss function with respect to input additionally.
It may cost extra computation to calculate the Hessian matrix.
Nonetheless, we exploit in this paper the ﬁrst-order derivative of the loss function to approximate the product of Hessian matrix and the vector, which is shown in Algorithm 2.
Therefore, our proposed method requires backward propagation three times and forward prorogation once.
Speciﬁcally, the ﬁrst backward prorogation is used to approximate the
Hessian matrix, the second one is used to evaluate the adversarial perturbation, and the last time is to update the parameters of DNNs.
In contrast to traditional adversarial training methods, our proposed method need merely one additional backward propagation which is acceptable in practice.
E XPERIMENT
To validate the efﬁcacy of our proposed method, we conduct a series of experiments on benchmark data including MNIST, CIFAR-10, and SVHN.
In these experiments, we compare our proposed ATRM with other competitive methods.
For MNIST datasets we use the same baseline as (Rasmus et al., 2015).
The same base structure called ’conv-large’ is used on dataset CIFARand SVHN, which follows (Miyato et al., 2018).
Speciﬁcally, ’conv-large’ means a large convolutional neural network with seven convolutional layers and three fully-connected layers with dropout where the size of all the convolutional kernels is 3 × 3 .
We ﬁrst implement our proposed ATRM on handwriting dataset MNIST.
Since there is no previous adversarial research on this baseline model on this dataset, we conduct the experiment on two methods (adversarial training with l ∞ and l 2 constraint) for comparison.
The model is trained with 60 , 000 labeled samples without any data augmentation and is tested with 10 , 000 samples.
We train the model with a batch size of 32 and the maximum 500 epochs.
There are two hyper parameters { ζ, ξ } in Algorithm 1 and Algorithm 2.
We set ζ to a very small value, which is 10 − 6 in this paper.
We tune the value of ξ in the range of { 0 . 1 , 0 . 2 , 0 . 5 , 1 , 2 , 5 , 10 } .
We use the set of hyper parameters that achieved the best performance on the validation set of size 5 , 000 , selected randomly from the pool of training samples of size 60 , 000 .
Table 1 lists the performance of various methods including our proposed ATRM and other competitive methods.
Except for ATRM, we also conduct the same experiment for the baseline model and traditional adversarial training methods with l 2 -ball and l ∞ -ball constraint.
We conduct the experiment with this same setting for ﬁve times and calculate the mean and standard deviation.
As observed, our proposed ATRM demonstrates the best performance.
In particular, all the adversarial methods achieve remarkably good performance, while the ATRM shows a further improvement.
This validates the advantages of our proposed method over the other methods, especially the other traditional adversarial methods.
[Table]
For CIFAR-10, we train our proposed model ATRM with 50 , 000 labeled samples with data augmentation as conducted in ( Miyato et al., 2018)
(translation and horizontal ﬂip).
The test dataset of
CIFAR-10 involves 10 , 000 samples.
Similar to the experiment on MNIST, we set ζ to a small vaule 10 − 6 , and tune the value of ξ in the range of { 1 , 2 , 5 , 8 , 10 } .
We run the experiments for ﬁve times and report the average performance and corresponding standard deviation.
Table 2 summarizes the results of different methods on CIFAR-10 dataset.
In this experiment, we intentionally compare our proposed method ATRM with the other two very deep models, i.e., the densely connected network (DenseNet) with 190 layers and very deep residual network (ResNet) with 1 , 001 layers.
Overall, our proposed method demonstrates competitive performance.
Though not as good as the very deep networks DenseNet and ResNet, the proposed ATRM shows superior performance to those adversarial learning methods and all the other remaining approaches.
Once again, this shows that adversarial learning should be conducted on the geometric manifold rather than the traditional Euclidean space.
The Street View House Numbers (SVHN) dataset contains 32 × 32 colored digit images.
There are 73 , 257 images in the training set, 26 , 032 images in the test set, and 531 , 131 images for additional training.
We train our model using the same setting as (Huang et al., 2017).
As can be clearly observed in Table 3, our method demonstrates the best performance.
It is even much better than the very deep networks (DenseNet and ResNet).
More importantly, our method achieves an obvious improvement compared with the traditional adversarial training algorithms.
[Table]
[Table] [Figure] convergence curves on three datasets Finally, we have also done the experiments to prove the convergence for our proposed method.
We have plotted the convergence curves for both training set and validation set of all three datasets as
Figure 3 showing.
C ONCLUSION
We present the novel framework called Adversarial Training with Riemannian Manifold (ATRM) which generalizes the traditional adversarial training method to Riemannian space.
For traditional adversarial training methods, the worst perturbation is often searched with the gradient ∇ x L .
However, when the data lay on a geometric manifold deﬁned as a Riemannian manifold, the gradient ∇ x L is not the steepest direction leading the adversarial perturbation is not the worst one.
We present a series of theory showing that our method leads to the steepest direction of the loss function in Riemannian space.
In practice, we also develop a practical algorithm guaranteeing the decent direction for the loss function at each epoch.
Experiments demonstrate encouraging results on benchmark data including MNIST, CIFAR-10 and SVHN.
A F IND THE ADVERSARIAL PERTURBATION WITHIN l 2 - BALL ON MANIFOLD
Recall our goal is to maximize the value of the problem:
[Equation]
Since L ( x ) is independent of ϵ and the worst perturbation has the norm σ then we have:
[Equation]
Then we apply the Lagrangian multiplier method on this problem:
[Equation]
Making the ﬁrst derivative of (21) with respect to ϵ zero [Equation] we have:
[Equation]
Then we have:
[Equation]
Therefore:
[Equation]
B F IND THE ADVERSARIAL PERTURBATION WITH l p - BALL ON MANIFOLD
The optimization problem is:
[Equation]
Similar to Appendix A [Equation] we reduce the problem to:
[Equation]
We solve it with the Lagrangian multiplier method again and set r = ϵ T S and f ( r ) ≡ ∥ r ∥ p = σ .
We have
[Equation]
Then we make the ﬁrst derivative respect to r :
[Equation]
If we sum over two sides, we have
[Equation]
By combining (22) and (23), we have
[Equation]
Since r = ϵ T S , we have
[Equation]
C F IND THE APPROXIMATION FOR | H | (cid:0) 1 ∇ x L ( y i ; x i ; (cid:18) ) .
We develop a modiﬁed BFGS method to approximate the | H | − 1 .
To solve the memory problem, we use the simpliﬁed Limited-memory BFGS (L-BFGS) to approximate | H | − 1 ∇ x L
( y i , x i , θ ) directly (based on BFGS).
We ﬁrst present Theorem C.1.
Theorem C.1.
(Sherman Morrison formula (Bartlett, 1951)) Let A ∈ R n × n be an invertible square matrix and v, u ∈ R n be column vectors.
Then A + uv T is invertible if and only if 1+ v T A − 1 u ̸ = 0 .
And its inverse is given by
[Equation]
We then take the second order Taylor expansion for L ( x + ϵ 1 ) (where ϵ 1 is a small value):
[Equation]
If we make the ﬁrst derivative on both sides with respect to ϵ 1 , we can have:
[Equation]
where H denotes the Hessian matrix of L ( x ) with respect to ϵ 1 .
We can deﬁne y =
∇L ( x + ϵ 1 ) − ∇L ( x ) , then [Equation]
Next, we use modiﬁed BFGS algorithm to approximate H and let
[Equation]
where B represents the approximation for H , I is identity matrix, and ∆ M denotes difference matrix.
Our aim is to evaluate ∆ M .
First we deﬁne
[Equation]
where a, b ∈ R and u, v ∈ R N are undetermined.
It is easy to get that ∆ M is symmetric.
We combine (27), (26) and (25):
[Equation]
We can further assume
[Equation]
Therefore, we have
[Equation]
If we combine (28) and (30), we have
[Equation]
Therefore, we can set
[Equation]
By combining (32) and (30), we have
[Equation]
[Equation]
Then, we combine (26), (32), and (33):
[Equation]
(34) can be viewed as a simpliﬁed BFGS method with one step approximation.
When the dimension of s and ϵ 1 increases, a large amount of memory is needed to store the matrix ss T and ϵ 1 ϵ T 1 .
To solve this problem, we then use the simpliﬁed L-BFGS method to evaluate directly | H | − 1 ∇ x L
( y i , x i , θ ) .
We ﬁrst use Theorem C.1 to reformulate (34) as:
[Equation]
Then, we can easily implement the method L-BFGS as (Byrd et al., 1994) and get Algorithm 3.
[-y = g 1 − g6: s = ζg7: ρ = 1 y T s 8: α = ρs T g9: q = g 1 − αy 10: r 0 = q11: β = ρy T r12: r 1 = r 
-]
D P ROOF FOR L EMMA 2.
Let ξ (0) and ξ ( a ) be two closed points on Riemannian manifold, where ξ ( t ) is a shortest curve connecting these two points.
Then, we have the square of geodesic distance between these two points:
[Equation]
where dθ i = ˙
ξ ( t ) i is the i th component of small vector dθ and g ij ( t ) is the metric tensor.
1: function A PPRO HD( y i , x i , θ, ζ ) 2: Set ζ to a very small value 3: g 0 = ∇ x L ( y i , x i , θ ) 4: g 1 = ∇ x L ( y i , x i + ζg 0 , θ ) 5: y = g 1 − g6: s = ζg7: ρ = 1 y T s 8: α =
[Equation]
T g9:
[Equation]
10: r 0 = q11: β =
[Equation]
T r12: r
[Equation]
=
[Equation]
13: return r14: end function
Proof. We evaluate the geodesic distance between ξ (0) and ξ ( a ) through Theorem 2.1:
[Equation]
Since ξ (0) and ξ ( a ) are closed points, a → 0 .
Then we have
[Equation]
Therefore, we have
[Equation]
E P ROOF FOR L EMMA 2.
Assume H be a symmetric square matrix in R n × R n and r ∈ R n be a vector.
Then we have | r T Hr | ≤ r T | H | r
and | H | represents the matrix with taking the absolute value of each eigenvalue of H .
Proof.
Assume { e 1 , e 2 , ..., e n } and { λ 1 , λ 2 , ..., λ n } are the eigenvectors and corresponding eigenvalues of matrix H .
Then we reformulate | r T Hr | with eigenvectors and corresponding eigenvalues as:
[Equation]
[Equation]
We can now use the triangle inequality | ∑ i r i | ≤ ∑ i | r i | and we have:
[Equation]
F P ROOF FOR L EMMA 2.
Let A be a real symmetric positive deﬁnite matrix in R n × R n .
Then we have a unique positive deﬁnite matrix S in R n × R n so that
A
=
S 2 .
Proof.
To prove existence: Since A is a real symmetric positive deﬁnite matrix, we have A =
P T diag ( λ 1 , ..., λ n ) P , where P is an orthogonal matrix and { λ i } are the Eigen values of A ( λ i > 0 ).
We can ﬁnd S
= P T diag ( λ 12 1 , ..., λ 12 n ) that A = S 2 .
To prove uniqueness: Let B be another positive deﬁnite matrix and A = B 2 .
Since B is positive deﬁnite, we have A = T T diag ( µ 1 , ..., µ n ) T , where T is an orthogonal matrix and { µ i } are Eigen values of B ( µ i > 0 ).
We have A = P T diag ( λ 1 , ..., λ n ) P , therefore [Equation]
[Equation]
Let U =
( u ij ) n × n = PT −
and we have:
[Equation]
42) which is equivalent to:
[Equation]
When λ ̸ = µ 2 j , u ij = 0 , we have λ 12 i = u ij µ j . When λ i = µ 2 j , we also have λ 12 i = u ij µ j .
Therefore:
[Equation]
Then we have:
[Equation] [Equation]