Privacy-Preserving Logistic Regression Training with A Faster Gradient Variant
AfﬁliationAddress
Abstract
Logistic regression training over encrypted data has been an attractive idea tosecurity concerns for years.
In this paper, we propose a faster gradient variantcalled quadratic gradient to implement logistic regression training in a homomorphic encryption domain, the core of which can be seen as an extensionof the simpliﬁed ﬁxed Hessian [5].
We enhance Nesterov’s accelerated gradient(NAG) and Adaptive Gradient Algorithm (Adagrad) respectively with this gradientvariant and evaluate the enhanced algorithms on several datasets.
Experimentalresults show that the enhanced methods have a state-of-the-art performance inconvergence speed compared to the naive ﬁrst-order gradient methods.
We thenadopt the enhanced NAG method to implement homomorphic logistic regressiontraining and obtain a comparable result by only 3 iterations.
Introduction
Logistic regression (LR) is a widely used classiﬁcation technology especially in medical risk assessment due to its simplicity but powerful performance.
Given a person’s healthcare data related to acertain disease, we can train an LR model capable of telling whether or not this person is likely todevelop this disease.
However, such personal health information is highly private to individuals.
Theprivacy concern, therefore, becomes a major obstacle for individuals to share their biomedical data.
The most secure solution is to encrypt the data into ciphertexts ﬁrst by Homomorphic Encryption(HE) and then securely outsource the ciphertexts to the cloud, without allowing the cloud to accessthe data directly.
iDASH is an annual competition that aims to call for implementing interesting cryptographic schemes in a biological context.
Since 2014, iDASH has included the theme of genomicsand biomedical privacy.
The third track of the 2017 iDASH competition and the second track ofthe 2018 iDASH competition were both to develop homomorphic-encryption-based solutions forbuilding an LR model over encrypted data.
Several studies on logistic regression models are based on homomorphic encryption.
Aono et al.
[2]only used an additive HE scheme and left some of the challenging HE computations to a trustedclient.
Kim et al.
[14] discussed the problem of performing LR training in an encrypted environment.
They used the full batch gradient descent in the training process and the least square method to get theapproximation of the sigmoid function.
In the iDASH 2017 competition, Bonte and Vercauteren [5],
Kim et al.
[12], Chen et al.
[6], and Crawford et al.
[8] all investigated the same problem that Kim etal.
[14] studied.
In the iDASH competition of 2018, Kim et al.
[13] and Blatt et al.
[3] further workedon it for an efﬁcient packing and semi-parallel algorithm.
The papers most relevant to this workare [5] and [12].
Bonte and Vercauteren [5] developed a practical algorithm called the simpliﬁedﬁxed Hessian (SFH) method.
Our study complements their work and adopts the ciphertext packingtechnique proposed by Kim et al.
[12] for efﬁcient homomorphic computation.
Our speciﬁc contributions in this paper are as follows:
We propose a new gradient variant, quadratic gradient , which can unite the ﬁrst-order
gradient method and the second-order Newton’s method as one.2.
We develop two enhanced gradient methods by equipping the original methods withquadratic gradient .
The resulting methods show a better performance in the convergence speed.3.
We adopt the enhanced Nesterov’s accelerated gradient to implement privacy-preservinglogistical regression training, to our best knowledge, which seems to be the best candidate
without compromising much on computation and storage.
Preliminaries
We adopt the square brackets “ [ ] ” to denote the index of a vector or matrix element in what follows.
For example, for a vector v ∈ R ( n ) and a matrix M ∈ R m × n , v [ i ] or v [ i ] means the i -th element ofvector v and M [ i ][ j ] or M [ i ][ j ] the j -th element in the i -th row of M .
2.1 Fully Homomorphic Encryption
Fully Homomorphic Encryption (FHE) is a type of cryptographic scheme that can be used to computean arbitrary number of additions and multiplications directly on the encrypted data.
It was not until2009 that Gentry constructed the ﬁrst FHE scheme via a bootstrapping operation [9].
FHE schemesthemselves are computationally time-consuming; the choice of dataset encoding matters likewiseto the efﬁciency.
In addition to these two limits, how to manage the magnitude of plaintext [11]also contributes to the slowdown.
Cheon et al.
[7] proposed a method to construct an HE schemewith a rescaling procedure which could eliminate this technical bottleneck effectively.
We adopttheir open-source implementation HEAAN while implementing our homomorphic LR algorithms.
Itis inevitable to pack a vector of multiple plaintexts into a single ciphertext for yielding a betteramortized time of homomorphic computation.
HEAAN supports a parallel technique (aka SIMD ) topack multiple numbers in a single polynomial by virtue of the Chinese Remainder Theorem andprovides rotation operation on plaintext slots.
The underlying HE scheme in HEAAN is well describedin [12, 14, 10].
2.2 Database Encoding Method
Kim et al.
[12] devised an efﬁcient and promising database-encoding method by using SIMD technique,which could make full use of the computation and storage resources.
Suppose that a database has atraining dataset consisting of n samples with (1 + d ) covariates, they packed the training dataset Zinto a single ciphertext in a row-by-row manner:
Training Dataset: Matrix Z z [1][0] z [1][1] . . .
z [1][ d ] z [2][0] z
z [2][ d ] .
z [ n ][0] z
[ n ][1] . . .
z [ n ][ d ]
Database Encoding Method [12] (cid:55)−−−−−−−−−−−−−−−−−→ Enc 
 z [1][0] z [1][1] . . .
z [1][ d ] z [2][0] z
z [2][ d ] .
z [ n ][0] z
[ n ][1] . . .
z [ n ][ d ]   (cid:121)
Encoding (cid:120) (cid:2)
z [1][0] . . .
z [1][ d ] z [2][0] . . .
z [2][ d ] . . .
z [ n ][0] . . .
z [ n ][ d ] (cid:3) (cid:120)  (cid:121) Encrypt (cid:120) 
Enc (cid:2) z
z [1][ d ] z [2][0] . . .
z [2][ d ] . . .
[Equation]
n ]
[Equation] [Equation]
n ]
[Equation] [Equation]
Using this encoding scheme, we can manipulate the data matrix Z by performing HE operations on theciphertext Enc [ Z ] , with the help of only three HE operations - rotation, addition and multiplication.
For example, if we want the ﬁrst column of Enc [ Z ] alone and ﬁlter out the other columns, we can design a constant matrix F consisting of ones in the ﬁrst column and zeros in the rest columns andthen multiply Enc [ Z ] by Enc [ F ] , obtaining the resulting ciphertext Enc [ Z p ] :
Enc [ F ] ⊗ Enc
[ Z ] =
Enc [ Z p ] ( where “ ⊗ ” means the component-wise HE multiplication )
=
Enc   1 0 . . .
.
 ⊗ Enc   z [1][0] z [1][1] . . .
z [1][ d ] z [2][0] z
z [2][ d ] .
z [ n ][0] z
[ n ][1] . . .
z [ n ][ d ]   =
Enc   z [1][0] 0 . .
.z
z [ n ][0] 0 . .
.  .
Han et al.
[10] introduced several operations to manipulate the ciphertexts, such as a procedure named“ SumColVec ” to compute the summation of the columns of a matrix.
By dint of these basic operations,more complex calculations such as computing the gradients in logistic regression models are feasible.
2.3 Logistic Regression
Logistic regression is widely used in binary classiﬁcation tasks to infer whether a binary-valuedvariable belongs to a certain class or not.
LR can be generalized from linear regression [15] bymapping the whole real line ( β T x ) to (0 , 1) via the sigmoid function σ ( z ) =
1 / (1+exp( − z )) , wherethe vector β ∈ R (1+ d ) is the main parameter of LR and the vector x
= (1 , x 1 , . .
, x d ) ∈ R (1+ d ) theinput covariate.
Thus logistic regression can be formulated with the class label y ∈ {± 1 } as follows:
[Equation]
LR sets a threshold (usually 0 . 5 ) and compares its output with it to decide the resulting class label.
The logistic regression problem can be transformed into an optimization problem that seeks a parameter β to maximize L ( β ) =
(cid:81) ni
=1 Pr( y i | x i , β ) or its log-likelihood function l ( β ) for convenience inthe calculation:
[Equation]
where n is the number of examples in the training dataset.
LR does not have a closed form ofmaximizing l ( β ) and two main methods are adopted to estimate the parameters of an LR model:(a) gradient descent method via the gradient; and (b) Newton’s method by the Hessian matrix.
Thegradient and Hessian of the log-likelihood function l ( β ) are given by, respectively:
[Equation]
where S is a diagonal matrix with entries S ii
= ( σ ( y i β T x i ) − 1) σ ( y i β T x i ) and X the dataset.
The log-likelihood function l ( β ) of LR has at most a unique global maximum [1], where its gradientis zero.
Newton’s method is a second-order technique to numerically ﬁnd the roots of a real-valueddifferentiable function, and thus can be used to solve the β in ∇ β l ( β ) = 0 for LR.
Technical Details
It is quite time-consuming to compute the Hessian matrix and its inverse in Newton’s method for each iteration.
One way to limit this downside is to replace the varying Hessian with a ﬁxed matrix ¯
H .
This novel technique is called the ﬁxed Hessian Newton’s method.
Böhning and Lindsay [4] have shown that the convergence of Newton’s method is guaranteed as long as ¯ H ≤ ∇ 2 β l ( β ) , where ¯ H is a symmetric negative-deﬁnite matrix independent of β and “ ≤ ” denotes the Loewner ordering in the sense that the difference ∇ 2 β l ( β ) − ¯ H is non-negative deﬁnite.
With such a ﬁxed Hessian matrix ¯ H , the iteration for Newton’s method can be simpliﬁed to:
[Equation]
Böhning and Lindsay also suggest the ﬁxed matrix ¯ H
= − 14 X T X is a good lower bound for the
Hessian of the log-likelihood function l ( β ) in LR.
3.1 the Simpliﬁed Fixed Hessian method
Bonte and Vercauteren [5] simplify this lower bound ¯ H further due to the need for inverting the ﬁxed Hessian in the encrypted domain.
They replace the matrix ¯ H with a diagonal matrix B whose diagonal elements are simply the sums of each row in ¯ H .
They also suggest a speciﬁc order of calculation to get B more efﬁciently.
Their new approximation B of the ﬁxed Hessian is:
[Equation]
where ¯ h ki is the element of ¯ H .
This diagonal matrix B is in a very simple form and can be obtainedfrom ¯ H without much difﬁculty.
The inverse of B can be approximated in the encrypted form bymeans of computing the inverse of every diagonal element of B via the iterative of Newton’s methodwith an appropriate start value.
Their simpliﬁed ﬁxed Hessian method can be formulated as follows:
[Equation]
b dd   ·   ∇ 0 ∇...
∇ d   = β t − 
 b 00 · ∇ 0 b 11 · ∇ 1 .
b dd · ∇ d   , where b ii is the reciprocal of (cid:80) di =0 ¯ h 0 i and ∇ i is the element of ∇ β l ( β ) .
Consider a special situation: if b 00 , . .
, b dd are all the same value − η with η > 0, the iterative formulaof the SFH method can be given as:
[Equation]
which is the same as the formula of the naive gradient ascent method.
Such coincident is just whatthe idea behind this work comes from: there is some relation between the Hessian matrix and thelearning rate of the gradient (descent) method.
We consider b ii · ∇ i as a new enhanced gradientvariant’s element and assign a new learning rate to it.
As long as we ensure that this new learningrate decreases from a positive ﬂoating-point number greater than 1 (such as 2) to 1 in a boundednumber of iteration steps, the ﬁxed Hessian Newton’s method guarantees the algorithm will convergeeventually.
The SFH method proposed by Bonte and Vercauteren [5] has two limitations: (a) in the constructionof the simpliﬁed ﬁxed Hessian matrix, all entries in the symmetric matrix ¯ H need to be non-positive.
For machine learning applications the datasets will be in advance normalized into the range [0,1],meeting the convergence condition of the SFH method.
However, for other cases such as numericaloptimization, it doesn’t always hold; and (b) the simpliﬁed ﬁxed Hessian matrix B that Bonte and
Vercauteren [5] constructed can still be singular, especially when the dataset is a high-dimensionalsparse matrix, such as the datasets from the MNIST database.
We complement their work by removingthese limitations so as to generalize this simpliﬁed ﬁxed Hessian to be invertible in any case andpropose a faster gradient variant, which we term quadratic gradient .
3.2 Quadratic Gradient
Suppose that a differentiable scalar-valued function F ( x ) has its gradient g and Hessian matrix H ,with any matrix ¯ H ≤ H in the Loewner ordering as follows:
[Equation]
 
[Equation]
¯ H =   ¯ h¯ h. . .
¯ h
d
¯ h ¯ h. . .
¯ h 1 d .
¯ h d¯ h d. . .
¯ h dd   , where ∇ 2 ij = ∇ 2 ji = ∂ 2 F ∂x i ∂x j .
We construct a new Hessian matrix ˜ B as follows:
[Equation]
where ε is a small positive constant to avoid division by zero (usually set to 1 e − 8 ).
As long as ˜ B satisﬁes the convergence condition of the above ﬁxed Hessian method, ˜ B ≤ H , wecan use this approximation ˜ B of the Hessian matrix as a lower bound.
Since we already assume that¯ H ≤ H , it will sufﬁce to show that ˜ B ≤ ¯ H .
We prove ˜ B ≤ ¯ H in a similar way that [5] did.
Lemma 1.
Let A ∈ R n × n be a symmetric matrix, and let B be the diagonal matrix whose diagonalentries B kk = − ε − (cid:80) i | A ki | for k = 1 , . .
. , n , then B ≤ A .
Proof.
By deﬁnition of the Loewner ordering, we have to prove the difference matrix C =
A − Bis non-negative deﬁnite, which means that all the eigenvalues of C need to be non-negative.
Byconstruction of C we have that C ij =
A ij + ε + (cid:80) nk
=1 | A ik | for i = j and C ij =
A ij for i (cid:54) =
j .
By means of Gerschgorin’s circle theorem, we can bound every eigenvalue λ of C in the sense that| λ − C ii | ≤ (cid:80) i (cid:54) =
j | C ij | for some index i ∈ { 1 , 2 , . . . , n } .
We conclude that λ ≥ A ii + ε + | A ii | ≥ε > 0 for all eigenvalues λ and thus that B ≤ A .
Deﬁnition 3.1 ( Quadratic Gradient ) .
Given such a ˜ B above, we deﬁne the quadratic gradientas G = ¯ B · g with a new learning rate η , where ¯ B is a diagonal matrix with diagonal entries¯ B kk = 1 / | ˜ B kk | , and η should be always no less than 1 and decrease to 1 in a limited numberof iteration steps.
Note that G is still a column vector of the same size as the gradient g .
Tomaximize or minimize the function F ( x ) , we can use the iterative formulas: x k +
= x k + η · G orx k +
= x
k − η · G , just like the naive gradient.
We point out here that ¯ H could be the Hessian matrix H itself and ˜ B further optimized to: ˜ B kk =¯ h kk + | ¯ h kk | + ε − (cid:80) d i =
| ¯ h ki | .
In our experiments, we use ¯ H = − 1 4 X T X to construct our ˜ B .
3.3 Two Enhanced Methods
Quadratic
Gradient can be used to enhance NAG and Adagrad.
NAG is a different variant of the momentum method to give the momentum term much moreprescience.
The iterative formulas of the gradient ascent method for NAG are as follows:
[Equation]
where V t +1 is the intermediate variable used for updating the ﬁnal weight β t +1 and γ t ∈ (0 , 1) is asmoothing parameter of moving average to evaluate the gradient at an approximate future position[12].
The enhanced NAG is to replace (3) with V t +1 = β t
+ (1 + α t ) · G .
Our enhanced NAGmethod is described in Algorithm 1 .
Adagrad is a gradient-based algorithm suitable for dealing with sparse data.
The updated operationsof Adagrad and its quadratic-gradient version, for every parameter β
[ i ] at each iteration step t , are as
1: Set
H ←
(cid:46) ¯ H ∈ R (1+ d ) × (1+ d ) 2: Set V ← 0 , W ← 0 , ¯ B ←(cid:46)
V ∈ R (1+ d ) , W ∈ R (1+ d ) , ¯ B ∈ R (1+ d ) × (1+ d ) 3: for i := 0 to d do 4: ¯ B
[ i ][ i ] ← ε (cid:46) ε is a small positive constant such as 1 e −5: for j := 0 to d do 6: ¯ B
[ i ][ i ] ← ¯ B [ i ][ i ] + | ¯ H
[ i ][ j ] | 7: end for 8: end for 9: Set α 0 ← 0 .
01 , α 1 ← 0 .
5 × (1 + (cid:112) 1 + 4 × α 20 ) 10: for count :
= 1 to κ do 11:
Set Z ←(cid:46) inputs for sigmoid function 12: for i := 1 to n do 13: for j := 0 to d do 14:
Z [ i ] ← Z [ i ] + Y [ i ] × V
[ j ] × X
[ i ][ j ] 15: end for 16: end for 17:
Set σ ←(cid:46)
σ ∈ R n is to store the outputs of the sigmoid function 18: for i :
= 1 to n do 19: σ [ i ] ← 1 / (1 + exp( − Z [ i ])) 20: end for 21:
Set g ←22: for j := 0 to d do 23: for i := 1 to n do 24: g
[ j ] ← g [ j ] + (1 − σ [ i ])
× Y [ i ] × X
[ i ][ j ] 25: end for 26: end for 27:
Set G ←28: for j := 0 to d do 29:
G
[ j ] ← ¯ B [ j ][ j ] × g [ j ] 30: end for 31:
Set η ← (1 − α 0 ) /α 1 , γ ← 1 / ( n × count ) (cid:46) training data 32: for j := 0 to d do 33: w temp ← V [ j ] + (1 + γ ) × G
[ j ] 34:
V [ j ] ← (1 − η ) × w temp + η × W
[ j ] 35:
W [ j ] ← w temp 36: end for 37: α α α
×
1 + 4 × α
38: end for 39: return V follows, respectively:
[Equation]
] · g
[Equation] [Equation]
· G
[Equation]
Performance Evaluation We evaluate the performance of various algorithms in the clear using the
Python programming language on the same desktop computer with an Intel Core CPU G640 at1.60 GHz and 7.3 GB RAM.
Since our focus is on how fast the algorithms converge in the trainingphase, the loss function, maximum likelihood estimation (MLE), is selected as the only indicator.
Weevaluate four algorithms, NAG, Adagrad, and their quadratic-gradient versions (denoted as Enhanced
NAG and Enhanced Adagrad, respectively) on the datasets that Kim et al.
[12] adopted: the iDASHgenomic dataset (iDASH), the Myocardial Infarction dataset from Edinburgh (Edinburgh), Low Birthweight Study (lbw), Nhanes III (nhanes3), Prostate Cancer study (pcs), and Umaru Impact Studydatasets (uis).
The genomic dataset is provided by the third task in the iDASH competition of 2017,which consists of 1579 records.
Each record has 103 binary genotypes and a binary phenotypeindicating if the patient has cancer.
The other ﬁve datasets all have a single binary dependent variable.
Figures 1 and 2 show that except for the enhanced Adagrad method on the iDASH genomic datasetour enhanced methods all converge faster than their original ones in other cases.
[Figure]
The iDASH dataset
[Figure]
x i m u m L i k e li hood
E s ti m a ti
[Figure]
In all the Python experiments, the time to calculate the ¯ B in quadratic gradient G before running theiterations and the time to run each iteration for various algorithms are negligible (few seconds).
Privacy-preserving Logistic Regression Training
Adagrad method is not a practical solution for homomorphic LR due to its frequent inversionoperations.
It seems plausible that the enhanced NAG is probably the best choice for privacypreserving LR training.
We adopt the enhanced NAG method to implement privacy-preservinglogistic regression training.
The difﬁculty in applying the quadratic gradient is to invert the diagonal
30 − 1 ,− 1 ,− 1 ,−−
Iteration Number
M a x i m u m L i k e li hood
E s ti m a ti on
NAGEnhancedNAG (a) The iDASH dataset    −−−−
Iteration Number
NAGEnhancedNAG (b) The Edinburgh dataset    −−−−
Iteration Number
NAGEnhancedNAG (c) The lbw dataset    −− 0 .− 0 .
Iteration Number
M a x i m u m L i k e li hood
E s ti m a ti on
NAGEnhancedNAG (d) The nhanes3 dataset    −−−−
Iteration Number
NAGEnhancedNAG (e) The pcs dataset    −−−−−
Iteration Number
NAGEnhancedNAG (f)
The uis dataset matrix ˜ B in order to obtain ¯ B .
We leave the computation of matrix ¯ B to data owner and let thedata owner upload the ciphertext encrypting the ¯ B to the cloud.
Since data owner has to prepare thedataset and normalize it, it would also be practicable for the data owner to calculate the ¯ B owing tono leaking of sensitive data information.
Privacy-preserving logistic regression training via homomorphic encryption technique faces a difﬁcultdilemma that no homomorphic schemes are capable of directly calculating the sigmoid function in the
LR model.
A common solution is to replace the sigmoid function with a polynomial approximationby using the widely adopted least square method.
We can call a function named “ polyfit( · ) ”in the Python package Numpy to ﬁt the polynomial in a least-square sense.
We adopt the degree5 polynomial approximation g ( x ) by which Kim et al.
[12] used the least square approach toapproximate the sigmoid function over the domain [ − 8 , 8] : g ( x ) = 0 .
5 + 0 .
19131 · x − 0 .
0045963 ·x 3 + 0 . 0000412332 · x 5 .
Given the training dataset X ∈ R n × (1+ d ) and training label Y ∈ R n × 1 , we adopt the same methodthat
Kim et al.
[12] used to encrypt the data matrix consisting of the training data combined withtraining-label information into a single ciphertext ct Z .
The weight vector β (0) consisting of zeros andthe diagnoal elements of ¯ B are copied n times to form two matrices.
The data owner then encrypt thetwo matrices into two ciphertexts ct (0) β and ct ¯ B , respectively.
The ciphertexts ct Z , ct (0) β and ct ¯ B areas follows:
[Equation] [Equation]
1 x n. . .
x nd   , Y =   y 1 y...
y n   , ct Z
=
Enc   yy 1 x. . .
y 1 x 1 d y y 2 x. . .
y 2 x 2 d .
y n y n x n. . .
y n x nd   , ct (0) β
= Enc   β (0) β (0) . . .
β (0) d β (0)β (0). . .
β (0) d ... β (0) β (0) . . .
β (0) d   , ct ¯ B
= Enc   ¯ B [0][0] ¯ B [1][1] . . .
¯ B
[ d ][ d ] ¯ B [0][0] ¯ B [1][1] . . .
¯ B
[ d ][ d ] .
¯ B [0][0] ¯ B [1][1] . . .
¯ B
[ d ][ d ]   , where ¯ B [ i ][ i ] is the diagonal element of ¯ B that is built from − 14 X T X .
The pulbic cloud takes the three ciphertexts ct Z , ct (0) β and ct ¯ B and evaluates the enhanced NAGalgorithm to ﬁnd a decent weight vector by updating the vector ct (0) β .
Refer to [12] for a detaileddescription about how to calculate the gradient by HE programming.
Implementation We implement the enhanced NAG based on HE with the library HEAAN .
The C++source code is publicly available at https://anonymous.4open.science/r/IDASH2017-245B .
All the experiments on the ciphertexts were conducted on a public cloud with 32 vCPUs and 64 GB
RAM.
For a fair comparison with [12], we utilized the same 10-fold cross-validation (CV) technique on thesame iDASH dataset consisting of 1579 samples with 18 features and the same 5-fold CV techniqueon the other ﬁve datasets.
Like [12], We consider the average accuracy and the Area Under the Curve(AUC) as the main indicators.
Tables 1 and 2 show the two experiment results, respectively.
The twotables also provide the average evaluation running time for each iteration.
We adopt the same packingmethod that Kim et al.
[12] proposed and hence our solution has similar storage of ciphertexts to [12]with some extra ciphertexts to encrypt the ¯ B .
The parameters of HEAAN we set are same to [12]:
logN = 16 , logQ = 1200 , logp = 30 , slots =32768 , which ensure the security level λ = 80 .
Refer [12] for the details of these parameters.
Sinceour enhanced NAG method need to consume more modulus to preserve the precision of ¯ B , we uselogp = 60 to encrypt the matrix ¯ B and thus only can perform 3 iterations of the enhanced NAGmethod.
Yet despite only 3 iterations, our enhanced NAG method still produces a comparable result.
[Table] [Table]
Conclusion
In this paper
[Table]
we proposed a faster gradient variant called quadratic gradient
[Table]
and implementedthe quadratic-gradient version of NAG in the encrypted domain to train the logistic regression model.
The quadratic gradient presented in this work can be constructed from the Hessian matrix directly,and thus somehow integrates the second-order Newton’s method and the ﬁrst-order gradient (descent)method together.
There is a good chance that quadratic gradient could accelerate other gradientmethods such as RMSprop and Adam, which is an open future work.
A Appendix
Optionally include extra information (complete