A DAPTIVE O PTIMIZERS WITH S PARSE G ROUP L ASSO
Anonymous authors
Paper under double-blind review
A BSTRACT
We develop a novel framework that adds the regularizers to a family of adaptive optimizers in deep learning, such as M OMENTUM , A DAGRAD , A DAM , AMSG RAD , A DA H ESSIAN , and create a new class of optimizers, which are named
G ROUP M OMENTUM , G ROUP
A DAGRAD , G ROUP
A DAM , G ROUP AMSG RAD and G ROUP
A DA H ESSIAN , etc., accordingly.
We establish theoretically proven convergence guarantees in the stochastic convex settings, based on primal-dual methods.
We evaluate the regularized effect of our new optimizers on three largescale real-world ad click datasets with state-of-the-art deep learning models.
The experimental results reveal that compared with the original optimizers with the post-processing procedure which use the magnitude pruning method, the performance of the models can be signiﬁcantly improved on the same sparsity level.
Furthermore, in comparison to the cases without magnitude pruning, our methods can achieve extremely high sparsity with signiﬁcantly better or highly competitive performance.
I NTRODUCTION
With the development of deep learning, deep neural network (DNN) models have been widely used in various machine learning scenarios such as search, recommendation and advertisement, and achieved signiﬁcant improvements.
In the last decades, different kinds of optimization methods based on the variations of stochastic gradient descent (SGD) have been invented for training DNN models.
However, most optimizers cannot directly produce sparsity which has been proven effective and efﬁcient for saving computational resource and improving model performance especially in the scenarios of very high-dimensional data.
Meanwhile, the simple rounding approach is very unreliable due to the inherent low accuracy of these optimizers.
In this paper, we develop a new class of optimization methods, that adds the regularizers especially sparse group lasso to prevalent adaptive optimizers, and retains the characteristics of the respective optimizers.
Compared with the original optimizers with the post-processing procedure which use the magnitude pruning method, the performance of the models can be signiﬁcantly improved on the same sparsity level.
Furthermore, in comparison to the cases without magnitude pruning, the new optimizers can achieve extremely high sparsity with signiﬁcantly better or highly competitive performance.
In this section, we describe the two types of optimization methods, and explain the motivation of our work.
1. A DAPTIVE O PTIMIZATION M ETHODS
Due to the simplicity and effectiveness, adaptive optimization methods (Robbins & Monro, 1951;
Polyak, 1964; Duchi et al., 2011; Zeiler, 2012; Kingma & Ba, 2015; Reddi et al., 2018; Yao et al., 2020) have become the de-facto algorithms used in deep learning.
There are multiple variants, but they can be represented using the general update formula (Reddi et al., 2018):
[Equation]
where α t is the step size, m t is the ﬁrst moment term which is the weighted average of gradient g t and V t is the so called second moment term that adjusts updated velocity of variable x t in each direction.
Here, √ V t :=
V 1 / 2 t , m t / √ V t := √ V t − 1 · m t .
By setting different m t , V t and α t , we can derive different adaptive optimizers including M OMENTUM (Polyak, 1964), A DAGRAD (Duchi et al., 2011), A DAM (Kingma & Ba, 2015), AMSG RAD (Reddi et al., 2018) and A DA H ESSIAN (Yao et al., 2020), etc.
See Table 1.
[Table]
1. R EGULARIZED O PTIMIZATION M ETHODS
Follow-the-regularized-leader (FTRL) (McMahan & Streeter, 2010; McMahan et al., 2013) has been widely used in click-through rates (CTR) prediction problems
[Table]
which adds (cid:96) 1 -regularization (lasso) to logistic regression and can effectively balance the performance of the model and the sparsity of features.
The update formula (McMahan et al., 2013) is:
[Equation]
where g 1: t = (cid:80) ts =1 g s , 12 (cid:80) ts =1 σ s (cid:107) x − x s (cid:107) 22 is the strong convex term that stabilizes the algorithm and λ 1 (cid:107) x (cid:107) 1 is the regularization term that produces sparsity.
However, it doesn’t work well in DNN models since one input feature can correspond to multiple weights and lasso only can make single weight zero hence can’t effectively delete zeros features.
To solve above problem, Ni et al.
(2019) adds the (cid:96) 21 -regularization (group lasso) to FTRL, which is named G-FTRL.
Yang et al.
(2010) conducts the research on a group lasso method for online learning that adds (cid:96) 21 -regularization to the algorithm of Dual Averaging (DA) (Nesterov, 2009), which is named DA-GL.
Even so, these two methods cannot been applied to other optimizers.
Different scenarios are suitable for different optimizers in the deep learning ﬁelds.
For example, M OMENTUM (Polyak, 1964) is typically used in computer vision; A DAM (Kingma & Ba, 2015) is used for training transformer models for natural language processing; and A DAGRAD (Duchi et al., 2011) is used for recommendation systems.
If we want to produce sparsity of the model in some scenario, we have to change optimizer which probably inﬂuence the performance of the model.
1. M OTIVATION
Eq.
(1) can be rewritten into this form:
[Equation]
Furthermore, we can rewrite Eq.
(3) into
[Equation] [Equation]
where m 1: t = (cid:80) ts =
m s , (cid:80) ts =1 Q s /α s = √ V t /α t .
It is easy to prove that Eq.
(3) and Eq.
(4) are equivalent using the method of induction.
The matrices Q s can be interpreted as generalized learning rates.
To our best knowledge, V t of Eq.
(1) of all the adaptive optimization methods are diagonal for the computation simplicity.
Therefore, we consider Q s as diagonal matrices throughout this paper.
We ﬁnd that Eq.
(4) is similar to Eq.
(2) except for the regularization term.
Therefore, we add the regularization term Ψ
( x ) to Eq.
(4), which is the sparse group lasso penalty also including (cid:96) 2 - regularization that can diffuse weights of neural networks.
The concrete formula is:
[Equation]
5) where λ 1 , λ 21 , λ 2 are regularization parameters of (cid:96) 1 , (cid:96) 21 , (cid:96) 2 respectively, G is the total number of groups of weights, x g is the weights of group g and d x g is the size of group g .
In DNN models, each group is deﬁned as the set of outgoing weights from a unit which can be an input feature, or a hidden neuron, or a bias unit (see, e.g., Scardapane et al.
A t can be arbitrary positive matrix satisfying A t +1 (cid:23)
A t , e.g., A t = I .
In Section 2.1, we let A t =
( (cid:80) ts =1 Q
gs
α s + λ 2 I ) just for solving the closed-form solution directly, where Q gs is a diagonal matrix whose diagonal elements are part of Q s corresponding to x g .
The ultimate update formula is:
[Equation]
1. O UTLINE OF C ONTENTS
The rest of the paper is organized as follows.
In Section 1.5, we introduce the necessary notations and technical background.
In Section 2, we present the closed-form solution of Eq.
(4) and the algorithm of general framework of adaptive optimization methods with sparse group lasso.
We prove the algorithm is equivalent to adaptive optimization methods when regularization terms vanish.
In the end, we give two concrete examples of the algorithm.
In Section 3, we derive the regret bounds of the method and convergence rates.
In Section 4, we validate the performance of new optimizers in the public datasets.
In Section 5, we summarize the conclusion.
Appendices A-B list the details of G ROUP A DAM and Group Adagrad respectively.
Appendices C-F contain technical proofs of our main results and Appendix G includes the details of the empirical results of Section 4.4.
1. N OTATIONS AND T ECHNICAL B ACKGROUND
We use lowercase letters to denote scalars and vectors, and uppercase letters to denote matrices.
We denote a sequence of vectors by subscripts, that is, x 1 , . . .
, x t , and entries of each vector by an additional subscript, e.g., x t,i .
We use the notation g 1: t as a shorthand for (cid:80) ts =1 g s .
Similarly we write m 1: t for a sum of the ﬁrst moment m t , and f 1: t to denote the function f 1: t ( x ) = (cid:80) t s =1 f s ( x ) .
Let M t =
[ m 1 · · · m t ] denote the matrix obtained by concatenating the vector sequence { m t } t
≥and M t,i denote the i -th row of this matrix which amounts to the concatenation of the i -th component of each vector.
The notation A (cid:23) 0 (resp.
A (cid:31) 0 ) for a matrix A means that A is symmetric and positive semideﬁnite (resp.
deﬁnite).
Similarly, the notations A (cid:23) B and A (cid:31)
B mean that
A − B (cid:23) 0 and A − B (cid:31) 0 respectively, and both tacitly assume that A and B are symmetric.
Given
A (cid:23) 0 , we write A 12 for the square root of A , the unique X (cid:23) 0 such that XX =
A (McMahan &
Streeter (2010), Section 1.4).
Let E be a ﬁnite-dimension real vector space, endowed with the Mahalanobis norm (cid:107) · (cid:107)
A which is denoted by (cid:107)·(cid:107)
A = (cid:112) (cid:104)· , A ·(cid:105) as induced by A (cid:31) 0 .
Let E ∗ be the vector space of all linear functions on E .
The dual space E ∗ is endowed with the dual norm (cid:107) · (cid:107) ∗ A
= (cid:112) (cid:104)· , A − 1 ·(cid:105) .
Let Q be a closed convex set in E .
A continuous function h ( x ) is called strongly convex on Q with norm (cid:107)·(cid:107) H if Q ⊆ dom h
and there exists a constant σ > 0 such that for all x, y ∈ Q and α ∈ [0 , 1] we have
[Equation]
The constant σ is called the convexity parameter of h ( x ) , or the modulus of strong convexity.
We also denote by (cid:107) · (cid:107)
h
= (cid:107) ·
[Equation]
H .
Further, if h is differential, we have
[Equation]
We use online convex optimization as our analysis framework.
On each round t = 1 , . . .
, T , a convex loss function f t : Q (cid:55)→ R is chosen, and we pick a point x t ∈ Q hence get loss f t ( x t ) .
Our goal is minimizing the regret which is deﬁned as the quantity
[Equation]
Online convex optimization can be seen as a generalization of stochastic convex optimization.
Any regret minimizing algorithm can be converted to a stochastic optimization algorithm with convergence rate O
( R T /T )
using an online-to-batch conversion technique (Littlestone, 1989).
In this paper, we assume Q ≡ E = R n , hence we have E ∗ =
R n .
We write s T x or s · x for the standard inner product between s, x ∈ R n .
For the standard Euclidean norm, (cid:107) x (cid:107) =
(cid:107) x (cid:107) 2 = (cid:112) (cid:104) x, x (cid:105) and (cid:107) s (cid:107) ∗
=
(cid:107) s (cid:107) 2 .
We also use (cid:107) x (cid:107)
= (cid:80) ni
=1 | x ( i ) | and (cid:107) x (cid:107) ∞ =
max i | x ( i ) | to denote (cid:96) 1 -norm
and (cid:96) ∞ -norm respectively, where x ( i ) is the i -th element of x .
A LGORITHM 2. C LOSED - FORM S OLUTION
We will derive the closed-form solution of Eq.
(6) with speciﬁc A t and Algorithm 1 with slight modiﬁcation in this section.
We have the following theorem.
Theorem 1.
Given A t = ( (cid:80) ts =1 Q gs 2 α s + λ 2 I ) of Eq.
(5) , z t =
z t − 1 + m t − Q t α t x t at each iteration t = 1 , . . .
, T and z 0 = 0 , the optimal solution of Eq.
(6) is updated accordingly as follows:
[Equation]
where the i -th element of s t is deﬁned as
[Equation] [Equation]
otherwise
[Equation]
(9) ˜ s t is deﬁned as
[Equation] [Equation]
and (cid:80) ts =1 Q s α s is the diagonal and positive deﬁnite matrix.
The proof of Theorem 1 is given in Appendix C.
We slightly modify (8) where we let ˜ s t = s t .
Our purpose is to let every entry of the group have the same effect of (cid:96) 21 -regularization.
Hence, we get
Algorithm 1.
Furthermore, we have the following theorem which shows the relationship between
Algorithm 1 and adaptive optimization methods.
The proof is given in Appendix D.
Theorem 2.
If regularization terms of Algorithm 1 vanish, Algorithm 1 is equivalent to Eq.
2. C ONCRETE E XAMPLES
Using Algorithm 1, we can easily derive the new optimizers based on A DAM (Kingma & Ba, 2015),
A DAGRAD (Duchi et al., 2011) which we call G ROUP A DAM , G ROUP
A DAGRAD respectively.
G ROUP A DAM
The detail of the algorithm is given in Appendix A. From Theorem 2, we know that when λ 1 , λ 2 , λ 21 are all zeros, Algorithm 2 is equivalent to A DAM (Kingma & Ba, 2015).
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
G ROUP A DAGRAD
The detail of the algorithm is given in Appendix B. Similarly, from Theorem 2, when λ 1 , λ 2 , λare all zeros, Algorithm 3 is equivalent to A DAGRAD (Duchi et al., 2011).
Furthermore, we can ﬁnd that when λ 21 = 0 , Algorithm 3 is equivalent to FTRL (McMahan et al., 2013).
Therefore, G ROUP
A DAGRAD can also be called G ROUP FTRL from the research of Ni et al.
Similarly, G ROUP M OMENTUM , G ROUP AMSG RAD , G ROUP
A DA H ESSIAN , etc., can be derived from M OMENTUM (Polyak, 1964), AMSG RAD (Reddi et al., 2018), A DA H ESSIAN (Yao et al., 2020), etc., with the same framework and we will not list the details.
C ONVERGENCE AND R EGRET A NALYSIS
Using the framework developed in Nesterov (2009)
; Xiao (2010); Duchi et al.
(2011), we have the following theorem providing the bound of the regret.
Theorem 3.
Let the sequence { x t } be deﬁned by the update (6) and
[Equation]
where c is an arbitrary constant vector.
Suppose f t ( x ) is convex for any t ≥ 1 and there exists an optimal solution x ∗ of (cid:80)
Tt =1 f t ( x ) , i.e., x ∗ =
arg min x ∈Q (cid:80)
Tt =1 f t ( x ) , which satisﬁes the condition
[Equation]
where m t is the weighted average of the gradient f t ( x t ) and [ T ] = { 1 , . .
. , T } for simplicity.
Without loss of generality, we assume
[Equation] [Equation]
where γ < 1 and m 0 = 0 .
Then
[Equation] [Equation]
where (cid:107)·(cid:107) h ∗ t is the dual norm of (cid:107)·(cid:107) h t .
h t is 1 -strongly convex with respect to (cid:107)·(cid:107) √ V t /α t for t ∈ [ T ] and h 0 is 1 -strongly convex with respect to (cid:107) · (cid:107) 2 .
The proof of Theorem 3 is given in Appendix E.
Since in most of adaptive optimizers, V t is the weighted average of diag ( g 2 t ) , without loss of generality, we assume α t = α and
[Equation]
where V
=
and η ≤ 1 .
Hence, we have the following lemma whose proof is given in Appendix F.1.
Lemma 1.
Suppose V t is the weighted average of the square of the gradient which is deﬁned by (15) , α t
= α , m t is deﬁned by (13) and V t satisﬁes the following arbitrary conditions:
η = 1 , 2.
η < 1 , η ≥ γ and κV t (cid:23) V t − 1 for all t ≥ 1 where κ < 1 .
Then we have
[Equation]
where ν = max( γ, κ ) and d is the dimension of x t .
We can always add δ 2 I to V t at each step to ensure V t (cid:31) 0 .
Therefore, h t ( x ) is 1 -strongly convex with respect to (cid:107) · (cid:107) √
δ 2 I + V t /α t .
Let δ ≥ max t ∈
[ T ] (cid:107) g t (cid:107) ∞ , for t > 1 , we have
[Equation]
For t = 1 , we have
[Equation]
From (17), (18) and Lemma 1, we have
Lemma 2.
Suppose V t , m t , α t , ν , d are deﬁned the same as Lemma 1,
max t ∈
[ T ] (cid:107) g t (cid:107) ∞ ≤ δ , (cid:107) · (cid:107) 2 h ∗ t = (cid:68) · , α t ( δ 2 I + V t ) − 12 · (cid:69) for t ≥ 1 and (cid:107) · (cid:107) 2 h ∗
= (cid:68) · , α 1 (cid:0) ( δ 2 + 1) I (cid:1) − 12 · (cid:69) .
Then
[Equation]
Therefore, from Theorem 3 and Lemma 2, we have
Corollary 1.
Suppose V t , m t , α t , h ∗ t , ν , d are deﬁned the same as Lemma 2, there exist constants
G , D 1 , D 2 such that max t ∈
[ T ] (cid:107) g t (cid:107) ∞
≤ G ≤ δ , (cid:107) x ∗ (cid:107) ∞ ≤ D 1 and max t ∈ [ T ] (cid:107) x t − x ∗ (cid:107) ∞ ≤ D 2 .
Then
[Equation]
The proof of Corollary 1 is given in F.2.
Furthermore, from Corollary 1, we have
Corollary 2.
Suppose m t is deﬁned as (13) , α t = α and satisﬁes the condition (19) .
There exist constants G , D 1 , D 2 such that tG 2 I (cid:23) V t , max t ∈ [ T ] (cid:107) g t (cid:107) ∞ ≤ G , (cid:107) x ∗ (cid:107) ∞ ≤ D 1 and max t ∈ [ T ] (cid:107) x t − x ∗ (cid:107) ∞ ≤ D 2 .
Then
[Equation]
Therefore, we know that the regret of the update (6) is O ( √ T ) and can achieve the optimal convergence rate O (1 / √ T ) under the conditions of Corollary 1 or Corollary 2.
E XPERIMENTS 4. E XPERIMENT S ETUP
We test the algorithms on three different large-scale real-world datasets with different neural network structures.
These datasets are various display ads logs for the purpose of predicting ads CTR.
The details are as follows.
a) The Avazu CTR dataset (Avazu, 2015) contains approximately 40M samples and 22 categorical features over 10 days.
In order to handle categorical data, we use the one-hot-encoding based embedding technique (see, e.g., Wang et al.
(2017), Section 2.1 or Naumov et al.
(2019), Section 2.1.1) and get 9.4M features in total.
For this dataset, the samples from the ﬁrst 9 days (containing 8.7M one-hot features) are used for training, while the rest is for testing.
Our DNN model follows the basic structure of most deep CTR models.
Speciﬁcally, the model comprises one embedding layer, which maps each one-hot feature into 16-dimensional embeddings, and four fully connected layers (with output dimension of 64, 32, 16 and 1, respectively) in sequence.
b)
The iPinYou dataset 2 (iPinYou, 2013) is another real-world dataset for ad click logs overdays.
The dataset contains 16 categorical features 3 .
After one-hot encoding, we get a dataset containing 19.5M instances with 1033.1K input dimensions.
We keep the original train/test splitting scheme, where the training set contains 15.4M samples with 937.7K one-hot features.
We use Outer Product-based Neural Network (OPNN) (Qu et al., 2016), and follow the standard settings of Qu et al.
(2016), i.e., one embedding layer with the embedding dimension of 10, one product layer and three hidden layers of size 512, 256, 128 respectively where we set dropout rate at 0.5.
c)
The third dataset is the Criteo Display Ads dataset (Criteo, 2014) which contains approximately 46M samples over 7 days.
There
are 13 integer features and 26 categorical features.
After onehot encoding of categorical features, we have total 33.8M features.
We split the dataset intopartitions in chronological order and select the earliest 6 parts for training which contains 29.6M features and the rest for testing though the dataset has no timestamp.
We use Deep & Cross
Network (DCN) (Wang et al., 2017) and choose the following settings 4 : one embedding layer with embedding dimension 8, two deep layers of size 64 each, and two cross layers.
For the convenience of discussion, we use MLP, OPNN and DCN to represent the aforementioned three datasets coupled with their corresponding models.
It is obvious that the embedding layer has most of parameters of the neural networks when the features have very high dimension, therefore we just add the regularization terms to the embedding layer.
Furthermore, each embedding vector is considered as a group, and a visual comparison between (cid:96) 1 , (cid:96) 21 and mixed regularization effect is given in Fig.
2 of Scardapane et al.
We treat the training set as the streaming data, hence we train 1 epoch with a batch size ofand do the validation.
The experiments are conducted with 4-9 workers and 2-3 parameter servers, which depends on the different sizes of the datasets.
We use the area under the receiver-operator curve (AUC) as the evaluation criterion since it is widely used in evaluating classiﬁcation problems.
Besides, some work validates AUC as a good measurement in CTR estimation (Graepel et al., 2010).
We explore 5 learning rates from 1 e-5 to 1 e-1 with increments of 10x and choose the one with the best AUC for each new optimizer in the case of no regularization terms (It is equivalent to the original optimizer according to Theorem 2).
All the experiments are run 5 times repeatedly and tested statistical signiﬁcance using t-test.
Without loss of generality, we choose two new optimizers to validate the performance, which are G ROUP
A
DAM
and G ROUP A DA G RAD .
4. A DAM VS . G ROUP A DAM
First, we compare the performance of the two optimizers on the same sparsity level.
We keep λ 1 , λ 2 be zeros and choose different values of λ 21 of Algorithm 2, i.e., G ROUP A DAM , and achieve the same sparsity with A DAM that uses the magnitude pruning method, i.e., sort the norm of embedding vector from largest to smallest, and keep top N embedding vectors which depend on the sparsity when ﬁnish the training.
Table 2 reports the average results of the two optimizers in the three datasets.
Note that G ROUP A DAM signiﬁcantly outperforms A DAM on the AUC metric on the same sparsity level for most experiments.
Furthermore, as shown in Figure 1, the same (cid:96) 21 -regularization strength λ 21 has different effects of sparsity and accuracy on different datasets.
The best choice of λ 21 depends on the dataset as well as the application (For example, if the memory of serving resource is limited, sparsity might be relative more important).
One can trade off accuracy to get more sparsity by increasing the value of λ
[Table] [Table]
1 e0.
5 e0.
(4.2e-3) 0.
(4.2e-3) 9.11e1 e0.
(0.038) 3.73e0.
(9.2e-3) 0. (9.2e-3) 0.
(2.5e-3) 0.
(2.5e-3) 5.97e-
[Figure]
vectors are not equal to 0 divided by the total number of features present in the training data).
The y-axis is AUC. Next, we compare the performance of A DAM without post-processing procedure, i.e., no magnitude pruning, and G ROUP A DAM with appropriate regularization terms which we choose in Table 3 on the AUC metric.
In general, good default settings of λ 2 is 1 e-5.
The results are shown in Table 4.
Note that compared with A DAM , G ROUP A DAM with appropriate regularization terms can achieve signiﬁcantly better or highly competitive performance with producing extremely high sparsity.
4. A DAGRAD VS . G ROUP A DAGRAD
We compare with the performance of A DAGRAD without magnitude pruning and G ROUP
A DAGRAD with appropriate regularization terms which we choose in Table 5 on the AUC metric.
The results are shown in Table 6.
Again note that in comparison to A DAGRAD , G ROUP
A DAGRAD can not only achieve signiﬁcantly better or highly competitive performance of AUC, but also effectively and efﬁciently reduce the dimensions of the features.
[Table] [Table]
[Table] [Table]
4. D ISCUSSION
In this section we will discuss the hyperparameters of emdedding dimension, (cid:96) 1 -regularization
and (cid:96) 21 -regularization to show how these hyperparameters affect the effects of regularization.
Embedding Dimension Table 7 of Appendix G reports the average results of different embedding dimensions of MLP, whose optimizer is G ROUP A DAM and regularization terms are same to MLP of Table 5.
Note that the sparsity increases with the growth of the embedding dimension.
The reason is that the square root of the embedding dimension is the multiplier of (cid:96) 21 -regularization.
(cid:96) 1 vs. (cid:96)
From lines 8 and 10 of Algorithm 1, we know that if z t has the same elements, the values of (cid:96) 1 and (cid:96) 21 , i.e., λ 1 and λ 21 , have the same regularization effects.
However, this situation almost cannot be happen in reality.
Without loss of generality, we set optimizer, λ 2 and embedding dimension be G ROUP
A DAM , 1 e-5 and 16 respectively, and choose different values of λ 1 , λ 21 .
The results on MLP are shown in Table 8 of Appendix G.
It is obvious that (cid:96) 21 -regularization is much more effective than (cid:96) 1 -regularization in producing sparsity.
For example, when λ 1 = 0 and λ 21 = 5 e-
3 , the feature sparsity is 0.136, while for λ 1 = 5 e- 3 and λ 21 = 0 , the feature sparsity is 0.470.
Therefore, if just want to produce sparsity, we can only tune λ 21 and use default settings for λ 2 and λ 1 , i.e., λ 2 = 1 e-5 and λ 1 = 0 .
C ONCLUSION
In this paper, we propose a novel framework that adds the regularization terms to a family of adaptive optimizers for producing sparsity of DNN models.
We apply this framework to create a new class of optimizers.
We provide closed-form solutions and algorithms with slight modiﬁcation.
We built the relation between new and original optimizers, i.e., our new optimizers become equivalent with the corresponding original ones, once the regularization terms vanish.
We theoretically prove the convergence rate of the regret and also conduct empirical evaluation on the proposed optimizers in comparison to the original optimizers with and without magnitude pruning.
The results clearly demonstrate the advantages of our proposed optimizers in both getting signiﬁcantly better performance and producing sparsity.
Finally, it would be interesting in the future to investigate the convergence in non-convex settings and evaluate our optimizers on more applications from ﬁelds such as compute vision, natural language processing and etc.
B G ROUP A DAGRAD
1: Input
[Table]
parameters
[Table] [Table]
[Table] [Table]
x
[Table]
R
[Table] [Table]
size
[Table]
, initialize z
[Table]
= 0 , V
=2: for t = 1 to T do 3: g t = ∇ f t ( x t ) 4: m t =
g t 5:
V t = (cid:26) V t − 1 + diag ( g 2 t ) + (cid:15)
I t
= 1 V t − 1 + diag
( g 2 t ) t >6:
Q t = √ V t − √ V t −7: z t ← z t − 1 + m t − 1 α Q t x t 8: for i ∈ { 1 , . . . , n } do 9: s t,i = (cid:26)if | z t,i | ≤ λ 1 sign( z t,i ) λ 1 − z t,i otherwise.
10: end for 11: x t +1 =
( √ V t α + 2 λ 2 I ) − 1 max(1 − (cid:113)
d xgt λ 21 (cid:107) s t (cid:107) 2 , 0)
s t 12: end for
C P ROOF OF T HEOREM
Proof.
[Equation]
Deﬁne z t − 1 = m 1: t − 1 − (cid:80)
[Equation]
s x s
[Equation]
2 ) and we can calculate z t as
[Equation]
By substituting (23), (22) is simpliﬁed to be
[Equation]
By substituting Ψ t ( x ) (Eq.
(5)) into (24)
, we get
[Equation]
Since the objective of (25) is component-wise and element-wise, we can focus on the solution in one group, say g , and one entry, say i , in the g -th group.
Let (cid:80) ts =1 Q gs 2 α s = diag
( σ gt ) where σ gt = ( σ gt, 1 , . . .
, σ gt,d xg ) .
The objective of (25) on x gt +1 ,i is
[Equation]
26) where Φ( x gt +1 ,i ) =
λ 21 √ d x g (cid:107) ( σ gt,i + λ 2 ) 12 x gt +1 ,i (cid:107)
+ (cid:107) ( σ gt,i + λ 2 ) 12 x gt +1 ,i (cid:107) 22 is a non-negative function and Φ( x gt +1 ,i ) =
iff x gt +1 ,i =
for all i ∈ { 1 , . . .
d x g } .
We discuss the optimal solution of (26) in three cases:
a) If z gt,i = 0 , then x gt +1 ,i = 0 .
b) If z gt,i > 0 , then x gt +1 ,i ≤ 0 .
Otherwise, if x gt +1 ,i > 0 , we have Ω( − x gt +1 ,i ) <
Ω( x gt
+1 ,i ) , which contradicts the minimization value of Ω( x ) on x gt +1 ,i .
Next, if z gt,i ≤ λ 1 , then x gt +1 ,i = 0 .
Otherwise, if x gt +1 ,i < 0 , we have Ω( x gt +1 ,i ) =
( z gt,i − λ 1 ) x gt
+1 ,
i + Φ( x g,it +1 ) > Ω(0) , which also contradicts the minimization value of Ω( x ) on x gt +1 ,i .
Third, z gt,i > λ 1 ( ∀ i = 1 , . . .
, d x g ) .
The objective of (26) for the g -th group, Ω( x gt +1 ) , becomes
[Equation]
c) If z gt,i < 0 , the analysis is similar to b).
We have x gt +1 ,i ≥ 0 .
When − z gt,i ≤ λ 1 , x gt +1 ,i = 0 .
When − z gt,i > λ 1 ( ∀ i = 1 , . . .
, d x g ) , we have Ω( x gt +1 ) =
( z gt + λ 1 1 d xg ) · x gt +1 + Φ( x gt +1 ) .
From
a
), b), c)
above
we have
[Equation]
where the i -th element of s gt is deﬁned same as (9).
Deﬁne
[Equation]
By substituting (28) into (27), we get
[Equation]
where ˜ s gt =
( diag ( σ gt )+ λ 2 I ) − 1 s gt which is deﬁned same as (10).
This is unconstrained non-smooth optimization problem.
Its optimality condition (see Rockafellar (1970), Section 27) states that y gt + is an optimal solution if and only if there exists ξ ∈ ∂ (cid:107)
y gt
+1 (cid:107) 2 such that
[Equation]
[Equation]
The subdifferential of (cid:107) y (cid:107) 2 is
[Equation]
Similarly to the analysis of (cid:96) 1 -regularization, we discuss the solution of (30) in two different cases: a) If (cid:107) ˜ s gt (cid:107) 2 ≤ λ 21 √ d x g , then y gt +
= 0 and ξ =
[Equation]
d
[Equation]
∂ (cid:107)
(cid:107) 2 satisfy (30).
We also show that there is no solution other than y gt +
= 0 .
Without loss of generality, we assume y gt
+1 ,i (cid:54) =
for all i ∈ { 1 , . . .
, d x g } , then ξ = y gt +1 (cid:107) y gt
+1 (cid:107) 2 , and
[Equation]
From (31), we can derive
[Equation]
Furthermore, we have
[Equation] [Equation]
where (cid:107) y gt
+1 (cid:107) 2 > 0 and (cid:107) ˜ s gt (cid:107) 2 − λ 21 √ d x g ≤ 0 contradict each other.
b)
If
(cid:107) ˜ s
gt
(cid:107) 2 > λ 21 √ d x
g , then from (31) and (32), we get
[Equation]
We replace y gt +1 of (33) by x gt +1 using (28), then we have
[Equation]
Combine a) and b) above, we ﬁnish the proof
[Equation]
D P ROOF OF T HEOREM
Proof.
We use the method of induction.
a) When t = 1 , then Algorithm 1 becomes
[Equation]
which equals to
b) Assume t =
T , Eq.
(35) are true.
[Equation]
For t = T + 1 , we have
[Equation]
Hence, we complete the proof.
E P ROOF OF T HEOREM
Proof.
Let h t ( x ) =
[Equation]
t
[Equation]
It is easy to verify that for all t ∈
[ T ] , h t ( x ) is 1 -strongly convex with respect to (cid:107) · (cid:107) √ V t /α t which √
V t α t = (cid:80) ts =1 Q s α s , and h 0 ( x ) is 1 -strongly convex with respect to (cid:107) · (cid:107) 2 .
From (7), we have
[Equation]
where in the ﬁrst and second inequality above, we use the convexity of f t ( x ) and the condition (12) respectively.
We deﬁne h ∗ t ( u ) to be the conjugate dual of Ψ t ( x ) + h t ( x ) :
[Equation]
where Ψ 0 ( x ) = 0 .
Since h t ( x ) is 1 -strongly convex with respect to the norm (cid:107) · (cid:107) h t , the function h ∗ t has 1 -Lipschitz continuous gradients with respect to (cid:107) · (cid:107) h ∗ t (see, Nesterov (2005), Theorem 1):
[Equation]
37) and
[Equation]
As a trivial corollary of (37), we have the following inequality:
[Equation]
[Equation]
Since h t +1 ( x ) ≥ h t ( x ) and Ψ t +1 ( x ) ≥ Ψ t ( x ) , from (38), (39), (6), we have
[Equation]
[Equation]
where the last equality above follows from h ∗ 0 (0)
=
and (11) which deduces x 1 = ∇ h ∗ 0 (0) .
By substituting (40), (36) becomes
[Equation]
F A DDITIONAL P ROOFS F. P ROOF OF L EMMA
Proof.
Let V t =
diag ( σ t ) where σ t is the vector of the diagonal elements of V t .
For i -th entry of σ t , by substituting (13) into (15), we have
[Equation]
Next, we will discuss the value of η in two cases.
a) η = 1 .
From (42), we have
[Equation] [Equation]
Recalling the deﬁnition of M t,i in Section 1.5, from (43), we have
T (cid:88)
[Equation]
where the last inequality above follows from Appendix C of Duchi et al.
Therefore, we get
[Equation] [Equation]
b) η < 1 .
We assume η ≥ γ and κV t (cid:23) V t − 1 where κ < 1 , then we have
[Equation]
Hence, we get
[Equation]
which deduces the same conclusion (44) of a).
Combine a) and b), we complete the proof.
F. P ROOF OF C OROLLARY
Proof.
From the deﬁnition of m t (13), V t (15), we have
[Equation]
Hence, we have
[Equation] [Equation]
Combining (46), (47), (48), we complete the proof.
G A DDITIONAL E MPIRICAL R ESULTS
[Table]
It is signiﬁcantly better than embedding dimensions of 4, 8 but has no difference in 95% conﬁdence level of the embedding dimension of
[Table] [Table]