UMEC: U NIFIED MODEL AND EMBEDDING COMPRESSION FOR EFFICIENT RECOMMENDATION SYSTEMS
Anonymous authors
Paper under double-blind review
A BSTRACT
The recommendation system (RS) plays an important role in the content recommendation and retrieval scenarios.
The core part of the system is the ranking neural network, which is usually a bottleneck of whole system performance during online inference.
Hammering an efﬁcient neural network based recommendation system involves entangled challenges of compressing both the network parameters and the feature embedding inputs.
We propose a uniﬁed model and embedding compression ( UMEC ) framework to jointly learn input feature selection and neural network compression together, which is formulated as a resource-constrained optimization problem using the alternating direction method of multipliers (ADMM) algorithm.
Experimental results on public benchmarks show that our UMEC framework notably outperforms other non-integrated baseline methods.
I NTRODUCTION
As the core component of a recommendation system (RS), recommendation models (RM) based on ranking neural networks are widely adopted in general content recommendation and retrieval applications.
In general, an effective recommendation model consists of two components: a feature embedding sub-model and a prediction sub-model, as illustrated in Figure 1.
Usually, RM adopts neural networks to serve for two sub-models.
Formally, we denote a RM as f ( · ; W ) , where W is the learnable parameters of f .
For the inference, the model f takes the input feature data x to predict the conﬁdence of a content, serving the recommendation applications.
Speciﬁcally, we further deﬁne the embedding and prediction sub-models as f e ( · ; W e ) and f p ( · ; W p ) respectively, where W e and W p are their own learnable parameters and W = {W e , W p } .
The embedding feature, v := f e ( x ; W e ) , is the input of f p with the input data x .
Hence, we can express the RM as f ( · ; W ) (cid:44) f p ( f e ( · ; W e ); W p ) .
Given a ranking training loss (cid:96) ( · ) (i.e., binary cross entropy (BCE) loss), the ranking model learning goal can be written as [Equation] where y is a groud-truth label, and D is the training dataset.
[Equation]
[Figure]
Nowadays, extremely large scale datas are poured into the recommendation system to predict user behaviors in many applications.
In the online inference procedure, the most heavy computation component is the layer-wise product between the hidden output vectors and the model parameters W , for a neural network based RM.
A slimmed neural network structure would save a great amount of power consumption during the inference.
Hence, the main idea of a RM compression is to slim the structural complexity of f ( W ) and reduce the dimension of hidden output vectors.
To obtain an efﬁcient ranking model (for example,
MLP based) for a RS, some existing model compression methods can be applied on MLP structure directly.
For example, Li et al.
(2016) removes the entire ﬁlters in the network together with their connecting feature maps in terms of magnitudes, which can also be applied on MLP structure to remove a speciﬁc neuron as well as its connections.
Molchanov et al.
(2019) approximates the importance of a neuron (ﬁlter) to the ﬁnal loss by using the ﬁrst and second-order Taylor expansions, which can also be applied to prune the neurons without hassle.
There are also some compression methods focusing on dimension reduction of embedding feature vectors.
Ginart et al.
(2019) proposes a mixed dimension embedding scheme by designing
non-uniform embedding dimensions scaling with popularity of features.
Joglekar et al.
(2020) uses Neuron Input Search (NIS) method to learn the embedding dimensions for the sparse categorical features.
However, the performance of these compression methods highly depends on the hyper-parameter tuning and the background knowledge of the speciﬁc recommendation model.
For example, an embedding dimension compression method may require the user to search the best dimension with multiple training and search rounds.
We would like to solve the RM compression problem with a uniﬁed resource constrained optimization framework.
It only relies on the resource consumption of the RM f ( W ) to compress both the MLP and the embedding dimensions without expensive hyper-parameters tuning.
Our novel uniﬁed model and embedding compression ( UMEC ) method directly satisﬁes both the requirement of resource consumption of the ranking neural network model and the prediction accuracy goal of the ranking model, with a uniﬁed gradient-based training strategy.
We reformulate the optimization of training loss associated with hard constraints into a minimax optimization problem solved by ADMM method (Boyd et al., 2011).
To summarize our contributions, we present the merits of UMEC as the following:
• To the best of our knowledge, UMEC is the ﬁrst uniﬁed optimization framework for the recommendation system scenarios.
Unlike those existing works that treat selection of input feature and compression of model as two individual problems, UMEC jointly learns both together via unifying both original prediction learning goal and the model compression related hard constraints.
• We reformulate the joint input feature selection and model compression task as a constrained optimization problem.
We convert resource constraints and L 0 sparsity constraints into soft optimization energy terms and solve the whole optimization using ADMM methods.
• Extensive experiments performed over an advertising dataset show that our method largely outperforms previous state-of-the-art input feature selection methods and model compression methods.
R ELATED W ORK
Recommendation Models With the recent development of deep learning techniques, plenty of works have proposed learning-based recommendation systems to grapple with the personalized recommendation tasks.
Hidasi et al.
(2015) applies recurrent neural networks (RNN) on long session data to model the whole session and achieve better accuracy than the traditional matrix factorization approaches.
Covington et al.
(2016) proposes a high-level recommendation algorithm for YouTube videos by specifying a candidate generation model followed by a separate ranking model, which demonstrates a signiﬁcant performance boost thanks to neural networks.
Wu et al.
(2019) proposes a recommendation algorithm based on graph neural network to better capture the complex transitions among items, achieving more accurate item embedding.
Their method outperforms traditional sequential approaches.
Naumov et al.
(2019) exploits categorical features and designs a parallelism scheme for the embedding tables to alleviate the limited memory problem.
Huang et al.
(2020) proposes a multi-attention based model to mitigate the low efﬁciency problem in the group recommendation scenarios.
Input Feature Compression for Recommendation Models
The enormous input features for practical recommendation scenarios necessitate the selection of useful ones to save memory resource and facilitate the computational efﬁciency during inference.
In factorization machines(FM)-based and context-aware recommendation systems, for example, Mao et al.
(2017) conducts input feature selection in an automatic manner by feature ranking and feature sub-sampling, while managing to improve the prediction quality.
Ginart et al.
(2019) designs a mixed dimension embedding layers scheme and adjusts the dimension of a particular embedding feature according to the frequency of that item.
Joglekar et al.
(2020) performs Neural Input Search (NIS) method to search the embedding sizes for the sparse categorical features, and designs the multi-size embedding framework to leverage the model capacity more efﬁciently.
Model Compression Model compression aims to reduce the complexity of Deep Neural Network (DNN) models to achieving inference efﬁciency and resource saving.
Generally, model compression techniques can be categorized into pruning (Han et al., 2015; Wen et al., 2016; Molchanov et al., 2019), quantization (Hubara et al., 2017; Ajanthan et al., 2019) and distillation (Hinton et al., 2015;
Polino et al., 2018; Tung & Mori, 2019).
Han et al.
(2015) proposes a magnitude-based and elementwise pruning technique, which does not guarantee the reduction of computation efﬁciency, energy and memory costs.
Structured pruning can solve this issue by removing whole certain type of structures from the network, such as ﬁlter pruning and channel pruning methods (Li et al., 2016;
He et al., 2017; Molchanov et al., 2019).
In the meanwhile, researchers also have great interest in investigating resource-constrained compression, which utilizes practical resource requirements(e.g., energy, latency, Flops) to guide the compression (He et al., 2018;
Yang et al., 2019; Liu et al., 2019).
U NIFIED M ODEL AND E MBEDDING C OMPRESSION ON RS
For an arbitrary NN-based recommendation model f ( · ) in a RS, the terminology of “compression” refers to reduce the power consumption with respect to all the computation operations for the inference.
The learning parameters of f ( W ) can be expressed as W :=
(cid:8) W ( l ) | l ∈ [ L ] (cid:9) , where there are L layers in the f ( W ) .
For an arbitrary layer l , the layer’s weight is denoted as W ( l ) ∈ R d out × d in .
The recommendation inference on the layer l is executed as
[Equation]
where x ( l ) ∈ R d in is the input of l -th layer, and b ( l ) is the corresponding learning bias term.
Since the RM consists of two components: feature embedding sub-model and neural network prediction sub-model, the “compression” can be interpreted as shrinking the dimension of embedding feature v and reducing the number of parameters in W and b .
Perspective of model parameters W and b It is to reduce the input or output neurons for the speciﬁc layer l .
For example, the i -th neuron of x ( l +1) and j -th neuron of x ( l ) are chosen to remove, then the weight matrix W ( l ) will be reshaped to ( d out − 1) ×
( d in − 1) .
Perspective of feature embedding vectors v
The reduction of d in would result in the elimination on the input features.
For example, v (1) is deﬁned as the concatenation of a series of embedding vectors, v (1) =
[ e 1 ; e 2 ; · · · ; e n ] , where [ · ; · ; · ] represents the concatenation operator, and n denotes the number of embedding feature vectors in the RS.
Without loss of generality, for the ﬁrst embedding vector e 1 ∈ R d 1 , the zero-out of ﬁrst d 1 columns in W (1) will result in the elimination of the whole embedding vectors for e 1 .
On the other hand, the compression on f e ( W e ) would reduce the feature embedding vectors’ dimensions.
3. A COMPRESSION EXAMPLE : G ROUP L ASSO FOR W EIGHT P RUNING
We ﬁrst consider a simple example with a RS model consists of f e ( W e ) and f p ( W p ) .
For this RS, to reduce the computation consumption, we present a straightforward model compression method with group lasso.
The method considers both the feature embedding selection and network slimming.
We only need to apply the sparse regularization terms on the weight matrices.
The optimization problem can be expressed as
[Equation] [Equation]
where (cid:96) ( W ) is the original objective function .
The λ s are hyper-parameters to control the importance of these regularization terms.
g · denotes the feature groups for the input layer.
(cid:107) · (cid:107) 1 indicates the
L 1 norm.
The optimization problem (1) restricts the group-wise sparsity for the input layer and the neuron-wise sparsity (or group size = 1) for the rest model layers while training with the original objective function (cid:96) ( W ) .
The optimization problem can be solved with the projected gradient descent method.
However, the optimization problem (1) has its own limitation.
The hyper-parameters λ s only have the latent impact on the model size and we could not directly affect the model size.
It is not friendly where the users would like to restrict the model size to a speciﬁc number.
On the other hand, the hyper-parameters also make the optimization problem hard to tune, requiring multiple arguments searching rounds.
To avoid these shortages, we are supposed to have new approaches to handle the compression of RS.
3. R ESOURCE C ONSTRAINED M ODEL C OMPRESSION
In this section, we propose a novel uniﬁed model and embedding compression method on RS and treat it as a constrained optimization problem.
We discuss how to solve the problem using a gradient-based optimization algorithm.
Computation Resource Function for RS Model To avoid the hyper-parameter tuning in the optimization of problem (1)
, we deﬁne the resource function to measure the resource consumption for a speciﬁc RS model structure, which can be used to guide the compression of structure.
For example, if considering the computation Flops, the resource of an MLP can be deﬁned as
[Equation]
where d ( l ) in and d ( l ) out indicate the number of input and output neurons.
For a pruned layer l , the input and output dimension are restricted by the number of pruned neurons, annotated as s ( l ) and s ( l +1) .
Therefore, we can deﬁne a resource consumption function in terms of s from each layer as
[Equation]
For simplicity, the resource function R Flops only takes the s as the number of neurons.
In a more general case, s can be substituted with the product between the number of pruned groups and the size of each pruned group.
Resource Constrained RS Model Compression With the deﬁnition of the resource function, we can formulate the following optimization problem,
[Equation]
where R budget denotes the upper bound of the overall model computation Flops setted by users.
Since we consider the group-wise sparsity for each layer, we utilize the set of variables s := { s (1) , s (2) , . . .
, s ( L ) } to control the model’s Flops.
The s ( l ) deﬁnes the lower bound of the l -th layer’s zero group number for the pruned model.
Therefore, we reformulate the optimization problem (2) to be
[Equation]
where I ( · ) is an indicator function with the output { 0 , 1 } .
When the condition is satisﬁed, the indicator function results in 1, otherwise, it would be 0.
(cid:80)
i
I (cid:18)(cid:13)(cid:13)(cid:13) W ( l ) .,i
(cid:13)(cid:13)(cid:13) 2 2 =
(cid:19) denote the zero group number for the l -th layer.
The resource constraint (3b) bounds the resource consumption of the recommendation model.
The inequalities in both (3c) and (3d) regularize the Flops consumption for each layer.
3. O PTIMIZATION
To address the optimization problem (3), we adopts the alternating direction method of multipliers (ADMM) for the reformulation.
In details, the optimization problem contains non-continuous indicator function in constraint (3c, 3d), and non-convex constraint (3b), which make the problem difﬁcult to solve.
Therefore, we ﬁrst reformulate the inequality constraints as soft regularizations and introduce Minimax optimization with dual variables.
Then we tackle the non-differentiable objective function using self-deﬁned numerical differentiation.
At last, we summarize all the optimization details into a gradient-based optimization formulation.
Minimax Optimization Reformulation Let (cid:107) W · ,g i (cid:107) s, 2 be the bottom- ( s, 2) “norm”, which denotes the Frobenius-norm of the sub-matrix (group) composed of bottom- s groups, by sorting from top to bottom in terms of norm of each group.
The equivalent formulation of the sparsity constraint (3c, 3d) can be
[Equation]
The above equivalence (4) is proved by Tono et al.
(2017), where the LHS is called DC (difference of convex functions) representation of the L 0 -constraint.
With the transformation, we can reformulate the problem in Eq.
(3), into this minimax optimization problem:
[Equation]
where y and z are dual variables.
The operator (cid:100)·(cid:101) indicates the ceiling function.
In the problem (5), the importance of regularization terms are tuned automatically based on the minimax optimization.
The sparsity loss S ( y , s , W ) :=
y
[Equation] [Equation]
W (1) .,g
(cid:13)(cid:13)(cid:13) 2 (cid:100) s (1) (cid:101) , 2 + (cid:80)
Ll [-=
-]=
2 y ( l ) (cid:13)(cid:13) W ( l ) .,.
(cid:13)(cid:13) 2 (cid:100)
s
( l ) (cid:101) , 2 and the resource loss z
( R Flops ( s ) − R budget ) are introduced to substitute the original constraints (3c, 3d).
Therefore, the optimal solution to the new problem (5) can serve as the optimal for the original problem (3).
To solve it, we present the details of updating rules for all learnable variables in the appendix.
We wrap up all the updating rules together to be a uniﬁed optimization algorithm, as described in
Algorithm 1.
In this algorithm, we only show an example that the input layer is column-wise grouped.
For a more general case, the column-wise group sparsity can be applied to an arbitrary layer in the
RS model.
Prune and Finetune Solving the minimax optimization problem (5) will make some channel weights extremely close but not exactly equal to zeros.
In order to extract the slimmed network, for each W ( l ) , we discard the s ( l ) groups with smallest L 2 norms (cid:107) W ( l ) · ,g i (cid:107) 2 (extremely close to 0) after we have solved problem (5).
Then we ﬁnetune the explicitly pruned network to further improve the performance.
E XPERIMENTS
To demonstrate the effectiveness of UMEC , we run experiments on a standard public RM over the real-world recommendation dataset.
And we’d like to answer the following questions: Whether our uniﬁed optimization framework that jointly considers input feature selection and prediction model compression has superiority over previous state-of-the-art RS compression methods which solve the problem from only one aspect?
If yes, why?
4. E XPERIMENTAL S ETUP
Dataset and Model We conduct all our experiments on the state-of-the-art ads CTR prediction model named DLRM (Naumov et al., 2019), as illustrated in Figure 1.
Following Naumov et al.
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Figure]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Figure] [Table]
[Equation]
[Table]
use the Criteo AI Labs Ad Kaggle dataset 1 for all experiments.
The Criteo AI Labs Ad
Kaggle dataset contains approximately 45 million click log samples collected over seven days.
Each sample contains 13 continuous and 26 categorical input features.
Following the ofﬁcial setting, we split the data of the last day into validation and testing sets, and use data from the rest six days as training set.
Baseline methods Latency of an ads CTR prediction model mainly comes from two sources: feature embedding and prediction model inference.
Feature embedding layers map input categorical values into a vector space.
Both the number of input categorical values and the output feature space dimension can affect the latency of feature embedding stage.
The inference latency of predication model mainly depends on the prediction model size.
Following the above analyses, methods to accelerate ads CTR prediction models mainly fall into three categories:
• Prediction model compression: compressing the prediction model with traditional model compression methods, such as Li et al.
; Molchanov et al.
; Yang et al.
• Input feature selection: reducing the number of input categorical features by ﬁltering out non-essential ones.
Embedding dimension reduction: shrinking the dimension of embedding layers’ output space.
Two recent works (Ginart et al., 2019; Joglekar et al., 2020) fall into this category.
Our uniﬁed optimization framework jointly considers input feature selection and model compression.
We will further show that our method can be generalized to also consider input feature dimensionreduction.
So we compare our method with baselines from all three categories to show the superiority of our method over those baselines tackling RS compression problem from only one aspect.
General Training and Evaluation Details For the general training and evaluation settings in our experiments, please see the appendix A for more details.
Evaluation Metrics We evaluate the performance with the following two terminologies: Accuracy and Compression Ratio (CR) .
Accuracy: CTR prediction accuracy on the validation set.
Compression ratio (CR): the ratio between the pruned Flops and original model Flops.
More formally,
CR = 1 − p f ( W ) /p
f ( W 0 ) , where p f ( · ) calculates the Flops of a (
4. P REDICTION MODEL COMPRESSION FOR RS M ODEL
We design and conduct extensive experiments to examine the compression performance among our method and several state-of-the-arts model compression methods.
The baseline methods include handcrafted structures, one-shot magnitude pruning (MP) (Li et al., 2016), and two recent state-ofthe-art methods, namely Taylor pruning (TP) (Molchanov et al., 2019) and energy-constrained model compression (ECC) (Yang et al., 2019).
In Figure 2, we demonstrate the best accuracy on validation set under different compression ratios (CRs) on the Criteo AI Labs Ad Kaggle dataset.
For more results on the the Criteo Ad Terabyte dataset, please see the appendix A.
Several observations can be drawn from this comparison ﬁgure.
First, our method continuously outperforms all other methods with a margin of at least 0.01%.
Second, only our method can successfully compress the original dense model to the ideal budget while still manages to maintain the performance without any degradation, within a certain CR scope.
These results show that our joint model compression and input selection method can achieve much better balance of efﬁciency vs accuracy than traditional model compression methods.
4. I NPUT F EATURE S ELECTION FOR RS M ODEL
We evaluate the effectiveness of our proposed method on input feature selection task.
The input feature selection task can be easily realized using our proposed framework, by keeping the resource constraint for the features groups in input layer ( i.e. , Eq.
(3c))
and ignoring constraint for weights of hidden layers( i.e. , Eq.
(3d)).
We use group Lasso as self-designed baseline.
The loss function of group
Lasso is shown in Eq.
Since we are only considering input feature selection in this sub-section, the sparsity regularization on hidden layers (e.g. , the second term in S ( y , s , W )) is ignored.
We tune hyper-parameter λ 1 among 0.0005, 0.001, 0.005, 0.01 in order to get different compression ratios.
After group Lasso training, all input features whose L 2 norms are below a threshold th will be removed, and the Flops for the new model are calculated accordingly.
We empirically ﬁnd that setting th to 0 .
01 can achieve good performance for group Lasso and thus using it as the default value.
For both our method and group Lasso, we train and ﬁne-tune for 1 epoch and 2 epochs, respectively.
In Figure 3, the performance on the holdout validation set at the end of the compressing process is reported.
We demonstrate the effectiveness of our method in input feature selection task, by showing its superior performance under all the compress ratios.
Another observation is that the pruned model of both methods will fail to recover the performance of the original dense model if too many input features are removed.
For example, at 0.21 compression ratio, which corresponds to removing 9 out of the total 27 input features, both methods have considerable drop of accuracy.
This indicates that reducing number of input features will cause considerable harm to the model’s performance.
For the input layer, the mentioned dense model has 27 features in total.
For the two hidden layers, it has 512 and 256 neurons respectively.
The convergency of the binary cross entropy (BCE) loss during training is shown in Figure 5 in the appendix.
4. E MBEDDING D IMENSION R EDUCTION FOR RS M ODEL
We show that our method can also be applied on embedding dimension reduction.
In this new scenario, W in Eq.
(3) is embedding layer weights, instead of previous prediction model weights.
Eq.
(3d) is removed ( i.e. , no longer to prune the input dimension).
The rest parts of the objective function remain the same.
We compare our method with the state-of-the-art embedding dimension reduction method named mixed dimension embedding layers (MDEL) (Ginart et al., 2019).
Following Ginart et al.
(2019), we use size of the embedding layers to help deﬁne compression ratio.
More speciﬁcally, in this setting, CR =
− p s ( W ) /p
s ( W 0 ) , where p s ( · ) calculates the number of parameters in a (sub-)model.
W and W 0 are the pruned and original dense embedding sub-models, respectively.
To evaluate performance change after pruning, we deﬁne ∆ acc =
Acc W − Acc W 0 , where Acc W and Acc W 0 are
CTR prediction accuracy of a pruned and original dense model, respectively.
[Table]
Experiment results are shown in Table 1.
As we can see, both methods have slightly improvement on accuracy at around 50%
CR, while the accuracy improvement of our method is larger than that of MDEL.
Also, our method can still achieve better accuracy improvements than MDEL with even 10% larger CR.
These results show that our ADMM based compression algorithm is better than
MDEL.
4. S PARSITY A NALYSES OF UMEC
Below we present how the sparsity of the prediction model evolves during the pruning phase in
Section 4.2.
As a recall, the aforementioned prediction model consists of one input layer and two hidden layers, with 27 input features, 512 and 256 neurons respectively.
As shown in Figure 4, when
R budget = 0 .
5 × p f
( W 0 ) , the pruned input features/neurons converges to 0, 102 and 250 respectively.
During training, last hidden layer getting the most percentage of neurons removed and the input layer getting the least percentage.
We can come to several conclusions from this result.
First, such phenomenon aligns with the observation in Section 4.3 and serves as an evidence that we need enough number of input features to adequately represent the raw input information and contribute to the downstream classiﬁcation task.
Second, to perform the relatively easier task, i.e., binary classiﬁcation, the requirement of neurons from last hidden layer falls far below the original design, thus considerable resource consumption can be saved here.
Third, it demonstrates that our resource-constrained uniﬁed optimization plays a smart role in globally ﬁnding the optimal resource allocations across all layers.
C ONCLUSION
The goal of this paper is to tackle with the new problem that jointly compressing input features and neural network structures in the recommendation models.
We propose UMEC framework by integrating these two objectives into one uniﬁed constrained optimization problem, solved by ADMM method.
We conduct extensive experiments and demonstrate the effectiveness of our proposed UMEC method by observing its superior performance than other state-of-the-art baseline methods
A A PPENDIX A. U PDATE R ULE FOR ALL LEARNABLE VARIABLES
To solve the optimization problem (5) we utilize the ADMM strategy to solve a series of sub-problems as follows.
Update W
The optimization problem on W can be solved with proximal-SGD (Nitanda, 2014).
The deﬁnition of proximal operator is as follows:
[Equation]
where ¯ W is the direct stochastic gradient descent update of W .
The sub-problem above has the closed form solution for W ∗ :
= (cid:8) W (1) ∗ , W (2) ∗ , . .
W
[Equation]
L [Equation] [Equation] where W i denotes the i -th sub-matrix, and W is the weight for an arbitrary layer.
Update dual variables z and y
For the sub-optimization problem, we have
[Equation]
and we can adopt the gradient ascent method as the update rule.
Update s The optimization on s relies on both sparsity and resource loss.
However, both of them are non-differentiable due to the ceiling function (cid:100) s (cid:101) .
Straight-through estimator (STE) (Bengio et al., 2013) is an effective approach for optimization through the non-differentiable functions.
The main idea is to adopt some simple proxy to be the derivative of the non-differentiable formulation.
Thus, the back-propagation can be applied as what happened in the differentiable case.
For (cid:107) W (cid:107) 2 s, 2 , we can apply the numerical differentiation (cid:107) W (cid:107) 2 s +1 , 2 − (cid:107) W (cid:107) 2 s, 2 as the proxy derivative of (cid:107) W (cid:107) 2 s, 2 with respect to s :
[Equation]
where W 2 is the column-group wise L 2 format of W , Dim( W ) is the total number of column groups of W , and W 2 least − j is the j -th least column group in W 2 .
For a general resource consumption function (e.g., Flops), the non-differentiable part of R is the ceiling function (cid:100) s (cid:101) , for which we can use a common STE (Bengio et al., 2013): ˜ ∂ (cid:100) s
[Equation]
= 1 .
[Equation]
Now we can wrap up all the update rules together to be a uniﬁed optimization algorithm 1.
In this algorithm, we only show an example for the input layer to be column grouped.
For a more general case, the column-group wise sparsity can be applied to an arbitrary layer in the RS model.
A. G ENERAL T RAINING AND E VALUATION D ETAILS
[Figure]
Training Loss
We follow the general settings in Naumov et al.
(2019) in all our experiments: we use SGD optimizer with learning rate 0.1 to optimize the BCE loss ( (cid:96) ( W ) in our method); we set training batch size to 1024, initial feature embedding dimension to 16, and use the default three-layer MLP prediction model with hidden dimensions 512 and 256.
For our method, we use Adam optimizer (Kingma & Ba, 2014) for s and SGD optimizer for W , y , and z .
We set learning rates η 1 , η 2 , η 3 , η 4 in Algorithm 1 to be [0 . 1 , 0 . 05 , 0 . 1 , 2 . 0] respectively, and set τ to be 306, which equals to 1 epoch.
After the training process in Algorithm 1, we zero out the redundant neurons (as described in the last paragraph in Section 3) and keep the model weights of the exacted sub-network as initialization values for the following ﬁne-tune process.
We then perform ﬁne-tuning for 2 epochs.
For fair comparison, we also ﬁne-tune 2 epochs for all baseline methods.
Following Naumov et al.
(2019), the best accuracy on validation set is reported. [Table]
A. E NERGY C ONSUMPTION AND I NFERENCE L ATENCY OF N ETWORK
We evaluate the energy cost and latency of all models on a GTX 2080 Ti GPU.
Following Yang et al.
(2019), we use the nvidia-smi utility 2 to monitor the energy consumption.
We follow the settings in Fu et al.
(2020) to measure inference latency on the aforementioned real-device.
All experiments are implemented with PyTorch.
As shown in Table 2, our method achieves best accuracy with least energy consumption and latency among all compared compression methods.
A. E XPERIMENTAL R ESULTS ON T ERABYTE D ATASET
We also provide the experimental results for the Criteo Ad Terabyte dataset 3 .
The full dataset consists of approximately 4.4 billion click log samples collected over 24 days.
For quick demonstration purpose, we perform uniform sampling with 12.5% sampling rate from the data among ﬁrst 6 days, where the ﬁrst 5 days are used as the training data and the 6th day is split into validating and testing data.
The experimental results over the full dataset following the same sampling scheme in Naumov et al.
(2019) will be included in the camera ready version once the paper gets accepted.
As shown in
Figure 6 and Figure 7, our proposed method consistently outperforms all other baseline methods.
magenta line represents the accuracy achieved by the original dense DLRM model
A. E XPERIMENTS USING CASCADED PIPELINE
We provide experimental results under cascaded pipeline using UMEC , where the compression processes for the input feature and the prediction model are carried out sequentially.
We choose to evaluate the results on the Criteo AI Labs Ad Kaggle dataset.
Full results will come out soon.