Adaptive Mixture of Low-Rank Factorizations for Compact Neural Modeling
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Modern deep neural networks have a large amount of weights, which make themdifﬁcult to deploy on computation constrained devices such as mobile phones.
Onecommon approach to reduce the model size and computational cost is to use lowrank factorization to approximate a weight matrix.
However, performing standardlow-rank factorization with a small rank can hurt the model expressiveness andsigniﬁcantly decrease the performance.
In this work, we propose to use a mixtureof multiple low-rank factorizations to model a large weight matrix, and the mixturecoefﬁcients are computed dynamically depending on its input.
We demonstratethe effectiveness of the proposed approach on both language modeling and imageclassiﬁcation tasks.
Experiments show that our method not only improves thecomputation efﬁciency but also maintains (sometimes outperforms) its accuracycompared with the full-rank counterparts.
Introduction
Modern neural networks usually contain millions of parameters [5, 9], and they are difﬁcult to bedeployed on mobile devices with limited computation resources.
To solve this problem, modelcompression techniques are proposed in recent years.
Low-rank factorization is a popular way ofreducing the matrix size.
It has been extensively explored in the literature [6, 7, 4, 11].
Mathematically,a large weight matrix W ∈ R m × n is factorized to two small rank- d matrices U ∈ R m × d , V ∈ R n × dwith W = UV T .
Since both U and V are dense, no sparsity support is required from specializedhardware.
It naturally ﬁts the general-purpose, off-the-shelf CPUs and GPUs.
To signiﬁcantly reduce the model size and computation, the rank d in the low-rank factorization needsto be small.
However, a small rank can limit the expressiveness of the model [10] and lead to worseperformance.
To understand the limitations, given a n -dim feature vector h , we observe that V T h ,as in U ( V T h ) , is a linear projection from a high-dimensional space ( n dims) to a low-dimensionalspace ( d dims).
This can lead to a signiﬁcant loss of information.
The conﬂict between the rank dand the model expressiveness prevents us from obtaining a both compact and accurate model.
To address the dilemma, we propose to increase the expressiveness by learning an adaptive, inputdependent factorization, rather than performing a ﬁxed factorization of a weight matrix.
To do so, weuse a mixture of multiple low-rank factorizations.
The mixing weights are computed based on theinput.
This creates an adaptive linear projection from a high-dimensional space to a low-dimensionalspace.
Compared to the conventional low-rank factorization, the proposed approach can signiﬁcantlyimprove its performance while only introducing a small additional cost.
[Figure]
pooling can be to compute π make it efﬁcient
Adaptive mixture of low-rank factorizations
We propose to use an unnormalized learned mixture of low-rank factorizations whose mixing weightsare computed adaptively based on the input.
More speciﬁcally, denoting the input by h and thenumber of mixture components by K , we decompose a large weight matrix by
[Equation]
where π ( · ) : R n → R K is the function which maps each input to its mixture coefﬁcients.
Forexample, π can be a small neural network.
This introduces a small amount of extra parameters andcomputation.
We will later discuss the details of efﬁcient ways to implement the mixture function π .
If π k , k = 1 , ..., K , is chosen to be constant (input independent), it can be absorbed into either
U ( k ) or V ( k ) .
Thus, the proposed method reduces to the low-rank factorization.
This is evidencedby rewriting W as W =
[ π 1 U (1) , ..., π K U ( K ) ]
[ V (1) , ..., V ( K ) ] (cid:62) .
In other words, the conventionallow-rank factorization can be considered as a special case of our method.
Figure 1 depicts theproposed framework.
Adaptive mixing weights π ( h ) .
The mixing weights can encode important information that we canuse to increase the expressiveness of the projected low-dimensional space.
Under our framework, thegeneration of the mixing weights π ( h ) is ﬂexible.
A straight-forward approach is to use a non-lineartransformation of the input to the weight matrix.
For example, π ( h ) =
σ ( Ph ) , where σ is a non-lineartransformation, such as sigmoid or hyperbolic tangent function, and P ∈ R K × n is an extra weightmatrix.
This adds some extra parameters and computation to the model since the linear projectionthat we construct is R n → R K .
To further reduce the parameter and computation in the mixingweights π , we propose the following strategies.
Pooling before projection.
We do not require the whole input to compute the mixture function π .
Instead, we can apply pooling to the input h before projection.
For example, a global average poolingcan be applied if the input is a 3D tensor (for images); for a 1D vector, we can segment the vectorand average each segmentations.
By applying pooling, we can both save the computation and bettercapture the global information.
Random projection.
To reduce the number of parameters in the linear projection of h , we can use arandom matrix P random in place of a fully adjustable P , i.e. π ( h )
=
σ ( P random h ) .
Note that we cansimply save a seed to recover the random matrix, but it still requires the same amount of memory andcomputation as the fully adjustable linear projection of h .
Increased expressiveness.
The adaptive mixing weights introduce a non-linear transformationinto the high-to-low-dimensional projection that can be more expressive.
Since each W ( h ) is adata-dependent low-rank matrix, there is no constant linear weight independent to the input (even afull-rank matrix) that can mimic the transformation W ( h ) .
Generating the whole weight matricescan be very expensive.
Our method can be seen as a swift approach to generate the weights byadaptively adjusting mixing weights for the linear bottleneck.
It assigns weights into groups anddynamically controls them at the group level.
[Figure]
[Table]
Experiments
Recurrent neural networks for language modeling.
Recurrent neural networks (RNNs) are widelyused in language modeling, machine translation and sequence modeling in general.
We adopt thesame Long Short Term Memory (LSTM) models and follow the settings from a previous state-ofthe-art model [12] for language modeling and use Penn Tree Bank (PTB) as well as Text8 datasets.
More speciﬁcally, we use the medium sized model introduced in [12].
We test three variants of the proposed model against regular low-rank factorization, each with differentways of computing mixing weights, namely (1) MIX-ALL-PJ: direct linear projection of the inputvector h , (2) MIX-POOL-PJ: linear projection after segment-based mean pooling of the input vectorh , and (3) MIX-RND-PJ: use a random projection for the input vector h .
Among these adaptiveprojection methods, MIX-ALL-PJ has a large amount of extra parameters, MIX-POOL-PJ has a smallamount of extra parameters, and MIX-RND-PJ has no extra parameters.
We compute the FLOPs of asingle time-step of applying LSTM, and the perplexity associated to different settings.
The results are shown in Figure 2.
Firstly, with adaptive mixtures, the low-rank factorization modelachieved 40% reduction in FLOPs, and even surpassed the performance of the full matrix baselineby decreasing the perplexity by 1.7 points.
Secondly, the use of adaptive mixtures can signiﬁcantlyimprove the performance compared with regular, non-adaptive low-rank factorization.
Thirdly, usingpooling before projection can be a good choice for computing the mixing weights π .
It not onlyreduces the computation and parameter size, but can better capture the global information and achievebetter accuracy.
CNN for image recognition.
We further demonstrate the effectiveness of the proposed approach oncompressing CNN models on ImageNet [2].
We chose to use modern compact CNN models as thebaseline (which are harder to compress), rather than using the bulky CNN models (which is easier tocompress).
Speciﬁcally, we choose to compress the point-wise convolution in depth-wise separableconvolutions [1], MobileNet [3, 8] in particular.
Table 1 shows the comparison of different state-of-art compact convolutional models.
We observedthat compared to the regular low-rank factorization of MobileNet model (a.k.a. MobileNet V2),the proposed method achieves signiﬁcantly better results (2.5% and 2% for two