WSN ET : L EARNING C OMPACT AND E FFICIENT N ETWORKS WITH W EIGHT S AMPLING
Anonymous authors
Paper under double-blind review
A BSTRACT
We present a novel network architecture for learning compact and efﬁcient deep neural networks, where the weights of convolution ﬁlters and fully connected layers, instead of learned independently, are sampled from a compact set of parameters that enforces weight sharing in the learning process.
Speciﬁcally, we consider learning compact and efﬁcient 1D convolutional neural networks for audio classiﬁcation in this work.
We show that our novel way of weight sampling allows not only weight sharing but also computation sharing, and therefore, we can learn much smaller and efﬁcient but yet competitive networks compared to baseline networks with same number of convolution ﬁlters.
Extensive experiments on multiple audio classiﬁcation datasets verify the effectiveness of our approach.
Combining weight quantization, we demonstrate that we can learn models that are up to 180 × smaller than the baselines without noticeable performance drop.
I NTRODUCTION
Despite deep neural networks (DNNs) have achieved remarkable success in various applications, e g. sound classiﬁcation, speech recognition and natural language processing, they usually suffer from following two problems caused by their inherent huge parameter space.
First, most of stateof-the-art deep architectures are prone to over-ﬁtting even trained on large datasets Simonyan &
Zisserman (2015); Szegedy et al.
Secondly, as they usually consume large storage memory and energy Han et al.
, DNNs are difﬁcult to embed into devices with limited memory and power (like portable devices or chips).
There have been a lot of research on compressing DNNs Han et al.
; Anwar et al.
; Li et al.
; Collins & Kohli (2014).
Nevertheless, they either compress the deep models with large sacriﬁce of classiﬁcation performance or only achieve relatively small compression ratio (tens of times).
To tackle the above problems, we propose a W eight S ampling deep neural network ( i.e. WSNet) to signiﬁcantly reduce the model size of deep networks, speciﬁcally 1D convolution neural networks in this paper, with more than 100 × smaller size at negligible loss of performance drop or achieve even better performance than baseline models.
In network architecture design, there are two novel components proposed in WSNet: weight sampling and channel sharing.
These two components enable the WSNet to learn a compact network with Weight Sharing Filters which are much smaller than that of independently trained ﬁlters in baseline model.
With weight sampling, all ﬁlters that are to perform convolution operations are sampled from the weight sharing ﬁlters, and similar weight sampling is also applied to fully connected layer weights.
With channel sharing, we ﬁrst learn the weight sharing ﬁlters which have less channels than needed and repeat the channels of extracted ﬁlters until reaching the number of channels in actual ﬁlters.
These two components make it possible to learn highly compact and efﬁcient WSNet.
We also propose a ﬁlter augmentation approach to make up the accuracy loss caused by compressing weights.
Combined with weight quantization method, WSNet is able to further reduce the model size during inferencing without sacriﬁcing the loss of accuracy.
For demonstrating the effectiveness of WSNet, we conduct extensive experiments on the challenging acoustic scene classiﬁcation and music detection tasks.
On all tested datasets, including MusicDet200K (a self-collected dataset, as detailed in Section 4), ESC-50 Piczak (2015a), UrbanSound8K J. et al.
and
DCASE Stowell et al.
(2015), WSNet signiﬁcantly reduces the model sizes of the baseline models by 100 × with comparable or even higher classiﬁcation accuracy.
When compressing more than 160 × , WSNet is only subject to slight accuracy drop.
Above experimental results strongly evident the capability of WSNet on learning compact and efﬁcient networks.
Although the experiments in this paper are mostly limited to one dimensional convolution neural networks, the same idea can naturally be extended to 2D CNNs.
In addition, we also propose to use integral image method Viola & Jones (2001) to accelerate WSNet in both training and inference.
With densely sampled weight sharing ﬁlters, integral image method could dramatically reduce the computation time and make WSNet much faster in computation.
The integral image method can also enable sampling ﬁlters with different size, from the weight sharing ﬁlters with little computational overhead.
This could provide much more powerful feature representation with the same number of weights and similar computation.
R ELATED W ORKS 2. A UDIO CLASSIFICATION
In this paper we considered Acoustic scene classiﬁcation tasks as well as music detection task.
Acoustic scene classiﬁcation (ASC) is a task focusing on classifying the surrounding environment where an audio stream is generated given the audio input Barchiesi et al.
It can be applied in many different applications such as audio tagging Cai et al.
, audio collections management
Landone et al.
, robotic navigation Chu et al.
(2006), intelligent wearable interfaces Xu et al.
(2008), context adaptive tasks Schilit et al.
(1994), and etc.
Music detection is a related task to determine whether a small segment of audio is music or not.
It is usually treated as a binary classiﬁcation problem given an audio segment as input, i.e. , to classify the segment into two categories, music or non-music.
Like in many other areas, deep learning based approaches has been recently proven to improve the performance over conventional baselines in audio classiﬁcation tasks.
Among the various types of deep learning approaches proposed, convolutional neural networks (CNN) have been the most popular in learning the representation of the audio signal Valenti et al.
Salamon & Bello (2017).
SoundNet Aytar et al. stands out among different CNNs applied for sound classiﬁcation.
One reason is because the SoundNet is trained from very large amounts of unlabeled sound data using visual information as a bridge, compared to many other networks that are trained with much smaller scale labeled dataset.
At the same time, the SoundNet directly takes the one dimensional raw wave signals as input so that there is no need to calculate audio speciﬁc features like MFCC Pols (1966) Davis &
Mermelstein (1980) and spectrogram Flanagan (1972).
The network contains all one dimensional convolutional layers.
The soundnet yields signiﬁcant performance improvements over the state-ofthe-art results on standard benchmarks for acoustic scene/object classiﬁcation.
2. D EEP M ODEL C OMPRESSION
Early approaches for deep model compression include optimal brain damage LeCun et al.
and optimal brain surgeon Hassibi & Stork (1993) that prune the connections in networks based on the second order information.
However, those methods are not feasible for deep networks due to high computational complexity.
Some recent works in network compression include Han et al.
Collins & Kohli (2014); Anwar et al.
; Lebedev & Lempitsky (2015);
Kim et al.
; Luo et al.
; Li et al.
(2017), in which Han et al.
; Anwar et al.
(2017) pruned connections in a progressively greedy way.
Collins & Kohli (2014); Lebedev & Lempitsky (2015);
Kim et al.
(2015) used sparsity related regularizer.
Li et al.
; Luo et al.
(2017) focused on ﬁlter level pruning by using different criterion.
Although those works can reduce model size signiﬁcantly, they also suffer from dramatic performance loss.
Sindhwani et al.
; Denton et al.
(2014) sought a low-rank approximation of the model Han et al.
combined pruning, quantization and Huffman coding techniques to provide rather high compression ratios.
However, those methods also suffer from large performance drop.
Iandola et al.
(2016) proposed to reduce parameters by re-designing the network architectures with “Fire” module.
Jin et al.
(2016) proposed an iterative hard thresholding method, but only achieve small compression ratios.
There are also works trying to compress a model by using binning methods Gong et al.
but they can only be applied over fully connected layers.
[Figure]
sampling
Channel sharing sampling
Repeat C times ） , （   m  ） , , （ n m   ） , , （ n m 
Filters
Stride: S
Repeat C times
S n )1 (      
C m m    …
[Figure]
In contrast, our method can be applied for both convolution layers and fully connected layers.
The models trained with WSNet do not only offer large compression ratios compared to the baseline models, but also achieve comparable performance.
M ETHOD
In this section, we describe the technical details of WSNet.
Firstly, we present those two most critical components in WSNet: weight sampling and channel sharing.
Second, we present a ﬁlter augmentation method to compensate the loss caused by the compression.
Third, we demonstrate that the weight quantization can be applied on WSNet to further reduce the model size.
Finally, we propose an integral image method for accelerating WSNet in both training and inference.
Before diving into the details, we ﬁrst introduce the notations used in the following text.
We use a tuple containing three elements i.e. ( (cid:96), m, n ) to represent the shape of weights W in a convolution layer of WSNet.
(cid:96), m, n denote the ﬁlter size, ﬁlter channels, and the number of ﬁlters that are actually used in convolution.
In each layer of WSNet, all ﬁlters used in actual convolutions are sampled from a learned weight sharing ﬁlter which has shape of ( (cid:96) ∗ , m ∗ ) .
The goal of training
WSNet is to learn more compact DNNs which satisfy the condition (cid:96) ∗ m ∗ < (cid:96)mn .
We deﬁne the compression ratio of a layer as (cid:96)mn(cid:96) ∗ m ∗ .
In the following, we will show that WSNet reduces the network’s size by compressing the weights in two dimensions: the spatial dimension and the channel dimension.
3. W EIGHT SAMPLING
During the training phase of traditional DNNs, the ﬁlters in a layer are learned independently.
There are two disadvantages of this strategy.
Firstly, it makes the DNNs have a large number of parameters, which impedes them to be deployed in computation resource constrained platforms.
Second, due to the over-parameterization of deep models, the network is prone to overﬁtting and get stuck in local minimum.
Therefore, we propose a novel weight sampling method to reuse the weights among ﬁlters.
Speciﬁcally, in WSNet, all ﬁlters with the shape ( (cid:96), m, n ) to be used in convolutions are sampled from a weight sharing ﬁlter with the shape ( (cid:96) ∗ , m ∗ ) , as illustrated in Figure 1.
By scanning the weight sharing ﬁlter with a window size of (cid:96) and stride of S , we could sample out n ﬁlters with ﬁlter size of (cid:96) .
Formally, the equation between the ﬁlter size of the weight sharing ﬁlter and the sampled ﬁlters is:
[Equation]
In weight sampling, the compression ratio (cid:96)m ∗ n (cid:96) ∗ m ∗ ≈ (cid:96)S .
Note that since the minimal value of S is 1, thus the minimal value of (cid:96) ∗ is (cid:96) + n − 1 and the maximal compression ratio is therefore (cid:96) .
For the brevity of description, In the following text, S will be used as the value of compression ratio where no confusion is caused.
Note we use Eq.
1 to strictly calculate the ﬁnal compression ratios of
WSNet in all tables in the experiments.
3. C HANNEL SHARING
Although it has been experimentally veriﬁed that the weight sampling strategy could compresses deep models with negligible loss of classiﬁcation accuracy, it has a disadvantage.
Just as mentioned in above section 3.1, the maximum compression ratio is the ﬁlter size (cid:96) .
What’s more, aggressively reducing the size of ﬁlters in the spatial dimension might do harm to the ﬁnal performance of WSNet due to its learning capability might be largely reduced by forcing sharing large proportions of weights among ﬁlters.
For seeking more compact networks, we propose a channel sharing strategy to compress DNNs along the channel dimension.
As illustrated in Figure 1, each ﬁlter after weight sampling is repeated C times to generate the actual ﬁlter used in convolution.
The equation between the channels of ﬁlters before and after channel sharing is:
[Equation]
In the following text, we use C to denote the compression ratio along the channel dimension.
As introduced later in Experiments, we observe that channel sharing is able to signiﬁcantly reduce the model size while achieving comparable performance to baselines.
Note one advantage of channel sharing is that the maximum compression ratio can be as large as m ∗ which make it possible to more aggressively compress deep models.
Although above analysis is conducted on convolution layers, the weight sampling and channel sharing can be conveniently applied to the fully connected layers.
For a fully connected layer, we treat its spatial dimension and channel dimension as one dimension, along which the weight sampling is performed to compress the size of weights.
3. F ILTER A UGMENTATION
The performance of WSNet might get adversely affected when performing aggressively model compression ( i.e. S and C are large).
To make up the performance loss, we propose to augment ﬁlters for layers with signiﬁcantly reduced size to enhance their learning capability.
Speciﬁcally, we sample more ﬁlters with stride ¯ S ( ¯ S < S ) from the weight sharing ﬁlter.
In order to retain the architecture of the following layer, we append a 1 × 1 convolution layer with shape of (1 , ¯ n, n ) to reduce the channels of augmented ﬁlters.
It has been experimentally veriﬁed that ﬁlter augmentation can effectively boost the performance of WSNet.
However since the ﬁlter augmentation also brings extra parameters to WSNet ( i.e. the weights in the added 1 × 1 convolution layer), it is generally only used in lower layers where the number of ﬁlters ( n ) is small (such that the size of added parameters to WSNet is small).
Besides, one can also conduct channel sharing to compress the weights of the newly introduced 1 × 1 convolution layer.
3. W EIGHT Q UANTIZATION
Weight quantization has been used in previous literatures Han et al.
(2016) to compresses DNNs by reducing the number of bits required to represent each weight.
By making multiple weights share the same weights, the storage space of DNNs can be largely saved.
More concretely, in our experiments, the weights in each layer are linearly quantized to q bins where q is a pre-deﬁned number. All the weights in the same bin share the same value, thus for each weight, we only need to store a small index into a table of shared weights.
The size of each bin is calculated as
[Figure]
(max( W ) − min( W )) /q .
The compression ratio by weight quantization is calculated as follows, given q bins, we only need log 2 ( q ) bits to encode the index.
For the weight sharing ﬁlter in WSNet which uses 4 bytes ﬂoat number (32 bits) to represent a weight, the compression ratio of weight quantization can be calculated as
[Equation]
Since the condition (cid:96) ∗ m ∗ (cid:29) q generally holds in most layers of WSNet, the compression rate by weight quantization is approximately
[Equation]
Note different from previous literatures Han et al.
(2015) which learns the weight quantization during training , we perform weight quantization after the training of WSNet is ﬁnished.
In the experiments, we ﬁnd that it is sufﬁcient to use such a simpler method to achieve signiﬁcant compression ratio without loss of accuracy.
3. I NTEGRAL I MAGE FOR E FFICIENT 1D C ONVOLUTION
The weight sharing scheme makes an accelerated algorithm using 1D integral image for the convolution operation between the 1D weight sharing ﬁlter and the input, which is illustrated in Figure 2.
Speciﬁcally, we ﬁrst compute the the outer product of the 1D weight sharing ﬁlter and the 1D input, and compute the 1D integral image along all the crossing lines of 45 degree.
In the example showed in Figure 2, the convolution result of ﬁlter f 1 and the input signal s 1 are the sum of elements in the green segment on the yellow crossing line of 45 degree, and convolution result of ﬁlter fand the input signal s 2 are the sum of elements in the red segment on the same yellow line.
With the 1D integral image computed for this yellow line, these two convolution results can be computed efﬁciently by two subtraction operations.
In this way, the 1D convolution can be carried out in a more efﬁcient way than traditional convolutions.
E XPERIMENTS
In this section, we present the details and analysis of the results in our experiments.
Extensive ablation studies are conducted to verify the effectiveness of the proposed WSNet on learning compact and efﬁcient networks.
On all tested datasets, WSNet is able to improve over baseline networks on classiﬁcation accuracy while achieves 100x compression ratio.
When compressing up to ( e.g. 180x ), there is only slight performance drop observed compared to baseline networks.
4. E XPERIMENTAL S ETTINGS
Datasets We collect a large-scale music detection dataset (MusicDet200K) from public available social media platforms ( e.g Facebook, Twitter, etc. ) to conduct experiments with.
For fair comparison with previous literatures, we also test WSNet on three standard, publicly available datasets, i.e
ESC-50, UrbanSound8K and DCASE, the details of which are as follows.
MusicDet200K aims to assign a binary label to each sample to indicate whether it is music or not.
MusicDet200K has overall 220,000 annotated sound clips.
Each has a time duration of 4 seconds and is resampled to 16000 Hz and normalized.
Among all samples, we use 200,000/20,000/18,
[Table]
as train/val/test set.
The samples belonging to “non-music” count for around 70% of all samples, which means if we trivially assign all samples to be ”non-music”, the classiﬁcation accuracy is 70%.
ESC-50 is a collection of 2000 short (5 seconds) environmental recordings comprising 50 equally balanced classes of sound events in 5 major groups (animals, natural soundscapes and water sounds, human non-speech sounds, interior/domestic sounds and exterior/urban noises) divided into 5 folds for comparable cross-validation.
Following Aytar et al.
[Table]
we extract 10 sound clips from each recording with length of 1 second and time step of 0.5s ( i.e. two neighboring clips have 0.5 seconds of themselves overlapped).
Therefore, in each cross-validation, the number of training samples is 16000.
In testing, we average over ten clips of each recording for the ﬁnal classiﬁcation results.
UrbanSound8K is a collection of 8732 short (around 4 seconds) recordings of various urban sound sources ( air conditioner , car horn , playing children , dog bark , drilling , engine idling , gun shot , jackhammer , siren , street music ).
Like in ESC-50, we extract 8 clips with time length of 1 second from each recording.
For those that are less than 1 seconds, we pad them with zeros and repeat for 8 times.
DCASE is used in the Detection and Classiﬁcation of Acoustic Scenes and Events Challenge (DCASE).
It contains 10 acoustic scene categories, 10 training examples per category and 100 testing examples.
Each sample is a 30 seconds audio recording.
During training, we evenly extractsound clips from each recording.
Evaluation criteria To demonstrate that WSNet is capable of effectively compressing CNN at small cost of accuracy losses, two evaluation criteria are used in our experiments: compression ratio and sound classiﬁcation accuracy.
Baseline networks The network we used on MusicDet200K consists of 7 convolutional layers and 2 fully connected layers, using which we verify the capability of WSNet on effectively learning compact representations for both convolutional layers and fully connected layers.
For fair comparison with others literatures, we ﬁrstly modify the state-of-the-art SoundNet Aytar et al.
by applying pooling layers to all
but the last convolutional layer.
As can be seen in Table 5, this modiﬁcation largely boost the performance of original SoundNet.
We apply this modiﬁed SoundNet on all three public datasets as the baseline network.
The architectures of those two baseline networks are shown in Table 1 and Table 2, respectively.
Implementation details WS-Net is implemented in Tensorﬂow Abadi et al.
Following Aytar et al.
(2016), the Adam Kingma & Ba (2014) optimizer and a ﬁxed learning rate of 0.
and momentum term of 0.9 and batch size of 64 are used throughout experiments.
We initialized all the weights to zero mean Gaussian noise with a standard deviation of 0.01.
In WSNet 1 , the dropout ratio for the dropout layers Srivastava et al.
(2014) after each fully connected layer is set to be 0.8.
The overall training is carried on 100,000 iterations.
[Table]
4. R ESULTS AND ANALYSIS 4.2. M USIC D ET 200K
Ablation analysis Through extensive control experiments, we investigate the effects of each component in WSNet on the networks’ compression ratios and classiﬁcation accuracies.
The comparative study results of different settings of WSNet are listed in Table
[Table]
(1) Weight sampling We test the performance of WSNet when using compression ratios of 2,and 8 respectively.
As can be seen from Table 3, S 2 and S 4 slightly outperforms the classiﬁcation accuracy of the baseline model, possibly by reducing the overﬁtting.
When compressing 8 times, the classiﬁcation accuracy of S 8 only drops by 0.6%.
Note that the maximum compression ratio along the spatial dimension is equal to the ﬁlter size, thus for the layer “conv7” which has ﬁlter size of 4, its compression ratio is still 4 in S 8 .
Above results demonstrate that beneﬁting from the weight-sharing strategy used by WSNet can achieve comparative performance by compressing the network in the spatial dimension.
(2) Channel sharing Three compression ratios: 2, 4 and 8 are used to compress the network along the channel dimension.
It can be observed from Table 3 that C 2 and C 4 and C 8 achieves linear compression ratios while incurring negligible loss of accuracy.
Actually, C 2 can even boost the accuracy upon baseline models, demonstrating the high efﬁciency of sharing channels in WSNet.
In the next, we compress in both the spatial dimension and channel dimension ( i.e. S 4 C 4 SC † 4 and
S 8 C 8 SC † 8 )
, it reveals that WSNet is capable of obtaining much higher compression ratios while leading to small performance decrease (less than 1%).
(3) Filter augmentation Filter augmentation is used to enhance the learning capability of WSNet in the case of aggressive compression ( i.e. S and C are large) and make up the performance loss caused by compression.
As shown in Table 3, after extracting 2 times more ﬁlters in conv1, conv2 and conv3, S 8 C 8 SC † 8 A 2 signiﬁcantly outperforms the S 8 C 8 SC † 8 .
Above results demonstrate that effectiveness of ﬁlter augmentation to boost the performance of WSNet.
(4) Weight quantization Weight quantization is used to further compress WSNet.
It can be observed from Table 3 that after using q = 256 quantization bins to represent each weight by one byte ( i.e. 8bits), S 8 C 8 SC † 15 A 2 Q 4 ) is able to compress the baseline network at 168 × while incurring only 0.1% accuracy loss.
Above result demonstrates that the weight quantization method is complementary with WSNet on jointly compressing deep networks.
Visualization To show that WSNets learn more compact parameters compared to baselines by using weight sampling and channel sharing, we visualize the learned ﬁlters in conv1 (has shape of (32 , 1 , 32) ) from the baseline model and WSNet ( i.e. S 4 C 4 SC † 4 in Table 3) in Figure 3.
For easy comparison, we ﬁrstly reshape the ﬁlters in the baseline model to have the same length of the learned weight sharing ﬁlters in WSNet.
More concretely: the ﬁlters in the baseline model are assigned to different segments of equal length by a stride of 4, and the overlapping weights are averaged.
As evidenced in Figure 3, the learned weight sharing ﬁlters in WSNet demonstrate wider spectrum between yellow and dark.
This could be explained as followed: since the ﬁlters in WSNet has to be learned in a much smaller parameter space, each weight is encouraged to capture different patterns
[Figure] [Figure]
[Table]
3M (1 × )
S                 89.
2 × S                 89.
4 × S                 88.
5.7 ×
C                 88.
2 × C                 88.
4 × C                 88.
8 ×
S 4 C 4 SC †                 88.
11.1 × S 8 C 8 SC †                 88.
23 ×
S 8 C 8 SC † 8 A                 89.
20 × S 8 C 8 SC † 15 A                 88.
42 × S 8 C 8 SC † 15 A 2 Q
[Table] [Table]
[Table] [Table]
in the input to be compact, therefore having the larger variations in ﬁlters from WSNet than that from the baseline network.
4.2. ESC-
The comparison of WSNet with other state-of-the-arts on ESC-50 is listed in Table 5.
The settings of WSNet used on ESC-50, UrbanSound8K and DCASE are listed in Table 4.
Compared with the baseline model, WSNet is able to signiﬁcantly reduce the model size by 25 times and 45 times, while at the same time improve the accuracy of baseline model by 0.5% and 0.1% respectively.
Such promising results again demonstrates the effectiveness of WSNet on compressing networks.
After applying weight quantization to WSNet, the compression ratio reaches 180 while the accuracy only slightly drops by 0.2%.
Beneﬁted from a transfer learning approach, the SoundNet Aytar et al.
(2016) that is trained using a large number of unlabeled videos achieves better accuracy than our network.
However, since the training method is orthogonal to WSNet, it is expectable that WSNet can also achieve higher accuracies by training similarly as SoundNet Aytar et al.
(2016) on unlabeled videos.
4.2. U RBAN S OUND 8K
We report the comparsion results of WSNet with state-of-the-arts in Table 6.
It can be again observed that WSNet signiﬁcantly reduces the model size of baseline while obtaining comparative results.
Both Piczak (2015b) and Salamon & Bello (2015) uses log-scaled mel-spectrograms extracted from recordings during training.
In comparison, the proposed WSNet simply takes the raw wave of recordings as input, saving the time of extracting complex features as done in Piczak (2015b); Salamon & Bello (2015).
[Table]
4.2. DCASE
As one can observe in Table 7, WSNet outperforms the baseline model by 1% with only 1% size of the baseline model.
When compressing at a more aggressive ratio, i.e. 180 × , the classiﬁcation accuracy of WSNet is only one percentage lower than the baseline model, demonstrating the effectiveness of the proposed weight sampling strategy in WSNet.
C ONCLUSION
In this paper, we present a W eight S ampling networks (WSNet) to learn efﬁcient and compact networks.
There are two novel components in WSNet: weight sampling and channel sharing, through which the weights in each layer is compressed in the spatial dimension and channel dimension respectively.
In weight sampling, all ﬁlters taking part in convolution are sampled from the learned weight sharing ﬁlters.
In channel sharing, channels in a ﬁlter are shared through repeating.
Extensive experiments on four audio classiﬁcation datasets including MusicDet200K, ESC-50, UrbanSound8K and DCASE demonstrates the effectiveness of WSNet.
It is demonstrated that WSNet is able to achieve a compression ratio as high as 180 without signiﬁcant performance drop.