WSN ET : C OMPACT AND E FFICIENT N ETWORKS WITH W EIGHT S AMPLING
Anonymous authors
Paper under double-blind review
A BSTRACT
We present a new approach and a novel architecture, termed WSNet, for learning compact and efﬁcient deep neural networks.
Existing approaches conventionally learn full model parameters independently at ﬁrst and then compress them via ad hoc processing like model pruning or ﬁlter factorization.
Different from them,
WSNet proposes learning model parameters by sampling from a compact set of learnable parameters, which naturally enforces parameter sharing throughout the learning process.
We show that such novel weight sampling approach (and induced WSNet) promotes both weights and computation sharing favorably.
It can more efﬁciently learn much smaller networks with competitive performance, compared to baseline networks with equal number of convolution ﬁlters.
Speciﬁcally, we consider learning compact and efﬁcient 1D convolutional neural networks for audio classiﬁcation.
Extensive experiments on multiple audio classiﬁcation datasets verify the effectiveness of WSNet.
Combined with weight quantization, the resulted models are up to 180 × smaller and theoretically up to 16 × faster than the well-established baselines, without noticeable performance drop
I NTRODUCTION
Despite deep neural networks (DNNs) have achieved remarkable success in various applications, e.g. audio classiﬁcation, speech recognition and natural language processing, they usually suffer following two problems from their inherent huge parameter space.
First, most of state-of-the-art deep architectures are prone to over-ﬁtting even trained on large datasets (Simonyan & Zisserman, 2015;
Szegedy et al., 2015).
Secondly, as they usually consume large storage memory and energy (Han et al., 2016), DNNs are difﬁcult to embed into devices with limited memory and power (like portable devices or chips).
Most existing networks explore to reduce the computational budget through network pruning (Han et al., 2015; Anwar et al., 2017; Li et al., 2017; Collins & Kohli, 2014), ﬁlter factorization (Jaderberg et al., 2014; Lebedev et al., 2014), low bit representation (Rastegari et al., 2016) for weights and knowledge transfering (Hinton et al., 2015).
Different from all the above works that ignore the strong dependencies among weights and learn ﬁlters independently based on existing network architectures, this paper proposes to explicitly enforce the parameter sharing among ﬁlters to more effectively learn compact and efﬁcient deep networks.
In this paper, we propose a W eight S ampling deep neural network ( i.e. WSNet) to signiﬁcantly reduce both the model size and computation cost of deep networks, achieving more than 100 × smaller size and up to 16 × speedup at negligible performance drop or even achieving better performance than the baselines ( i.e. conventional networks that learn ﬁlters independently).
Speciﬁcally, WSNet is parameterized by layer-wise condensed ﬁlters from which each ﬁlter participating in actual convolutions can be directly sampled, in both spatial and channel dimensions.
Since condensed ﬁlters have much fewer parameters than independently trained ﬁlters as in conventional CNNs, learning by sampling from them makes WSNet a more compact model compared with conventional CNNs.
In addition, to reduce the ubiquitous computational redundancy in convolving the overlapped ﬁlters and input patches, we propose an integral image based method to dramatically reduce computation cost of WSNet in both training and inference.
The integral image method is also advantageous by enabling weight sampling with different ﬁlter size with little computational overhead to enhance the learning capability of WSNet.
For demonstrating efﬁcacy of WSNet, we conduct extensive experiments on the challenging acoustic scene classiﬁcation and music detection tasks.
On all the test datasets, including MusicDet200K (a self-collected dataset, as detailed in Section 4), ESC-50 (Piczak, 2015a), UrbanSound8K (Salamon et al., 2014) and DCASE (Stowell et al., 2015a), WSNet signiﬁcantly reduces the model sizes of the baselines by 100 × with comparable or even higher classiﬁcation accuracy.
When compressing more than 180 × , WSNet is only subject to negligible accuracy drop.
At the same time, WSNet also signiﬁcantly reduces the computation cost (up to 16 × ).
Such results strongly evident the capability of WSNet on learning compact and efﬁcient networks.
Note although the experiments in this paper are mostly limited to 1D CNNs, the same approach can be naturally generalized to 2D CNNs which we will explore in the future.
R ELATED W ORKS 2. A UDIO CLASSIFICATION
In this paper we considered Acoustic Scene Classiﬁcation (ASC) tasks as well as music detection task.
ASC aims to classify the surrounding environment where an audio stream is generated given the audio input (Barchiesi et al., 2015).
It can be applied in many different applications such as audio tagging (Cai et al., 2006), audio collections management (Landone et al., 2007), robotic navigation (Chu et al., 2006), intelligent wearable interfaces (Xu et al., 2008), context adaptive tasks (Schilit et al., 1994), and etc.
Music detection is a related task to determine whether a small segment of audio is music or not.
It is usually treated as a binary classiﬁcation problem given an audio segment as input, i.e. , to classify the segment into two categories, music or non-music.
Like in many other areas, convolutional neural networks (CNN) have been widely used in audio classiﬁcation tasks (Valenti et al., 2016)
(Salamon & Bello, 2017).
SoundNet (Aytar et al., 2016) stands out among different CNNs for sound classiﬁcation due to two reasons:
ﬁrst it is trained from the large-scale unlabeled sound data using visual information as a bridge, while many other networks are trained with smaller dataset.
Secondly, SoundNet directly takes the one dimensional raw wave signals as input so that there is no need to calculate time-consuming audio speciﬁc features, e.g.
MFCC (Pols et al., 1966) (Davis & Mermelstein, 1980) and spectrogram (Flanagan, 1972).
SoundNet has yielded signiﬁcant performance improvements over the state-of-the-art results on standard benchmarks for acoustic scene classiﬁcation.
In this paper, we demonstrate that the proposed WSNet achieves comparable or even better performance than SoundNet with signiﬁcantly smaller size and faster speed.
2. D EEP M ODEL C OMPRESSION AND A CCELERATION
Early approaches for deep model compression include (LeCun et al., 1989; Hassibi & Stork, 1993) that prune the connections in networks based on the second order information.
Most recent works in network compression adopts weight pruning (Han et al., 2015; Collins & Kohli, 2014; Anwar et al., 2017; Lebedev & Lempitsky, 2016;
Kim et al., 2015; Luo et al., 2017; Li et al., 2017), ﬁlter decomposition (Sindhwani et al., 2015; Denton et al., 2014; Jaderberg et al., 2014), hashed networks (Chen et al., 2015; 2016) and weight quantization (Han et al., 2016).
However, although those works reduce model size, they also suffer from large performance drop.
Bucilu et al.
(2006) and Ba & Caruana (2014) are based on student-teacher approches which may be difﬁcult to be applied in new tasks since they require training a teacher network in advance.
Denil et al.
(2013) predicts parameters based on a few number of weight values.
Jin et al.
(2016) proposes an iterative hard thresholding method, but only achieve relatively small compression ratios.
Gong et al.
(2014) uses binning method which can only be applied over fully connected layers.
Hinton et al.
(2015) compresses deep models by transferring the knowledge from pre-trained larger networks to smaller networks.
In contrast, WSNet is able to learn compact representation for both convolution layers and fully connected layers from scratch.
The deep models learned by WSNet can signiﬁcantly reduce model size compared to the baselines with comparable or even better performance.
In terms of deep model acceleration, the factorization and quantization methods listed above can also reduce computation latency in inference.
While irregular prunning (as done in most prunning methods (Han et al., 2016)) even incur computational overhead, grouped pruning (Lebedev & Lempitsky, 2016) is able to accelerate networks.
FFT (Mathieu et al., 2013) and LCNN (Bagherinezhad et al., 2016) are also used to speedup computation in pratice.
Comparatively, WSNet is superior by learning networks which have both smaller model size and faster computation versus baselines.
2. E FFICIENT M ODEL D ESIGN
WSNet presents a class of novel models that has the appealling properties of small model size and small computation cost.
Some recently proposed efﬁcient model architectures include the class of
Inception models (Szegedy et al., 2015; Ioffe & Szegedy, 2015; Chollet, 2016) which adopts depthwise separable convolutions, the class of Residual models (He et al., 2016; Xie et al., 2017; Chen et al., 2017) which uses residual path for efﬁcient optimization and the factorized networks which use fully factorized convolutions.
MobileNet (Howard et al., 2017) and Flattened networks (Jin et al., 2014) are based on factorization convolutions.
ShufﬂeNet (Zhang et al., 2017) uses group convolution and channel shufﬂe to reduce computational cost.
Compared with above works, WSNet presents a new model design strategy which is more ﬂexible and generalizable: the parameters in deep networks can be obtained conveniently from a more compact representation, e.g. through the weight sampling method proposed in this paper or other more complex methods based on the learned statistic models.
M ETHOD
In this section, we describe details of the proposed WSNet for 1D CNNs.
First, the notations are introduced.
Secondly, we elaborate on the core components in WSNet: weight sampling along the spatial dimension and channel dimension.
Thirdly, we introduce the denser weight sampling to enhance the learning capability of WSNet.
Finally, we propose an integral image method for accelerating WSNet in both training and inference.
3. N OTATIONS
Before diving into the details, we ﬁrst introduce the notations used in this paper.
The traditional 1D convolution layer takes as input the feature map F ∈ R T × M and produces an output feature map G ∈ R T × N where ( T, M, N ) denotes the spatial length of input, the channel of input and the number of ﬁlters respectively.
Note here we assume the output has the same spatial size as input which holds true by using zero padded convolution.
The 1D convolution kernel K used in the actual convolution of WSNet has the shape of ( L, M, N ) where L is the kernel size.
Let k n , n ∈ { 1 , · · · N } denotes a ﬁlter and f t , t ∈ { 1 , · · · T } denotes a input patch that spatially spans from t to t + L − 1 , then the convolution assuming stride one and zero padding is computed as:
[Equation]
where · stands for the vector inner product.
Note we omit the element-wise activation function to simplify the notation.
In WSNet, instead of learning each weight independently, K is obtained by sampling from a learned condensed ﬁlter Φ which has the shape of ( L ∗ , M ∗ ) .
The goal of training WSNet is thus cast to learn more compact DNNs which satisfy the condition of L ∗ M ∗ < LMN .
To quantize the advantage of WSNet in achieving compact networks, we deﬁne the compactness of K in a learned layer in
WSNet w.r.t.
the conventional layer with independently learned weights as:
[Equation]
In the following, we show that WSNet learn compact networks by sampling weights in two dimensions: the spatial dimension and the channel dimension.
3. W EIGHT SAMPLING 3.2. A LONG SPATIAL DIMENSION
In conventional CNNs, the ﬁlters in a layer are learned independently which suffers two disadvantages.
Firstly, the resulted DNNs have a large number of parameters, which impedes them to be
[Figure]
deployed in computation resource constrained platforms.
Second, such over-parameterization makes the network prone to overﬁtting and getting stuck in (extra introduced) local minimums.
To solve these two problems, a novel weight sampling method is proposed to efﬁciently reuse the weights among ﬁlters.
Speciﬁcally, in each convolutional layer of WSNet, all convolutional ﬁlters K are sampled from the condensed ﬁlter Φ , as illustrated in Figure 1.
By scanning the weight sharing ﬁlter with a window size of L and stride of S , we could sample out N ﬁlters with ﬁlter size of L .
Formally, the equation between the ﬁlter size of the condensed ﬁlter and the sampled ﬁlters is:
[Equation]
The compactness along spatial dimension is LM ∗ N L ∗ M ∗ ≈ LS .
Note that since the minimal value of S is 1, the minimal value of L ∗ ( i.e. the minimum spatial length of the condensed ﬁlter) is L + N −and the maximal achievable compactness is therefore L .
3.2. A LONG C HANNEL DIMENSION
Although it is experimentally veriﬁed that the weight sampling strategy could learn compact deep models with negligible loss of classiﬁcation accuracy (see Section 4), the maximal compactness is limited by the ﬁlter size L , as mentioned in Section 3.2.1.
For seeking more compact networks without such limitation, we propose a channel sharing strategy for WSNet to learn by weight sampling along the channel dimension.
As illustrated in Figure 1 (top panel), the actual ﬁlter used in convolution is generated by repeating sampling for C times.
The relation between the channels of ﬁlters before and after channel sampling is:
[Equation]
Therefore, the compactness of WSNet along the channel dimension achieves C .
As introduced later in Experiments (Section 4), we observe that the repeated weight sampling along the channel dimension signiﬁcantly reduces the model size of WSNet without signiﬁcant performance drop.
One notable advantage of channel sharing is that the maximum compactness can be as large as M ( i.e. when the condensed ﬁlter has channel of 1) which paves the way for learning much more aggressively smaller models ( e.g. more than 100 × smaller models than baselines).
The above analysis for weight sampling along spatial/channel dimensions can be conveniently generalized from convolution layers to fully connected layers.
For a fully connected layer, we treat its weights as a ﬂattened vector with channel of 1, along which the spatial sampling (ref.
Section 3.2.1) is performed to reduce the size of learnable parameters.
For example, for the fully connected layer “fc1” in the baseline network in Table 1, its ﬁlter size, channel number and ﬁlter number are 1536, 1 and 256 respectively.
We can therefore perform spatial sampling for “fc1” to learn a more compact representation.
Compared with convolutional layers which generally have small ﬁlter sizes and thus have limited compactnesses along the spatial dimenstion, the fully connected layers can achieve larger compactnesses along the spatial dimension without harming the performance, as demonstrated in experimental results (ref.
to Section 4.2).
3.2. T HE TRAINING OF CONDENSED FILTERS
WSNet is trained from the scratch in a similar way of conventional deep convolutional networks by using standard error back-propagation.
Since every weight K l,m,n in the convolutional kernel K is sampled from the condensed ﬁlter Φ along the spatial and channel dimension, the only difference is the gradient of Φ i,j is the summation of all gradients of weights that are tied to it.
Therefore, by simply recording the position mapping M : ( i, j ) →
( l, m, n ) from Φ i,j to all the tied weights in
K , the gradient of Φ i,j is calculated as:
[Equation]
where L is the conventional cross-entropy loss function.
In open-sourced machine learning libraries which represent computation as graphs, such as TensorFlow Abadi et al.
(2016), Equation (4) can be calculated automatically.
3. D ENSER W EIGHT S AMPLING
The performance of WSNet might be adversely affected when the size of condensed ﬁlter is decreased aggressively ( i.e. when S and C are large).
To enhance the learning capability of WSNet, we could sample more ﬁlters for layers with signiﬁcantly reduced sizes.
Speciﬁcally, we use a smaller sampling stride ¯ S ( ¯ S < S ) when performing spatial sampling.
In order to keep the shape of weights unchanged in the following layer, we append a 1 × 1 convolution layer with the shape of (1 , ¯ n, n ) to reduce the channels of densely sampled ﬁlters.
It is experimentally veriﬁed that denser weight sampling can effectively improve the performance of WSNet in Section 4.
However, since it also brings extra parameters and computational cost to WSNet, denser weight sampling is only used in lower layers of WSNet whose ﬁlter number ( n ) is small.
Besides, one can also conduct channel sampling on the added 1 × 1 convolution layers to further reduce their sizes.
3. E FFICIENT C OMPUTATION WITH INTEGRAL IMAGE
According to Equation 1, the computation cost in terms of the number of multiplications and adds ( i.e. Mult-Adds) in a conventional convolutional layer is:
[Equation]
However, as illustrated in Figure 2, since all ﬁlters in a layer in WSNet are sampled from a condensed ﬁlter Φ with stride S , calculating the results of convolution in the conventional way as in Eq.
(1) incurs severe computational redundance.
Concretely, as can be seen from Eq.
(1), one item in the ouput feature map is equal to the summation of L inner products between the row vector of f and the column vector of k .
Therefore, when two overlapped ﬁlters that are sampled from the condensed ﬁlter ( e.g. k 1 and k 2 in Fig. 2) convolves with the overlapped input windows ( e.g. f 1 and f 2 in
Fig. 2)), some partially repeated calculations exist ( e.g. the calculations highlight in green and indicated by arrow in Fig. 2).
To eliminate such redundancy in convolution and speed-up WSNet, we propose a novel integral image method to enable efﬁcient computation via sharing computations.
We ﬁrst calculate an inner product map P ∈ R T × L ∗ which stores the inner products between each row vector in the input feature map ( i.e. F ) and each column vector in the condensed ﬁlter ( i.e. Φ ): [Equation] ] and v [Equation]
[Equation] [Equation] otherwise. [Equation]
v...
I (u 1 + L -1,v 1 + L -1)
I (u 1 ,v 1 ) v 2 + Lv 1 + Lu 2 u 1 + L -1u 2 + Lvu
For notation deﬁnitions refer to Sec. 3.1.
The comparisons of computation costs between
WSNet and the baselines conventional are introduced in
The integral image for speeding-up convolution is denoted as I .
It has the same size as P and can be conveniently obtained throught below formulation:
[Equation]
Based on I , all convolutional results can be obtained in time complexity of O (1) as follows
[Equation]
Recall that the n -th ﬁlter lies in the spatial range of ( nS, nS + L − 1) in the condensed ﬁlter Φ .
Since G ∈ R T × N , it thus takes TN times of calculating Eq.
(8) to get G .
In Eq.
(6) ∼ Eq.
(8), we omit the case of padding for clear description.
When zero padding is applied, we can freely get the convolutional results for the padded areas even without using Eq.
(8) since I ( u, v ) =
I ( T, v − 1) , u >
T .
Based on Eq.
(6) ∼ Eq.
(8), the computation cost of the proposed integral image method is
[Equation]
Note the computation cost of P ( i.e. Eq.
(6)) is the dominating term in Eq.
Based on Eq.
Eq.
(9) and Eq.
(2), the theoretical acceleration ratio is
[Equation]
Recall that L is the ﬁlter size and S is the pre-deﬁned stride when sampling ﬁlters from the condensed ﬁlter Φ (ref.
to Eq.
In practice, we adopt a variant of above method to further boost the computation efﬁciency of WSNet, as illustrated in Fig 3.
In Eq.
(6), we repeat Φ by C times along the channel dimension to channelwrapping v 2 + Lv 1 + Lu 1 + L -1u
+ Lvvuu channle of demonstrated make it equal with the channel of the input F .
However, we could ﬁrst wrap the channels of F by accumulating the values with interval of L along its channel dimension to a thinner feature map ˜F ∈ R T × M ∗ which has the same channel number as Φ , i.e. ˜F ( i, j )
= (cid:80) C − 1 c =
F ( i, j + cM ∗ ) .
Both
Eq.
(7) and Eq.
(8) remain the same.
Then the computational cost is reduced to
[Equation]
where the ﬁrst item is the computational cost of warping the channels of F to obtain ˜F .
Since the dominating term ( i.e. Eq.
(6)) in Eq (10) is smaller than in Eq.
(9), the overall computation cost is thus largely reduced.
By combining Eq.
(10) and Eq.
(5), the theoretical acceleration compared to the baseline is
[Equation]
Finally, we note that the integral image method applied in WSNet naturally takes advantage of the property in weight sampling: redundant computations exist between overlapped ﬁlters and input patches.
Different from other deep model speedup methods (Sindhwani et al., 2015; Denton et al., 2014) which require to solve time-consuming optimization problems and incur performance drop, the integral image method can be seamlessly embeded in WSNet without negatively affecting the ﬁnal performance.
E XPERIMENTS
In this section, we present the details and analysis of the results in our experiments.
Extensive ablation studies are conducted to verify the effectiveness of the proposed WSNet on learning compact and efﬁcient networks.
On all tested datasets, WSNet is able to improve the classiﬁcation performance over the baseline networks while using 100 × smaller models.
When using even smaller ( e.g.
180 × ) model size, WSNet achieves comparable performance w.r.t the baselines.
In addition, WSNet achieves 2 × ∼ 4 × acceleration compared to the baselines with a much smaller model (more than 100 × smaller).
4. E XPERIMENTAL S ETTINGS
Datasets We collect a large-scale music detection dataset (MusicDet200K) from publicly available platforms ( e.g. Facebook, Twitter, etc. ) for conducting experiments.
For fair comparison with previous literatures, we also test WSNet on three standard, publicly available datasets, i.e ESC-50,
UrbanSound8K and DCASE.
The details of used datasets are as follows.
MusicDet200K aims to assign a sample a binary label to indicate whether it is music or not.
MusicDet200K has overall 238,000 annotated sound clips.
Each has a time duration of 4 seconds and is resampled to 16000 Hz and normalized (Piczak, 2015b).
Among all samples, we use 200,000/20,000/18,000 as train/val/test set.
The samples belonging to “non-music” count for 70% of all samples, which means if we trivially assign all samples to be ”non-music”, the classiﬁcation accuracy is 70%.
ESC-50 (Piczak, 2015a) is a collection of 2000 short (5 seconds) environmental recordings comprising 50 equally balanced classes of sound events in 5 major groups ( animals , natural soundscapes and water sounds , human non-speech sounds , interior/domestic sounds and exterior/urban noises ) divided into 5 folds for cross-validation.
Following Aytar et al.
, we extract 10 sound clips from each recording with length of 1 second and time step of 0.5 second ( i.e. two neighboring clips have 0.5 seconds overlapped).
Therefore, in each cross-validation, the number of training samples is 16000.
In testing, we average over ten clips of each recording for the ﬁnal classiﬁcation result.
UrbanSound8K (Salamon et al., 2014) is a collection of 8732 short (around 4 seconds) recordings of various urban sound sources ( air conditioner , car horn , playing children , dog bark , drilling , engine idling , gun shot , jackhammer , siren and street music ).
Like in ESC-50, we extract 8 clips with time length of 1 second and time step of 0.5 second from each recording.
For those that are less thansecond, we pad them with zeros and repeat for 8 times ( i.e. time step is 0.5 second).
DCASE (Stowell et al., 2015a) is used in the Detection and Classiﬁcation of Acoustic Scenes and
Events Challenge (DCASE).
It contains 10 acoustic scene categories, 10 training examples per category and 100 testing examples.
Each sample is a 30 seconds audio recording.
During training, we evenly extract 12 sound clips with time length of 5 seconds and time step of 2.5 seconds from each recording.
Evaluation criteria To demonstrate that WSNet is capable of learning more compact and efﬁcient models than conventional CNNs, three evaluation criteria are used in our experiments: model size, the number of multiply and adds in calculation (mult-adds) and classiﬁcation accuracy.
Baseline networks To test the scability of WSNet to different network architectures ( e.g. whether having fully connected layers or not), two baseline networks are used in comparision.
The baseline network used on MusicDet200K consists of 7 convolutional layers and 2 fully connected layers, using which we demonstrate the effectiveness of WSNet on both convolutional layers and fully connected layers.
For fair comparison with previous literatures, we ﬁrstly modify the state-of-theart SoundNet (Aytar et al., 2016) by applying pooling layers to all but the last convolutional layer.
As can be seen in Table 5, this modiﬁcation signiﬁcantly boosts the performance of original SoundNet.
We then use the modiﬁed SoundNet as baseline on all three public datasets.
The architectures of those two baseline networks are shown in Table 1 and Table 2 respectively.
Weight Quantization Similar to other works (Han et al., 2016; Rastegari et al., 2016), we apply weight quantization to further reduce the size of WSNet.
Speciﬁcally, the weights in each layer are linearly quantized to q bins where q is a pre-deﬁned number.
By setting all weights in the same bin to the same value, we only need to store a small index of the shared weight for each weight.
The size of each bin is calculated as (max( Φ ) − min( Φ )) /q .
Given q bins, we only need log 2 ( q ) bits to encode the index.
Assuming each weight in WSNet is represented using 4 bytes ﬂoat number ( bits) without weight quantization, the ratio of each layer’s size before and after weight quantization is 32 L ∗ M ∗ L ∗ M ∗ log 2 ( q )+32 q .
Recall that L ∗ and M ∗ are the spatial size and the channel number of condensed ﬁlter.
Since the condition L ∗ M ∗ (cid:29) q generally holds in most layers of WSNet, weight quantization is able to reduce the model size by a factor of 32 log 2 ( q ) .
Different from (Han et al., 2016; Rastegari et al., 2016) which learns the quantization during training, we apply weight quantization to WSNet
[Table] [Table]
after its training.
In the experiments
[Table]
we ﬁnd that such an off-line way is sufﬁcient to reduce model size without losing accuracy.
Implementation details WSNet is implemented and trained from scratch in Tensorﬂow (Abadi et al., 2016).
Following Aytar et al.
[Table]
the Adam (Kingma & Ba, 2014) optimizer and a ﬁxed learning rate of 0.001 and momentum term of 0.9 and batch size of 64 are used throughout experiments.
We initialized all the weights to zero mean gaussian noise with a standard deviation of 0.01.
In the network used on MusicDet200K
[Table]
the dropout ratio for the dropout layers (Srivastava et al., 2014) after each fully connected layer is set to be 0.8.
The overall training takes 100,000 iterations.
4. R ESULTS AND ANALYSIS 4.2. M USIC D ET 200K
Ablation analysis Through controled experiments, we investigate the effects of each component in WSNet on the model size, computational cost and classiﬁcation accuracy.
The comparative study results of different settings of WSNet are listed in Table 3.
For clear description, we name WSNets with different settings by the combination of symbols S/C/SC † /D/
Q. Please refer to the caption of
Table 3 for detailed meanings.
(1) Spatial sampling.
We test the performance of WSNet by using different sampling stride S in spatial sampling.
As listed in Table 3, S 2 and S 4 slightly outperforms the classiﬁcation accuracy of the baseline, possibly due to reducing the overﬁtting of models.
When the sampling stride is 8, i.e. the compactness in spatial dimension is 8 (ref.
to Section 3.2.1), the classiﬁcation accuracy of
S 8 only drops slightly by 0.6%.
Note that the maximum compactness along the spatial dimension is equal to the ﬁlter size, thus for the layer “conv7” which has a ﬁlter size of 4, its compactness is limited by 4 (highlighted by underline in Table 3) in S 8 .
Above results clearly demonstrate that the
[Table]
11 11 11 1 88.8 ± 0.
4 × 1 ×
S2 12 12 12 12 1 89.0 ± 0.
2 × 1 × S4 14 14 14 14 1 89.0 ± 0.
4 × 1.8 × S8 18 18 18 14 1 88.3 ± 0.
5.7 × 3.4 ×
C1 21 21 21 21 2 89.1 ± 0.
2 × 1 × C1 41 41 41 41 4 88.7 ±
[Table]
4 × 1.4 × C1 81 81 81 81 8 88.6 ± 0.
8 × 2.
[Table]
×
S
[Table]
SC †4 44 44 44 44 4 88.
[Table]
± 0.
11.1 × 5.7 × S
[Table]
8 SC †8 88 88 88 84 8 88.4 ± 0.
23 × 16.4 ×
S 8 C 8 SC † 8 D8 88 88 88 84 8 89.2 ± 0.
20 × 3.8 × S 8 C 8 SC † 15 D8 88 88 88 88 8 88.6 ± 0.
42 × 3.8 × S 8 C 8 SC † 8 Q8 88 88 88 84 8 88.4 ± 0.
92 × 16.4 × S 8 C 8 SC
[Table]
spatial sampling enables WSNet to learn signiﬁcantly smaller model with comparable accuracies w.r.t.
the baseline.
(2) Channel sampling.
Three different compactness along the channel dimension, i.e. 2, 4 and 8 are tested by comparing with baslines.
It can be observed from Table 3 that C 2 and C 4 and Chave linearly reduced model size without incurring noticeable drop of accuracy.
In fact, C 2 can even improve the accuracy upon the baseline, demonstrating the effectiveness of channel sampling in WSNet.
When learning more compact models, C 8 demonstrates better performance compared to
S 8 tha has the same compactness in the spatial dimension, which suggests we should focus on the channel sampling when the compactness along the spatial dimension is high.
We then simultaneously perform weight sampling on both the spatial and channel dimensions.
As demonstrated by the results of S 4 C 4 SC † 4 and S 8 C 8 SC † 8 , WSNet can learn highly compact models (more than 20 × smaller than baselines) without noticeable performance drop (less than 0.5%).
(3) Denser weight sampling.
Denser weight sampling is used to enhance the learning capability of WSNet with aggressive compactness ( i.e. when S and C are large) and make up the performance loss caused by sharing too much parameters among ﬁlters.
As shown in Table 3, by sampling 2 × more ﬁlters in conv1, conv2 and conv3, S 8 C 8 SC † 8 D 2 signiﬁcantly outperforms the S 8 C 8 SC † 8 .
Above results demonstrate the effectiveness of denser weight sampling to boost the performance of WSNet.
(4) Integral image for efﬁcient computation.
As evidenced in the last column in Table 3, the proposed integral image method consistently reduces the computation cost of WSNet.
For S 8 C 8 SC †which is 23 × smaller than the baseline, the computation cost (in terms of #mult-adds) is signiﬁcantly reduced by 16.4 times.
Due to the extra computation cost brought by the 1 × 1 convolution in denser ﬁlter sampling, S 8 C 8 SC † 8 D 2 achieves lower acceleration (3.8 × ).
Group convolution (Xie et al.,
[Table]
[Table] [Table]
(Aytar et
[Table] [Table]
[Table]
2017) can be used to alleviate the computation cost of the added 1 × 1 convolution layers.
We will explore this direction in our future work.
(5) Weight quantization.
It can be observed from Table 3 that by using 256 bins to represent each weight by one byte ( i.e. 8bits), S 8 C 8 SC † 15 A 2 Q 4 is reduced to 1/168 of the baseline’s model size while incurring only 0.1% accuracy loss.
Above result demonstrates that the weight quantization is complementary to WSNet and they can be used jointly to effectively reduce the model size of
WSNet.
In this paper, since we do not explore using weight quantization to accelerate models, the
WSNets before and after weight quantization have the same computational cost.
4.2. ESC-
The comparison of WSNet with other state-of-the-arts on ESC-50 is listed in Table 5.
The settings of WSNet used on ESC-50, UrbanSound8K and DCASE are listed in Table 4.
Compared with the baseline, WSNet is able to signiﬁcantly reduce the model size of the baseline by 25 times andtimes, while at the same time improving the accuracy of the baseline by 0.5% and 0.1% respectively.
The computation costs of WSNet are listed in Table 4, from which one can observe that
WSNet achieves higher computational efﬁciency by reducing the #Mult-Adds of the baseline by 2.3 × and 2.4 × , respectively.
Such promising results again demonstrate the effectiveness of WSNet on learning compact and efﬁcient networks.
After applying weight quantization to WSNet, its model size is reduced to only 1/180 of the baseline while the accuracy only slightly drops by 0.2%.
Compared with the SoundNet trained from scratch with provided data, WSNets signiﬁcantly outperform its classiﬁcation accuracy by over 10% with more than 100 × smaller models.
Using a transfer learning approach, SoundNet (Aytar et al., 2016) that is pre-trained using a large number of unlabeled videos achieves better accuracy than WSNet.
However, since the training method is orthogonal to WSNet, we believe that WSNet can achieve better performance by training in a similar way as SoundNet (Aytar et al., 2016) on a large amount of unlabeled video data.
[Table] [Table]
13M
4.2. U RBAN S OUND 8K
We report the comparsion results of WSNet with state-of-the-arts on UrbanSound8k in Table 6.
It is again observed that WSNet signiﬁcantly reduces the model size of baseline while obtaining comparative results.
Both Piczak (2015b) and Salamon & Bello (2015) use pre-computed 2D features after log-mel transformation as input.
In comparison, the proposed WSNet simply takes the raw wave of recordings as input, avoiding the time-consuming feature extraction as done in Piczak (2015b);
Salamon & Bello (2015).
4.2. DCASE
As evidenced in Table 7, WSNet outperforms the classiﬁcation accuracy of the baseline by 1% with a 100 × smaller model.
When using an even more compact model, i.e. 180 × smaller in model size, the classiﬁcation accuracy of WSNet is only one percentage lower than the baseline ( i.e. has only one more incorrectly classiﬁed sample), verifying the effectiveness of WSNet.
Compared with
SoundNet (Aytar et al., 2016) that utilizes a large number of unlabeled data during training, WSNet (S 8 C 4 D 2 Q 4 ) that is 100 × smaller achieves comparable results by only using the provided data.
C ONCLUSION
In this paper, we present a class of W eight S ampling networks (WSNet) which are highly compact and efﬁcient.
A novel weight sampling method is proposed to sample ﬁlters from condensed ﬁlters which are much smaller than the independently trained ﬁlters in conventional networks.
The weight sampling in conducted in two dimensions of the condensed ﬁlters, i.e. by spatial sampling and channel sampling.
Taking advantage of the overlapping property of the ﬁlters in WSNet, we propose an integral image method for efﬁcient computation.
Extensive experiments on four audio classiﬁcation datasets including MusicDet200K, ESC-50, UrbanSound8K and DCASE clearly demonstrate that
WSNet can learn compact and efﬁcient networks with competitive performance
S UPPLEMENTARY M ATERIAL 6. ABLATIVE STUDY OF WSN ET ON ESC-
We conduct extensive ablative study of WSNet on the public dataset ESC-50 to investigate the effects of each component of WSNet on ﬁnal performance.
The results are listed in Table 8.
[Table]
8 × 2.8 ×
S 4 C4 44 44 44 44 465.6 ± 0.
16 × 6.3 × S 8 C4 44 84 84 88 865.2 ± 0.
60 × 18.1 ×
S 8 C 4 D4 44 44 44 48 466.5 ± 0.
25 × 2.3 × S 8 C 4 D 2 Q4 44 44 44 48 466.2 ± 0.
100 × 2.3 × S 8 C 8 D4 44 84 84 88 866.1 ± 0.
45 × 2.4 × S 8 C 8 D 2 Q4 44 84 84 88 865.8 ± 0.
180 × 2.4 ×
[Figure] [Figure]
6. C OMPARISON BETWEEN WSN ET AND BASELINES WITH REDUCED FILTERS
To further verify WSNet’s capacity of learning compact models, we conduct experiments on ESC-50 and MusicDet200K to compare WSNet with baselines compressed in an intuitive way, i.e. reducing the number of ﬁlters in each layer.
If #ﬁlters in each layer is reduced by T , the overall #parameters in baselines is reduced by T 2 ( i.e. the compression ratio of model size is T 2 ).
In Figure 4 and Figure 5, we plot how baseline accuracy varies with respect to different compression ratios and the accuracies of WSNet with the same model size of compressed baselines.
As shown in Figure 4 and Figure 5, WSNet outperforms baselines by a large magin across all compression ratios.
Particularly, when the comparison ratios are large ( e.g. 45 on ESC-50 and 42 on MusicDet200K),
[Table] [Table]
[Table]
baselines suffer severe performance drop.
In contrast, WSNet achieves comparable accuracies with full-size baselines without signiﬁcant drop (66.1 versus 66.0 on ESC-50 and 88.6 versus 88.9 on MusicDet200K).
This clearly demonstrates the effectiveness of weight sampling methods proposed in WSNet.
6. WSN ET ON 2D CONVNETS 6.3. E XTENSION OF WSN ET FROM 1D CONVNET TO 2D CONVNET
In this paper, we focus on WSNet with 1D convnets.
Comprehensive experiments clearly demonstrate its advantages in learning compact and computation-efﬁcient networks.
We note that WSNet is general and can also be applied to build 2D convnets.
In 2D convnets, each ﬁlter has three dimensions including two spatial dimensions ( i.e. along X and Y directions) and one channel dimension.
One straightforward extension of
WSNet to 2D convnets is as follows: for spatial sampling, each ﬁlter is sampled out as a patch (with the same number of channels as in condensed ﬁlter) from condensed ﬁlter.
Channel sampling remains the same as in 1D convnets, i.e. repeat sampling in the channel dimension of condensed ﬁlter.
Following the notations for WSNet with 1D convnets (ref. to Sec. 3.1), we denote the ﬁlters in one layer as K ∈ R w × h × M × N where ( w, h, M, N ) denote the width and height of each ﬁlter, the number of channels and the number of ﬁlters respectively.
The condensed ﬁlter Φ has the shape of ( W, H, M ∗ ) .
The relations between the shape of condensed ﬁlter and each sampled ﬁlter are:
[Equation]
where S w and S h are the sampling strides along two spatial dimensions and C is the compactness of WSNet along channel dimension.
The compactness (please refer to Sec.
3.1 for deniﬁnition) of WSNet along spatial dimension is WHwhN .
However, such straightforward extension of WSNet to 2D convnets is not optimum due to following two reasons: (1) Compared to 1D ﬁlters, 2D ﬁlters present stronger spatial dependencies between the two spatial dimensions.
Nave extension may fail to capture such dependencies.
(2) It is not easy to use the integral image method for speeding up WSNet in 2D convnets as in 1D convnets.
Because of above problems, we believe there are more sophisticated and effective methods for applying WSNet to 2D convnets and we would like to explore in our future work.
Nevertheless, we conduct following preliminary experiments on 2D convents using above intuitive extension and verify the effectiveness of WSNet in image classiﬁcation tasks on MNIST and
CIFAR
[Table] [Table]
P RELIMINARY EXPERIMENTAL RESULTS
Since both WSNet and HashNet Chen et al.
(2015; 2016) explore weights tying, we compare them on MNIST and CIFAR10.
For fair comparison, we use the same baselines used in Chen et al.
The baseline used for MNIST is a 3-layer fully connected network with a single hidden layer containing 1,000 hidden units.
The conﬁguration of the baseline network used for CIFAR10 is listed in Table 9.
All hyperparameters used training including learning rate, momentum, drop out and so on follow Chen et al.
For each dataset, we hold out 20% of the whole training samples to form a validation set.
The comparison results between WSNet and HashNet on MNIST/CIFAR10 are listed in Table 10/Table 11, respectively.
As one can observe in Table 10 and Table 11, when learning networks with the same sizes, WSNet achieves signiﬁcant lower error rates than HashNet on both datasets.
Above results clearly demonstrate the advantages of WSNet in learning compact models.
Besides, we also conduct experiment on CIFAR10 with the state-of-the-art ResNet
He et al.
(2016) network as baseline.
Both the network architecture and training hyperparameters follow He et al.
WSNet is able to achieve 20 × smaller model size with slight performance drop (0.6%).
Such promising results further demonstrate the effectiveness of WSNet.