W AVELET F EATURE M APS C OMPRESSION FOR L OW B ANDWIDTH C ONVOLUTIONAL N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Quantization is one of the most effective techniques for compressing Convolutional
Neural Networks (CNNs), which are known for requiring extensive computational resources.
However, aggressive quantization may cause severe degradation in the prediction accuracy of such networks, especially in image-to-image tasks such as semantic segmentation and depth prediction.
In this paper, we propose
Wavelet Compressed Convolution (WCC)—a novel approach for activation maps compression for 1 × 1 convolutions (the workhorse of modern CNNs).
WCC achieves compression ratios and computational savings that are equivalent to low bit quantization rates at a relatively minimal loss of accuracy.
To this end, we use a hardware-friendly Haar-wavelet transform, known for its effectiveness in image compression, and deﬁne the convolution on the compressed activation map.
WCC can be utilized with any 1 × 1 convolution in an existing network architecture.
By combining WCC with light quantization, we show that we achieve compression rates equal to 2-bit and 1-bit with minimal degradation in image-to-image tasks.
I NTRODUCTION
Over the past years, Convolutional Neural Networks (CNNs) have brought signiﬁcant improvement in processing images, video, and audio (LeCun et al., 2015; Krizhevsky et al., 2017).
However, CNNs require signiﬁcant computational and memory costs, which makes the usage of CNNs difﬁcult in applications where computing power is limited, e.g., on edge devices.
To address this limitation, several approaches have been proposed to reduce the computational costs of neural networks.
Among the popular ones are weight pruning (Han et al., 2015; Guo et al., 2016), architecture search (Howard et al., 2019), and quantization (Li et al., 2017; Banner et al., 2018a).
In principle, all these approaches can be applied simultaneously on top of each other to reduce the computational costs of CNNs.
In this work, we focus on quantization.
This approach relieves the computational cost of CNNs by quantizing their weights and activation (feature) maps using low numerical precision so that they can be stored and applied as ﬁxed point integers (Hubara et al., 2017; Banner et al., 2018a).
In particular, it is common to apply aggressive quantization (less than 4-bit precision) to compress the activation maps (Esser et al., 2020).
Yet, it is known that compressing natural images using uniform quantization is sub-optimal.
Indeed, applying aggressive quantization in certain CNNs can lead to signiﬁcant degradation in the accuracy of the network.
The impact is especially evident for the image-to-image tasks such as semantic segmentation (Tang et al., 2019) and depth prediction (Lee et al., 2020), where each pixel has to be assigned a value.
In a recent work that targets quantized U-Nets (Tang et al., 2019), activations bit rates are kept relatively high (about 8 bits) while the weight bit rates are lower (down to 2 bits).
Beyond that, we note that the majority of the quantization works are applied and tested on image classiﬁcation (Esser et al., 2020; Li et al., 2020, and references therein).
This work aims to improve the compression of the activation maps by introducing Wavelet Compressed Convolution (WCC) layers.
These layers utilize the Haar-wavelet transform (Daubechies, 1992) to compress the activation maps before applying convolutions.
This wavelet transform and its inverse can be applied efﬁciently in linear complexity for each channel, using additions and subtractions, thanks to the simplicity of the Haar basis.
The main idea of our approach is to keep the same top k entries (in magnitude) of the transformed activations maps with respect to all channels (dubbed as joint shrinkage) and perform the convolution in the wavelet domain on the compressed signals, saving signiﬁcant computations.
We show that the transform and shrinkage operations
[Figure]
compression commute with the 1 × 1 convolution, the heart of modern CNNs.
This procedure is applied along with modest quantization to reduce the computational costs further.
We demonstrate the effectiveness and ﬂexibility of our WCC by applying it in popular network architectures:
DeeplabV3plus (Chen et al., 2018) with MobileNetV2 encoder (Sandler et al., 2018) for semantic segmentation, and BTS (Lee et al., 2020) with ResNeXt50 encoder (Xie et al., 2017) for depth prediction.
We show that using WCC dramatically improves the results over aggressive quantization for the same compression rates while retaining the baseline network architecture.
R ELATED WORK
Quantized Neural Networks:
Quantized neural networks have been quite popular recently and are exhibiting impressive progress in the goal for true network compression and efﬁcient CNN deployment.
Quantization methods include Zhou et al.
; Zhang et al.
; Banner et al.
(2018b), and in particular Li et al.
Esser et al.
; Choi et al.
(2019), which show that the clipping parameters—a highly important parameter in quantization schemes—can be learned through optimization.
Beyond that, there are more sophisticated methods to improve the mentioned schemes.
For example, dynamic quantization schemes utilize different bit allocations at every layer (Dong et al., 2020; Cai & Vasconcelos, 2020; Uhlich et al., 2020).
Non-uniform methods can improve the quantization accuracy (Yamamoto, 2021) but require a look-up table, which reduces hardware efﬁciency.
Quantization methods can also be enhanced by combination with pruning (Tung & Mori, 2018) and knowledge distillation for better training (Kim et al., 2019).
The works above focus on image classiﬁcation.
When considering image-to-image tasks like semantic segmentation, networks tend to be more sensitive to quantization of the activations.
In the work of AskariHemmat et al.
(2019), targeting segmentation of medical images, the lowest bit rate for the activations is 4 bits, and signiﬁcant degradation in the performance is evident compared tobits.
These results are consistent with the work of Tang et al.
(2019) that was mentioned earlier, which uses a higher bit rate for the activations than for the weights.
The work of Xu et al.
(2018) uses weight (only) quantization for medical image segmentation as an attempt to remove noise and not for computational efﬁciency.
The recent work of Liu et al.
(2021) shows both a sophisticated post-training quantization scheme and includes ﬁne-tuned semantic segmentation results.
Again a signiﬁcant degradation is observed when going from 6 to 4 bits.
One exception is the work of Heinrich et al.
(2018) that uses ternary networks (equivalent to 2 bits here) and segments one medical data set relatively well compared to its full precision baseline.
In this work, we focus on the simplest possible quantization scheme: uniform (across all weights and activations), quantization-aware training, with per-layer clipping parameters.
That is to ensure hardware compatibility and efﬁcient use of available training data.
In principle, one can run our platform regardless of the quantization type and scenario (e.g., non-uniform/dynamic quantization).
Also, since we target the compression of the activations, then any method that is focused on the weights (e.g., pruning (Tung & Mori, 2018)) can be combined here as well.
Wavelet transforms and compression: Wavelet transforms are widely used in image processing (Porwik & Lisowska, 2004).
For example, the JPEG2000 format (Rabbani, 2002) uses the wavelet domain to represent images as highly sparse feature maps.
Recently, wavelet transforms have been used to deﬁne convolutions and architectures in CNNs for various of imaging tasks.
For example, the work of Huang et al.
(2017) presents a network architecture for super-resolution, where the wavelet coefﬁcients are predicted and used to reconstruct the high-resolution image.
The works of Duan et al.
(2017); Williams & Li (2018) use wavelet transforms in place of pooling operators to improve CNNs performance.
The former uses dual-tree complex wavelet transform (Kingsbury, 1998), and the latter learns the wavelet basis as part of the network optimization.
In these cases, the wavelet transform is not used for compression, but to better preserve information with its low-pass ﬁlters.
The work of Wolter et al.
(2020) proposes a wavelet-based approach to learn basis functions for the wavelet transform to compresses the weights of linear layers, as opposed to compression of the activation as we apply here.
We use the Haar transform (as opposed to a learned one) for its hardware efﬁciency.
Using a different transform (known or learned) is also possible, at the corresponding computational cost.
The work of Sun et al.
(2021) introduces quantization in the wavelet domain, somewhat similarly to this work.
However, the authors suggest to improve the quantization scheme by learning a different clipping parameter per wavelet component, but without the feature shrinkage stage, which is the heart of our approach (we use the same clipping parameter for the whole layer using 8 bits for hardware efﬁciency).
As mentioned before, one can use our WCC together with different types of quantization schemes, in particular including Sun et al.
(2021), taking the additional hardware complexity of using different clipping parameter per component into account.
Lastly, Liu et al.
(2018) suggest using a modiﬁed U-Net architecture for image-to-image translation.
There, wavelet transforms are used for down-sampling, and the inverse is used for up-sampling.
This work is architecture-speciﬁc, and the method can not be easily integrated into other architectures.
In contrast, here we propose the WCC layer to easily replace 1 × 1 convolutions regardless of the network architecture.
Hence our framework is, in principle, also suitable for post-training quantization (Banner et al., 2019), where the data is not available, and the original network is not retrained.
B ACKGROUND
Quantization-aware training Quantization schemes can be divided into post- and quantizationaware training schemes.
Post training schemes perform model training and model quantization separately, which is most suitable when the training data is not available during the quantization phase (Banner et al., 2019; Nagel et al., 2019;
Cai et al., 2020).
On the other hand, quantization-aware training schemes are used to adapt the model’s weights as an additional training phase.
Such schemes do require training data but generally provide better performance.
Quantization schemes can also be divided into uniform vs. non-uniform methods, where the latter is more accurate but the former is more hardware friendly (Li et al., 2020;
Jung et al., 2019).
Lastly, quantization schemes can utilize quantization parameters per channel within each layer or utilize these parameters only per layer (where all the channels share the same parameters).
Similar to before, per-channel methods are difﬁcult to exploit in hardware, while per-layer methods are less accurate but are more feasible for deployment on edge devices.
This paper focuses on per-layer and uniform quantization-aware training for both weights and activations and aims to improve this with wavelet transforms.
Other quantization schemes can be applied within our wavelet compression framework as well.
In quantization-aware training, we set the values of the weights to belong to a small set so that after training, the application of the network can be carried out in integer arithmetic, i.e., activation maps are quantized as well.
Even though that we use discontinuous rounding functions throughout the network, quantization-aware training schemes utilize gradient-based methods to optimize the network’s weights (Han et al., 2015; Yin et al., 2019).
In a nutshell, when training, we iterate on the ﬂoating-point values of the weights.
During the forward pass, both the weights and activation maps are quantized, while during the backward pass, the Straight Through Estimator (STE) approach is used (Bengio et al., 2013), where we ignore the rounding function, whose exact derivative is zero.
The speciﬁc quantization scheme that we use is based on Li et al.
First, the pointwise quantization operator is deﬁned by:
[Equation]
where t ∈ [ − 1 , 1] or t ∈ [0 , 1] for signed or unsigned quantization, respectively 1 .
The parameter b is the number of bits that are used to represent t .
During the forward pass, each number undergoes three steps: scale, clip, and round.
That is, to get the quantized version of any number, we apply: [Equation]
signed
[Equation]
Here, x, x b are the real-valued and quantized tensors, and α is the clipping parameter.
The parameters α in (2) play an important role in the error generated at each quantization and should be chosen carefully.
The works of Li et al.
Esser et al.
(2020) introduced an effective gradient-based optimization to ﬁnd the clipping values α for each layer, again using the STE approximation.
This enables the quantized network to be trained in an end-to-end manner with backpropagation.
To further improve the optimization, weight normalization is also used before each quantization.
Haar wavelet transform and compression.
In this section, we describe the Haar-wavelet transform in deep learning language and its usage for compression.
See Vyas et al.
(2018) for more details on the use of wavelets for image compression.
Given an image channel x , the one-level Haar transform can be achieved by a separable 2D stride two convolution with the following kernel:
[Equation]
These kernels can also be expressed as a composition of separable 1D kernels [1 , 1] / √ 2 and [1 , − 1] / √ 2 .
The result of the convolution [ y 1 , y 2 , y 3 , y 4 ] =
Conv ( W , x ) has four channels, each of which has half the resolution of x .
The leftmost kernel in (3) is an averaging kernel, and the three right kernels are edge-detectors.
This, together with the fact that images are piece-wise smooth, leads to relatively sparse images y 2 , y 3 , y 4 .
Hence, if we retain only the few top-magnitude entries in these vectors, we keep most of the information, as most of the entries we drop are zeros.
That is the main idea of wavelet compression.
We denote the Haar-wavelet transform by y = HWT ( x ) , where y is deﬁned as the concatenation of the vectors y 1 , ..., y 4 into one.
Since the kernels in (3) form an orthonormal basis, applying the inverse transform is obtained by the transposed convolution of (3):
[Equation]
transposed ( W , [ y 1 , y 2 , y 3 , y 4 ]) .
However, unlike y 2 , y 3 , y 4 , the averaged image y 1 is not sparse, and is just the down-sampled original x .
Hence, in the multilevel wavelet transform, we apply the kernel in (3) on y 1 to generate further down-sampled sparse channels.
For example, a 2-level Haar transform can be summarized as
[Equation]
and the resulting transformed image with 2-levels can be written as the concatenated vector
[Equation]
In this work we use 3 levels in all the experiments.
To apply compression we deﬁne the operator
T that extracts the top k entries in magnitude from the vector y :
y compressed =
T · HWT ( x ) .
To de-compress this vector we ﬁrst zero-ﬁll y compressed (multiply with T (cid:62) ) and apply the inverse Haar transform, which is the transposed convolutions in the opposite order.
W AVELET C OMPRESSED C ONVOLUTION
In this work, we aim to reduce the memory bandwidth and computational cost associated with convolutions performed on intermediate activation maps.
To this end, we apply the Haar-wavelet transform to compress the activation maps, in addition to light quantization of 8 bits.
Our method is most efﬁcient for scenarios with high-resolution feature maps (i.e., large images, whether in 2D or 3D), where the wavelet compression is most effective.
Such cases are mostly encountered in image-to-image tasks like semantic segmentation, depth prediction, image denoising, in-painting, super-resolution, and more.
Typically, in such scenarios, the size and memory bandwidth used for the weights are relatively small compared to those used for the features (e.g., point cloud segmentation).
In addition, since we have to predict each pixel, typical architectures do not aggressively down-sample the images, as opposed to multi-class image classiﬁcation, where activation maps become small, and the number of channels grows towards the end of the network.
Convolution in the wavelet domain: Since we focus on computational efﬁciency, we use the Haar transform, as it is the simplest and most computationally efﬁcient wavelet variant.
Indeed, the Haar transform involves binary pooling-like operations, which include only additions, subtractions, and a bit-wise shift.
We focus on the compression of fully-coupled 1 × 1 convolutions, as these are the workhorse of lightweight and efﬁcient architectures like MobileNet (Sandler et al., 2018), ShufﬂeNet (Ma et al., 2018), EfﬁcientNet (Tan & Le, 2019), and ResNeXt (Xie et al., 2017).
All these modern architectures rely on 1 × 1 convolutions in addition to grouped or depthwise spatial convolutions (i.e., with 3 × 3 or larger kernels), which comprise a small part of the computational effort in the network—the 1 × 1 operations dominate the inference cost (see Appendix B).
The main idea of our method is to transform and compress the input using the wavelet transform prior to the 1 ×convolution, then apply it in the wavelet domain on a fraction of the input size.
Since the wavelet compression is applied separately on each channel, it commutes with the 1 × 1 convolution.
Hence, applying the convolution in the wavelet domain is equivalent to applying it in the spatial domain.
Joint hard shrinkage: Prior to the 1 × 1 convolution, the spatial wavelet transform is applied, and we get sparse feature maps.
Since the different channels result from the same input image propagated through the network, they typically include patterns at similar locations, and hence the sparsity pattern of their wavelet-domain representation is relatively similar.
This idea of redundancy in the channel space is exploited in the works of Han et al. ; Eliasof et al. Bae et al. (2021), where part of the channels are used to represent the others.
Therefore, we perform a joint shrinkage operation between all the channels, in which we zero and remove the entries with the smallest feature norms across channels, resulting in a compressed representation of the activation maps 2 .
The locations of the non-zeros in the original image are kept in a single index list or a bit-map for all the channels in the layer, as they are needed for the inverse transform back to the spatial domain.
Lastly, we also apply light 8-bit quantization to the transformed images to further improve the compression rate.
The weights and wavelet-domain activations are quantized using the symmetric scheme as described in section 3.
More precisely, the advantage of the wavelet transforms is their ability to compress images.
Denote the Haar transform matrix as H , i.e., Hx = HWT ( x ) .
Then, for most natural images we have that
[Equation]
where T is the shrinkage operator described above, and because of its orthogonality, H (cid:62) is the inverse transform IHWT.
Our WCC layer is deﬁned by:
[Equation]
where K 1 × 1 is the learned convolution matrix.
The workﬂow is illustrated in Figure 2.
Note that the convolution operates on the compressed domain, hence, if T can signiﬁcantly reduce the dimensions of the channels, this leads to major savings in computations.
Our method aims at compression only.
Hence, we show that a 1 × 1 convolution kernel can be applied both in the spatial and in the compressed wavelet domain.
By its deﬁnition, we can write a 1 ×convolution as a summation over channels.
That is:
[Equation]
where k ij ∈ R are the weights of the convolution tensor.
Now, suppose we wish to compress the result y , through (6).
Because T and H are separable and spatial, we get by simple linear arithmetic:
[Equation]
Hence, the wavelet compression operator, which is known to be highly efﬁcient for natural images, commutes with the 1 × 1 convolution, so the latter can be applied on the compressed signal and get the same result as compressing the result directly.
This introduces an opportunity to save computations on the one hand and use more accurate compression on the other.
The description above is suited for 1 × 1 convolutions only, while many CNN architectures involve spatial convolutions with larger kernels, strides, etc.
The main concept is that fully coupled convolutions that mix all channels are expensive, inefﬁcient, and redundant when used with large spatial kernels (Ephrath et al., 2020).
The spatial mixing can be obtained using separable convolutions at less cost without losing efﬁciency (Chen et al., 2018).
Since we aim at saving computations, separating the kernels in the architecture is recommended even before discussing any type of lossy compression.
Furthermore, separable convolutions (and the Haar transform) can be applied separately in chunks of channels or together as part of the 1 × 1 convolution using specialized implementation.
Hence the memory bandwidth can be reduced.
The 1 × 1 convolution, on the other hand, involves all channels at once; hence it is harder to reduce the bandwidth in this case.
4. C OMPUTATIONAL C OSTS IN B IT O PERATIONS (BOP S )
To evaluate the computational cost involved in WCC we use the measure of Bit-Operations (BOPs)
(Wang et al., 2020; Louizos et al., 2018).
First, the number of Multiply-And-Accumulate (MAC) operations in a convolutional layer is given by
[Equation]
where C in and C out are the number of input and output channels, ( N W , N H ) is the size of the input, ( K W , K H ) is the size of the kernel, and ( S W , S H ) is the stride value.
The BOPs count is then
[Equation]
where b w and b a denote the number of bits used for weight and activations.
As described in section 3, the Haar transform is separable between the input channels, and can be viewed as four 2 × 2 convolutions with stride (2 , 2) and binary weights.
Hence, the one-level transform requires 4 · C in · W · H · b a BOPs.
The transform can be used with more levels of compression explained in section 3, on down-scaled inputs, resulting in a total of
[Equation]
BOPs, where L is the level of compression.
Similarly, the inverse-transform result in the same calculation, only with C out in place of C in .
To demonstrate the relatively small cost of the compression, consider a 1 × 1 convolution with C in = 160 , C out = 960 , input size of (34 , 34) , and quantization b w = b a = 8 (which is part of a network used later in section 5).
This layer costs 11 , 364 M BOPs.
Using a 3 levels wavelet transform and its inverse for this layer results in 54 M BOPs, a negligible cost which allows for better compression, as we demonstrate next.
R ESULTS
In this section, we evaluate our WCC layer’s performance in image-to-image tasks—semantic segmentation and depth estimation—where the feature maps are relatively large and suitable for wavelet compression.
We compare our method to quantization-aware training based on the work of Li et al.
(2020), which we also use in our method for 8-bit quantization of the shrunk signals.
Our code is implemented in PyTorch (Paszke et al., 2017) based on Torchvision implementations, including the same data augmentation.
We ran our experiments on NVIDIA 24GB RTX 3090 GPU.
We detail our experimental setup for each section separately.
[Figure]
5. Q UALITATIVE C OMPRESSION A SSESSMENT
Before describing experiments with common datasets and tasks, we ﬁrst qualitatively demonstrate the advantage of our approach.
First, we compare the MSE of standard quantization vs our approach on a feature map from a pretrained MobileNetV3 applied on a batch of 1000 images from the ImageNet dataset (Krizhevsky et al., 2017).
We consider an effective bit rate range of [2 , 8] .
The quantization of the shrunk wavelet coefﬁcients is 8 bit, and multiplying it with the compression ratio yields the effective bit rate.
E.g., 8 bit and 25% shrinkage is effectively 2 bits.
This also agrees with the BOPs measure in Eq.
Figure 1 shows the MSE comparison per effective bit rate.
One can see that the information loss based on the MSE is more signiﬁcant in standard quantization.
We note that the quantization parameters in this experiment were chosen by an exhaustive search for minimizing the
MSE for every dot in the plot.
Furthermore, in Figure 3 we can see a visualization comparison of a typical activation map compressed by a standard quantization and our method.
It is obvious that the perceptive degradation in the image approximation is more signiﬁcant in the standard quantization.
This is not surprising since wavelet image compression is more advanced than uniform quantization.
5. S EMANTIC S EGMENTATION
Semantic segmentation is the task of assigning a label to every pixel in the input image.
CNNs have shown signiﬁcant improvements over traditional methods but at a high computational cost.
Our model of choice here is the popular DeeplabV3plus (Chen et al., 2018) with MobileNetV2 backbone (Sandler et al., 2018).
We evaluated the proposed method on the Cityscapes and Pascal VOC.
The
Cityscapes dataset (Cordts et al., 2016) contains images of urban street scenes.
The images are of size 1024x2048 with pixel-level annotation of 19 classes.
During training, we used a random crop of size 768x768 and no crop for the validation set.
The Pascal VOC (Everingham et al., 2015) dataset contains images of size 513x513 with pixel-level annotation of 20 foreground object classes and a background class.
We augmented the dataset similarly to Chen et al.
[Figure]
Precision
Wavelet
Cityscapes
Pascal VOC (Weight / Activation) shrinkage
BOPs (B) mIoU BOPs (B) mIoU
Full precision
None 36, 0.
8bit / 8bit
None 2, 0.
8bit / 6bit
None 1, 0.
8bit / 4bit
None 1, 0.
8bit / 8bit 100% 2, 0.
8bit / 8bit 50% 1, 0.
8bit / 8bit 25% 0.
8bit / 8bit 12.5% 0.
4bit / 8bit
None 1, 0.
4bit / 6bit
None 0.
4bit / 4bit
None 0.
4bit / 8bit 100% 1, 0.
[Table]
We used two conﬁgurations for the weights—4 and 8 bits—and for each of them we used different compression rates for the activations, both for the standard quantization and our WCC layer.
Like before, our WCC layer with a shrinkage rate of 50% has nearly equal BOPs to reducing the activation bits from 8 to 4.
Likewise, a shrinkage rate of 25% nearly equals to 2 bits.
The training scheme used is similar to Li et al.
We ﬁrst trained a network in full precision and then gradually reduced the bit rates (this was used for the WCC as well).
For all datasets and model optimization, we used SGD with momentum 0.9, weight decay 10 − 4 , learning rate decay 0 . 9 , and set the batch size to 16.
For Cityscapes, we trained the full precision model for 160 epochs with a base learning rate of 0.1, and each of the retrains was trained for 50 epochs with a base learning rate of 0.01.
For Pascal VOC, we trained the full precision model for 50 epochs with a base learning rate of 0.01, and each of the retrains was trained for 25 epochs with a base learning rate of 0.002.
Table 1 shows the performance and BOPs of each model, from which we can see that our model achieves superior performances at high compression conﬁgurations, outperforming the 4-bit activation even with compression rates equivalent to 2- and 1-bits.
Figure 4 shows a visual example.
[Table]
Comparison to Liu et al.
This work deals with zero-shot quantization and shows results on quantization-aware training (ﬁne-tuned) for Cityscapes, albeit using 256 × 256 input images.
In their paper, the 8bit/8bit ﬁne-tuned result has a mIoU of 0.613 compared to about 0.7 here, in both the standard and wavelet cases.
Our standard 4bit/8bit setting achieves 0.67 vs. 0.6 for 6bit/6bit by Liu et al.
Furthermore, when considering 4bit weights, all our wavelet compression results (up to 12.5%) outperform or are on par with the 4bit/4bit result of 0.56 by Liu et al.
5. M ONOCULAR D EPTH E STIMATION
In this section, we apply our compression technique on Big-to-Small (BTS, Lee et al., 2020), a network architecture for monocular depth estimation—a task of estimating scene depth using a single image.
We evaluated the results on the KITTI dataset (Geiger et al., 2013), containing images of autonomous driving scenarios, each of size ∼ 1241x376 pixels.
The dataset splits are as in BTS, using the strategy of Eigen et al.
As in BTS, we use a maximum value of 80 meters for prediction when testing, and the performance evaluation is based on the cropping scheme of Garg et al.
We did not modify the initial convolution, the ﬁnal convolutions, and the depth prediction layer when applying the compression.
Using any compression technique on these layers resulted in a signiﬁcant degradation in the results.
In our experiments, we focus on a ResNeXt50 backbone.
The compressed models are ﬁne-tuned from a 8bit/8bit quantized network, trained from scratch.
In local planar guidance layers, we use 8bit/4bit quantization instead of our wavelet scheme.
When training, we use the same conﬁguration presented in the BTS repository.
We use the AdamW optimizer (Loshchilov & Hutter, 2019), a learning rate of 0.00014, a weight decay of 0.001, a batch size of 4 images, and train for 50 epochs.
Table 2 shows a comparison between the standard quantization and WCC for BTS.
Using our WCC layer with 50% compression resulted in comparable scores to the alternative method of 8bit/4bit quantization (with similar BOPs).
Moreover, when applying a compression factor of 25% (50% on low spatial-sized tensors), we achieve superior results to the quantized alternative.
This comparison is presented qualitatively in Appendix A.
Finally, in the full 25% compression experiment, we see that even when aggressively compressing the low spatial-sized tensors, we achieve better results than using low-precision standard quantization.
C ONCLUSION
In this work, we presented a new approach for feature map compression, aiming to reduce the memory bandwidth and computational cost of image-to-image CNNs, where typically the resolution is high.
Our approach is based on the classical Haar-wavelet image compression, which has been used for years in standard devices and simple hardware, and in real-time.
We save the computational cost by applying 1 × 1 convolutions on the shrunk wavelet domain.
We show that this approach surpasses aggressive quantization using the same bit rate at minimal additional costs of the transform.
R EPRODUCIBILITY S TATEMENT
The code for running and reproducing the experiments reported in this paper is attached as supplementary material.
A BTS COMPRESSION RESULTS
[Figure]
B A N OTE ON 1 × 1 - CONVOLUTIONS
In the case when a certain CNN use 3 × 3 convolution only, one can split it to two convolution, a depthwise- 3 × 3 and a 1 × 1 (Wang et al., 2017; Vanhoucke, 2014).
The depthwise conv involves with C in · 3 · 3 · N W · N H MAC operations (assuming no strides), while the 1 × 1 conv includes
C in · C out · N W · N H MAC operations ( C out / 9 times more expensive than depthwise).
Meaning, for a large enough C out , the 3 × 3 convolution has about 8-9 times more MAC operations than the depthwise- 3 × 3 convolution and 1 × 1 convolution.
Some models, such as the ones referenced in section 4, are deﬁned based on that concept.
For example, MobilenetV2 consists of residual blocks that perform 1 × 1 , depthwise- 3 × 3 , and an additional 1 × 1 , and for an image input of size 1024 × 2048 ( e.g. cityscapes), the 1 × 1 -conv has a MAC count of 18,022M, while the 3 × 3 -conv has a MAC count of 1,056M (a full breakdown is provided in Table 3).
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
InvRes6 conv         56,623,
InvRes6 conv         201,326,
InvRes7 conv         206,069,
InvRes7 conv         14,155,
InvRes7 conv         100,663,
InvRes8 conv         210,862,
InvRes8 conv         28,311,
InvRes8 conv         201,326,
InvRes9 conv         210,862,
InvRes9 conv         28,311,
InvRes9 conv         201,326,
InvRes10 conv         210,862,
InvRes10 conv         28,311,
InvRes10 conv         201,326,
InvRes11 conv         210,862,
InvRes11 conv         28,311,
InvRes11 conv         301,989,
InvRes12 conv         474,439,
InvRes12 conv         42,467,
InvRes12 conv         452,984,
InvRes13 conv         474,439,
InvRes13 conv         42,467,
InvRes13 conv         452,984,
InvRes14 conv         474,439,
InvRes14 conv         42,467,
InvRes14 conv         754,974,
InvRes15 conv         1,378,713,
InvRes15 conv         70,778,
InvRes15 conv         1,258,291,
InvRes16 conv         1,378,713,
InvRes16 conv         70,778,
InvRes16 conv         1,258,291,
[Table]