L IFT P OOL : B IDIRECTIONAL C ONV N ET P OOLING
Anonymous authors
Paper under double-blind review
A BSTRACT
Pooling is a critical operation in convolutional neural networks for increasing receptive ﬁelds and improving robustness to input variations.
Most existing pooling operations downsample the feature maps, which is a lossy process.
Moreover, they are not invertible: upsampling a downscaled feature map can not recover the lost information in the downsampling.
By adopting the philosophy of the classical Lifting Scheme from signal processing, we propose LiftPool for bidirectional pooling layers, including LiftDownPool and LiftUpPool.
LiftDownPool decomposes a feature map into various downsized sub-bands, each of which contains correlated structures or details of the feature map.
As the pooling function in LiftDownPool is perfectly invertible, by performing LiftDownPool backwards, a corresponding up-pooling layer LiftUpPool is able to generate a reﬁned upsampled feature map using the detail sub-bands, which is useful for image-toimage translation challenges.
Experiments show the proposed methods achieve better results on image classiﬁcation and semantic segmentation, using various backbones.
Moreover, LiftDownPool offers better robustness to input corruptions and perturbations.
I NTRODUCTION
Spatial pooling has been a critical ConvNet operation since its inception (Fukushima, 1979; LeCun et al., 1990; Krizhevsky et al., 2012;
He et al., 2016; Chen et al., 2018).
It is crucial that a pooling layer maintains the most important activations for the network’s discriminability (Saeedan et al., 2018; Boureau et al., 2010).
Several simple operations, such as average pooling or max pooling, have been explored for aggregating features in a local area.
Springenberg et al.
(2015) employ a convolutional layer with an increased stride to replace a pooling layer, which is equivalent to downsampling.
While effective and efﬁcient, simply using the average or maximum activation may ignore local structures.
In addition, as these functions are not invertible, upsampling the downscaled feature maps can not recover the lost information.
Different from existing pooling operations, we propose in this paper a bidirectional pooling called LiftPool, including LiftDownPool which preserves correlated structures in feature maps as well as details when downsizing the feature maps, and LiftUpPool for generating ﬁner upsampled feature maps.
LiftPool is inspired by the classical Lifting Scheme (Sweldens, 1998) from signal processing, which is commonly used for information compression (Pesquet-Popescu & Bottreau, 2001), reconstruction (Dogiwal et al., 2014) and denoising (Wu et al., 2004).
The Lifting Scheme decomposes an input signal into various sub-bands with downscaled size and this process is perfectly invertible.
Applying the idea of Lifting Scheme, LiftDownPool factorizes an input feature map into several downsized spatial sub-bands with different correlation structures.
As shown in Figure 1, for an image feature map, the LL sub-band is an approximation removing several details.
The LH , HL and HH represent details along horizontal, vertical and diagonal directions.
LiftDownPool respects preserving any sub-band(s) as the pooled result.
Moreover, due to the invertibility of the pooling function, LiftUpPool is introduced for upsampling feature maps.
Upsampling a feature map is more challenging as seen for the MaxUpPool (Badrinarayanan et al., 2017), which generates an output with many ‘holes’ (shown in Figure 1).
LiftUpPool utilizes the recorded details to recover a reﬁned output by performing LiftDownPool backwards.
We analyze the proposed LiftPool from several viewpoints.
LiftDownPool allows a ﬂexible choice for any sub-band(s) as the pooled result.
It outperforms baselines on image classiﬁcation with various ConvNet backbones.
LiftDownPool also presents better stability to corruptions and pertur-
[Figure]
bations of inputs.
By performing LiftDownPool backwards, LiftUpPool generates a reﬁned upsampling feature map for semantic segmentation.
M ETHODS
The down-pooling operator is formulated as minimizing the information loss caused by downsizing feature maps, as in image downscaling by Saeedan et al.
; Kobayashi (2019a).
The Lifting
Scheme (Sweldens, 1998) naturally matches the problem.
The Lifting Scheme was originally designed to exploit the correlated structures present in signals to build a downsized approximation and several detail sub-bands in the spatial domain (Daubechies & Sweldens, 1998).
The inverse transform is realizable and always provides a perfect reconstruction of the input.
LiftPool is derived from the Lifting Scheme for bidirectional pooling layers.
2. L IFT D OWN P OOL
Taking a one-dimension (1D) signal as an example, LiftDownPool decomposes a given signal x =[ x 1 , x 2 , x 3 , ..., x n ] , x n ∈ R into a downscaled approximation signal s and a difference signal d by, s , d = F ( x ) .
(1) where F ( · )=
f update ◦ f predict ◦ f split ( · ) , consists of three functions: split (downsample), predict and update.
And ◦ indicates the function composition operator.
The LiftDownPool-1D is illustrated in
Figure 2(a).
Speciﬁcally,
Split f split : x (cid:55)→ ( x e , x o ) .
The given signal x is split into two disjoint sets x e =[ x 2 , x 4 , ..., x 2 k ] with even indices and x o =[ x 1 , x 3 , ..., x 2 k +1 ] with odd indices.
The two sets are typically highly correlated.
Predict f predict : ( x e , x o ) (cid:55)→ d .
Given one set e.g . x e , another set x o is able to be predicted by a predictor P ( · ) .
The predictor is not required to be precise, so the difference, or detail with high-pass coefﬁcients d is deﬁned as:
[Equation] [Equation]
[Figure]
[Equation]
LiftDownPool
1D generate details d and an approximation s .
(b) LiftUpPool-1D.
By running
LiftDownPool backwards, x e and x o are generated from s and d , and then merged to x .
Update f update : ( x e , d ) (cid:55)→ s .
Taking x e as an approximation of x causes a serious aliasing because x e is simply downsampled from x .
Particularly, the running average of x e is not the same as that of x .
To correct it, a smoothed version s is generated by adding U ( d ) to x e :
[Equation]
The update procedure is equivalent to applying a low-pass ﬁlter to x .
Thus, s with low-pass coefﬁcients is taken as an approximation of the original signal.
The classiﬁc Lifting Scheme method applies pre-deﬁned low-pass ﬁlters and high-pass ﬁlters to decompose an image into four sub-bands.
However, pre-designing ﬁlters in P ( · ) and U ( · ) is difﬁcult (Zheng et al., 2010).
Earlier, Zheng et al.
(2010) proposed to optimize these ﬁlters by a back-propagation network.
All functions in LiftDownPool are differentiable.
P ( · ) and U ( · ) are able to be simply implemented by convolution operators followed by non-linear ReLU operators (Glorot et al., 2011).
Speciﬁcally, we design P ( · ) and U ( · ) as:
[Equation]
() ◦ Conv(kernel= K ,stride=1,groups= C ) (4)
U (
[Equation]
= Conv(kernel=1,stride=1,groups= C ) ◦ ReLU() ◦
Conv(kernel= K ,stride=1,groups= C ) (5)
Where K is the kernel size and C is the number of input channels.
We prefer to learn the ﬁlters in
P ( · ) and U ( · ) with deep neural networks in an end-to-end fashion.
To that end, two constraints need to be added to the ﬁnal loss function.
s is the downsized approximation of x .
As s is updated from x e according to Eq 3, s is essentially close to x e .
Thus, s is naturally required to be close to x o as well.
Therefore, one constraint term c u is for minimising the L2-norm distance between s and x o .
With Eq 3,
[Equation]
The other constraint term c p is for minimising the detail d , with Eq 2,
[Equation]
The total loss is:
[Equation]
Where L task is the loss for a speciﬁc task, like classiﬁcation or semantic segmentation loss.
Our experiments show that the two terms bring good regularization to the model.
LiftDownPool-2D is easily decomposed into several 1D LiftDownPool operators.
Following the standard Lifting Scheme, we ﬁrst perform a LiftDownPool-1D along the horizontal direction to obtain an approximation part s (low frequency in the horizontal direction) and a difference part d (high frequency in the horizontal direction).
Then, for each of the two parts, we apply a LiftDownPool1D along the vertical direction.
By doing so, s is further decomposed into LL (low frequency in vertical and horizontal directions) and LH (low frequency in the vertical direction and high frequency in the horizontal direction).
d is further decomposed into HL (high frequency in the vertical direction and low frequency in the horizontal direction) and HH (high frequency in vertical and horizontal directions).
We can ﬂexibly choose sub-band(s) for down-pooling and keep the other sub-band(s) for reversing the operation.
Naturally, LiftDownPool-1D can be generalized further for any n -dimensional signal.
In Figure 3, we show several feature maps from the ﬁrst LiftDownPool layer based on VGG13.
LL has smoothed features with less details.
LH , HL and HH capture the details along horizontal, vertical and diagonal directions.
[Equation]
[Figure]
Discussion MaxPool is usually formulated as ﬁrst performing Max and then downsampling: MaxPool k,s = downsample s ◦
Max k
(Zhang, 2019).
By contrast, LiftDownPool is:
LiftDownPool k,s = update k ◦ predict k ◦ downsample s .
First downsampling and then performing two lifting steps (prediction and updating) helps anti-aliasing.
A simple analysis is provided in the Appendix.
As shown in Figure 1, LiftDownPool keeps more structured information and better reduces aliasing then MaxPool.
2. L IFT U P P OOL
LiftPool inherits the invertibility of the Lifting Scheme.
Taking the 1D signal as an example, LiftUpPool generates an upsampled signal x from s , d by:
[Equation]
where G ( · )=
f merge ◦ f predict ◦ f update ( · ) , including functions: update, predict and merge.
Speciﬁcally, s , d (cid:55)→ x e , d
(cid:55)→ x e , x o (cid:55)→ x is realized by:
[Equation]
[Equation] [Equation]
Discussion Up-pooling has been used in image-to-image translation tasks such as semantic segmentation (Chen et al., 2017), super-resolution (Shi et al., 2016), and image colorization (Zhao et al., 2018).
It is generally used in encoder-decoder networks such as SegNet (Badrinarayanan et al., 2017) and UNet (Ronneberger et al., 2015).
However, most existing pooling functions are not invertible.
Taking MaxPool as the baseline, it is required to record the maximum indices during max pooling.
For simplicity, we use the down-pooled results as inputs to the up-pooling in Figure 1.
When performing MaxUpPool, the values of the input feature maps are directly ﬁlled on the corresponding maximum indices of the output and other indices will be given zeros.
By doing so, the output looks sparse and loses much of the structured information, which is harmful for generating good-resolution outputs.
LiftUpPool performing an inverse transformation of LiftDownPool, is able to produce ﬁner outputs by using the multiphase sub-bands.
R ELATED W ORK
Taking the average over a feature map region was the pooling mechanism of choice in the Neocognitron (Fukushima, 1979; 1980) and LeNet (LeCun et al., 1990).
Average pooling is equivalent to blurred-downsampling.
Max pooling later proved even more effective (Scherer et al., 2010) and became popular in deep ConvNets.
Yet, averaging activations or picking the maximum activations causes loss of details.
Zeiler & Fergus (2013) and Zhai et al.
(2017) introduced a stochastic process to pooling and downsampling, respectively, for a better regularization.
Lee et al.
(2016) mixed
AveragePool and MaxPool by a gated mask to adapt to complex and variable input patterns.
Saeedan et al.
(2018) introduced detail-preserving pooling (DPP) for maintaining structured details.
By contrast, Zhang (2019) proposed a BlurPool by applying a low-pass ﬁlter, which removes details.
Interestingly, both methods improved image classiﬁcation, indicating that (empirically) determining the best pooling strategy is beneﬁcial (Saeedan et al., 2018).
Williams & Li (2018) introduced the wavelet transform into pooling for reducing jagged edges and other artifacts.
Rippel et al.
(2015) suggested pooling in the frequency domain, which enabled ﬂexibility in the choice of the pooling output dimensionality.
Pooling based on a probabilistic model was proposed in (Kobayashi, 2019a) and (Kobayashi, 2019b).
Kobayashi (2019a)
ﬁrst used a Gaussian distribution to model the local activations and then aggregates the activations into the two statistics of mean and standard deviation.
Kobayashi (2019b) estimated parameters from global statistics in the input feature map, to ﬂexibly represent various pooling types.
Our proposed LiftDownPool decomposes the input feature map into a downsized approximation and several details.
It is ﬂexible to choose any sub-band(s) as pooled result.
While existing pooling functions are not invertible, our proposed LiftPool is able to perform both down-pooling and up-pooling.
Previously, MaxUpPool (Badrinarayanan et al., 2017) was introduced for semantic segmentation.
As the max pooling function is not invertible, the lost details can not be recovered during up-pooling.
Hence, the output suffers from aliasing.
Although adding a
BlurPool to MaxUpPool may help to reduce the aliasing (Zhang, 2019), several details are still lost.
LiftUpPool, performing the LiftDownPool functions backwards, is capable of producing a reﬁned high-resolution output with the help of the details sub-bands.
Earlier, Zheng et al.
(2010) introduce back-propagation for Lifting Scheme to perform nonlinear wavelet decomposition.
They propose an update-ﬁrst Lifting Scheme and use back-propagation to replace the Updater and Predictor in the Lifting Scheme.
By this way, they realize a backpropagation neural network in lifting steps for signal processing.
There is no pooling layer used.
We develop down-pooling and up-pooling layers by leveraging the idea of Lifting Scheme for image processing.
We utilize convolution layers and ReLU layers to implement the Updater and Predictor, which are optimized end-to-end with the deep neural networks.
Our developed pooling layers are easily plugged into various backbones.
Recently, Rodriguez et al.
(2020) introduce the Lifting
Scheme for multiresolution analysis in a network.
Speciﬁcally, they develop an adaptive wavelet network by stacking several convolution layers and Lifting Scheme layers.
They focus on an interpretable network by integrating multiresolution analysis, rather than pooling.
Our paper aims at developing a pooling layer by utilizing the lifting steps.
We develop a down-pooling which constructs various sub-bands with different information, and an up-pooling which generates reﬁned upsampled feature maps.
E XPERIMENTS 4. C ONV N ET T ESTBEDS
Image Classiﬁcation We ﬁrst verify the proposed LiftDownPool for image classiﬁcation on
CIFAR-100 (Krizhevsky & Hinton, 2009) with 32 × 32 low-resolution images.
CIFAR-100 hasclasses with 600 images each.
There are 500 training images and 100 testing images per class.
A
VGG13 (Simonyan & Zisserman, 2015) network is trained on this dataset.
For experiments conducted on CIFAR-100, we repeat each experiment three times with different initial random seeds during training and report the averaged error rate with the standard deviation.
We also report on ImageNet (Deng et al., 2009) with 1.2M training images and 5000 validation images for 1000 classes.
We plug the LiftDownPool into several popular ConvNet backbonesto verify its generalizability for image classiﬁcation.
We replace the local pooling layers by LiftDownPool in all the networks.
Error rate is utilized as the evaluation metric.
Speciﬁc training settings are seen in Appendix.
Semantic Segmentation We also test the LiftDownPool and LiftUpPool for semantic segmentation on PASCAL-VOC12 (Everingham et al., 2010), which contains 20 foreground object classes and one background class.
An augmented version with 10582 training images and 1449 validation images is used.
We consider SegNet (Badrinarayanan et al., 2017) with VGG13 and
DeeplabV3Plus (Chen et al., 2018) with ResNet50 as ConvNets for semantic segmentation.
The performance is measured in terms of pixel mean-intersection-over-union ( mIoU ) across the 21 classes.
Code and ConvNets for all experiments will be released.
[Table] [Table]
[Table] [Table]
WaveletPool (Williams & Li, 2018)
[Table] [Table]
[Table]
BlurPool (cid:63) (Zhang, 2019)
[Table] [Table]
[Table] [Table]
DPP (cid:63) (Saeedan et al., 2018)
[Table]
SpectralPool (Rippel et al., 2015)
[Table]
GatedPool (Lee et al., 2016)
[Table]
MixedPool (Lee et al., 2016)
[Table]
(Kobayashi, 2019b)
[Table]
Kobayashi
[Table] [Table]
4. A BLATION S TUDY
Flexibility We ﬁrst test VGG13 on CIFAR-100.
Different from previous pooling methods, LiftDownPool generates four sub-bands, each of which contains a different type of information.
LiftDownPool allows to ﬂexibly choose which sub-band(s) to keep as the ﬁnal pooled results.
In Table 1, we show the Top-1 error rate for the classiﬁcation based on different sub-bands.
Interestingly, it is observed that vertical details contribute more for image classiﬁcation.
Low-pass coefﬁcients and high-pass coefﬁcients along horizontal direction get similar error rate.
Whether the two spatial dimensions should be treated equally, we leave for our future work.
To realize a less lossy pooling, we combine all the sub-bands by summing them up with almost no additional compute cost.
Such a pooling signiﬁcantly improves the results.
In addition, the constrains c u and c p help to decrease the error rate.
Moreover, seen from Table 1 and Table 3, we further conclude LiftDownPool outperforms other baselines even based on any single sub-band.
We believe the learned LiftDownPool provides an effective regularization to the model.
[Table]
LiftDownPool achieves better results with only a few more parameters than MaxPool.
[Table]
Effectiveness Table 2 ablates the performance when varying kernel sizes for the ﬁlters in P ( · ) and
U ( · ) .
A larger kernel size, covering more local information, performs slightly better.
When kernel size equals 7, it brings more computations but no performance gain.
Unless speciﬁed otherwise, we use for all experiments from now on a kernel size of 5 and we sum up all the sub-bands.
We also compare our LiftDownPool with the commonly-used MaxPool, AveragePool, as well as the convolutional layer with stride 2 (Springenberg et al., 2015), which is called Skip by Kobayashi (2019a).
Seen from Table 3, LiftDownPool outperforms other pooling methods on CIFAR-100.
Generalizability We apply LiftDownPool to several backbones including ResNet18,
ResNet50 (He et al., 2016) and MobileNet-V2 (Sandler et al., 2018) on ImageNet.
In Table 4, LiftDownPool has 2% lower Top-1 error rate than MaxPool and AveragePool.
While combining MaxPool and AveragePool in a Gated (Lee et al., 2016) or Mixed (Lee et al., 2016) fashion, still has a 1% gap with LiftDownPool.
Gauss (Kobayashi, 2019a) and GFGP (Kobayashi, 2019b) are comparable to LiftDownPool with ResNet50, but not with lighter networks.
Compared to Spectral pooling (Rippel et al., 2015) and Wavelet pooling (Williams & Li, 2018), which are applied in the frequency or space-frequency domain, LiftDownPool offers an advantage by learning correlated structures and details in the spatial domain.
Compared to DPP (Saeedan et al., 2018), which preserves details, and BlurPool (Zhang, 2019), smoothing feature maps by a low-pass ﬁlter, our LiftDownPool retains all sub-bands which proves to be more powerful for image classiﬁcation.
Stochastic approaches like S 3 Pool obtain poor results on the large-scale dataset because randomness in pooling hampers network training, as earlier observed by Kobayashi (2019b).
To conclude,
LiftDownPool performs better no matter what backbone is used.
Parameter Efﬁciency.
For all pooling layers in one network, we use the same kernel size in LiftPool.
Besides the two hyper-parameters: the kernel size K and the balancing weights for the losses, there are no extra hyper-parameters introduced.
For the trainable parameters, recall P or U has a 1D convolution, so each has C × K + C parameters where K is the kernel size of the ﬁrst 1D convolution and C is the number of the input channels.
A 2D LiftPool has 3 × ( C × K + C ) parameters.
We compare our LiftDownPool to two other parameterized pooling methods using ResNet50 on ImageNet:
GFGP and GaussPool.
We achieve a lower error rate compared to GFGP and GaussPool with less parameters.
With only 3K more parameters than MaxPool, we reduce the error rate by 1.9%.
Our performance boost is due to the LiftPool scheme, not the added capacity.
[Figure] [Figure]
4. S TABILITY A NALYSIS
Out-of-distribution Robustness A good down-pooling method is expected to be stable to perturbations and noise.
Following Zhang (2019), we test the robustness of LiftDownPool to corruptions on ImageNet-C and stability to perturbations on ImageNet-P using ResNet50.
Both datasets come from Hendrycks & Dietterich (2019).
We report the mean Corruption Error ( mCE ) and mean
Flip Rate ( mFR ) for the two tasks, with both unnormalized raw values and normalized values by
AlexNet’s mCE and mFR following Hendrycks & Dietterich (2019).
We evaluate all the pooling methods on the two datasets.
We ﬁnd SpectralPool, MixedPool and GFGP do not work on the two datasets.
Thus, we do not report them in Table 5.
From Table 5, LiftDownPool effectively reduces raw mCE compared to the baselines.
We show CE for each corruption type for further analysis in Figure 9 in the Appendix.
LiftDownPool enables robustness to both “high-frequency” corruptions, such as noise or spatter, and “low-frequency” corruptions, like blur and jpeg compression.
We believe LiftDownPool beneﬁts from the mechanism that all sub-bands are used.
A similar conclusion is obtained for robustness to perturbations on
ImageNet-P from Table 5 and Figure 9 in the Appendix.
ImageNet-P contains short video clips of a single image with small perturbations added.
Such perturbations are generated by several types of noise, blur, geometric changes, and simulated weather conditions.
The metric FR measures how often the Top-1 classiﬁcation changes in consecutive frames.
It is designed for testing a model’s stability under small deformations.
Again, LiftDownPool achieves lower FR for most perturbations.
Shift Robustness We then test the shift-invariance of our model.
Following Zhang (2019), we use classiﬁcation consistency to measure the shift-invariance.
It represents how often the network outputs the same classiﬁcation, given the same image with two different shifts.
We test the models with varying backbones trained on ImageNet.
In Figure 4, LiftDownPool boosts classiﬁcation accuracy as well as consistency no matter which backbone is used.
Besides, we have other interesting ﬁndings.
The deeper ResNet50 network has more stable shift-invariance.
Various pooling methods including MaxPool, Skip, AveragePool, do not make signiﬁcant difference on consistency.
How-
[Table]
LiftUpPool performs much better with only a few more parameters.
[Table]
ever, a lighter ResNet18 network is inﬂuenced much by the pooling method.
LiftDownPool brings more than 10% improvement on consistency using ResNet18.
We leave for future work how the depth of the network affects the shift-invariance of the network itself.
4. R ESULTS FOR S EMANTIC S EGMENTATION
LiftUpPool for Semantic Segmentation LiftDownPool functions are invertible as described in
Eq 10 and Eq 11.
It naturally beneﬁts a corresponding up-pooling operation, which is popularly used in Encoder-Decoder networks for image-to-image translation tasks.
Usually, the Encoder downsizes feature maps layer by layer to generate a high-level embedding for understanding the image.
Then the Decoder needs to translate the embedding with a tiny spatial size to a required map with the same spatial size as the original input image.
Interpreting details is pivotal for producing high-resolution outputs.
We replace all down-pooling and up-pooling layers with LiftDownPool and LiftUpPool in SegNet for semantic segmentation on PASCAL-VOC12.
For LiftDownPool we only keep the
LL sub
band.
For LiftUpPool, the detail-preserving sub-bands LH , HL and HH are used for generating upsampled feature maps.
MaxUpPool is taken as the baseline.
We also test MaxUpPool followed by a BlurPool (Zhang, 2019), which is expected to help anti-aliasing.
Table 7 reveals
LiftUpPool improves over the baselines with a considerable margin.
As illustrated in Figure 1,
MaxUpPool is unable to compensate for the lost details.
Although BlurPool helps smoothing local areas, it can only provide a small improvement.
As LiftUpPool is capable of reﬁning the feature map by fusing it with details, it is beneﬁcial for per-pixel prediction tasks like semantic segmentation.
We show several examples for semantic segmentation in Figure 5.
LiftUpPool is observed more precise on details and edges.
We also show the feature maps per predicted class in the Appendix.
Semantic Segmentation with DeepLabV3Plus As discussed, LiftDownPool helps to lift ConvNets on accuracy and stability for image classiﬁcation.
ImageNet-trained ConvNets often serve as the backbones for downstream tuning, it is expected to transfer the nature of LiftDownPool to other tasks.
We still consider semantic segmentation as our example.
We leverage the state-of-the-art
DeeplabV3Plus-ResNet50 (Chen et al., 2018).
The input image has size 512 × 512 .
The output feature map of the encoder is 32 × 32 .
The decoder upsamples the feature map to 128 × 128 and concatenates them with the low-level feature map for the ﬁnal pixel-level classiﬁcation.
Same as before, all local pooling layers are replaced by LiftDownPool.
We use the pre-trained weights for image classiﬁcation to initialize the corresponding model.
As shown in Table 8, LiftDownPool outperforms all the baselines with considerable gaps.
C ONCLUSION
Applying classical signal processing theory to modern deep neural networks, we propose a novel pooling method: LiftPool.
LiftPool is able to perform both down-pooling and up-pooling.
LiftDownPool improves both accuracy and robustness for image classiﬁcation.
LiftUpPool, generating reﬁned upsampling feature maps, outperforms MaxUpPool by a considerable margin on semantic segmentation.
Future work may focus on applying LiftPool to ﬁne-grained image classiﬁcation or super-resolution tasks, which have high requirements for details.
A A PPENDIX
We show additional analysis and results for robustness and semantic segmentation in this part.
LiftDownPool vs. MaxPool
We make a schematic diagram in Figure 6 for further illustrating the difference between MaxPool and LiftDownPool, MaxUpPool and LiftUpPool.
Taking kernel size 2, stride 2 as an example, MaxPool selects the maximum activations in a local neighbourhood.
Hence, it looses 75% information.
The lost details could be important for image recognition.
LiftDownPool decomposes a feature map into LL , LH , HL and HH .
LL containing low-pass coefﬁcients is an approximation of the input.
It is designed for capturing correlated structures of the input.
Other sub-bands contain detail coefﬁcients along different directions.
The pooling is implemented by summing up all the sub-bands.
The ﬁnal pooled result containing both the approximation and details is expected to be more effective for image classiﬁcation.
LiftUpPool vs. MaxUpPool
The pooling function in MaxPool is not invertible.
MaxPool records the maximum indices for performing the corresponding MaxUpPool.
MaxUpPool takes the activations at the corresponding positions for the recorded maximum indices on the output.
For other indices, there will be zeros.
The ﬁnal upsampled output has many ‘holes’.
On the contrary, the pooling functions in LiftDownPool is invertible.
Leveraging the property, LiftUpPool, performing a
LiftDownPool backwards, is able to generate a reﬁned output from an input, including the recorded details.
Experiment Settings For the VGG13 (Simonyan & Zisserman, 2015) network trained on CIFAR100, it is optimized by SGD with a batch size of 100, weight decay of 0.0005, momentum of 0.9.
The learning rate starts from 0.1 and is reduced by multiplying 0.1 after 80 and 120 epochs for a total of 160 epochs.
We train ResNets for 100 epochs and MobileNet for 150 epochs on ImageNet, following the standard training recipe from the public PyTorch (Paszke et al., 2017) repository.
We set λ u and λ p equal to 0.01 for all the networks.
High-resolution Feature Maps Visualization
By using ResNet50 with input size 224 × 224 , we extract the feature maps of an image from the ﬁrst pooling layer.
We show the high-resolution feature maps in Figure 7.
We only show the LL sub-band from LiftDownPool.
Compared to MaxPool,
LiftDownPool better maintains the local structure.
Anti-aliasing LiftDownPool effectively reduces aliasing following the Lifting Scheme (Sweldens, 1998) compared to naive downsizing.
Figure 8(b) provides a simple illustration of LiftDownPool.
The dash line is an original signal x .
According to Eq 1, the predictor P ( · ) for the odd part x 2 k +
[Figure]
could easily take the average of its two even neighbors.
[Equation]
Thus, if x is linear in a local area, the detail d k is zero.
The prediction step takes care of some of the spatial correlation.
If an approximation s of the original signal x is simply taken from the even part x e , it is really downsizing the signal shaped in the red line.
There is serious aliasing.
The running average of x e is not the same as that of the original signal x .
The updater U ( · ) in Eq 2 is for correcting this by replacing x e with smoothed values s .
Speciﬁcally, U ( · ) restores the correct running average and thus reduces aliasing.
[Equation]
As shown in Figure 8, d k is the difference between the odd sample x 2 k +1 and the average of two even samples.
This causes a loss d k / 2 in the area with the red shade.
To preserve the running average, this area is redistributed to the two neighbouring even samples x 2 k and x 2 k +2 , which shapes a coarser piecewise linear signal s in the solid line.
The signal after LiftDownPool, drawn in solid line, covers the same area with the original signal dash line.
LiftDownPool reduces aliasing compared to the downsizing drawn in the solid line in (a).
Out-of-distribution Robustness
We show the robustness of pooling methods per kind of corruption and perturbation in Figure 9.
Corruption Error ( CE ) is the metric of the robustness to corruptions on ImageNet-C. And Flip Rate ( FR ) is reported for the robustness to perturbation on ImageNet-P.
Following (Hendrycks & Dietterich, 2019), we report both unnormalized raw values and normalized values by AlexNet’s CE and FR .
Lower values are better.
As seen in Figure 9(a) and (c), LiftDownPool gets the lowest CE for most of the “high frequency” corruptions including gaussian noise and spatter, as well as the “low frequency” corruptions such as motion blur, zoom blur.
In Figure 9(b) and (d), it clearly shows LiftDownPool has less sensitivity to most of the perturbations such as speckle noise and gaussian blur.
Visualization of Up-pooling In Figure 10, we show the feature map for each predicted category from the last layer of SegNet using varying up-pooling methods.
Using MaxUpPool, the feature maps look noisy and less continuous due to the fact that MaxUpPool generates the output with many ‘zeros’, where there is no information.
By applying a BlurPool following the MaxUpPool, the feature maps turn more smooth, while still with less details.
LiftUpPool, beneﬁting from the recorded details during LiftDownPool, produces ﬁner feature maps for each category.
It has smooth edges, continuous segmentation maps and less aliasing.
[Figure]
[Equation]
[Figure]
[Figure] [Figure]