F REQUENCY P OOLING : S HIFT - EQUIVALENT AND A NTI -A LIASING D OWN S AMPLING
Anonymous authors
Paper under double-blind review
A BSTRACT
Convolutional layer utilizes the shift-equivalent prior of images which makes it a great success for image processing.
However, commonly used down sampling methods in convolutional neural networks (CNNs), such as max-pooling, averagepooling, and strided-convolution, are not shift-equivalent.
This destroys the shiftequivalent property of CNNs and degrades their performance.
In this paper, we propose a novel pooling method which is strict shift equivalent and anti-aliasing in theory.
This is achieved by (inverse)
Discrete Fourier Transform and we call our method frequency pooling.
Experiments on image classiﬁcations show that frequency pooling improves accuracy and robustness w.r.t shifts of CNNs.
I NTRODUCTION
Convolutional neural networks (CNNs) have achieved great success on image processing Goodfellow et al.
natural language processing Yin et al.
(2017), game playing Mnih et al.
(2013) and so on.
One of the reasons is that convolutions utilize the shift-equivalent prior of signals.
Modern
CNNs include not only convolutional layers but also down sampling or pooling layers.
As an important part of CNNs, pooling layers are used to reduce spatial resolution of feature maps, aggregate spatial information, and reduce computational and memory cost.
Based on classical Nyquist sampling theorem Nyquist (1928), the sampling rate must be at least as twice as the highest frequency of a signal.
Otherwise, frequency aliasing will appear, i.e. highfrequencies of the signal alias into low-frequencies.
To anti-alias, a traditional solution is that applying low-pass ﬁlter to the signal before down sampling it.
Following the spirit of blurred down sampling, early CNNs use average pooling to achieve down sampling Lecun et al.
Later, empirical evidence suggests max-pooling Scherer et al.
and strided-convolutions Long et al.
(2015) provide better performance.
They are widely used in CNNs but they don’t consider about anti-aliasing.
Shift-equivalent is another expected property of pooling.
Because shift-nonequivalent poolings destroy the shift-equivalent of CNNs and thus the shift-equivalent prior of signals is not fully utilized.
Unfortunately, they are not shift-equivalent.
Worse, small shifts in the input can drastically change the output when stacking multiple max-pooling or strided convolutions Engstrom et al.
(2017); Azulay & Weiss (2018); Zhang (2019).
Recently, Zhang (2019) propose anti-aliasing pooling (AA-pooling) by low-pass ﬁltering before down sampling.
They observe increased accuracy and better generalization on image classiﬁcation when low-pass ﬁltering is integrated correctly.
Speciﬁcally, they decompose a pooling operator with down sampling factor s into two parts: a pooling operator with factor 1 and a blur ﬁlter with factor s .
Although AA-pooling reduces the aliasing effects and makes the outputs more stable w.r.t input shifts, it is not strict shift-equivalent in theory.
In this paper, we propose a novel pooling which is strict shift equivalent in theory.
We ﬁrst transform a signal or image into frequency domain via Discrete Fourier Transform (DFT).
Then we only retain its low frequencies, i.e. the frequencies which are smaller than half of resolution of the down sampled signal.
Finally, we transform the low frequencies back into time domain via inverse DFT.
We call our method frequency pooling (F-pooling).
We summarize our contributions as followings:
[Figure]
• We formally deﬁne shift-equivalent of functions which contain down sampling operations.
A suitable up sampling operation U must be involved in the deﬁnition.
Without it, the deﬁnition is ill-posed.
• We prove that F-pooling is the optimal anti-aliasing down sampling operation given U .
We also prove that F-pooling is shift equivalent.
To best of our knowledge, F-pooling is the ﬁrst pooling method which has those properties.
• Experiments on CIFAR-100 and a subset of ImageNet demonstrate that F-pooling remarkably increases accuracy and robustness w.r.t shifts of commonly-used network architectures.
F-pooling has other two properties: 1) it adapts to variable input size and variable down sampling factor; 2) it is a global operation while previous pooling methods are local operations.
However, they are not the main point of this paper.
Instead, we focus on shift-equivalent and anti-aliasing.
M ETHOD
In this section, we ﬁrst deﬁne shift-equivalent for CNNs formally.
Then we describe F-pooling in detail and prove its properties.
Finally we discuss our implementation and analyze its computational complexity.
2. W HAT IS SHIFT - EQUIVALENT
Denote X as an h 0 × w 0 × c tensor where h 0 , w 0 , and c are the height, width, and number of channels of X respectively.
Denote Y as an h 1 × w 1 × c tensor.
We suppose h 0 > h 1 and w 0 > w 1 .
M : X → Y is a pooling operation and U : Y → X is a up sampling operation.
We say M is shift-equivalent if and only if
[Equation]
[Figure]
That is, if there is a suitable up sampling operation U which makes UM and Shift (cid:52) h, (cid:52) w commutable, then M is shift-equivalent.
The shift operation is required to be circular or periodic.
When a shifted element hits the edge, it rolls to other side.
[Equation]
where % is the modulus function.
Our deﬁnition of shift-equivalent is similar with the one in Zhang (2019).
The difference is that we introduce an up sampling U to make this deﬁnition more strict.
Without introducing U , the deﬁnition of shift-equivalent for pooling is ill-posed .
Suppose M down samples signals by a factor s .
Without introducing U , one may deﬁne shift-equivalent as follows:
[Equation]
However, Shift (cid:52) h/s, (cid:52) w/s is not operable for discrete signals when (cid:52) h % s (cid:54) = 0 or (cid:52)
h % s (cid:54) = 0 .
To make the shifted element which is not on lattice operable, one must interpolate or up sample the down sampled signals.
This leads to the deﬁnition in equation 1.
As in equation 1, to make M shift-equivalent, one must ﬁnd the corresponding up sampling operation U .
For a given M , the ﬁrst line in equation 1 may hold for some up sampling operations.
But it may not hold for other up sampling operations.
2. F- POOLING
The basic idea of F-pooling is removing the high frequencies of signals and reconstructing the signals only using the low frequencies.
In this paper, high frequencies mean the frequencies which are beyond Nyquist frequency , i.e. half of signal resolution.
And low frequencies mean the frequencies which are not higher than Nyquist frequency .
To remove high frequencies, we ﬁrst transform signals into frequency domain via DFT.
Then we remove high frequencies and retain the low frequencies.
Finally, we transform the low frequencies back to time domain via inverse DFT (IDFT).
Figuregives an illustration of the forward process of F-pooling.
Since 2D (inverse) DFT can be decomposed into two 1D (inverse) DFT, we provide the formal representation of F-pooling in 1D case.
Denote x ∈ R N as input signal and y ∈ C M .
Without loss of generality, we suppose M is even.
F N ∈ C N × N is the so-called DFT-matrix:
[Equation]

[Equation]
where ω N = e − 2 πi/N .
F ∗ N is the inverse DFT-matrix.
By deﬁnition, 1 N F N F ∗ N =
I where I is identity matrix.
We denote T µ as a operation which selects the ﬁrst µ rows and the last µ rows of a matrix.
[Equation]
That is we select the basis of frequencies ranged in [ − µ, µ ) , due to periodicity of DFT.
When applying T µ to a signal, we obtain its lowest µ frequencies.
Then the function of F-pooling for 1D signals is represented as:
[Equation]
where P ∈ C M × N is the transform matrix of F-pooling.
Now we formalize F-pooling for CNNs.
Recall that X is an h 0 × w 0 × c tensor
and Y is an h 1 × w 1 × c tensor.
We apply F-pooling to each channel of X .
[Equation]
where P h ∈ C h 1 × h 0 and P w ∈ C w 1 × w 0 are the transform matrices of F-pooling along vertical direction and horizontal direction respectively.
Since F-pooling can be represented as two times matrix-matrix multiplications, its back propagation rule is easily derived.
As mentioned in section 2.1, the choice of up sampling operation U is important.
In this paper, we set U to the re-sampling method which is widely used in signal processing community.
Speciﬁcally, we transform a signal into frequency domain.
Then we zero pad the transformed signal to match the resolution of output.
Finally, we transform it back to time domain.
This process is some-what symmetrical with F-pooling and can also be represented by matrix multiplications.
For 1D signals, we have:
[Equation]
where is Z is the zero padding operation.
[Equation]
U is formalized in a similar way for 2D signals.
2. O PTIMAL ANTI - ALIASING DOWN SAMPLING
In this section, we prove that F-pooling is the optimal anti-aliasing down sampling operation under a given up sampling operation U .
We focus on 1D case because it is easy to extend the conclusion into 2D case.
Given U , the optimal anti-aliasing down sampling is obtained by solving the following problem:
[Equation]
where A is a set of all possible anti-aliasing down sampling operations.
That is, we hope to ﬁnd an anti-aliasing down sampling which minimizes the reconstruction error.
Based on Nyquist sampling theory, M must remove high frequencies of signals and this holds for F-pooling.
We focus on the optimality of F-pooling.
We decompose x into low frequencies part x l and high frequencies part x h , i.e. x l = 1 N F ∗ N D M 2 F N x and x h = x − x l .
D µ is a diagonal matrix whose the ﬁrst and the last µ diagonal elements equal to 1 while others equal to 0.
[Equation]
equation 13 holds because the third term and the forth term of equation 12 equal to 0.
M removes high frequencies and U doesn’t introduce new frequencies.
Thus UM x only contains low frequencies.
Due to the orthogonality of frequency spectrum, (cid:104)UM x , x h (cid:105) = 0 .
Similarly, (cid:104) x l , x h (cid:105) = 0 .
When M is F-pooling, we have
[Equation]
[Equation] [Equation]
16) equation 16 holds due to the deﬁnition of operations T , Z , and D .
See appendix A for proof.
Since the ﬁst term of equation 13 equals to 0, equation 10 is minimized.
Thus, we have proved that
F-pooling is the optimal anti-aliasing down sampling operation.
[Figure]
2. S HIFT - EQUIVALENT
Based on shift theorem of Fourier transform, we have
[Equation]
where S (cid:52) h ∈ C N whose k th element is e − 2 πkN (cid:52) h and (cid:12) is element-wise multiplication.
Combining with equation 16, we have
[Equation]
We have proved that F-pooling is shift-equivalent.
We highlight that equation 22 is not hold if other up sampling operations are used, such as linear interpolation.
The proofs in section 2.3 and 2.4 can be easily extended for 2D signals.
2. D EALING WITH IMAGINARY PART
Generally, the output of F-pooling, i.e. M x contains both real part and imaginary part.
However, for commonly used CNNs, the feature maps must be real.
Thus one must ignore the imaginary part of F-pooling.
On the other hand, M x is treated as complex in the proofs in section 2.3 and 2.4.
If we ignore the imaginary part, the conclusions in section 2.3 and 2.4 are no longer hold.
That is
F-pooling is no longer the optimal anti-aliasing down sampling (but still anti-aliasing) and F-pooling is no longer strict shift-equivalent.
Fortunately, when the resolution of down sampled signals is odd, the imaginary part of M x is zero.
Suppose M = 2 µ + 1 , the frequencies ranged in [ − u, u ] are retained.
Due to its symmetry, M x only contains real part.
If M = 2 µ , the frequencies ranged in [ − u, u ) are retained.
In this case,
M x contains imaginary part.
But we can recover the symmetry and eliminate the imaginary part by setting ( − u ) th frequency to zero.
We call this trick odd padding .
A drawback of odd padding is that more information is lost during down sampling which may reduces accuracy.
Whether M x contains imaginary part is very important in theory.
It determines the theoretical nature of F-pooling.
In practice, we ﬁnd there are no signiﬁcant differences if we don’t use odd padding .
In ﬁgure 1, odd padding is used.
On real-world datasets, i.e. CIFAR and ImageNet, odd padding is not used in this paper.
We show the effects of odd padding in appendix B.
2. I MPLEMENTATION
We implement F-pooling in PyTorch Paszke et al.
Theoretically, it is best to implement
F-pooling with fast Fourier transform (FFT) and inverse FFT.
Suppose F-pooling down samples
[Table]
[Table]
an h 0 × w 0 × c tensor to a h 1 × w 1 × c tensor.
Then its time complexity is ch 0 w 0 log( w 0 h 0 )) .
Unfortunately, we ﬁnd such a implementation in PyTorch is comparatively slow.
Instead, we implement F-pooling via 1 × 1 convolutions.
As in equation 7, 2D F-pooling can be represented as two times matrix-matrix multiplications along vertical direction and horizontal direction respectively.
This is equivalent to 1 × 1 convolutions along vertical direction and direction horizontal.
First, we use two 1 × 1 × w 0 × w 1 convolutions to compute the real part and imaginary part of X : , : ,i P ∗ w in equation 7.
This requires 2 ch 0 w 0 w 1 ﬂoat multiplications.
Now X is mapped to an h 0 × w 1 × c tensor.
We use another two 1 × 1 × h 0 × h 1 convolutions to compute the real part of Y .
This requires 2 ch 0 h 1 w 1 ﬂoat multiplications.
Thus the total number of ﬂoat multiplications is 2 cw 1 ( h 0 w 0 + h 0 h 1 ) .
If we ﬁrst use two 1 × 1 × h 0 × h 1 convolutions then use two 1 × 1 × w 0 × w 1 convolutions, the total number of ﬂoat multiplications becomes 2 ch 1 ( h 0 w 0 + w 0 w 1 ) .
Its time complexity is O
( h 0 w 0 ( h 1 + w 1 )) which is higher than the optimal complexity.
F-pooling requires much more computational costs than average pooling or max-pooling.
F-pooling is faster than a convolutional layer when the resolution of feature maps is smaller than the number of channels, as the situations of image classiﬁcations.
Moreover, the number of pooling layers is limited compared with the number of convolutional layers.
Thus, introducing F-pooling will not introduce too many computational costs into CNNs.
Zhang (2019) claims that it is important to integrate anti-aliasing pooling into CNNs in a correct way.
In this paper, we follow their settings.
Speciﬁcally, a max-pooling with stride s is replaced with a max-pooling with stride 1 and an F-pooling with stride s .
A convolution with stride s is replaced with a convolution with stride 1 and an F-pooling with stride s .
An average pooling with stride s is replaced with an F-pooling with stride s .
See the illustration in ﬁgure 3.
E XPERIMENTS 3. 1D SIGNALS
We test the shift-equivalent of F-pooling on 1D signals.
In the ﬁrst row of ﬁgure 1, the original signal is a sine curve with period 16 and 256 discrete elements.
In the second row of ﬁgure 1, the original signal is a randomly selected row of a 512 × 512 image.
We apply max-pooling, average pooling, and F-pooling with stride 4 to those signals.
As shown in ﬁgure 1, F-pooling is perfectly shift-equivalent.
And average pooling is better than max-pooling from shift-equivalent perspective.
odd padding is used here.
3. I MAGE CLASSIFICATION
CIFAR-100: 1 we test classiﬁcation of low-resolution 32 ×
32 color images.
This dataset contains 50k
[Table]
sub-ImageNet : we then test classiﬁcation of high-resolution 224 × 224 color images.
Original
ImageNet dataset Russakovsky et al.
(2015) contains 1.28M training and 50k validation images, classiﬁed into one of 1,000 categories.
Each category has 1,280 images in average.
To reduce the computational resources for training, we use a subset of ImageNet (sub-ImageNet) in this work.
We randomly select 200 categories from 1,000 categories.
For each category, we randomly selectimages.
Thus, we collect 100k images for training.
And we select corresponding 10k validation images.
All models are trained on a single GPU with batchsize 64 and 100 epochs.
We decrease the initial learning rate by a factor 10 every 40 epochs.
For other hyper-parameters, we follow PyTorch’s ofﬁcial training script.
We train CIFAR-100 using resnet-
He et al.
(2016) and densenet-41 Huang et al.
We train sub-ImageNet using resnet-50 He et al.
(2016), densenet-121 Huang et al.
(2017), and mobilenet-v
Sandler et al.
Those models are widely used as benchmarks.
Max-pooling, average pooling, and strided convolution are covered in those models.
We also compare F-pooling with AA-pooling
Zhang (2019).
For a model trained on sub-ImageNet, its ﬁrst down sampling operation is kept to reduce computational costs.
This setting is also used in Zhang (2019).
The results on CIFARare averaged by 3 runs.
odd padding is not used for both datasets.
We study not only accuracy but also consistency.
We follow the metric of consistency used in Zhang (2019): we check how often the network outputs the same classiﬁcation, given the same image with two different shifts.
[Equation]
We only evaluate diagonal shifts in this paper.
For CIFAR-100, the number of shifted pixels ranges from -7 to 7.
For sub-ImageNet, it ranges from -63 to 63.
Accuracy and consistency on sub-ImageNet is shown in table 1.
See appendix C for loss curves.
Results on CIFAR-100 with shift augment are shown in table 2.
Results on CIFAR-100 without shift augment are shown in table 3. .
Based on those results, we conclude that
• F-pooling is signiﬁcantly and consistently better than original pooling methods in terms of accuracy and consistency.
• F-pooling is comparable with AA-pooling in term of accuracy and consistency.
• Consistency on ImageNet for all methods is much better than CIFAR-100’s.
A probable cause is that the resolution of CIFAR-100 images is too low.
3. S TRICT SHIFT - EQUIVALENT CNN S
CNNs with F-pooling in section 3.2 are not strict shift-equivalent.
There are two reasons: 1) odd padding is not used.
When we ignore the imaginary part of F-pooling output, strict shift-equivalent is destroyed; 2) Convolution layers are not strict shift-equivalent because of padding.
The smaller the resolution, the larger effects of padding.
Correspondingly, we can make the whole CNN strict shift-equivalent by: 1) use odd padding or set the resolution of F-pooling’s output to odd; 2) set the convolutional kernel at all layers to 1 × 1 .
This has been veriﬁed by our experiments.
But we have not yet trained such CNNs on CIFAR-100 or ImageNet.
R ELATED W ORKS
Pooling which reduces the resolution of feature maps is an important part of CNNs.
Early CNNs
Lecun et al.
(1989) use average pooling which is good for anti-aliasing.
Later empirical evidence suggests max-pooling Scherer et al.
and strided-convolutions Long et al.
(2015) provide better performance.
However, small shifts in the input can drastically change the output when stacking multiple max-pooling or strided-convolutions Engstrom et al.
; Azulay & Weiss (2018).
Other variants such as Graham (2014); He et al.
; Lee et al.
(we just list a few of them), focus on extending the functionality of pooling Lee et al.
or making pooling adjusted to variable input size Graham (2014)
; He et al.
Recently, Zhang (2019) shows that CNNs will have better shift-equivalent and anti-aliasing properties when low-pass ﬁltering is integrated correctly.
However, their method is not strict shiftequivalent and anti-aliasing in theory.
Williams & Li (2018) propose Wavelet-pooling.
They decompose a signal via wavelet transform and retain the lowest sub-band.
This process is repeated until the designed down sampling factor is met.
The spirit of Wavelet-pooling is similar to F-pooling.
However, they claim that Wavelet-pooling is better than others because it is a global transform instead of a local transform.
They neither focus on shift-equivalent nor prove Wavelet-pooling is shift-equivalent or not.
F-pooling is a complex transformation because DFT and IDFT are involved in it.
Many works integrate complex transformations or complex values into neural networks.
Amin & Murase (2009) study single-layered complex-valued neural networks for real-valued classiﬁcation problems.
Complex numbers represented in polar coordinates are more suitable to deal with rotations.
Based on this, Cohen et al.
(2018) propose spherical CNNs which are rotation-equivalent to deal with signals projected from spherical surface.
Trabelsi et al.
(2018) propose general deep complex CNNs.
They adjust batch normalization and non-linear activations for complex CNNs.
F-pooling utilizes shift theorem of DFT and achieves shift-equivalent.
This is a new success for the combination of complex transformations and neural networks.
C ONCLUSIONS
In this paper, we have proposed frequency pooling (F-pooling) for CNNs.
As the name suggested,
F-pooling reduces the dimension of signals in frequency domain.
We have deﬁned shift-equivalent of functions which contain down sampling operations by introducing an up sampling operation.
Under this deﬁnition, we have proved that F-pooling is the optimal anti-aliasing down sampling operation and F-pooling is shift-equivalent.
We have integrated F-pooling into modern CNNs.
We have veriﬁed that F-pooling remarkably increases accuracy and robustness w.r.t shifts of modern
CNNs.
Several issues, such as the padding of convolutions and the loss of imaginary part of F-pooling, make the whole model shift-nonequivalent.
We will train CNNs which are shift-equivalent as a whole in the future.
A P ROOF OF OPERATION
We prove D M
=
Z M 2 T M 2 .
By deﬁnition, T can be represented as an M × N matrix and Z can be represented as a N × M matrix.
[Equation]
24) where I M is an M × M identity matrix and
M is an M × M zero matrix.
Z M 2 is equal to the transpose of T M 2 by deﬁnition.
Then
[Equation]
(25) which is the same as D M 2 by deﬁnition.
B E FFECT OF odd padding C L OSS CURVES ON SUB -I MAGE N ET
[Figure]
r
[Figure]
i n
[Figure]
t r
[Figure]