M AKING C ONVOLUTIONAL N ETWORKS S HIFT -I NVARIANT A GAIN
Anonymous authors
Paper under double-blind review
A BSTRACT
Modern convolutional networks are not shift-invariant, despite their convolutional nature: small shifts in the input can cause drastic changes in the internal feature maps and output.
In this paper, we isolate the cause – the downsampling operation in convolutional and pooling layers – and apply the appropriate signal processing ﬁx – low-pass ﬁltering before downsampling.
This simple architectural modiﬁcation boosts the shift-equivariance of the internal representations and consequently, shift-invariance of the output.
Importantly, this is achieved while maintaining downstream classiﬁcation performance.
In addition, incorporating the inductive bias of shift-invariance largely removes the need for shift-based data augmentation.
Lastly, we observe that the modiﬁcation induces spatially-smoother learned convolutional kernels.
Our results suggest that this classical signal processing technique has a place in modern deep networks
I NTRODUCTION
Deep convolutional neural networks (CNNs) are designed to perform high-level tasks and be robust to low-level nuisance factors.
For example, small shifts in the input should simply shift the internal feature maps (shift-equivariance), and leave the output relatively unaffected (shift-invariance).
This property has been explicitly engineered through convolutional and pooling layers, where the same function is applied on a local region across the image in a sliding window fashion.
However, recent work (Engstrom et al., 2017; Azulay & Weiss, 2018) has found that small shifts can drastically change the output of a classiﬁcation network.
Why is this the case?
Shift-invariance is lost when spatial resolution is lost, for example, from pooling layers.
Our insight is that conventional strided-pooling, as shown in Fig.
1 (top), is inherently composed of two operations: (1) evaluating the pooling operator densely (without striding), and (2) downsampling.
Naive downsampling loses shift-equivariance, as high-frequency components of the signal alias into low-frequencies.
This phenomenon is commonly illustrated in movies, where wheels appear to spin backwards, due to the frame rate not meeting the Nyquist sampling criterion (known as the Stroboscopic effect).
Separating these operations is important, as it allows us to keep the pooling operation, while applying the appropriate ﬁx to the downsampling operation.
We propose to add the signal processing tool of low-pass ﬁltering before downsampling , as shown in Fig.
1 (bottom).
By low-pass ﬁltering, the high-frequency components of the signal are reduced, reducing aliasing and better preserving shift-equivariance.
This ultimately cascades into better shiftinvariance in the output.
We show example classiﬁcation instabilities in Fig.
A potential concern is that over-aggressive low-pass ﬁltering can result in heavy loss of information.
However, we ﬁnd that with a reasonable selection of low-pass ﬁlter weights, we can maintain classiﬁcation performance while increasing shift-invariance.
Furthermore, we show that without shift-based data augmentation, incorporating this inductive bias actually improves performance.
We ﬁnd that the learned ﬁlters also naturally become smoother after adding the blurring layer.
These results indicate that incorporating this small modiﬁcation not only induces shift-invariance, but causes the network to learn a smoother feature extractor.
In summary, our contributions are as follows:
[Figure]
Pooling Layer
[Figure]
Pooling Layer (Equivalent Interpretation)
[Figure]
PoolBlurDownsample (Proposed)
[Figure]
• We isolate the cause for loss of shift-invariance – downsampling.
Separating the downsampling from pooling enables us to keep the desired pooling, while ﬁxing the loss of shift-equivariance.
We propose to low-pass ﬁlter before downsampling, a common signal processing technique.
• We validate on a classiﬁcation task, and demonstrate increased shift-equivariance in the features and shift-invariance in the output.
• In addition, we observe large improvements in classiﬁcation performance when training without shift-augmentation, indicating more efﬁcient usage of data.
R ELATED W ORK
Local connectivity and weight sharing have been a central tenet of neural networks, including the
Neocognitron (Fukushima & Miyake, 1982), LeNet (LeCun et al., 1998) and modern networks such as Alexnet (Krizhevsky et al., 2012), VGG (Simonyan & Zisserman, 2014), ResNet (He et al., 2016), and DenseNet (Huang et al., 2017).
In biological systems, local connectivity was famously discovered observed in a cat’s visual system by Hubel & Wiesel (1962).
Recent work has strived to build in additional types of invariances, such as rotation, reﬂection, and scaling (Sifre & Mallat, 2013; Bruna & Mallat, 2013;
Esteves et al., 2017; Kanazawa et al., 2014; Worrall et al., 2017; Cohen & Welling, 2016).
Our work focusses on the elusive goal of shift-invariance.
Though properties such as shift-equivariance have been engineered into networks, what factors and invariances does an emergent representation actually learn?
Analysis of deep networks have included qualitative approaches, such as showing patches which activate hidden units (Girshick et al., 2014;
Zhou et al., 2014), actively maximizing hidden units (Mordvintsev et al., 2015), and mapping features back into pixel space (Dosovitskiy & Brox, 2016a;b; Mahendran & Vedaldi, 2015; Zeiler & Fergus, 2014; Nguyen et al., 2017; H´enaff & Simoncelli, 2015).
Our analysis is focused on a speciﬁc, low-level property and is complementary to these qualitative approaches.
A more quantitative approach for analyzing networks is measuring representation or output changes (or robustness to changes) in response to manually generated perturbations to the input, such as image transformations (Goodfellow et al., 2009; Lenc & Vedaldi, 2015; Azulay & Weiss, 2018),
[Figure]
geometric transforms (Ruderman et al., 2018; Fawzi & Frossard, 2015), and CG renderings with various shape, poses, and colors (Aubry & Russell, 2015).
A related line of work is in adversarial examples, where directed perturbations in the input can result in large changes in the output.
These perturbations can be directly on pixels (Goodfellow et al., 2014a;b), a single pixel (Su et al., 2017), small deformations (Xiao et al., 2018), or even afﬁne transformations (Engstrom et al., 2017).
We aim make the network robust to the simplest of these types of attacks and perturbations: shifts.
Both H ´ enaff & Simoncelli (2015) and Azulay & Weiss (2018) identify that modern deep networks ignore the Nyquist sampling criterion when downsampling.
In our work, we propose and empirically validate an easily adoptable ﬁx which minimally perturbs the existing network architecture.
Classic hand-engineered computer vision and image processing representations, such as
SIFT (Lowe, 1999), wavelets, and image pyramids (Burt & Adelson, 1987; Adelson et al., 1984) also extract features in a sliding window manner, often with some subsampling factor.
As discussed in Simoncelli et al.
(1992), literal shift-equivariance cannot hold when with subsampling.
Shiftequivariance can be recovered if features are extracted densely, for example textons (Leung & Malik, 2001), the Stationary Wavelet Transform (Fowler, 2005), and DenseSIFT (Vedaldi & Fulkerson, 2010).
Deep networks can also be evaluated densely, by removing striding and making appropriate changes to subsequent layers by using ´a trous /dilated convolutions (Chen et al., 2014; 2018; Yu &
Koltun, 2015).
This comes at great computation and memory cost.
Our work investigates achieving shift-equivariance with minimal additional computation, by blurring before subsampling.
Blurring before downsampling is fundamental technique in signal processing (Oppenheim et al., 1999), image processing (Gonzalez & Woods, 1992), computer graphics (Foley et al., 1995), and vision (Szeliski, 2010).
In deep learning, average pooling (LeCun et al., 1990) is a form of blurring.
Scherer et al.
(2010) ﬁnds max-pooling to more effective than variants of blurred-downsampling, under the assumption that they are alternatives.
Conversely, we show that they are compatible.
M ETHODS 3. P RELIMINARIES
Deep convolutional networks as feature extractors Let an image with resolution H × W be represented by X ∈ R
H
× W × 3 .
An L -layer CNN can be expressed as a feature extractor F l ( X ) ∈
R H l × W × C l , with layer l ∈ [0 , L ] , spatial resolution H l × W l and C l channels.
Each feature map can also be upsampled to original resolution, (cid:101) F l ( X ) ∈
R H l × W l × C l .
Shift-equivariance and shift-invariance A representation (cid:101) F is shift-equivariant if shifting the input produces a shifted feature map, meaning that shifting and feature extraction are commutable.
We more rigorously deﬁne the Shift function in Eqn. 4.
[Equation]
[Figure]
A representation is shift-invariant if shifting the input results in an identical representation.
[Equation]
For modern classiﬁers, layer l = 0 is the raw pixels, and ﬁnal layer L is a probability distribution over D classes, F L ∈ ∆ 1 × 1 × D .
The net typically progressively reduces spatial resolution, until all resolution is lost and features are of shape R 1 × 1 × C l .
A common technique, such as used in (Lin et al., 2013;
He et al., 2016; Huang et al., 2017), is to average across the entire convolutional feature map spatially, and use fully-connected layers in all subsequent layers, which can be expressed as 1 × 1 convolutions (Long et al., 2015).
In such a setting, as proven by Azulay & Weiss (2018), shift-invariance on the output will necessarily emerge from shift-equivariance in the convolutional features.
Modulo-N shift-equivariance/invariance In some cases, the deﬁnitions in Equations 1, 2 may hold only when shifts (∆ h, ∆ w ) are integer multiples of N.
We refer to these scenarios as moduloN shift-equivariance or invariance.
For example, modulo-2 shift-invariance means that even-pixel shifts of the input result in an identical representation, but odd-pixel shifts may not.
3. C ONVENTIONAL P OOLING VS P ROPOSED P OOL -B LUR -D OWNSAMPLE
Conventional strided pooling breaks shift-equivariance In Fig.
3, we show an example 1-D signal [0 , 0 , 1 , 1 , 0 , 0 , 1 , 1] .
Max-pooling (kernel k
= 2 , stride s = 2 ) will result in [0 , 1 , 0 , 1] .
Simply shifting the input by one index results a dramatically different answer of [1 , 1 , 1 , 1] .
Shift-equivariance is lost.
As seen in the bottom-left, both of these results are inherently downsampling from an intermediate signal – the input signal densely max-pooled ( k = 2 , s = 1 ).
We can write a maxpooling layer as a composition of two functions, max-pooling densely evaluated, followed by naive downsampling: MaxPool k,s ( X ) =
Downsample s ( MaxPool k, 1 ( X )) .
Max-pooling preserves shiftequivariance (when evaluated densely), but naive downsampling does not.
Blurring before downsampling better preserves shift-equivariance
We propose to low-pass ﬁlter the intermediate signal before downsampling, as shown in Fig.
3(bot-right).
We deﬁne our MaxPoolBlurDownsample operator below.
[Equation]
Sampling from the low-pass ﬁltered signal gives [ . 5 , 1 , .
5 , 1] and [ .
75] (Fig.
3 bot-right).
These are closer to each other and better representations of the intermediate signal.
The method allows for a choice of blur kernel.
In image processing, small kernels are often used across applications such as edge detection (Canny, 1986) and image pyramids (Adelson et al., 1984).
We try a number of kernels, ranging from size 2 × 2 to 7 × 7 .
As the blur kernels are separable, it
[Figure]
[Table] [Table]
[Table] [Table]
can be implemented as a series of two convolutions (vertical blur followed by horizontal), and added computation scales linearly with k blur , rather than quadratically.
E XPERIMENTS 4. E XPERIMENTAL S ETUP
Data, architecture, training schedule We test on CIFAR10 classiﬁcation (Krizhevsky & Hinton, 2009), which consists of 50k training and 10k testing images at resolution 32 × 32 .
We use the
VGG13 architecture (Simonyan & Zisserman, 2014) from the PyTorch framework (Paszke et al., 2017) 1 and will make code available.
Each block consists of 2 Conv-BatchNorm-ReLU chunks, followed by MaxPool , doubling feature channels and halving spatial resolution until all resolution is lost.
A ﬁnal softmax predicts a probability vector.
We use stochastic gradient descent (SGD) with momentum 0 .
9 and batch size 128.
We train for 100 epochs at initial learning rate 0 .
1 and 50 additional epochs at 0 .
01 and 0 .
Low-pass ﬁlter kernels We try a number of standard low-pass ﬁlters, shown in Table 1, ranging from size 2 × 2 to 7 × 7 .
All ﬁlters allow the DC signal pass and suppress (or completely kill) the highest frequency.
Variations in ﬁlters correspond to tradeoffs between location of the cutoff frequency, slope of the cutoff, and variation of lobes in the passband and stopband.
These properties are well-studied in the context of ﬁnite impulse response (FIR) ﬁlter design.
However, it is unclear which types of ﬁlters are best suited for deep networks, so we empirically investigate their effects.
Circular convolution and shifting Edge artifacts are an important consideration.
When an image is shifted, information is necessarily lost on one side, and has to be ﬁlled in on the other.
In all our experiments, we use circular shifting and convolution.
When the convolutional kernel hits the edge, it wraps to the other side.
When shifting, pixels are “rolled” off the edge to the other side.
[Equation]
, where % is the modulus function
[Equation]
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Table]
This modiﬁcation minorly decreases classiﬁcation performance, 93 .
8% vs 93 .
4% with data augmentation.
This could potentially be mitigated by additional padding, at the expense of memory and computation.
But more importantly, this methodology affords us a clean testbed.
Any loss in shift-equivariance or invariance is purely due to characteristics of the feature extractor.
4. A NALYSIS
We measure shift-equivariance/invariance in three ways, targeting different aspects.
We ﬁrst focus on the shift-equivariance of the internal layers.
We then check on the agreement of the hard output classiﬁcation.
Finally, we measure how much the soft predicted probability itself varies.
Feature distance (lower is
better).
We test how close shift-equivariance and invariance are to being fulﬁlled by computing d
( Shift ∆ h, ∆ w ( (cid:101) F ( X )) , (cid:101) F ( Shift ∆ h, ∆ w ( X ))) and d ( (cid:101) F ( X ) , (cid:101) F ( Shift ∆ h, ∆ w ( X ))
(left & right-hand sides of Eq. 1, 2), respectively.
We use cosine distance, which is commonly used for deep features (Kiros et al., 2015; Zhang et al., 2018).
[Figure]
Classiﬁcation consistency (higher is better).
Perhaps of greatest interest is the actual decisions the classiﬁer makes.
We can measure its consistency by checking how often the network outputs the same classiﬁcation, given the same image with two different shifts: E
[Equation]
Classiﬁcation variation (lower is better).
The metric above looks at the hard classiﬁcation, discounting classiﬁer conﬁdence.
Similar to Azulay & Weiss (2018), we trace the variation in probability of correct classiﬁcation, given different shifts.
We can capture the variation across all possible shifts: (cid:112) V ar h,w ( { P correct class ( Shift h,w ( X )) }} ) .
Table 1 shows results across a number of different low-pass ﬁlters, training with and without data augmentation.
We dissect the results below.
How shift-equivariant are deep features?
In Fig.
4 (top), we compute distance from shiftequivariance, as a function of all possible shift-offsets (∆ h, ∆ w ) and layers.
MaxPool layers are broken into two components – before and after downsampling.
Pixels are trivially shift-equivariant, as are all layers before the ﬁrst downsampling.
Once downsampling occurs in pool1(ds) , shiftequivariance is lost.
However, modulo-N shift-equivariance still holds, and each subsequent downsampling doubles the factor.
Additionally, we observe that before the downsampling operation, the pooling layer ﬁrst increases shift-equivariance (e.g., conv3 2 to pool3(dense) ).
This is consistent with the long-held intuition that pooling build invariances inside the network (LeCun et al., 1990) and isolates the downsampling operation as the culprit behind loss of shift-equivariance.
Does blurring before downsampling achieve better shift-equivariance?
In Fig.
4 (bottom), we add a blurring ﬁlter to the MaxPool layers, as proposed in Section 3, and again plot shiftequivariance maps for each layer.
Shift-equivariance is clearly better preserved.
In particular, the severe drop-offs in downsampling layers do not occur.
Improved shift-equivariance throughout the network cascades into more consistent classiﬁcations in the ﬁnal softmax layer.
Some selected examples are in Fig.
Our method stabilizes the classiﬁcations.
In Fig.
6, we show the distribution of classiﬁcation variations, before and after adding in the low-pass ﬁlter.
Even a small 2 × 2 ﬁlter, immediately variation.
As the ﬁlter size is increased, the output classiﬁcation variation decreases.
This has a larger effect when training without data augmentation, but is still observable when training with data augmentation.
Does shift-invariance degrade performance?
Our method produces more shift-equivariant feature maps and consequently, more shift-invariant outputs.
However, does this come at a cost?
We study the output classiﬁcation consistency versus classiﬁcation accuracy.
In Fig.
5 (left), we show results, trained without shift-based data augmentation.
Training with the baseline MaxPooling gives accuracy 91 .
6% and consistency 88 .
Our proposed change – with a 5 × 5 triangle ﬁlter improves accuracy to 93 .
3% and consistency to 98 .
This indicates that low-pass ﬁltering does not destroy the signal, or make learning harder.
On the contrary, preserving shift-equivariance serves as “built-in” augmentation, indicating more efﬁcient data usage.
[Figure]
We compute average smoothness of learned conv ﬁlters per layer (lower is smoother).
Baseline MaxPool is in black, and adding additional blurring is shown in colors.
Note that the learned convolutional layers be- come smoother, indicating that a smoother feature extractor is induced.
The Binomial- 7 ﬁlter produces consistently strong results, in both consistency and accuracy.
In principle, networks can learn to be shift-invariant from data.
Does adding shift-based data augmentation remove the beneﬁt from method?
Shift-based data augmentation with the baseline network results in consistency of 96 . 6% , lower than our method trained without data augmentation.
In addition, as seen in Fig.
5 (right), applying out method with data augmentation provides an immediate jump in classiﬁcation consistency, while maintaining accuracy.
From there, a clear tradeoff appears – higher amounts of shift-invariance can be achieved at the cost of decreased accuracy.
For example, very large rectangular ﬁlters over-aggressively smooth the signal.
Downstream applications may favor one factor over another, and the choice of ﬁlter allows one to explore this space.
Fig.
6 investigates the distribution of classiﬁcation variations.
Training with data augmentation with the baseline network reduces variation (black lines on both plots).
Our method reduces variation in both scenarios.
More aggressive ﬁltering further decreases variation.
How do the learned convolutional ﬁlters change with the proposed modiﬁcation?
We measure spatial smoothness using the normalized Total Variation (TV) metric proposed in Ruderman et al.
Our proposed change smooths the internal feature maps for purposes of downsampling.
As shown in Fig.
7, this induces smoother learned ﬁlters throughout the network.
Adding in more aggressive blur kernels further decreases the TV (increasing smoothness).
This indicates that our modiﬁcation actually induces a smoother feature extractor overall.
[Table]
How does the proposed method affect timing?
In
Tab. 2, we show the added time each element of the proposed method takes: evaluating the MaxPool layer at stride 1 instead of stride 2, and running a blurring ﬁlter.
Since the blurring ﬁlters are separable, time increases linearly with ﬁlter size.
The largest ﬁlter adds 12 .
3% per forward pass.
This is signiﬁcantly cheaper than evaluating multiple forward passes in an ensembling approach ( 1024 × computation to evaluate every shift), or evaluating each layer more densely by exchanging striding for dilation ( 4 × , 16 × , 64 × , 256 × computation for conv2 - conv5 , respectively).
These timings are on our VGG13 network setup.
With deeper networks, the relative added computation decreases.
C ONCLUSIONS AND D ISCUSSION
In summary, we show that shift-invariance is lost through a deep network, as downsampling in pooling layers do not meet the Nyquist criteria.
We propose a simple architectural modiﬁcation, following signal processing principles, to improve shift-equivariance.
This change allows the network architecture designer to keep their pooling layer of choice untouched.
We achieve higher consistency while maintaining classiﬁcation performance.
In addition, we show large improvements in both performance and consistency when training without data augmentation.
This is potentially applicable to online learning scenarios, where the data distribution is changing.
Future directions include exploring the potential beneﬁt to downstream applications, such as nearest-neighbor retrieval, improving temporal consistency in video models, robustness to adversarial examples, and high-level vision tasks such as detection.
Another possible future direction is learning the downsampling kernels.
Overall, our experiments indicate that this classical signal processing technique has a place in modern deep networks.
A PPENDIX A D ENSE N ET ARCHITECTURE
Blurring before downsampling can be applied to any strided layer in any network.
We provide an additional experiment using the DenseNet architecture (Huang et al., 2017).
In Fig.
[Figure]
we show classiﬁcation consistency vs. accuracy, similar to Fig.
5 for VGG13 in the main paper
[Figure]
Comparison to VGG13 for Baseline Network We use the DenseNet-40-12 architecture, from a publicly available implementation.
2 Relative to VGG13, DenseNet achieves higher performance ( 94 .
4% vs 93 .
, despite using fewer parameters (1M vs 9M).
DenseNet also starts with higher shift-invariance ( 97 .
7% vs. 96 . 6% ) for two reasons: (a) fewer downsampling layers (2 vs 5) and (b) already using blurring before downsampling, in the form of AveragePool layers, equivalent to using a Rectangle (2) ﬁlter.
We investigate the effects of replacing this 2 × 2 ﬁlter.
Results on DenseNet Our method improves the DenseNet results, and conﬁrms the ﬁndings in the main paper.
In some cases, results are actually stronger.
The primary ﬁndings are:
• As seen in Fig
(left), using a stronger low-pass ﬁlter, such as Binomial (5, 7) without data augmentation, provides competitive performance compared to the baseline trained with data augmentation.
For Binomial (7) , performance is actually better in both consistency and accuracy.
• As seen in Fig.
8 (right), when training with data augmentation, using ﬁlters such as Rect (3) , Triangle (3,5) , and Binomial (5,7) not only increase consistency, as expected, but also slightly increases accuracy, surprisingly.
The observations for DenseNet corroborate the results from VGG13 in the main paper, further demonstrating the effectiveness of blurring before downsampling.
B R OBUSTNESS TO SHIFT - BASED ADVERSARY
In the main paper, we show that using the proposed PoolBlurDownsample method increases the classiﬁcation consistency, while maintaining accuracy.
A logical consequence is increased accuracy in presence of a shift-based adversary.
We empirically conﬁrm this in Fig.
9 for VGG13 on CIFAR10.
We compute classiﬁcation accuracy as a function of maximum adversarial shift.
A max shift of 2 means the adversary can choose any of the 25 positions within a 5 × 5 window.
For the classiﬁer to “win”, it must correctly classify all of them correctly.
Max shift of 0 means that there is no adversary.
Conversely, a max shift of 16 means the image must be correctly classiﬁed at all 32 × 32 = 1024 positions.
Our primary observations are as follows:
• As seen in Fig.
9 (left), the baseline network (gray) is very sensitive to the adversary.
• Adding larger Binomial ﬁlters (from red to purple) increases robustness to the adversary.
In fact, Binomial (7) ﬁlter (purple) without augmentation outperforms the baseline (black) with augmentation.
• As seen in Fig.
9 (right), adding larger Binomial ﬁlters also increases adversarial robustness, even when training with augmentation.
These results corroborate the ﬁndings in the main paper, and demonstrate a use case: increased robustness to shift-based adversarial attack.
[Figure]
C E FFECT OF BLURRING BEFORE POOLING
In our proposed method, we break the strided-pooling operation into two, and blur in between.
This allows us to directly blur before downsampling, which has solid theoretical backing in sampling theory (Oppenheim et al., 1999), and is commonly used in image processing (Gonzalez & Woods, 1992), graphics (Foley et al., 1995), and computer vision (Szeliski, 2010).
Here, we empirically investigate blurring before pooling instead.
Fig.
10 shows the results by applying blurring ﬁrst (shown in the gray points), in comparison our proposed method (colored polygons, as shown before in Fig. 5).
We make the following observations:
• In Fig.
10 (right), when training with augmentation, blurring before pooling reduces performance for all ﬁlters .
For almost all ﬁlters (with few exceptions), both classiﬁcation accuracy and consistency are signiﬁcantly reduced.
• In Fig.
10 (left), when training without augmentation, the lower performing ﬁlters actually perform better when blurring before ﬁltering.
For the better ﬁlters, however, blurring before pooling lowers performance (similar shift-invariance, but lower accuracy).
The signal pre-pooling is undoubtedly related to the signal post-pooling.
Thus, blurring before pooling provides “second-hand” anti-aliasing, and still increases shift-invariance over the baseline.
Though it does empirically help in certain circumstances (the lower-performing ﬁlters, without augmentation), the best performing ﬁlters use the proposed PoolBlurDownsample ordering.
[Figure]
D A VERAGE ACCURACY ACROSS SPATIAL POSITIONS
In Figure
[Figure]
we show how accuracy systematically degrades as a function of spatial shift, when training without augmentation.
We observe the following:
• On the left, the baseline heatmap shows that classiﬁcation accuracy when testing with no shift, but quickly degrades when shifting.
• The proposed ﬁltering decreases the degradation.
Binomial-7 is largely consistent across all spatial positions.
• On the right, we plot the accuracy when making diagonal shifts.
As increased ﬁltering is added, classiﬁcation accuracy becomes consistent in all positions.
[Figure]
E F ILTER DISCUSSION E. F ILTER SELECTION
We use select standard low-pass ﬁlters to empirically test.
The ﬁlter weights are shown in Tab. 1 in the main paper.
Note that weights are normalized to sum to 1.
Rectangle , Triangle , and Binomial ﬁlters are discussed in textbooks such as (Szeliski, 2010).
Window and Least Squares are more advanced FIR ﬁlter design techniques
[Figure]
• Rectangle : a moving average, often referred to as a box ﬁlter.
The ﬁlter is a vector of length ones.
For example, Rect-2 , is [1 , 1] .
This ﬁlter, followed by subsampling, is equivalent to the AveragePooling layer.
• Triangle : linearly decreases weight of neighboring values.
This is equivalent of applying box ﬁltering twice.
For example, Triangle-3 is [1 , 2 , 1] , is two Rect-2 [1 , 1] ﬁlters convolved together, and Triangle-5 is [1 , 2 , 3 , 2 , 1] , is two Rect-3 [1 , 1 , 1] ﬁlters convolved together.
• Binomial : Filter used in Laplacian Pyramids (Burt & Adelson, 1987); [1 , 1] ﬁlter convolved with itself repeatedly.
Note that Binomial-2,3 is equivalent to Rectangle-2 and Triangle-3 , respectively.
• Window : ﬁlter produced using the window method ( firwin ), as described in “7.4 Optimum Approximations for FIR Filters” in Oppenheim et al.
• Least Squares : least squares error minimization ( firls ), from Python scipy.signal toolbox, as described in Selesnick (2005).
E. F ILTER SEPARABILITY
As discussed in Section 3.2, our ﬁlters are separable.
Consider a G ∈ R .
If G is rank-1, it can be decomposed (or separated) into G y G x , where G y ∈ R K × 1 and G x ∈ R 1 × K .
This is an important consideration when convolving G with signal X ∈ R H × W .
[Equation]
, where ∗ is convolution (5)
The left-hand side, evaluating the blur with 2-D convolution takes H × W × K × K multiply-adds, with runtime scaling quadratically with K 2 .
Meanwhile, evaluating a horizontal and vertical blur sequentially takes H × W × K multiply-adds each