Paper:
ERTIES OF CONVOLUTIONAL NEURAL NETWORKS
Convolutional Neural Networks (CNNs) learn highly discriminative representations from data, but how robust and structured are these representations? How does the data shape the internal network representation? We shed light on these questions by empirically measuring the invariance and equivariance properties of a large number of CNNs trained with various types of input transformations. We find that CNNs learn invariance wrt all 9 tested transformation types and that invariance extends to transformations outside the training range. We also measure the distance between CNN representations and show that similar input transformations lead to more similar internal representations. Transforms can be grouped by the way they affect the learned representation. Additionally, we also propose a loss function that aims to improve CNN equivariance.
1 INTRODUCTION
The overwhelming success of Convolutional Neural Networks (CNNs) is generally attributed to their ability to learn task-specific representations from large quantities of data. This has led to many state of the art results in areas such as image classification (He et al., 2015), semantic segmentation (Liu et al., 2015), and game-playing Go agents (Silver et al., 2016). However, our current understanding of CNN representations is limited, though there is a growing body of literature on visualization techniques for interpreting internal representations (Mahendran & Vedaldi, 2016; Dosovitskiy & Brox, 2015; Nguyen et al., 2016).
In this work, we aim to understand the effect that the training data has on three key properties of representations: invariance, equivariance, and equivalence (Lenc & Vedaldi, 2015). Invariance, with respect to a particular transformation (e.g. rotation by θ = 5), is achieved when the feature vectors for input images at a particular layer do not change when the inputs are transformed. This can be seen as a measure of the robustness of the representation. Equivariance is a generalization of invariance and allows the representation to change in predictable ways in response to input transformations. Equivariance is a rough measure of the structure of the representation space because we can reason about input space transformations in the more abstract representation space. Likewise, measuring equivalence among network representations trained on either the same or different data distributions gives insight into how the training data shapes the representation.
Data augmentation has been considered essential for top CNN performance since the seminal work of Krizhevsky et al. (2012). When input images are stochastically perturbed, less overfitting is observed because all inputs are unique. Since then, others have experimented with various data augmentation strategies with great success (Howard, 2013; Wu et al., 2015; He et al., 2014). While others have noted that data augmentation leads to greater representation invariance (Lenc & Vedaldi, 2015; Wu et al., 2015; Peng et al., 2014), we measure the amount of invariance achieved in response to 9 types of transforms at various magnitudes. We additionally measure the equivariance and pairwise representation distance of the resulting CNNs.
In this work we quantify the invariance and equivariance properties of 70 CNNs on two datasets trained using different input transformations. We find the representation at the penultimate layer
(fc7) is structured wrt almost all input transformations when no augmentation is used. Applying data augmentation effectively collapses the structure of the representation space, leading to input transformation invariance for all 9 transforms, including magnitudes of transforms that were not observed during training.
We measure the pairwise distances of the CNN representations by measuring the error of learned mappings between CNN representations. There is a strong bias towards similarity among CNNs trained with the same type of transformation, and we can group transformations based on their mutual average distances. Similar types of transformations (e.g. Color Jitter, Gaussian Noise) yield more similar representations compared to other dissimilar transform pairs.
We also propose a way to increase the equivariance of a CNN by finetuning on a novel loss function that simultaneously minimizes classification error for both transformed and untransformed inputs. This leads to an increase in equivariance while improving or slightly decreasing performance on the untransformed images (dataset dependent), though this trade-off can be controlled by weighting the terms of the loss function.
2 RELATED WORKS
Lenc & Vedaldi (2015) studied image representations and gave definitions for representation invariance, equivariance, and equivalence. They measured these properties wrt a limited number of transforms for the convolutional layers of the popular AlexNet architecture. We provide extensions to the definitions of these properties to measure the relative degree to which representations possess these properties. We also consider a wider variety of transformations and use the fc7 layer of AlexNet because it is the most invariant layer.
Convergent learning is the idea that identical networks (differing in random initialization) converge to the same representation (i.e.. representation equivalence). In Li et al. (2015), the authors show that there are many corresponding convolutional filters among CNNs for one-to-one, one-to-many, and many-to-many relationships. In complementary fashion, we measure representation distances between last fully connected layers in CNNs trained over different transformations of their inputs.
In Krizhevsky et al. (2012), applying random crops, horizontal flips and color jittering during training reduced the top-1 error by over 1%. Howard (2013) applied additional brightness, contrast, crop, and scale transformations at both training and test time to reduce validation top-1 error from 40.7% to 37.0%. Wu et al. (2015) adds rotation and photo filter effects such as lens distortion, vignetting, and color casting for further improvement. Similarly, we explore a series of 10 types of transformations applied at training time; however, we focus on how they affect the learned representation instead of optimizing classifier performance.
In Peng et al. (2014), synthetic images rendered from 3D CAD models are used to explore invariance in object detection networks for factors such as pose, object texture, and foreground/background color. Training a CNN on images that vary in these aspect results in greater invariance than training with less data variety. 3D CAD models are also used in Aubry & Russell (2015) to show that upper layers of CNNs disentangle and (locally) linearize independent object factors such as object viewpoint, style, color, and scale.
ManiTest (Fawzi & Frossard, 2015) defines classifier invariance as the average magnitude of the minimal transform that causes predictions to change. In contrast, we examine internal CNN representations and measure both representation distance and classifier performance.
3 INVARIANCE, EQUIVARIANCE, AND EQUIVALENCE
In this section, we provide definitions and intuition on the three properties of interest, borrowing basic definitions and notation from Lenc & Vedaldi (2015). Though we focus on CNN image representations, the following applies to any kind of data representation.
3.1 EQUIVARIANCE AND INVARIANCE
If the domain of interest (e.g. natural images) is X ⊂ Rnxn, a function φ : Rnxn → Rm assigns a feature vector to every input image and thus defines an image representation. We will use X to
denote the input space and Z the output space (i.e. φ : X → Z). φ may be equivariant to some transformations, but not others, so we say that φ is equivariant wrt g : X → X iff
∃Mg : Z → Z, ∀x ∈ X : φ(gx) ≈Mgφ(x) (1) This means that transformation by g in X corresponds to a transformation by Mg (if it exists) in Z. Thus if Mg exists, it informs us of the structure of the abstract space Z in terms of the concrete space X . If Mg is the identity transform, then φ maps x and gx to the same point in Z, which leads to invariance wrt g (i.e. φ(gx) ≈ φ(x)). This is related to the idea of whether CNNs collapse (invariance) or linearize (equivariance) view manifolds of 3D objects (Bakry et al., 2015; Aubry & Russell, 2015).
While the previous definitions suggest equivariance and invariance are all-or-nothing properties, we use the following definition to precisely define the ≈ used in Eq 1. We measure the equivariance of φ wrt a transform g as
Equivarainceφ(g;L) = min Mg
1
N N∑ i=1 L(φ(xi),Mgφ(gxi)) (2)
whereL is a suitable loss function such as classification error or L2 distance (Section 3.3). Invariance is similarly measured by fixing Mg as the identity function.
In practice, it is difficult to maximize Mg over the space of all functions, so we restrict Mg to be a simple parametric function (e.g. linear) and learn the parameters from data. Thus our results can be considered a lower bound on the actual equivariance of the representation. Eq. 2 also uses the paradigm that Mg attempts to undo in Z the transformation g performed in X . This is also done in Lenc & Vedaldi (2015) and allows us to compute classification based metrics by applying the same classifier to both φ(x) and Mgφ(gx).
3.2 EQUIVALENCE AND REPRESENTATION DISTANCE
In Lenc & Vedaldi (2015), φ1 is equivalent to φ2 if φ1 has the same information in the sense that ∃E1 : Z1 → Z2, ∀x ∈ X : φ2(x) ≈ E1φ1(x) (3)
However, this relation is only symmetric (as the name equivalence implies) iff E1 is invertible. If E1 is invertible, then E−11 satisfies φ1(x) ≈ E −1 1 φ2(x). In the other direction, if the relation is symmetric, then ∃E2 such that φ1(x) ≈ E2φ2(x) and such an E2 is an inverse of E1. In light of this, we propose renaming Eq 3 to be the sub-representation property with the the terminology that φ2 is a sub-representation of φ1. We then define φ1 and φ2 to be equivalent iff an invertible E1 exists. One example of sub-representation are two reprensetations characterized by (1) the RGB pixels of an image and (2) the corresponding grayscale intensities. The grayscale is a sub-representation of the RGB because we can recover the grayscale from the RGB, but not vice versa.
As with equivariance, we are interested in measuring the extent that two representations are distinct. Thus we define the distance between two representations as
D(φ1, φ2) = min E1 [L(E1φ1(x), φ2)] + min E2 [L(φ1, E2φ2(x))] (4)
As with Mg , finding the optimal E1, E2 is difficult, so we learn linear models from data. Choices for the loss function, L, are discussed in Section 3.3.
3.3 DISTANCE METRICS
The two distance metric we employ are normalized euclidean distance in the representation space and classification error.
LL2(ri, r ′ i) = ||ri − r′i||2 ||ri||2 , Lerr(ri, r ′ i) = 1− δ(yi, ŷi), ŷi := argmax j C(r′i)j
where ri = φ(xi), r′i =Mg(gxi), C(ri) is the predicted distribution of class labels, yi is the ground truth label of ri, and δ is the Kronecker delta function. For LL2, we normalize each dimension of r to have similar magnitude, similar to the normalization of Li et al. (2015). The classifier, C, is composed of the remaining layers of the CNN after φ.
4 IMPROVING EQUIVARIANCE
In this section, we propose a loss function for training a network to increase its invariance and equivariance. CNNs for classification tasks are typically trained with a cross entropy loss, i.e.
Ld(X,Y ) = 1
N N∑ i=1 H(yi, C(φ(xi))) (5)
where X is the training images, Y is the training labels, and H computes cross entropy.
Ld can be augmented by a term that biases the representation φ to be equivariant to some set of transforms G = {g1, g2, ..., gJ}. We propose the loss function, Le, which is minimized when the transformed images are both correctly classified and spatially close to the untransformed image inZ.
Le(X,Y ) = 1
NJ N∑ i=1 J∑ j=1 [ H(yi, C(x ′ ij)) + λ1||x′ij − φ(xi)||22 + λ2||φ(gjxi)− φ(xi)||22 ] (6)
where x′ij = Mgjφ(gjxi). The first term of the sum is the classification loss of the transformed images, while the second and third terms respectively enforce equivariance and invariance. In our experiments, we set λ1 = 504096 , λ2 = 25 4096 , though the results do not appear highly sensitive to this particular setting. Each equivariance mapping Mgj constitutes a new portion of the network that attempts to undo the transformation gj in the Z. As such, the parameters of Mgj must be learned along with the parameters of the network.
Combining Eq. 5 and 6, we arrive at our combined loss:
L(X,Y ) = Ld(X,Y ) + λ3Le(X,Y ) (7)
We found that λ3 = 0.3 provides good compromise between performance on the transformed and untransformed data, though λ3 can be tuned for individual performance requirements.
5 EXPERIMENTAL SETUP
Here we detail the experiments we conduct including the transformations, networks, and datasets.
For the first set of experiments, we simply measure the invariance and equivariance of the AlexNet1 architecture trained under various data augmentation schemes. We measure the representation of layer fc7 (denoted φfc7) after applying the ReLU non-linearity and dropout2.
1We used the reference CaffeNet architecture, which differs slightly from Krizhevsky et al. (2012) 2We follow the test time convention of halving neuron activations instead of zeroing out half the activations.
5.1 TRANSFORMATIONS
We examine a set of 10 transformation types. See Table 1 for brief descriptions or Appendix A for full details. We train 4 CNNs for each type of transform (only 2 for blurring, 1 for baseline) for a total of 35 networks per dataset. Each of the 4 CNN within a transform type is trained with different magnitudes of transformations. Training with data augmentation is performed by stochastically transforming each network input with a transform taken from a specified range.
After training, we measure invariance and equivariance wrt fixed transforms for each type. For example, the 4 CNNs trained on different ranges of rotations (θ ∈ [−5, 5], θ ∈ [−10, 10], θ ∈ [−15, 15], θ ∈ [−20, 20]) all have their equivariance measured wrt rotations of θ = ±{2.5, 5, 10, 15, 20, 25, 30, 40}. As a baseline, we trained a model with no data augmentation and measured the invariance and equivariance of that network wrt all transforms.
5.2 EQUIVARIANCE MAPPING
To measure equivariance wrt gj , we learn Mgj (Eq. 2) and measure the classification accuracy over the transformed images. Due to data augmentation during training, the network may already be invariant wrt gj , so we model Mgj as a linear residual mapping to bias it towards an identity mapping (He et al., 2015). That is Mgj (r) = σ(r + R(r)), where R is to be estimated from data and σ(x) = max(x, 0) is the ReLU non-linearity. We experimented with R as a linear mapping and as a neural network with a single hidden layer. In nearly all cases, the linear mapping outperformed the neural network, so we present results only for the linear mappings. Therefore, we have
Mgj (r) = σ(r +Wjr + bj) = σ((Wj + I)r + bj) (8)
where (Wj , bj) are parameters to be estimated from data.
Each Mgj is trained using fully online SGD for 10 epochs with a static learning rate of 0.001, using an L2 weight decay penalty of 10−5. Momentum with β = 0.9 is used. For learning Mgj , only (Wj , bj) are updated. The rest of the network parameters are treated as fixed. The loss to be minimized is ||Mgj (φfc7(gjx))− φfc7(x)||22.
5.3 REPRESENTATION DISTANCE MAPPING
Here we are interested in learning E1 (or equivalently E2) from Eq. 4 for each ordered pair (φi, φj) in a set of of network representations {φi}. We consider linear mappings with E1(r) = σ(Wr + b) and estimate (W, b) from data. Hyper-parameters for learning are the same as in Section 5.2. The loss to be minimized for each ordered pair is ||φi(x)− E1φj(x)||22.
5.4 FINETUNING EQUIVARIANCE
For this experiment, we take the CNNs trained with the most extreme parameter ranges for each transform type and finetune the representation using Eq. 7 as the loss function. For each CNN, we selected 4-6 transformations within the training range to be the G in Eq. 6.
We finetune for 150,000 weight updates with a mini-batch size of 10 for RVL-CDIP and 100 for ILSVRC. The learning rate is 0.0005 and decays by a factor of 10 every 60,000 updates. Momentum of β = 0.9 was used with no weight-decay regularization.
5.5 DATASETS
We use two large datasets with distinct properties. The first is the popular ILSVRC 2012 dataset (1.2M train / 50K validation), composed of natural images. The second is the RVL Complex Document Information Processing (RVL-CDIP) dataset (Harley et al., 2015) (320K train / 40K val / 40K test). It is composed of scanned tobacco litigation documents in grayscale format and each document image is labeled with one of 16 categories (e.g. letter, memo, email, form, news article, scientific publication). Examples images can be found in Figure 4 in Appendix A In contrast to ILSVRC, document images in RVL-CDIP are intrinsically 2D objects, have fixed zoom and location, and have little background area. We aim to compare and contrast the invariance and equivariance of the various transforms for these two datasets.
In our experiments CNNs are trained using the training splits, using the validation split for model selection. For learning equivariance mappings and representation distance mappings, we use a static randomly chosen 50K training images for ILSVRC and the validation split for RVL-CDIP. Thus the reported metrics are over the validation split for ILSVRC and test split for RVL-CDIP. We subsample for ILSVRC for computational reasons and use the validation split for RVL-CDIP because the CNN representation may have overfit the training data.
6 RESULTS
6.1 MEASURING INVARIANCE AND EQUIVARIANCE
Figure 1 shows the invariance and equivariance measurements for each type of transform on ILSVRC. Additional results for both ILSVRC and RVL-CDIP can be found in Appendix B.1. For each transform type, we compare the CNN trained with the most extreme transforms to the baseline model trained with no data augmentation. The transformations we measured range from no transform to approximately double the transformation magnitude seen during training.
In many instances (e.g. Blur, Noise), the baseline model’s performance deteriorates rapidly even for mild transformations. The large difference between the baseline invariance (black line) and equivariance (red line) indicate that Z is structured wrt most transforms (Color is one exception).
For CNNs trained with data augmentation, the equivariance mapping does not improve performance for transforms that are observed during training. This suggests that Z may not be (linearly) struc-
tured wrt those particular transforms possibly because the structure has been collapsed due to the training object. Some transforms (e.g. Noise, Blur, Shear), at large magnitudes, do show a gap between the equivariance and invariance lines, suggesting that Z is structured wrt these transforms. Equivariance performance does drop off outside the training range, but not near as sharply as the baseline model (e.g Figs. 7a, 7c, 7f), so CNN representations generalize somewhat to unseen transforms.
Figure 2 shows how various magnitudes of training augmentation affect the invariance/equivariance properties of the network for select transforms. Other transforms behaved similarly. Larger training ranges yields greater robustness for a wider range of transforms.
6.2 REPRESENTATION DISTANCE
In this second experiment, we measured the pairwise representation distances of 37 CNNs (34 w/ transforms and 3 baseline networks). Using equation 4 (L = LL2), we computed a pairwise distance matrix for the 37 network representations. We attempted a T-SNE embedding (Maaten & Hinton, 2008) visualization, but the distances seem intrinsically difficult to embed in 2D (see Appendix B.3). Comments on the asymmetry of the distances measured can be found in Appendix B.3.
There is, however, a strong bias towards networks of the same training transformation to be nearer to each other compared to random chance. We performed a K-NN analysis by counting the percentage of same-transform pairs that are K nearest neighbors (Figure 3a). Approximately 80% of sametransform pairs are within 5 neighbors of each other for RVL-CDIP (10 neighbors for ILSVRC). This indicates that networks trained with the same transform end up with more similar representations, though the strength of the transformation and network initialization also play a role.
We also visualize the average distance between CNNs grouped by transformation type (Figs 3c and 3b). Patterns emerge in both datasets. In RVL-CDIP, Crop is the most unique transform because the CNNs learn features as a different static image scale. On the other hand, Crop is not as unique for ILSVRC because objects appear at multiple sizes, so the features learned are not targeted at a single size. Blur, especially for ILSVRC, seems to give unique representations because it changes the local textures from which the representation is derived in a bottom-up fashion. Refining the representation in a top-down manner may be a further avenue for improving CNN representation robustness.
Rotation, Shear, and Perspective transforms are mutually similar for both datasets. This is likely because all three move local pixels in a rigid fashion. Another mutually similar group is Baseline, Color Jitter, and Gaussian Noise, which all operate on pixels independently. Elastic Deformations are somewhat similar to Shear, but not to other transforms.
6.3 IMPROVED EQUIVARIANCE
For RVL-CDIP, finetuning using Eq. 7 improves both equivariance and performance on untransformed images by ∼0.5% (Table 2). This may be because the original network was trained with a wide range of transform parameters, so the representation caters to no particular set of transform parameters. By introducing Mgj during finetuning, the representation can individualize to the untransformed image, while still avoiding the overfit that occurs without data augmentation. Results for ILSVRC were mixed, with finetuning improving equivariance for 3 of 9 transformations (see Table 4 in Appendix B).
7 CONCLUSION
This work gives the results from a large scale empirical study into the invariance and equivariance properties of CNNs trained under different input transformations. We quantify these properties for 70 CNNs across 10 types of transforms for 2 large datasets. CNNs are able to learn invariance to all transforms tried, and this invariance/equivariance extends somewhat to transforms outside the training range. We show evidence that the CNN linearizes its representation wrt Rotation, Perspective, and Shear transforms. In general, the baseline CNN showed very little invariance to even moderate transformations.
We also measured CNN representation distance between all pairs of 37 networks. There is a bias towards CNNs trained with the same type of transformation to have more similar representations. The analysis also revealed that similar types of transforms (e.g. Rotation, Shear) lead to more similar representations. We also proposed a joint loss function that moderately increases accuracy on both untransformed and transformed images, leading to an increase in equivariance for most transforms.
APPENDIX
A TRANSFORM DETAILS
For space reasons, exact details and parameterizations of the transforms used to train the CNNs were omitted from the main text, but are included here. Table 1 gives a brief explanation of the 9 transforms, but each transform will be explained in detail hereafter. Pixels that move into the image dimensions from outside the image (e.g. pixels rotated in) are set to be intensity 0 (see Figure 6b).
A.1 COLOR JITTER
Color Jitter adds a random value to each color channel (RGB or grayscale) of the input image. The random value drawn is separate for each channel, but the same value is applied to each spatial location, essentially making each color either brighter or darker. If the addition of the random value causes a pixel to go outside the normal [0, 255] range, it is truncated back to the range.
During CNN training, the random values are drawn from a Gaussian Distribution parameterized by a mean µ and standard deviation σ. Four CNNs were trained, with µ = 0 and σ ∈ {5, 10, 15, 20}. When measuring invariance/equivariance, we apply a deterministic color jitter transformation that is the same for all input images. We simply adjust the brightness of the image (all color channels tied) by a fixed amount in ±{2, 5, 10, 15, 20, 25, 30, 45}.
A.2 CROP
During training with Crop transformations, we first resize the input images to a larger size (such as 240x240, 256x256, 288x288, or 320x320), and then take a 227x227 crop from the larger imgae. The larger the original image, the more detail the CNN gets to see, but also a smaller percentage of the original image is captured in the input window. CNNs trained with other transformations have their input images resized to 227x227.
For measuring invariance/equivariance, the baseline transform for the Crop CNNs is the center crop, and the transforms are crops at other locations. For example, for invariance to upper left corner crops, we measured the difference in CNN activations between the network applied to the center crop and the network applied to the upper left crop. We measured 25 spatial locations arranged in a 5x5 grid.
A.3 ELASTIC DEFORMATIONS
Elastic Deformations are a way of locally distorting images by applying a smoothed random displacement field (Simard et al., 2003). Image transformations can be characterized by a backward mapping of pixel locations in the output image to locations in the input image. For elastic deformations, first an random displacement field is sampled that would map output pixels to random input locations in their local neighborhood. The size of the local neighborhood is controlled by an α parameter. This displacement field is then smoothed using Gaussian filtering with a specified σ. This causes local regions of the displacement field to point in the same direction.
For training CNNs, we used a fixed σ and sampled α for each input image. The four CNNs we trained with Elastic Deformations can be described as (α ∈ [0, 5], σ = 2), (α ∈ [0, 10], σ = 2), (α ∈ [0, 5], σ = 3), (α ∈ [0, 10], σ = 3). When measuring equivariance/invariance (in contrast to training) we apply the same exact transform to every image. For elastic deformations, this means we applied the same displacement field to each image. We did this for a number of displacement fields of various parameter settings. As shown in Figure 7c These settings included every combination of α ∈ {5, 10, 15} and σ ∈ {2, 2.5, 3, 3.5}.
A.4 GAUSSIAN BLUR
We used the standard Gaussian Blur transform which replaces each pixel by a weighted average of the neighboring pixels, where weight magnitudes are Gaussian wrt spatial distance. The shape of the Gaussian weighting function is controlled by σ, which is measured in pixels. We trained two CNNs with Gaussian Blur, where for each input image, we sampled a sigma from a uniform distribution (i.e. σ ∈ [0, 1.5] and σ ∈ [0, 3]). For measuring equivariance/invariance, we used σ ∈ [0.5, 1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, 5].
A.5 GAUSSIAN NOISE
Gaussian Noise is similar to Color Jitter except that we sample different random values for each spatial location. The strength of the noise is controlled by σ, which is measured in pixel intensity ([0, 255] scale). When training the four CNNs, σ is sampled from a uniform distribution for each input image. The four ranges used were [0, 5], [0, 10], [0, 15], [0, 20].
We measured the invariance/equivariance of Gaussian Noise transforms defined by σ ∈ [2.5, 5, 7.5, 10, 12.5, 15, 20, 25, 30, 40].
A.6 MIRROR
Mirroring refers to reflecting the input image over the horizontal or vertical image axes. We trained 3 CNNs with mirroring. The first performed horizontal mirroring with probability 0.5. The second performed vertical mirroring with probability 0.5. The third performed both types of mirroring with independent probability 0.5 for a total of 4 combinations of flips.
We measures the equivariance/invariance of all 4 combinations of flips.
A.7 PERSPECTIVE
Perspective transforms are a class of transforms with the constraint that straight lines in the input image remain straight in the output image. One way to parameterize (8 parameters) a perspective transform is to specify the output coordinates of the input unit square, so that the unit square is mapped to an arbitrary quadrilateral.
When training a CNN, for each image we first sampled σ from a uniform distribution. Then we sampled the displacement of the coordinates of the unit square from a Gaussian Distribution with mean µ = 0 and standard deviation σ. Thus the output coordinates for the upper left corner of the unit square are x = N(0, σ), y = N(0, σ). For the lower right corner, they would be x = 1 + N(0, σ), y = 1 + N(0, σ). Then the image is warped according to the perspective transform defined by the output coordinates of the unit square.
The ranges used for training are sigma = [0, 0.001], [0, 0.002], [0, 0.003], [0, .004]. For measuring invariance/equivariance, we sampled 10 perspective transforms from a range of equally spaced σ values in increasing magnitude along the x-axis of Figure 7f. Examples of these transforms are given in Figure 6.
A.8 ROTATION
A Rotation transform is specified by an angle θ and is always performed about the center of the image. During training, we sample θ from a uniform distribution for each image. We trained four CNNs with rotations with θ drawn from the following ranges: [−5, 5], [−10, 10], [−15, 15], [−20, 20]. For measuring equivariance/invariance, we used θ ∈ ±{2.5, 5, 10, 15, 20, 25, 30, 45}.
A.9 SHEAR
A Shear transform warps a square image into a parallelogram. We parameterize it by specifying a shear angle θ and an orientation that is either horizontal or vertical. While the original square image has corners that are 90 deg, the parallelogram that results has corners with angles of 90 + θ and 90− θ.
During training, we sample θ from a uniform distribution for each image. We trained four CNNs with shear with θ drawn from the following ranges: [−5, 5], [−10, 10], [−15, 15], [−20, 20]. Each shear has equal probability of horizontal or vertical orientation. For equivariance/invariance, we measured only horizontal shears with θ ∈ ±{5, 10, 15, 20, 25, 30}.
B ADDITIONAL RESULTS
B.1 INVARIANCE AND EQUIVARIANCE
In this section, we include additional results from the experiments in Section 6.1 that did not fit in the main text. Figure 1 presents invariance and equivariance measurements for the ILSVRC dataset (similar to Figure 7).
We observe several interesting differences to the results on RVL-CDIP. The baseline model for ILSVRC is significantly more invariant to vertical mirrors than horizontal mirrors, though the RVLCDIP baseline performs equally bad on both types of mirrors. Brightness increases affect ILSVRC trained CNNs more, though this is likely because natural images are more susceptible to image saturation because they occupy the full spectrum of pixel intensities, while document images tend to be bi-modal. The RVL-CDIP CNN trained on Gaussian Noise was able to learn invariance, while the corresponding ILSVRC CNN did not. However, both the baseline and noise CNN for ILSVRC can correct for the noise with an equivariant mapping.
Figures 8 and 9 show the invariance/equivariance for all 70 CNNs trained with data augmentation. In general, we see an overwhelming trend that training with greater variety of inputs transforms results in greater invariance and equivariance. The exception seems to be that the magnitude of color jittering does not highly affect the CNN robustness. Early CNN layers can easily become invariant to overall brightness changes by extracting information based on pixel differences, and prior work has shown that later layers encode information mostly by which neurons are non-zero, rather than the magnitude of the neuron activations (Agrawal et al., 2014).
For Crop transforms (Figure 8c), we examined 25 evenly spaced crops arranged in a 5x5 grid. The x-axis of the figure shows these crops ordered in scan-line order. The periodic nature of the graph shows that crops nearer the center of the image yield higher performance. There is virtually no difference between the invariance and equivariance measurements for the Crop CNNs, showing that the high level CNN features trained for classification are not predictive for image extrapolation. This is because the equivariance mapping for Crop transforms uses the CNN activations for some un-centered crop to predict the CNN activations over the center crop.
B.1.1 CNN ACCURACY
We also report the accuracy of each CNN in Table 3. While all transforms help improve invariance, we see that some input transformations help performance on the untransformed images, while other hurt performance. In particular, Crop and Shear transforms improve performance the most, while Elastic Deformations and Gaussian Blur seem to hurt performance most.
B.2 CROSS DATASET INVARIANCE/EQUIVARIANCE MEASUREMENTS
In this section, we describe and give results for a new experiment. While all previous experiments measure invariance/equivariance on the same dataset that the CNN was trained on (though with different splits), here we use a new dataset to measure the invariance/equivariance of CNNs trained on RVL-CDIP. The new dataset, ANDOC, is also composed of grayscale document images, though while RVL-CDIP is composed of scanned office documents, ANDOC is composed of digitized
historical documents, which differ significantly from modern office documents. As the RVL-CDIP CNNs were not trained to classify ANDOC documents, we resort to measuring the LL2 difference in the feature representations.
CNN RVL-CDIP ILSVRC CNN RVL-CDIP ILSVRC
In general, the measured invariance/equivariance is lower for ANDOC than for RVL-CDIP (see Figure 10 for examples). This makes sense because the CNNs learn to encode discriminative information about RVL-CDIP documents. However, the invariance and equivariance properties are not lost when switching to a new domain of data. In fact, for Mirror, Rotation, and Shear transforms, the invariance is higher, though this is likely due to less information about the input image being encoded for the new domain.
B.3 REPRESENTATION DISTANCE
In this section, we include additional results from the experiments in Section 6.2 that did not fit in the main text. Figure 11 shows an example T-SNE embedding of the CNN representations. There appears to be very little cluster structure and the T-SNE embeddings appear to be sensitive to the random initialization of the embedding vectors. The results from multiple runs of the T-SNE
embedding in general do not agree on either which points are outliers or on nearest neighbor pairs. This is likely because representation distances do not seem to follow the triangle inequality axiom.
While Eq. 4 yields a symmetric function by averaging two one-way distances, we find that for some pairs of CNNs, there is significant difference between the magnitudes of those one-way distances. In other words,
D′(φ1, φ2) = ∣∣∣min E1 [L(E1φ1(x), φ2)]−min E2 [L(φ1, E2φ2(x))] ∣∣∣ (9)
is large. Figure 12 plots a heat map of Eq. 9 applied pairwise to each CNN for each dataset. Optimization difficulties are one possible cause of large values in Figure 12. However, a more likely explanation is that some CNNs encode unique information about the input that cannot be predicted from the representations learned from other CNNs.
B.4 IMPROVED EQUIVARIANCE
Table 4 shows the results of finetuning using Eq. 7 for CNNs trained with different types of data augmentation. Overall, finetuning improves 3 of 9 transform types. We believe that performance does not improve for some transforms because the dataset naturally contains examples of those transforms due to different object poses. This is not the case for RVL-CDIP because the document images are always scanned in the same manner.

Review:
This paper is an extension of Lenc&Vedaldi15 paper, showing CNN representations at FC7 layer are to certain extent equivariant to various classes of transformations and that training with a certain group of transformation makes the representations more equivalent.
Authors performed a large amount of experiments, training over 30 networks with different forms of jitter, which is quite impressive. However it is rather difficult to find a main message of this work. Yes, authors measured the properties on a different layer than the Lenc&Vedaldi15, however it is hard to find some novel insights other than the known fact that jitter helps to achieve invariance. The evaluation seems to be mostly correct, however the paper does not seem to be solving the task advertised in its title really well.
Major issues are in the experiments with the representation distances:
* The selection of only FC7 is a bit controversial - it is followed only by a single classification layer (FC8) to the common output - class likelyhoods. Because the FC8 is just a linear projections, what the equivalence map does is just to re-project the FC8 weights of the attached network to the weights of the original network. Probably performing similar experiments but on more layers may be more useful (as the networks are already trained).
* The experiment with representation distance is missing what is the classification error on the testing dataset. This would answer whether the representations are actually compatible up to linear transformation at all...
* It is not clear for the experiment with K-NN whether this is measured per each test set example? After training the equivalence map? More clear would be to show that networks trained on similar group of jitter transformations are more compatible on the target task.
* The proposed method does not seem to improve equivariance consistently on all tasks. Especially with \lambda_1 and \lambda_2 having such small values, the loss is basically equal to simple data jitter as it just adds up the loss of the original and transformed image. Maybe the issue is in the selection of the FC7 layer?
In general, this paper shows some interesting results on the FC7 equivariance, but it does not seem to be drawing many interesting new observations out of these experiments. Due to some issues with the equivalence experiments and the finetuning of equivariance, I would not recommend acceptance of this manuscript. However, refining the experiments on already trained networks and restructuring this manuscript into more investigative work may lead to interesting contribution to the field.
There are also few minor issues:
* It is not experimentally verified that the new criterion for equivariance mapping helps to gain better results.
* The angles on page 1 and 5 are missing units (degrees?).
* On page three, "In practice, it is difficult... ", it is not M_g which is maximised/minimised, but the loss over the M_g
* Page 4, footnote 2 - if you are just halving the activations, it is hard to call it a dropout as this constant factor can be passed to the following/preceding weights
* Is the network for RVL-CDIP the same architecture as Alexnet?
* On page 7, Figure 3a+3b - in my opinion, turning the diagonal elements to white is really misleading, and probably even incorrect, as the distance between the same representations should be zero (which is also a way how to verify that the experiments are performed correctly).