Original: E NHANCED C ONVOLUTIONAL N EURALT ANGENT K ERNELS
Revised: E NHANCED C ONVOLUTIONAL N EURAL K ERNELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recent research shows that for training with (cid:96) 2 loss, convolutional neural networks ( CNN s) whose width (number of channels in convolutional layers) goes to inﬁnity, correspond to regression with respect to the CNN Gaussian Process kernel ( CNN-GP )  if only the last layer is trained, and correspond to regression with respect to the Convolutional Neural Tangent Kernel ( CNTK ) if all layers are trained.
Revised: Recent research shows that for training with (cid:96) 2 loss, convolutional neural networks ( CNN s) whose width (number of channels in convolutional layers) goes to inﬁnity correspond to regression with respect to the CNN Gaussian Process kernel ( CNN-GP ) (Novak et al., 2019) if only the last layer is trained, and correspond to regression with respect to the Convolutional Neural Tangent Kernel ( CNTK ) if all layers are trained.

Original: An exact algorithm to compute CNTK (Arora et al., 2019) yielded the ﬁnding that classiﬁcation accuracy of CNTK on CIFAR-10 is within 6 - 7% of that of the corresponding CNN architecture (best ﬁgure being around 78 %) which is interesting performance for a ﬁxed kernel.
Revised: An exact algorithm to compute CNTK (Arora et al., 2019) yielded the ﬁnding that classiﬁcation accuracy of CNTK on CIFAR-10 is within 6 - 7% of that of the corresponding CNN architecture (best ﬁgure being around 78 %) which is interesting performance for a ﬁxed kernel.

Original: Here we show how to signiﬁcantly enhance the performance of these kernels using two ideas.
Revised: Here we show how to signiﬁcantly enhance the performance of these kernels using two ideas.

Original: (1) Modifying the kernel using a new operation called Local Average
Revised: (1) Modifying the kernel using a new operation called Local Average

Original: Pooling ( LAP ) which preserves efﬁcient computability of the kernel and inherits the spirit of standard data augmentation using pixel shifts.
Revised: Pooling ( LAP ) which preserves efﬁcient computability of the kernel and inherits the spirit of standard data augmentation using pixel shifts.

Original: Earlier papers were unable to incorporate naive data augmentation because of the quadratic training cost of kernel regression.
Revised: Earlier papers were unable to incorporate naive data augmentation because of the quadratic training cost of kernel regression.

Original: This idea is inspired by Global Average Pooling ( GAP ), which we show for CNN-GP and CNTK, GAP is equivalent to full translation data augmentation.
Revised: This idea is inspired by Global Average Pooling ( GAP ), which we show for CNN-GP and CNTK is equivalent to full translation data augmentation.

Original: (2) Representing the input image using a pre-processing technique proposed by Coates et al.
Revised: (2) Representing the input image using a pre-processing technique proposed by Coates et al.

Original: (2011), which uses a single convolutional layer composed of random image patches.
Revised: (2011), which uses a single convolutional layer composed of random image patches.

Original: On CIFAR-10 the resulting kernel, CNN-GP with LAP and horizontal ﬂip data augmentation achieves 89% accuracy, matching the performance of
Revised: On CIFAR-10, the resulting kernel, CNN-GP with LAP and horizontal ﬂip data augmentation, achieves 89% accuracy, matching the performance of

Original: AlexNet (Krizhevsky et al., 2012). Note that this is the best such result we know of for a classiﬁer that is not a trained neural network.
Revised: AlexNet (Krizhevsky et al., 2012) , and outperforms the best previous classiﬁer that is not a trained neural network (Mairal, 2016).

Original: Similar improvements are obtained for Fashion-MNIST.
Revised: Similar improvements are obtained for Fashion-MNIST.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recent research shows that for training with (cid:96) 2 loss, convolutional neural networks ( CNN s) whose width (number of channels in convolutional layers) goes to inﬁnity, correspond to regression with respect to the CNN Gaussian Process kernel ( CNN-GP ) if only the last layer is trained, and correspond to regression with respect to the Convolutional Neural Tangent Kernel ( CNTK ) if all layers are trained (Jacot et al., 2018).
Revised: Recent research shows that for training with (cid:96) 2 loss, convolutional neural networks ( CNN s) whose width (number of channels in convolutional layers) goes to inﬁnity, correspond to regression with respect to the CNN Gaussian Process kernel ( CNN-GP ) if only the last layer is trained (Novak et al., 2019; Garriga-Alonso et al., 2019), and correspond to regression with respect to the Convolutional Neural Tangent Kernel ( CNTK ) if all layers are trained (Jacot et al., 2018; Allen-Zhu et al., 2018;

Original: An exact algorithm was given (Arora et al., 2019) to compute CNTK for
Revised: Du et al., 2019b; Arora et al., 2019). Novak et al. Garriga-Alonso et al.

Original:  CNN architectures, as well as those that include a Global Average Pooling ( GAP ) layer (deﬁned below).
Revised: (2019) also implemented CNN-GP and tested its empirical performance. An efﬁcient exact algorithm was given (Arora et al., 2019) to compute CNTK for CNN architectures, as well as those that include a Global Average Pooling ( GAP ) layer (deﬁned below).

Original: This is a ﬁxed kernel that inherits some beneﬁts of CNN s, including exploitation of locality via convolution, as well as multiple layers of processing.
Revised: This is a ﬁxed kernel that inherits some beneﬁts of CNN s, including exploitation of locality via convolution, as well as multiple layers of processing.

Original: For CIFAR-10, incorporating GAP into the kernel improves classiﬁcation accuracy by up to 10% compared to pure convolutional CNTK .
Revised: For CIFAR-10, incorporating GAP into the kernel improves classiﬁcation accuracy by up to 10% compared to pure convolutional CNTK .

Original: While this performance is encouraging for a ﬁxed kernel, the best accuracy is still under 78% , which is disappointing even compared to AlexNet.
Revised: While this performance is encouraging for a ﬁxed kernel, the best accuracy is still under 78% , which is disappointing even compared to AlexNet.

Original: One hope for improving the accuracy further is to somehow capture modern innovations such as batch normalization, data augmentation, residual layers, etc. in CNTK .
Revised: One hope for improving the accuracy further is to somehow capture modern innovations such as batch normalization, data augmentation, residual layers, etc. in CNTK .

Original: The current paper shows how to incorporate simple data augmentation.
Revised: The current paper shows how to incorporate simple data augmentation.

Original: Speciﬁcally, the idea of creating new training images from existing images using pixel translation and ﬂips, while assuming that these operations should not change the label.
Revised: Speciﬁcally, the idea of creating new training images from existing images using pixel translation and ﬂips, while assuming that these operations should not change the label.

Original: Since deep learning uses stochastic gradient descent (SGD), it is trivial to do such data augmentation on the ﬂy.
Revised: Since deep learning uses stochastic gradient descent (SGD), it is trivial to do such data augmentation on the ﬂy.

Original: However, it’s unclear how to efﬁciently incorporate data augmentation in kernel regression, since training time is quadratic in the number of training images.
Revised: However, it’s unclear how to efﬁciently incorporate data augmentation in kernel regression, since training time is quadratic in the number of training images.

Original: Thus somehow data augmentation has to be incorporated into the computation of the kernel itself.
Revised: Thus somehow data augmentation has to be incorporated into the computation of the kernel itself.

Original: The main observation here is that the above-mentioned algorithm for computing CNTK involves a dynamic programming whose recursion depth is equal to the depth of the corresponding ﬁnite CNN .
Revised: The main observation here is that the above-mentioned algorithm for computing CNTK involves a dynamic programming whose recursion depth is equal to the depth of the corresponding ﬁnite CNN .

Original: It is possible to impose symmetry constraints at any desired layer during this computation.
Revised: It is possible to impose symmetry constraints at any desired layer during this computation.

Original: In this viewpoint, it can be shown that prediction using CNTK / CNN-GP with GAP is equivalent to prediction using CNTK / CNN-GP without GAP but with full translation data augmentation with wraparound at the boundary.
Revised: In this viewpoint, it can be shown that prediction using CNTK / CNN-GP with GAP is equivalent to prediction using CNTK / CNN-GP without GAP but with full translation data augmentation with wraparound at the boundary.

Original: The translation invariance property implicitly assumed in data augmentation is exactly equivalent to an imposed symmetry constraint in the computation of the CNTK which in turn is derived from the pooling layer in the CNN .
Revised: The translation invariance property implicitly assumed in data augmentation is exactly equivalent to an imposed symmetry constraint in the computation of the CNTK which in turn is derived from the pooling layer in the CNN .

Original: See Section 4 for more details.
Revised: See Section 4 for more details.

Original: Thus GAP corresponds to full translation data augmentation scheme, but in practice such data augmentation creates unrealistic images (cf.
Revised: Thus GAP corresponds to full translation data augmentation scheme, but in practice such data augmentation creates unrealistic images (cf.

Original: Figure 1) and training on them can harm performance.
Revised: Figure 2) and training on them can harm performance.

Original: However, the idea of incorporating symmetry in the dynamic programming leads to a variant we call Local Average Pooling ( LAP ).
Revised: However, the idea of incorporating symmetry in the dynamic programming leads to a variant we call Local Average Pooling ( LAP ).

Original: This implicitly is like data augmentation where image labels are assumed to be invariant to small translation, say by a few pixels.
Revised: This implicitly is like data augmentation where image labels are assumed to be invariant to small translation, say by a few pixels.

Original: This operation also suggests a new pooling layer for CNN s which we call BBlur and also ﬁnd it beneﬁcial for CNN s in experiments.
Revised: Interestingly, LAP corresponds to a average pooling layer for CNN s, named box ﬁltering (Szeliski, 2010).

Original: Experimentally, we ﬁnd LAP signiﬁcantly enhances the performance as discussed below.
Revised: Experimentally, we ﬁnd LAP signiﬁcantly enhances the performance as discussed below.

Original: • In extensive experiments on CIFAR-10 and Fashion-MNIST, we ﬁnd LAP consistently improves performance of CNN-GP and CNTK .
Revised: • In extensive experiments on CIFAR-10 and Fashion-MNIST, we ﬁnd LAP consistently improves performance of CNN-GP and CNTK .

Original: In particular, we ﬁnd CNN-GP with LAP achieves 81% on CIFAR-10 dataset, outperforming the best previous kernel predictor by 3% .
Revised: In particular, we ﬁnd CNN-GP with LAP achieves 81% on CIFAR-10 dataset, outperforming the best previous kernel predictor by 3% .

Original: • When using the technique proposed by Coates et al.
Revised: • When using the technique proposed by Coates et al.

Original: (2011), which uses randomly sampled patches from training data as ﬁlters to do pre-processing, 1 CNN-GP with LAP and horizontal ﬂip data augmentation achieves 89% accuracy on CIFAR-10, matching the performance of AlexNet (Krizhevsky et al., 2012) and is the strongest classiﬁer that is not a trained neural
Revised: (2011), which uses randomly sampled patches from training data as ﬁlters to do pre-processing, 2 CNN-GP with LAP and horizontal ﬂip data augmentation achieves 89% accuracy on CIFAR-10, matching the performance of AlexNet (Krizhevsky et al., 2012) and is the strongest classiﬁer that is not a trained neural

Original: network.• We also derive a layer for CNN that corresponds to LAP and observe that it improves the performance on certain architectures.
Revised: network.• We also test performance of CNN s with an extra layer corresponding to LAP and observe that it improves the performance on certain architectures.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Data augmentation has long been known to improve the performance of neural network and kernel methods (Sietsma & Dow, 1991;
Revised: Data augmentation has long been known to improve the performance of neural network and kernel methods (Sietsma & Dow, 1991;

Original: Sch ¨ olkopf et al., 1996).
Revised: Sch ¨ olkopf et al., 1996).

Original: Theoretical study of data augmentation dates back to Chapelle et al.
Revised: Theoretical study of data augmentation dates back to Chapelle et al.

Original: Recently, Dao et al.
Revised: Recently, Dao et al.

Original: (2018) proposed a theoretical framework for understanding data augmentation and showed data augmentation with a kernel classiﬁer can have feature averaging and variance regularization effects.
Revised: (2018) proposed a theoretical framework for understanding data augmentation and showed data augmentation with a kernel classiﬁer can have feature averaging and variance regularization effects.

Original: More recently, Chen et al.
Revised: More recently, Chen et al.

Original: (2019) quantitatively shows in certain settings, data augmentation provably improves the classiﬁer performance.
Revised: (2019) quantitatively shows in certain settings, data augmentation provably improves the classiﬁer performance.

Original: For more comprehensive discussion on data augmentation and its properties, we refer readers to Dao et al.
Revised: For more comprehensive discussion on data augmentation and its properties, we refer readers to Dao et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: and references therein.
Revised: and references therein.

Original: CNN-GP and CNTK correspond to inﬁnitely wide CNN with different training strategies (only training the top layer or training all layers jointly).
Revised: CNN-GP and CNTK correspond to inﬁnitely wide CNN with different training strategies (only training the top layer or training all layers jointly).

Original: The correspondence between inﬁnite neural networks and kernel machines was ﬁrst noted by Neal (1996).
Revised: The correspondence between inﬁnite neural networks and kernel machines was ﬁrst noted by Neal (1996).

Original: More recently, this was extended to deep and convolutional neural networks (Lee et al., 2018;
Revised: More recently, this was extended to deep and convolutional neural networks (Lee et al., 2018;

Original: Matthews et al., 2018; Novak et al., 2019; Garriga-Alonso et al., 2019).
Revised: Matthews et al., 2018; Novak et al., 2019; Garriga-Alonso et al., 2019).

Original: These kernels correspond to  neural networks where only the last layer is trained.
Revised: These kernels correspond to inﬁnitely wide neural networks where only the last layer is trained.

Original: A recent line of work studied overparameterized neural networks where all layers are trained (Allen-Zhu et al., 2018; Du et al., 2019b; 2018; Li & Liang, 2018;
Revised: A recent line of work studied overparameterized neural networks where all layers are trained (Allen-Zhu et al., 2018; Du et al., 2019b; 2018; Li & Liang, 2018;

Original: Zou et al., 2018).
Revised: Zou et al., 2018).

Original: Their proofs imply the gradient kernel is close to a ﬁxed kernel which only depends the training data and neural network architecture.
Revised: Their proofs imply the gradient kernel is close to a ﬁxed kernel which only depends the training data and neural network architecture.

Original: These kernels thus correspond to  neural networks where are all layers are trained.
Revised: These kernels thus correspond to inﬁnitely wide neural networks where are all layers are trained.

Original: Jacot et al.
Revised: Jacot et al.

Original: (2018) named this kernel, neural tangent kernel ( NTK ).
Revised: (2018) named this kernel, neural tangent kernel ( NTK ).

Original: Arora et al.
Revised: Arora et al.

Original: (2019) formally proved inﬁnitely wide neural net predictor trained by gradient descent is equivalent to NTK predictor.
Revised: (2019) formally proved inﬁnitely wide neural net predictor trained by gradient descent is equivalent to NTK predictor.

Original: Recently, NTK s induced by various neural network architectures are derived and shown to achieve strong empirical performance (Arora et al., 2019; Yang, 2019; Du et al., 2019a).
Revised: Recently, NTK s induced by various neural network architectures are derived and shown to achieve strong empirical performance (Arora et al., 2019; Yang, 2019; Du et al., 2019a).

Original: Global Average Pooling ( GAP ) is ﬁrst proposed in Lin et al.
Revised: Global Average Pooling ( GAP ) was ﬁrst proposed in Lin et al.

Original: and is common in modern
Revised: and is common in modern

Original: CNN design (Springenberg et al., 2014;
Revised: CNN design (Springenberg et al., 2014;

Original: He et al., 2016; Huang et al., 2017).
Revised: He et al., 2016; Huang et al., 2017).

Original: However, current theoretical understanding on GAP is still rather limited.
Revised: However, current theoretical understanding on GAP is still rather limited.

Original: It has been conjectured in Lin et al.
Revised: It has been conjectured in Lin et al.

Original:  that GAP reduces the number of parameters in the last fully-connected layer and thus avoids overﬁtting, and  GAP is more robust to spatial translations of the input since it sums out the spatial information.
Revised: (2013) that GAP reduces the number of parameters in the last fully-connected layer and thus avoids overﬁtting, and that GAP is more robust to spatial translations of the input since it sums out the spatial information.

Original: In this work, we study GAP from the CNN-GP and CNTK perspective, and draw an interesting connection between GAP and data augmentation.
Revised: In this work, we study GAP from the CNN-GP and CNTK perspective, and draw an interesting connection between GAP and data augmentation.

Original: The method proposed in Coates et al.
Revised: Here we are interested in methods that are not trained neural networks. If the features are predeﬁned before seeing the data, Oyallon & Mallat (2015) proposed the scattering network which achieves 82% classiﬁcation accuracy on CIFAR-10. If one uses unsupervised learning methods to extract features, the method proposed in Coates et al.

Original: (2011) is one of the best-performing approaches on CIFARpreceding modern CNNs.
Revised: (2011) is one of the best-performing approaches on CIFAR-10 preceding modern CNNs.

Revised: To our knowledge, the best result via unsupervised learning method in this line is by Mairal (2016), who used the convolutional kernel network to achieve 86% accuracy on CIFAR-10.

Original: In this work we combine CNTK with LAP and the approach in Coates et al.
Revised: In this work we combine CNTK with LAP and the idea in Coates et al.

Original: (2011) to achieve the best performance for classiﬁers that are not trained neural networks.
Revised: (2011) to achieve the best performance for classiﬁers that are not trained neural networks.

Original:  P RELIMINARIES 3. N OTATION
Revised:  P RELIMINARIES 3. N OTATION

Original: We use bold-faced letters for vectors, matrices and tensors.
Revised: We use bold-faced letters for vectors, matrices and tensors.

Original: For a vector a , let [ a ] i be its i -th entry; for a matrix A , let [ A ] i,j be its ( i, j ) -th entry; for an order 4 tensor T , let [ T ] ij,i (cid:48)
Revised: For a vector a , let [ a ] i be its i -th entry; for a matrix A , let [ A ] i,j be its ( i, j ) -th entry; for an order 4 tensor T , let [ T ] ij,i (cid:48)

Original: j (cid:48) be its ( i, j, i (cid:48) , j (cid:48) )
Revised: j (cid:48) be its ( i, j, i (cid:48) , j (cid:48) )

Original: -th entry.
Revised: -th entry.

Original: For a symmetric tensor, wet let tr ( T ) =
Revised: For a symmetric tensor, wet let tr ( T ) =

Original: (cid:80) i,j T ij,ij .
Revised: (cid:80) i,j T ij,ij .

Original: For an order d tensor
Revised: For an order d tensor

Original: T ∈ R C 1 × C 2 × ...
Revised: T ∈ R C 1 × C 2 × ...

Original: × C d and an integer α ∈ [ C d ] , we use T ( α ) ∈ R C 1 × C 2 × ...
Revised: × C d and an integer α ∈ [ C d ] , we use T ( α ) ∈ R C 1 × C 2 × ...

Original: × C d − 1 to denote the order d − 1 tensor formed by ﬁxing the coordinate of the last dimension of T to be α .
Revised: × C d − 1 to denote the order d − 1 tensor formed by ﬁxing the coordinate of the last dimension of T to be α .

Original: 3. CNN , CNN-GP AND CNTK
Revised: 3. CNN , CNN-GP AND CNTK

Original: In this section we give formal deﬁnitions of CNN , CNN-GP and CNTK that we study in this paper.
Revised: In this section we give formal deﬁnitions of CNN , CNN-GP and CNTK that we study in this paper.

Original: Throughout the paper, we let P be the width and Q be the height of the image.
Revised: Throughout the paper, we let P be the width and Q be the height of the image.

Original: We use q ∈ Z + to denote the ﬁlter size.
Revised: We use q ∈ Z + to denote the ﬁlter size.

Original: In practice, q = 1 , 3 , 5 or 7 .
Revised: In practice, q = 1 , 3 , 5 or 7 .

Original: Padding Schemes.
Revised: Padding Schemes.

Original: In the deﬁnition of CNN , CNTK and CNN-GP , we may use different padding schemes.
Revised: In the deﬁnition of CNN , CNTK and CNN-GP , we may use different padding schemes.

Original: Let x ∈ R P × Q be an image.
Revised: Let x ∈ R P × Q be an matrix.

Original: For a given index pair ( i, j ) with i ≤ 0 , i ≥ P + 1 , j ≤or j ≥ Q + 1 , different padding schemes deﬁne different value for [ x ] i,j .
Revised: For a given index pair ( i, j ) with i ≤ 0 , i ≥ P + 1 , j ≤or j ≥ Q + 1 , different padding schemes deﬁne different value for [ x ] i,j .

Original: For circular padding , we deﬁne [ x ] i,j to be [ x ] i mod P,j mod Q .
Revised: For circular padding , we deﬁne [ x ] i,j to be [ x ] i mod P,j mod Q .

Original: For zero padding , we simply deﬁne [ x ] i,j to be 0 .
Revised: For zero padding , we simply deﬁne [ x ] i,j to be 0 .

Original: Note the difference between circular padding and zero padding occurs only on the boundary of images.
Revised: Note the difference between circular padding and zero padding occurs only on the boundary of images.

Original: We will prove our theoretical results for the circular padding scheme to avoid boundary effects.
Revised: We will prove our theoretical results for the circular padding scheme to avoid boundary effects.

Original: CNN .
Revised: CNN .

Original: Now we describe CNN with and without GAP .
Revised: Now we describe CNN with and without GAP .

Original: For any input image x , after L intermediate layers, we obtain x
Revised: For any input image x , after L intermediate layers, we obtain x

Original: ( L ) ∈
Revised: ( L ) ∈

Original: R P × Q × C ( L ) where C ( L ) is the number of channels of the last layer.
Revised: R P × Q × C ( L ) where C ( L ) is the number of channels of the last layer.

Original: See
Revised: See

Original: Section A for the deﬁnition of x ( L ) .
Revised: Section A for the deﬁnition of x ( L ) .

Original: For the output, there are two choices: with and without GAP .
Revised: For the output, there are two choices: with and without GAP .

Original: • Without GAP : the ﬁnal output is deﬁned as f ( θ , x ) = (cid:80) C ( L )
Revised: • Without GAP : the ﬁnal output is deﬁned as f ( θ , x ) = (cid:80) C ( L )

Original: α =1 (cid:68) W
Revised: α =1 (cid:68) W

Original: ( L +1)
Revised: ( L +1)

Original: ( α ) , x ( L ) ( α ) (cid:69) where x ( L ) ( α
Revised: ( α ) , x ( L ) ( α ) (cid:69) where x ( L ) ( α

Original: ) ∈ R P × Q , and W ( L +1)
Revised: ) ∈ R P × Q , and W ( L +1)

Original: ( α ) ∈ R P × Q is the weight of the last fully-connected layer.
Revised: ( α ) ∈ R P × Q is the weight of the last fully-connected layer.

Original: • With GAP : the ﬁnal output is deﬁned as f ( θ , x ) = 1 PQ (cid:80) C ( L ) α =1 W ( L +1) ( α ) 
Revised: • With GAP : the ﬁnal output is deﬁned as f ( θ , x ) = 1 PQ (cid:80) C ( L ) α =1 W ( L +1) ( α ) · (cid:80) ( i,j ) ∈ [ P ] × [ Q ] [Equation] L )

Original: [Equation]
Revised: [Equation]

Original: ×
Revised: α

Original: [Equation]
Revised: [Equation]

Original:  where W ( L +1) ( α ) ∈ R is the weight of the last fully-connected layer.
Revised: i,j where W ( L +1) ( α ) ∈ R is the weight of the last fully-connected layer.

Original: CNN-GP and CNTK .
Revised: CNN-GP and CNTK .

Original: Now we describe CNN-GP and CNTK .
Revised: Now we describe CNN-GP and CNTK .

Original: Let x , x (cid:48) be two input images.
Revised: Let x , x (cid:48) be two input images.

Original: We denotes the L -th layer’s CNN-GP kernel as Σ ( L ) ( x , x (cid:48) )
Revised: We denote the L -th layer’s CNN-GP kernel as Σ ( L ) ( x , x (cid:48) )

Original: ∈ R [ P ] ×
Revised: ∈ R [ P ] ×

Original: [ Q ] × [ P ] ×
Revised: [ Q ] × [ P ] ×

Original: [ Q ] and the L -th layer’s CNTK kernel as Θ ( L ) ( x , x (cid:48) )
Revised: [ Q ] and the L -th layer’s CNTK kernel as Θ ( L ) ( x , x (cid:48) )

Original: ∈ R [ P ] ×
Revised: ∈ R [ P ] ×

Original: [ Q ] × [ P ] × [ Q ] .
Revised: [ Q ] × [ P ] × [ Q ] .

Original: See Section A for the precise deﬁnitions of Σ ( L ) ( x , x (cid:48) ) and Θ ( L ) ( x , x (cid:48) ) .
Revised: See Section A for the precise deﬁnitions of Σ ( L ) ( x , x (cid:48) ) and Θ ( L ) ( x , x (cid:48) ) .

Original: For the output kernel value, again, there are two choices, without GAP (equivalent to using a fully-connected layer) or with GAP .
Revised: For the output kernel value, again, there are two choices, without GAP (equivalent to using a fully-connected layer) or with GAP .

Original: • Without GAP : the output of CNN-GP is Σ FC ( x , x (cid:48) 
Revised: • Without GAP : the output of CNN-GP is Σ FC ( x , x (cid:48) )

Original: [Equation]

Original: = tr (cid:0) Σ ( L ) ( x , x (cid:48) )
Revised: = tr (cid:0) Σ ( L ) ( x , x (cid:48) )

Original: (cid:1) and the output of CNTK is Θ FC ( x , x (cid:48) 
Revised: (cid:1) and the output of CNTK is Θ FC ( x , x (cid:48) )

Original: [Equation]

Original:  tr (cid:0) Θ ( L ) ( x , x (cid:48) 
Revised: = tr (cid:0) Θ ( L ) ( x , x (cid:48) )

Original: [Equation]

Original: (cid:1) .
Revised: (cid:1) .

Original: • With GAP : the output of CNN-GP is 
Revised: • With GAP : the output of CNN-GP is Σ

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original:  and the output of CNTK is 
Revised: , and the output of CNTK is Θ

Revised: [Equation]

Revised: [Equation]

Revised: =

Original: [Equation]
Revised: [Equation]

Original: Kernel Prediction.
Revised: Kernel Prediction.

Original: Lastly, we recall the formula for kernel regression.
Revised: Lastly, we recall the formula for kernel regression.

Original: For simplicity, throughout the paper, we will assume all kernels are invertible.
Revised: For simplicity, throughout the paper [Equation] we will assume all kernels are invertible.

Original: Given a kernel K ( x , x (cid:48) ) and a dataset ( X , y ) with data { ( x i , y i ) } Ni =1 , deﬁne K X ∈ R N × N where [ K X ] i,
Revised: Given a kernel K ( x , x (cid:48) ) and a dataset ( X , y ) with data { ( x i , y i ) } Ni =1 , deﬁne K X ∈ R N × N where [ K X ] i,

Original: j = K
Revised: j = K

Original: ( x i , x j ) .
Revised: ( x i , x j ) .

Original: The prediction for unseen data x (cid:48) is (cid:80) Ni
Revised: The prediction for unseen data x (cid:48) is (cid:80) Ni

Original: =

Revised: =

Original: α i K
Revised: α i K

Original: ( x (cid:48) , x i ) , where α = K − 1 X y .
Revised: ( x (cid:48) , x i ) , where α = K − 1 X y .

Original: 3. D ATA A UGMENTATION S CHEMES
Revised: 3. D ATA A UGMENTATION S CHEMES

Original: In this paper we consider two types of data augmentation schemes: translation and horizontal ﬂip.
Revised: In this paper we consider two types of data augmentation schemes: translation and horizontal ﬂip.

Original: Translation.
Revised: Translation.

Original: Given ( i, j ) ∈
Revised: Given ( i, j ) ∈

Original: [ P ] × [ Q ] , we deﬁne the translation operator T ij : R P × Q × C →
Revised: [ P ] × [ Q ] , we deﬁne the translation operator T ij : R P × Q × C →

Original: R P × Q × C : for an image x ∈ R P × Q × C , [ T ij ( x )] i (cid:48) ,j (cid:48) ,c
Revised: R P × Q × C : for an image x ∈ R P × Q × C , [ T ij ( x )] i (cid:48) ,j (cid:48) ,c

Original: = [ x ] i (cid:48)
Revised: = [ x ] i (cid:48)

Original: + i,j (cid:48)
Revised: + i,j (cid:48)

Original: + j,c for ( i (cid:48) , j (cid:48) , c ) ∈
Revised: + j,c for ( i (cid:48) , j (cid:48) , c ) ∈

Original: [ P ] ×
Revised: [ P ] ×

Original: [ Q ] × [ C ] .
Revised: [ Q ] × [ C ] .

Original: Here the precise deﬁnition of [ x ] i (cid:48)
Revised: Here the precise deﬁnition of [ x ] i (cid:48)

Original: + i,j (cid:48)
Revised: + i,j (cid:48)

Original: + j depends on the padding scheme.
Revised: + j,c depends on the padding scheme.

Original: Given a dataset
Revised: Given a dataset

Original: D = { ( x i , y i ) }
Revised: D = { ( x i , y i ) }

Original: Ni =1 , the full translation data augmentation scheme creates a new dataset D T
Revised: Ni =1 , the full translation data augmentation scheme creates a new dataset D T

Original: = { ( T ij ( x i ) , y i ) }
Revised: = { ( T ij ( x i ) , y i ) }

Original: ( i,j,n ) ∈
Revised: ( i,j,n ) ∈

Original: [ P ] ×
Revised: [ P ] ×

Original: [ Q ] × [ N ] and training is performed on D T .
Revised: [ Q ] × [ N ] and training is performed on D T .

Original: Horizontal Flip.
Revised: Horizontal Flip.

Original: Flip operator F : R P × Q × C → R P × Q × C : for an image x ∈ R P × Q × C , [ F ( x )]
Revised: Flip operator F : R P × Q × C → R P × Q × C : for an image x ∈ R P × Q × C , [ F ( x )]

Original: i,j,c
Revised: i,j,c

Original: =
Revised: =

Original: [ x ] P +1 − i,j,c for ( i, j, c ) ∈
Revised: [ x ] P +1 − i,j,c for ( i, j, c ) ∈

Original: [ P ] ×
Revised: [ P ] ×

Original: [ Q ] × [ C ] .
Revised: [ Q ] × [ C ] .

Original: Given a dataset D = { ( x i , y i ) } Ni =1 , the horizontal ﬂip augmentation scheme creates a new dataset of the form D F = { ( F ( x i ) , y i ) } Ni = and training is performed on D F ∪ D .
Revised: Given a dataset D = { ( x i , y i ) } Ni =1 , the horizontal ﬂip augmentation scheme creates a new dataset of the form D F = { ( F ( x i ) , y i ) } Ni = and training is performed on D F ∪ D .

Original:  E QUIVALENCE B ETWEEN A UGMENTED K ERNEL AND D ATA A UGMENTATION
Revised:  E QUIVALENCE B ETWEEN A UGMENTED K ERNEL AND D ATA A UGMENTATION

Original: In this section, we demonstrate the equivalence between using data augmentation and using a augmented kernel.
Revised: In this section, we demonstrate the equivalence between using data augmentation and using a augmented kernel.

Original: To formally discuss the equivalence, we use group theory to describe translation and horizontal ﬂip operators.
Revised: To formally discuss the equivalence, we use group theory to describe translation and horizontal ﬂip operators.

Original: We provide the deﬁnition of group in Section B for completeness.
Revised: We provide the deﬁnition of group in Section B for completeness.

Original: It is easy to verify that {F , I} , {T i,j } ( i,
Revised: It is easy to verify that {F , I} , {T i,j } ( i,

Original: j ) ∈
Revised: j ) ∈

Original: [ P ] ×
Revised: [ P ] ×

Original: [ Q ] , {T i,j ◦ F} ( i,j ) ∈
Revised: [ Q ] , {T i,j ◦ F} ( i,j ) ∈

Original: [ P ] × [ Q ] ∪ {T i,j } ( i,j ) ∈
Revised: [ P ] × [ Q ] ∪ {T i,j } ( i,j ) ∈

Original: [ P ] × [ Q ] are groups, where I is the identity map.
Revised: [ P ] × [ Q ] are groups, where I is the identity map.

Original: From now on, given a dataset ( X , y ) with data { ( x i , y i ) } Ni = and a group G , the augmented dataset ( X G , y G ) is deﬁned to be { g ( x i ) , y i } g ∈G ,i ∈
Revised: From now on, given a dataset ( X , y ) with data { ( x i , y i ) } Ni = and a group G , the augmented dataset ( X G , y G ) is deﬁned to be { g ( x i ) , y i } g ∈G ,i ∈

Original: [ N ] .
Revised: [ N ] .

Original: For kernel prediction for unseen data x (cid:48) on the augmented dataset, we have the following formula:(cid:80) i ∈
Revised: For kernel prediction for unseen data x (cid:48) on the augmented dataset, we have the following formula:

Original: [ N ] ,g ∈G (cid:101) α i,g K ( x (cid:48) , g ( x i )) , where (cid:101) α = K − 1 X G y G .
Revised: [Equation]

Original: To proceed, we deﬁne the concept of augmented kernel .
Revised: To proceed, we deﬁne the concept of augmented kernel .

Original: Let G be a ﬁnite group.
Revised: Let G be a ﬁnite group.

Original: Deﬁne the augmented kernel K G as K G ( x , x (cid:48) )
Revised: Deﬁne the augmented kernel K G as K G ( x , x (cid:48) )

Original: =
Revised: =

Original: E g ∈G  K ( g ( x ) ,  x (cid:48) )
Revised: E g ∈G E g (cid:48) ∈G K ( g ( x ) , g (cid:48) ( x (cid:48) ))

Original: where x , x (cid:48) are two inputs images.
Revised: where x , x (cid:48) are two inputs images.

Original: A key observation is that for CNTK and CNN-GP , when circular padding and GAP is adopted, these are actually the augmented kernels with the group G =
Revised: A key observation is that for CNTK and CNN-GP , when circular padding and GAP is adopted, these are actually the augmented kernels with the group G =

Original: {T i,j } ( i,j ) ∈
Revised: {T i,j } ( i,j ) ∈

Original: [ P ] × [ Q ] .
Revised: [ P ] × [ Q ] .

Original: Formally, we have Σ GAP ( x , x (cid:48) )
Revised: Formally, we have Σ GAP ( x , x (cid:48) )

Original: = 1 PQ Σ G FC ( x , x (cid:48) ) and Θ GAP ( x , x (cid:48) )
Revised: = 1 PQ Σ G FC ( x , x (cid:48) ) and Θ GAP ( x , x (cid:48) )

Original: = 1 PQ Θ G FC ( x , x (cid:48) ) .
Revised: = 1 PQ Θ G FC ( x , x (cid:48) ) .

Original: The proof for these two equations is just by checking the formula of these kernels and using deﬁnition of circular padding.
Revised: The proof for these two equations is just by checking the formula of these kernels and using deﬁnition of circular padding.

Original: By similar proof, one can observe the following invariance property of Σ GAP , Σ FC , Θ GAP and Θ FC , under all groups mentioned above, including {F , I} and {T i,j } ( i,j ) ∈ [ P ] × [ Q ] .
Revised: By similar proof, one can observe the following invariance property of Σ GAP , Σ FC , Θ GAP and Θ FC , under all groups mentioned above, including {F , I} and {T i,j } ( i,j ) ∈ [ P ] × [ Q ] .

Original: Deﬁnition 4.1.
Revised: Deﬁnition 4.1.

Original: A kernel K is invariant under a group G if and only if for any g ∈ G ,
Revised: A kernel K is invariant under a group G if and only if for any g ∈ G ,

Original: K ( g ( x ) , g ( x (cid:48) ))
Revised: K ( g ( x ) , g ( x (cid:48) ))

Original: =
Revised: =

Original: K ( x , x (cid:48) ) .
Revised: K ( x , x (cid:48) ) .

Original: Now the following theorem formally states the equivalence between using an augmented kernel on the dataset and using the kernel on the augmented dataset.
Revised: Now the following theorem formally states the equivalence between using an augmented kernel on the dataset and using the kernel on the augmented dataset.

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: Given a group G and a kernel K such that K is invariant under G , then the prediction of augmented kernel K G with dataset ( X , y ) is equal to that of kernel K and augmented dataset ( X G , y G ) .
Revised: Given a group G and a kernel K such that K is invariant under G , then the prediction of augmented kernel K G with dataset ( X , y ) is equal to that of kernel K and augmented dataset ( X G , y G ) .

Original: Namely, for any x (cid:48) ∈ R P × {+Q × C , (cid:80) Ni =
+}
Revised: Namely, for any x (cid:48) ∈ R P × {+Q × C , (cid:80) Ni =
+}

Original: [Equation]
Revised: α i K G ( x (cid:48) , x i ) =

Original: × C , (cid:80) 
Revised: (cid:80) i ∈

Original: [Equation]
Revised: [ N ] ,g ∈G (cid:101) α

Original:  x i )) where α = (cid:0) K G X (cid:1) − 1 y , (cid:101) α
Revised: i,g K ( x (cid:48) , g ( x i )) where α = (cid:0) K G X (cid:1) − 1 y , (cid:101) α

Original: = ( K X G ) − 1 y G .
Revised: = ( K X G ) − 1 y G .

Original: The proof is deferred to Appendix B. Two corollaries are directly followed.
Revised: The proof is deferred to Appendix B. Two corollaries are directly followed.

Original: Corollary 4.1.
Revised: Corollary 4.1.

Original: For G =
Revised: For G =

Original: {T i,j } ( i,j ) ∈
Revised: {T i,j } ( i,j ) ∈

Original: [ P ] × [ Q ] , for any given dataset D , the prediction of Σ GAP (or Θ
Revised: [ P ] × [ Q ] , for any given dataset D , the prediction of Σ GAP (or Θ

Original: GAP )  dataset D  prediction  Σ FC (or Θ FC )  augmented dataset D T .[Figure]
Revised: GAP ) with dataset D is equal to the prediction of Σ FC (or Θ FC ) with augmented dataset D T .

Original: Corollary 4.2.
Revised: Corollary 4.2.

Original: For G = {F , I} , for any given dataset D , the prediction of Σ G GAP (or Θ G GAP ) with dataset D is equal to the prediction of Σ GAP (or Θ GAP ) with augmented dataset D F ∪ D .
Revised: For G = {F , I} , for any given dataset D , the prediction of Σ G GAP (or Θ G GAP ) with dataset D is equal to the prediction of Σ GAP (or Θ GAP ) with augmented dataset D F ∪ D .

Original: Now we discuss implications of Theorem 4.1 and its corollaries.
Revised: Now we discuss implications of Theorem 4.1 and its corollaries.

Original: Naively applying data augmentation, with full translation on CNTK or CNN-GP for example, one needs to create a P 2 Q 2 times larger kernel matrix since there are PQ translation operators, which is often computationally infeasible.
Revised: Naively applying data augmentation, with full translation on CNTK or CNN-GP for example, one needs to create a P 2 Q 2 times larger kernel matrix since there are PQ translation operators, which is often computationally infeasible.

Original: Instead, we can directly use the augmented kernel ( Σ GAP or Θ GAP for the case of full translation on CNTK or CNN-GP ) for prediction, for which one only needs to create a kernel matrix that is as large as the original one.
Revised: Instead, we can directly use the augmented kernel ( Σ GAP or Θ GAP for the case of full translation on CNTK or CNN-GP ) for prediction, for which one only needs to create a kernel matrix that is as large as the original one.

Original: For horizontal ﬂip, although the augmentation kernel is not as conveniently computed as full translation, Corollary 4.2 still provides a more efﬁcient method for computing kernel value and solving kernel regression, since the augmented dataset is twice as large as the original dataset.
Revised: For horizontal ﬂip, although the augmentation kernel is not as conveniently computed as full translation, Corollary 4.2 still provides a more efﬁcient method for computing kernel value and solving kernel regression, since the augmented dataset is twice as large as the original dataset.

Original:  L OCAL A VERAGE P OOLING
Revised:  L OCAL A VERAGE P OOLING

Original: In this section, we introduce a new operation called Local Average Pooling ( LAP ).
Revised: In this section, we introduce a new operation called Local Average Pooling ( LAP ).

Original: As discussed in the introduction, full translation data augmentation can create unrealistic images.
Revised: As discussed in the introduction, full translation data augmentation can create unrealistic images.

Original: A natural idea is to do local translation data augmentation, i.e., restricting the distance of translation.
Revised: A natural idea is to do local translation data augmentation, i.e., restricting the distance of translation.

Original: More speciﬁcally, we only allow translation operations T ∆ i , ∆ j (cf.
Revised: More speciﬁcally, we only allow translation operations T ∆ i , ∆ j (cf.

Original: Section 3.3) for (∆ i , ∆ j ) ∈ [ − c, c ] ×
Revised: Section 3.3) for (∆ i , ∆ j ) ∈ [ − c, c ] ×

Original: [ − c, c ] where c is a parameter to control the amount of allowed translation.
Revised: [ − c, c ] where c is a parameter to control the amount of allowed translation.

Original: With a proper choice of the parameter c , translation data augmentation will not create unrealistic images (cf.
Revised: With a proper choice of the parameter c , translation data augmentation will not create unrealistic images (cf.

Original: Figure 1).
Revised: Figure 2).

Original: However, naive local translation data augmentation is computationally infeasible for kernel methods, even for moderate choice of c .
Revised: However, naive local translation data augmentation is computationally infeasible for kernel methods, even for moderate choice of c .

Original: To remedy this issue, in this section we introduce LAP , which is inspired by the connection between full translation data augmentation and GAP on CNN-GP and CNTK .
Revised: To remedy this issue, in this section we introduce LAP , which is inspired by the connection between full translation data augmentation and GAP on CNN-GP and CNTK .

Original: Here, for simplicity, we assume P = Q and derive the formula only for CNTK .
Revised: Here, for simplicity, we assume P = Q and derive the formula only for CNTK .

Original: Our formula can be generalized to CNN-GP in a straightforward manner.
Revised: Our formula can be generalized to CNN-GP in a straightforward manner.

Original: Recall that for two given images x and x (cid:48) , without GAP , the formula for output of CNTK is tr ( Θ ( x , x (cid:48) )) .
Revised: Recall that for two given images x and x (cid:48) , without GAP , the formula for output of CNTK is tr ( Θ ( x , x (cid:48) )) .

Original: With GAP , the formula for output of CNTK is
Revised: With GAP , the formula for output of CNTK is

Original: [Equation]
Revised: [Equation]

Original: With circular padding, the formula can be rewritten as
Revised: With circular padding, the formula can be rewritten as

Original: [Equation]
Revised: [Equation]

Original:  j , which is again equal to 
Revised: i ,j +∆ (cid:48) j , which is again equal to 1 P 2 E ∆ i , ∆ (cid:48) i , [Equation] i [Equation]

Revised: x )

Original: [Equation]
Revised: [Equation]

Revised: i

Revised: [Equation]

Revised: (cid:48)

Revised: [Equation]

Revised: x (cid:48)

Revised: [Equation]

Revised: (cid:17)(cid:17) .

Original: We ignore the 1 /P 2 scaling factor since it plays no role in kernel regression.
Revised: We ignore the 1 /P 2 scaling factor since it plays no role in kernel regression.

Original: Now we consider restricted translation operations T ∆ i , ∆ j with (∆ i , ∆ j ) ∈ [ − c, c ] × [ − c, c ] and derive the formula for LAP .
Revised: Now we consider restricted translation operations T ∆ i , ∆ j with (∆ i , ∆ j ) ∈ [ − c, c ] × [ − c, c ] and derive the formula for LAP .

Original: Assuming circular padding, we have
Revised: Assuming circular padding, we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Now we have derived the formula for LAP which is given in Equation 1.
Revised: Now we have derived the formula for LAP which is given in Equation 1.

Original: Notice that the formula in
Revised: Notice that the formula in

Original: Equation 1 is a well-deﬁned quantity for all padding schemes.
Revised: Equation 1 is a well-deﬁned quantity for all padding schemes.

Original: In particular, assuming zero padding, 
Revised: In particular, assuming zero padding, when c =

Original: [Table] when c = P , LAP is equivalent to GAP .
Revised: P , LAP is equivalent to GAP .

Original: When c = 0 , LAP is equivalent to no pooling layer.
Revised: When c = 0 , LAP is equivalent to no pooling layer.

Original: Another advantage of LAP is that it does not incur any extra computational cost, since the formula in Equation 1 can be rewritten as (cid:80) i,j,i (cid:48) ,j (cid:48) ∈
Revised: Another advantage of LAP is that it does not incur signiﬁcant additional computational cost, since the formula in Equation 1 can be rewritten as (cid:80) i,j,i (cid:48) ,j (cid:48) ∈

Original: [ P ] 4 [ w ] i,j,i (cid:48) ,j (cid:48) ·
Revised: [ P ] 4 [ w ] i,j,i (cid:48) ,j (cid:48) ·

Original: [ Θ ( x , x (cid:48) )] i,j,i (cid:48) ,j (cid:48) where each entry in the weight tensor w can be calculated in O (1) time.
Revised: [ Θ ( x , x (cid:48) )] i,j,i (cid:48) ,j (cid:48) where each entry in the weight tensor w can be calculated in O (1) time.

Original: Note that the GAP operation in CNN-GP and CNTK corresponds to the GAP layer in CNN s.
Revised: Note that the GAP operation in CNN-GP and CNTK corresponds to the GAP layer in CNN s.

Original: Here we observe the follow box blur layer that corresponds to LAP in CNN s. Box blur layer ( BBlur ) is a function R P × Q → R P × Q such that [ BBlur ( x )] i,j =1 (2 c +1) 2 (cid:80) ∆ i , ∆ j ∈ [ − c,c ] 2 x i +∆ i ,j +∆ j .
Revised: Here we observe the following box ﬁltering layer that corresponds to LAP in CNN s. Box ﬁltering layer ( BF ) is a function R P × Q → R P × Q such that [ BF ( x )] i,j =

Revised: [Equation]

Original: This is in fact the standard average pooling layer  with  pooling size 2 c +[-
 and stride-] 1 .
Revised: This is in fact a standard average pooling layer but with stride 1 and pooling size 2 c +[-
 and stride-] 1 .

Original: We prove the equivalence between LAP and box blur layer in Appendix C. In Section 6.3, we test BBlur on CNN s to verify its effectiveness.
Revised: We prove the equivalence between LAP and box ﬁltering layer in Appendix C. In Section 6.3, we test BF on CNN s to verify its effectiveness.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section we present our empirical ﬁndings on CIFAR-10 (Krizhevsky, 2009) and FashionMNIST (Xiao et al., 2017).
Revised: In this section we present our empirical ﬁndings on CIFAR-10 (Krizhevsky, 2009) and FashionMNIST (Xiao et al., 2017).

Original: The detailed experimental setup is reported in Appendix D. 
Revised: The detailed experimental setup is reported in Appendix D. When reporting test accuracies, the best result on the test set is in boldface and the result that corresponds to the hyper-parameter chosen by cross-validation is underlined.

Original: 6. A BLATION S TUDY ON CIFAR-10 AND F ASHION -MNIST
Revised: 6. A BLATION S TUDY ON CIFAR-10 AND F ASHION -MNIST

Original: We perform experiments to study the effect of different values of the c parameter in LAP and horizontal ﬂip data argumentation on CNTK and CNN-GP .
Revised: We perform experiments to study the effect of different values of the c parameter in LAP and horizontal ﬂip data argumentation on CNTK and CNN-GP .

Original: For experiments in this section we set the bias term in CNTK and CNN-GP to be β = 

Revised: For experiments in this section we set the bias term in CNTK and CNN-GP to be β =

Original: (cf.
Revised: (cf.

Original: Section A).
Revised: Section A).

Original: We use the same architecture for
Revised: We use the same architecture for

Original: CNTK and CNN-GP as in Arora et al.
Revised: CNTK and CNN-GP as in Arora et al.

Original: I.e., we stack multiple convolutional layers before the ﬁnal pooling layer.
Revised: I.e., we stack multiple convolutional layers before the ﬁnal pooling layer.

Original: We use d to denote the number of convolutions layers, and in our experiments we set d to be 5 , 8 , 11 or 14 , to study the effect of depth on CNTK and CNN-GP .
Revised: We use d to denote the number of convolutions layers, and in our experiments we set d to be 5 , 8 , 11 or 14 , to study the effect of depth on CNTK and CNN-GP .

Original: For CIFAR-10, we set the c parameter in LAP to be 0 , 4 , . . .
Revised: For CIFAR-10, we set the c parameter in LAP to be 0 , 4 , . . .

Original: , 32 , while for Fashion-MNIST we set the c parameter in LAP to be 0 , 4 , . .
Revised: , 32 , while for Fashion-MNIST we set the c parameter in LAP to be 0 , 4 , . .

Original: Notice that when c = 32 for CIFAR-10 or c = 28 for Fashion-MNIST,
Revised: Notice that when c = 32 for CIFAR-10 or c = 28 for Fashion-MNIST,

Original: LAP is equivalent to GAP , and when c = 0 , LAP is equivalent to no pooling layer.
Revised: LAP is equivalent to GAP , and when c = 0 , LAP is equivalent to no pooling layer.

Original: Results on
Revised: Results on

Original: CIFAR-10 are reported in Table 1-2.
Revised: CIFAR-10 are reported in Tables 1 and 3.

Original: Due to space constraint, results on Fashion-MNIST are reported in Table 5-6 in Appendix E.
Revised: Due to space constraint, results on Fashion-MNIST are reported in Tables 5 and 6 in Appendix E.

Original: In each table, for each combination of c and d , the ﬁrst number is the test accuracy without horizontal ﬂip data augmentation (in percentage), and the second number (in parentheses) is the test accuracy with horizontal ﬂip data augmentation.
Revised: In each table, for each combination of c and d , the ﬁrst number is the test accuracy without horizontal ﬂip data augmentation (in percentage), and the second number (in parentheses) is the test accuracy with horizontal ﬂip data augmentation.

Revised: To perform cross-validation to choose the hyper-parameters, we use the last 10000 samples in the training set of CIFAR-10 and Fashion-MNIST as the validation set and the rest samples as the training set.

Revised: We then use the full training set to report the test accuracy.

Revised: To perform cross-validation, we choose c , d , CNN or CNN-GP , and whether or not to adopt horizontal ﬂip based on the validation accuracy (shown in Appendix F).

Revised: With cross-validation, the resulting accuracy is 82.09% on CIFAR-10 and 94.07% on Fashion-MNIST.

Original: We made the following observations regarding our experimental results
Revised: We made the following observations regarding our experimental results

Original: • LAP with a proper choice of the parameter c signiﬁcantly improves the performance of CNTK and CNN-GP .
Revised: • LAP with a proper choice of the parameter c signiﬁcantly improves the performance of CNTK and CNN-GP .

Original: On CIFAR-10, the best-performing value of c is c = 12 or 16 , while on FashionMNIST the best-performing value of c is c = 4 .
Revised: On CIFAR-10, the best-performing value of c is c = 12 or 16 , while on FashionMNIST the best-performing value of c is c = 4 .

Original: We suspect this difference is due to the nature of the two datasets: CIFAR-10 contains real-life images and thus allow more translation, while Fashion-MNIST contains images with centered clothes and thus allow less translation.
Revised: We suspect this difference is due to the nature of the two datasets: CIFAR-10 contains real-life images and thus allow more translation, while Fashion-MNIST contains images with centered clothes and thus allow less translation.

Original: For both datasets, the best-performing value of c is consistent across all settings (depth, CNTK or CNNGP ) that we have considered.
Revised: For both datasets, the best-performing value of c is consistent across all settings (depth, CNTK or CNNGP ) that we have considered.

Original: • Horizontal ﬂip data augmentation is less effective on Fashion-MNIST than on CIFAR-10.
Revised: • Horizontal ﬂip data augmentation is less effective on Fashion-MNIST than on CIFAR-10.

Original: There are two possible explanations for this phenomenon.
Revised: There are two possible explanations for this phenomenon.

Original: First, most images in Fashion-MNIST are [Table] [Table] [Table]
Revised: First, most images in Fashion-MNIST are nearly horizontally symmetric (e.g., T-shirts and bags).

Original: [Table]

Original: nearly horizontally symmetric (e.g., T-shirts and bags)

Original: Second, CNTK and CNN-GP have already achieved a relatively high accuracy on Fashion-MNIST, and thus it is reasonable for horizontal ﬂip data augmentation to be less effective on this dataset.
Revised: Second, CNTK and CNN-GP have already achieved a relatively high accuracy on Fashion-MNIST, and thus it is reasonable for horizontal ﬂip data augmentation to be less effective on this dataset.

Original: • Finally, for CNTK , when c = 

Revised: • Finally, for CNTK , when c =

Original: (no pooling layer) and c = 32 ( GAP ) our reported test accuracies are close to those in Arora et al.
Revised: (no pooling layer) and c = 32 ( GAP ) our reported test accuracies are close to those in Arora et al.

Original: on CIFAR-10.
Revised: on CIFAR-10.

Original: For CNN-GP , when c = 0 (no pooling layer) our reported test accuracies are close to those in Novak et al.
Revised: For CNN-GP , when c = 0 (no pooling layer) our reported test accuracies are close to those in Novak et al.

Original:  on CIFAR-10 and Fashion-MNIST.
Revised: (2019) on CIFAR-10 and Fashion-MNIST.

Original: This suggests that we have reproduced previous reported results.
Revised: This suggests that we have reproduced previous reported results.

Original: 6. I MPROVING P ERFORMANCE ON CIFAR-10 U SING A DDITIONAL F EATURE E XTRACTOR
Revised: 6. I MPROVING P ERFORMANCE ON CIFAR-10 U SING R ANDOM P ATCHES L AYER

Original: Finally, we explore another interesting question: what is the limit of non-deep-neural-network methods on CIFAR-10?
Revised: Finally, we explore another interesting question: what is the best performance achievable via a method that is not a trained neural network?

Original: To further improve the performance, we combine CNTK and CNN-GP with LAP , together with the previous best-performing non-deep-neural-network method Coates et al.
Revised: To further improve the performance, we combine CNTK and CNN-GP with LAP , together with the unsupervised learning approach developed in Coates et al.

Original: Here we use the variant implemented in Recht et al.
Revised: Here we use the variant implemented in Recht et al.

Original: More speciﬁcally, we ﬁrst sample 2048 random image patches with size 5 × 5 from all training images.
Revised: More speciﬁcally, we ﬁrst sample 2048 random image patches with size 5 × 5 from all training images.

Original: Then for the sampled images patches, we subtract the mean of the patches, then normalize them to have unit  norm, and ﬁnally perform ZCA transformation to the resulting patches.
Revised: Then for the sampled images patches, we subtract the mean of the patches, then normalize them to have unit [Table] [Table] norm, and ﬁnally perform ZCA transformation to the resulting patches.

Original: We use the resulting patches asﬁlters of a convolutional layer with kernel size 5 , stride 1 and no dilation or padding.
Revised: We use the resulting patches as 2048 ﬁlters of a convolutional layer with kernel size 5 , stride 1 and no dilation or padding.

Original: For an input image x , we use conv ( x ) to denote the output of the convolutional layer.
Revised: For an input image x , we use conv ( x ) to denote the output of the convolutional layer.

Original: As in the implementation in Recht et al.
Revised: As in the implementation in Recht et al.

Original: (2019), we use ReLU( conv ( x ) − β feature ) and ReLU( − conv ( x ) − β feature ) as the input feature for CNTK and CNN-GP .
Revised: (2019), we use ReLU( conv ( x ) − β feature ) and ReLU( − conv ( x ) − β feature ) as the input feature for CNTK and CNN-GP .

Original: Here we ﬁx β feature = 1 as in Recht et al.
Revised: Here we ﬁx β feature = 1 as in Recht et al.

Original: and the bias term β in CNTK and CNN-GP to be β = 1 .
Revised: and the bias term β in CNTK and CNN-GP to be β = 1 .

Original: To make the output kernel value invariant under horizontal ﬂip (cf.
Revised: To make the output kernel value invariant under horizontal ﬂip (cf.

Original: Deﬁntion 4.1), for each image patch, we horizontally ﬂipped it and add the ﬂipped patch into the convolutional layer as a new ﬁlter.
Revised: Deﬁntion 4.1), for each image patch, we horizontally ﬂipped it and add the ﬂipped patch into the convolutional layer as a new ﬁlter.

Original: Thus, for an input CIFAR-10 image of size 32 × 32 , the dimension of the output feature is 8192 × 28 × 28 .
Revised: Thus, for an input CIFAR-10 image of size 32 × 32 , the dimension of the output feature is 8192 × 28 × 28 .

Original: To isolate the effect of randomness in the choices of the image patches, we ﬁx the random seed to be 0 throughout the experiment.
Revised: To isolate the effect of randomness in the choices of the image patches, we ﬁx the random seed to be 0 throughout the experiment.

Original: In this experiment, we set the value of the c parameter in LAP to be 4 , 8 , 12 , . . .
Revised: In this experiment, we set the value of the c parameter in LAP to be 4 , 8 , 12 , . . .

Original: , 20 to avoid small and large values of c .
Revised: , 20 to avoid small and large values of c .

Original: The results are reported in Table 3-4.
Revised: The results are reported in Tables 2 and 4.

Revised: Similar to the experiments in Section 6.1, again we set the hyper-parameters by cross-validation, and the resulting accuracy is 88.91%.

Original:  From our experimental results, it is evident that combining CNTK or CNN-GP with additional feature extractor can signiﬁcantly improve upon the performance of using solely CNTK or CNNGP , and that of using solely the feature extractor Coates et al.
Revised: See Appendix F for the validation accuracy for different hyper-parameters. From our experimental results, it is evident that combining CNTK or CNN-GP with additional feature extractor can signiﬁcantly improve upon the performance of using solely CNTK or CNNGP , and that of using solely the feature extractor Coates et al.

Original: Previously, it has been reported in Recht et al.
Revised: Previously, it has been reported in Recht et al.

Original: that using solely the feature extractor Coates et al.
Revised: that using solely the feature extractor Coates et al.

Original: (2011) (together with appropriate pooling layer) can only achieve a test accuracy of 84.2% using 256, 000 image [Table] [Table] [Figure]
Revised: (2011) (together with appropriate pooling layer) can only achieve a test accuracy of 84.2% using 256, 000 image patches, or 83.3% using 32, 000 image patches.

Original: [Table]

Original: [Figure]

Original: patches, or 83.3%

Original: 32, 000 image patches.

Original: Even with the help of horizontal data augmentation, the feature extractor Coates et al.
Revised: Even with the help of horizontal data augmentation, the feature extractor Coates et al.

Original: (2011) can only achieve a test accuracy of 85.6% using 256, 000 image patches, or 85.0% using 32, 000 image patches.
Revised: (2011) can only achieve a test accuracy of 85.6% using 256, 000 image patches, or 85.0% using 32, 000 image patches.

Original: Here we use signiﬁcantly less image patches (only 2048 ) but achieve a much better performance, with the help of CNTK and CNN-GP .
Revised: Here we use signiﬁcantly less image patches (only 2048 ) but achieve a much better performance, with the help of CNTK and CNN-GP .

Original: In particular, we achieve a performance of 88.92% on CIFAR-10, matching the performance of
Revised: In particular, we achieve a performance of 88.91% on CIFAR-10, matching the performance of

Original: AlexNet on the same dataset.
Revised: AlexNet on the same dataset.

Original: In the setting reported in Coates et al.
Revised: In the setting reported in Coates et al.

Original: (2011), increasing the number of sampled image patches will further improve the performance.
Revised: (2011), increasing the number of sampled image patches will further improve the performance.

Original: Here we also conjecture that in our setting, further increasing the number of sampled image patches can improve the performance and get close to modern CNN s. However, due the limitation on computational resources, we leave exploring the effect of number of sampled image patches as a future research direction.
Revised: Here we also conjecture that in our setting, further increasing the number of sampled image patches can improve the performance and get close to modern CNN s. However, due the limitation on computational resources, we leave exploring the effect of number of sampled image patches as a future research direction.

Original: 6. E XPERIMENTS ON CNN WITH BBlur
Revised: 6. E XPERIMENTS ON CNN WITH B OX F ILTERING L AYER

Original: In Figure 2, we verify the effectiveness of BBlur on a 10-layer CNN (with Batch Normalization) on
Revised: In Figure 1, we verify the effectiveness of BF on a 10-layer CNN (with Batch Normalization) on

Original: CIFAR-10.
Revised: CIFAR-10.

Original: The setting of this experiment is reported in Appendix F. Our network structure has no pooling layer except for the BBlur layer before the last fully-connected layer.
Revised: The setting of this experiment is reported in Appendix G. Our network structure has no pooling layer except for the BF layer before the last fully-connected layer.

Original: The fully-connected layer is ﬁxed during the training.
Revised: The fully-connected layer is ﬁxed during the training.

Original: Our experiment illustrates that even with a ﬁxed last FC layer, using 
Revised: Our experiment illustrates that even with a ﬁxed last FC layer, using [Table] [Table]

Original:  GAP could improve the performance of CNN , and challenges the conjecture that GAP reduces the number of parameters in the last fully-connected layer and thus avoids overﬁtting.
Revised: [Table] [Table] [Table] GAP could improve the performance of CNN .

Original: Our experiments also show that BBlur with appropriate choice of c achieves better performance than GAP . 
Revised: Our experiments also show that BF with appropriate choice of c achieves better performance than GAP . [Figure] [Table] [Figure]

Original: C ONCLUSION
Revised: [Table] [Figure] [Table] [Figure]

Original:  In this paper, inspired by the connection between full translation data augmentation and GAP , we derive a new operation, LAP , on CNTK and CNN-GP , which consistently improves the performance on image classiﬁcation tasks.
Revised: [Table] [Figure]  C ONCLUSION In this paper, inspired by the connection between full translation data augmentation and GAP , we derive a new operation, LAP , on CNTK and CNN-GP , which consistently improves the performance on image classiﬁcation tasks.

Original: Combining CNN-GP with LAP and the pre-processing technique proposed by Coates et al.
Revised: Combining CNN-GP with LAP and the pre-processing technique proposed by Coates et al.

Original: (2011), the resulting kernel achieves 89% accuracy on CIFAR-10, matching the performance of AlexNet and is the strongest classiﬁer that is not a trained neural network.
Revised: (2011), the resulting kernel achieves 89% accuracy on CIFAR-10, matching the performance of AlexNet and is the strongest classiﬁer that is not a trained neural network.

Original: Here we list a few future research directions.
Revised: Here we list a few future research directions.

Original: Is that possible to combine more modern techniqueson CNN , such as batch normalization and residual layers,with CNTK or CNN-GP , to further improve the performance?
Revised: Is it possible to develop analogs of CNTK or CNNGP incorporating modern techniques such as batch norm and residual layers, to further improve the performance?

Original: Moreover, it is an interesting direction to study other components in modern CNN s through the lens of CNTK and CNN-GP .
Revised: Moreover, it is an interesting direction to study other components in modern CNN s through the lens of CNTK and CNN-GP .

Original: A F ORMAL D EFINITIONS OF CNN-GP AND CNTK
Revised: A F ORMAL D EFINITIONS OF CNN-GP AND CNTK

Original: We add some additional notations.
Revised: We add some additional notations.

Original: Let I be the identity matrix, and [ n ] = { 1 , 2 , . . . , n } .
Revised: Let I be the identity matrix, and [ n ] = { 1 , 2 , . . . , n } .

Original: Let e i be an indicator vector with i -th entry being 1 and other entries being 0 , and let 1 denote the all-one vector.
Revised: Let e i be an indicator vector with i -th entry being 1 and other entries being 0 , and let 1 denote the all-one vector.

Original: We use (cid:12) to denote the pointwise product and ⊗ to denote the tensor product.
Revised: We use (cid:12) to denote the pointwise product and ⊗ to denote the tensor product.

Original: We use diag( · ) to transform a vector to a diagonal matrix.
Revised: We use diag( · ) to transform a vector to a diagonal matrix.

Original: We use σ ( · ) to denote the activation function, such as the rectiﬁed linear unit (ReLU) function: σ ( z )
Revised: We use σ ( · ) to denote the activation function, such as the rectiﬁed linear unit (ReLU) function: σ ( z )

Original: = max { z, 0 } , and ˙ σ ( · ) to denote the derivative of σ ( · ) .
Revised: = max { z, 0 } , and ˙ σ ( · ) to denote the derivative of σ ( · ) .

Original: We set c σ = 2 .
Revised: We set c σ = 2 .

Original: Denote by N ( µ , Σ ) the Gaussian distribution with mean µ and covariance Σ .
Revised: Denote by N ( µ , Σ ) the Gaussian distribution with mean µ and covariance Σ .

Original: Equation equation 2 shows patch [ w ∗ x ] ij depends on [ x ] i − q − 12 : i + q − 12 ,j − q − 12 : j + q − 12 .
Revised: Equation equation 2 shows patch [ w ∗ x ] ij depends on [ x ] i − q − 12 : i + q − 12 ,j − q − 12 : j + q − 12 .

Original: For ( i, j, i (cid:48) , j (cid:48) )
Revised: For ( i, j, i (cid:48) , j (cid:48) )

Original: ∈
Revised: ∈

Original: [ P ] ×
Revised: [ P ] ×

Original: [ Q ] × [ P ] × [ Q ] , deﬁne
Revised: [ Q ] × [ P ] × [ Q ] , deﬁne

Original: [Equation]
Revised: [Equation]

Original: Now we deﬁne the convolution operation.
Revised: Now we deﬁne the convolution operation.

Original: For a convolutional ﬁlter w ∈ R q × q and an image x ∈ R P × Q , the convolution operator is deﬁned as
Revised: For a convolutional ﬁlter w ∈ R q × q and an image x ∈ R P × Q , the convolution operator is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Now we formally deﬁne CNN .
Revised: Now we formally deﬁne CNN .

Original: • Let x (0) = x ∈ R P × Q × C (0) be the input image where C (0) is the initial number of channels.
Revised: • Let x (0) = x ∈ R P × Q × C (0) be the input image where C (0) is the initial number of channels.

Original: • For h = 1 , . . .
Revised: • For h = 1 , . . .

Original: , L , β = 1 , . . .
Revised: , L , β = 1 , . . .

Original: , C ( h ) , the intermediate outputs are deﬁned as
Revised: , C ( h ) , the intermediate outputs are deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: each W ( h ) ( α ) , ( β ) ∈ R q × q is a ﬁlter with Gaussian initialization and b ( β ) is a bias term with Gaussian initialization scaled by γ .
Revised: each W ( h ) ( α ) , ( β ) ∈ R q × q is a ﬁlter with Gaussian initialization and b ( β ) is a bias term with Gaussian initialization scaled by γ .

Original: CNN-GP and CNTK
Revised: CNN-GP and CNTK

Original: • For α = 1 , . . .
Revised: • For α = 1 , . . .

Original: , C (0) , ( i, j, i (cid:48) , j (cid:48) )
Revised: , C (0) , ( i, j, i (cid:48) , j (cid:48) )

Original: ∈
Revised: ∈

Original: [ P ] ×
Revised: [ P ] ×

Original: [ Q ] × [ P ] × [ Q ] , deﬁne K (0)( α ) ( x , x (cid:48) )
Revised: [ Q ] × [ P ] × [ Q ] , deﬁne K (0)( α ) ( x , x (cid:48) )

Original: =
Revised: =

Original: x ( α ) ⊗ x (cid:48)
Revised: x ( α ) ⊗ x (cid:48)

Original: ( α ) and (cid:104) Σ (0) ( x , x (cid:48) )
Revised: ( α ) and (cid:104) Σ (0) ( x , x (cid:48) )

Original: (cid:105) ij,i (cid:48)
Revised: (cid:105) ij,i (cid:48)

Original: j (cid:48)
Revised: j (cid:48)

Original: = 1 qC
Revised: = 1 qC

Original: [Equation]
Revised: [Equation]

Original: 0) (cid:88)
Revised: 0) (cid:88)

Original: α = tr
Revised: α = tr

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: α )
Revised: α )

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: x (cid:48)
Revised: x (cid:48)

Original: [Equation]
Revised: [Equation]

Original: • For h ∈ [ L ] , – For ( i, j, i (cid:48) , j (cid:48)
Revised: • For h ∈ [ L ] , – For ( i, j, i (cid:48) , j (cid:48)

Original: [Table]
Revised: [Table]

Original: [ P ]
Revised: [ P ]

Original: [Table]
Revised: [Table]

Original: [ Q ] × [ P ] ×
Revised: [ Q ] × [ P ] ×

Original: [ Q ] , deﬁne
Revised: [ Q ] , deﬁne

Original: [Table]
Revised: [Table]

Original: ( x
Revised: ( x

Original: [Equation]
Revised: [Equation]

Original: x (cid:48)
Revised: x (cid:48)

Original: [Equation]
Revised: [Equation]

Original: (cid:17)
Revised: (cid:17)

Original: [Equation]
Revised: [Equation]

Original: – Deﬁne Σ ( h ) ( x , x (cid:48) )
Revised: – Deﬁne Σ ( h ) ( x , x (cid:48) )

Original: ∈ R P × Q × P × Q , for ( i, j, i (cid:48) , j (cid:48) )
Revised: ∈ R P × Q × P × Q , for ( i, j, i (cid:48) , j (cid:48) )

Original: ∈
Revised: ∈

Original: [ P ] × [ Q ]
Revised: [ P ] × [ Q ]

Original: ×
Revised: ×

Original: [ P ]
Revised: [ P ]

Original: ×
Revised: ×

Original: [ Q
Revised: [ Q

Original: [Equation]
Revised: [Equation]

Original: Note that Σ ( x , x (cid:48) ) and ˙ Σ ( x , x (cid:48) ) share similar structures as their NTK counterparts (Jacot et al., 2018).
Revised: Note that Σ ( x , x (cid:48) ) and ˙ Σ ( x , x (cid:48) ) share similar structures as their NTK counterparts (Jacot et al., 2018).

Original: The only difference is that we have one more step, taking the trace over patches.
Revised: The only difference is that we have one more step, taking the trace over patches.

Original: This step represents the convolution operation in the corresponding CNN .
Revised: This step represents the convolution operation in the corresponding CNN .

Original: Next, we can use a recursion to compute the ﬁnal kernel value.
Revised: Next, we can use a recursion to compute the ﬁnal kernel value.

Original: First, we deﬁne Θ (0) ( x , x (cid:48) )
Revised: First, we deﬁne Θ (0) ( x , x (cid:48) )

Original: =
Revised: =

Original: Σ (0) ( x , x (cid:48) ) .
Revised: Σ (0) ( x , x (cid:48) ) .

Original: For h = 1 , . . .
Revised: For h = 1 , . . .

Original: , L and ( i, j, i (cid:48) , j (cid:48) ) ∈
Revised: , L and ( i, j, i (cid:48) , j (cid:48) ) ∈

Original: [ P ] ×
Revised: [ P ] ×

Original: [ Q ] × [ P ] × [ Q ] , we deﬁne (cid:104) Θ ( h ) ( x , x (cid:48) )
Revised: [ Q ] × [ P ] × [ Q ] , we deﬁne (cid:104) Θ ( h ) ( x , x (cid:48) )

Original: (cid:105) ij,i (cid:48)
Revised: (cid:105) ij,i (cid:48)

Original: j (cid:48)
Revised: j (cid:48)

Original: [Equation]
Revised: [Equation]

Original: B A DDITIONAL DEFINITION AND PROOF FOR S ECTION
Revised: B A DDITIONAL DEFINITION AND PROOF FOR S ECTION

Original: Deﬁnition B.1 (Group) .
Revised: Deﬁnition B.1 (Group) .

Original: ( G , ◦ ) is a group , if and only if
Revised: ( G , ◦ ) is a group , if and only if

Original: each element g ∈ G is a operator: R P × Q × C → R P × Q × C ; 2.
Revised: each element g ∈ G is a operator: R P × Q × C → R P × Q × C ; 2.

Original: ∀ g 1 , g 2 ∈ G , g 1 ◦ g 2 ∈ G , where ( g 1 ◦ g 2 )( x ) is deﬁned as g 1 ( g 2 ( x )) .
Revised: ∀ g 1 , g 2 ∈ G , g 1 ◦ g 2 ∈ G , where ( g 1 ◦ g 2 )( x ) is deﬁned as g 1 ( g 2 ( x )) .

Original: ∀ g 1 , g 2 , g 3 ∈ G , ( g 1 ◦ g 2 ) ◦ g 

Revised: ∀ g 1 , g 2 , g 3 ∈ G , ( g 1 ◦ g 2 ) ◦ g

Original: = g 1 ◦
Revised: = g 1 ◦

Original: ( g 2 ◦ g 3 ) .
Revised: ( g 2 ◦ g 3 ) .

Original: ∃ e ∈ G , such that ∀ g ∈ G , e ◦ g
Revised: ∃ e ∈ G , such that ∀ g ∈ G , e ◦ g

Original: =
Revised: =

Original: g ◦ e = g .
Revised: g ◦ e = g .

Original: ∀ g 1 ∈ G , ∃ g 2 ∈ G , such that g 1 ◦ g 2 =
Revised: ∀ g 1 ∈ G , ∃ g 2 ∈ G , such that g 1 ◦ g 2 =

Original: g 2 ◦ g 1 = e .
Revised: g 2 ◦ g 1 = e .

Original: We denote g 2 as the inverse of g 1 , namely, g − 1 1 .
Revised: We denote g 2 as the inverse of g 1 , namely, g − 1 1 .

Original: Proof of Theorem 4.1.
Revised: Proof of Theorem 4.1.

Original: Since we assume K G X and K X G are invertible, both α and (cid:101) α are uniquely deﬁned.
Revised: Since we assume K G X and K X G are invertible, both α and (cid:101) α are uniquely deﬁned.

Original: Now we claim (cid:101)
Revised: Now we claim (cid:101)

Original: α g = { (cid:101)
Revised: α g = { (cid:101)

Original: α
Revised: α

Original: i,g } i ∈ [ N ] ∈ R N is equal to α |G| for all g ∈ G .
Revised: i,g } i ∈ [ N ] ∈ R N is equal to α |G| for all g ∈ G .

Original: By the invariance of K under G , for all j ∈ [ N ] and g (cid:48) ∈ G ,
Revised: By the invariance of K under G , for all j ∈ [ N ] and g (cid:48) ∈ G ,

Original: [Equation]
Revised: [Equation]

Original: Note that (cid:101) α is deﬁned as the unique solution of K X G (cid:101)
Revised: Note that (cid:101) α is deﬁned as the unique solution of K X G (cid:101)

Original: α = y G , the claim has been veriﬁed.
Revised: α = y G , the claim has been veriﬁed.

Original: Similarly, we have
Revised: Similarly, we have

Original: [Equation]
Revised: [Equation]

Original: C E QUIVALENCE B ETWEEN LAP AND B OX B LUR L AYER .
Revised: C E QUIVALENCE B ETWEEN LAP AND B OX F ILTERING L AYER .

Original: For a CNN with a box blur layer before the ﬁnal fully-connected layer, the ﬁnal output is deﬁned as f ( θ , x ) = (cid:80) C ( L )
Revised: For a CNN with a box ﬁltering layer before the ﬁnal fully-connected layer, the ﬁnal output is deﬁned as f ( θ , x ) = (cid:80) C ( L )

Original: α =1 (cid:68) W
Revised: α =1 (cid:68) W

Original: ( L +1)
Revised: ( L +1)

Original: ( α ) , BBlur (cid:16) x ( L ) ( α ) (cid:17)(cid:69) , where x ( L ) ( α
Revised: ( α ) , BF (cid:16) x ( L ) ( α ) (cid:17)(cid:69) , where x ( L ) ( α

Original: ) ∈ R P × Q , and W ( L +1)
Revised: ) ∈ R P × Q , and W ( L +1)

Original: ( α ) ∈ R P × Q is the weight of the last fully-connected layer.
Revised: ( α ) ∈ R P × Q is the weight of the last fully-connected layer.

Original: Now we establish the equivalence between BBlur and LAP on CNTK .
Revised: Now we establish the equivalence between BF and LAP on CNTK .

Original: The equivalence on CNNGP can be derived similarly.
Revised: The equivalence on CNNGP can be derived similarly.

Original: Let Θ BBlur ( x , x (cid:48) )
Revised: Let Θ BF ( x , x (cid:48) )

Original: ∈ R [ P ] ×
Revised: ∈ R [ P ] ×

Original: [ Q ] × [ P ] × [ Q ] be the CNTK kernel of
Revised: [ Q ] × [ P ] × [ Q ] be the CNTK kernel of

Original: BBlur (cid:16) x ( L ) ( α ) (cid:17) .
Revised: BF (cid:16) x ( L ) ( α ) (cid:17) .

Original: Since BBlur is just a linear operation, we have [
Revised: Since BF is just a linear operation, we have [

Original: [Equation]
Revised: [Equation]

Original: By the formula of the output kernel value for CNTK without GAP , we obtain tr ( Θ BBlur ( x , x (cid:48) )) =
Revised: By the formula of the output kernel value for CNTK without GAP , we obtain tr ( Θ BF ( x , x (cid:48) )) =

Original: [Equation]
Revised: [Equation]

Revised: [Figure]

Original: D E XPERIMENTAL S ETUP IN S ECTION
Revised: D E XPERIMENTAL S ETUP IN S ECTION

Original: For both CIFAR-10 and Fashion-MNIST we use the full training set and report the test accuracy on the full test set.
Revised: For both CIFAR-10 and Fashion-MNIST we use the full training set and report the test accuracy on the full test set.

Original: Throughout this section we only consider 3 × 3 convolutional ﬁlters with stride 1 and no dilation.
Revised: Throughout this section we only consider 3 × 3 convolutional ﬁlters with stride 1 and no dilation.

Original: In the convolutional layers in CNTK and CNN-GP , we use zero padding with pad size 1 to ensure the input of each layer has the same size.
Revised: In the convolutional layers in CNTK and CNN-GP , we use zero padding with pad size 1 to ensure the input of each layer has the same size.

Original: We use zero padding for LAP throughout the experiment.
Revised: We use zero padding for LAP throughout the experiment.

Original: We perform standard preprocessing (mean subtraction and standard deviation division) for all images.
Revised: We perform standard preprocessing (mean subtraction and standard deviation division) for all images.

Original: In all experiments, we perform kernel ridge regression to utilize the calculated kernel values 3 .
Revised: In all experiments, we perform kernel ridge regression to utilize the calculated kernel values 4 .

Original: We normalize the kernel matrices so that all diagonal entries are ones.
Revised: We normalize the kernel matrices so that all diagonal entries are ones.

Original: Equivalently, we ensure all features have unit norm in RKHS.
Revised: Equivalently, we ensure all features have unit norm in RKHS.

Original: Since the resulting kernel matrices are usually ill-conditioned, we set the regularization term λ = 5 × 10 − 5 , to make inverting kernel matrices numerically stable.
Revised: Since the resulting kernel matrices are usually ill-conditioned, we set the regularization term λ = 5 × 10 − 5 , to make inverting kernel matrices numerically stable.

Original: We use one-hot encodings of the labels as regression targets.
Revised: We use one-hot encodings of the labels as regression targets.

Original: We use scipy.linalg.solve to solve the corresponding kernel ridge regression problem.
Revised: We use scipy.linalg.solve to solve the corresponding kernel ridge regression problem.

Original: The kernel value of CNTK and CNN-GP are calculated using the CuPy package.
Revised: The kernel value of CNTK and CNN-GP are calculated using the CuPy package.

Original: We write native
Revised: We write native

Original: CUDA codes to speed up the calculation of the kernel values.
Revised: CUDA codes to speed up the calculation of the kernel values.

Original: All experiments are performed on
Revised: All experiments are performed on

Original: Amazon Web Services (AWS), using (possibly multiple) NVIDIA Tesla V100 GPUs.
Revised: Amazon Web Services (AWS), using (possibly multiple) NVIDIA Tesla V100 GPUs.

Original: For efﬁciency considerations, all kernel values are computed with 32-bit precision.
Revised: For efﬁciency considerations, all kernel values are computed with 32-bit precision.

Original: One unique advantage of the dynamic programming algorithm for calculating CNTK and CNNGP is that we do not need repeat experiments for, say, different values of c in LAP and different depths.
Revised: One unique advantage of the dynamic programming algorithm for calculating CNTK and CNNGP is that we do not need repeat experiments for, say, different values of c in LAP and different depths.

Original: With our highly-optimized native CUDA codes, we spend roughly 1,000 GPU hours on calculating all kernel values for each dataset.
Revised: With our highly-optimized native CUDA codes, we spend roughly 1,000 GPU hours on calculating all kernel values for each dataset.

Revised: SVM but found it signiﬁcantly degrading the performance, and thus do not include the results.

Original: E T EST A CCURACY OF CNTK AND CNN-GP ON F ASHION -MNIST
Revised: E T EST A CCURACY OF CNTK AND CNN-GP ON F ASHION -MNIST

Revised: [Table]

Revised: [Table]

Revised: F V ALIDATION A CCURACY OF CNTK AND CNN-GP ON CIFAR-10 AND F ASHION -MNIST

Original: c d     92 . 
Revised: c d     64 . 26 (68 . 42) 64 . [Table] [Table] [Table]

Original: 97 (93 . 19) c
Revised: [Table] [Table] [Table] [Table]

Original: d     91 .
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: F S ETTING OF THE E XPERIMENT IN S ECTION 6. 
Revised: G S ETTING OF THE E XPERIMENT IN S ECTION 6.

Original: The total number of training epochs is 80, and the learning rate is 0.1 initially, decayed by 10 at epoch 40 and 60 respectively.
Revised: The total number of training epochs is 80, and the learning rate is 0.1 initially, decayed by 10 at epoch 40 and 60 respectively.

Original: The momentum is 0.9 and the weight decay factor is 0.0005.
Revised: The momentum is 0.9 and the weight decay factor is 0.0005.

Original: In
Revised: In

Original: Figure 2, the blue line reports the average test accuracy of the last 10 epochs, while the red line reports the best test accuracy of the total 80 epochs.
Revised: Figure 1, the blue line reports the average test accuracy of the last 10 epochs, while the red line reports the best test accuracy of the total 80 epochs.

Original: Each experiment is repeated for 3 times.
Revised: Each experiment is repeated for 3 times.

Original: We use circular padding for both convolutional layers and the BBlur layer.
Revised: We use circular padding for both convolutional layers and the BF layer.

Original: The last data point with largest x -coordinate reported in Figure 2 corresponds to GAP .
Revised: The last data point with largest x -coordinate reported in Figure 1 corresponds to GAP .