T HE U NREASONABLE E FFECTIVENESS OF P ATCHES IN D EEP C ONVOLUTIONAL K ERNELS M ETHODS
A BSTRACT
A recent line of work showed that various forms of convolutional kernel methods can be competitive with standard supervised deep convolutional networks on datasets like CIFAR-10, obtaining accuracies in the range of 87 − 90% while being more amenable to theoretical analysis.
In this work, we highlight the importance of a data-dependent feature extraction step that is key to the obtain good performance in convolutional kernel methods.
This step typically corresponds to a whitened dictionary of patches, and gives rise to a data-driven convolutional kernel methods.
We extensively study its effect, demonstrating it is the key ingredient for high performance of these methods.
Speciﬁcally, we show that one of the simplest instances of such kernel methods, based on a single layer of image patches followed by a linear classiﬁer is already obtaining classiﬁcation accuracies on CIFARin the same range as previous more sophisticated convolutional kernel methods.
We scale this method to the challenging ImageNet dataset, showing such a simple approach can exceed all existing non-learned representation methods.
This is a new baseline for object recognition without representation learning methods, that initiates the investigation of convolutional kernel models on ImageNet.
We conduct experiments to analyze the dictionary that we used, our ablations showing they exhibit low-dimensional properties.
I NTRODUCTION
Understanding the success of deep convolutional neural networks on images remains challenging because images are high-dimensional signals and deep neural networks are highly-non linear models with a substantial amount of parameters: yet, the curse of dimensionality is seemingly avoided by these models.
This problem has received a plethora of interest from the machine learning community.
One approach taken by several authors (Mairal, 2016; Li et al., 2019; Shankar et al., 2020; Lu et al., 2014) has been to construct simpler models with more tractable analytical properties (Jacot et al., 2018; Rahimi and Recht, 2008), that still share various elements with standard deep learning models.
Those simpler models are based on kernel methods with a particular choice of kernel that provides a convolutional representation of the data.
In general, these methods are able to achieve reasonable performances on the CIFAR-10 dataset.
However, despite their simplicity compared to deep learning models, it remains unclear which ones of the multiple ingredients they rely on are essential.
Moreover, due to their computational cost, it remains open to what extend they achieve similar performances on more complex datasets such as ImageNet.
In this work, we show that an additional implicit ingredient, common to all those methods, consists in a data-dependent feature extraction step that makes the convolutional kernel data-driven (as opposed to purely handcrafted) and is key for obtaining good performances.
Data driven convolutional kernels compute a similarity between two images x and y , using both their translation invariances and statistics from the training set of images X .
In particular, we focus on similarities K that are obtained by ﬁrst standardizing a representation Φ of the input images and then feeding it to a predeﬁned kernel k :
[Equation]
where a rescaling and shift is (potentially) performed by a diagonal afﬁne operator
L =
L (Φ , X ) and is mainly necessary for the optimization step Jin et al.
(2009): it is typically a standardization.
The kernel K ( x, y ) is said to be data-driven if Φ depends on training set X , and data-independent otherwise.
This, for instance, is the case if a dictionary is computed from the data (Li et al., 2019;
Mairal, 2016; Mairal et al., 2014) or a ZCA (Shankar et al., 2020) is incorporated in this representation.
The convolutional structure of the kernel K can come either from the choice of the representation Φ (convolutions with a dictionary of patches (Coates et al., 2011)) or by design of the predeﬁned kernel k (Shankar et al., 2020), or a combination of both (Li et al., 2019; Mairal, 2016).
One of the goal of this paper is to clearly state that kernel methods for vision do require to be data-driven and this is explicitly responsible for their success.
We thus investigate, to what extent this common step is responsible for the success of those methods, via a shallow model.
Our methodology is based on ablation experiments: we would like to measure the effect of incorporating data, while reducing other side effects related to the design of Φ , such as the depth of Φ or the implicit bias of a potential optimization procedure.
Consequently, we focus on 1-hidden layer neural networks of any widths, which have favorable properties, like the ability to be a universal approximator under non-restrictive conditions.
The output linear layer shall be optimized for a classiﬁcation task, and we consider ﬁrst layers which are predeﬁned and kept ﬁxed, similarly to Coates et al.
We will see below that simply initializing the weights of the ﬁrst layer with whitened patches leads to a signiﬁcant improvement of performances, compared to a random initialization, a wavelet initialization or even a learning procedure.
This patch initialization is used by several works (Li et al., 2019; Mairal, 2016) and is implicitly responsible for their good performances.
Other works rely on a whitening step followed by very deep kernels (Shankar et al., 2020), yet we noticed that this was not sufﬁcient in our context.
Here, we also try to understand why incorporating whitened patches is helpful for classiﬁcation.
Informally, this method can be thought as one of the simplest possible in the context of deep convolutional kernel methods, and we show that the depth or the non-linearities of such kernels play a minor role compared to the use of patches.
In our work, we decompose and analyze each step of our feature design, on gold-standard datasets and ﬁnd that a method based solely on patches and simple non-linearities is actually a strong baseline for image classiﬁcation.
We investigate the effect of patch-based pre-processing for image classiﬁcation through a simple baseline representation that does not involve learning (up to a linear classiﬁer) on both CIFARand ImageNet datasets: the path from CIFAR-10 to ImageNet had never been explored until now in this context.
Thus, we believe our baseline to be of high interest for understanding ImageNet’s convolutional kernel methods, which almost systematically rely on a patch (or descriptor of a patch) encoding step.
Indeed, this method is straightforward and involves limited ad-hoc feature engineering compared to deep learning approach: here, contrary to (Mairal, 2016;
Coates et al., 2011; Recht et al., 2019; Shankar et al., 2020;
Li et al., 2019)
we employ modern techniques that are necessary for scalability (from thousands to million of samples) but can still be understood through the lens of kernel methods (e.g., convolutional classiﬁer, data augmentation, ...).
Our work allows to understand the relative improvement of such encoding step and we show that our method is a challenging baseline for classiﬁcation on Imagenet: we outperform by a large margin the classiﬁcation accuracy of former attempts to get rid of representation learning on the large-scale ImageNet dataset.
While the literature provides a detailed analysis of the behavior of a dictionary of patches for image compression (Wallace, 1992), texture synthesis (Efros and Leung, 1999) or image inpainting (Criminisi et al., 2004), we have a limited knowledge and understanding of it in the context of image classiﬁcation.
The behavior of those dictionaries of patches in some classiﬁcation methods is still not well understood, despite often being the very ﬁrst component of many classic vision pipelines (Perronnin et al., 2010; Lowe, 2004; Oyallon et al., 2018b).
Here, we proposed a reﬁned analysis: we deﬁne a Euclidean distance between patches and we show that the decision boundary between image classes can be approximated using a rough description of the image patches neighborhood: it is implied for instance by the fame low-dimensional manifold hypothesis (Fefferman et al., 2016).
Our paper is structured as follows:
ﬁrst, we discuss the related works in Sec. 2.
Then, Sec. 3 explains precisely how our visual representation is built.
In Sec. 4, we present experimental results on the vision datasets CIFAR-10 and the large scale ImageNet.
The ﬁnal Sec. 4.3 is a collection of numerical experiments to understand better the dictionary of patches that we used.
Our code as well as commands to reproduce our results are available here: https://github.com/louity/patches .
R ELATED WORK
The seminal works by Coates et al.
(2011) and Coates and Ng (2011) study patch-based representations for classiﬁcation on CIFAR-10.
They set the ﬁrst baseline for a single-layer convolutional network initialized with random patches, and they show it can achieve a non-trivial performance ( ∼ 80% ) on the CIFAR-10 dataset.
Recht et al.
(2019) published an implementation of this technique and conducted numerous experiments with hundreds of thousands of random patches, improving the accuracy ( ∼ 85% ) on this dataset.
However, both works lack two key ingredients: online optimization procedure (which allows to scale up to ImageNet) and well-designed linear classiﬁer (as we propose a factorization of our linear classiﬁer).
Recently, (Li et al., 2019; Shankar et al., 2020) proposed to handcraft kernels, combined with deep learning tools, in order to obtain high-performances on CIFAR-10.
Those performances match standard supervised methods ( ∼ 90% ) which involve end-to-end learning of deep neural networks.
Note that the line of work (Li et al., 2019; Shankar et al., 2020; Mairal, 2016) employs a wellengineered combination of patch-extracted representation and a cascade of kernels (possibly some neural tangent kernels).
While their works suggest that patch extraction is crucial, the relative improvement due to basic-hyper parameters such as the number of patches or the classiﬁer choice is unclear, as well as the limit of their approach to more challenging dataset.
We address those issues.
From a kernel methods perspective, a dictionary of random patches can be viewed as the building block of a random features method (Rahimi and Recht, 2008) that makes kernel methods computationally tractable.
Rudi et al.
(2017) provided convergence rates and released an efﬁcient implementation of such a method.
However, previously mentioned kernel methods (Mairal, 2016; Li et al., 2019;
Shankar et al., 2020) have not been tested on ImageNet to our knowledge.
Simple methods involving solely a single-layer of features have been tested on the ImageNetdataset 1 , using for example SIFT, color histogram and Gabor texture encoding of the image with
K -nearest neighbors, yet there is a substential gap in accuracy that we attempt to ﬁll in this work on ImageNet-2012 (or simply ImageNet).
We note also that CNNs with random weights have been tested on ImageNet, yielding to low accuracies ( ∼ 20% top-1, (Arandjelovic et al., 2017)).
The Scattering Transform (Mallat, 2012) is also a deep non-linear operator that does not involve representation learning, which has been tested on ImageNet ( ∼ 45% top-5 accuracy (Zarka et al., 2019) and CIFAR-10 ( ∼ 80% , (Oyallon and Mallat, 2015)) and is related to the HoG and SIFT transforms (Oyallon et al., 2018a).
Some works also study directly patch encoders that achieve competitive accuracy on ImageNet but involve deep cascade of layers that are difﬁcult to interpret (Oyallon et al., 2017; Zarka et al., 2019;
Brendel et al., 2019).
Here, we focus on shallow classiﬁers.
M ETHOD
We ﬁrst introduce our preliminary notations to describe an image.
A patch p of size P of a larger image x , is a restriction of that image to a squared domain of surface P 2 .
We denote by N 2 the size of the natural image x and require that P ≤ N .
Hence, for a spatial index i of the image, p i,x represents the patch of image x located at i .
We further introduce the collection of all overlapping patches of that image, denoted by: P x = { p i,x , i ∈ I} where I is a spatial index set such that |I| =
( N − P + 1) 2 .
Fig.
1 corresponds to an overview of our classiﬁcation pipeline that consist of 3 steps: an initial whitening step of a dictionary
D of random patches, followed by a nearest neighbor quantization of images patches via D that are ﬁnally spatially averaged.
Whitening We describe the single pre-processing step that we used on our image data, namely a whitening procedure on patches.
Here, we view natural image patches of size P 2 as samples from a random vector of mean µ and covariance Σ .
We then consider whitening operators which act at the level of each image patch by ﬁrst subtracting its mean µ then applying the linear transformation
W =
( λ I + Σ) − 1 / 2 to the centered patch.
The additional whitening regularization with parameter λ was used to avoid ill-conditioning effects.
[Figure]
[Table]
/YgQJGHfYEo5Nwy9FA4MWSCIIqTkh1xHEI0gSNsSfShh7kTZ4cl2ol0BtowYPL5QsvcvxMx9Difeq7sTG/ki7XU/K9mRWLYcGLih5HAPpotGkZUE4GWRqUNCMNI0KkEiBiRt2poDBlEQgY6t0WQyWOairGYwTJ0zYoh+bZabl3m+RTBETgGp8AAddACN6ANOgCBJ/ACXsGb8qy8K5/K16y1oOQzh2BOys8vGACmRQ==</latexit> …
The whitening operation is deﬁned up to an isometry, but the Euclidean distance between whitened patches (i.e., the Mahanobolis distance (Chandra et al., 1936)) is not affected by the choice of such isometry (choices leading to PCA, ZCA, ...), as discussed in Appendix A.
In practice, the mean and covariance are estimated empirically from the training set to construct the whitening operators.
For the sake of simplicity, we only consider whitened patches, and unless explicitly stated, we assume that each patch p is already whitened, which holds in particular for the collection of patches in
P x of any image x .
Once this whitening step is performed, the Euclidean distance over patches is approximatively isotropic and is used in the next section to represent our signals.
Q -Nearest Neighbors on patches The basic idea of this algorithm is to compare the distances between each patch of an image and a ﬁxed dictionary of patches D , with size |D| that is the number of patches extracted.
Note that we also propose a variant where we simply use a soft-assignment operator.
For a ﬁxed dataset, this dictionary D is obtained by uniformly sampling patches from images over the whole training set.
We augment D into ∪ d ∈D { d, − d } because it allows the dictionary of patches to be contrast invariant and we observe it leads to better classiﬁcation accuracies; we still refer to it as D .
An illustration is given by Fig.
Once the dictionary D is ﬁxed, for each patch p i,x we consider the set C i,x of pairwise distances C i,x = {(cid:107) p i,x − d (cid:107) , d ∈ D} .
For each whitened patch we encode the Q -Nearest Neighbors of p i,x from the set
D , for some Q ∈ N .
More formally, we consider τ i,x the Q -th smallest element of
C i,x , and we deﬁne the Q -Nearest Neighbors binary encoding as follow, for ( d, i ) ∈ D × I :
[Equation]
Eq. 2 can be viewed as a Vector Quantization (VQ) step with hard-assignment (Coates and Ng, 2011).
The representation φ encodes the patch neighborhood in a subset of randomly selected patches and can be seen as a crude description of the topological geometry of the image patches.
Moreover, it allows to view the distance between two images x, y as a Hamming distance between the patches neighborhood encoding as:
[Equation]
In order to reduce the computational burden of our method, we perform an intermediary averagepooling step.
Indeed, we subdivide I in squared overlapping regions I j ⊂ I , leading to the representation Φ deﬁned, for d [Equation] by
[Equation]
Hence, the resulting kernel is simply given by K ( x, y )
= (cid:104) Φ( x ) , Φ( y ) (cid:105) .
Implementation details can be found in Appendix B.
The next section describes our classiﬁcation pipeline, as we feed our representation Φ to a linear classiﬁer on challenging datasets.
E XPERIMENTS
We train shallow classiﬁers, i.e. linear classiﬁer and 1-hidden layer CNN ( 1-layer ) on top of our representation Φ on two major image classiﬁcation datasets, CIFAR-10 and ImageNet, which consist respectively of 50 k small and 1 .
2 M large color images divided respectively into 10 and 1 k classes.
For training, we systematically used mini-batch SGD with momentum of 0.9, no weight decay and using the cross-entropy loss.
Classiﬁer parametrization In each experiments, the spatial subdivisions I j are implemented as an average pooling with kernel size k 1 and stride s 1 .
We then apply a 2D batch-normalization (Ioffe and Szegedy, 2015) in order to standardize our features on the ﬂy before feeding them to a linear classiﬁer.
In order to reduce the memory footprint of this linear classiﬁer (following the same line of idea of a "bottleneck" (He et al., 2016)), we factorize it into two convolutional operators.
The ﬁrst one with kernel size k 2 and stride 1 reduces the number of channels from D to c 2 and the second one with kernel size k 3 and stride 1 outputs a number of channel equal to the number of image classes.
Then we apply a global average pooling.
For the 1-hidden layer experiment, we simply add a ReLU non linearity between the ﬁrst and the second convolutional layer.
4. CIFAR-
Implementation details Our data augmentation consists in horizontal random ﬂips and random crops of size 32 2 after reﬂect-padding with 4 pixels.
For the dictionary, we choose a patch size of
P = 6 and tested various sizes of the dictionary |D| and whitening regularization λ = 0 .
In all cases, we used Q = 0 .|D| .
The classiﬁer is trained for 175 epoch with a learning rate decay of 0.1 at epochs 100 and 150.
The initial learning rate is 0 .
003 for |D| = 2 k and 0 .
001 for larger |D| .
Single layer experiments For the linear classiﬁcation experiments, we used an average pooling of size k 1 = 5 and stride s 1 = 3 , k 2 = 1 and c 2 = 128 for the ﬁrst convolutional operator and k
=
for the second one.
Our results are reported and compared in Tab. 1a.
First, note that contrary to experiments done by Coates et al.
(2011), our methods has surprisingly good accuracy despite the hard-assignment due to VQ.
Sparse coding, soft-thresholding and orthogonal matching pursuit based representations used by Coates and Ng (2011)
; Recht et al.
(2019) can be seen as soft-assignment
VQ and yield comparable classiﬁcation accuracy (resp.
81.5% with 6 .
10 3 patches and 85.6% with 2 .
10 5 patches).
However, these representations contain much more information than hard-assignment [Table] [Table] [Table]
[Table]
VQ as they allow to reconstruct a large part of the signal.
We get better accuracy with only coarse topological information on the image patches, suggesting that this information is highly relevant for classiﬁcation.
To obtain comparable accuracies with a linear classiﬁer, we use a single binary encoding step compared to Mairal (2016) and we need a much smaller number of patches than Recht et al.
; Coates and Ng (2011).
Moreover, Recht et al.
(2019) is the only work in the litterature, besides us, that achieves good performance using solely a linear model with depth one.
To test the VQ importance, we replace the hard-assignment VQ implemented with a binary non-linearity 1 (cid:107) p i,x − d (cid:107)≤ τ i,x (see Eq. 2) by a soft-assignment VQ with a sigmoid function (1 + e (cid:107) p i,x − d (cid:107)− τ i,x ) − 1 .
The accuracy increases by 0 .
2% , showing that the use soft-assignment in VQ which is crucial for performance in Coates and Ng (2011) does not affect much the performances of our representation.
Importance of data-driven representations As we see in Tab.1c, the data-driven representation is crucial for good performance of handcrafted kernel classiﬁers.
We remind that a data-independent kernel is built without using the dataset, which is for instance the case with a neural network randomly initialized.
The accuracies from Shankar et al.
(2020) correspond to Myrtle5 (CNN and kernel), because the authors only report an accuracy without ZCA for this model.
As a sanity check, we consider D whose atoms are sampled from a Gaussian white noise: this step leads to a drop of 8.1%.
This is aligned with the ﬁnding of each work we compared to: performances drop if no ZCA is applied or if patches are not extracted.
Using a dictionary of size |D| = 2048 , the same model trained end-to-end (including the learning of
D ) yields to the same accuracy (- 0.1 %), showing that here, sampling patches is as efﬁcient as optimizing them.
Note that our method also outperforms linearized deep neural networks (Samarin et al., 2020), i.e. trained in a lazy regime (Chizat et al., 2019).
Non-linear classiﬁcation experiments To test the discriminative power of our features, we use a 1-hidden layer classiﬁer with ReLU non-linearity and an average pooling of size k 1 = 3 and stride s 1 = 2 , k 2 = 3 , c 2 = 2048 and k 3 = 7 .
Our results are reported and compared with other non-linear classiﬁcation methods in Tab.1b.
Using a shallow non-linear classiﬁer, our method is competitive with end-to-end trained methods (Li et al., 2019; Shankar et al., 2020; Krizhevsky et al., 2012).
This further indicates the relevance of patches neighborhood information for classiﬁcation task.
Hyper-parameter analysis CIFAR-10 is a relatively small dataset that allows fast benchmarking, thus we conducted several ablation experiments in order to understand the relative improvement due to each hyper-parameter of our pipeline.
We thus vary the size of the dictionary |D| , the patch size P , the number of nearest neighbors Q and the whitening regularization λ which are the hyper-parameters of Φ .
Results are shown in Fig.
Note that even a relatively small number of patches is competitive with much more complicated representations, such as Oyallon and Mallat (2015).
While it is possible to slightly optimize the performances according to P or Q , the ﬂuctuations remain minor compared to other factors, which indicate that the performances of our method are relatively stable w.r.t.
this set of hyper-parameters.
The whitening regularization behaves similarly to a thresholding operator on the eigenvalues of Σ 1 / 2 , as it penalizes larger eigenvalues.
Interestingly, we note that under a certain threshold, this hyper-parameter does almost not affect the classiﬁcation performances.
This goes in hand with both a fast eigenvalue decay and a stability to noise, that we discuss further in Sec. 4.3.
I MAGE N ET
Implementation details To reduce the computational overhead of our method on ImageNet, we followed the same approach as Chrabaszcz et al.
(2017): we reduce the resolution to 64 2 , instead of the standard 224 2 length.
They observed that this does not alter much the top-performances of standard models ( 5% to 10% drop of accuracy on average), and we also believe it introduces a useful dimensionality reduction, as it removes high-frequency part of images that are unstable (Mallat, 1999).
We set the patch size to P = 6 and the whitening regularization to λ = 10 − 2 .
Since ImageNet is a much larger than CIFAR-10, we restricted to |D| = 2048 patches.
As for CIFAR-10, we set Q = 0 .|D| .
The parameters of the linear convolutional classiﬁer are chosen to be: k 1 = 10 , s 1 = 6 , k 2 = 1 , c 2 = 256 , k 3 = 7 .
For the 1-hidden layer experiment, we used kernel size of k 2 = 3 for the ﬁrst convolution.
Our models are trained during 60 epochs with an initial learning rate of 0.003 decayed by a factor 10 at epochs 40 and 50.
During training, similarly to Chrabaszcz et al.
we use random ﬂip and we select random crops of size 64 , after a reﬂect-padding of size 8.
At testing, we simply resize the image to 64 .
Note this procedure differs slightly from the usual procedure, which consists in resizing images while maintaining ratios, before a random cropping.
Classiﬁcation experiments Tab.2a reports the accuracy of our method, as well as the accuracy of comparable methods.
Despite a smaller image resolution, our method outperforms by a large margin ( ∼ 10% Top5) the Scattering Transform (Mallat, 2012), which was the previous state-of-the-artmethod in the context of no-representation learning.
Note that our representation uses only 2 .
10randomly selected patches which is a tiny fraction of the billions of ImageNet patches. [Table] [Table] Brendel
[Table]
[Table] In Tab.2b [Table] we compare our performances with supervised models trained end-to-end
[Table]
which also use convolutions with small receptive ﬁelds.
Here [Table] D = 2 k .
BagNets (Brendel et al., 2019) have shown that competitive classiﬁcation accuracies can be obtained with patch-encoding that consists of 50 layers.
The performance obtained by our shallow experiment with a 1-hidden layer classiﬁer is competitive with a BagNet with similar patch-size.
It suggests once again that hard-assignment
VQ does not degrade much of the classiﬁcation information.
We also note that our approach with a linear classiﬁer outperforms supervised shallow baselines that consists of 1 or 2 hidden-layers CNN (Belilovsky et al., 2018), which indicates that a patch based representation is a non-trivial baseline.
To measure the importance of the resolution on the performances, we run a linear classiﬁcation experiment on ImageNet images with twice bigger resolution ( N = 128 2 , Q = 12 , k 1 = 20 , s 1 = 12 ).
We observe that it improves classiﬁcation performances.
Note that the patches used are in a space of dimension 432 (cid:29) 1 : this improvement is surprising since distance to nearest neighbors are known to be meaningless in high-dimension (Beyer et al., 1999).
This shows a form of low-dimensionality in the natural image patches, that we study in the next Section.
4. D ICTIONARY STRUCTURE
The performance obtained with the surprisingly simple classiﬁer hints to a low dimensional structure in the classiﬁcation problem that is exploited by the patch based classiﬁer we proposed.
This motivates us to further analyse the structure of the dictionary of patches to uncover a lower dimensional structure and to investigate how the whitening, which highly affects performance, relates to such lower-dimensional structure.
Spectrum of D
As a preliminary analysis, we propose to analyse the singular values (spectrum) of Σ 1 / 2 sorted by a decreasing order as λ 1 ≥ ... ≥ λ d ext with d ext = 3 P 2 being the extrinsic dimension (number of colored pixels in each patch).
From this spectrum, it is straightforward to compute the covariance dimension d cov of the patches deﬁned as the smallest number of dimensions needed to explain 95% of the total variance.
In other words, d cov is the smallest index such that (cid:80) d cov i =
1 λ i ≥ 0 .
95 (cid:80)
d ext i =
λ i .
Fig.
4 (top) shows the spectrum for several values of P , normalized by λ 1 on CIFAR-10 and ImageNet-32.
The ﬁrst observation is that patches from ImageNetdataset tend to be better conditioned than those from CIFAR-10 with a conditioning ratio of 10for ImageNet vs 10 3 for CIFAR-10.
This is probably due to the use of more diverse images than on CIFAR-10.
Second, note that the spectrum tends to decay at an exponential rate (linear rate in semi-logarithmic scale).
This rate decreases as the size of the patch increases (from dark brown to light brown) suggesting an increased covariance dimension for larger patches.
This is further conﬁrmed in Fig.
4(bottom-left) which shows the covariance dimension d cov as a function of the [Figure] extrinsic dimension d ext , with and without whitening.
Before whitening, this linear dimension is much smaller than the ambient dimension: whitening the patches increases the linear dimensionality of the patches, which still increases at a linear growth as a function of P 2 .
Intrinsic dimension of D
We propose to reﬁne our measure of linear dimensionality to a nonlinear measure of the intrinsic dimension.
Under the assumption of low-dimensional manifold, if the manifold is non-linear, the linear dimensionality is only an upper bound of the true dimensionality of image patches.
To get more accurate non-linear estimates [Figure] we propose to use the notion of intrinsic dimension d int introduced in (Levina and Bickel, 2004).
It relies on a local estimate of the dimension around a patch point p , obtained by ﬁnding the k -Nearest Neighbors to this patch in the whole dataset and estimating how much the Euclidean distance τ k ( p ) between the k -Nearest Neighbor and patch p varies as k increases up to K ∈ N :
[Equation]
In high dimensional spaces, it is possible to have many neighbors that are equi-distant to p , thus τ k ( p ) would barely vary as k increases.
As a result the estimate d int ( p ) will have large values.
Similarly, a small dimension means large variations of τ k ( p ) since it is not possible to pack as many equidistant neighbors of p .
This results in a smaller value for d int ( p ) .
An overall estimate of the d int is then obtained by averaging the local estimate d int ( p ) over all patches, i.e. d int = 1 |D| (cid:80) p ∈D
d int ( p ) .
Fig.
4 (bottom-right) shows the intrinsic dimension estimated using K = 4 · 10 3 and a dictionary of size |D| = 16 ·
In all cases, the estimated intrinsic dimension d int is much smaller than the extrinsic dimension d ext = 3 P 2 .
Moreover, it grows even more slowly than the linear dimension when the patch size P increases.
Finally, even after whitening, d int is only about 10% of the total dimension, which is a strong evidence that the natural image patches are low dimensional.
C ONCLUSION
In this work, we shed light on data-driven kernels: we emphasize that they are a necessary steps of any methods which perform well on challenging datasets.
We study this phenomenon through ablation experiments: we used a shallow, predeﬁned visual representations, which is not optimized by gradient descent.
Surprisingly, this method is highly competitive with others, despite using only whitened patches.
Due to limited computational resources, we restricted ourselves on ImageNet to small image resolutions and relatively small number of patches.
Conducting proper large scale experiments is thus one of the next research directions.
A CKNOWLEDGEMENTS
EO was supported by a GPU donation from NVIDIA.
This work was granted access to the HPC resources of IDRIS under the allocation 2021-AD011011216R1 made by GENCI.
This work was partly supported by ANR-19-CHIA “SCAI”.
EB acknowledges funding from IVADO fundamentals grant.
The authors would like to thank Alberto Bietti, Bogdan Cirstea, Lénaic Chizat, Arnak Dalayan, Corentin Dancette, Stéphane Mallat, Arthur Mensch, Thomas Pumir, John Zarka for helpful comments and suggestions. Julien Mairal provided additional numerical results that were helpful to this project A M AHANALOBIS DISTANCE AND WHITENING
The Mahalanobis distance (Chandra et al., 1936; McLachlan, 1999) between two samples x and x (cid:48) drawn from a random vector X with covariance Σ is deﬁned as
[Equation]
If the random vector X has identity covariance, it is simply the usual euclidian distance :
[Equation]
Using the diagonalization of the coraviance matrix, Σ = P Λ P T , the afﬁne whitening operators of the random vector X are the operators
[Equation]
For example, the PCA whitening operator is
[Equation]
and the ZCA whitening operator is
[Equation]
For all whitening operator w we have
[Equation]
since
[Equation]
I MPLEMENTATION OF THE PATCHES K- NEAREST - NEIGHBORS ENCODING
In this section, we explicitly write the whitened patches with the whitening operator W .
Recall that we consider the following set of euclidean pairwise distances:
[Equation]
For each image patch we encode the K nearest neighbors of Wp i,x in the set Wd, d ∈ D , for some
K ∈ 1 . . .
|D| .
We can use the square distance instead of the distance since it doesn’t change the K nearest neighbors.
We have
[Equation]
The term (cid:107)
Wp i,x (cid:107) 2 doesn’t affect the K nearest neighbors, so the K nearest neighbors are the K smallest values of
[Equation]
This can be implemented in a convolution of the image using − W T Wd as ﬁlters and (cid:107)
Wd (cid:107) 2 / 2 as bias term, followed by a "vectorwise" non-linearity that binary encodes the K smallest values in the channel dimension.
Once this is computed, we can then easily compute
[Equation]
which is the quantity needed to compute the K nearest neighbors in the set of negative patches
D .
This is a computationally efﬁcient way of doubling the number of patches while making the representation invariant to negative transform.
C A BLATION STUDY ON CIFAR-
For this ablation study on CIFAR-10, the reference experiment uses |D| = 2048 patches, a patch size Q = 6 a number of neighbors K = 0 .× 2048 = 820 and a whitening regularizer λ
= 1 e − 3 , and yields 82.5% accuracy.
Figure 5 shows the results in high resolution.
We further performed an experiment, where we replaced the patches of CIFAR-10 by the patches of ImageNet: this leads to a drop of 0 .
4% accuracy compared to the reference model.
Note that the same model without data augmentation performs about 2% worse than the reference model.
Number of patches |D| varies in
512 , 1024 , 2048 , 4096 } , number of neighbors K varies in { 10 , 50 , 100 , 500 , 800 , 1000 , 1500 } , patch size Q varies in { 4 , 5 , 6 , 7 , 8 } , whitening regularization λ varies in { 0 , 10 − 5 , 10 − 4 , 10 − 3 , 10 − 2 , 10 − 1 , 1 , 10 } .
D I NTRINSIC DIMENSION ESTIMATE
The following estimate of the intrinsic dimension d int is introduced in Levina and Bickel (2004) as follows
[Equation]
where τ k ( p ) is the euclidean distance between the patch p
and it’s k -th nearest neighbor int the training set.