T HE I NTRINSIC D IMENSION OF I MAGES AND I TS I MPACT ON L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
It is widely believed that natural image data exhibits low-dimensional structure despite being embedded in a high-dimensional pixel space.
This idea underlies a common intuition for the success of deep learning and has been exploited for enhanced regularization and adversarial robustness.
In this work, we apply dimension estimation tools to popular datasets and investigate the role of low dimensional structure in neural network learning.
We ﬁnd that common natural image datasets indeed have very low intrinsic dimension relative to the high number of pixels in the images.
Additionally, we ﬁnd that low dimensional datasets are easier for neural networks to learn.
We validate our dimension estimation tools on synthetic data generated by GANs in which we can manipulate intrinsic dimension.
I NTRODUCTION
The idea that real-world data distributions can be described by very few variables underpins machine learning research from manifold learning to dimension reduction (Besold & Spokoiny, 2019;
Fodor, 2002).
The number of variables needed to describe a data distribution is known as its intrinsic dimension (ID).
In applications, such as crystallography, computer graphics, and ecology, practitioners depend on data having low intrinsic dimension (Valle & Oganov, 2010;
Desbrun et al., 2002; Laughlin, 2014).
The utility of representations which are low-dimensional has motivated a variety of deep learning techniques including autoencoders and regularization methods (Hinton & Salakhutdinov, 2006; Vincent et al., 2010; Gonzalez & Balajewicz, 2018; Zhu et al., 2018).
It is also known that dimensionality plays a strong role in learning function approximations and non-linear class boundaries.
The exponential cost of learning in high dimensions is easily captured by the trivial case of sampling a function on a cube; in d dimensions, sampling only the cube vertices would require 2 d measurements.
Similar behaviors emerge in learning theory.
It is known that learning a manifold requires a number of samples that grows exponentially with the manifold’s intrinsic dimension (Narayanan & Mitter, 2010).
Similarly, the number of samples needed to learn a well-conditioned decision boundary between two classes is an exponential function of the intrinsic dimension of the manifold on which the classes lie (Narayanan & Niyogi, 2009).
Furthermore, these learning bounds have no dependence on the ambient dimension in which manifold-structured datasets live.
In light of the exponentially large sample complexity of learning high-dimensional functions, the ability of neural networks to learn from image data is remarkable.
Networks learn complex decision boundaries from small amounts of image data (often just a few hundred or thousand samples per class).
At the same time, generative adversarial networks (GANs) are able to learn image “manifolds” from merely a few thousand samples.
The seemingly low number of samples needed to learn these manifolds strongly suggests that image datasets have extremely low-dimensional structure.
Despite the established role of low dimensional data in deep learning, little is known about the intrinsic dimension of popular datasets and the impact of dimensionality on the performance of neural networks.
Computational methods for estimating intrinsic dimension enable these measurements.
We adopt tools from the dimension estimation literature to shed light on dimensionality in settings of interest to the deep learning community.
Our contributions can be summarized as follows:
[Figure]
• We verify the reliability of intrinsic dimension estimation on high-dimensional data using generative adversarial networks (GANs), a setting in which we can a priori upper-bound the intrinsic dimension of generated data by the dimension of the latent noise vector.
• We measure the dimensionality of popular datasets such as MNIST, CIFAR-10, and ImageNet.
In our experiments, we ﬁnd that natural image datasets whose images contain thousands of pixels can, in fact, be described by orders of magnitude fewer variables.
For example, we estimate that ImageNet, despite containing 224 × 224 × 3 = 150528 pixels per image, only has intrinsic dimension between 26 and 43 ; see Figure 1.
• We train classiﬁers on data, synthetic and real, of various intrinsic dimension and ﬁnd that this variable correlates closely with the number of samples needed for learning.
On the other hand, we ﬁnd that extrinsic dimension, the dimension of the ambient space in which data is embedded, has little impact on generalization.
Together, these results put experimental weight behind the hypothesis that the unintuitively low dimensionality of natural images is being exploited by deep networks, and suggest that a characterization of this structure is an essential building block for a successful theory of deep learning.
R ELATED W ORK
While the hypothesis that natural images lie on or near a low-dimensional manifold is controversial, Goodfellow et al. (2016) argue that the low-dimensional manifold assumption is at least approximately correct for images, supported by two observations. First, natural images are locally connected, with each image surrounded by other highly similar images reachable through image transformations (e.g., contrast, brightness).
Second, natural images seem to lie on a low-dimensional structure, as the probability distribution of images is highly concentrated; uniformly sampled pixels can hardly assemble a meaningful image.
It is widely believed that the combination of natural scenes and sensor properties yields very sparse and concentrated image distributions, as has been supported by several empirical studies on image patches (Lee et al., 2003; Donoho & Grimes, 2005; Carlsson et al., 2008).
This observation motivated work on efﬁcient coding (Olshausen & Field, 1996) and served as a prior in computer vision (Peyr´e, 2009).
Further, rigorous experiments have been conducted clearly supporting the low-dimensional manifold hypothesis for many image datasets (Ruderman, 1994; Sch¨olkopf et al., 1998; Roweis & Saul, 2000; Tenenbaum et al., 2000;
Brand, 2003). Fefferman et al. (2016) also proposed principled algorithms for verifying the manifold hypothesis.
The generalization literature seeks to understand why some models generalize better from training data to test data than others.
One line of work suggests that the loss landscape geometry explains why neural networks generalize well (Huang et al., 2019).
Other generalization work predicts that data with low dimension, along with other properties which do not include extrinsic dimension, characterize the generalization difﬁculty of classiﬁcation problems (Narayanan & Niyogi, 2009).
In the context of deep learning, Gong et al.
(2019) found that neural network features are low- dimensional.
Ansuini et al.
(2019) further found that the intrinsic dimension of features decreases in late layers of neural networks and observed interesting trends in the dimension of features in early layers. In contrast to Gong et al.
and Ansuini et al.
(2019), who ﬁnd that the intrinsic dimension of internal representations is inversely correlated with high performance, we study the dimensionality of data and its impact on performance, and we make a similar ﬁnding.
Zhu et al.
(2018) recently proposed a regularizer derived from the intrinsic dimension of images augmented with their corresponding feature vectors.
Another line of work in deep learning has found that neural networks rely heavily on textures and that these textures are low-dimensional (Geirhos et al., 2018;
Brendel & Bethge, 2019).
Similarly, some have suggested that natural images can be represented as mixtures of textures which lie on a low-dimensional manifold (Vacher & Coen-Cagli, 2019; Vacher et al., 2020).
I NTRINSIC D IMENSION E STIMATION
Given a set of sample points P ⊂ R N , it is common to assume that P lies on or near a lowdimensional manifold M ⊆ R N of intrinsic dimension dim( M ) =
d (cid:28) N .
As a measure of the degrees of freedom in a dataset, as well as the information content, there is great interest in estimating the intrinsic dimension d .
In the remainder of this section, we brieﬂy describe the dimension estimation method we use in this paper; for further information, see (Kim et al., 2019) and references therein.
One of the main approaches to intrinsic dimension estimation is to examine a neighborhood around each point in the dataset, and compute the Euclidean distance to the k th nearest neighbor.
Assuming that density is constant within small neighborhoods, the Maximum Likelihood Estimation (MLE) of Levina & Bickel (2005) uses a Poisson process to model the number of points found by random sampling within a given radius around each sample point.
By relating the rate of this process to the surface area of the sphere, the likelihood equations yield an estimate of the ID at a given point x as:
[Equation]
where T j ( x ) is the Euclidean ( (cid:96) 2 ) distance from x to its j th nearest neighbor.
Levina & Bickel (2005) propose to average the local estimates at each point to obtain a global {+estimate ¯ m k =n (cid:80) ni =
 ˆ m k ( x i ) .+}
MacKay & Ghahramani (2005) suggestion a correction based on averaging of inverses [Equation] where n is the number of samples. We use Equation (2) as our MLE estimator throughout this paper.
Since the geometry of natural images is complex and unknown, we face two challenges when verifying the accuracy of MLE on natural image datasets.
First, we need to choose a proper value of k .
As shown by MacKay & Ghahramani (2005), the positive bias of the corrected estimator Equation (2) increases as k increases, but the variance decreases. In order to navigate this bias-variance tradeoff, we try various values of k in Section 4. Second, in addition to the aforementioned local uniformity assumption, MLE assumes that data arises as a sequence of i.i.d. random variables which can be written as a continuous and sufﬁciently smooth function of a random variable with smooth density, which may or may not be true for natural image datasets.
While the truth of these assumptions is unknown on natural images, we verify the accuracy of our MLE estimates in a controlled setting in the following section.
We brieﬂy discuss other notable techniques for dimensionality estimation.
GeoMLE (Gomtsyan et al., 2019) attempts to account for non-uniformity of density and nonlinearity of manifold using a polynomial regression of standard MLE based on distances to nearest neighbors in different sized neighborhoods.
However, GeoMLE chooses to approximate averages of ˆ m k ( x i ) , instead of averaging its reciprocal like Equation (2), resulting in a potentially wrong maximum likelihood estimator.
As a result, we ﬁnd its estimation deviates signiﬁcantly from expected dimensionalities.
TwoNN (Facco et al., 2017) is based on the ratio of the distances to the ﬁrst and second nearest neighbors.
Finally, the approach of (Granata & Carnevale, 2016) considers the distribution of geodesic distances over the data manifold, approximated by distances through kNN graphs, compared to the distribution of distances over hyperspheres of varying dimension.
Unlike MLE, our preliminary experiments suggest that these techniques do not provide reasonable estimates for some natural and synthetic images which are key to this work; please refer to Appendix A.5 for further discussion.
Dimensionality estimates are often applied on “simple” manifolds or toy datasets where the dimensionality is known, and so the accuracy of the methods can be validated.
Image manifolds, by contrast, are highly complex, may contain many symmetries and modes, and are of unknown dimension.
In principle, there is no reason why MLE-based dimensionality estimates cannot be applied to image datasets.
However, because we lack knowledge of the exact dimensionality of image datasets, we cannot directly verify that MLE-based dimensionality estimates scale up to the complexity of image structures.
There is an inherent uncertainty in estimating the ID of a given dataset.
First, we cannot be sure if the dataset actually resembles a sampling of points on or near a manifold.
Second, there are typically no guarantees that the sampling satisﬁes the conditions assumed by the ID estimators we are using.
Towards a principled application of ID estimates to the learning context, we begin by validating that
MLE methods can generate accurate dimensionality estimates for complex image structures.
We do this by generating image datasets using generative models for which the intrinsic dimensionality is known.
We believe such validations are essential to put recent ﬁndings in perspective
(Gong et al., 2019; Ansuini et al., 2019).
GAN Images We use the BigGAN variant with 128 latent entries and outputs of size 128 × 128 ×trained on the ImageNet dataset (Deng et al., 2009).
Using this GAN, we generate datasets with a varying number of images, where we ﬁx most entries of the latent vectors to zero leaving only d free entries to be chosen at random.
As we increase the number of free entries, we expect the intrinsic dimension to increase, and the output can only be at most d -dimensional; see Section A.1 for further discussion.
In particular, we create several synthetic datasets of varying intrinsic dimensionality using the ImageNet class, basenji , and check if the estimates match our expectation.
As seen in Figure 2, we observe increasing diversity with increasing intrinsic dimension.
In Figure 3, we show convergence of the MLE estimate on basenji data with dimension bounded above by 10 .
We observe that the estimates can be sensitive to the choice of k as discussed in prior work; see Appendix A.2 for additional GAN classes.
Scaling to large datasets.
We develop a practical approach for estimating the ID of large datasets such as ImageNet.
In this approach, we randomly select a fraction α of the dataset as anchors.
Then, we evaluate the MLE estimate using only the anchor points, where nearest-neighbors are computed over the entire dataset.
Note that, when anchors are chosen randomly, this acceleration has no impact on the expected value of the result.
See Appendix A.3 for an evaluation of this approach.
T HE I NTRINSIC D IMENSION OF P OPULAR D ATASETS
In this section, we measure the intrinsic dimensions of a number of popular datasets including
MNIST (Deng, 2012), SVHN (Netzer et al., 2011), CIFAR-10 and CIFAR-100 (Krizhevsky et al., 2009), ImageNet (Deng et al., 2009), MS-COCO (Lin et al., 2014), and CelebA (Liu et al., 2015).
Using three different parameter settings for the MLE ID estimator, we ﬁnd that the ID is indeed much smaller than the number of pixels; see Table 5.
Notice that the rank order of datasets by dimension does not depend on the choice of k .
A comparison of state-of-the-art (SOTA) classiﬁcation accuracy on each respective dataset 1 with the dimension estimates suggests a negative correlation d =d =d =d =d = [Figure] [Figure]
i
[Figure]
n s [Figure] between the intrinsic dimension and test accuracy. In the next section, we take a closer look at this phenomenon through a series of dedicated experiments.
I NTRINSIC D IMENSION AND G ENERALIZATION
Learning theory work has established that learning a manifold requires a number of samples that grows exponentially with the manifold’s intrinsic dimension (Narayanan & Mitter, 2010), but the required number of samples is independent of the extrinsic dimension.
Speciﬁcally, the number of samples needed to learn a well-conditioned decision boundary between two classes is an exponential [Table] function of the intrinsic dimension of the manifold on which the classes lie (Narayanan & Niyogi, 2009).
We leverage dimension estimation tools to empirically verify these theoretical ﬁndings using a family of binary classiﬁcation problems deﬁned over both synthetic and real datasets of varying intrinsic dimension.
We then train classiﬁers on these problems and measure test accuracy.
In these experiments, we observe a connection between the intrinsic dimension of data and generalization.
Speciﬁcally, we ﬁnd that classiﬁcation problems on data of lower intrinsic-dimensionality are easier to solve.
6. S YNTHETIC GAN DATA : S AMPLE COMPLEXITY DEPENDS ON INTRINSIC ( NOT EXTRINSIC ) DIMENSIONALITY
The synthetic GAN data generation technique described in Section 4 provides a unique opportunity to test the relationship between generalization and the intrinsic/extrinsic dimensionality of images.
By creating datasets with controlled intrinsic dimensionality, we may compare their sample complexity , that is the number of samples required to obtain a given level of test error.
Speciﬁcally we test the following two hypotheses (1) data of lower intrinsic dimensionality has lower sample complexity than that of higher intrinsic dimensionality and (2) extrinsic dimensionality is irrelevant for sample complexity.
To investigate hypothesis (1), we create four synthetic datasets of varying intrinsic dimensionality: 16 , 32 , 64 , 128 , ﬁxed extrinsic dimensionality: 3 × 32 × 32 , and two classes: basenji and beagle .
For each dataset we ﬁx a test set of size N = 1700 .
For all experiments, we use the ResNet- (width = 64 ) architecture (He et al., 2016).
We then train models until they ﬁt their entire training set with increasing amounts of training samples and measure the test error.
We show these results in Figure 4.
Observing the varying rates of growth, we see that data of higher intrinsic dimension requires more samples to achieve a given test error.
For hypothesis (2), we carry out the same experiment with the roles of intrinsic and extrinsic dimension switched.
We create four synthetic datasets of varying extrinsic dimensionality by resizing the images with nearest-neighbor interpolation.
Speciﬁcally we create 6 datasets of square, 3-channel images of sizes 16 , 32 , 64 , 128 , 256 , ﬁxed intrinsic dimensionality of size 128 , and all other experimental details the same.
We show these results in Figure 5.
Observing the lack of variable growth rates, we see that extrinsic dimension has little to no effect on sample complexity.
We conclude by noting that, to the best of our knowledge, this is the ﬁrst such experimental result to demonstrate that intrinsic but not extrinsic dimensionality matters for the generalization of deep networks .
6. R EAL D ATA : I NTRINSIC DIMENSIONALITY MATTERS FOR GENERALIZATION
Next, we examine the sample complexity of binary classiﬁcation tasks from four common image datasets: MNIST, SVHN, CIFAR-10, and ImageNet.
This case differs from the synthetic case in that we have no control over each dataset’s intrinsic dimension.
Instead, we estimate it via the MLE method discussed in Section 3.
To account for variable difﬁculty of classes, we randomly sampleclass pairs from each dataset and run the previously described sample complexity experiment.
Note that these subsets differ from those used in Table 5, where the estimates are taken from the entire dataset and across all classes.
[Figure]
[Figure]
On these sampled subsets, we ﬁnd the MLE estimates as shown in Table 2.
We note that these estimates are consistent with expectation, e.g. MNIST is qualitatively less complex then SVHN and CIFAR-10.
We conduct the same sample complexity experiment as the previous section on the datasets.
Because these datasets are ordinarily of varying extrinsic dimensionality, we resize all to size 32 × 32 ×(before applying MLE).
We report results in Figure 6, where we overall observe trends ordered by intrinsic dimensionality estimate.
These results are consistent with expectation of the relative hardness of each dataset.
However, there are some notable differences from the synthetic case.
Several unexpected cross-over points exist in the low-sample regime, and the gap between SVHN and CIFAR-10 is smaller than one may expect based on their estimated intrinsic dimension.
From these observations we conclude that intrinsic dimensionality is indeed relevant to generalization on real data, but it is not the only feature of data that inﬂuences sample complexity. MNIST SVHN CIFARImageNet k
=7.5 (0.2) 8.5 (0.1) 11.4 (0.2) 15.4 (0.8) k
=9.8 (0.3) 11.6 (0.1) 15.9 (0.2) 19.8 (0.9) k
= [Table]
6. R EAL DATA : A DDING NOISE CHANGES DIMENSIONALITY TO AFFECT GENERALIZATION
In this section, we examine an alternative technique for changing the intrinsic dimension of a real dataset: adding noise to images.
Here we leverage the fact that uniformly sampled noise in [0 , 1] d has dimension d .
We thus add independent noise, drawn uniformly from a ﬁxed randomly oriented d -dimensional unit hypercube embedded in pixel space, to each sample in a dataset.
This procedure [Figure] ensures that the dataset has dimension at least d .
Since the natural data we use has low dimension, and the hypercubes have high dimension, this procedure speciﬁcally increases dimensionality.
We note that estimation error may occur when there is insufﬁcient number of samples required to achieve a proper dimension estimate.
Since the variation in images in a dataset may still be dominated by non-noise directions, we expect to underestimate the new increased dimensions of these noised datasets.
Starting with CIFAR-10 data, we add noise of varying dimensions, where we replace pixels at random in the image.
We only add noise to an image once to keep the augmented dataset the same size as the original.
We use the following noise dimensionalities: 256 , 512 , 1024 , 2048 , 2560 .
The estimated dimensions of the noised datasets are listed in Table 3.
We see that intrinsic dimension increases with increasing noise dimensionality, but dimensionality does not saturate to the maximum true dimension, likely due to a poverty of samples.
On these noisy CIFAR-10 datasets, we again carry out the sample complexity experiment of the previous sections.
We show results in Figure 7.
We observe sample complexity largely in the same order as intrinsic dimension. d =d =d =d =d =d =k =19. 136.1 k =25.
177.7 k =27. [Figure] 6. M ANIPULATING THE INTRINSIC DIMENSIONALITY OF FONTS
In this section [Table] we describe a ﬁnal technique for studying the effect of intrinsic dimensionality on sample complexity on the recently proposed FONTS dataset (Stutz et al., 2019).
Beginning with a collection of characters and font types, termed a prototype set by the authors, FONTS datasets are constructed using a ﬁxed set of data augmentations: scaling, translation, rotation, and sheering.
In principle, these augmentations each increase the intrinsic dimension of the prototype set.
Furthermore, by varying the number of augmentations used, we can synthetically alter the intrinsic dimension.
We construct 5 FONTS datasets in this way, FONTS- { 0 , 1 , 2 , 3 , 4 } , where the sufﬁx denotes the number of transformations used in the data generation process.
The MLE estimates on each of the datasets are given in Table
[Table]
Consistent with expectation
[Table]
we observe that MLE methods consistently resolve the increased dimesionality of transformed datasets.
Carrying out the sample complexity experiment of the previous section [Table] we report results in Figure 8.
We observe again that, on the whole, sample complexity is ordered by intrinsic dimension. [Table] [Figure] of transforms
[Figure]
D ISCUSSION
In this work, we measure the intrinsic dimension of popular image datasets and show that the intrinsic dimension of data matters for deep learning.
While there may be many factors, such as class separation and the number of classes, which determine generalization, we build the case that intrinsic dimension is one of these important factors.
Along the way, we introduce a technique for using GANs to synthesize data while manipulating dimensionality.
This technique is useful not only for validating dimension estimation methods but also for examining the learning behavior of neural networks under a dimension-controlled environment.
In addition to synthetic data, we verify that dimension plays a large role in learning on natural data.
Our results support the commonly held belief that low dimensional structure underpins the success of deep learning on high-resolution data.
A V ALIDATION OF ID E STIMATES
In this section, we present additional discussion results and discussion relevant to the ID estimation and related validation experiments in Section 4.
A. GAN PROPERTIES
We devise a method for validating ID measurements in a controlled setting using images generated by GANs.
To justify this method, we ﬁrst note that the image of R d under a locally Lipschitz function can be a manifold with dimension at most d .
Then, consider that the BigGAN generator, a convolutional neural network with ReLU activations, is a function with this property (Brock et al., 2018).
Speciﬁcally, BigGAN can be written as a composition of linear functions, translations, and ReLU activation functions.
Individually, these operations do not increase dimension, and by a composition property, their composition cannot increase dimensionality either.
The more general fact that the image of R d under a locally Lipschitz function can be a manifold with dimension at most d follows from Sard’s theorem (Sard, 1942).
A. C ONVERGENCE FOR M ORE GAN C LASSES
We include additional results on the estimation of ID for synthetic GAN images from various ImageNet classes with d = 10 free entries.
As observed earlier in Section 4, the MLE estimates are sensitive to the choice of k , where we expect the ID to be close to 10 given the way we sample the latent vectors to use for the GAN.
We note that for a number of classes, all choices of k we considered seem to underestimate the ID.
[Figure]
D i m e n s i o n a li t y
E
[Figure]
t i m a t e
[Figure]
k
[Figure]
k
[Figure]
k
[Figure]
k
[Figure]
k= k
[Figure]
A. S UBSAMPLING FOR L ARGE D ATASETS
In Figure 12 we validate the anchor approximation on basenji data of dimension 10 for varying anchor ratio α .
Then, in Figure 13 we validate the anchor approximation on tree-frog data of dimension 32 for varying k while ﬁxing the anchor ratio at α = 0 . 001 . A. R ELATIONSHIP BETWEEN k IN THE MLE METHOD AND DATASET DIMENSION
The choice of k may affect the dimensionality estimates. To better understand this relationship, we conducted additional studies for the MLE method using synthetic GAN data generated as described in Section 4. Table 5 shows the estimation results for various k and basenji data of intrinsic dimensionality 2 , 4 , 8 , 16 , 32 , 64 , 128 . We use a ﬁxed number of samples ( n = 10000 ).
We observe that the estimated intrinsic dimension increases with k , and large values can yield overestimates for
ID.
These results agree with the work of Levina & Bickel (2005) who suggest that low values of k
[Figure]
o n
[Figure] li [Figure] E s t
i
[Figure] a t [Figure] [Figure]
may produce an estimator with higher variance while higher values of k produce an estimator with higher positive bias.
Given that we have access to large amounts of synthetic data
[Figure]
we opt for lower values of k in our work.
We do not choose a particular value of k
[Figure]
and we report all experiments with multiple values
[Figure]
[Figure]
[Figure]
D i m e n s i o n a li t y
E s t i m a t e tree-frog
k= k= k= k= k= k= k= k d        1.
A. C OMPARING MLE TO O THER E STIMATORS IN A C ONTROLLED S ETTING
To validate MLE in comparison to other estimators, we evaluate three other dimensionality estimation methods: GeoMLE (Gomtsyan et al., 2019), TwoNN (Facco et al., 2017) and kNN graph distances (Granata & Carnevale, 2016).
For GeoMLE, we sample a total of 20 bootstrap subsets ( M = 20 ) and use k 1 = 20 , k 2 = 55 as recommended by Gomtsyan et al.
To extend the kNN graph distance method to large datasets, we randomly sample a subset of samples (ﬁxed to 10,000 for datasets with more than 10,000 samples) and use shortest graph distances to their k nearest neighbors to estimate the IDs.
Other settings are as default in the implementations of Granata &
Carnevale (2016).
On basenji 10 , as in Figure 3, we observe that each other method does not achieve an accurate estimate, thus motivating our focus on MLE.
Notably, GeoMLE and TwoNN severely overestimate dimension, while the kNN graph distance method severely underestimates dimension.
The implementation of each method was validated on simple hypercube data with uniform distribution.
On
MNIST, CIFAR-10, CIFAR-100, and SVHN, the results of these other estimation methods also deviate from expectation (Table 6).
For example, TwoNN assigns a signiﬁcantly higher dimension estimate to MNIST than to CIFAR-100, which contradicts both intuition and the results of other estimators.
We set k = 4 and number of bins to 1000, and use default settings for all other parameters including r MAX .
[Figure]
n a li t
[Figure]
s t
i
[Figure]
[Figure]
A. ID BY I MAGE N ET C LASS
Figure 17 contains histograms of intrinsic dimension estimates on all individual ImageNet classes for several values of k .
We use this ﬁgure to suggest that the class dimensions have high variance.
Unfortunately, each ImageNet class contains insufﬁcient data in order to obtain convergent estimates.
Thus, we suspect that the values in this ﬁgure are underestimates.
[Figure]
[Table]
MLE Dimensionality Estimate
F r e q u e n
[Figure]
y k
[Equation]
3, mean
[Equation]
[Equation]
[Equation]
k= 4, mean=18.4 +/- 2.
{+k= 5, mean=20.
+}
+/-
k=10, mean=21.4 +/- 2. k=50, mean=17.7 +/- 2.