LIDL: L OCAL I NTRINSIC D IMENSION ESTIMATION USING APPROXIMATE L IKELIHOOD
A BSTRACT
Most of the existing methods for estimating the local intrinsic dimension of a data distribution do not scale well to high dimensions because they rely on investigating the nearest neighbors structure, which may cause problems due to the curse of dimensionality.
We attempt to address that challenge by proposing a new method for Local Intrinsic Dimension estimation using approximate Likelihood (LIDL) which makes use of the recent progress in likelihood estimation in high dimensions—the normalizing ﬂow methods.
We empirically show that on standard benchmarks for this problem, our method yields more accurate estimates than previous algorithms and that, unlike other methods, it scales well to problems with thousands of dimensions.
What is more, we anticipate this new approach to improve further with continuing advances in the density estimation literature.
I NTRODUCTION
One of the important problems in representation learning is estimating the intrinsic dimensionality (ID) of lowerdimensional data embedded in a higher-dimensional observed space (Ansuini et al., 2019; Li et al., 2018; Rubenstein et al., 2018), which we will refer to as IDE.
It is a well-studied problem in the context of dimensionality reduction, clustering, and classiﬁcation problems (Camastra & Staiano, 2016; Kleindessner & Luxburg, 2015;
Vapnik, 2013).
ID is also relevant for some prototypebased clustering algorithms (Claussen & Villmann, 2005;
Struski et al., 2018).
ID estimation can be used as a powerful analytical tool to study the process of training and representation learning in deep neural networks (Li et al., 2018;
Ansuini et al., 2019).
Also in the context of representation learning, Rubenstein et al.
(2018) show how the mismatch between the latent space dimensionality and the dataset ID may hurt the performance of auto-encoderbased generative models like VAE (Kingma & Welling, 2014),
WAE (Tolstikhin et al., 2017), or CWAE (Knop et al., 2020).
IDE methods can be divided into two broad categories: global and local (Camastra & Staiano, 2016).
Global methods aim to give a single estimate of the dimensionality of the entire dataset.
However, reducing the description of a dataset’s ID structure to just a single number might discard the nuanced structure, such as when the data lies on a union of manifolds with different numbers of dimensions (which is the case for real-world datasets).
On the contrary, the local methods (Carter et al., 2009) try to estimate the local intrinsic dimensionality (LID) of data manifold in the neighborhood of an arbitrary point on that manifold.
This approach gives more insight into the nature of the dataset and provides more options to summarize the dimensionality of the manifold from the global perspective – similarly to how the estimate of the density of a random variable provides richer information than just the estimates of its summary statistics.
Studying the local dimensionality allows to reason on the level of single samples, which can lead to new insights e.g. by looking at single datapoints and how their properties are related to their LID estimates.
A good review and comparison of a wide range of methods for global and local ID estimation can be found in (Camastra & Staiano, 2016).
Most of the existing methods for LID estimation (Kleindessner & Luxburg, 2015; Levina & Bickel, 2004; Hino et al., 2017; Camastra & Staiano, 2016; Rozza et al., 2012; Ceruti et al., 2014; Camastra & Vinciarelli, 2002) tend to analyze the local structure of the data manifold by investigating data point’s nearest neighbours structure or pairwise distances.
Unfortunately, such approaches generally do not scale well to high dimensions.
To address this problem, we propose a new method for
Local Intrinsic Dimension estimation using approximate Likelihood (LIDL), which circumvents that challenge by making use of the recent progress in normalizing ﬂows (NF) (Dinh et al., 2014;
Rezende & Mohamed, 2015; Kingma & Dhariwal, 2018).
Our method makes use of the observation that the local dimensionality of the manifold can be estimated using approximate likelihood of the data distribution perturbed with a small Gaussian noise of varying magnitude.
Our contributions are: we introduce a new way of local intrinsic dimension estimation, that uses approximate likelihood instead of a nearest neighbour approach; we show that our method outperforms other approaches on standard benchmark problems for this task (see Fig. 1); we demonstrate that our method scales to high-dimensional settings (see Table 2) and works on image datasets (Fig. 4).
L OCAL I NTRINSIC D IMENSIONALITY
We often know that a particular dataset comes from some T -dimensional manifold, but we observe it after it has been embedded into a higher-dimensional space of R M , T < M .
While such setting is common for structured problems, it is also hypothesised to hold to some extent in less structured settings, e.g., for natural images.
This is termed the manifold hypothesis (Fefferman et al., 2016) and it motivates applying the intrinsic dimension estimation methods to less structured datasets.
As pointed out in the introduction, the data does not necessarily come from a single T -dimensional manifold, but instead it might lie on a union of manifolds of different dimensions, which might or might not intersect.
Paying particular attention to such settings is important, because they allow to showcase some advantages of the local over the global ID estimation methods.
To give the problem a more formal framing we adapt a deﬁnition from (Kleindessner & Luxburg, 2015) in Appendix A.
M ETHOD
Let us start with building an intuition about the core observation that underpins our method.
First, in the context of the deﬁnition of the problem setting from the previous section, note that in the absence of noise η i , the probability density of the data in the observed space p D ( x ) has to be described using a Dirac delta function.
Unfortunately, this makes it impossible for computational methods to express the probability density on M because the density becomes numerically singular.
This is illustrated in the top row, middle column of Fig.
2 as an example of a 1D set being embedded in a 2D space.
To circumvent this problem, consider perturbing the data points from the original distribution p D ( x ) with a small Gaussian noise (cid:15) δ ∼ N M ( 0 , δI M ) , such that (cid:15) δ ∈ R M , I M denotes an M -dimensional identity matrix, δ is a parameter determining the noise magnitude, and
N K denotes a K -dimensional normal distribution.
The summation of random variables, the original data and the noise, corresponds to a convolution of the probability density functions such that the resulting density is p δ ( x )
= p D ( x ) ∗ N M ( x | 0 , δI M ) , which no longer causes numerical issues and can be approxi- mated using density estimation methods.
Adding the noise not only allows us to approximate the density of the perturbed data, but is also the key to estimating the local intrinsic dimension.
Locally, around every point x , space M can be divided into two linear subspaces: subspace that is locally tangent to the manifold M x T , and the subspace orthogonal to the manifold M x O .
The the fact the convolution with a normal distribution displaces some of the probability mass from the tangent subspace into the orthogonal subspace decreases the amount of probability mass in the tangent subspace in proportion to the dimensionality of the orthogonal subspace.
See Fig.
2 to build an intuition about this process.
We assume that the density along the locally tangent subspace is locally constant such that the convolution along the tangent subspace has no effect on the density in that subspace, more discussion in Appendix B.
[Table]
+Wl6WfQFeRGsh6jxDgqqm5kkQ0T4IZEQVF
[Table]
[Figure]
) =
p D ( x ) ∗ N M ( x | 0 , δI M ) .
Middle and bottom rows correspond to two different noise magnitudes, δ 1 and δ 2 = 2 δ 1 , respectively.
Note that the symbol δ is overloaded, the Dirac delta term δ ( · ) should not be confused with δ specifying the magnitude of the noise.
The density colors are consistent per-column (for the middle and bottom rows), such that the colors in each of the columns are directly comparable, and the maximum value of the colorbar gives the maximum value of the density on a given plot.
Let’s say that our goal is to determine the local intrinsic dimensionality for the neighbourhood of point (0 , 0) .
The key observation that will allow us to answer that question is that for different dimensionalities of X (i.e., different columns) the difference between densities p δ 1 (0 , 0) and p δ 2 (0 , 0) (i.e., middle and bottom rows) differs.
The precise relationship between those quantities is dictated by Eq.
1 what allows us to determine the local intrinsic dimension at a particular point on the distribution by evaluating densities for different values of δ .
In this particular case we observe that: for the left column the density remains constant as the noise increases; for the middle column it is halved when the noise amplitude is doubled; for the right column the density is quartered.
Having access only to the evaluations p δ 1 (0 , 0) and p δ 2 (0 , 0) , by following Eq.
1 we would conclude that T = 2 , 1 , 0 for the neighbourhood of point (0,0) for columns left to right, respectively.
Consider how the probability density is displaced by the act of convolution at x ( n ) , which is the n th point in the dataset.
Three distinct effects take place: 1.
The probability density from point x ( n ) is displaced into neighbouring points in the tangent subspace, 2.
The probability density from point x ( n ) is displaced into neighbouring points in the orthogonal subspace, 3.
The probability density from the neighbouring points in the tangent subspace is displaced onto the point x ( n ) .
By the assumption above and the symmetry of the isotropic normal distribution as the convolution kernel, the effects 1 and 3 cancel out.
This implies that the amount of mass displaced from any point x ( n ) in the tangent subspace will depend on the dimensionality of the local tangent and orthogonal subspaces.
This implies that the density p δ ( x ( n ) ) for a point x ( n ) can be expressed as a product of two independent probability distributions, one for each subspace we are considering.
The ﬁrst term is the original density with support on the tangent space which is an integral of p D over the orthogonal subspace p T ( x ( n ) ) =
(cid:82) p D ( x ( n ) )
d M x ( n ) O .
To deﬁne the second term ﬁrst let O ( n ) =
M − T ( n ) be the dimensionality of the orthogonal subspace at point x ( n ) .
Now, the second term is the probability density of the component of the noise (cid:15) δ in the O ( n ) –dimensional orthogonal subspace, i.e., N O ( n ) ( 0 , δI O ( n ) ) .
Then, p δ ( x ( n ) ) =
p T ( x ( n ) ) · N O ( n ) ( 0 | 0 , δI O ( n ) )
=
p T ( x ( n ) ) · (
[Equation]
) − O ( n ) / 2 , leading to
[Equation]
If we are able to evaluate p δ ( x ( n ) ) for at least two different values of δ , then we can use this relationship to estimate the value of O ( n ) for a datapoint x ( n ) .
Since we know M we obtain an estimate of the local intrinsic dimension T ( n ) .
LIDL Algorithm
Now, let us consider how to use the method proposed above in practice.
To evaluate the probabilities p δ ( x ) we use NF, which these days can scale even to high-dimensional data like images.
We learn q δ ( x ) : a model that approximates p δ ( x ) .
NF are trained on the data points from the dataset with a
Gaussian noise of appropriate magnitude δ added to them.
For each datapoint in every new batch, we sample a new noise perturbation (cid:15) δ .
When we learn separate models for different values δ i we can use linear regression to obtain an estimate of O ( n ) at some particular point x ( n ) .
This estimate can then be used to calculate T ( n ) , the LID.
To estimate LID for a set of N points S
= { x (1) , ..., x ( n ) , ..., x ( N ) } in the dataset D we have to ﬁt d > 1 models F i ( i = 1 , ..., d ) to d datasets D i .
Each D
i is perturbed version of the original D with different noise N M (0 , δ i I M ) added to the whole dataset.
The probability density for the same coordinates is different when estimated by different F
i and it decreases monotonically as the value of δ i increases.
If we then calculate the values log q
i ( x ( n ) ) for the same point x ( n ) , we can use the linear regression to ﬁt Eq.
1, where the slope is equal to O ( n ) =
M − T ( n ) .
The full algorithm is presented in detail in Algorithm 1 in Appendix C.
E XPERIMENTS
We ran a series of experiments to verify LIDL, to compare it with other state-of-the-art algorithms for LID estimation and to analyze how it behaves on real-world datasets.
4. L OLLIPOP DATASET
For the ﬁrst experiment, we used an synthetic 1D/2D dataset shown in Fig.
We trained the MAF (Papamakarios et al., 2017) model to ﬁt 8 models and estimated the average density per model for the points from ”head” and ”stick” parts of the dataset.
Than we ﬁtted regression to those averages.
Calculated LID estimate for points in the ”head” of the lollipop was 1 . 96 , and for points from the ”stick” part estimated LID was 1 . 00 .
4. C OMPARISON WITH OTHER ALGORITHMS ON SYNTHETIC DATASETS
We collated LIDL with other LID estimation algorithms (Levina & Bickel, 2004; Kleindessner &
Luxburg, 2015) for intrinsic dimension estimation by comparing it with estimates in Table 1 from
Kleindessner & Luxburg (2015).
Each algorithm in Kleindessner & Luxburg (2015) was tested on a dataset of size 1000, so we used the data set of the same size for LIDL training (we usedexamples from this set for training and 250 for validation).
We used MAF and RQ-NSF (Durkan et al., 2019) for this experiments.
The results of this comparison (each one with a standard deviation calculated using 10 experiments) are presented in Table 1 and in Fig.
From this experiments we can clearly see, that LIDL yields result close to the original dimensionality for low-dimensional datasets and gives better estimates in higher dimensions.
[Table] [Table]
4. H IGH - DIMENSIONAL SYNTHETIC DATASETS
We compared LIDL with CorrDim on four datasets of size 10K with Gaussian distribution embedded in higher dimensional space with ID equals 1, 10, 100, and 1000 dimensions.
Results are presented in Table 2 and show
[Table]
that LIDL scales
4. I MAGE DATASETS
We ran LIDL on MNIST (image size 32x32x1) (LeCun & Cortes, 2010), FMNIST (32x32x1) (Xiao et al., 2017) and Celeb-A (64x64x3)
(Liu et al., 2015) datasets using Glow (Kingma &
Dhariwal, 2018) as a density model.
Before training all images were normalized to have pixel values between ( − 0 . 5 , 0 . 5) .
Estimated dimensionalities for MNIST images span roughly from 40 to 250, for FMNIST those numbers are 100 and 900, and for Celeb-A we estimated dimensionalities between 2500 and 6500.
After sorting the dataset according to the local intrinsic dimension of individual data points, we observed that visually more complicated examples have higher estimated dimensionalities.
Some small, medium and high dimensional images from those datasets are shown in Fig.
4 and 7.
More details about experiments can be found in Appendix D and E.
C ONCLUSIONS
We introduced the algorithm for LID estimation based on NF as density estimators, provided the theoretical justiﬁcation for it, and showed that it can scale to datasets of thousands of dimensions.
Our approach, however, is limited by the ability of NF models to scale to even higher dimensions.
For now, we are not able to cope with datasets of images consisting of millions of pixels.
We hope that current intensive research on NF models will make them able to scale to those datasets eventually and automatically make LIDL to be able to estimate LID for them as well.
Appendices A P ROBLEM DEFINITION
Let us assume the data manifold is a union of J separate manifolds, and we will use a discrete indexing variable e , 1 ≤ e ≤ J .
Let X e ⊆
R T e be a set of low-dimensional sets of dimensionality
T e , ϕ e : X e → M ⊆ R M a set of smooth embeddings of X e into a high-dimensional space M , p X e a probability density function supported on
X e , and p e a probability mass function over the indexing variable e .
Assume that a datapoint ˜x ( n ) is drawn by ﬁrst sampling a random variable E ( n ) ∼ p
e which decides which of the submanifolds the datapoint will drawn from, and then drawing the sample from an appropriate distribution ˜x ( n ) ∼ p X E ( n ) .
Next, the samples ˜x ( n ) are embedded into the observation space R M via a corresponding ϕ E ( n ) , possibly disturbed by noise η ( n ) ∈ R M , resulting in the samples x ( n ) =
ϕ E ( n ) (cid:0) ˜x ( n ) (cid:1) + η ( n ) .
The resulting probability density function supported on M is denoted as p D ( x ) .
Given a dataset of samples x ( n ) , the task of LID estimation is to infer the corresponding values T E ( n ) .
B T HE ASSUMPTION OF CONSTANT DENSITY ALONG THE TANGENT SUBSPACE
Let us revisit the assumption we made in the previous section about the density along the tangent subspace p T being constant, e.g., as per Fig.
5a.
Naturally, in general the density p T will not be constant and so we need to question ourselves how much of an error will that introduce to our LID estimates.
To answer that question, consider that the degree of violation of our assumption is a product of two factors: the magnitudes of δ 1 , δ 2 used to make the LID estimates, and the rate of change of p T which worst case could be analyzed by considering by Lipschitz constants of the distributions p X e and the associated functions ϕ e .
The smaller the δ values, the larger the rate of change of p T can be while keeping the degree of violation of our assumption ﬁxed.
This means that we can alleviate this violation by using small values of δ (as much as numerical precision allows), but it does introduce limitations on the smoothness of p X e and ϕ e —we will be able to perform better LID estimation for smoother densities.
[Figure]
C A LGORITHM
LIDL algorithm is written out in details in Algorithm 1.
Algorithm 1: LIDL:
Local Intrinsic Dimension estimation using approximate Likelihood.
Input: Dataset D ; d – number of models to estimate;
M – dimensionality of the data space; ∆ =
( δ 1 , ..., δ d ) – list of values of δ i ;
Set S of N points from R M , at which we want to estimate LID;
Result:
List of T j ’s – LID estimates for points in S ; for δ i in ∆ do initialize model F i ; while
F i not converged do Sample batch b from D ;
Sample noise n i from N (0 , δ i I ) ; b i ← b + n i ; Make training step on
F i using b i ; end endfor x j in S do for F i in ( F 1 , ..., F d ) do Estimate likelihood q ij at point x j using model F i ; end
Calculate regression coefﬁcient O j for a list of d pairs: ((log q 1 j , log δ 1 ) , ..., (log q dj , log δ d )) ;
T
j ← M − O j ; end
D M ORE ON LIDL ESTIMATES ON MNIST
We also used LIDL estimates for MNIST dataset to analyse how the LID distribution looks for different classes.
Empirical CDFs of those values for each digit separately are presented in Fig.
Classes 1s and 7s have the lowest LIDs and 8s and 0s have the highest dimensionalities, which is consistent with the visual complexity of those classes.
We can ask a question: how LIDL estimates are close to underlying LID?
The estimated IDs reported for some MNIST digits in Table 1 in (Kleindessner & Luxburg, 2015) are between 3.07 and 18.
On the other hand Cavallari et al.
(Cavallari et al., 2018) and Wang et al.
(Wang et al., 2016) used auto-encoder representation of MNIST as an input to SVN digit classiﬁer and they achieved the best classiﬁcation results for an auto-encoder with latent space size greater than 100.
This means that we need more than 100 dimensions to encode an average MNIST digit, which is more consistent with our result than with (Kleindessner & Luxburg, 2015).
E NF IMPLEMENTATIONS
For non-image datasets we used MAF and RQ-NSF implementations from nﬂows library (Durkan et al., 2020).
For images we used PyTorch implementation of glow from https://github.
com/
rosinality/glow-pytorch .
F A CKNOWLEDGEMENTS
I wish to thank various people for their contribution to this project: Marek Cygan for overall support during this project and valuable and constructive suggestions on the manuscript; Maciej Dziubi´nski,
Dominik Filipak, Piotr Kozakowski and Maciej ´Sliwowski for their valuable and constructive suggestions on the manuscript and Wacława Tempczyk for her help with method derivation.