L ABEL S MOOTHED E MBEDDING
H YPOTHESIS
FOR
O UT - OF -D ISTRIBUTION D ETECTION
Anonymous authors
Paper under double-blind review
A BSTRACT
Detecting out-of- distribution (OOD) examples is critical in many applications
We propose an unsupervised method to detect OOD samples using a k -NN density estimate with respect to a classiﬁcation model’s intermediate activations on indistribution samples.
We leverage a recent insight about label smoothing, which we call the Label Smoothed Embedding Hypothesis and show that one of the implications is that the k -NN density estimator performs better as an OOD detection method both theoretically and empirically when the model is trained with label smoothing.
Finally, we show that our proposal outperforms many OOD baselines and we also provide new ﬁnite-sample high-probability statistical results for k -NN density estimation’s ability to detect OOD examples
I NTRODUCTION
Identifying out-of-distribution examples has a wide range of applications in machine learning including fraud detection in credit cards (Awoyemi et al., 2017) and insurance claims (Bhowmik, 2011), fault detection and diagnosis in critical systems (Zhao et al., 2013), segmentations in medical imaging to ﬁnd abnormalities (Prastawa et al., 2004), network intrusion detection (Zhang & Zulkernine, 2006), patient monitoring and alerting Hauskrecht et al.
counter-terrorism
(Skillicorn, 2008) and anti-money laundering (Labib et al., 2020).
Out-of-distribution detection is highly related to the classical line of work in anomaly and outlier detection.
Such methods include density-based (Ester et al., 1996),
one-class SVM (Schölkopf et al., 2001), and isolation forest (Liu et al., 2008).
However, these classical methods often aren’t immediately practical on large and possibly high-dimensional modern datasets.
More recently, Hendrycks & Gimpel (2016) proposed a simple baseline for detecting out-ofdistribution examples by using a neural network’s softmax predictions, which has motivated many works since then that leverage deep learning (Lakshminarayanan et al., 2016; Liang et al., 2017; Lee et al., 2017).
However, the majority of the works still ultimately use the neural network’s softmax predictions which suffers from the following weakness.
The uncertainty in the softmax function cannot distinguish between the following situations: (1) the example is actually in-distribution but there is high uncertainty in its predictions and (2) the example is actually out of distribution.
This is largely because the softmax probabilities sum to 1 and thus must assign the probability weights accordingly.
This has motivated recent explorations in estimating conformal sets for neural networks (Park et al., 2019; Angelopoulos et al., 2020) which can distinguish between the two cases.
In this paper, we circumvent the above-mentioned weakness by avoiding using the softmax probabilities altogether.
To this end, we approach OOD detection with an alternative paradigm – we leverage the intermediate embeddings of the neural network and nearest neighbors.
Our intuition is backed by recent work in which the effectiveness of using nearest-neighbor based methods on these embeddings have been demonstrated on a range of problems such as uncertainty estimation (Jiang et al., 2018), adversarial robustness (Papernot & McDaniel, 2018), and noisy labels (Bahri et al., 2020).
In this work, we explore using k -NN density estimation to detect OOD examples by computing its density on the embedding layers.
To this end, it’s worth noting that k -NN density estimation is a unsupervised technique, which makes it very different from the aforementioned deep k -NN work (Bahri et al., 2020) which leverages the label information of the nearest neighbors.
One key intuition here is that low k -NN density examples might be OOD candidates as it implies that these examples are far from the training examples in the embedding space.
In order for density estimation to be effective on the intermediate embeddings, the data must have good clusterability (Ackerman & Ben-David, 2009), meaning that examples in the same class should be close together in distance in the embeddings, while examples not in the same class should be far apart.
While much work has been done for the speciﬁc problem of clustering deep learning embeddings (Xie et al., 2016a; Hershey et al., 2016)
many of these ideas are not applicable to density estimation.
In this paper, we use a much simpler but effective approach of label smoothing, which involves training the neural network on a soft label obtained by taking a weighted average between the original one-hot encoded label and the uniform distribution over labels.
We leverage a key insight about the effect of label smoothing on the embeddings (Müller et al., 2019): training with label smoothing has the effect of contracting the intermediate activations of the examples within the same class to be closer together at a faster rate relative to examples in different classes.
This results in embeddings that have better clusterability.
We call this the Label Smoothed
Embedding Hypothesis , which we deﬁne below.
Hypothesis 1 (Label Smoothed
Embedding Hypothesis (Müller et al., 2019)) .
Training with label smoothing contracts the intermediate embeddings of the examples in a neural network, where examples within the same class move closer towards each other in distance at a faster rate than examples in different classes.
We refer interested readers to Müller et al.
(2019) for 2D visualizations of this effect on the model’s penultimate layer.
We will later portray the same phenomenon using k -NN density estimation.
We summarize our contributions as follows:
(1) We propose a new procedure that uses label smoothing along with aggregating the k -NN density estimator across various intermediate representations to obtain an OOD score.
(2) We show a number of new theoretical results for the k -NN density estimator in the context of OOD detection, including guarantees on the recall and precision of identifying OOD examples, the preservation of the ranking w.r.t.
the true density, and a result that provides intuition for why the Label Smoothed
Embedding Hypothesis improves the k -NN based OOD score.
(3) We experimentally validate the effectiveness of our method and the beneﬁts of label smoothing on benchmark image classiﬁcation datasets, comparing against recent baselines, including one that uses k -NN in a different way, as well as classical alternatives to the k -NN but applied in the same way.
The comparison against these ablative models highlight the discriminative power of the k -NN density estimator for OOD detection.
(4) We conduct ablations to study the performance impact of the three hyper-parameters of our method - (1) the amount of label smoothing, (2) which intermediate layers to use, and (3) number of neighbors k .
A LGORITHM
We start by deﬁning the foundational quantity in our method.
Deﬁnition 1.
Deﬁne the k -NN radius of x ∈ R D as
[Equation]
When X is implicit, we drop it from the notation for brevity.
Our method goes as follows: upon training a classiﬁcation neural network on a sample X in from some distribution f in , the intermediate representations of X in should be close together (in the Euclidean sense), possibly clustered by class label.
Meanwhile, out-of-distribution points should be further away from the training manifold - that is, r k
( g i ( x out ); X in ) >
r k ( g i ( x in ); X in ) for x in ∼ f in , x out ∼ f out , where g i maps the input space to the output of the i -th layer of the trained model.
Thus, for ﬁxed layer i , we propose the following statistic:
[Equation]
Since Q depends on unknown f in we estimate it using cross-validation:
[Equation]
Letting ˆ T i be our statistic using ˆ Q , we now aggregate across M layers to form our ﬁnal statistic:
[Equation]
We use a one-sided threshold rule on ˆ T - namely, if ˆ T > t we predict out-of-distribution, otherwise we do not.
With key quantities now deﬁned, we use the k -NN radius to substantiate (1) the claim that in and out-of-distribution points are different distances away from the training points and (2)
Hypothesis 1, that label smoothing causes in-distribution points to contract to the training points faster than OOD ones.
This provides the grounding for why a statistic based on the k -NN radius using a label smoothed model is a powerful discriminator.
Figure 2 shows the distribution of 1 -NN distances for three layers as well as our proposed aggregate statistic on two dataset pairs.
Across layers and datasets, we see some separability between in and out-of-distributions points.
Label smoothing has the effect of shrinking these distances for both in/out classes but the effect is larger for in points, making the distributions even more separable and thereby improving the performance of our method.
T HEORETICAL R ESULTS
In this section, we provide statistical guarantees for using the k -NN radius as a method for out of distribution detection.
To do this, we assume that the features of the data lie on compact support
X ⊂ R d and that examples are drawn i.i.d.
from this.
We assume that there exists a density function f : R d → R corresponding to the distribution of the feature space.
This density function can serve as a proxy for how much an example is out of distribution.
The difﬁculty is that this underlying density function is unknown in practice.
Fortunately, we can show that the k -NN radius method approximates the information conveyed by f based on a ﬁnite sample drawn from f .
For the theory, we deﬁne an out of distribution example as an example where x (cid:54)∈ X .
Thus, f ( x ) = 0 for such examples.
3. O UT OF DISTRIBUTION DETECTION HIGH - RECALL RESULT
In this section, we give a result about identifying out of distribution examples based on the k -NN radius with perfect recall if we were to use a particular threshold.
That is, any example that is indeed out of distribution (i.e. has 0 density) will have k -NN radius above that threshold.
We also give a guarantee that the false-positives (i.e. those examples with k -NN radius higher than that quantity which were not out-of-distribution examples) were of low-density to begin with.
Our results hold with high-probability uniformly across all of R d .
As we will see, as n grows and k/n → 0 , we ﬁnd that the k -NN radius method using the speciﬁed threshold is able to identify which examples are in-distribution vs out-of-distribution.
Our result requires a smoothness assumption on the density function shown below.
This smoothness assumption ensures a relationship between the density of a point and the probability mass of balls around that point which is used in the proofs.
Assumption 1 (Smoothness) .
f is β -Holder continuous for some 0 < β ≤ 1 .
i.e. | f ( x ) − f
( x (cid:48) )
| ≤
C β | x − x (cid:48) | β .
We now give our result below.
Theorem 1.
Suppose that Assumption 1 holds and that 0 < δ < 1 and k ≥ 2 8 · log (2 /δ )
2 · d log n .
If we choose
[Equation]
then the following holds uniformly for all x ∈ R d with probability at least 1 − δ : (1) If f ( x ) = 0 , then r k ( x ) ≥ r .
(2) If r k ( x ) ≥ r , then f ( x ) ≤ λ .
In words, it says that the set of points x ∈ R d satisfying r k ( x ) (cid:38) ( k/n ) 1 / ( β + d ) , is guaranteed to contain all of the outliers and does not contain any points whose density exceeds a cutoff (i.e. f ( x ) (cid:38) ( k/n ) β/ ( β + d ) ).
These quantities all go to 0 as k/n → 0 and thus with enough samples, asymptotically are able to distinguish between out-of-distribution and in-distribution examples.
We can assume the following condition on the boundary smoothness of the density as is done in a recent analysis of k -NN density estimation (Zhao & Lai, 2020).
Assumption 2 (Boundary smoothness) .
There exists 0 < η ≤ 1 such that for any t > 0 , f satisﬁes
[Equation]
where P represents the distribution of in-distribution examples during evaluation.
Then, Theorem 1 has the following consequence on the precision and recall of the k -NN density based out of distribution detection method.
Corollary 1.
Suppose that Assumptions 1 and 2 hold and that 0 < δ < 1 and k ≥ 2 8 · log (2 /δ ) 2 · d log n .
Then if we choose
[Equation]
then the following holds with probability at least 1 − δ .
Let us classify an example x ∈ R d as out of distribution if r k ( x ) ≥ r and in-distribution otherwise.
Then, this classiﬁer will identify all of the out-of-distribution examples (perfect recall) and falsely identify in-distribution examples as out-of-distribution with probability (error in precision)
[Equation]
3. R ANKING PRESERVATION RESULT
We next give the following result saying that if the gap in density between two points is large enough, then their rankings will be preserved w.r.t.
the k -NN radius.
Theorem 2.
Suppose that Assumption 1 holds and that 0 < δ < 1 and k ≥ 2 8 · log (2 /δ )
2 · d log n .
Deﬁne C δ,n := 16 log(2 /δ )
√ d log n .
Then there exists a constant C depending on f such that the following holds with probability at least 1 − δ uniformly for all pairs of points x 1 , x 2 ∈ R d .
If f ( x 1 ) > f ( x 2 ) + (cid:15) k,n , where
[Equation]
then, we have r k ( x 1 ) <
r k ( x 2 ) .
We note that as n, k → ∞ , k/n → 0 , and log n/ √ k → 0 , we have that (cid:15) k,n → 0 and thus asymptotically, the k -NN radius preserves the ranking by density in the case of non-ties.
3. P ERFORMANCE UNDER LABEL SMOOTHING EMBEDDING HYPOTHESIS
In this section, we provide some theoretical intuition behind why the observed label smoothing embedding hypothesis can lead to better performance for the k -NN density-based approach on embeddings learned with label smoothing.
We make an assumption that our in-distribution has a convex set X ⊆ R d as its support with uniformly lower bounded density and that applying label smoothing has the effect of contracting the space R d in the following way: for points in X the contraction is with respect to a point of origin x 0 in the interior of X so that points in X move closer to the origin and for outlier points, they move closer to the boundary of X .
We ensure that the former happens at a faster rate than the latter and show the following guarantee, which says that under certain regularity conditions on the density and X , we have that the ratio of the k -NN distance between an out-of-distribution point and an in-distribution point increases after this mapping.
This suggest that under such transformations such as ones induced by what’s implied by the label smoothed embedding hypothesis, the k -NN distance becomes a better score at separating the in-distribution examples from the out-of-distribution examples.
Proposition 1 (Improvement of k -NN OOD with Label Smoothed Embedding Hypothesis) .
Let f has convex and bounded support X ⊆ R d and let x 0 be an interior point of X and additionally assume that there exists r 0 , c 0 > 0 such that for all 0 < r < r 0 and x ∈ X , we have Vol ( B ( x, r ) ∩ X ) ≤ c 0 · Vol ( B ( x, r )) holds (to ensure that X ’s boundaries have regularity and are full dimensional) and that f ( x ) ≥ λ 0 for all x ∈ X for some λ 0 > 0 .
Deﬁne mapping φ : R d → R d such that φ ( x ) = γ in · ( x − x 0 ) − x 0 if x ∈ X and otherwise, φ ( x ) = γ out · ( x − Proj X
( x )) − Proj X
( x ) otherwise, for some 0 < γ in < γ out < 1 .
(Proj X ( x ) denotes the projection of x onto the boundary of convex set X ).
We see that φ contracts the points where points in X contract at a faster rate than those outside of X .
Suppose our training set consists of n examples X [ n ] drawn from f , and denote by φ ( X [ n ] ) the image of those examples w.r.t.
φ .
Let 0 < δ < 1 and r min > 0 and k satisﬁes
[Equation]
and n is sufﬁciently large depending on f .
Then with probability at least 1 − δ , the following holds uniformly among all r min > 0 , choices of x in ∈ X (in-distribution example) and x out such that d ( x, X ) ≥ r min (out-of-distribution example with margin).
The following holds.
[Equation]
where r k ( x, A ) denotes the k -NN distance of x w.r.t.
dataset A .
E XPERIMENTS 4. S ETUP
We validate our method on MNIST (LeCun et al., 1998), Fashion MNIST (Xiao et al., 2017), SVHN (cropped to 32x32x3) (Netzer et al., 2011), CIFAR10 (32x32x3) (Krizhevsky et al., 2009), and
CelebA (32x32x3) (Liu et al., 2015).
In CelebA, we train against the binary label “smiling”.
We train models on the train split of each of these datasets, and then test OOD binary classiﬁcation performance for a variety of OOD datasets, while always keeping the in-distribution to be the test split of the dataset used for training.
Thus, a dataset pairing denoted “A → B” means that the classiﬁcation model is trained on A’s train and is evaluated for OOD detection using A’s test as in-distribution points and B’s test as out-of-distribution points.
In addition to the aforementioned, we form OOD datasets by corrupting the in-distribution test sets - by ﬂipping images left and right (HFlip) as well as up and down (VFlip) - and we also use the validation split of ImageNet (32x32x3), the test splits of
KMNIST (28x28x1), EMNIST digits (28x28x1), and Omniglot (32x32x3).
All datasets are available as Tensorﬂow Datasets 1 .
We measure the OOD detectors’ ROC-AUC, sample-weighting to ensure balance between in and out-of-distribution samples (since they can have different sizes).
For MNIST and Fashion MNIST, we train a 3-layer ReLU-activated DNN, with 256 units per layer, for 20 epochs.
For SVHN, CIFAR10, and CelebA, we train the convolutional LeNet5 (LeCun et al., 2015) forepochs.
We use 128 batch size and Adam optimizer with default learning rate 0.001 throughout.
All methods were implemented in TensorFlow and trained on a cloud environment.
We estimate we used a total of 10k CPU hours for all of our experiments.
For embedding-based methods, we aggregate over 3 layers for the DNN and 4 dense layers for LeNet5, including the logits.
For our method, we always use Euclidean distance between embeddings, k = 1 and label smoothing α = 0 .
These could likely be tuned for better performance in the presence of a validation OOD dataset sufﬁciently similar to the unknown test set.
We do not do this since we assume the absence of such dataset.
4. B ASELINES
We validate our method against the following recent baselines:
Control.
We use the model’s maximum softmax conﬁdence, as suggested by Hendrycks & Gimpel (2016).
The lower the conﬁdence, the more likely the example is to be OOD.
Robust Deep k -NN.
This method, proposed in Papernot & McDaniel (2018) leverages k -NN for a query input as follows: it computes the label distribution of the query point’s nearest training points for each layer and then computes a layer-aggregated p -value-based non-conformity score against a held-out calibration set.
Queries that have high disagreement, or impurity, in their nearest neighbor label set are suspected to be OOD.
We use 10% of the training set for calibration, k = 50 , and cosine similarity, as described in the paper.
DeConf.
Hsu et al.
(2020) improves over the popular method ODIN (Liang et al., 2017) by freeing it from the needs of tuning on OOD data.
It consists of two components - a learned “conﬁdence decomposition” derived from the model’s penultimate layer, and a modiﬁed method for perturbing inputs optimally for OOD detection using a Fast-Sign-Gradient-esque strategy.
We use the “h” branch of the cosine similarity variant described in the paper.
We searched the perturbation hyperparameter (cid:15)
[Table]
over the range listed in the paper, but found that it never helped OOD in our setting.
We thus reports numbers for (cid:15)
= 0 .
SVM.
We learn a one-class SVM (Schölkopf et al., 1999) on the intermediate embedding layers and then aggregate the outlier scores across layers in the same way we propose in our method.
We use an
RBF kernel.
Isolation Forest.
This is similar to SVM, but uses an isolation forest (Liu et al., 2008) withestimators at each layer.
4. R ESULTS
Our main results are shown in Table 1.
We observe that label smoothing nearly always improved our method, denoted k -NN, and that the method is competitive, outperforming the rest on the most number of dataset pairs.
SVM, Isolation Forest serve as key ablative models, since they leverage the same intermediate layer representations as our method and their layer-level scores are combined in the same way.
Interestingly, we see that the k -NN consistently outperforms them, revealing the discriminative power of the k -NN radius distance.
Robust k -NN also uses the same layer embeddings and k -NN, but in a different manner.
Crucially, it performs OOD detection by means of the nearest training example neighbors’ class label distribution.
Given that we outperform Robust k -NN more often than not we might conjecture that the distance has more discriminative power for OOD detection than class label distribution .
We were surprised that DeConf routinely did worse than the simple control, despite having implementing the method following the paper closely.
4. A BLATIONS
In this section, we study the impact of three factors on our method’s performance: (1) the number of neighbors, k , (2) the amount of label smoothing α , and (3) the intermediate layers used.
Impact of k .
In Figure 2 we plot the impact of k on OOD detection for two dataset pairings: MNIST → Fashion MNIST and SVHN → CIFAR10 with and without label smoothing.
We see that larger k degrades ROC-AUC monotonically, but the effect is rather small.
We thus recommend a default of k = 1 .
k = 1 has the added beneﬁt of being more efﬁcient in most implementations of index-based large-scale nearest-neighbor lookup systems.
Impact of Label Smoothing α .
We now consider the effect of label smoothing amount α on ROCAUC in Figure 2.
We see, interestingly, that performance mostly increases monotonically with larger α until it reaches a critical point, after which it declines monotonically.
While this optimal point may be data and model dependent and thus hard to estimate, we’ve found that selecting a ﬁxed value like 0.1 works well in most cases.
Impact of Intermediate Layer .
Our method aggregates k -NN distance scores across intermediate layers.
We depict the effect of different choices of a single layer on Fashion MNIST and CelebA in
Figure 3.
We ﬁnd that label smoothing generally boosts performance for each layer individually and that while no single layer is always optimal, the penultimate layer performs fairly well across the datasets.
R ELATED W ORK
Out-of-Distribution Detection.
OOD detection has classically been studied under names such as outlier, anomaly, or novelty detection.
One line of work are density-based methods: Ester et al.
(1996) presents a density-based clustering algorithm which is also an outlier detection algorithm by identifying noise points which are points whose (cid:15) -neighborhood has fewer than a certain number of points.
Breunig et al.
Kriegel et al.
(2009) propose local outlier scores based on the degree to which how isolated the datapoint is with respect to its neighborhood via density estimation.
Another line of work uses k -NN density estimates (Ramaswamy et al., 2000; Angiulli & Pizzuti, 2002; Hautamaki et al., 2004; Dang et al., 2015).
We use the k -NN density estimator, but use it in conjunction with the embeddings of a neural network trained with label smoothing.
Other classical approaches include the one-class SVM (Schölkopf et al., 2001; Chen et al., 2001), isolation forest (Liu et al., 2008).
A slew of recent methods have been proposed for OOD.
We refer interested readers to a survey.
Label Smoothing.
Label smoothing has received much attention lately; we give a brief review here.
It has been shown to improve model calibration (and therefore the generation quality of autoregressive sequence models like machine translation) but has been seen to hurt teacher-to-student knowledge distillation (Pereyra et al., 2017; Xie et al., 2016b; Chorowski & Jaitly, 2016; Gao et al., 2020; Lukasik et al., 2020b; Müller et al., 2019).
Müller et al.
(2019) show visually that label smoothing encourages the penultimate layer representations of the training examples from the same class to group in tight clusters.
Lukasik et al.
(2020a) shows that label smoothing makes models more robust to label noise in the training data (to a level competitive with noisy label correction methods), and, furthermore, smoothing the teacher is beneﬁcial when distilling from noisy data.
Chen et al.
(2020) corroborates the beneﬁts of smoothing for noisy labels and provides a theoretical framework wherein the optimal smoothing parameter α can be identiﬁed.
LS has been seen to hurt performance on sparse distributions (Meister et al., 2020) and decrease robustness to adversarial attacks (Zantedeschi et al., 2017).
Yuan et al.
(2020) casts knowledge distillation (KD) as a type of learned label smoothing regularization, showing that part of KD’s success stems from its ability to regularize soft labels in the same way as LS.
They then propose Teacher-free KD that achieves comparable performance to normal KD with a superior teacher.
k -NN Density Estimation Theory.
Statistical guarantees for k -NN density estimation has had a long history (Fukunaga & Hostetler, 1973; Devroye & Wagner, 1977; Mack, 1983; Buturovi´c, 1993; Biau et al., 2011; Kung et al., 2012).
Most works focus on showing convergence guarantees under metrics like L 2 risk or are asymptotic.
Dasgupta & Kpotufe (2014) provided the ﬁrst ﬁnitesample uniform rates, which to our knowledge is the strongest result so far.
Our analysis uses similar techniques, which they also borrow from Chaudhuri & Dasgupta (2010); however our results are for the application of OOD detection whereas Dasgupta & Kpotufe (2014)’s goal was mode estimation.
As a result, our results hold with high probability uniformly in the input space, while having ﬁnite-sample guarantees and provide new theoretical insights into the use of k -NN for OOD detection.
C ONCLUSION
In this work we put forward the Label Smoothing Embedding Hypothesis and proposed a deep k -NN density-based method for out-of-distribution detection that leverages the separability of intermediate layer embeddings and showed how label smoothing the model improves our method. Appendix A P ROOFS
We need the following result giving guarantees between the probability measure on the true balls and the empirical balls.
Lemma 1 (Uniform convergence of balls (Chaudhuri & Dasgupta, 2010)) .
Let F be the distribution corresponding to f and F n be the empirical distribution corresponding to the sample X .
Pick 0 < δ < 1 .
Assume that k ≥ d log n .
Then with probability at least 1 − δ , for every ball B ⊂ R D we have
[Equation]
where C δ,n = 16 log(2 /δ ) √ d log n
Remark.
For the rest of the paper, many results are qualiﬁed to hold with probability at least 1 − δ .
This is precisely the event in which Lemma 1 holds.
Remark.
If δ = 1 /n , then C δ,n = O ((log n ) 3 / 2 ) .
Proof of Theorem 1.
Suppose that x satisﬁes f ( x ) = 0 .
Then we have
[Equation]
Therefore, by Lemma 1, we have that r k ( x ) ≤ r .
Now for the second part, we prove the contrapositive.
Suppose that f ( x ) > λ .
Then we have
[Equation]
Therefore, by Lemma 1, we have that f ( x ) ≥ λ , as desired.
Proof of Theorem 2.
We borrow some proof techniques used in Dasgupta & Kpotufe (2014) to give uniform bounds on | f ( x ) − f k ( x ) | where f k is the k -NN density estimator deﬁned as
[Equation]
It is also clear that r ≤ C (cid:48) · ( k/n ) 1 /d for some C (cid:48) depending on f .
If we choose r such that
[Equation]
Then, we have by Lemma 1 that r k ( x ) > r .
Thus, we have
[Equation]
for some C 1 > 0 depending on f .
The argument for the other direction is similar: we instead choose r such that
[Equation]
Again it’s clear that r ≤ C (cid:48)(cid:48) · ( k/n ) 1 /d for some C (cid:48)(cid:48) depending on f .
Next, we have by Lemmathat
r k ( x ) ≤ r .
Thus, we have
[Equation]
for some C 2 depending on f .
Therefore, there exists C 0 depending on f such that
[Equation]
Finally, we have that setting C = 2 · C 0 , we have
[Equation]
then it immediately follows that r k ( x 1 ) <
r k ( x 2 ) , as desired.
Proof of Proposition 1.
Deﬁne r u := max x ∈X r k ( x ) .
We have
[Equation]
Next, we have
[Equation]
since all pairwise distances within X are scaled by γ in through our mapping φ .
Thus, it sufﬁces to have
[Equation]
which is equivalent to having
[Equation]
Which holds when
[Equation]
This holds because we have r u ≤ (cid:16)
[Equation]
nv d (cid:17) 1 /d by Lemma 1, and the result follows by the condition on k .