DIME: An Information-Theoretic Difﬁculty Measure for AI Datasets
Abstract
Evaluating the relative difﬁculty of widely-used benchmark datasets across time andacross data modalities is important for accurately measuring progress in machinelearning.
To help tackle this problem, we propose DIME , an information-theoretic
DI fﬁculty ME asure for datasets, based on Fano’s inequality and a neural networkestimation of the conditional entropy of the sample-label distribution.
DIME can bedecomposed into components attributable to the data distribution and the numberof samples.
DIME can also compute per-class difﬁculty scores.
Through extensiveexperiments on both vision and language datasets, we show that DIME is wellaligned with empirically observed performance of state-of-the-art machine learningmodels.
We hope that DIME can aid future dataset design and model-trainingstrategies.
Introduction
Empirical machine learning research relies heavily on comparing performance of algorithms on afew standard benchmark datasets.
Moreover, researchers frequently introduce new datasets that theybelieve to be more challenging than existing benchmarks.
However, we lack objective measuresof dataset difﬁculty that are independent of the choices made about algorithm- and model-design.
Moreover, it is also hard to compare algorithmic progress across data modalities, such as languageand vision.
So, for instance, it is difﬁcult to compare the relative progress made on a sentimentanalysis benchmark such as the Stanford Sentiment Treebank (SST)
(Socher et al., 2013) and animage classiﬁcation benchmark, like CIFAR-10 (Krizhevsky, 2009).
With these challenges in mind,we propose a model-agnostic and modality-agnostic measure for comparing how difﬁcult it is toperform supervised learning on a given dataset.
For i.i.d.
dataset examples, we argue that the difﬁculty of a dataset can be decoupled into tworelatively independent sources:(a) approximation complexity, the number of samples required toapproximate the true distribution up to certain accuracy, and (b) distributional complexity, the intrinsicdifﬁculty involved in modeling the statistical relationship between the labels and features.
We focus our analysis on the second source of the intrinsic difﬁculty in supervised learning, andconsider the most common setting where features X are continuous and the labels Y are discrete.
Toprovide a model-agnostic measure, we turn to the information-theoretic approach.
Indeed, Fano’sinequality suggests the lowest possible probability of the 0 - 1 error P e is lower bounded by termsrelated to the conditional entropy H ( Y | X ) .
Naturally we consider designing an estimator for thislower bound of P e as a difﬁculty measure for learning Y given X .
To estimate H ( Y | X ) , noticethat H ( Y | X )
=
H ( Y ) − I ( X, Y ) , where I ( X, Y ) is the mutual information.
Direct estimation of 33 I ( X, Y ) is hard for the hybrid case where X is continuous and Y is discrete.
Our key observationis that we can express I ( X, Y ) as a weighted average of the KL divergence between X | Y =
y and
D I
M
E networks
Related Work: Although conditional entropy and mutual information estimation have been extensively studied, research has focused on purely discrete or continuous data.
Nair et al.
(2006) wereamong the ﬁrst to study the theoretical properties for the case of mixture of discrete and continuousvariables.
Ross (2014), Gao et al.
(2017) and Beknazaryan et al.
(2019) proposed approaches forestimating mutual information for the mixture case based on density ratio estimation (e.g.,binning,kNN or kernel methods), which is unsatisfactory for high dimensional data such as image and text.
We use neural network estimation, inspired by Belghazi et al.
(2018), to avoid these issues.
However,we emphasize that the technique for estimating the mutual information in Belghazi et al.
(2018)cannot be applied directly to the hybrid case, as argued before.
More importantly, we are the ﬁrst toconnect the hybrid conditional entropy with the difﬁculty measure for datasets.
Designing a Dataset Difﬁculty Measure2.1 Main Idea
For the continuous feature and discrete label setting, we use ( X, Y ) to represent feature-label pairsand assume X ∈ X ⊂ R d x , and Y ∈ Y ⊂
Z + .
We denote the joint distribution of ( X, Y ) as P XYand the marginal distributions as P X and P Y respectively.
Throughout this paper, we always assumethat the feature-label pairs in the datasets, both training and testing, are sampled i.i.d.
from a staticdistribution P XY .
As Y is discrete, a natural measure that characterizes the difﬁculty of the datadistribution is the best probability of the 0-1 error that can be achieved by any estimator.
Deﬁnition 1.
Model-Agnostic Error
[Equation]
While the computation of general P e is almost impossible, Fano’s inequality establishes a lowerbound of P e by terms related to the conditional entropy.
Deﬁnition 2.
Hybrid conditional entropy
[Equation]
where P ( Y = y | X ) :
=
E [ 1 ( Y = y )
| X ] .
Theorem 1.
Fano’s inequality for continuous feature-discrete label case : Let P e be the minimumerror probability as deﬁned in (1) .
Then we have
[Equation]
where H ( P e ) :=
−P e log P e − (1 − P e ) log(1 − P e ) .
A detailed proof is given in Appendix C.
Our proof is based on the proof of Lemma 2.10, Tsybakov(2004).
Note that most of the proofs of Fano’s inequality are designed for discrete random variables(e.g.
Cover & Thomas (2012)).
2.2 Estimating the Lower Bound of P e
Motivated by Fano’s inequality, we just need to estimate H ( Y | X ) .
We rewrite it as (see Appendix Bfor a proof)
[Equation]
with
[Equation]
Remark 1.
In some benchmark datasets with balanced classes (e.g., CIFAR-10 and MNIST),
[Equation]
X | Y
=
y || X ) .
This indicates that if a dataset has moreclasses and the distributions of the features for different classes are closer to each other on average,then H ( Y | X ) would be larger and thus pushes P e larger, implying that the classiﬁcation task on itwould be harder due to the intrinsic distributional complexity.
We now discuss the practical design of an estimator for KL ( X | Y =
y || X ) .
(Notice that both of therandom varibles are continuous.)
Many non-parametric KL divergence estimators such as k NN basedtechniques (Noshad et al., 2017; Perez-Cruz, 2008; Wang et al., 2009) are not satisfying as they oftenrely on some knowledge of the internal dimension of the data manifold.
As an alternative, we designa neural network-based estimator for KL divergence estimation, inspired by Belghazi et al.
The estimator is based on the Donsker-Varadhan representation:
[Equation]
for two probability measures P , Q .
If we parameterize the function T using a neural network T θ ,we can obtain an estimator of (cid:100) KL ( X | y || X ) for each y ∈ Y .
Then by equation (4), we get (cid:98) H ( Y | X ) ,which further leads to (cid:99) P e by (1), an estimator of the lower bound of P e .
We report (cid:99) P e as our datatsetdifﬁculty measure DIME.
In our empirical evaluation, we optimize the empirical average instead of the expectation.
Howeverthis may cause issues such as overﬁtting.
To mitigate this issue, we split the data into separate trainingand validation sets.
The neural network models are trained on the training set and the estimation ismade using the validation set.
See Algorithm 1 in Appendix A.1 for details.
Remark 2.
Given that we are using a neural network model to estimate the KL divergence between
P ( X | y ) and P ( X ) , according to Equation (5), (cid:100) KL ( X | y || X ) ≤ KL ( X | y || X ) .
As a consequence,the estimated mutual information (cid:98) I ( X ; Y ) ≤ I ( X ; Y ) , which leads to a larger (cid:98) H ( Y | X ) .
In the end,
DIME could be larger as compared to true lower bound on P e
.The hope is, if the function class of theneural network is large enough, the gap between (cid:100) KL ( X | y || X ) and KL ( X
| y || X ) is small enough sothat DIME won’t deviate too much from the true lower bound of P e .
Experiments and Results
3.1 Sanity-check Experiments
Experiments with toy datasets: We generate three toy datasets and report their DIME socres, asshown in Figure 1.
The details are given in Appendix A.2, which contains a label corruption test aswell.
3.2 Evaluating benchmark datasets in vision and language
We evaluate DIME using popular image and language domain datasets for various classiﬁcation tasks.
We use a simple multi-layer perceptron (MLP) with residual layers with ReLU activation functions,containing three hidden layers of 4096 neurons.
(The details about the datasets and neural network wechoose are given in Appendix A.3.)
In addition to dataset statistics, we report the estimated entropy(cid:98) H ( Y ) , (cid:98) I
( X, Y ) , (cid:98) H ( Y | X ) , and DIME.
We also report the state-of-the-art error on the dataset fromrecent literature (Wan et al., 2013; Huang et al., 2018; Cubuk et al., 2019; Yang et al., 2019; Patroet al., 2018; Cer et al., 2018; Melis et al., 2019).
From the results summarized in Table 1, we makethe following empirical observations:
Corpus classes (cid:98) H ( Y ) (cid:98) I ( X, Y ) (cid:98) H ( Y | X )
DIME
SOTA Error
[Table]
Language Modeling
[Table]
First, DIME aligns well with the state-of-the-art results on the benchmark datasets, indicating that therelative difﬁculty of datasets is reﬂected in the performance of latest machine learning algorithms.
However, notably, DIME is lower than the reported state-of-the-art error for all the variants of
EMNIST.
This could indicate that since EMNIST is a relatively new, less-investigated dataset, wemight see model performance improving over time.
Second, a larger number of classes lead to a higher value of DIME in general, which is expectedgiven its dependence on (cid:98) H ( Y ) .
But (cid:98) H ( Y ) does not always dominate the value of the measure.
Forexample, MNIST has 10 classes, but its DIME is much smaller compared to IMDb which has onlytwo classes.
This relative difference in difﬁculty of MNIST and IMDb is also reﬂected in their SOTAerrors, validating our measure.
Finally, for datasets such as MNIST, Fashion MNIST, EMNIST, FakeData, and DBPedia, DIMEprovides a tight error bound.
But for certain datasets such as CIFAR-10, CIFAR-100, and Tiny
ImageNet, DIME seem slightly pessimistic.
As we discussed in Remark 2, DIME could be largerthan the true lower bound of P e since we constrain the function space to be a neural network model.
While the relative order of difﬁculty suggested by DIME is reasonable, its large value also suggeststhat an MLP may not have enough capacity for accurate P e estimation.
While it is an open questionhow to choose a model that is large enough to approximate P e yet easy to optimize, Equation (5)suggests that as the model size grows, the gap between DIME and true lower bound of P e becomessmaller.
We verify this by calculating DIME with increasing number of neurons for a three-layer
MLP.
See Figure 4 in Appendix A.5 for details.
As we mentioned in the abstract, we can also use DIME to compute per-class difﬁculty score andreﬂect the effect of number of samples on data distribution, etc.
See Appendix A.4 for the additionaluse cases for DIME.
Discussion
We would like to use this section to discuss some common questions proposed by the reviewers andreaders.1. Why don’t we use a cross entropy estimation to approximate the conditional entropy H ( Y | X ) (which is also an upper bound of H ( Y | X ) )?
We want to emphasize that our motivation here is to design a metric for datasetsto capture the intrinsic complexity of the sample-label distribution in supervised learning.
Our intuition is that if a dataset has a large number of classes and the distributions of thefeatures for each class are close to each other, then it would be difﬁcult to accurately predictthe labels given the features.
Therefore, Our starting point is to deﬁne a metric of thefollowing form:
[Equation]
for some increasing function f and some distance function D over distributions. We ﬁnd agood choice of ( f, D ) is (log , KL ) and this exactly leads to the deﬁnition of H ( Y | X ) . Thenwe start to consider how to estimate the KL divergence for some high-dimensional data. Therefore, even if we can use a neural network via a cross entropy approach to approximate
H ( Y | X ) —
-and we expect the approximation would be more accurate than our approachas the distribution of X | Y is generally much more complicated than that of Y | X —-westill avoid this approach since it is just meaningless here, given that the spirit of this paperis to show that the distance over the distributions of features for each class is an essentialindicator of the complexity of the label-feature distributions. Since our focus and interestare the distributions X | Y = y , an approach to approximate Y | X is just another trial to dothe prediction and thus, is not appropriate here to provide a model-free measure.
On the other hand, we have to point out that one main limitation of our approach is that itis pretty challenging to approximate the distribution of X | Y (i.e. achieving the supremumin (5)). A simple MLP neural network, as we used here, seems far from satisfying giventhat the fact that DIME is almost always larger than the SOTA error in Table 1. Therefore,it seems worthwhile to try some neural networks with more capacity or more complicatedarchitectures like convnets, LSTMs, transformers, etc. (Of course, then it would be morecontroversial to claim our metric "model-free".)2.
Why do you use MINE (Belghazi et al.
(2018)) to estimate the mutual information (which is not a valid lower We actually use a MINE style estimator to estimate the KL divergence instead ofthe mutual information. So some other methods to estimate the mutual information (see e.g. Alemi et al.
for an upper bound; Oord et al.
(2018) for a lower bound) are not exactlyrelevant here.
That being said, it is deﬁnitely worthwhile to try these methods to directlyestimate the mutual information (epsecially those for upper bounds such that we end upwith a valid lower bound of H ( Y | X ) ) and compare the results with the current ones.
Weleave this for future explorations. A Supplementary Results
A.1 Algorithm for DIME
Algorithm 1 DIME
Initialize |Y| neural networks: T θ 1 , T θ 2 , . .
, T θ |Y| , one per class.
Initialize class counters k 1 = 0 , . . .
, k |Y| = 0 , one per class.
Initialize the sample counter k = 0 .
for t in 0 , . . .
, T do
Draw a batch of b training examples { ( x i , y i ) } , and b evaluation examples { ( x ei , y ei ) } .
for each class c ∈ { 1 , . . .
, |Y|} do
Find samples of the c -th class:
S c = { i | y i =
c } , S ec =
{ i | y e i =
c }
Train θ c = arg max θ c 1 | S c | (cid:80) S c T θ c ( x i ) − log[ 1 b (cid:80) j exp T θ c ( x j )]
Evaluate (cid:100) KL θ c = 1 | S ec | (cid:80) S ec T
θ c ( x ei ) − log[ 1 b (cid:80) j exp T θ c ( x ej )]
If (cid:100) KL θ c stops increasing, pause the training for T θ c .
Update counters: k c + = | S ec | .
end for
Update the sample counter: k + = b
If all neural networks stop updating, break.
end for
Estimate the class probability (cid:98) p c = k c k , c
= { 1 , . .
. , |Y|} .
Calculate (cid:98) H ( Y | X ) using (4), (cid:98) p c , and (cid:100) KL
θ c , c
= { 1 , . .
. , |Y|} .
Solve (3) and output the solution.
A.2 Supplement Results for Section 3.
Experiments with toy datasets: We generate three two-dimensional 2-class toy datasets with20,000 samples each with increasing level of difﬁculty (Figure 1).
Samples in Toy Data 1 and Toy
Data 2 are generated using random Gaussian variables by scaling differently in different dimensionsand then by rotating, while the two classes in Toy Data 3 are simply uniform and Gaussian randomvariables.
Toy Data 1, which is relatively easy to be separated, gets a DIME of 0.001.
Toy Data 2,which is a bit harder due to a small overlapping region around the axis origin, gets a DIME of 0.013.
And Toy Data 3, which is the hardest due to overlapping supports of the two class, gets a DIME of0.263.
In sum, DIME accurately reﬂects the relative difﬁculty of the toy examples.
Label corruption test: Next, we estimate DIME for the case of label corruption.
If we assignrandom labels to a fraction of the training samples, intuitively the dataset should become moredifﬁcult.
We experiment with ﬁve datasets with ten classes each—MNIST, Fashion MNIST, CIFAR10, EMNIST (digits), and SVHN—and perform label corruption on their training sets in steps of0.1 from 0 to 1.
We observe a consistent increase in DIME across all the datasets with increasinglabel corruption (Figure 2-(left)).
For some datasets such as MNIST, DIME is almost tight given thefraction of randomly corrupted labels.
Label Corruption Rate
D
I M
E
CIFAR-10 MNIST
FASHION SVHN
EMNIST-LETTERS 0.
Fraction of Samples
[Figure]
[Figure]
Results for
A.3.1 Choice of Benchmark Datasets
We evaluate the following image classiﬁcation datasets: MNIST (LeCun et al., 1998), Extended
MNIST (EMNIST)
(Cohen et al., 2017), Fashion MNIST (Xiao et al., 2017), CIFAR-10 (Krizhevsky,2009), CIFAR-100 (Krizhevsky, 2009), Tiny ImageNet 1 , and SVHN (Netzer et al., 2011).
The
EMNIST dataset has several different splits, which include splits by digits, letters, merge, class,and balanced.
For a controlled evaluation study, we also add the PyTorch Fakedata dataset 2 whichcontains 10 -classes of randomly-generated 32 × 32 grayscale images.
Among natural language processing datasets, we investigate three supervised scenarios: sentimentanalysis, text classiﬁcation, and language modeling.
For sentiment analysis, we analyze IMDb (Maaset al., 2011), and Stanford Sentiment Treebank (SST) (Socher et al., 2013).
In the case of SST, weutilize both ﬁne-grained labels (SST-5) and binary labels (SST-2).
For the text classiﬁcation task, weuse the TREC dataset (Li & Roth, 2002), AG News, DBPedia, as well as Yelp Review dataset (Zhanget al., 2015).
For language modeling task, we analyze the character level modeling for Penn Tree
Bank (Marcus et al., 1994).
For DIME estimation, we do not perform any preprocessing on the data except for simple rescaling.
For image data we rescale all the pixel values to ﬂoats in [0 , 1] .
By data processing inequality (Cover& Thomas, 2012), the rescaling will not cause any change on H ( Y | X ) and P e since the rescalingfunction is invertible.
A.3.2 Choice of Neural Network Estimator
To estimate DIME for image datasets, we use a simple multi-layer perceptron (MLP) with residuallayers with Rectiﬁed Linear Unit (ReLU) activation functions.
The network has three hidden layersof 4096 neurons.
For language datasets, we use an embedding layer of dimension 1500 , followed bythree layers of 256 hidden neurons with residual connections and ReLU activations.
We found that ahigher dimension of the embedding helps achieve a tighter bound.
The embedding layer is initializedwith a concatenation of ﬁve pretrained embedding vectors: GloVe-840B-300d, GloVe-42B-300d,
GloVe-twitter-27B-200d, GloVe-6B-300d, FastText-en-300d, and CharNGram-100d.
The inputsequences are truncated or padded to the length of 128.
The MLP is operating on the token dimensioninstead of the embedding dimension to take the order of the sequence into consideration.
For example,the ﬁrst layer of MLP is 128-by-256 instead of 1500-by-256.
The last fully-connected layer ﬂattenseverything into a vector and projects it to a scalar.
For optimizing the MLP, we use SGD with initiallearning rate of 0.1 and anneal it to 0.01 and 0.001 if the objective stops updating.
We use the test setas evaluation set, as described in Algorithm 1. tiny
imagenet
com/
https
com
A.4 Additional Use Cases for DIME
Dataset difﬁculty per class: Algorithm 1 can also estimate the relative difﬁculty of classes withina dataset.
For this purpose, we simply rank the classes by their (cid:100) KL ( X | y || X ) .
A smaller value of(cid:100) KL ( X | y || X ) indicates the class is harder to classify.
As examples in Figure 3 show, ‘cat’ and ‘bird’are the most difﬁcult classes in CIFAR-10, while ‘automobile’ and ‘ship’ are the easiest.
In addition,‘shirt’ and ‘pullover’ are the most difﬁcult classes in Fashion MNIST, while ‘sandal’ and ‘trouser’ arethe easiest.
Controlled Study with CIFAR Subsets:
We obtain a 10-class subset from CIFAR-100 by choosing subclasses from the ‘large carnivores’ and ‘large omnivores and herbivores’ superclasses.
Theseclasses—bear, leopard, lion, tiger, wolf, camel, cattle, chimpanzee, elephant, and kangaroo—shouldbe intuitively harder to classify as compared to CIFAR-10, which includes disparate classes suchas bird, truck, and ship.
Note that the image data for both of these datasets comes from the same,larger ‘80 million tiny images’ dataset (Torralba et al., 2008).
We ﬁnd that the DIME for this 10-class
CIFAR-100 subset is 0.504, signiﬁcantly larger than a DIME of 0.340 for CIFAR-10.
Effect of number of samples: With increasing number of samples, the empirical distribution ofboth training and testing tend towards the population distribution, making the distance between themsmaller.
It should also make the dataset easier, which is indeed reﬂected in a decreasing value of
DIME with increasing number of samples (Figure 2-(right)).
But we also observe that our measurebecomes stable with increasing number of samples.
This suggests that our method can be used toestimate the relative difﬁculty of new datasets in domains where data is hard to collect (e.g., medicaldatasets, ﬁne-grained image classiﬁcation) and make determinations if additional data collection isrequired to make the problem easier to solve.
A.5 Model Complexity Experiment
Equation (5) suggests when the model class is larger the gap between DIME and the true lower boundon P e becomes smaller.
This is veriﬁed in our experiment shown in the middle of Figure 4, wherewe use neural networks with three hidden layers of varying number of neurons to calculate DIME.
In the experiment we try 32, 64, 128, 256, 512, 1024, and 2048 neurons for each hidden layer.
Asthe number of hidden neurons grows, DIME roughly decreases as expected.
Still on CIFAR-10 thedecrease is pretty monotonic.
For simple datasets such as MNIST, Fashion MNIST and EMNIST(byletters) the numbers got stabilized pretty quickly.
B Hybrid conditional entropy
Consider the usual classiﬁcation setting, X , representing the feature, is a random vector in R p ; Y ,representing the label, is a random variable(or vector) in { 1 , . .
. , |Y|} .
Assume X is a continuousvariable with density p X ( x ) for x in some compact domain Ω ⊂ X ⊂ R p .
Can we deﬁne a hybridconditional entropy H ( Y | X ) while X is continuous and Y is discrete?
We give a formal deﬁnitionfor this mixed-pair case and demonstrate that H ( Y | X ) deﬁned in our way can be a good indicator forthe hardness of classiﬁcation on the data ( X, Y ) , which is consistent with the classical deﬁnition inthe sense that both of them give the intuition that how much information or uncertainty is left for Ygiven X .
Our deﬁnition is
[Equation]
[Figure]
[Figure]
[Figure]
where P ( Y = y | X ) :
=
E [ 1 ( Y = y ) | X ] .
By the deﬁnition of conditional expectation (or probability),it is easy to check that
[Equation]
where p ( x | Y
=
y ) , x ∈ R p is the conditional density function of X given Y =
y .
Denotep y :
= P ( Y = y ) , we have
[Equation]
This proves (4) .
We summerize some basic properties of H ( Y | X ) under our deﬁnition (7) :
Lemma 1.
[Equation]
(ii) H ( Y | X ) =
H ( Y ) if and only if X and Y are independent.
(iii) H ( Y | X )
=
if and only if Y is a function of X .
That is, there exist |Y| subsets ofΩ : Ω 1 , . .
, Ω |Y| such that P ( X ∈ Ω y ∩
Ω y (cid:48) )
=
for any y (cid:54) =
y (cid:48) ∈ { 1 , . .
. , |Y|} ,
P ( X ∈ ∪ |Y| y =
1 Ω y )
= 1 and P ( Y = y | X ∈ Ω y )
= 1 for any y ∈ { 1 , . .
. , |Y|} .
Proof.
(i)
This is trivial by (7) , (9) and the fact that KL divergence is always nonnegative.(ii) By (9) , it is easy to get
H ( Y | [Equation]
, ∀ y
[Equation]
, |Y
[Equation]
⇐⇒ P ( X
∈
A, Y = y )
=
P ( X ∈ A ) P ( Y = y ) , for any measurable set A in X , any y ∈ { 1 , . .
. , |Y|} .
⇐⇒ X , Y are independent.
iii) Sufﬁciency:
If Y is a function of X , then by assumptions, for any y 1 (cid:54) =
y 2 , we have
P ( X ∈ Ω y 1 , Y = y 2 ) ≤ P ( X ∈ Ω y 1 ) − P ( X ∈
Ω
y 1 , Y = y 1 ) = 0 .
Therefore, p ( x | Y = y 2 ) = 0 for x ∈ Ω y 1 (a.e.).
Then for any y ∈ { 1 , . . . , |Y|} , we have
KL ( X | Y = y ) =
(cid:90) Ω y p
( x | Y = y ) log p ( x | Y =
y )
p X ( x ) dx (10)
Notice that for x ∈ Ω y (a.e.), p X ( x )
=
(cid:80) |Y| y (cid:48)
=1 p y (cid:48) p ( x | Y
=
y (cid:48) )
= p y p ( x | Y = y ) , and thus
KL ( X | Y =
y )
= − log p y .
Then by (9) , H ( Y | X ) = 0 .
Necessity: Assume H ( Y | X )
= 0 .
Notice that p X ( x ) = (cid:80) |Y| y =1 p y p
( x | Y = y ) , ∀ x ∈ Ω .
Therefore, for any y ∈ { 1 , . . . , |Y|} ,
[Equation]
Combing (9) and (12) , we know if H ( Y | X ) = 0 , then
[Equation]
Deﬁne Ω
y := { x ∈ Ω : p ( x | Y = y ) > 0 } , for every y ∈ { 1 , . .
. , |Y|} .
Without lossof generality, we assume p X ( x ) > 0 for every x ∈ Ω .
(Otherwise, we can replace Ω byΩ ∩ { p X ( x ) > 0 } and the following argument is still true.)
Then we have
[Equation]
By (13) , we have
[Equation]
Therefore, P ( Y = y
| X ∈ Ω y )
= 1 .
Lastly, we show P ( X ∈ Ω y ∩ Ω y (cid:48) )
=
for any y (cid:54) =
y (cid:48) .
This is immediate if we notice that by replacing Ω y in (14) with Ω y ∩ Ω y (cid:48) , we can have
[Equation]
Changing the postion of y and y (cid:48) , we have
[Equation]
Thus, P ( X ∈ Ω y ∩ Ω y (cid:48) )
= 0 .
Before the end of the proof, we want to add a remark that the condition P ( X ∈ Ω y ∩ Ω y (cid:48) )
=0 is actually just a consequence of P ( Y = y | X ∈ Ω y )
= 1 , ∀ y .
We add it explicitly in theconditions to make it clear that Ω 1 , . . .
, Ω |Y| is a partition of Ω according to the value of Y .
C Proof of Fano’s Inequality for Continuous Features
We see the above essential properties for H ( Y | X ) have been preserved for the mixture case.
Furthermore, we show Fano’s inequality still holds for our case.
Our proof here is based on the proof of
Lemma 2.10, Tsybakov (2004).
Proof.
(Proof of Theorem 1)
Assume P e ( f ) :
= P ( f ( X (cid:54) = Y )) .
Then
[Equation]
where p ( x | Y
=
y ) is the density function of X | Y
=
y and p ( x ) is the density of X .
Denote p
[Equation]
| x [Equation]
:= P
[Equation]
Y
[Equation]
y [Equation] p
[Equation]
x | Y = y ) p ( x ) , we have
[Equation]
Deﬁne g ( x ) :
= H ( x ) + x log( |Y| − 1) , for x ∈ [0 , 1] .
Here, H ( x ) = − x log x − (1 − x ) log(1 − x ) .
As g is a concave function, by Jensen’s Inequality, we have
[Equation]
We claim:
[Equation]
If the claim ( 17 ) is true, then
[Equation]
By the deﬁnition of P e , there exists a sequence { f k } ∞ k =1 such that P e ( f k ) → P e .
Therefore,
[Equation]
This is exactly (3) .
So it sufﬁces to show the claim ( 17 ) is true.
We may assume that
[Equation]
x ) p
[Equation]
y | x ) (cid:54) =
(otherwise, it is trivial).
Let y
= f
( x ) , then
[Equation]
= y
[Equation]
[Equation]
[Equation]
y (cid:54) = y
[Equation]
( y | x )
[Equation]
q ( y | x ) , where
[Equation]
(cid:88)
[Equation]
q ( y | x [Equation] q ( y | x )
[Equation]
Then by Jensen’s Inequality, we have(cid:88)
[Equation]
(cid:54) [Equation] q ( y | x
[Equation]
( y | x
[Equation]
log( |Y| − 1) .
Therefore,
[Equation] [Equation]