DIME: An Information-Theoretic Difﬁculty Measure for AI Datasets
Abstract
Evaluating the relative difﬁculty of widely-used benchmark datasets across time and across data modalities is important for accurately measuring progress in machine learning.
To help tackle this problem, we propose DIME , an information-theoretic
DI fﬁculty ME asure for datasets, based on Fano’s inequality and a neural network estimation of the conditional entropy of the sample-label distribution.
DIME can be decomposed into components attributable to the data distribution and the number of samples.
DIME can also compute per-class difﬁculty scores.
Through extensive experiments on both vision and language datasets, we show that DIME is wellaligned with empirically observed performance of state-of-the-art machine learning models.
We hope that DIME can aid future dataset design and model-training strategies.
Introduction
Empirical machine learning research relies heavily on comparing performance of algorithms on a few standard benchmark datasets.
Moreover, researchers frequently introduce new datasets that they believe to be more challenging than existing benchmarks.
However, we lack objective measures of dataset difﬁculty that are independent of the choices made about algorithm- and model-design.
Moreover it is also hard to compare algorithmic progress across data modalities, such as language and vision.
So, for instance, it is difﬁcult to compare the relative progress made on a sentiment analysis benchmark such as the Stanford Sentiment Treebank (SST)
(Socher et al., 2013) and an image classiﬁcation benchmark, like CIFAR-10 (Krizhevsky, 2009).
With these challenges in mind, we propose a model-agnostic and modality-agnostic measure for comparing how difﬁcult it is to perform supervised learning on a given dataset.
For i.i.d.
dataset examples, we argue that the difﬁculty of a dataset can be decoupled into two relatively independent sources:(a) approximation complexity, the number of samples required to approximate the true distribution up to certain accuracy, and (b) distributional complexity, the intrinsic difﬁculty involved in modeling the statistical relationship between the labels and features.
We focus our analysis on the second source of the intrinsic difﬁculty in supervised learning, and consider the most common setting where features X are continuous and the labels Y are discrete.
To provide a model-agnostic measure, we turn to the information-theoretic approach.
Indeed, Fano’s inequality suggests the lowest possible probability of the 0 - 1 error P e is lower bounded by terms D I M E related to the conditional entropy H ( Y | X ) .
Naturally we consider designing an estimator for this lower bound of P e as a difﬁculty measure for learning Y given X .
To estimate H ( Y | X ) , notice that H ( Y | X )
=
H ( Y ) − I ( X, Y ) , where I ( X, Y ) is the mutual information.
Direct estimation of I ( X, Y ) is hard for the hybrid case where X is continuous and Y is discrete.
Our key observation is that we can express I ( X, Y ) as a weighted average of the KL divergence between X | Y =
y and
X for each label y (see equation (4)).
Then we may use neural networks to approximate the KL divergence based on Donsker-Varadhan representation.
Related Work: Although conditional entropy and mutual information estimation have been extensively studied, research has focused on purely discrete or continuous data.
Nair et al.
(2006) were among the ﬁrst to study the theoretical properties for the case of mixture of discrete and continuous variables.
Ross (2014), Gao et al.
(2017) and Beknazaryan et al.
(2019) proposed approaches for estimating mutual information for the mixture case based on density ratio estimation (e.g.,binning, kNN or kernel methods), which is unsatisfactory for high dimensional data such as image and text.
We use neural network estimation, inspired by Belghazi et al.
(2018), to avoid these issues.
However, we emphasize that the technique for estimating the mutual information in Belghazi et al.
(2018) cannot be applied directly to the hybrid case, as argued before.
More importantly, we are the ﬁrst to connect the hybrid conditional entropy with the difﬁculty measure for datasets.
Designing a Dataset Difﬁculty Measure 2.1 Main Idea
For the continuous feature and discrete label setting, we use ( X, Y ) to represent feature-label pairs and assume X ∈ X ⊂ R d x , and Y ∈ Y ⊂
Z + .
We denote the joint distribution of ( X, Y ) as P XY and the marginal distributions as P X and P Y respectively.
Throughout this paper, we always assume that the feature-label pairs in the datasets, both training and testing, are sampled i.i.d.
from a static distribution P XY .
As Y is discrete, a natural measure that characterizes the difﬁculty of the data distribution is the best probability of the 0-1 error that can be achieved by any estimator.
Deﬁnition 1.
Model-Agnostic Error
[Equation]
While the computation of general P e is almost impossible, Fano’s inequality establishes a lower bound of P e by terms related to the conditional entropy.
Deﬁnition 2.
Hybrid conditional entropy
[Equation]
where P ( Y = y | X ) :
=
E [ 1 ( Y = y )
| X ] .
Theorem 1.
Fano’s inequality for continuous feature-discrete label case : Let P e be the minimum error probability as deﬁned in (1) .
Then we have
[Equation]
where H ( P e ) :=
−P e log P e − (1 − P e ) log(1 − P e ) .
A detailed proof is given in Appendix C.
Our proof is based on the proof of Lemma 2.10, Tsybakov (2004).
Note that most of the proofs of Fano’s inequality are designed for discrete random variables (e.g. Cover & Thomas (2012)).
2.2 Estimating the Lower Bound of P e
Motivated by Fano’s inequality, we just need to estimate H ( Y | X ) .
We rewrite it as (see Appendix B for a proof)
[Equation]
with
[Equation]
Remark 1.
In some benchmark datasets with balanced classes (e.g., CIFAR-10 and MNIST),
H ( Y | X ) = log |Y| − 1 |Y| (cid:80) |Y| y =1 KL
( X | Y =
y || X ) .
This indicates that if a dataset has more classes and the distributions of the features for different classes are closer to each other on average, then H ( Y | X ) would be larger and thus pushes P e larger, implying that the classiﬁcation task on it would be harder due to the intrinsic distributional complexity.
We now discuss the practical design of an estimator for KL ( X | Y =
y || X ) .
(Notice that both of the random varibles are continuous.)
Many non-parametric KL divergence estimators such as k NN based techniques (Noshad et al., 2017; Perez-Cruz, 2008; Wang et al., 2009) are not satisfying as they often rely on some knowledge of the internal dimension of the data manifold.
As an alternative, we design a neural network-based estimator for KL divergence estimation, inspired by Belghazi et al.
The estimator is based on the Donsker-Varadhan representation:
[Equation]
[Equation]
for two probability measures P , Q .
If we parameterize the function T using a neural network T θ , we can obtain an estimator of (cid:100) KL ( X | y || X ) for each y ∈ Y .
Then by equation (4), we get (cid:98) H ( Y | X ) , which further leads to (cid:99) P e by (1), an estimator of the lower bound of P e .
We report (cid:99) P e as our datatset difﬁculty measure DIME.
In our empirical evaluation, we optimize the empirical average instead of the expectation.
However this may cause issues such as overﬁtting.
To mitigate this issue, we split the data into separate training and validation sets.
The neural network models are trained on the training set and the estimation is made using the validation set.
See Algorithm 1 in Appendix A.1 for details.
Remark 2.
Given that we are using a neural network model to estimate the KL divergence between
P ( X | y ) and P ( X ) , according to Equation (5), (cid:100) KL ( X | y || X ) ≤ KL ( X | y || X ) .
As a consequence, the estimated mutual information (cid:98) I ( X ; Y ) ≤ I ( X ; Y ) , which leads to a larger (cid:98) H ( Y | X ) .
In the end,
DIME could be larger as compared to true lower bound on P e
.The hope is, if the function class of the neural network is large enough, the gap between (cid:100) KL ( X | y || X ) and KL ( X
| y || X ) is small enough so that DIME won’t deviate too much from the true lower bound of P e .
Experiments and Results 3.1 Sanity-check Experiments
Experiments with toy datasets: We generate three toy datasets and report their DIME socres, as shown in Figure 1.
The details are given in Appendix A.2, which contains a label corruption test as well.
Evaluating benchmark datasets in vision and language
We evaluate DIME using popular image and language domain datasets for various classiﬁcation tasks.
We use a simple multi-layer perceptron (MLP) with residual layers with ReLU activation functions, containing three hidden layers of 4096 neurons.
(The details about the datasets and neural network
we
Corpus classes (cid:98) H ( Y ) (cid:98) I ( X, Y ) (cid:98) H ( Y | X )
DIME
SOTA Error
[Table]
Language Modeling
[Table]
choose are given in Appendix A.3.)
In addition to dataset statistics, we report the estimated entropy (cid:98) H ( Y ) , (cid:98) I ( X, Y ) , (cid:98) H ( Y | X ) , and DIME.
We also report the state-of-the-art error on the dataset from recent literature (Wan et al., 2013; Huang et al., 2018; Cubuk et al., 2019;
Yang et al., 2019; Patro et al., 2018; Cer et al., 2018; Melis et al., 2019).
From the results summarized in Table 1, we make the following empirical observations:
First, DIME aligns well with the state-of-the-art results on the benchmark datasets, indicating that the relative difﬁculty of datasets is reﬂected in the performance of latest machine learning algorithms.
However, notably, DIME is lower than the reported state-of-the-art error for all the variants of
EMNIST.
This could indicate that since EMNIST is a relatively new, less-investigated dataset, we might see model performance improving over time.
Second, a larger number of classes lead to a higher value of DIME in general, which is expected given its dependence on (cid:98) H ( Y ) .
But (cid:98) H ( Y ) does not always dominate the value of the measure.
For example, MNIST has 10 classes, but its DIME is much smaller compared to IMDb which has only two classes.
This relative difference in difﬁculty of MNIST and IMDb is also reﬂected in their SOTA errors, validating our measure.
Finally, for datasets such as MNIST, Fashion MNIST, EMNIST, FakeData, and DBPedia, DIME provides a tight error bound.
But for certain datasets such as CIFAR-10, CIFAR-100, and Tiny
ImageNet, DIME seem slightly pessimistic.
As we discussed in Remark 2, DIME could be larger than the true lower bound of P e since we constrain the function space to be a neural network model.
While the relative order of difﬁculty suggested by DIME is reasonable, its large value also suggests that an MLP may not have enough capacity for accurate P e estimation.
While it is an open question how to choose a model that is large enough to approximate P e yet easy to optimize, Equation (5) suggests that as the model size grows, the gap between DIME and true lower bound of P e becomes smaller.
We verify this by calculating DIME with increasing number of neurons for a three-layer
MLP.
See Figure 4 in Appendix A.5 for details.
As we mentioned in the abstract, we can also use DIME to compute per-class difﬁculty score and reﬂect the effect of number of samples on data distribution, etc.
See Appendix A.4 for the additional use cases for DIME.
Discussion
We would like to use this section to discuss some common questions proposed by the reviewers and readers.
Why don’t we use a cross entropy estimation to approximate the conditional entropy H ( Y | X ) (which is also an upper bound of H ( Y | X ) )?
We want to emphasize that our motivation here is to design a metric for datasets to capture the intrinsic complexity of the sample-label distribution in supervised learning.
Our intuition is that if a dataset has a large number of classes and the distributions of the features for each class are close to each other, then it would be difﬁcult to accurately predict the labels given the features.
Therefore, Our starting point is to deﬁne a metric of the following form:
[Equation]
for some increasing function f and some distance function D over distributions.
We ﬁnd a good choice of ( f, D ) is (log , KL ) and this exactly leads to the deﬁnition of H ( Y | X ) .
Then we start to consider how to estimate the KL divergence for some high-dimensional data.
Therefore, even if we can use a neural network via a cross entropy approach to approximate H ( Y | X ) —-and we expect the approximation would be more accurate than our approach as the distribution of X | Y is generally much more complicated than that of Y | X —-we still avoid this approach since it is just meaningless here, given that the spirit of this paper is to show that the distance over the distributions of features for each class is an essential indicator of the complexity of the label-feature distributions.
Since our focus and interest are the distributions X | Y
=
y , an approach to approximate Y | X is just another trial to do the prediction and thus, is not appropriate here to provide a model-free measure.
On the other hand, we have to point out that one main limitation of our approach is that it is pretty challenging to approximate the distribution of X | Y (i.e. achieving the supremum in (5)).
A simple MLP neural network, as we used here, seems far from satisfying given that the fact that DIME is almost always larger than the SOTA error in Table 1.
Therefore, it seems worthwhile to try some neural networks with more capacity or more complicated architectures like convnets, LSTMs, transformers, etc.
(Of course, then it would be more controversial to claim our metric "model-free".) 2.
Why do you use MINE (Belghazi et al.
(2018)) to estimate the mutual information (which itself is not a valid lower bound, as shown by Poole et al.
We actually use a MINE style estimator to estimate the KL divergence instead of the mutual information.
So some other methods to estimate the mutual information (see e.g.
Alemi et al.
for an upper bound; Oord et al.
(2018) for a lower bound) are not exactly relevant here.
That being said, it is deﬁnitely worthwhile to try these methods to directly estimate the mutual information (epsecially those for upper bounds such that we end up with a valid lower bound of H ( Y | X ) ) and compare the results with the current ones.
We leave this for future explorations.
A Supplementary Results for Experiments A.1 Algorithm for DIME Algorithm
[Table]
Initialize |Y| neural networks: T θ 1 , T θ 2 , . .
, T θ |Y| , one per class.
Initialize class counters k 1 = 0 , . . .
, k |Y| = 0 , one per class.
Initialize the sample counter k = 0 .
for t in 0 , . . .
, T do
Draw a batch of b training examples { ( x i , y i ) } , and b evaluation examples { ( x ei , y ei ) } .
for each class c ∈ { 1 , . . .
, |Y|} do
Find samples of the c -th class:
S c = { i | y i =
c } , S ec =
{ i | y e i =
c }
Train θ c = arg max θ c 1 | S c | (cid:80) S c T θ c ( x i ) − log[ 1 b (cid:80) j exp T θ c ( x j )]
Evaluate (cid:100) KL θ c = 1 | S ec | (cid:80) S ec T
θ c ( x ei ) − log[ 1 b (cid:80) j exp T θ c ( x ej )]
If (cid:100) KL θ c stops increasing, pause the training for T θ c .
Update counters: k c + = | S ec | .
end for
Update the sample counter: k + = b
If all neural networks stop updating, break.
end for
Estimate the class probability (cid:98) p c = k c k , c
= { 1 , . .
. , |Y|} .
Calculate (cid:98) H ( Y | X ) using (4), (cid:98) p c , and (cid:100) KL
θ c , c
= { 1 , . .
. , |Y|} .
Solve (3) and output the solution.
A.2 Supplement Results for Section [-3.
-]3.
Experiments with toy datasets: We generate three two-dimensional 2-class toy datasets with 20,000 samples each with increasing level of difﬁculty (Figure 1).
Samples in Toy Data 1 and Toy
Data 2 are generated using random Gaussian variables by scaling differently in different dimensions and then by rotating, while the two classes in Toy Data 3 are simply uniform and Gaussian random variables.
Toy Data 1, which is relatively easy to be separated, gets a DIME of 0.001.
Toy Data 2, which is a bit harder due to a small overlapping region around the axis origin, gets a DIME of 0.013.
And Toy Data 3, which is the hardest due to overlapping supports of the two class, gets a DIME of 0.263.
In sum, DIME accurately reﬂects the relative difﬁculty of the toy examples.
Label corruption test: Next, we estimate DIME for the case of label corruption.
If we assign random labels to a fraction of the training samples, intuitively the dataset should become more difﬁcult.
We experiment with ﬁve datasets with ten classes each—MNIST, Fashion MNIST, CIFAR10, EMNIST (digits), and SVHN—and perform label corruption on their training sets in steps of 0.1 from 0 to 1.
We observe a consistent increase in DIME across all the datasets with increasing label corruption (Figure 2-(left)).
For some datasets such as MNIST, DIME is almost tight given the fraction of randomly corrupted labels.
Label Corruption Rate
D
I M
E
CIFAR-10 MNIST
FASHION SVHN
EMNIST-LETTERS 0.
Fraction of Samples
[Figure] [Figure]
Results for
A.3.1 Choice of Benchmark Datasets
We evaluate the following image classiﬁcation datasets: MNIST (LeCun et al., 1998), Extended
MNIST (EMNIST)
(Cohen et al., 2017), Fashion MNIST (Xiao et al., 2017), CIFAR-10 (Krizhevsky, 2009), CIFAR-100 (Krizhevsky, 2009), Tiny ImageNet 3 , and SVHN (Netzer et al., 2011).
The
EMNIST dataset has several different splits, which include splits by digits, letters, merge, class, and balanced.
For a controlled evaluation study, we also add the PyTorch Fakedata dataset 4 which contains 10 -classes of randomly-generated 32 × 32 grayscale images.
Among natural language processing datasets, we investigate three supervised scenarios: sentiment analysis, text classiﬁcation, and language modeling.
For sentiment analysis, we analyze IMDb (Maas et al., 2011), and Stanford Sentiment Treebank (SST) (Socher et al., 2013).
In the case of SST, we utilize both ﬁne-grained labels (SST-5) and binary labels (SST-2).
For the text classiﬁcation task, we use the TREC dataset (Li & Roth, 2002), AG News, DBPedia, as well as Yelp Review dataset (Zhang et al., 2015).
For language modeling task, we analyze the character level modeling for Penn Tree
Bank (Marcus et al., 1994).
For DIME estimation, we do not perform any preprocessing on the data except for simple rescaling.
For image data we rescale all the pixel values to ﬂoats in [0 , 1] .
By data processing inequality (Cover & Thomas, 2012), the rescaling will not cause any change on H ( Y | X ) and P e since the rescaling function is invertible.
A.3.2 Choice of Neural Network Estimator
To estimate DIME for image datasets, we use a simple multi-layer perceptron (MLP) with residual layers with Rectiﬁed Linear Unit (ReLU) activation functions.
The network has three hidden layers of 4096 neurons.
For language datasets, we use an embedding layer of dimension 1500 , followed by three layers of 256 hidden neurons with residual connections and ReLU activations.
We found that a higher dimension of the embedding helps achieve a tighter bound.
The embedding layer is initialized with a concatenation of ﬁve pretrained embedding vectors: GloVe-840B-300d, GloVe-42B-300d,
GloVe-twitter-27B-200d, GloVe-6B-300d, FastText-en-300d, and CharNGram-100d.
The input sequences are truncated or padded to the length of 128.
The MLP is operating on the token dimension instead of the embedding dimension to take the order of the sequence into consideration.
For example, the ﬁrst layer of MLP is 128-by-256 instead of 1500-by-256.
The last fully-connected layer ﬂattens everything into a vector and projects it to a scalar.
For optimizing the MLP, we use SGD with initial learning rate of 0.1 and anneal it to 0.01 and 0.001 if the objective stops updating.
We use the test set as evaluation set, as described in Algorithm 1.
A.4 Additional Use Cases for DIME
Dataset difﬁculty per class: Algorithm 1 can also estimate the relative difﬁculty of classes within a dataset.
For this purpose, we simply rank the classes by their (cid:100) KL ( X | y || X ) .
A smaller value of (cid:100) KL ( X | y || X ) indicates the class is harder to classify.
As examples in Figure 3 show, ‘cat’ and ‘bird’ are the most difﬁcult classes in CIFAR-10, while ‘automobile’ and ‘ship’ are the easiest.
In addition, ‘shirt’ and ‘pullover’ are the most difﬁcult classes in Fashion MNIST, while ‘sandal’ and ‘trouser’ are the easiest.
Controlled Study with CIFAR Subsets:
We obtain a 10-class subset from CIFAR-100 by choosing subclasses from the ‘large carnivores’ and ‘large omnivores and herbivores’ superclasses.
These classes—bear, leopard, lion, tiger, wolf, camel, cattle, chimpanzee, elephant, and kangaroo—should be intuitively harder to classify as compared to CIFAR-10, which includes disparate classes such as bird, truck, and ship.
Note that the image data for both of these datasets comes from the same, larger ‘80 million tiny images’ dataset (Torralba et al., 2008).
We ﬁnd that the DIME for this 10-class
CIFAR-100 subset is 0.504, signiﬁcantly larger than a DIME of 0.340 for CIFAR-10.
Effect of number of samples: With increasing number of samples, the empirical distribution of both training and testing tend towards the population distribution, making the distance between them smaller.
It should also make the dataset easier, which is indeed reﬂected in a decreasing value of
DIME with increasing number of samples (Figure 2-(right)).
But we also observe that our measure becomes stable with increasing number of samples.
This suggests that our method can be used to estimate the relative difﬁculty of new datasets in domains where data is hard to collect (e.g., medical datasets, ﬁne-grained image classiﬁcation) and make determinations if additional data collection is required to make the problem easier to solve.
A.5 Model Complexity Experiment
Equation (5) suggests when the model class is larger the gap between DIME and the true lower bound on P e becomes smaller.
This is veriﬁed in our experiment shown in the middle of Figure 4, where we use neural networks with three hidden layers of varying number of neurons to calculate DIME.
In the experiment we try 32, 64, 128, 256, 512, 1024, and 2048 neurons for each hidden layer.
As the number of hidden neurons grows, DIME roughly decreases as expected.
Still on CIFAR-10 the decrease is pretty monotonic.
For simple datasets such as MNIST, Fashion MNIST and EMNIST (byletters) the numbers got stabilized pretty quickly.
B Hybrid conditional entropy
Consider the usual classiﬁcation setting, X , representing the feature, is a random vector in R p ; Y , representing the label, is a random variable(or vector) in { 1 , . .
. , |Y|} .
Assume X is a continuous variable with density p X ( x ) for x in some compact domain Ω ⊂ X ⊂ R p .
Can we deﬁne a hybrid conditional entropy H ( Y | X ) while X is continuous and Y is discrete?
We give a formal deﬁnition for this mixed-pair case and demonstrate that H ( Y | X ) deﬁned in our way can be a good indicator for the hardness of classiﬁcation on the data ( X, Y ) , which is consistent with the classical deﬁnition in the sense that both of them give the intuition that how much information or uncertainty is left for Y given X .
Our deﬁnition is
[Equation]
[Figure] [Figure]
[Figure]
where P ( Y = y | X ) :
=
E [ 1 ( Y = y ) | X ] .
By the deﬁnition of conditional expectation (or probability), it is easy to check that
[Equation]
where p ( x | Y
=
y ) , x ∈ R p is the conditional density function of X given Y =
y .
Denote p y :
= P ( Y = y ) , we have
[Equation]
This proves (4) .
We summerize some basic properties of H ( Y | X ) under our deﬁnition (7) :
Lemma 1.
[Equation]
(ii) H ( Y | X ) =
H ( Y ) if and only if X and Y are independent.
(iii) H ( Y | X )
=
if and only if Y is a function of X
That is, there exist |Y| subsets of Ω : Ω 1 , . . .
, Ω |Y| such that P ( X ∈ Ω y ∩
Ω y (cid:48) )
=
for any y (cid:54) =
y (cid:48) ∈ { 1 , . .
. , |Y|} , P ( X ∈ ∪ |Y| y =1 Ω y )
= 1 and P ( Y = y | X ∈ Ω y )
= 1 for any y ∈ { 1 , . .
. , |Y|} .
Proof.
(i)
This is trivial by (7) , (9) and the fact that KL divergence is always nonnegative. (ii) By (9) , it is easy to get
[Equation]
X
[Equation]
A
[Equation]
P
[Equation]
Y = y ) , for any measurable set A in X , any y ∈
, | Y|
[Equation] X , Y are independent.
(iii) Sufﬁciency:
If Y is a function of X , then by assumptions, for any y 1 (cid:54) =
y 2 , we have
P
[Equation] Therefore, p ( x | Y = y 2 ) = 0 for x ∈ Ω y 1 (a.e.).
Then for any y ∈ { 1 , . . . , |Y|} , we have
KL ( X | Y = [Equation]
) = (cid:90)
[Equation]
Notice that for x ∈ Ω y (a.e.), p X ( x )
=
(cid:80) |Y| y (cid:48)
=1 p y (cid:48) p ( x | Y
=
y (cid:48) )
= p y p ( x | Y = y ) , and thus
[Equation]
Then by (9) , H ( Y | X )
= 0 .
Necessity: Assume H ( Y | X )
= 0 .
Notice that p X ( x ) = (cid:80) |Y| y =1 p y p
( x | Y = y ) , ∀ x ∈ Ω .
Therefore, for any y ∈ { 1 , . . .
, |Y|} ,
KL ( X
[Equation]
Combing (9) and (12) , we know if H ( Y | X ) = 0 , then
[Equation]
Deﬁne Ω
y := { x ∈ Ω : p ( x | Y = y ) > 0 } , for every y ∈ { 1 , . .
. , |Y|} .
Without loss of generality, we assume p X ( x ) > 0 for every x ∈ Ω .
(Otherwise, we can replace Ω by Ω ∩ { p X ( x ) > 0 } and the following argument is still true.)
Then we have
[Equation]
By (13) , we have
[Equation]
Therefore, P ( Y = y
| X ∈ Ω y )
= 1 .
Lastly, we show P ( X ∈ Ω y ∩ Ω y (cid:48) )
=
for any y (cid:54) =
y (cid:48) .
This is immediate if we notice that by replacing Ω y in (14) with Ω y ∩ Ω y (cid:48) , we can have P ( X ∈ Ω y ∩ Ω y
[Equation]
) =
P ( X ∈ Ω y ∩ Ω y (cid:48) , Y = y ) .
Changing the postion of y and y (cid:48) , we have
[Equation]
Thus, P ( X ∈ Ω y ∩ Ω y (cid:48) )
= 0 .
Before the end of the proof, we want to add a remark that the condition P ( X ∈ Ω y ∩ Ω y (cid:48) )
= 0 is actually just a consequence of P ( Y = y | X ∈ Ω y )
= 1 , ∀ y .
We add it explicitly in the conditions to make it clear that Ω 1 , . . .
, Ω |Y| is a partition of Ω according to the value of Y .
C Proof of Fano’s Inequality for Continuous Features
We see the above essential properties for H ( Y | X ) have been preserved for the mixture case.
Furthermore, we show Fano’s inequality still holds for our case.
Our proof here is based on the proof of
Lemma 2.10, Tsybakov (2004).
Proof.
(Proof of Theorem 1)
Assume P e ( f ) :
= P ( f ( X (cid:54) = Y )) .
Then
P
[Equation]
where p ( x | Y
=
y ) is the density function of X | Y
=
y and p ( x ) is the density of X .
Denote p ( y | x ) := P
[Equation]
Y = y ) p
[Equation]
x
[Equation]
Y
[Equation]
y
[Equation]
p ( x ) , we have
[Equation]
Deﬁne g ( x ) :
= H ( x ) + x log( |Y| − 1) , for x ∈ [0 , 1] .
Here, H ( x ) = − x log x − (1 − x ) log(1 − x ) .
As g is a concave function, by Jensen’s Inequality, we have
[Equation]
We claim:
[Equation]
If the claim ( 17 ) is true, then
[Equation]
By the deﬁnition of P e , there exists a sequence { f k } ∞ k =1 such that P e ( f k ) → P e .
Therefore,
[Equation]
P
[Equation]
This is exactly (3) .
So it sufﬁces to show the claim ( 17 ) is true.
We may assume that
[Equation]
p
[Equation]
y | x [Equation] =
(otherwise, it is trivial).
Let y
= f
( x ) , then
[Equation]
[Equation]
y
[Equation]
x
[Equation]
, where
[Equation]
Then by Jensen’s Inequality, we have (cid:88)
[Equation]
[Equation]
Therefore, |Y| (cid:88) y =
[Equation]
[Equation] [Equation]