Original: L EARNABILITY FOR THE I NFORMATION B OTTLENECK
Revised: L EARNABILITY FOR THE I NFORMATION B OTTLENECK

Revised: Tailin Wu

Original: Anonymous authors
Revised: MIT tailin@mit edu Ian Fischer Google AI iansf@google.com

Original: Paper under double-blind review
Revised: Isaac L. Chuang MIT ichuang@mit.edu Max Tegmark MIT tegmark@mit.edu

Original: A BSTRACT
Revised: A BSTRACT

Original: Compressed representations generalize better (Shamir et al., 2010), which may be crucial when learning from limited or noisy labeled data.
Revised: Compressed representations generalize better (Shamir et al., 2010), which may be crucial when learning from limited or noisy labeled data.

Original: The Information Bottleneck (IB) method (Tishby et al.
Revised: The Information Bottleneck (IB) method (Tishby et al.

Original: (2000)) provides an insightful and principled approach for balancing compression and prediction in representation learning.
Revised: (2000)) provides an insightful and principled approach for balancing compression and prediction in representation learning.

Original: The IB objective I ( X ; Z ) − βI ( Y ; Z ) employs a Lagrange multiplier β to tune this trade-off.
Revised: The IB objective I ( X ; Z ) − βI ( Y ; Z ) employs a Lagrange multiplier β to tune this trade-off.

Original: However, there is little theoretical guidance for how to select β .
Revised: However, there is little theoretical guidance for how to select β .

Original: There is also a lack of theoretical understanding about the relationship between β , the dataset, model capacity, and learnability .
Revised: There is also a lack of theoretical understanding about the relationship between β , the dataset, model capacity, and learnability .

Original: In this work, we show that if β is improperly chosen, learning cannot happen: the trivial representation P ( Z | X ) =
Revised: In this work, we show that if β is improperly chosen, learning cannot happen: the trivial representation P ( Z | X ) =

Original: P ( Z ) becomes the global minimum of the IB objective.
Revised: P ( Z ) becomes the global minimum of the IB objective.

Original: We show how this can be avoided, by identifying a sharp phase transition between the unlearnable and the learnable which arises as β varies.
Revised: We show how this can be avoided, by identifying a sharp phase transition between the unlearnable and the learnable which arises as β varies.

Original: This phase transition deﬁnes the concept of IB-Learnability .
Revised: This phase transition deﬁnes the concept of IB-Learnability .

Original: We prove several sufﬁcient conditions for IB-Learnability, providing theoretical guidance for selecting β .
Revised: We prove several sufﬁcient conditions for IB-Learnability, providing theoretical guidance for selecting β .

Original: We further show that IB-learnability is determined by the largest conﬁdent , typical , and imbalanced subset of the training examples.
Revised: We further show that IB-learnability is determined by the largest conﬁdent , typical , and imbalanced subset of the training examples.

Original: We give a practical algorithm to estimate the minimum β for a given dataset.
Revised: We give a practical algorithm to estimate the minimum β for a given dataset.

Original: We test our theoretical results on synthetic datasets, MNIST, and CIFAR10 with noisy labels, and make the surprising observation that accuracy may be non-monotonic in β .
Revised: We test our theoretical results on synthetic datasets, MNIST, and CIFAR10 with noisy labels, and make the surprising observation that accuracy may be non-monotonic in β .

Original:  I NTRODUCTION AND R ELATED W ORK
Revised:  I NTRODUCTION AND R ELATED W ORK

Original: Compressed representations generalize better (Shamir et al., 2010), which is likely to be particularly important when learning from limited or noisy labels, as otherwise we should expect our models to overﬁt to the noise.
Revised: Compressed representations generalize better (Shamir et al., 2010), which is likely to be particularly important when learning from limited or noisy labels, as otherwise we should expect our models to overﬁt to the noise.

Original: Tishby et al.
Revised: Tishby et al.

Original: (2000) introduced the Information Bottleneck (IB) objective function which learns a representation Z of observed variables ( X, Y ) that retains as little information about X as possible, but simultaneously captures as much information about Y as possible: 
Revised: (2000) introduced the Information Bottleneck (IB) objective function which learns a representation Z of observed variables ( X, Y ) that retains as little information about X as possible, but simultaneously captures as much information about Y as possible: [Equation] I

Original: [Equation]
Revised: [Equation]

Original: I ( X ; Y ) =
Revised: X [Equation] Y ) =

Original: (cid:82) dx dy p ( x, y ) log p ( x,y ) p ( x ) p ( y ) is the mutual information.
Revised: (cid:82) dx dy p ( x, y ) log p ( x,y ) p ( x ) p ( y ) is the mutual information.

Original: The hyperparameter β controls the trade-off between compression and prediction, in the same spirit as Rate-Distortion Theory (Shannon, 1948), but with a learned representation function P ( Z | X ) that automatically captures some part of the “semantically meaningful” information, where the semantics are determined by the observed relationship between X and Y .
Revised: The hyperparameter β controls the trade-off between compression and prediction, in the same spirit as Rate-Distortion Theory (Shannon, 1948), but with a learned representation function P ( Z | X ) that automatically captures some part of the “semantically meaningful” information, where the semantics are determined by the observed relationship between X and Y .

Original: The IB framework has been studied extensively and applied to a variety of problems (Chechik et al.
Revised: The IB framework has been studied extensively and applied to a variety of problems (Chechik et al.

Original: ; Rey & Roth (2012); Alemi et al.
Revised: ; Rey & Roth (2012); Alemi et al.

Original: ; Chalk et al.
Revised: ; Chalk et al.

Original: ; Strouse & Schwab (2017); Achille & Soatto (2017); Achille & Soatto (2018); Fischer (2018); Achille et al.
Revised: ; Strouse & Schwab (2017); Achille & Soatto (2017); Achille & Soatto (2018); Fischer (2018); Achille et al.

Original: ; Kolchinsky et al.
Revised: ; Kolchinsky et al.

Original: (2019), etc.)
Revised: (2019), etc.)

Original: , but a core issue remains: how should we select β ?
Revised: , but a core issue remains: how should we select β ?

Original: In the original work, the authors recommend sweeping β > 1 , which can be prohibitively expensive in practice, but also leaves open interesting theoretical questions around the relationship between β , P ( Z | X ) , and the observed data, P ( X, Y ) .
Revised: In the original work, the authors recommend sweeping β > 1 , which can be prohibitively expensive in practice, but also leaves open interesting theoretical questions around the relationship between β , P ( Z | X ) , and the observed data, P ( X, Y ) .

Original: For example, under how much label noise will IB at a given β still be able to learn a useful representation?
Revised: For example, under how much label noise will IB at a given β still be able to learn a useful representation?

Original: This work begins to answer some of those questions by characterizing the onset of learning.
Revised: This work begins to answer some of those questions by characterizing the onset of learning.

Original: Speciﬁcally:
Revised: Speciﬁcally:

Original: • We show that improperly chosen β may result in a failure to learn: the trivial solution P ( Z | X ) =
Revised: • We show that improperly chosen β may result in a failure to learn: the trivial solution P ( Z | X ) =

Original: P ( Z ) becomes the global minimum of the IB objective, even for β (cid:29) 1 .
Revised: P ( Z ) becomes the global minimum of the IB objective, even for β (cid:29) 1 .

Original: • We introduce the concept of IB-Learnability , and show that when we vary β , the IB objective will undergo a phase transition from the inability to learn to the ability to learn.
Revised: • We introduce the concept of IB-Learnability , and show that when we vary β , the IB objective will undergo a phase transition from the inability to learn to the ability to learn.

Original: • Using the second-order variation, we derive sufﬁcient conditions for IB-Learnability, which provide theoretical guidance for choosing a good β .
Revised: • Using the second-order variation, we derive sufﬁcient conditions for IB-Learnability, which provide theoretical guidance for choosing a good β .

Original:  We show that IB-learnability is determined by the largest conﬁdent , typical , and imbalanced subset of the training examples, reveal its relationship with the slope of the Pareto frontier at the origin on the information plane
Revised: • We show that IB-learnability is determined by the largest conﬁdent , typical , and imbalanced subset of the training examples, reveal its relationship with the slope of the Pareto frontier at the origin on the information plane

Original: I ( Y ; Z ) vs. I ( X ; Z ) , and discuss its relation with model capacity.
Revised: I ( Y ; Z ) vs. I ( X ; Z ) , and discuss its relation with model capacity.

Original: We use our main results to demonstrate on synthetic datasets, MNIST (LeCun et al., 1998), and CIFAR10 (Krizhevsky & Hinton, 2009) under noisy labels that the theoretical prediction for IB-Learnability closely matches experiment.
Revised: We use our main results to demonstrate on synthetic datasets, MNIST (LeCun et al., 1998), and CIFAR10 (Krizhevsky & Hinton, 2009) under noisy labels that the theoretical prediction for IB-Learnability closely matches experiment.

Original: We present an algorithm for estimating the onset of IB-Learnability, and demonstrate that it does a good job of estimating the theoretical predictions and the empirical results.
Revised: We present an algorithm for estimating the onset of IB-Learnability, and demonstrate that it does a good job of estimating the theoretical predictions and the empirical results.

Original: Finally, we observe discontinuities in the Pareto frontier of the information plane as β increases, and those dicontinuities correspond to accuracy decreasing as β increases.
Revised: Finally, we observe discontinuities in the Pareto frontier of the information plane as β increases, and those dicontinuities correspond to accuracy decreasing as β increases.

Original:  IB-L EARNABILITY AND ITS S UFFICIENT C ONDITIONS
Revised:  IB-L EARNABILITY AND ITS S UFFICIENT C ONDITIONS

Original: We are given instances of ( x, y ) ∈
Revised: We are given instances of ( x, y ) ∈

Original: X × Y drawn from a distribution with probability (density) P ( X, Y ) , where unless otherwise stated, both X and Y can be discrete or continuous variables.
Revised: X × Y drawn from a distribution with probability (density) P ( X, Y ) , where unless otherwise stated, both X and Y can be discrete or continuous variables.

Original: ( X, Y ) is our training data , and may be characterized by different types of noise.
Revised: ( X, Y ) is our training data , and may be characterized by different types of noise.

Original: We can learn a representation Z of X with conditional probability 1 p ( z | x ) , such that X, Y, Z obey the Markov chain Z ← X ↔ Y .
Revised: We can learn a representation Z of X with conditional probability 1 p ( z | x ) , such that X, Y, Z obey the Markov chain Z ← X ↔ Y .

Original: Eq.
Revised: Eq.

Original: (1) above gives the IB objective with Lagrange multiplier β , IB β
Revised: (1) above gives the IB objective with Lagrange multiplier β , IB β

Original: ( X, Y ; Z ) , which is a functional of p ( z | x ) :
Revised: ( X, Y ; Z ) , which is a functional of p ( z | x ) :

Original: IB β ( X, Y ; Z ) =
Revised: IB β ( X, Y ; Z ) =

Original: IB β [ p ( z | x )] .
Revised: IB β [ p ( z | x )] .

Original: The IB learning task is to ﬁnd a conditional probability p
Revised: The IB learning task is to ﬁnd a conditional probability p

Original: ( z | x ) that minimizes IB β
Revised: ( z | x ) that minimizes IB β

Original: ( X, Y ; Z ) .
Revised: ( X, Y ; Z ) .

Original: The larger β , the more the objective favors making a good prediction for Y .
Revised: The larger β , the more the objective favors making a good prediction for Y .

Original: Conversely, the smaller β , the more the objective favors learning a concise representation.
Revised: Conversely, the smaller β , the more the objective favors learning a concise representation.

Original: How can we select β such that the IB objective learns a useful representation?
Revised: How can we select β such that the IB objective learns a useful representation?

Original: In practice, the selection of β is done empirically.
Revised: In practice, the selection of β is done empirically.

Original: Indeed, Tishby et al.
Revised: Indeed, Tishby et al.

Original: (2000) recommends “sweeping β ”.
Revised: (2000) recommends “sweeping β ”.

Original: In this section, we provide theoretical guidance for choosing β by introducing the concept of IB-Learnability and providing a series of IB-learnable conditions.
Revised: In this section, we provide theoretical guidance for choosing β by introducing the concept of IB-Learnability and providing a series of IB-learnable conditions.

Original: Deﬁnition 1 (IB β -Learnability) .
Revised: Deﬁnition 1 (IB β -Learnability) .

Original: ( X, Y ) is IB β -learnable if there exists a Z given by some p 1 ( z | x ) , such that
Revised: ( X, Y ) is IB β -learnable if there exists a Z given by some p 1 ( z | x ) , such that

Original: IB β ( X, Y ; Z ) | p 1 ( z | x ) <
Revised: IB β ( X, Y ; Z ) | p 1 ( z | x ) <

Original: IB β ( X, Y ; Z ) | p ( z | x )=
Revised: IB β ( X, Y ; Z ) | p ( z | x )=

Original: p ( z ) , where p ( z | x ) = p ( z ) characterizes the trivial representation such that Z = Z trivial is independent of X .
Revised: p ( z ) , where p ( z | x ) = p ( z ) characterizes the trivial representation such that Z = Z trivial is independent of X .

Original: If ( X ; Y ) is IB β -learnable, then when IB β ( X, Y ; Z ) is globally minimized, it will not learn a trivial representation.
Revised: If ( X ; Y ) is IB β -learnable, then when IB β ( X, Y ; Z ) is globally minimized, it will not learn a trivial representation.

Original: If ( X ; Y ) is not IB β -learnable, then when IB β ( X, Y ; Z ) is globally minimized, it may learn a trivial representation.
Revised: If ( X ; Y ) is not IB β -learnable, then when IB β ( X, Y ; Z ) is globally minimized, it may learn a trivial representation.

Original: Necessary condition for IB-Learnability.
Revised: Necessary condition for IB-Learnability.

Original: From Deﬁnition 1, we can see that IB β -Learnability for any dataset ( X ; Y ) requires β > 1 .
Revised: From Deﬁnition 1, we can see that IB β -Learnability for any dataset ( X ; Y ) requires β > 1 .

Original: In fact, from the Markov chain Z ← X ↔ Y , we have I ( Y ; Z ) ≤
Revised: In fact, from the Markov chain Z ← X ↔ Y , we have I ( Y ; Z ) ≤

Original: I ( X ; Z ) via the dataprocessing inequality.
Revised: I ( X ; Z ) via the dataprocessing inequality.

Original: If β ≤ 1 , then since I ( X ; Z ) ≥ 

Revised: If β ≤ 1 , then since I ( X ; Z ) ≥

Original: and I ( Y ; Z ) ≥ 0 , we have that min( I ( X ; Z ) − βI ( Y ; Z ))
Revised: and I ( Y ; Z ) ≥ 0 , we have that min( I ( X ; Z ) − βI ( Y ; Z ))

Original: = 
 =
Revised: = 
 =

Original: IB β
Revised: IB β

Original: ( X, Y ; Z trivial ) .
Revised: ( X, Y ; Z trivial ) .

Original: Hence ( X, Y ) is not IB β -learnable for β ≤ 1 .
Revised: Hence ( X, Y ) is not IB β -learnable for β ≤ 1 .

Original: Theorem 1 characterizes the IB β -Learnability range for β (see Appendix B for the proof):
Revised: Theorem 1 characterizes the IB β -Learnability range for β (see Appendix B for the proof):

Original: Theorem 1.
Revised: Theorem 1.

Original: If ( X, Y ) is IB β 1 -learnable, then for any β 2 > β 1 , it is IB β 2 -learnable.
Revised: If ( X, Y ) is IB β 1 -learnable, then for any β 2 > β 1 , it is IB β 2 -learnable.

Original: Based on Theorem 1, the range of β such that ( X, Y ) is IB β -learnable has the form β ∈ ( β 0 , + ∞ ) .
Revised: Based on Theorem 1, the range of β such that ( X, Y ) is IB β -learnable has the form β ∈ ( β 0 , + ∞ ) .

Original: Thus, β 0 is the threshold of IB-Learnability.
Revised: Thus, β 0 is the threshold of IB-Learnability.

Original: Furthermore, the trivial representation is a stationary solution for the IB objective:
Revised: Furthermore, the trivial representation is a stationary solution for the IB objective:

Original: Lemma 1.1.
Revised: Lemma 1.1.

Original: p ( z | x ) = p ( z ) is a stationary solution for IB β ( X, Y ; Z ) .
Revised: p ( z | x ) = p ( z ) is a stationary solution for IB β ( X, Y ; Z ) .

Original: The proof in Appendix E shows that the ﬁrst-order variation δ IB β [ p ( z | x )] = 0 vanishes at the trivial representation.
Revised: The proof in Appendix E shows that the ﬁrst-order variation δ IB β [ p ( z | x )] = 0 vanishes at the trivial representation.

Original: Lemma 1.1 yields our strategy for ﬁnding sufﬁcient conditions for learnability: ﬁnd conditions such that p ( z | x )
Revised: Lemma 1.1 yields our strategy for ﬁnding sufﬁcient conditions for learnability: ﬁnd conditions such that p ( z | x )

Original: =
Revised: =

Original: p ( z ) is not a local minimum for the functional IB β [ p ( z | x )] .
Revised: p ( z ) is not a local minimum for the functional IB β [ p ( z | x )] .

Original: By requiring that the second order variation δ 2 IB β [ p ( z | x )] <at the trivial representation (Suff.
Revised: By requiring that the second order variation δ 2 IB β [ p ( z | x )] <at the trivial representation (Suff.

Original: Cond.
Revised: Cond.

Original: 1, Appendix C), and constructing a special form of perturbation at the trivial representation (Suff.
Revised: 1, Appendix C), and constructing a special form of perturbation at the trivial representation (Suff.

Original: Cond.
Revised: Cond.

Original: 2, Appendix F), we arrive at the key result of this paper (see Appendix G for the proof) 2 :
Revised: 2, Appendix F), we arrive at the key result of this paper (see Appendix G for the proof) 2 :

Original: Theorem 2 ( Conﬁdent Subset Suff.
Revised: Theorem 2 ( Conﬁdent Subset Suff.

Original: Cond. ) .
Revised: Cond. ) .

Original: A sufﬁcient condition for ( X, Y ) to be IB β -learnable is X and Y are not independent, and
Revised: A sufﬁcient condition for ( X, Y ) to be IB β -learnable is X and Y are not independent, and

Original: [Equation]
Revised: [Equation]

Original:  where Ω x denotes the event that x ∈ Ω x , with probability p (Ω x ) .
Revised: (2) where Ω x denotes the event that x ∈ Ω x , with probability p (Ω x ) .

Original: Moreover, (inf Ω x ⊂X β 0 (Ω x )) − 1 gives a lower bound on the slope of the Pareto frontier at the origin of the information plane I
Revised: Moreover, (inf Ω x ⊂X β 0 (Ω x )) − 1 gives a lower bound on the slope of the Pareto frontier at the origin of the information plane I

Original: ( Y ; Z ) vs. I ( X ; Z ) .
Revised: ( Y ; Z ) vs. I ( X ; Z ) .

Original: [Equation]

Original: Characteristics of dataset leading to low β 0 .
Revised: Characteristics of dataset leading to low β 0 .

Original: From Eq.
Revised: From Eq.

Original: (2), we see that three characteristics of the subset Ω x ⊂ X lead to low β 0 : (1) conﬁdence: p ( y | Ω x ) is large; (2) typicality and size: the number of elements in Ω x is large, or the elements in Ω x are typical, leading to a large probability of p (Ω x ) ; (3) imbalance: p ( y ) is small for the subset Ω x , but large for its complement.
Revised: (2), we see that three characteristics of the subset Ω x ⊂ X lead to low β 0 : (1) conﬁdence: p ( y | Ω x ) is large; (2) typicality and size: the number of elements in Ω x is large, or the elements in Ω x are typical, leading to a large probability of p (Ω x ) ; (3) imbalance: p ( y ) is small for the subset Ω x , but large for its complement.

Original: In summary, β 0 will be determined by the largest conﬁdent , typical and imbalanced subset of examples, or an equilibrium of those characteristics.
Revised: In summary, β 0 will be determined by the largest conﬁdent , typical and imbalanced subset of examples, or an equilibrium of those characteristics.

Original: Theorem 2 immediately leads to two important corollaries under special problem structures: classiﬁcation with classconditional noisy labels (Angluin & Laird (1988)) and deterministic mappings.
Revised: Theorem 2 immediately leads to two important corollaries under special problem structures: classiﬁcation with classconditional noisy labels (Angluin & Laird (1988)) and deterministic mappings.

Original: Corollary 2.1.
Revised: Corollary 2.1.

Original: Suppose that the true class labels are y ∗ , and the input space belonging to each y ∗ has no overlap.
Revised: Suppose that the true class labels are y ∗ , and the input space belonging to each y ∗ has no overlap.

Original: We only observe the corrupted labels y with class-conditional noise p ( y | x, y ∗ )
Revised: We only observe the corrupted labels y with class-conditional noise p ( y | x, y ∗ )

Original: = p ( y | y ∗ ) (cid:54) =
Revised: = p ( y | y ∗ ) (cid:54) =

Original: p ( y ) .
Revised: p ( y ) .

Original: Then a sufﬁcient condition for IB β -Learnability is:
Revised: Then a sufﬁcient condition for IB β -Learnability is:

Original: [Equation]
Revised: [Equation]

Original: Corollary 2.2.
Revised: Corollary 2.2.

Original: For classiﬁcation problems, if Y is a deterministic function of X and not independent of X , then a necessary and sufﬁcient condition for IB β -Learnability is β > β 

Revised: For classiﬁcation problems, if Y is a deterministic function of X and not independent of X , then a necessary and sufﬁcient condition for IB β -Learnability is β > β

Original: = 1 .
Revised: = 1 .

Original: Therefore, if we ﬁnd that β 0 > 1 for a classiﬁcation task, we may infer that Y is not a deterministic function of X , i.e. either some classes have overlap, or the labels are noisy.
Revised: Therefore, if we ﬁnd that β 0 > 1 for a classiﬁcation task, we may infer that Y is not a deterministic function of X , i.e. either some classes have overlap, or the labels are noisy.

Original: However, ﬁnite models may add effective class overlap if they have insufﬁcient capacity for the learning task.
Revised: However, ﬁnite models may add effective class overlap if they have insufﬁcient capacity for the learning task.

Original: This may translate into a higher observed β 0 , even when learning deterministic functions.
Revised: This may translate into a higher observed β 0 , even when learning deterministic functions.

Original: Proofs are provided in Appendix H.
Revised: Proofs are provided in Appendix H.

Original:  E STIMATING THE IB-L EARNABILITY CONDITION
Revised:  E STIMATING THE IB-L EARNABILITY CONDITION

Original: Based on Theorem 2, for general classiﬁcation tasks we suggest Algorithm 1 in Appendix J to empirically estimate an upper-bound ˜ β 0 ≥ β 0 .
Revised: Based on Theorem 2, for general classiﬁcation tasks we suggest Algorithm 1 in Appendix J to empirically estimate an upper-bound ˜ β 0 ≥ β 0 .

Original: Here, we give the intuition behind the algorithm.
Revised: Here, we give the intuition behind the algorithm.

Original: First, we train a single maximum likelihood model on the dataset.
Revised: First, we train a single maximum likelihood model on the dataset.

Original: That model provides estimates for all p ( y | x ) in the training set.
Revised: That model provides estimates for all p ( y | x ) in the training set.

Original: Since learnability is deﬁned with respect to the training data , it is correct to directly use the empirical probability of p ( x ) and p ( y ) in the training data.
Revised: Since learnability is deﬁned with respect to the training data , it is correct to directly use the empirical probability of p ( x ) and p ( y ) in the training data.

Original: Given p ( x ) , p ( y ) , and p ( y | x ) , and the understanding that we are seaching for a conﬁdent subset Ω x , we can then perform an efﬁcient targeted search of the exponential space of subsets of the training data.
Revised: Given p ( x ) , p ( y ) , and p ( y | x ) , and the understanding that we are seaching for a conﬁdent subset Ω x , we can then perform an efﬁcient targeted search of the exponential space of subsets of the training data.

Original: The algorithm returns the lowest estimate of ˜ β 0 found during that process.
Revised: The algorithm returns the lowest estimate of ˜ β 0 found during that process.

Original: After estimating ˜ β 0 , we can then use it for learning with IB, either directly, or as an anchor for a region where we can perform a much smaller sweep than we otherwise would have.
Revised: After estimating ˜ β 0 , we can then use it for learning with IB, either directly, or as an anchor for a region where we can perform a much smaller sweep than we otherwise would have.

Original: This may be particularly important for very noisy datasets, where β 0 can be very large.
Revised: This may be particularly important for very noisy datasets, where β 0 can be very large.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: To test our theoretical results and Alg. 1, we perform experiments on synthetic datasets, MNIST, and CIFAR10.
Revised: To test our theoretical results and Alg. 1, we perform experiments on synthetic datasets, MNIST, and CIFAR10.

Original: Additional experiment details are provided in Appendix K.
Revised: Additional experiment details are provided in Appendix K.

Original: Synthetic datasets.
Revised: Synthetic datasets.

Original: We generate a set of synthetic datasets with varying class-conditional noise rates.
Revised: We generate a set of synthetic datasets with varying class-conditional noise rates.

Original: Fig.
Revised: Fig.

Original: 1 shows the results of sweeping β to ﬁnd the empirical onset of learning, and compares that onset to the predicted onset using Eq.
Revised: 1 shows the results of sweeping β to ﬁnd the empirical onset of learning, and compares that onset to the predicted onset using Eq.

Original: Clearly the estimate provides a tight upper bound in this simple setting.
Revised: Clearly the estimate provides a tight upper bound in this simple setting.

Original: Also note that β 0 grows exponentially as the label noise increases, underscoring that improperly-chosen β may result in an inability to learn useful representations, and the importance of theoretically-guided β selection as opposed to sweeping β in general.
Revised: Also note that β 0 grows exponentially as the label noise increases, underscoring that improperly-chosen β may result in an inability to learn useful representations, and the importance of theoretically-guided β selection as opposed to sweeping β in general.

Original: MNIST.
Revised: MNIST.

Original: We perform binary classiﬁcation with digits 0 and 1, but again add class-conditional noise to the labels with varying noise rates ρ .
Revised: We perform binary classiﬁcation with digits 0 and 1, but again add class-conditional noise to the labels with varying noise rates ρ .

Original: To explore how the model capacity inﬂuences the onset of learning, for each dataset we train two sets of Variational Information Bottleneck (Alemi et al., 2016)
Revised: To explore how the model capacity inﬂuences the onset of learning, for each dataset we train two sets of Variational Information Bottleneck (Alemi et al., 2016)

Original: (VIB) models differing only by the number of neurons in their hidden layers of the encoder: one with n = 128 neurons, the other with n = 512 neurons.
Revised: (VIB) models differing only by the number of neurons in their hidden layers of the encoder: one with n = 128 neurons, the other with n = 512 neurons.

Original: Insufﬁcient capacity will result in more uncertainty of Y given X from the point of view of the model, so we expect β 0 , observed for the n = 128 model to be larger.
Revised: Insufﬁcient capacity will result in more uncertainty of Y given X from the point of view of the model, so we expect β 0 , observed for the n = 128 model to be larger.

Original: Fig.
Revised: Fig.

Original: 1 conﬁrms this prediction.
Revised: 1 conﬁrms this prediction.

Original: It also shows the β 0 , estimated and β 0 , predicted given by
Revised: It also shows the β 0 , estimated and β 0 , predicted given by

Original: Algorithm 1 and Eq.
Revised: Algorithm 1 and Eq.

Original: (3), respectively.
Revised: (3), respectively.

Original: We see that Algorithm 1 does a good job estimating the onset of learning for the large-capacity model, and that the estimated results line up well with the theoretical predictions.
Revised: We see that Algorithm 1 does a good job estimating the onset of learning for the large-capacity model, and that the estimated results line up well with the theoretical predictions.

Original: [Figure]
Revised: [Figure]

Original: CIFAR10 forgetting.
Revised: CIFAR10 forgetting.

Original: For CIFAR10 (Krizhevsky & Hinton, 2009), we study how forgetting varies with β .
Revised: For CIFAR10 (Krizhevsky & Hinton, 2009), we study how forgetting varies with β .

Original: In other words, given a VIB model trained at some high β 2 , if we anneal it down to some much lower β 1 , what accuracy does the model converge to?
Revised: In other words, given a VIB model trained at some high β 2 , if we anneal it down to some much lower β 1 , what accuracy does the model converge to?

Original: We estimated β 0 = 1 . 0483 on a version of CIFAR10 with 20% label noise using Alg. 1.
Revised: We estimated β 0 = 1 . 0483 on a version of CIFAR10 with 20% label noise using Alg. 1.

Original: The lowest β with performance above chance was β = 1 . 048 .
Revised: The lowest β with performance above chance was β = 1 . 048 .

Original: See Appendix K.1 for experiment details.
Revised: See Appendix K.1 for experiment details.

Original: As can be seen in Fig.
Revised: As can be seen in Fig.

Original: 1 (d) , there are large discontinuities in the Pareto frontier, even though we vary β in very small increments.
Revised: 1 (d) , there are large discontinuities in the Pareto frontier, even though we vary β in very small increments.

Original: Those discontinuities start at points on the Pareto frontier where many values of β yield essentially the same I ( X ; Z ) and I
Revised: Those discontinuities start at points on the Pareto frontier where many values of β yield essentially the same I ( X ; Z ) and I

Original: ( Y ; Z ) , and end when β crosses apparent phase transitions that give large increases in both
Revised: ( Y ; Z ) , and end when β crosses apparent phase transitions that give large increases in both

Original: I ( X ; Z ) and I ( Y ; Z ) (marked with red arrows).
Revised: I ( X ; Z ) and I ( Y ; Z ) (marked with red arrows).

Original: Fig.
Revised: Fig.

Original: 1 (c) shows that the lowest value of β in each such region tends to have the highest accuracy.
Revised: 1 (c) shows that the lowest value of β in each such region tends to have the highest accuracy.

Original: A primary empirical result of our work is the following: some datasets have non-monotonic performance in regions where multiple values of β cluster together.
Revised: A primary empirical result of our work is the following: some datasets have non-monotonic performance in regions where multiple values of β cluster together.

Original: This surprising behavior is important to check for when training IB models.
Revised: This surprising behavior is important to check for when training IB models.

Original: More thorough study is needed, but based on our initial results, we may expect that reducing β to the minimal value that achieves a particular point on the information plane yields better representations.
Revised: More thorough study is needed, but based on our initial results, we may expect that reducing β to the minimal value that achieves a particular point on the information plane yields better representations.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we have presented theoretical results for predicting the onset of learning, and have shown that it is determined by the largest conﬁdent, typical and imbalanced subset of the examples.
Revised: In this paper, we have presented theoretical results for predicting the onset of learning, and have shown that it is determined by the largest conﬁdent, typical and imbalanced subset of the examples.

Original: We gave a practical algorithm for predicting the transition, and showed that those predictions are accurate, even in cases of extreme label noise.
Revised: We gave a practical algorithm for predicting the transition, and showed that those predictions are accurate, even in cases of extreme label noise.

Original: We have also observed a surprising non-monotonic relationship between β and accuracy, and shown its relationship to discontinuities in the Pareto frontier of the information plane.
Revised: We have also observed a surprising non-monotonic relationship between β and accuracy, and shown its relationship to discontinuities in the Pareto frontier of the information plane.

Original: We believe these results will provide theoretical and practical guidance for choosing β in the IB framework for balancing prediction and compression.
Revised: We believe these results will provide theoretical and practical guidance for choosing β in the IB framework for balancing prediction and compression.

Original: Our work also raises other questions, such as whether there are other phase transitions in learnability that might be identiﬁed.
Revised: Our work also raises other questions, such as whether there are other phase transitions in learnability that might be identiﬁed.

Original: We hope to address some of those questions in future work.Appendix
Revised: We hope to address some of those questions in future work.

Original: The structure of the Appendix is as follows.

Original: In Appendix A, we provide preliminaries for the ﬁrst-order and secondorder variations on functionals.

Original: Then we prove Theorem 1 in Appendix B.

Original: In Appendix C, we state and prove Sufﬁcient

Original: Condition 1 for IB β -learnability.

Original: In Appendix D, we calculate the ﬁrst and second variations of IB β [ p ( z | x )] at the trivial representation p ( z | x )

Original: = p ( z ) , which is used in proving the Sufﬁcient Condition 2 IB β -learnability (Appendix

Original: F).

Original: After these preparations, we prove the key result of this paper, Theorem 2, in Appendix G.

Original: Then two important corollaries 2.1, 2.2 are proved in Appendix H. We provide additional discussions and insights for Theorem 2 in

Original: Appendix I, and Algorithm 1 for estimation of an upper bound ˜ β 0 ≥ β 0 in Appendix J. Finally in Appendix K, we provide details for the experiments.

Original: A P RELIMINARIES : FIRST - ORDER AND SECOND - ORDER VARIATION
Revised: A P RELIMINARIES : FIRST - ORDER AND SECOND - ORDER VARIATION

Original: Let functional F [ f ( x )] be deﬁned on some normed linear space R .
Revised: Let functional F [ f ( x )] be deﬁned on some normed linear space R .

Original: Let us add a perturbative function (cid:15)h ( x ) to f ( x ) , and now the functional F [ f ( x ) + (cid:15)h ( x )] can be expanded as∆ [Equation] [Equation] ( x )] =
Revised: Let us add a perturbative function (cid:15)h ( x ) to f ( x ) , and now the functional F [ f ( x ) + (cid:15)h ( x )] can be expanded as

Original: [Equation]
Revised: [Equation]

Original: [ f ( x ) + (cid:15)h ( x )] − F [ f ( x )] =

Original: [Equation]

Original: [Equation]

Original: ( x )]

Original: + ϕ 2 [ f ( x )]

Original: + O ( (cid:15)

Original: || h || 2 ) where || h || denotes the norm of h , ϕ 

Revised: where || h || denotes the norm of h , ϕ

Original: [ f ( x )] = (cid:15) dF [ f ( x )] d(cid:15) is a linear functional of (cid:15)h ( x ) , and is called the ﬁrst-order variation , denoted as δF [ f ( x )] .
Revised: [ f ( x )] = (cid:15) dF [ f ( x )] d(cid:15) is a linear functional of (cid:15)h ( x ) , and is called the ﬁrst-order variation , denoted as δF [ f ( x )] .

Original: ϕ 

Revised: ϕ

Original: [ f ( x )] = 12 (cid:15) 2 d 2 F [ f ( x )]
Revised: [ f ( x )] = 12 (cid:15) 2 d 2 F [ f ( x )]

Original: d(cid:15)is a quadratic functional of (cid:15)h ( x ) , and is called the secondorder variation , denoted as δ 2 F [ f ( x )] .
Revised: d(cid:15)is a quadratic functional of (cid:15)h ( x ) , and is called the secondorder variation , denoted as δ 2 F [ f ( x )] .

Original: If δF [ f ( x )] = 0 , we call f ( x ) a stationary solution for the functional F [ · ] .
Revised: If δF [ f ( x )] = 0 , we call f ( x ) a stationary solution for the functional F [ · ] .

Original: If ∆ F [ f ( x )] ≥ 0 for all h ( x ) such that f ( x ) + (cid:15)h ( x ) is at the neighborhood of f ( x ) , we call f ( x ) a (local) minimum of F [ · ] .
Revised: If ∆ F [ f ( x )] ≥ 0 for all h ( x ) such that f ( x ) + (cid:15)h ( x ) is at the neighborhood of f ( x ) , we call f ( x ) a (local) minimum of F [ · ] .

Original: B P ROOF OF T HEOREM
Revised: B P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: At the trivial representation p ( z | x ) = p ( z ) , we have I ( X ; Z ) = 0 , and I ( Y ; Z ) = 

Revised: At the trivial representation p ( z | x ) = p ( z ) , we have I ( X ; Z ) = 0 , and I ( Y ; Z ) =

Original: due to the Markov chain,
Revised: due to the Markov chain,

Original: so IB β ( X, Y ; Z ) | p ( z | x )=
Revised: so IB β ( X, Y ; Z ) | p ( z | x )=

Original: p ( z ) = 

Revised: p ( z ) =

Original: for any β .
Revised: for any β .

Original: Since ( X, Y ) is IB β 1 -learnable, there exists a Z given by a p 1 ( z | x ) such that IB β 1 ( X, Y ; Z ) | p 1 ( z | x ) < 0 .
Revised: Since ( X, Y ) is IB β 1 -learnable, there exists a Z given by a p 1 ( z | x ) such that IB β 1 ( X, Y ; Z ) | p 1 ( z | x ) < 0 .

Original: Since β 2 > β 1 , and I ( Y ; Z ) ≥ 0 , we have IB β 2 ( X, Y ; Z ) | p 1 ( z | x ) ≤
Revised: Since β 2 > β 1 , and I ( Y ; Z ) ≥ 0 , we have IB β 2 ( X, Y ; Z ) | p 1 ( z | x ) ≤

Original: IB β 1 ( X, Y ; Z ) | p 1 ( z | x ) < 

Revised: IB β 1 ( X, Y ; Z ) | p 1 ( z | x ) <

Original: = IB β 2 ( X, Y ; Z ) | p ( z | x )=
Revised: = IB β 2 ( X, Y ; Z ) | p ( z | x )=

Original: p ( z ) .
Revised: p ( z ) .

Original: Therefore, ( X, Y ) is IB β 2 -learnable.
Revised: Therefore, ( X, Y ) is IB β 2 -learnable.

Original: C S UFFICIENT CONDITION 1 AND PROOF
Revised: C S UFFICIENT CONDITION 1 AND PROOF

Original: In this section, we prove the Sufﬁcient Condition 1 for IB β -learnability, which will lay the foundation for the Sufﬁcient condition 2 (Appendix F) and the Conﬁdent Subset Sufﬁcient condition (key result of this paper, Theorem 2) that follow.
Revised: In this section, we prove the Sufﬁcient Condition 1 for IB β -learnability, which will lay the foundation for the Sufﬁcient condition 2 (Appendix F) and the Conﬁdent Subset Sufﬁcient condition (key result of this paper, Theorem 2) that follow.

Original: Theorem 3 ( Suff.
Revised: Theorem 3 ( Suff.

Original: Cond. 1 ) .
Revised: Cond. 1 ) .

Original: A sufﬁcient condition for ( X, Y ) to be IB β -learnable is that there exists a perturbation function h
Revised: A sufﬁcient condition for ( X, Y ) to be IB β -learnable is that there exists a perturbation function h

Original: ( z | x ) with 3 (cid:82) h ( z | x ) dz = 0 ,
Revised: ( z | x ) with 3 (cid:82) h ( z | x ) dz = 0 ,

Original: such that the second-order variation δ 2 IB β [ p ( z | x )] < 0 at the trivial representation p ( z | x )
Revised: such that the second-order variation δ 2 IB β [ p ( z | x )] < 0 at the trivial representation p ( z | x )

Original: = p ( z ) .
Revised: = p ( z ) .

Original: Proof.
Revised: Proof.

Original: To prove Theorem 3, we use the Theorem 1 of Chapter 5 of Gelfand et al.
Revised: To prove Theorem 3, we use the Theorem 1 of Chapter 5 of Gelfand et al.

Original: (2000) which gives a necessary condition for F [ f ( x )] to have a minimum at f 0 ( x ) .
Revised: (2000) which gives a necessary condition for F [ f ( x )] to have a minimum at f 0 ( x ) .

Original: Adapting to our notation, we have:
Revised: Adapting to our notation, we have:

Original: Theorem 4 (Gelfand et al.
Revised: Theorem 4 (Gelfand et al.

Original: A necessary condition for the functional F [ f ( x )] to have a minimum at f ( x ) = f 0 ( x ) is that for f ( x ) = f 0 ( x ) and all admissible (cid:15)h ( x ) ,
Revised: A necessary condition for the functional F [ f ( x )] to have a minimum at f ( x ) = f 0 ( x ) is that for f ( x ) = f 0 ( x ) and all admissible (cid:15)h ( x ) ,

Original: [Equation]
Revised: [Equation]

Original: Applying to our functional IB β [ p ( z | x )] , an immediate result of Theorem 4 is that, if at p ( z | x ) = p ( z ) , there exists an (cid:15)h ( z | x ) such that δ 2 IB β [ p ( z | x )] < 0 , then p ( z | x ) = p ( z ) is not a minimum for IB β [ p ( z | x )] .
Revised: Applying to our functional IB β [ p ( z | x )] , an immediate result of Theorem 4 is that, if at p ( z | x ) = p ( z ) , there exists an (cid:15)h ( z | x ) such that δ 2 IB β [ p ( z | x )] < 0 , then p ( z | x ) = p ( z ) is not a minimum for IB β [ p ( z | x )] .

Original: Using the deﬁnition of
Revised: Using the deﬁnition of

Original: IB β learnability, we have that ( X, Y ) is IB β -learnable.
Revised: IB β learnability, we have that ( X, Y ) is IB β -learnable.

Original: Intuitively, if δ 2 IB β [ p ( z | x )] (cid:12)(cid:12) p ( z | x )= p ( z ) < 0 , we can always ﬁnd a p (cid:48)
Revised: Intuitively, if δ 2 IB β [ p ( z | x )] (cid:12)(cid:12) p ( z | x )= p ( z ) < 0 , we can always ﬁnd a p (cid:48)

Original: ( z | x ) = p ( z | x ) + h ( z | x ) in the neighborhood of the trivial representation p ( z | x ) = p ( z ) , such that IB β [ p (cid:48)
Revised: ( z | x ) = p ( z | x ) + h ( z | x ) in the neighborhood of the trivial representation p ( z | x ) = p ( z ) , such that IB β [ p (cid:48)

Original: ( z | x )] < IB β
Revised: ( z | x )] < IB β

Original: [ p ( z | x )] , thus satisfying the deﬁnition for
Revised: [ p ( z | x )] , thus satisfying the deﬁnition for

Original: IB β -Learnability.
Revised: IB β -Learnability.

Original: To make Theorem 3 more practical, we perturb p ( z | x ) around the trivial solution p (cid:48)
Revised: To make Theorem 3 more practical, we perturb p ( z | x ) around the trivial solution p (cid:48)

Original: ( z | x ) = p ( z | x ) + (cid:15)h ( z | x ) , and expand IB β
Revised: ( z | x ) = p ( z | x ) + (cid:15)h ( z | x ) , and expand IB β

Original: [ p ( z | x ) + h ( z | x )] − IB β
Revised: [ p ( z | x ) + h ( z | x )] − IB β

Original: [ p ( z | x )] to the second order of (cid:15) .
Revised: [ p ( z | x )] to the second order of (cid:15) .

Original: We can then prove Theorem 5:
Revised: We can then prove Theorem 5:

Original: D F IRST - AND SECOND - ORDER VARIATIONS OF IB β [ p ( z | x )]
Revised: D F IRST - AND SECOND - ORDER VARIATIONS OF IB β [ p ( z | x )]

Original: In this section, we derive the ﬁrst- and second-order variations of IB β [ p ( z | x )] , which are needed for proving Lemma 1.1 and Theorem 5.
Revised: In this section, we derive the ﬁrst- and second-order variations of IB β [ p ( z | x )] , which are needed for proving Lemma 1.1 and Theorem 5.

Original: Lemma 4.1.
Revised: Lemma 4.1.

Original: Using perturbative function h ( z | x ) , we have
Revised: Using perturbative function h ( z | x ) , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Since IB β [ p ( z | x )] = I ( X ; Z ) − βI ( Y ; Z ) , let us calculate the ﬁrst and second-order variation of I ( X ; Z ) and
Revised: Since IB β [ p ( z | x )] = I ( X ; Z ) − βI ( Y ; Z ) , let us calculate the ﬁrst and second-order variation of I ( X ; Z ) and

Original: I ( Y ; Z ) w.r.t.
Revised: I ( Y ; Z ) w.r.t.

Original: p ( z | x ) , respectively.
Revised: p ( z | x ) , respectively.

Original: Through this derivation, we use (cid:15)h ( z | x ) as a perturbative function, for ease of deciding different orders of variations.
Revised: Through this derivation, we use (cid:15)h ( z | x ) as a perturbative function, for ease of deciding different orders of variations.

Original: We will ﬁnally absorb (cid:15) into h ( z | x ) .
Revised: We will ﬁnally absorb (cid:15) into h ( z | x ) .

Original: Denote I ( X ; Z ) =
Revised: Denote I ( X ; Z ) =

Original: F 1 [ p ( z | x )] .
Revised: F 1 [ p ( z | x )] .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Since
Revised: Since

Original: [Equation]
Revised: [Equation]

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Expanding F
Revised: Expanding F

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: | x )] to the second order of (cid:15) , we have
Revised: x )] to the second order of (cid:15) , we have

Original: [Equation]
Revised: [Equation]

Original: z )

Original: [Equation]
Revised: [Equation]

Original: Collecting the ﬁrst order terms of (cid:15) , we have δF
Revised: Collecting the ﬁrst order terms of (cid:15) , we have δF

Original: [Equation]
Revised: [Equation]

Original: [ p ( z | x )]
Revised: [ p [Equation] z | x )]

Original: [Equation]
Revised: [Equation]

Original: Collecting the second order terms of (cid:15) 2 , we have δ 2 F 1 [ p ( z | x )] =
Revised: Collecting the second order terms of (cid:15) 2 , we have δ 2 F 1 [ p ( z | x )] =

Original: [Equation]
Revised: [Equation]

Original: p ( z ) (cid:19)
Revised: p ( z [Equation] (cid:19)

Original: [Equation]
Revised: [Equation]

Original: − h ( z | x ) p ( z | x ) (cid:82) h ( z | x (cid:48) )
Revised: − h ( z | x ) p ( z | x ) (cid:82) h ( z | x (cid:48) )

Original: p ( x (cid:48) ) dx (cid:48)
Revised: p ( x (cid:48) ) dx (cid:48)

Original: p ( z ) − (cid:18) h
Revised: p ( z ) − (cid:18) h

Original: ( z | x ) p ( z | x ) − (cid:82) h ( z | x (cid:48) )
Revised: ( z | x ) p ( z | x ) − (cid:82) h ( z | x (cid:48) )

Original: p ( x (cid:48) ) dx (cid:48)
Revised: p ( x (cid:48) ) dx (cid:48)

Original: p ( z ) (cid:19) [Equation] (cid:21) 
Revised: p ( z ) (cid:19) 2 (cid:21) + (cid:15)

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: h [Equation] Now let us calculate the ﬁrst and second-order variation of F 2 [ p ( z | x )]
Revised: Now let us calculate the ﬁrst and second-order variation of F 2 [ p ( z | x )]

Original: = I ( Z ; Y ) .
Revised: = I ( Z ; Y ) .

Original: We have
Revised: We have

Original: F 2 [ p
Revised: F 2 [ p

Original: [Equation]
Revised: [Equation]

Original: z | x )] =
Revised: z | x )] =

Original: I

Original: [Equation]
Revised: [Equation]

Original: ; Z ) =
Revised: ; Z ) =

Original: [Equation]
Revised: [Equation]

Original: log
Revised: log

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]

Original: x

Original: [Equation]
Revised: [Equation]

Original:  y )log
Revised: x, y )

Original: [Equation]
Revised: [Equation]

Original: y

Original: [Equation]
Revised: [Equation]

Original: ) p

Original: [Equation]

Original: y )

Original: [Equation]
Revised: [Equation]

Original: z )
Revised: z )

Original: Using the Markov chain Z ← X ↔ Y , we have
Revised: Using the Markov chain Z ← X ↔ Y , we have

Original: [Equation]
Revised: [Equation]

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: Then expanding F 2 [ p ( z | x ) + (cid:15)h ( z | x )] to the second order of (cid:15)
Revised: Then expanding F 2 [ p ( z | x ) + (cid:15)h ( z | x )] to the second order of (cid:15)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: F 2 [ p( z | x
Revised: F 2 [ p

Original: [Equation]
Revised: [Equation]

Original: +
Revised: z

Original: [Equation]
Revised: [Equation]

Original: z | x )]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: + O ( (cid:15) 3 )
Revised: + O ( (cid:15) 3 )

Original: Collecting the ﬁrst order terms of (cid:15)
Revised: Collecting the ﬁrst order terms of (cid:15)

Original: [Equation]
Revised: [Equation]

Original: we have δF
Revised: we have δF

Original: [Equation]
Revised: [Equation]

Original: [ p
Revised: [ p

Original: [Equation]
Revised: [Equation]

Original: z
Revised: z

Original: [Equation]
Revised: [Equation]

Original: x )]
Revised: x [Equation]

Original: [Equation]
Revised: [Equation]

Original: Collecting the second order terms of (cid:15)
Revised: Collecting the second order terms of (cid:15)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: [ p
Revised: [ p

Original: [Equation]
Revised: [Equation]

Original: z | x )]
Revised: z | x )]

Original: [Equation]
Revised: [Equation]

Original: Finally
Revised: Finally

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Revised: [Equation]

Original: h ( z | x
Revised: [Table] [Equation] [Table] [Equation]

Original: [Table]
Revised: [Table]

Original: get rid of the (cid:15)
Revised: [Equation] [Table] [Equation]

Original: [Table]
Revised: [Table]

Original: and obtain the ﬁnal expression in Lemma
Revised: h ( z | x )

Original: [Table]
Revised: [Table]

Revised: we get rid of the (cid:15) factor and obtain the ﬁnal expression in Lemma 4.

Original: [Table]
Revised: [Table]

Original: E P ROOF OF L EMMA 1.Proof 
Revised: E P ROOF OF L EMMA 1.

Revised: Proof.

Original: Using Lemma 4.1, we have
Revised: Using Lemma 4.1, we have

Original: [Equation]
Revised: [Equation]

Original: Let p ( z | x ) =
Revised: Let p ( z | x ) =

Original: p ( z ) (the trivial representation), we have that log p ( z | x ) p ( z ) ≡ 0 .
Revised: p ( z ) (the trivial representation), we have that log p ( z | x ) p ( z ) ≡ 0 .

Original: Therefore, the two integrals are both 0.
Revised: Therefore, the two integrals are both 0.

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: Therefore, the p ( z | x ) = p ( z ) is a stationary solution for IB β [ p ( z | x )] .
Revised: Therefore, the p ( z | x ) = p ( z ) is a stationary solution for IB β [ p ( z | x )] .

Original: F S UFFICIENT CONDITION 2 AND PROOF F. S TATEMENT OF THE THEOREM
Revised: F S UFFICIENT CONDITION 2 AND PROOF F. S TATEMENT OF THE THEOREM

Original: Theorem 5 ( Suff.
Revised: Theorem 5 ( Suff.

Original: Cond. 2 ) .
Revised: Cond. 2 ) .

Original: A sufﬁcient condition for ( X, Y ) to be IB β -learnable is X and Y are not independent, and
Revised: A sufﬁcient condition for ( X, Y ) to be IB β -learnable is X and Y are not independent, and

Original: [Equation]
Revised: [Equation]

Original: where the functional β 

Revised: where the functional β

Original: [ h ( x )] is given by
Revised: [ h ( x )] is given by

Original: [Equation]
Revised: [Equation]

Original: Moreover, we have that (cid:0) inf h ( x ) β [ h ( x )] (cid:1) − 1 is a lower bound of the slope of the Pareto frontier in the information plane
Revised: Moreover, we have that (cid:0) inf h ( x ) β [ h ( x )] (cid:1) − 1 is a lower bound of the slope of the Pareto frontier in the information plane

Original: I
Revised: I

Original: ( Y ; Z ) vs. I ( X ; Z ) at the origin.
Revised: ( Y ; Z ) vs. I ( X ; Z ) at the origin.

Original: The proof is given in Appendix F, which also gives a construction for h ( z | x ) for Theorem 3 for any h ( x ) satisfying
Revised: The proof is given in Appendix F, which also gives a construction for h ( z | x ) for Theorem 3 for any h ( x ) satisfying

Original: Theorem 5, and shows that the converse is also true: if there exists h ( z | x ) suth that the condition in Theorem 3 is true, then we can ﬁnd h ( x ) satisfying the the condition in Theorem 5.
Revised: Theorem 5, and shows that the converse is also true: if there exists h ( z | x ) suth that the condition in Theorem 3 is true, then we can ﬁnd h ( x ) satisfying the the condition in Theorem 5.

Original: The geometric meaning of ( β 

Revised: The geometric meaning of ( β

Original: [ h ( x )]) − 1 is as follows.
Revised: [ h ( x )]) − 1 is as follows.

Original: It equals ∆ I ( Y ; Z ) ∆ I ( X ; Z ) (cid:12)(cid:12) p ( z | x )=
Revised: It equals ∆ I ( Y ; Z ) ∆ I ( X ; Z ) (cid:12)(cid:12) p ( z | x )=

Original: p ( z ) under a perturbation function of the form
Revised: p ( z ) under a perturbation function of the form

Original: h 1 ( z | x ) =
Revised: h 1 ( z | x ) =

Original: h ( x ) h 2 ( z ) (satisfying (cid:82)
Revised: h ( x ) h 2 ( z ) (satisfying (cid:82)

Original: h 2 ( z ) dz = 

Revised: h 2 ( z ) dz =

Original: and (cid:82) h 22 ( z ) p ( z ) dz > 0 ) at the trivial representation, where ∆ I ( X ; Z ) =
Revised: and (cid:82) h 22 ( z ) p ( z ) dz > 0 ) at the trivial representation, where ∆ I ( X ; Z ) =

Original: I ( X ; Z ) | p ( z | x )+ h 1 ( z | x ) − I ( X ; Z ) | p ( z | x ) and similarly for I ( Y ; Z ) .
Revised: I ( X ; Z ) | p ( z | x )+ h 1 ( z | x ) − I ( X ; Z ) | p ( z | x ) and similarly for I ( Y ; Z ) .

Original: Since the ﬁrst order variation vanishes for both I ( X ; Z ) and I ( Y ; Z ) , we have ∆ I ( Y ; Z ) ∆ I ( X ; Z )
Revised: Since the ﬁrst order variation vanishes for both I ( X ; Z ) and I ( Y ; Z ) , we have ∆ I ( Y ; Z ) ∆ I ( X ; Z )

Original: =
Revised: =

Original: δ 2 I ( Y ; Z )
Revised: δ 2 I ( Y ; Z )

Original: δ 

Revised: δ

Original: I ( X ; Z ) , which turns out to be equal to ( β 

Revised: I ( X ; Z ) , which turns out to be equal to ( β

Original: [ h ( x )]) − 1 .
Revised: [ h ( x )]) − 1 .

Original: Therefore, (cid:0) inf h ( x ) β [ h ( x )] (cid:1) − 1 gives the highest ∆ I ( Y ; Z ) ∆ I ( X ; Z ) (cid:12)(cid:12) p ( z | x )= p ( z ) under the class of perturbation functions h 1 ( z | x )
Revised: Therefore, (cid:0) inf h ( x ) β [ h ( x )] (cid:1) − 1 gives the highest ∆ I ( Y ; Z ) ∆ I ( X ; Z ) (cid:12)(cid:12) p ( z | x )= p ( z ) under the class of perturbation functions h 1 ( z | x )

Original: =
Revised: =

Original: h 2 ( z ) h
Revised: h 2 ( z ) h

Original: ( x ) , and provides a lower bound of sup h ( z | x ) ∆ I ( Y ; Z ) ∆
Revised: ( x ) , and provides a lower bound of sup h ( z | x ) ∆ I ( Y ; Z ) ∆

Original: I ( X ; Z ) (cid:12)(cid:12) p ( z | x )= p ( z ) , which is the slope of the Pareto frontier in the information plane I ( Y ; Z ) vs. I ( X ; Z ) at the origin.
Revised: I ( X ; Z ) (cid:12)(cid:12) p ( z | x )= p ( z ) , which is the slope of the Pareto frontier in the information plane I ( Y ; Z ) vs. I ( X ; Z ) at the origin.

Original: Theorem 5 in essence states that as long as β − 1 is lower than this lower bound of the slope of the Pareto frontier, ( X ; Y ) is IB β -learnable.
Revised: Theorem 5 in essence states that as long as β − 1 is lower than this lower bound of the slope of the Pareto frontier, ( X ; Y ) is IB β -learnable.

Original: From Theorem 5, we see that it still has an inﬁmum over an arbitrary function h ( x ) , which is not easy to estimate.
Revised: From Theorem 5, we see that it still has an inﬁmum over an arbitrary function h ( x ) , which is not easy to estimate.

Original: To get rid of h ( x ) , we can use a speciﬁc functional form for h ( x ) in Eq.
Revised: To get rid of h ( x ) , we can use a speciﬁc functional form for h ( x ) in Eq.

Original: (5), and obtain a stronger sufﬁcient condition for
Revised: (5), and obtain a stronger sufﬁcient condition for

Original: IB β -Learnability.
Revised: IB β -Learnability.

Original: But we want to choose h ( x ) as near to the inﬁmum as possible.
Revised: But we want to choose h ( x ) as near to the inﬁmum as possible.

Original: To do this, we note the following characteristics for the R.H.S of Eq.
Revised: To do this, we note the following characteristics for the R.H.S of Eq.

Original: • We can set h ( x ) to be nonzero if x ∈ Ω x for some region Ω x ⊂ X and 0 otherwise.
Revised: • We can set h ( x ) to be nonzero if x ∈ Ω x for some region Ω x ⊂ X and 0 otherwise.

Original: Then we obtain the following sufﬁcient condition:
Revised: Then we obtain the following sufﬁcient condition:

Original: [Equation]
Revised: [Equation]

Original: • The numerator of the R.H.S. of Eq.
Revised: • The numerator of the R.H.S. of Eq.

Original: (6) attains its minimum when h ( x ) is a constant within Ω x .
Revised: (6) attains its minimum when h ( x ) is a constant within Ω x .

Original: This can be proved using the Cauchy-Schwarz inequality: (cid:104) u, u (cid:105)(cid:104) v, v (cid:105) ≥ (cid:104) u, v (cid:105) 2 , setting u ( x ) = h ( x ) (cid:112) p ( x ) , v ( x ) =
Revised: This can be proved using the Cauchy-Schwarz inequality: (cid:104) u, u (cid:105)(cid:104) v, v (cid:105) ≥ (cid:104) u, v (cid:105) 2 , setting u ( x ) = h ( x ) (cid:112) p ( x ) , v ( x ) =

Original: (cid:112) p ( x ) , and deﬁning the inner product as (cid:104) u, v (cid:105) =
Revised: (cid:112) p ( x ) , and deﬁning the inner product as (cid:104) u, v (cid:105) =

Original: (cid:82) u ( x ) v ( x ) dx .
Revised: (cid:82) u ( x ) v ( x ) dx .

Original: Therefore, the numerator of the R.H.S. of Eq.
Revised: Therefore, the numerator of the R.H.S. of Eq.

Original: (6) ≥
Revised: (6) ≥

Original: [Equation]
Revised: [Equation]

Original:  and attains equality when u ( x ) v ( x ) 
Revised: − 1 , and attains equality when u ( x ) v ( x ) =

Original: = h ( x ) is constant.
Revised: h ( x ) is constant.

Original: Based on these observations, we can let h ( x ) be a nonzero constant inside some region Ω x ⊂ X and 0 otherwise, and the inﬁmum over an arbitrary function h ( x ) is simpliﬁed to inﬁmum over Ω x ⊂ X , and we obtain the conﬁdent subset sufﬁcient condition (Theorem 2) for IB β -Learnability, which is a key result of this paper.
Revised: Based on these observations, we can let h ( x ) be a nonzero constant inside some region Ω x ⊂ X and 0 otherwise, and the inﬁmum over an arbitrary function h ( x ) is simpliﬁed to inﬁmum over Ω x ⊂ X , and we obtain the conﬁdent subset sufﬁcient condition (Theorem 2) for IB β -Learnability, which is a key result of this paper.

Original: F. P ROOF OF T HEOREM 5 (S UFF . C OND . 2)
Revised: F. P ROOF OF T HEOREM 5 (S UFF . C OND . 2)

Original: Proof.
Revised: Proof.

Original: Firstly, from the necessary condition of β > 1 in Section 2, we have that any sufﬁcient condition for IB βlearnability should be able to deduce β > 1 .
Revised: Firstly, from the necessary condition of β > 1 in Section 2, we have that any sufﬁcient condition for IB βlearnability should be able to deduce β > 1 .

Original: Now using Theorem 3, a sufﬁcient condition for ( X, Y ) to be IB β -learnable is that there exists h ( z | x ) with (cid:82) h ( z | x ) dx = 

Revised: Now using Theorem 3, a sufﬁcient condition for ( X, Y ) to be IB β -learnable is that there exists h ( z | x ) with (cid:82) h ( z | x ) dx =

Original: such that δ 2 IB β [ p ( z | x )] < 0 at p ( z | x ) = p ( x ) .
Revised: such that δ 2 IB β [ p ( z | x )] < 0 at p ( z | x ) = p ( x ) .

Original: At the trivial representation, p ( z | x ) = p ( z ) and hence p ( x, z ) = p ( x ) p ( z ) .
Revised: At the trivial representation, p ( z | x ) = p ( z ) and hence p ( x, z ) = p ( x ) p ( z ) .

Original: Due to the Markov chain Z ← X ↔ Y , we have p ( y, z ) =
Revised: Due to the Markov chain Z ← X ↔ Y , we have p ( y, z ) =

Original: p ( y ) p ( z ) .
Revised: p ( y ) p ( z ) .

Original: Substituting them into the δ 2 IB β [ p ( z | x )] in Lemma 4.1, the condition becomes: there exists h ( z | x ) with (cid:82) h ( z | x ) dz = 0 , such that
Revised: Substituting them into the δ 2 IB β [ p ( z | x )] in Lemma 4.1, the condition becomes: there exists h ( z | x ) with (cid:82) h ( z | x ) dz = 0 , such that

Original: [Equation]
Revised: [Equation]

Original: Rearranging terms and simplifying, we have
Revised: Rearranging terms and simplifying, we have

Original: [Equation]
Revised: [Equation]

Revised: <

Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Now we prove that the condition that ∃ h ( z | x ) s.t.
Revised: Now we prove that the condition that ∃ h ( z | x ) s.t.

Original: (cid:82) dzp ( z ) G [ h ( z | x )] < 0 is equivalent to the condition that ∃ h ( x ) s.t.
Revised: (cid:82) dzp ( z ) G [ h ( z | x )] < 0 is equivalent to the condition that ∃ h ( x ) s.t.

Original: G [ h ( x )] < 0 .
Revised: G [ h ( x )] < 0 .

Original: If ∀ h ( z | x ) , G [ h ( z | x )] ≥ 0 , then we have ∀ h
Revised: If ∀ h ( z | x ) , G [ h ( z | x )] ≥ 0 , then we have ∀ h

Original: ( z | x ) , (cid:82) dzp ( z ) G [ h ( z | x )] ≥ 0 .
Revised: ( z | x ) , (cid:82) dzp ( z ) G [ h ( z | x )] ≥ 0 .

Original: Therefore, if ∃ h ( z | x ) s.t.
Revised: Therefore, if ∃ h ( z | x ) s.t.

Original: (cid:82) dzp ( z ) G [ h ( z | x )] < 0 , we have that ∃ h
Revised: (cid:82) dzp ( z ) G [ h ( z | x )] < 0 , we have that ∃ h

Original: ( z | x ) s.t.
Revised: ( z | x ) s.t.

Original: G [ h ( z | x )] < 0 .
Revised: G [ h ( z | x )] < 0 .

Original: Since the functional G [ h ( z | x )] does not contain integration over z , we can treat the z in G [ h ( z | x )] as a parameter and we have that ∃ h ( x ) s.t.
Revised: Since the functional G [ h ( z | x )] does not contain integration over z , we can treat the z in G [ h ( z | x )] as a parameter and we have that ∃ h ( x ) s.t.

Original: G [ h ( x )] < 0 .
Revised: G [ h ( x )] < 0 .

Original: Conversely, if there exists an certain function h ( x ) such that G [ h ( x )] < 0 , we can ﬁnd some h 2 ( z ) such that (cid:82) h 2 ( z ) dz = 0 and (cid:82) h 22 ( z ) p ( z ) dz > 0 , and let h 1 ( z | x )
Revised: Conversely, if there exists an certain function h ( x ) such that G [ h ( x )] < 0 , we can ﬁnd some h 2 ( z ) such that (cid:82) h 2 ( z ) dz = 0 and (cid:82) h 22 ( z ) p ( z ) dz > 0 , and let h 1 ( z | x )

Original: =
Revised: =

Original: h ( x ) h 2 ( z ) .
Revised: h ( x ) h 2 ( z ) .

Original: Now we have
Revised: Now we have

Original: [Equation]
Revised: [Equation]

Original: In other words, the condition Eq.
Revised: In other words, the condition Eq.

Original: (7) is equivalent to requiring that there exists an h ( x ) such that G [ h ( x )] < 0 .
Revised: (7) is equivalent to requiring that there exists an h ( x ) such that G [ h ( x )] < 0 .

Original: Hence, a sufﬁcient condition for IB β -learnability is that there exists an h ( x ) such that
Revised: Hence, a sufﬁcient condition for IB β -learnability is that there exists an h ( x ) such that

Original: [Equation]
Revised: [Equation]

Original: When h ( x ) = C = const in the entire input space X , Eq.
Revised: When h ( x ) = C = const in the entire input space X , Eq.

Original: (8) becomes:
Revised: (8) becomes:

Original: [Equation]
Revised: [Equation]

Original: which cannot be true.
Revised: which cannot be true.

Original: Therefore, h ( x ) = const cannot satisfy Eq.
Revised: Therefore, h ( x ) = const cannot satisfy Eq.

Original: Rearranging terms and simplifying, and note that (cid:2) (cid:82) dxh ( x ) p ( x ) (cid:3) 2 > 0 due to h ( x ) (cid:54)≡ 0 = const, we have β
Revised: Rearranging terms and simplifying, and note that (cid:2) (cid:82) dxh ( x ) p ( x ) (cid:3) 2 > 0 due to h ( x ) (cid:54)≡ 0 = const, we have β

Original: [Equation]
Revised: [Equation]

Original: For the R.H.S. of Eq.
Revised: For the R.H.S. of Eq.

Original: (9), let us show that it is greater than 0.
Revised: (9), let us show that it is greater than 0.

Original: Using Cauchy-Schwarz inequality: (cid:104) u, u (cid:105)(cid:104) v, v (cid:105) ≥ (cid:104) u, v (cid:105) 2 , and setting u ( x )
Revised: Using Cauchy-Schwarz inequality: (cid:104) u, u (cid:105)(cid:104) v, v (cid:105) ≥ (cid:104) u, v (cid:105) 2 , and setting u ( x )

Original: = h ( x ) (cid:112) p ( x ) , v ( x ) =
Revised: = h ( x ) (cid:112) p ( x ) , v ( x ) =

Original: (cid:112) p ( x ) , and deﬁning the inner product as (cid:104) u, v (cid:105) =
Revised: (cid:112) p ( x ) , and deﬁning the inner product as (cid:104) u, v (cid:105) =

Original: (cid:82) u ( x ) v ( x ) dx .
Revised: (cid:82) u ( x ) v ( x ) dx .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: It attains equality when u ( x ) v ( x )
Revised: It attains equality when u ( x ) v ( x )

Original: = h ( x ) is constant.
Revised: = h ( x ) is constant.

Original: Since h ( x ) cannot be constant, we have that the R.H.S. of Eq.
Revised: Since h ( x ) cannot be constant, we have that the R.H.S. of Eq.

Original: (9) is greater than 0.
Revised: (9) is greater than 0.

Original: For the L.H.S. of Eq.
Revised: For the L.H.S. of Eq.

Original: (9), due to the necessary condition that β > 0 , if
Revised: (9), due to the necessary condition that β > 0 , if

Original: [Equation]
Revised: [Equation]

Original: cannot hold.
Revised: cannot hold.

Original: Then the h ( x ) such that Eq.
Revised: Then the h ( x ) such that Eq.

Original: (9) holds is for those that satisﬁes
Revised: (9) holds is for those that satisﬁes

Original: [Equation]
Revised: [Equation]

Original: i.e.
Revised: i.e.

Original: [Equation]
Revised: [Equation]

Original: We see this constraint contains the requirement that h ( x ) (cid:54)≡ const.
Revised: We see this constraint contains the requirement that h ( x ) (cid:54)≡ const.

Original: Written in the form of expectations, we have
Revised: Written in the form of expectations, we have

Original: [Equation]
Revised: [Equation]

Original: Since the square function is convex, using Jensen’s inequality on the outer expectation on the L.H.S. of Eq.
Revised: Since the square function is convex, using Jensen’s inequality on the outer expectation on the L.H.S. of Eq.

Original: (10), we have
Revised: (10), we have

Original: [Equation]
Revised: [Equation]

Original: The equality holds iff E x ∼ p ( x | y )
Revised: The equality holds iff E x ∼ p ( x | y )

Original: [ h ( x )] is constant w.r.t.
Revised: [ h ( x )] is constant w.r.t.

Original: y , i.e. Y is independent of X .
Revised: y , i.e. Y is independent of X .

Original: Therefore, in order for Eq.
Revised: Therefore, in order for Eq.

Original: (10) to hold, we require that Y is not independent of X .
Revised: (10) to hold, we require that Y is not independent of X .

Original: Using Jensen’s inequality on the innter expectation on the L.H.S. of Eq.
Revised: Using Jensen’s inequality on the innter expectation on the L.H.S. of Eq.

Original: (10), we have
Revised: (10), we have

Original: [Equation]
Revised: [Equation]

Original: The equality holds when h ( x ) is a constant.
Revised: The equality holds when h ( x ) is a constant.

Original: Since we require that h ( x ) is not a constant, we have that the equality cannot be reached.
Revised: Since we require that h ( x ) is not a constant, we have that the equality cannot be reached.

Original: Under the constraint that Y is not independent of X , we can divide both sides of Eq. 8, and obtain the condition: there exists an h ( x ) such that
Revised: Under the constraint that Y is not independent of X , we can divide both sides of Eq. 8, and obtain the condition: there exists an h ( x ) such that

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e.
Revised: e.

Original: [Equation]
Revised: [Equation]

Original: Written in the form of expectations, we have
Revised: Written in the form of expectations, we have

Original: [Equation]
Revised: [Equation]

Original: We can absorb the constraint Eq.
Revised: We can absorb the constraint Eq.

Original: (10) into the above formula
Revised: (10) into the above formula

Original: [Equation]
Revised: [Equation]

Original: and get
Revised: and get

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: which proves the condition of Theorem 5.
Revised: which proves the condition of Theorem 5.

Original: Furthermore, from Eq.
Revised: Furthermore, from Eq.

Original: (11) we have
Revised: (11) we have

Original: [Equation]
Revised: [Equation]

Original: for h ( x ) (cid:54)≡ const, which satisﬁes the necessary condition of β > 1 in Section 2.
Revised: for h ( x ) (cid:54)≡ const, which satisﬁes the necessary condition of β > 1 in Section 2.

Original: Proof of lower bound of slope of the Pareto frontier at the origin:
Revised: Proof of lower bound of slope of the Pareto frontier at the origin:

Original: Now we prove the second statement of Theorem 5.
Revised: Now we prove the second statement of Theorem 5.

Original: Since δI ( X ; Z ) = 0 and δI ( Y ; Z ) = 

Revised: Since δI ( X ; Z ) = 0 and δI ( Y ; Z ) =

Original: according to Lemma 1.1, we have (cid:16) ∆ I ( Y ; Z ) ∆ I ( X ; Z ) (cid:17) − 1 = (cid:16) δ 2 I ( Y ; Z )
Revised: according to Lemma 1.1, we have (cid:16) ∆ I ( Y ; Z ) ∆ I ( X ; Z ) (cid:17) − 1 = (cid:16) δ 2 I ( Y ; Z )

Original: δ 2 I ( X ; Z ) (cid:17) − 1 .
Revised: δ 2 I ( X ; Z ) (cid:17) − 1 .

Original: Substituting into the expression of δ 2 I ( Y ; Z ) and δ 2 I ( X ; Z ) from Lemma 4.1, we have[Equation] p
Revised: Substituting into the expression of δ 2 I ( Y ; Z ) and δ 2 I ( X ; Z ) from Lemma 4.1, we have

Original: [Equation]
Revised: [Equation]

Original: E x ∼ p ( x | y )
Revised: E x ∼ p ( x | y )

Original: [ h ( x )]
Revised: [ h ( x )]

Original: E x ∼ p ( x ) [ h ( x )] (cid:17) 2 (cid:21) −= β 

Revised: E x ∼ p ( x ) [ h ( x )] (cid:17) 2 (cid:21) −= β

Original: [ h ( x )]
Revised: [ h ( x )]

Original: Therefore, (cid:0) inf h ( x ) β 

Revised: Therefore, (cid:0) inf h ( x ) β

Original: [ h ( x )] (cid:1) − 1 gives the largest slope of ∆ I
Revised: [ h ( x )] (cid:1) − 1 gives the largest slope of ∆ I

Original: ( Y ; Z )
Revised: ( Y ; Z )

Original: vs. ∆
Revised: vs. ∆

Original: I ( X ; Z ) for perturbation function of the form h 1 ( z | x ) = h ( x ) h 2 ( z ) satisfying (cid:82) h 2 ( z ) dz = 0 and (cid:82) h 22 ( z ) p ( z ) dz > 0 , which is a lower bound of slope of ∆ I ( Y ; Z ) vs. ∆ I ( X ; Z ) for all possible perturbation function h 1 ( z | x ) .
Revised: I ( X ; Z ) for perturbation function of the form h 1 ( z | x ) = h ( x ) h 2 ( z ) satisfying (cid:82) h 2 ( z ) dz = 0 and (cid:82) h 22 ( z ) p ( z ) dz > 0 , which is a lower bound of slope of ∆ I ( Y ; Z ) vs. ∆ I ( X ; Z ) for all possible perturbation function h 1 ( z | x ) .

Original: The latter is the slope of the Pareto frontier of the I ( Y ; Z ) vs. I ( X ; Z ) curve at the origin.
Revised: The latter is the slope of the Pareto frontier of the I ( Y ; Z ) vs. I ( X ; Z ) curve at the origin.

Original: Inﬂection point for general Z : If we do not assume that Z is at the origin of the information plane, but at some general stationary solution Z ∗ with p ( z | x ) , we deﬁne
Revised: Inﬂection point for general Z : If we do not assume that Z is at the origin of the information plane, but at some general stationary solution Z ∗ with p ( z | x ) , we deﬁne

Original: [Equation]
Revised: [Equation]

Original: which reduces to β 

Revised: which reduces to β

Original: [ h ( x )] when p ( z | x ) = p ( z ) .
Revised: [ h ( x )] when p ( z | x ) = p ( z ) .

Original: When
Revised: When

Original: [Equation]
Revised: [Equation]

Original: It becomes a non-stable solution (non-minimum), and we will have other Z that achieves a better IB β ( X, Y ; Z ) than the current Z ∗ .
Revised: It becomes a non-stable solution (non-minimum), and we will have other Z that achieves a better IB β ( X, Y ; Z ) than the current Z ∗ .

Original: Multiple phase transitions To discuss multiple phase transitions, let us ﬁrst obtain the β (1) for stationary solution for the IB objective.
Revised: Multiple phase transitions To discuss multiple phase transitions, let us ﬁrst obtain the β (1) for stationary solution for the IB objective.

Original: At a stationary solution for IB β [ p ( z | x )] , for valid perturbation h ( z | x ) satisfying (cid:82) dzh ( z | x ) =for any x , we have δ (cid:2) IB β
Revised: At a stationary solution for IB β [ p ( z | x )] , for valid perturbation h ( z | x ) satisfying (cid:82) dzh ( z | x ) =for any x , we have δ (cid:2) IB β

Original: [ p ( z | x )] − (cid:82) dzdxλ ( x ) p ( z | x ) (cid:3) = 

Revised: [ p ( z | x )] − (cid:82) dzdxλ ( x ) p ( z | x ) (cid:3) =

Original: as a constraint optimization with λ ( x ) as Lagrangian multipliers.
Revised: as a constraint optimization with λ ( x ) as Lagrangian multipliers.

Original: Using Eq.
Revised: Using Eq.

Original: (4), we have
Revised: (4), we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: The last equality is due to that the ﬁrst equality is always true for any function
Revised: The last equality is due to that the ﬁrst equality is always true for any function

Original: h ( z | x ) .
Revised: h ( z | x ) .

Original: So we can take out the (cid:82) dxdzh ( z | x ) factor.
Revised: So we can take out the (cid:82) dxdzh ( z | x ) factor.

Original: λ ( x ) is used for normalization of p ( z | x ) .
Revised: λ ( x ) is used for normalization of p ( z | x ) .

Original: Eq.
Revised: Eq.

Original: (13) is equivalent to the result of the self-consistent equation in Tishby et al.
Revised: (13) is equivalent to the result of the self-consistent equation in Tishby et al.

Original: Eq.
Revised: Eq.

Original: (13) and Eq.
Revised: (13) and Eq.

Original: (12) provide us with an ideal tool to study multiple phase transitions.
Revised: (12) provide us with an ideal tool to study multiple phase transitions.

Original: For each β , at the minimization of the IB objective, Eq.
Revised: For each β , at the minimization of the IB objective, Eq.

Original: (13) is satisﬁed by some Z ∗ that is at the Pareto frontier on the I ( Y ; Z ) vs. I ( X ; Z ) plane.
Revised: (13) is satisﬁed by some Z ∗ that is at the Pareto frontier on the I ( Y ; Z ) vs. I ( X ; Z ) plane.

Original: As we increase β , the inf h ( x ) β (2) [ h ( x )] may remain stable for a wide range of β , until β is greater than inf h ( x ) β (2)
Revised: As we increase β , the inf h ( x ) β (2) [ h ( x )] may remain stable for a wide range of β , until β is greater than inf h ( x ) β (2)

Original: [ h ( x )] , at which point we will have a phase transition where suddenly there is a better Z = Z ∗∗ that achieves much lower
Revised: [ h ( x )] , at which point we will have a phase transition where suddenly there is a better Z = Z ∗∗ that achieves much lower

Original: IB β ( X, Y ; Z ) value.
Revised: IB β ( X, Y ; Z ) value.

Original: For example, we can rewrite Eq.
Revised: For example, we can rewrite Eq.

Original: (13) as
Revised: (13) as

Original: [Equation]
Revised: [Equation]

Original: where ˜ λ ( x )
Revised: where ˜ λ ( x )

Original: = λ ( x ) p ( x ) .
Revised: = λ ( x ) p ( x ) .

Original: By substituting into Eq.
Revised: By substituting into Eq.

Original: (12), we may proceed and get useful results.
Revised: (12), we may proceed and get useful results.

Original: G P ROOF OF T HEOREM 2 ( CONFIDENT SUBSET SUFFICIENT CONDITION )
Revised: G P ROOF OF T HEOREM 2 ( CONFIDENT SUBSET SUFFICIENT CONDITION )

Original: Proof.
Revised: Proof.

Original: According to Theorem 5, a sufﬁcient condition for ( X, Y ) to be IB β -learnable is that X and Y are not independent, and
Revised: According to Theorem 5, a sufﬁcient condition for ( X, Y ) to be IB β -learnable is that X and Y are not independent, and

Original: [Equation]
Revised: [Equation]

Original: We can assume a speciﬁc form of h ( x ) , and obtain a (potentially stronger) sufﬁcient condition.
Revised: We can assume a speciﬁc form of h ( x ) , and obtain a (potentially stronger) sufﬁcient condition.

Original: Speciﬁcally, we let
Revised: Speciﬁcally, we let

Original: [Equation]
Revised: [Equation]

Original: ∈ Ω [Equation] 0 , otherwise (16) for certain Ω x ⊂ X .
Revised: Ω x 0 , otherwise (16) for certain Ω x ⊂ X .

Original: Substituting into Eq.
Revised: Substituting into Eq.

Original: (16), we have that a sufﬁcient condition for ( X, Y ) to be IB β -learnable is
Revised: (16), we have that a sufﬁcient condition for ( X, Y ) to be IB β -learnable is

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where p (Ω x ) =
Revised: where p (Ω x ) =

Original: (cid:82) x ∈ Ω x p ( x ) dx .
Revised: (cid:82) x ∈ Ω x p ( x ) dx .

Original: The denominator of Eq.
Revised: The denominator of Eq.

Original: (17) is
Revised: (17) is

Original: [Equation]
Revised: [Equation]

Original: Using the inequality x − 1 ≥ log x , we have
Revised: Using the inequality x − 1 ≥ log x , we have

Original: [Equation]
Revised: [Equation]

Original: Both equalities hold iff p ( y | Ω x ) ≡ p ( y ) , at which the denominator of Eq.
Revised: Both equalities hold iff p ( y | Ω x ) ≡ p ( y ) , at which the denominator of Eq.

Original: (17) is equal to 0 and the expression inside the inﬁmum diverge, which will not contribute to the inﬁmum.
Revised: (17) is equal to 0 and the expression inside the inﬁmum diverge, which will not contribute to the inﬁmum.

Original: Except this scenario, the denominator is greater than 0.
Revised: Except this scenario, the denominator is greater than 0.

Original: Substituting into Eq.
Revised: Substituting into Eq.

Original: (17), we have that a sufﬁcient condition for ( X, Y ) to be IB β -learnable is
Revised: (17), we have that a sufﬁcient condition for ( X, Y ) to be IB β -learnable is

Original: [Equation]
Revised: [Equation]

Original: Since Ω x is a subset of X , by the deﬁnition of h ( x ) in Eq.
Revised: Since Ω x is a subset of X , by the deﬁnition of h ( x ) in Eq.

Original: (16), h ( x ) is not a constant in the entire X .
Revised: (16), h ( x ) is not a constant in the entire X .

Original: Hence the numerator of Eq.
Revised: Hence the numerator of Eq.

Original: (18) is positive.
Revised: (18) is positive.

Original: Since its denominator is also positive, we can then neglect the “ > 0 ”, and obtain the condition in Theorem 2.
Revised: Since its denominator is also positive, we can then neglect the “ > 0 ”, and obtain the condition in Theorem 2.

Original: Since the h ( x ) used in this theorem is a subset of the h ( x ) used in Theorem 5, the inﬁmum for Eq.
Revised: Since the h ( x ) used in this theorem is a subset of the h ( x ) used in Theorem 5, the inﬁmum for Eq.

Original: (2) is greater than or equal to the inﬁmum in Eq.
Revised: (2) is greater than or equal to the inﬁmum in Eq.

Original: Therefore, according to the second statement of Theorem 5, we have that the (inf Ω x β 0 (Ω x )) − 1 is also a lower bound of the slope for the Pareto frontier of I ( Y ; Z ) vs. I ( X ; Z ) curve.
Revised: Therefore, according to the second statement of Theorem 5, we have that the (inf Ω x β 0 (Ω x )) − 1 is also a lower bound of the slope for the Pareto frontier of I ( Y ; Z ) vs. I ( X ; Z ) curve.

Original: Now we prove that the condition Eq.
Revised: Now we prove that the condition Eq.

Original: (2) is invariant to invertible mappings of X .
Revised: (2) is invariant to invertible mappings of X .

Original: In fact, if X (cid:48) =
Revised: In fact, if X (cid:48) =

Original: g ( X ) is a uniquely invertible map (if X is continuous, g is additionally required to be continuous), let X (cid:48)
Revised: g ( X ) is a uniquely invertible map (if X is continuous, g is additionally required to be continuous), let X (cid:48)

Original: = { g ( x ) | x ∈ Ω x } , and denote g (Ω x ) ≡ { g ( x ) | x ∈ Ω x } for any Ω x ⊂ X , we have p ( g (Ω x )) =
Revised: = { g ( x ) | x ∈ Ω x } , and denote g (Ω x ) ≡ { g ( x ) | x ∈ Ω x } for any Ω x ⊂ X , we have p ( g (Ω x )) =

Original: p (Ω x ) , and p ( y | g (Ω x ))
Revised: p (Ω x ) , and p ( y | g (Ω x ))

Original: =
Revised: =

Original: p ( y | Ω x ) .
Revised: p ( y | Ω x ) .

Original: Then for dataset ( X, Y ) , let Ω (cid:48) x
Revised: Then for dataset ( X, Y ) , let Ω (cid:48) x

Original: = g (Ω x ) , we have
Revised: = g (Ω x ) , we have

Original: [Equation]
Revised: [Equation]

Original: Additionally we have X (cid:48)
Revised: Additionally we have X (cid:48)

Original: =
Revised: =

Original: g ( X ) .
Revised: g ( X ) .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: For dataset ( X (cid:48) , Y ) =
Revised: For dataset ( X (cid:48) , Y ) =

Original: ( g ( X ) , Y ) , applying Theorem 2 we have that a sufﬁcient condition for it to be IB β -learnable is
Revised: ( g ( X ) , Y ) , applying Theorem 2 we have that a sufﬁcient condition for it to be IB β -learnable is

Original: [Equation]
Revised: [Equation]

Original: where the equality is due to Eq.
Revised: where the equality is due to Eq.

Original: Comparing with the condition for IB β -learnability for ( X, Y ) (Eq.
Revised: Comparing with the condition for IB β -learnability for ( X, Y ) (Eq.

Original: , we see that they are the same.
Revised: , we see that they are the same.

Original: Therefore, the condition given by Theorem 2 is invariant to invertible mapping of X .
Revised: Therefore, the condition given by Theorem 2 is invariant to invertible mapping of X .

Original: H P ROOF OF C OROLLARY 2.1 AND C OROLLARY 2. H. P ROOF OF C OROLLARY 2. 
Revised: H P ROOF OF C OROLLARY 2.1 AND C OROLLARY 2. H. P ROOF OF C OROLLARY 2.

Original: Proof.
Revised: Proof.

Original: We use Theorem 2.
Revised: We use Theorem 2.

Original: Let Ω x contain all elements x whose true class is y ∗ for some certain y ∗ , and 0 otherwise.
Revised: Let Ω x contain all elements x whose true class is y ∗ for some certain y ∗ , and 0 otherwise.

Original: Then we obtain a (potentially stronger) sufﬁcient condition.
Revised: Then we obtain a (potentially stronger) sufﬁcient condition.

Original: Since the probability p ( y | y ∗ , x ) = p ( y | y ∗ ) is classconditional, we have
Revised: Since the probability p ( y | y ∗ , x ) = p ( y | y ∗ ) is classconditional, we have

Original: [Equation]
Revised: [Equation]

Original: By requiring β > inf y ∗ 1 p ( y ∗ ) −
Revised: By requiring β > inf y ∗ 1 p ( y ∗ ) −

Original: E y ∼ p ( y | y ∗ ) (cid:2) p
Revised: E y ∼ p ( y | y ∗ ) (cid:2) p

Original: ( y | y ∗ )
Revised: ( y | y ∗ )

Original: p ( y ) − 1 (cid:3) , we obtain a sufﬁcient condition for IB β learnability.
Revised: p ( y ) − 1 (cid:3) , we obtain a sufﬁcient condition for IB β learnability.

Original: H. P ROOF OF C OROLLARY 2. 
Revised: H. P ROOF OF C OROLLARY 2.

Original: Proof.
Revised: Proof.

Original: We again use Theorem 2.
Revised: We again use Theorem 2.

Original: Since Y is a deterministic function of X , let Y =
Revised: Since Y is a deterministic function of X , let Y =

Original: f ( X ) .
Revised: f ( X ) .

Original: Since it is classiﬁcation problem, Y contains at least one value y such that its probability p ( y ) > 0 , we let Ω  contain only x such that f ( x ) =
Revised: Since it is classiﬁcation problem, Y contains at least one value y such that its probability p ( y ) > 0 , we let Ω x contain only x such that f ( x ) =

Original: y .
Revised: y .

Original: Substituting into Eq.
Revised: Substituting into Eq.

Original: (2), we have
Revised: (2), we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, the sufﬁcient condition becomes β > 1 .
Revised: Therefore, the sufﬁcient condition becomes β > 1 .

Original: Furthermore, since a necessary condition for IB β -learnability is β > 1 (Section 2), we have that β > β 

Revised: Furthermore, since a necessary condition for IB β -learnability is β > 1 (Section 2), we have that β > β

Original: = 1 is a necessary and sufﬁcient condition.
Revised: = 1 is a necessary and sufﬁcient condition.

Original: I A DDITIONAL DISCUSSIONS FOR T HEOREM
Revised: I A DDITIONAL DISCUSSIONS FOR T HEOREM

Original: Similarity to information measures.
Revised: Similarity to information measures.

Original: The denominator of Eq.
Revised: The denominator of Eq.

Original: (2) is closely related to mutual information.
Revised: (2) is closely related to mutual information.

Original: Using the inequality x − 1 ≥ log ( x ) for x > 0 , it becomes:
Revised: Using the inequality x − 1 ≥ log ( x ) for x > 0 , it becomes:

Original: [Equation]
Revised: [Equation]

Original: where ˜ I
Revised: where ˜ I

Original: (Ω x ; Y ) is the mutual information “density” at Ω x ⊂ X .
Revised: (Ω x ; Y ) is the mutual information “density” at Ω x ⊂ X .

Original: Of course, this quantity is also D KL [ p ( y | Ω x ) || p
Revised: Of course, this quantity is also D KL [ p ( y | Ω x ) || p

Original: ( y )] , so we know that the denominator of Eq.
Revised: ( y )] , so we know that the denominator of Eq.

Original: (2) is non-negative.
Revised: (2) is non-negative.

Original: Incidentally, E y ∼ p ( y | Ω x ) (cid:2) p ( y | Ω x ) p ( y ) − 1 (cid:3) is the density of “rational mutual information” (Lin & Tegmark (2016)) at Ω x .
Revised: Incidentally, E y ∼ p ( y | Ω x ) (cid:2) p ( y | Ω x ) p ( y ) − 1 (cid:3) is the density of “rational mutual information” (Lin & Tegmark (2016)) at Ω x .

Original: Similarly, the numerator is related to the self-information of Ω x :
Revised: Similarly, the numerator is related to the self-information of Ω x :

Original: [Equation]
Revised: [Equation]

Original: so we can estimate the phase transition as:
Revised: so we can estimate the phase transition as:

Original: [Equation]
Revised: [Equation]

Original: Since Eq.
Revised: Since Eq.

Original: (22) uses upper bounds on both the numerator and the denominator, it does not give us a bound on β 0 .
Revised: (22) uses upper bounds on both the numerator and the denominator, it does not give us a bound on β 0 .

Original: Multiple phase transitions.
Revised: Multiple phase transitions.

Original: Based on this characterization of Ω x , we can hypothesize datasets with multiple learnability phase transitions.
Revised: Based on this characterization of Ω x , we can hypothesize datasets with multiple learnability phase transitions.

Original: Speciﬁcally, consider a region Ω x 0 that is small but “typical”, consists of all elements conﬁdently predicted as y 0 by p ( y | x ) , and where y 0 is the least common class.
Revised: Speciﬁcally, consider a region Ω x 0 that is small but “typical”, consists of all elements conﬁdently predicted as y 0 by p ( y | x ) , and where y 0 is the least common class.

Original: By construction, this Ω x 0 will dominate the inﬁmum in Eq.
Revised: By construction, this Ω x 0 will dominate the inﬁmum in Eq.

Original: (2), resulting in a small value of β 0 .
Revised: (2), resulting in a small value of β 0 .

Original: However, the remaining X − Ω x 0 effectively form a new dataset,
Revised: However, the remaining X − Ω x 0 effectively form a new dataset,

Original: X 1 .
Revised: X 1 .

Original: At exactly β 0 , we may have that the current encoder, p 0 ( z | x ) , has no mutual information with the remaining classes in X 1 ; i.e., I ( Y 1 ; Z 0 )
Revised: At exactly β 0 , we may have that the current encoder, p 0 ( z | x ) , has no mutual information with the remaining classes in X 1 ; i.e., I ( Y 1 ; Z 0 )

Original: = 0 .
Revised: = 0 .

Original: In this case, Deﬁnition 1 applies to p 0 ( z | x ) with respect to I ( X 1 ; Z 1 ) .
Revised: In this case, Deﬁnition 1 applies to p 0 ( z | x ) with respect to I ( X 1 ; Z 1 ) .

Original: We might expect to see that, at β 0 , learning will plateau until we get to some β 1 > β 0 that deﬁnes the phase transition for X 1 .
Revised: We might expect to see that, at β 0 , learning will plateau until we get to some β 1 > β 0 that deﬁnes the phase transition for X 1 .

Original: Clearly this process could repeat many times, with each new dataset X i being distinctly more difﬁcult to learn than
Revised: Clearly this process could repeat many times, with each new dataset X i being distinctly more difﬁcult to learn than

Original: X i − 1 .
Revised: X i − 1 .

Original: The end of Appendix F gives a more detailed analysis on multiple phase transitions.
Revised: The end of Appendix F gives a more detailed analysis on multiple phase transitions.

Original: Estimating model capacity.
Revised: Estimating model capacity.

Original: The observation that a model can’t distinguish between cluster overlap in the data and its own lack of capacity gives an interesting way to use IB-Learnability to measure the capacity of a set of models relative to the task they are being used to solve.
Revised: The observation that a model can’t distinguish between cluster overlap in the data and its own lack of capacity gives an interesting way to use IB-Learnability to measure the capacity of a set of models relative to the task they are being used to solve.

Original: [Figure]
Revised: [Figure]

Original: Learnability and the Information Plane.
Revised: Learnability and the Information Plane.

Original: Many of our results can be interpreted in terms of the geometry of the
Revised: Many of our results can be interpreted in terms of the geometry of the

Original: Pareto frontier illustrated in Fig. 2, which describes the trade-off between increasing I ( Y ; Z ) and decreasing I ( X ; Z ) .
Revised: Pareto frontier illustrated in Fig. 2, which describes the trade-off between increasing I ( Y ; Z ) and decreasing I ( X ; Z ) .

Original: At any point on this frontier that minimizes IB min β ≡ min I ( X ; Z ) − βI ( Y ; Z ) , the frontier will have slope β − 1 if it is differentiable.
Revised: At any point on this frontier that minimizes IB min β ≡ min I ( X ; Z ) − βI ( Y ; Z ) , the frontier will have slope β − 1 if it is differentiable.

Original: If the frontier is also concave (has negative second derivative), then this slope β − 1 will take its maximum β − 1 0 at the origin, which implies IB β -Learnability for β > β 0 , so that the threshold for IB β -Learnability is simply the inverse slope of the frontier at the origin.
Revised: If the frontier is also concave (has negative second derivative), then this slope β − 1 will take its maximum β − 1 0 at the origin, which implies IB β -Learnability for β > β 0 , so that the threshold for IB β -Learnability is simply the inverse slope of the frontier at the origin.

Original: More generally, as long as the Pareto frontier is differentiable, the threshold for IB β -learnability is the inverse of its maximum slope.
Revised: More generally, as long as the Pareto frontier is differentiable, the threshold for IB β -learnability is the inverse of its maximum slope.

Original: Indeed, Theorem 2 gives lower bounds of the slope of the Pareto frontier at the origin.
Revised: Indeed, Theorem 2 gives lower bounds of the slope of the Pareto frontier at the origin.

Original: The shaded region I ( Y ; Z ) > I ( X ; Z ) is impossible by the data processing inequality, since Z depends on Y only via
Revised: The shaded region I ( Y ; Z ) > I ( X ; Z ) is impossible by the data processing inequality, since Z depends on Y only via

Original: X .
Revised: X .

Original: Indeed, it is easy to see that if the Pareto Frontier is differentiable, its slope must always lie in the unit interval [0 , 1] .
Revised: Indeed, it is easy to see that if the Pareto Frontier is differentiable, its slope must always lie in the unit interval [0 , 1] .

Original: Moreover, both the horizontal and vertical axes are bounded, since 0 ≤ I ( X ; Z ) ≤ H ( X ) and 0 ≤
Revised: Moreover, both the horizontal and vertical axes are bounded, since 0 ≤ I ( X ; Z ) ≤ H ( X ) and 0 ≤

Original: I ( Y ; Z ) ≤ H ( Y ) , so if the Pareto Frontier is differentiable, then its slope lies in some compact interval [ β − 1 1 , β − 1 0 ] , where 1 ≤ β 0 ≤ β 1 .
Revised: I ( Y ; Z ) ≤ H ( Y ) , so if the Pareto Frontier is differentiable, then its slope lies in some compact interval [ β − 1 1 , β − 1 0 ] , where 1 ≤ β 0 ≤ β 1 .

Original: This means that we lack IB β -learnability for β < β 0 , which makes the origin the optimal point.
Revised: This means that we lack IB β -learnability for β < β 0 , which makes the origin the optimal point.

Original: If the frontier is convex, then we achieve optimality at the upper right endpoint if β > β 1 , otherwise on the frontier at the location between the two endpoints where the frontier slope is β − 1 .
Revised: If the frontier is convex, then we achieve optimality at the upper right endpoint if β > β 1 , otherwise on the frontier at the location between the two endpoints where the frontier slope is β − 1 .

Original: Learnability and contraction coefﬁcient If we regard the true mapping from X to Y as a channel with transition kernel P Y | X , we can deﬁne contraction coefﬁcient η KL ( P 
Revised: Learnability and contraction coefﬁcient If we regard the true mapping from X to Y as a channel with transition kernel P Y | X , we can deﬁne contraction coefﬁcient η KL ( P Y | X ) = sup Q ; P :0 < D KL ( P

Original: [Equation]
Revised: [Equation]

Original: |
Revised: ) < ∞ D KL

Original: [Equation]
Revised: [Equation]

Original: = [Equation] [Equation]
Revised: Y | X ◦ P

Original: [Equation] | [Equation] ) D KL ( P || Q ) (Polyanskiy & Wu (2017)) as a measure of how much it keeps the two distributions P and Q intact (as opposed to being drawn nearer measured by KL-divergence) after pushing forward through the channel.
Revised: || P Y | X ◦ Q ) D KL ( P || Q ) (Polyanskiy & Wu (2017)) as a measure of how much it keeps the two distributions P and Q intact (as opposed to being drawn nearer measured by KL-divergence) after pushing forward through the channel.

Original: By Polyanskiy & Wu (2017) we have η KL ( P Y | X ) =
Revised: By Polyanskiy & Wu (2017) we have η KL ( P Y | X ) =

Original: sup Z I ( Y ; Z ) I (
Revised: sup Z I ( Y ; Z ) I (

Original: X ; Z ) , which is the slope β − 1of the Pareto frontier at the origin.
Revised: X ; Z ) , which is the slope β − 1of the Pareto frontier at the origin.

Original: By the analysis of the information plane above, we have that as long as β − 1 < η KL
Revised: By the analysis of the information plane above, we have that as long as β − 1 < η KL

Original: ( P Y | X ) =
Revised: ( P Y | X ) =

Original: β − 1 0 , it is IB β -learnable.
Revised: β − 1 0 , it is IB β -learnable.

Original: Furthermore, with Theorem 5, we have (cid:0) inf h ( x ) β 

Revised: Furthermore, with Theorem 5, we have (cid:0) inf h ( x ) β

Original: [ h ( x )] (cid:1) − 1 ≤ sup h ( z | x ) ∆ I
Revised: [ h ( x )] (cid:1) − 1 ≤ sup h ( z | x ) ∆ I

Original: ( Y ; Z ) ∆
Revised: ( Y ; Z ) ∆

Original: I ( X ; Z ) (cid:12)(cid:12) p ( z | x )= p ( z ) ≤ sup Z I ( Y ; Z ) I ( X ; Z ) , therefore
Revised: I ( X ; Z ) (cid:12)(cid:12) p ( z | x )= p ( z ) ≤ sup Z I ( Y ; Z ) I ( X ; Z ) , therefore

Original: (cid:0) inf h ( x ) β 

Revised: (cid:0) inf h ( x ) β

Original: [ h ( x )] (cid:1) − 1 ≤ η KL ( P Y | X ) .
Revised: [ h ( x )] (cid:1) − 1 ≤ η KL ( P Y | X ) .

Original: Theorem 5 hence also provides a lower bound for the contraction coefﬁcient η KL ( P Y | X ) .
Revised: Theorem 5 hence also provides a lower bound for the contraction coefﬁcient η KL ( P Y | X ) .

Original: Similarly for Theorem 2.
Revised: Similarly for Theorem 2.

Original: J A LGORITHM FOR ESTIMATING THE IB-L EARNABILITY CONDITION
Revised: J A LGORITHM FOR ESTIMATING THE IB-L EARNABILITY CONDITION

Original: In Alg.
Revised: In Alg.

Original: 1 we present a detailed algorithm for estimating β 0 .
Revised: 1 we present a detailed algorithm for estimating β 0 .

Original: Require : Dataset D = { ( x i , y i ) } , i = 1 , 2 , ...
Revised: Require : Dataset D = { ( x i , y i ) } , i = 1 , 2 , ...

Original: N .
Revised: N .

Original: The number of classes is C .
Revised: The number of classes is C .

Original: Require ε : tolerance for estimating β
Revised: Require ε : tolerance for estimating β

Original: Learn a maximum likelihood model p θ ( y | x ) using the dataset D .
Revised: Learn a maximum likelihood model p θ ( y | x ) using the dataset D .

Original: Subroutine Get β
Revised: Subroutine Get β

Original: ( P y | x , p y , Ω) 
Revised: ( P y | x , p y , Ω) of

Original: ← y | x y | x
Revised: Ω

Original: p y
Revised: P y

Original: Ω j ← 1 n i ∈ Ω ( P y | x) ij , j= 1 , 2 , ...
Revised: x j

Original: ← Nn −
Revised: [Table]

Original: K S ETTINGS FOR THE EXPERIMENTS
Revised: K S ETTINGS FOR THE EXPERIMENTS

Original: We use the Variational Information Bottleneck (VIB) objective by Alemi et al.
Revised: We use the Variational Information Bottleneck (VIB) objective by Alemi et al.

Original: For the synthetic experiment, the latent Z has dimension of 2.
Revised: For the synthetic experiment, the latent Z has dimension of 2.

Original: The encoder is a neural net with 2 hidden layers, each of which has 128 neurons with ReLU activation.
Revised: The encoder is a neural net with 2 hidden layers, each of which has 128 neurons with ReLU activation.

Original: The last layer has linear activation and 4 output neurons, with the ﬁrst two parameterizes the mean of a Gaussian and the last two parameterizes the log variance of the Gaussian.
Revised: The last layer has linear activation and 4 output neurons, with the ﬁrst two parameterizes the mean of a Gaussian and the last two parameterizes the log variance of the Gaussian.

Original: The decoder is a neural net with 1 hidden layers with 128 neurons and ReLU activation.
Revised: The decoder is a neural net with 1 hidden layers with 128 neurons and ReLU activation.

Original: Its last layer has linear activation and outputs the logit for the class labels.
Revised: Its last layer has linear activation and outputs the logit for the class labels.

Original: It uses a mixture of Gaussian prior with 500 components (for the experiment with class overlap, 256 components), each of which is a 2D Gaussian with learnable mean and log variance, and the weights for the components are also learnable.
Revised: It uses a mixture of Gaussian prior with 500 components (for the experiment with class overlap, 256 components), each of which is a 2D Gaussian with learnable mean and log variance, and the weights for the components are also learnable.

Original: For the MNIST experiment, the architecture is mostly the same, except the following: (1) for Z , we let it have dimension of 256.
Revised: For the MNIST experiment, the architecture is mostly the same, except the following: (1) for Z , we let it have dimension of 256.

Original: For the prior, we use standard Gaussian with diagonal covariance matrix.
Revised: For the prior, we use standard Gaussian with diagonal covariance matrix.

Original: For all experiments, we use Adam (Kingma & Ba (2014)) optimizer with default parameters.
Revised: For all experiments, we use Adam (Kingma & Ba (2014)) optimizer with default parameters.

Original: We do not add any regularization.
Revised: We do not add any regularization.

Original: We use learning rate of 10 − 4 and have a learning rate decay of 1 1+0 .
Revised: We use learning rate of 10 − 4 and have a learning rate decay of 1 1+0 .

Original: 01 × epoch .
Revised: 01 × epoch .

Original: We train in totalepochs with batch size of 500.
Revised: We train in totalepochs with batch size of 500.

Original: All experiments has train-test split of 5:1, and we report the accuracy on the test set, w.r.t.
Revised: All experiments has train-test split of 5:1, and we report the accuracy on the test set, w.r.t.

Original: the true labels.
Revised: the true labels.

Original: For estimation of β 0 , exp in Fig.
Revised: For estimation of β 0 , exp in Fig.

Original: 1, in the accuracy vs. β i curve, we take the mean and standard deviation of the accuracy for the lowest 5 β i values, denoting as µ β , σ β .
Revised: 1, in the accuracy vs. β i curve, we take the mean and standard deviation of the accuracy for the lowest 5 β i values, denoting as µ β , σ β .

Original: When β i is greater than µ β + 3 σ β , we regard it as learning a non-trivial representation, and take the average of β i and β i − 1 as the experimentally estimated onset of learning.
Revised: When β i is greater than µ β + 3 σ β , we regard it as learning a non-trivial representation, and take the average of β i and β i − 1 as the experimentally estimated onset of learning.

Original: We also inspect manually and conﬁrm that it is consistent with human intuition.
Revised: We also inspect manually and conﬁrm that it is consistent with human intuition.

Original: For the estimating β 0 , estimated using Alg. 1, at step 7 we use the following discrete search algorithm.
Revised: For the estimating β 0 , estimated using Alg. 1, at step 7 we use the following discrete search algorithm.

Original: We gradually narrow down the range [ a, b ] of i upper , starting from [1 , N ] .
Revised: We gradually narrow down the range [ a, b ] of i upper , starting from [1 , N ] .

Original: At each iteration, we set a tentative new range [ a (cid:48) , b (cid:48) ] , where a (cid:48)
Revised: At each iteration, we set a tentative new range [ a (cid:48) , b (cid:48) ] , where a (cid:48)

Original: = 0 .
Revised: = 0 .

Original: 8 a + 0 .
Revised: 8 a + 0 .

Original: 2 b , b (cid:48)
Revised: 2 b , b (cid:48)

Original: = 0 .
Revised: = 0 .

Original: 2 a + 0 .
Revised: 2 a + 0 .

Original: 8 b , and calculate ˜ β 0 ,a (cid:48)
Revised: 8 b , and calculate ˜ β 0 ,a (cid:48)

Original: =
Revised: =

Original: Get β
Revised: Get β

Original: ( P y | x , p y , Ω a (cid:48) ) , ˜ β 0 ,b (cid:48)
Revised: ( P y | x , p y , Ω a (cid:48) ) , ˜ β 0 ,b (cid:48)

Original: =
Revised: =

Original: Get β
Revised: Get β

Original: ( P y | x , p y , Ω b (cid:48) )
Revised: ( P y | x , p y , Ω b (cid:48) )

Original: where Ω a (cid:48)
Revised: where Ω a (cid:48)

Original: = { 1 , 2 , ...a (cid:48) } and Ω b (cid:48)
Revised: = { 1 , 2 , ...a (cid:48) } and Ω b (cid:48)

Original: = { 1 , 2 , ...
Revised: = { 1 , 2 , ...

Original: b (cid:48) } .
Revised: b (cid:48) } .

Original: If ˜ β 0 ,a (cid:48) < ˜ β 0 ,a , let a ← a (cid:48) .
Revised: If ˜ β 0 ,a (cid:48) < ˜ β 0 ,a , let a ← a (cid:48) .

Original: If ˜ β 0 ,b (cid:48) < ˜ β 0 ,b , let b ← b (cid:48) .
Revised: If ˜ β 0 ,b (cid:48) < ˜ β 0 ,b , let b ← b (cid:48) .

Original: In other words, we narrow down the range of i upper if we ﬁnd that the Ω given by the left or right boundary gives a lower ˜ β
Revised: In other words, we narrow down the range of i upper if we ﬁnd that the Ω given by the left or right boundary gives a lower ˜ β

Original: [Table]
Revised: [Table]

Original: value.
Revised: value.

Original: The process stops when both ˜ β 0 ,a (cid:48) and ˜ β 0 ,b (cid:48) stops improving (which we ﬁnd always happens when b (cid:48)
Revised: The process stops when both ˜ β 0 ,a (cid:48) and ˜ β 0 ,b (cid:48) stops improving (which we ﬁnd always happens when b (cid:48)

Original: = a (cid:48) +1 ), and we return the smaller of the ﬁnal ˜ β 0 ,a (cid:48) and ˜ β 0 ,b (cid:48) as ˜ β 0 .
Revised: = a (cid:48) +1 ), and we return the smaller of the ﬁnal ˜ β 0 ,a (cid:48) and ˜ β 0 ,b (cid:48) as ˜ β 0 .

Original: K. CIFAR10 D ETAILS
Revised: K. CIFAR10 D ETAILS

Original: We trained a deterministic 28x10 wide resnet (He et al., 2016; Zagoruyko & Komodakis, 2016), using the open source implementation from Cubuk et al.
Revised: We trained a deterministic 28x10 wide resnet (He et al., 2016; Zagoruyko & Komodakis, 2016), using the open source implementation from Cubuk et al.

Original: However, we extended the ﬁnal 10 dimensional logits of that model through another 3 layer MLP classiﬁer, in order to keep the inference network architecture identical between this model and the
Revised: However, we extended the ﬁnal 10 dimensional logits of that model through another 3 layer MLP classiﬁer, in order to keep the inference network architecture identical between this model and the

Original: VIB models we describe below.
Revised: VIB models we describe below.

Original: During training, we dynamically added label noise according to the class confusion matrix in Tab.
Revised: During training, we dynamically added label noise according to the class confusion matrix in Tab.

Original: K.1.
Revised: K.1.

Original: The mean label noise averaged across the 10 classes is 20%.
Revised: The mean label noise averaged across the 10 classes is 20%.

Original: After that model had converged, we used it to estimate β 0 with Alg. 1.
Revised: After that model had converged, we used it to estimate β 0 with Alg. 1.

Original: Even with 20% label noise, β 0 was estimated to be 1.0483.
Revised: Even with 20% label noise, β 0 was estimated to be 1.0483.

Original: We then trained 73 different VIB models using the same 28x10 wide resnet architecture for the encoder, parameterizing the mean of a 10-dimensional unit variance Gaussian.
Revised: We then trained 73 different VIB models using the same 28x10 wide resnet architecture for the encoder, parameterizing the mean of a 10-dimensional unit variance Gaussian.

Original: Samples from the encoder distribution were fed to the samelayer MLP classiﬁer architecture used in the deterministic model.
Revised: Samples from the encoder distribution were fed to the samelayer MLP classiﬁer architecture used in the deterministic model.

Original: The marginal distributions were mixtures offully covariate 10-dimensional Gaussians, all parameters of which are trained.
Revised: The marginal distributions were mixtures offully covariate 10-dimensional Gaussians, all parameters of which are trained.

Original: The VIB models had β ranging from 1.02 to 2.0 by steps of 0.02, plus an extra set ranging from 1.04 to 1.06 by steps of 0.001 to ensure we captured the empirical β 0 with high precision.
Revised: The VIB models had β ranging from 1.02 to 2.0 by steps of 0.02, plus an extra set ranging from 1.04 to 1.06 by steps of 0.001 to ensure we captured the empirical β 0 with high precision.

Original: However, this particular VIB architecture does not start learning until β > 2 . 5 , so none of these models would train as described.
Revised: However, this particular VIB architecture does not start learning until β > 2 . 5 , so none of these models would train as described.

Original: 4 Instead, we started them all at β = 100 , and annealed β down to the corresponding target over 10, training gradient steps.
Revised: 4 Instead, we started them all at β = 100 , and annealed β down to the corresponding target over 10, training gradient steps.

Original: The models continued to train for another 200,000 gradient steps after that.
Revised: The models continued to train for another 200,000 gradient steps after that.

Original: In all cases, the models converged to essentially their ﬁnal accuracy within 20,000 additional gradient steps after annealing was completed.
Revised: In all cases, the models converged to essentially their ﬁnal accuracy within 20,000 additional gradient steps after annealing was completed.

Original: They were stable over the remaining ∼ 180 , 000 gradient steps.
Revised: They were stable over the remaining ∼ 180 , 000 gradient steps.