Original: L OSSLESS

Original: C

Original:  OMPRESSION  G ENERALIZATION IN O VERPARAMETERIZED M 
Revised: L OSSLESS C OMPRESSION & G ENERALIZATION IN O VERPARAMETERIZED M ODELS : T HE C ASE OF B OOSTING

Original: ODELS :

Original: T

Original: HE

Original: C ASE

Original: OF

Original: B OOSTING

Original: Anonymous authors Paper under double-blind review A BSTRACT Successful learning algorithms like DNNs, kernel methods or ensemble learning methods, have been known to produce models that exhibit good generalization despite being drawn from overparameterized model families.
Revised: Successful learning algorithms like DNNs, kernel methods or ensemble learning methods, have been known to produce models that exhibit good generalization despite being drawn from overparameterized model families.

Original: This observation has put in question the convex relationship between model complexity and generalization.
Revised: This observation has put in question the convex relationship between model complexity and generalization.

Original: We instead propose rethinking the relevant notion of model complexity for the purposes of assessing the complexity of models trained on a given dataset.
Revised: We instead propose rethinking the relevant notion of model complexity for the purposes of assessing the complexity of models trained on a given dataset.

Original: Borrowing from information theory, we identify the optimal model one can train on a given dataset as one achieving its lossless maximal compression .
Revised: Borrowing from information theory, we identify the optimal model one can train on a given dataset as one achieving its lossless maximal compression .

Original: In the noiseless dataset setting, it can be shown that such a model coincides with an average margin maximizer of the training data.
Revised: In the noiseless dataset setting, it can be shown that such a model coincides with an average margin maximizer of the training data.

Original: Experimental results on gradient boosting conﬁrm our observations and show that the minimal generalization error is attained in expectation by models achieving lossless maximal compression of the training data.
Revised: Experimental results on gradient boosting conﬁrm our observations and show that the minimal generalization error is attained in expectation by models achieving lossless maximal compression of the training data.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recent theoretical and empirical results have demonstrated that contrary to the traditionally-held belief that overparameterized models 1 are likely to overﬁt, in practice, models produced by contemporary learning algorithms like DNNs, kernel methods or ensemble learning methods, can exhibit good generalization despite being drawn from overparameterized model families Wyner et al.
Revised: Recent theoretical and empirical results have demonstrated that contrary to the traditionally-held belief that overparameterized models 1 are likely to overﬁt, in practice, models produced by contemporary learning algorithms like DNNs, kernel methods or ensemble learning methods, can exhibit good generalization despite being drawn from overparameterized model families Wyner et al.

Original: Belkin et al.
Revised: Belkin et al.

Original: ; Hastie et al.
Revised: ; Hastie et al.

Original: , even without the use of explicit regularization B ¨ uhlmann & Hothorn (2007); Zhang et al.
Revised: , even without the use of explicit regularization B ¨ uhlmann & Hothorn (2007); Zhang et al.

Original: ; Kawaguchi et al.
Revised: ; Kawaguchi et al.

Original: The related “ double descent ” behavior of generalization errors Belkin et al.
Revised: The related “ double descent ” behavior of generalization errors Belkin et al.

Original: ; Loog et al.
Revised: ; Loog et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: (2020) has revealed situations in which increasing model complexity, causes generalization performance to ﬁrst deteriorate and then improve (even to do so repeatedly), prompting us to rethink our understanding of the relation between generalization and model complexity.
Revised: (2020) has revealed situations in which increasing model complexity, causes generalization performance to ﬁrst deteriorate and then improve (even to do so repeatedly), prompting us to rethink our understanding of the relation between generalization and model complexity.

Original: We propose an alternative approach for reconciling these recent ﬁndings to the established theory: by rethinking the notion of model complexity that is useful in such a setting.
Revised: We propose an alternative approach for reconciling these recent ﬁndings to the established theory: by rethinking the notion of model complexity that is useful in such a setting.

Original: The goal is to compare  models trained in a supervised manner on a given training dataset and for doing so we need to move beyond naive parameter counts  of the trained models or worst-case measures of the richness of their respective model family, like the VC-dimension Vapnik & Chervonenkis (2015) or the Rademacher complexity Bartlett & Mendelson (2002).
Revised: The goal is to compare the potential of overﬁtting of models trained in a supervised manner on a given training dataset and for doing so we need to move beyond naive parameter counts 2 of the trained models or worstcase measures of the richness of their respective model family 3 , like the VC-dimension Vapnik & Chervonenkis (2015) or the Rademacher complexity Bartlett & Mendelson (2002).

Original: Taking an information-theoretic perspective to learning Tishby et al.
Revised: Taking an information-theoretic perspective to learning, inspired by the information bottleneck principle Tishby et al.

Original: ; Tishby & Zaslavsky (2015); Shwartz-Ziv & Tishby (2017), we treat the features & targets of the training data, as well as the outputs of the trained model as random variables.
Revised: ; Tishby & Zaslavsky (2015); Shwartz-Ziv & Tishby (2017), we treat the features & targets of the training data, as well as the outputs of the trained model as random 1 Models whose degrees of freedom exceed the number of training datapoints.

Revised: Although sparse models tend to be more resistant to overﬁtting, one can easily come up with counterex- amples.

Revised: Sparsity can make models more robust (in the sense that their output will not change much for small perturbations of their input) but it is neither a necessary, nor a sufﬁcient condition for robustness.

Revised: Successful learning algorithms should, after all, have inductive biases that steer them towards models that are robust, regardless of the capacity of the underlying model family.

Original:  We then identify the ideal model as one that achieves lossless maximal compression (LMC) of the training dataset, i.e. extracts from the features all the useful information for predicting the target and no more.
Revised: variables. We then identify the ideal model as one that achieves lossless maximal compression (LMC) of the training dataset, i.e. extracts from the features all the useful information for predicting the target and no more.

Original: We show that in the noiseless binary classiﬁcation setting, LMC models are equivalent to average margin maximizers , models known to exhibit good generalization guarantees Vapnik (1982); Schapire et al.
Revised: We show that in the noiseless binary classiﬁcation setting, LMC models are equivalent to average margin maximizers , models known to exhibit good generalization guarantees Vapnik (1982); Schapire et al.

Original: (1998); Sokoli´c et al.
Revised: (1998); Sokoli´c et al.

Original:  Dziugaite & Roy (2017); Neyshabur et al.
Revised: (2017); Dziugaite & Roy (2017); Neyshabur et al.

Original: ; Wei et al.
Revised: ; Wei et al.

Original: In the noisy case -when margin maximizers are known to perform poorly Kalai & Servedio (2003);
Revised: In the noisy case -when margin maximizers are known to perform poorly Kalai & Servedio (2003);

Original: Servedio (2003); Bootkrajang  Kab´an (2013) ; Poggio et al.
Revised: Servedio (2003); Bootkrajang & Kab´an (2013); Poggio et al.

Original: (2017)- we show that the aforementioned equivalence collapses, as LMC models also capture label uncertainty while margin maximizersare not even guaranteed to be lossless.
Revised: (2017)- we show that the aforementioned equivalence collapses, as LMC models also capture label uncertainty while margin maximizers are not even guaranteed to be lossless.

Original: We posit that adequate overparameterization guarantees losslessness (in the noiseless case this means interpolation) and the inductive biases of successful machine learning algorithms guide them towards maximal compression, yielding LMC models.
Revised: We posit that adequate overparameterization guarantees losslessness (in the noiseless case this means interpolation) and the inductive biases of successful machine learning algorithms guide them towards maximal compression, yielding LMC models.

Original: This explains their surprisingly good generalization properties Wyner et al.
Revised: This explains their surprisingly good generalization properties Wyner et al.

Original: ; Belkin et al.
Revised: ; Belkin et al.

Original: ; Hastie et al.
Revised: ; Hastie et al.

Original: ; Muthukumar et al.
Revised: ; Muthukumar et al.

Original: ; Bartlett et al.
Revised: ; Bartlett et al.

Original: ; Muthukumar et al.
Revised: ; Muthukumar et al.

Original: We support our observations with theoretical arguments and empirical evidence, using gradient boosting as an example and identify interesting directions for future work.
Revised: We support our observations with theoretical arguments and empirical evidence, using gradient boosting as an example and identify interesting directions for future work.

Original:  A N INFORMATION - THEORETIC VIEW OF MODEL COMPLEXITY
Revised:  A N INFORMATION - THEORETIC VIEW OF MODEL COMPLEXITY

Original: We shall focus on the case of binary classiﬁcation and we will treat the features X & targets Y of the training dataset S , as well as the outputs (scores) of the trained model F as random variables.
Revised: We shall focus on the case of binary classiﬁcation and we will treat the features X & targets Y of the training dataset S , as well as the outputs (scores) of the trained model F as random variables.

Original: Being a deterministic transformation of X , F cannot contain more information 2 than X .
Revised: Being a deterministic transformation of X , F cannot contain more information 4 than X .

Original: So, H ( F | X ) = 

Revised: So, H ( F | X ) =

Original: ⇐⇒ I
Revised: ⇐⇒ I

Original: ( X ; F ) =
Revised: ( X ; F ) =

Original: H ( F ) ≤ H ( X ) .
Revised: H ( F ) ≤ H ( X ) .

Original: We shall now deﬁne the following properties:
Revised: We shall now deﬁne the following properties:

Original: Noiselessness: The dataset S is noiseless if and only if H ( Y | X ) = 0 .
Revised: Noiselessness: The dataset S is noiseless if and only if H ( Y | X ) = 0 .

Original: Otherwise , S is noisy and H ( Y | X ) > 0 .
Revised: Otherwise, S is noisy and H ( Y | X ) > 0 .

Original: A noiseless training dataset S is one in which no datapoints with the same feature vector have different labels
Revised: A noiseless training dataset S is one in which no datapoints with the same feature vector have different labels.

Original: For such a dataset, there exists a model that can achieve zero empirical risk ( training error) , i.e. that can perfectly classify the training data
Revised: For such a dataset, there exists a model that can achieve zero empirical risk (training error), i.e. that can perfectly classify the training data.

Original: In

Original: other words, the features

Original: X

Original:  contain all information to perfectly describe the target Y .
Revised: In other words, the features X , contain all information to perfectly describe the target Y .

Original: Losslessness: The model F is lossless on the dataset S if and only if I ( F ; Y )
Revised: Losslessness: The model F is lossless on the dataset S if and only if I ( F ; Y )

Original: =
Revised: =

Original: I ( Y ; X ) .
Revised: I ( Y ; X ) .

Original: Otherwise, the model is lossy on S and I ( F ; Y ) <
Revised: Otherwise, the model is lossy on S and I ( F ; Y ) <

Original: I ( Y ; X ) .
Revised: I ( Y ; X ) .

Original: A lossless model F on a dataset S is one that captures all the information in features X that is relevant for describing the target Y 3 .
Revised: A lossless model F on a dataset S is one that captures all the information in features X that is relevant for describing the target Y 5 .

Original: If a model F is lossless on a training set S , its output can be used to describe the target Y with the only source of training error being the irreducible class overlap in the training set.
Revised: If a model F is lossless on a training set S , its output can be used to describe the target Y with the only source of training error being the irreducible class overlap in the training set.

Original: Maximal Compression: The model F is a maximal compressor of the dataset S if and only if
Revised: Maximal Compression: The model F is a maximal compressor of the dataset S if and only if

Original: I ( F ; X ) =
Revised: I ( F ; X ) =

Original: I ( F ; Y ) .
Revised: I ( F ; Y ) .

Original: Otherwise, the model is undercompressed on S and I ( F ; X ) >
Revised: Otherwise, the model is undercompressed on S and I ( F ; X ) >

Original: I ( F ; Y ) .
Revised: I ( F ; Y ) .

Original: A model F that is a maximal compressor of a training dataset S is one that only captures from the features X information relevant for describing the target Y .
Revised: A model F that is a maximal compressor of a training dataset S is one that only captures from the features X information relevant for describing the target Y .

Original: It does not necessarily capture all that information; this special case, merits a deﬁnition of its own given below.
Revised: It does not necessarily capture all that information; this special case, merits a deﬁnition of its own given below.

Original: Lossless Maximal Compression - (LMC): The model F is a lossless maximal compressor (LMC) of the training dataset S if and only if it is lossless on S and a maximal compressor on S .
Revised: Lossless Maximal Compression - (LMC): The model F is a lossless maximal compressor (LMC) of the training dataset S if and only if it is lossless on S and a maximal compressor on S .

Original: A model F that is an LMC of a training dataset S is one that only captures from the features X all the information relevant for describing the target Y 4 .
Revised: A model F that is an LMC of a training dataset S is one that only captures from the features X all the information relevant for describing the target Y 6 .

Original: From an information-theoretic perspective, an
Revised: From an information-theoretic perspective, an

Original: LMC of S is the optimal classiﬁcation model that can be constructed from S .
Revised: LMC of S is the optimal classiﬁcation model that can be constructed from S .

Original: Average Margin Maximization: A model F is an average margin maximizer of a training dataset
Revised: Average Margin Maximization: A model F is an average margin maximizer of a training dataset

Original: S if and only if there exists some invertible transformation g such that g ( F ) =
Revised: S if and only if there exists some invertible transformation g such that g ( F ) =

Original: Y .
Revised: Y .

Original: In other words, an average margin maximizer would assign the same score s + to all positively labelled training examples and the same score s − to all negatively labelled training examples.
Revised: In other words, an average margin maximizer would assign the same score s + to all positively labelled training examples and the same score s − to all negatively labelled training examples.

Original: From the above deﬁnitions the following propositions can easily be derived Nikolaou et al.
Revised: From the above deﬁnitions the following propositions can easily be derived Nikolaou et al.

Original: (I)
Revised: (I)

Original: The model F is an LMC of a training dataset S , if and only if I(F;X)
Revised: The model F is an LMC of a training dataset S , if and only if I(F;X)

Original: = I(F;Y) =
Revised: = I(F;Y) =

Original: I(Y;X).
Revised: I(Y;X).

Original: (II)
Revised: (II)

Original: Any LMC model on a noiseless dataset S is also an average margin maximizer on S and viceversa.
Revised: Any LMC model on a noiseless dataset S is also an average margin maximizer on S and viceversa.

Original: 2 Information-theoretic quantities refer to estimates obtained on the training data (empirical distribution).

Original: Equivalently: the r

Original: (III)
Revised: (III)

Original: In the noisy case, the above equivalence no longer holds.
Revised: In the noisy case, the above equivalence no longer holds.

Original: A lossless model is one that also captures the uncertainty introduced by the ambiguous labelling of a feature vector i.e. P ( Y | X ) . 
Revised: A lossless model is one that also captures the uncertainty introduced by the ambiguous labelling of a feature vector i e. P ( Y | X ) . So

Revised: a

Original: So a lossless model should assign different scores F to examplescorresponding to different P
Revised: lossless model should assign different scores F to examples

Original:  ( Y | X ) , even if theyhave the same label Y .
Revised: corresponding to different P ( Y | X ) , even if they

Revised: have the

Revised: same

Original:  An average margin maximizer would therefore not even necessarily be lossless, let alone a LMC on a noisy dataset.
Revised: label Y . An average margin maximizer, assigning only 2 values s

Revised: +

Revised: to positives s

Revised: − to

Revised: negatives (

Revised: regardless of

Revised: label uncertainty)

Revised: would therefore not even necessarily be

Revised: lossless,

Revised: let alone

Revised: a LMC on a

Revised: noisy

Revised: dataset.

Original:  E MPIRICAL E VIDENCE 3. E XPERIMENTAL S ETUP
Revised:  E MPIRICAL E VIDENCE 3. E XPERIMENTAL S ETUP

Original: Boosting, a method that explicitly maximizes the margins of the training examples, can be shown empirically to also converge to LMC models on noiseless datasets.
Revised: Boosting, a method that explicitly maximizes the margins of the training examples, can be shown empirically to also converge to LMC models on noiseless datasets.

Original: After lossless maximal compression is achieved, so is the minimal generalization error, as estimated by the error on the test set.
Revised: After lossless maximal compression is achieved, so is the minimal generalization error, as estimated by the error on the test set.

Original: To demonstrate this, we plot the trajectory of the boosting ensemble on the entropy-normalized information plane , I ( F ; Y ) /H
Revised: To demonstrate this, we plot the trajectory of the boosting ensemble on the entropy-normalized information plane , I ( F ; Y ) /H

Original: ( Y )
Revised: ( Y )

Original: vs. I ( F ; X ) /H
Revised: vs. I ( F ; X ) /H

Original: ( X ) Tishby & Zaslavsky (2015); Shwartz-Ziv & Tishby (2017).
Revised: ( X ) Tishby & Zaslavsky (2015); Shwartz-Ziv & Tishby (2017).

Original: For each boosting round t , F = F t denotes the random variable of which the ensemble’s outputs are realizations 5 .
Revised: For each boosting round t , F = F t denotes the random variable of which the ensemble’s outputs are realizations.

Revised: The boosting ensemble consisted of a maximum of T = 100 decision trees (i.e. rounds of boosting) of maximal depth 6 .

Revised: No shrinkage of the updates or subsampling of the examples was performed (both are techniques to counter overﬁtting), and the exponential loss function was used (i.e. the loss minimized by AdaBoost).

Original: 3. E XPERIMENTAL R ESULTS
Revised: We performed no hyperparameter optimization. Plotting trajectories on the information plane follows Tishby & Zaslavsky (2015); Shwartz-Ziv & Tishby (2017). All information-theoretic quantities were estimated on the training data by ﬁrst discretizing the features & model outputs in b = 100 equal-sized bins 7 , then using maximum likelihood estimators. The joint r.v.

Revised: X was then constructed by the discretized features X 1 , X 2 , . . .

Revised: , X d as X = (cid:80) di

Revised: =1 X i b

Revised: i − 1 .

Revised: We plot average results across 100 runs with different train-test splits ( 50% – 50% ) on the same original data.

Revised: We also visualize the trajectories obtained by some random individual runs to showcase that although they can vary signiﬁcantly from one another, they all follow the same general pattern 8 .

Revised: 3. E XPERIMENTAL R ESULTS

Original: In Figure 1 we present some example trajectories ona noiseless datasets.
Revised: In Figure 1 we present some example trajectories on noiseless datasets.

Original: Below follows a summary of our observations:
Revised: Below follows a summary of our observations:

Original: Boosting leads to lossless maximal compression: The boosting ensemble traces a trajectory on the information plane that leads to the LMC point and once it reaches it in never escapes.
Revised: Boosting leads to lossless maximal compression: The boosting ensemble traces a trajectory on the information plane that leads to the LMC point and once it reaches it in never escapes.

Original: Lossless maximal compression coincides with margin maximization: The image on the information plane of the models that minimize the margin coincides with the LMC point.
Revised: Lossless maximal compression coincides with margin maximization: The image on the information plane of the models that minimize the margin coincides with the LMC point.

Original: Lossless maximal compression coincides with maximal generalization: The point of the ensemble’s trajectory corresponding to the minimal test error coincides –on average– with the LMC point on the information plane (and so does the margin maximization point).
Revised: Lossless maximal compression coincides with maximal generalization: The point of the ensemble’s trajectory corresponding to the minimal test error coincides –on average– with the LMC point on the information plane (and so does the margin maximization point).

Original: Average trajectory shape: After the 
Revised: Average trajectory shape: After the training error is minimized, the test error can be further

Original: training error is minimized , the test
Revised: [Table]

Original: error can be further decreased by
Revised: [Figure]

Original:  training for more rounds –a known result in boosting, explained via margin theory Schapire et al.
Revised: decreased by training for more rounds –a known result in boosting, explained via margin theory Schapire et al.

Original: Here we give an information-theoretic interpretation.
Revised: Here we give an information-theoretic interpretation.

Original: Training until training error minimization, amounts to achieving losslessness.
Revised: Training until training error minimization, amounts to achieving losslessness.

Original: Subsequent rounds result in travelling along the line of maximal I ( F ; X ) on the information plane, towards the LMC point.
Revised: Subsequent rounds result in travelling along the line of maximal I ( F ; X ) on the information plane, towards the LMC point.

Original: This compresses the model (relieves it of remaining information from X irrelevant for predicting
Revised: This compresses the model (relieves it of remaining information from X irrelevant for predicting

Original: Y ), decreasing its effective complexity 6 .
Revised: Y ), decreasing its effective complexity 9 .

Original: Training in boosting consists of 2 (typically distinct) phases: A similar behaviour was observed in Shwartz-Ziv & Tishby (2017) for the trajectories of the representations learned by DNNs.
Revised: Training in boosting consists of 2 (typically distinct) phases: A similar behaviour was observed in Shwartz-Ziv & Tishby (2017) for the trajectories of the representations learned by DNNs.

Original: Following the terminology of Shwartz-Ziv & Tishby (2017), these are the empirical risk minimization (ERM) phase , when I ( F ; Y ) increases (the model better ﬁts the training data) but typically so does
Revised: Following the terminology of Shwartz-Ziv & Tishby (2017), these are the empirical risk minimization (ERM) phase , when I ( F ; Y ) increases (the model better ﬁts the training data) but typically so does

Original: I ( F ; X ) (the model uses more information from X ) and the compression phase , when I ( F ; X ) decreases (the model uses increasingly less information from X , reducing its effective complexity), without decreasing I ( F ; Y ) .
Revised: I ( F ; X ) (the model uses more information from X ) and the compression phase , when I ( F ; X ) decreases (the model uses increasingly less information from X , reducing its effective complexity), without decreasing I ( F ; Y ) .

Original: The ERM phase is usually much shorter than the compression phase, as is the case with DNNs Shwartz-Ziv & Tishby (2017).
Revised: The ERM phase is usually much shorter than the compression phase, as is the case with DNNs Shwartz-Ziv & Tishby (2017).

Original: Early stopping does not improve generalization in gradient boosting: As long as losslessness can be achieved, additional boosting rounds do not hurt generalization.
Revised: Early stopping does not improve generalization in gradient boosting: As long as losslessness can be achieved, additional boosting rounds do not hurt generalization.

Original: Once the model reaches the LMC point on the information plane, it never escapes it.
Revised: Once the model reaches the LMC point on the information plane, it never escapes it.

Original: This suggests that early stopping with boosting is unnecessary for improving generalization.
Revised: This suggests that early stopping with boosting is unnecessary for improving generalization.

Original: This result agrees with recent observations in boosting Wyner et al.
Revised: This result agrees with recent observations in boosting Wyner et al.

Original: (2015), DNNs Belkin et al.
Revised: (2015), DNNs Belkin et al.

Original: and also linear models trained under general margin losses Soudry et al.
Revised: and also linear models trained under general margin losses Soudry et al.

Original: Consistency across datasets, hyperparameter & discretization settings: These observations hold across different datasets, hyperparameter settings and entropy estimation choices.
Revised: Consistency across datasets, hyperparameter & discretization settings: These observations hold across different datasets, hyperparameter settings and entropy estimation choices.

Original: [Table] [Figure] Margin maximization as a built-in regularization mechanism: Regularization methods like subsampling or shrinkage are not the main reason why boosting regularizes.
Revised: Margin maximization as a built-in regularization mechanism: Regularization methods like subsampling or shrinkage are not the main reason why boosting regularizes.

Original: Their contribution is small compared to the algorithm’s built-in regularization mechanism: margin maximization, which as we saw amounts to lossless maximal compression of the training dataset.
Revised: Their contribution is small compared to the algorithm’s built-in regularization mechanism: margin maximization, which as we saw amounts to lossless maximal compression of the training dataset.

Original: DNNs have also been observed to perform implicit regularization and additional regularization control (e.g. dropout or batch normalization) not to be the main contributor to their good generalization Zhang etal.
Revised: DNNs have also been observed to perform implicit regularization and additional regularization control ( e .g. dropout or batch normalization) not to be the main contributor to their good generalization Zhang et

Original: Shwartz-Ziv & Tishby (2017); Kawaguchi et al.
Revised: al Shwartz -Ziv & Tishby (2017) Kawaguchi et al

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We characterized from an information theoretic perspective, models trained on a given training set w.r.t.
Revised: We characterized from an information theoretic perspective, models trained on a given training set w.r.t.

Original: the information they capture from it.
Revised: the information they capture from it.

Original: We identiﬁed an ideal model trained on a given dataset as its lossless maximal compressor (LMC): one capturing all the information from the features relevant for predicting the target and no more.
Revised: We identiﬁed an ideal model trained on a given dataset as its lossless maximal compressor (LMC): one capturing all the information from the features relevant for predicting the target and no more.

Original: We established that an LMC is –in the case of noiseless classiﬁcation– equivalent to a margin maximizer of the dataset.
Revised: We established that an LMC is –in the case of noiseless classiﬁcation– equivalent to an average margin maximizer of the dataset.

Original: In the noisy case the above equivalence collapses and  margin maximizers are suboptimal from an information-theoretic point.
Revised: In the noisy case the above equivalence collapses and average margin maximizers are suboptimal from an information-theoretic point.

Original: Our experiments on gradient boosting, demonstrate that indeed, margin maximization amounts to lossless maximal compression on noiseless data.
Revised: Our experiments on gradient boosting, demonstrate that indeed, margin maximization amounts to lossless maximal compression on noiseless data.

Original: The evolution of the model constructed by boosting, traces a trajectory on the information plane that leads to the LMC point which also coincides with the point of margin maximization and the point on average exhibiting the best generalization.
Revised: The evolution of the model constructed by boosting, traces a trajectory on the information plane that leads to the LMC point which also coincides with the point of margin maximization and the point on average exhibiting the best generalization.

Original: This work gives an information-theoretic interpretation of margin maximization and provides us with a principled way to deﬁne model complexity for the purposes of generalization, thus shedding more light on the success of methods like gradient boosting and identifying situations when they would underperform.
Revised: This work gives an information-theoretic interpretation of margin maximization and provides us with a principled way to deﬁne model complexity for the purposes of generalization, thus shedding more light on the success of methods like gradient boosting and identifying situations in which they would underperform.

Original: It also opens various directions for future work.
Revised: It also opens various directions for future work.

Original: For instance, exploring how these concepts can be applied in model selection or to inform learning algorithm design to more efﬁciently traverse the information plane to reach the LMC point.
Revised: For instance, exploring how these concepts can be applied in model selection or to inform learning algorithm design to more efﬁciently traverse the information plane to reach the LMC point.

Original: It would also be of interest to identify the analogue of the LMC in learning tasks other than classiﬁcation, like ranking or regression. 
Revised: It would also be of interest to identify the analogue of the LMC in learning tasks other than classiﬁcation, like ranking or regression. A CKNOWLEDGEMENTS

Revised: This project was partially supported by the EPSRC Doctoral Prize Fellowship at the University of

Revised: Manchester and the EU Horizon 2020 research & innovation programme [grant No 758892, ExoAI].

Revised: N. Nikolaou also graciously acknowledges the support of the NVIDIA Corporation’s GPU grant.