Original: T HE EFFECTIVENESS OF LAYER - BY - LAYER TRAINING USING THE INFORMATION BOTTLENECK PRINCIPLE
Revised: T HE EFFECTIVENESS OF LAYER - BY - LAYER TRAINING USING THE INFORMATION BOTTLENECK PRINCIPLE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The recently proposed information bottleneck (IB) theory of deep nets suggests that during training, each layer attempts to maximize its mutual information (MI) with the target labels (so as to allow good prediction accuracy), while minimizing its MI with the input (leading to effective compression and thus good generalization).
Revised: The recently proposed information bottleneck (IB) theory of deep nets suggests that during training, each layer attempts to maximize its mutual information (MI) with the target labels (so as to allow good prediction accuracy), while minimizing its MI with the input (leading to effective compression and thus good generalization).

Original: To date, evidence of this phenomenon has been indirect and aroused controversy due to theoretical and practical complications.
Revised: To date, evidence of this phenomenon has been indirect and aroused controversy due to theoretical and practical complications.

Original: In particular, it has been pointed out that the MI with the input is theoretically inﬁnite in many cases of interest, and that the MI with the target is fundamentally difﬁcult to estimate in high dimensions.
Revised: In particular, it has been pointed out that the MI with the input is theoretically inﬁnite in many cases of interest, and that the MI with the target is fundamentally difﬁcult to estimate in high dimensions.

Original: As a consequence, the validity of this theory has been questioned.
Revised: As a consequence, the validity of this theory has been questioned.

Original: In this paper, we overcome these obstacles by two means.
Revised: In this paper, we overcome these obstacles by two means.

Original: First, as previously suggested, we replace the MI with the input by a noise-regularized version, which ensures it is ﬁnite.
Revised: First, as previously suggested, we replace the MI with the input by a noise-regularized version, which ensures it is ﬁnite.

Original: As we show, this modiﬁed penalty in fact acts as a form of weight-decay regularization.
Revised: As we show, this modiﬁed penalty in fact acts as a form of weight-decay regularization.

Original: Second, to obtain accurate (noise regularized) MI estimates between an intermediate representation and the input, we incorporate the strong prior-knowledge we have about their relation, into the recently proposed MI estimator of Belghazi et al.
Revised: Second, to obtain accurate (noise regularized) MI estimates between an intermediate representation and the input, we incorporate the strong prior-knowledge we have about their relation, into the recently proposed MI estimator of Belghazi et al.

Original: With this scheme, we are able to stably train each layer independently to explicitly optimize the IB functional.
Revised: With this scheme, we are able to stably train each layer independently to explicitly optimize the IB functional.

Original: Surprisingly, this leads to enhanced prediction accuracy, thus directly validating the
Revised: Surprisingly, this leads to enhanced prediction accuracy, thus directly validating the

Original: IB theory of deep nets for the ﬁrst time.
Revised: IB theory of deep nets for the ﬁrst time.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural nets (DNN) have shown unparalleled success in many ﬁelds, yet the theoretical understanding of these models is lagging behind.
Revised: Deep neural nets (DNN) have shown unparalleled success in many ﬁelds, yet the theoretical understanding of these models is lagging behind.

Original: Among the various attempts to shed light on this ﬁeld,
Revised: Among the various attempts to shed light on this ﬁeld,

Original: Tishby & Zaslavsky (2015) have recently suggested a link between DNN training and the Information Bottleneck (IB) principle of (Tishby et al., 1999).
Revised: Tishby & Zaslavsky (2015) have recently suggested a link between DNN training and the Information Bottleneck (IB) principle of Tishby et al.

Original: Their theory claims that optimal performance is attained if each layer simultaneously attempts to maximize its mutual information (MI) with the target space, while minimizing its MI with the input space.
Revised: Their theory claims that optimal performance is attained if each layer simultaneously attempts to maximize its mutual information (MI) with the target space, while minimizing its MI with the input space.

Original: The logic is that the DNN layers should compress the latent representation so as to allow for good generalization, while retaining only the information essential for target prediction.
Revised: The logic is that the DNN layers should compress the latent representation so as to allow for good generalization, while retaining only the information essential for target prediction.

Original: Moreover, it was hypothesized that training DNNs through minimization of the popular cross-entropy loss, implicitly achieves this goal.
Revised: Moreover, it was hypothesized that training DNNs through minimization of the popular cross-entropy loss, implicitly achieves this goal.

Original: More formally, it was suggested that during training, each DNN layer approaches the maximal value of the IB Langragian
Revised: More formally, it was suggested that during training, each DNN layer approaches the maximal value of the IB Langragian

Original: [Equation]
Revised: [Equation]

Original: where I denotes MI, X and Y are the input and target random variables, L i is the latent representation at the output of the i th hidden layer, and β is a trade-off parameter.
Revised: where I denotes MI, X and Y are the input and target random variables, L i is the latent representation at the output of the i th hidden layer, and β is a trade-off parameter.

Original: This elegant connection between DNN training and the IB principle was later supported by empirical experiments conducted by the original authors (Shwartz-Ziv & Tishby, 2017), as well as by others (Alemi et al., 2018;
Revised: This elegant connection between DNN training and the IB principle was later supported by empirical experiments conducted by Shwartz-Ziv & Tishby (2017), as well as by others (Alemi et al., 2018;

Original: Belghazi et al., 2018; Kolchinsky et al., 2017; Saxe et al., 2017).
Revised: Belghazi et al., 2018; Kolchinsky et al., 2017; Saxe et al., 2018).

Original: These works either (i) showed the effect of optimizing the IB functional in (1) for classiﬁcation tasks, or (ii) analyzed the dynamics of each layer in the information plane (deﬁned by axes
Revised: These works either (i) showed the effect of optimizing the IB functional in (1) for classiﬁcation tasks, or (ii) analyzed the dynamics of each layer in the information plane (deﬁned by axes

Original: I ( X ; L ) , I ( Y ; L ) ) during training with the cross-entropy loss.
Revised: I ( X ; L ) , I ( Y ; L ) ) during training with the cross-entropy loss.

Original: And indeed, these works have shown such a scheme is beneﬁcial in terms of classiﬁcation accuracy, and that the IB functional tends to decrease during training with the cross-entropy loss.
Revised: And indeed, these works have shown such a scheme is beneﬁcial in terms of classiﬁcation accuracy, and that the IB functional tends to increase during training with the crossentropy loss.

Original: Yet in all these demonstrations, nets were trained in an endto-end fashion, where the IB objective was either enforced only on a single layer L i , or not at all (trained only with a cross-entropy loss).
Revised: Yet in all these demonstrations, nets were trained in an end-to-end fashion, where the IB objective was either enforced only on a single layer L i , or not at all (trained only with a crossentropy loss).

Original: This implies that the empirical evidence on the validity of the theory is circumstantial. Indeed, there is no direct evidence that every layer acts to optimize (1) during training, and it is not currently clear whether such a layer-by-layer objective would  yield useful representations. 
Revised: Therefore, it remains unclear whether directly optimizing the IB functional of each layer, as suggested by Tishby & Zaslavsky (2015), would indeed yield useful representations. The goal of this paper is to experimentally examine this hypothesis.

Original: In addition to the lack of explicit validation, criticism on the rationale behind the IB theory of deep learning has surfaced lately.
Revised: In addition to the lack of explicit validation, criticism on the rationale behind the IB theory of deep learning has surfaced lately.

Original: One major concern highlighted in the recent works of Saxe et al.
Revised: One major concern highlighted in the recent works of Saxe et al.

Original: (2017) and Amjad & Geiger (2018), is that for deterministic DNNs with continuous inputs, the term
Revised: (2018) and Amjad & Geiger (2018), is that for deterministic DNNs with continuous inputs, the term

Original: I ( X ; L i ) is always inﬁnite so that the IB principle is in fact meaningless.
Revised: I ( X ; L i ) is always inﬁnite so that the IB principle is in fact meaningless.

Original: The authors suggest to resolve this by resorting to stochastic DNNs via the introduction of noise after each layer, which ensures that I ( X ; L i ) is ﬁnite.
Revised: The authors suggest to resolve this by resorting to stochastic DNNs via the introduction of noise after each layer, which ensures that I ( X ; L i ) is ﬁnite.

Original: And indeed, all attempts to train DNNs with the IB objective thus far employed stochastic nets (Alemi et al., 2018;
Revised: And indeed, all attempts to train DNNs with the IB objective thus far employed stochastic nets (Alemi et al., 2018;

Original: Belghazi et al., 2018; Kolchinsky et al., 2017), which in retrospect was a key element in their success.
Revised: Belghazi et al., 2018; Kolchinsky et al., 2017).

Original: Yet, deterministic DNNs are far more popular for most tasks, raising questions on the applicability of the IB theory for these models.
Revised: Yet, deterministic DNNs are far more popular for most tasks, raising questions on the applicability of the IB theory for these models.

Original: In this paper, we focus onapplying and validating (a regularized version of) the IB theory to deterministic DNNs trained on high-dimensional data.
Revised: In this paper, we focus on validating (a regularized version of) the IB theory to deterministic DNNs trained on high-dimensional data.

Original: To alleviate the explosion of MI with the input space, we introduce noise only for quantifying the MI between the input X and hidden layer L i (as also suggested by Saxe et al.
Revised: To alleviate the explosion of MI with the input space, we introduce noise only for quantifying the MI between the input X and hidden layer L i (as also suggested by Saxe et al.

Original: (2017)), and not into the DNN itself.
Revised: (2018)), and not into the DNN itself.

Original: Namely, we replace the ﬁrst term in (1) by I ( X ; L i + ε ) , where ε is noise.
Revised: Namely, we replace the ﬁrst term in (1) by I ( X ; L i + ε ) , where ε is noise.

Original: As we show, the resulting term can be interpreted as a weight decay penalty, which aligns with common practice in DNN training.
Revised: As we show, the resulting term can be interpreted as a weight decay penalty, which aligns with common practice in DNN training.

Original: To estimate MI, we use an auxiliary net, similarly to the suggestion by Belghazi et al.
Revised: To estimate MI, we use an auxiliary net, similarly to the suggestion by Belghazi et al.

Original: (2018), and in contrast to applying a variational lower-bound as in (Alemi et al., 2018; Kolchinsky et al., 2017).
Revised: (2018), and in contrast to applying a variational lower-bound as in (Alemi et al., 2018; Kolchinsky et al., 2017).

Original: However, as opposed to the original mutual information neural estimator (MINE) of Belghazi et al.
Revised: However, as opposed to the original mutual information neural estimator (MINE) of Belghazi et al.

Original: (2018), which fails to accurately estimate I ( X ; L i + ε ) in our setting 1 , we tailor the architecture of the auxiliary net speciﬁcally for our case in which L i is a known deterministic function of X .
Revised: (2018), which fails to accurately estimate I ( X ; L i + ε ) in our setting 1 , we tailor the architecture of the auxiliary net speciﬁcally for our case in which L i is a known deterministic function of X .

Original: This modiﬁcation results in signiﬁcantly more accurate estimates, as we show in Section 4.
Revised: This modiﬁcation results in signiﬁcantly more accurate estimates, as we show in Section 4.

Original: Equipped with this estimation strategy, we directly test the validity of the IB theory of deep learning for deterministic models.
Revised: Equipped with this estimation strategy, we directly test the validity of the IB theory of deep learning for deterministic models.

Original: This is accomplished by training a DNN while guaranteeing that each layer optimizes the IB functional in (1).
Revised: This is accomplished by training a DNN while guaranteeing that each layer optimizes the IB functional in (1).

Original: Starting from the ﬁrst hidden layer, each layer is trained independently, and frozen before moving on to the next layer.
Revised: Starting from the ﬁrst hidden layer, each layer is trained independently, and frozen before moving on to the next layer.

Original: We ﬁnd that such a training process is not only useful, but surprisingly consistently leads to slightly better performance than end-toend training with the cross-entropy loss.
Revised: We ﬁnd that such a training process is not only useful, but surprisingly consistently leads to slightly better performance than end-toend training with the cross-entropy loss.

Original: This is thus the ﬁrst explicit conﬁrmation of the theory suggested by Tishby & Zaslavsky (2015), subject to adapting the deﬁnition of I ( X ; L ) so as to make this quantity meaningful.
Revised: This is thus the ﬁrst explicit conﬁrmation of the theory suggested by Tishby & Zaslavsky (2015), subject to adapting the deﬁnition of I ( X ; L ) so as to make this quantity meaningful.

Revised: Please note that whether layer-wise training with the IB principle is generally beneﬁcial over other layer-wise training schemes, or over other regularized end-to-end schemes, is out of this paper’s scope.

Original: Accompanying the original theory were several key observations, the most notable being the emergence of two-phase training dynamics (Shwartz-Ziv & Tishby, 2017).
Revised: Accompanying the original theory were several key observations, the most notable being the emergence of two-phase training dynamics (Shwartz-Ziv & Tishby, 2017).

Original: It was hypothesized that the second “compression” phase is responsible for enhancing the generalization capability.
Revised: It was hypothesized that the second “compression” phase is responsible for enhancing the generalization capability.

Original: Yet Saxe et al.
Revised: Yet Saxe et al.

Original: (2017) contradict this claim, showing that this phase may not exist when training with a crossentropy loss.
Revised: (2018) contradict this claim, showing that this phase may not exist when training with a crossentropy loss.

Original: Our experiments align with those of Saxe et al.
Revised: Our experiments align with those of Saxe et al.

Original: (2017), though we do observe these two phases when training explicitly with the IB functional.
Revised: (2018), though we do observe these two phases when training explicitly with the IB functional.

Original: Surprisingly, when training with crossentropy and weight decay regularization, the two-phase dynamics are also apparent.
Revised: Surprisingly, when training with crossentropy and weight decay regularization, the two-phase dynamics are also apparent.

Original: This suggests that this two-phase pattern is induced by regularization, and is thus indeed linked to generalization.
Revised: This suggests that this two-phase pattern is induced by regularization, and is thus indeed linked to generalization.

Original:  R ELATED W ORKDeep neural nets and The Information Bottleneck
Revised:  R ELATED W ORK

Original:  The Information Bottleneck (IB) technique for summarizing a random variable X while maintaining maximal mutual information with a desirable output Y , was ﬁrst introduced by Tishby et al.
Revised: Deep neural nets and The Information Bottleneck The Information Bottleneck (IB) technique for summarizing a random variable X while maintaining maximal mutual information with a desirable output Y , was ﬁrst introduced by Tishby et al.

Original: It is designed to ﬁnd the optimal trade-off between prediction accuracy and representation complexity, by compressing X while retaining the essential information for predicting Y .
Revised: It is designed to ﬁnd the optimal trade-off between prediction accuracy and representation complexity, by compressing X while retaining the essential information for predicting Y .

Original: The IB optimization problem can be directly solved in discrete settings, as well as in certain simple families of continuous distributions, like jointly Gaussian random variables (Chechik et al., 2005).
Revised: The IB optimization problem can be directly solved in discrete settings, as well as in certain simple families of continuous distributions, like jointly Gaussian random variables (Chechik et al., 2005).

Original: However, in general high-dimensional continuous scenarios, exact optimization becomes impractical due to the intractable continuous MI.
Revised: However, in general high-dimensional continuous scenarios, exact optimization becomes impractical due to the intractable continuous MI.

Original: The IB principle was recently linked to DNNs by Tishby & Zaslavsky (2015), which hypothesized that DNN layers converge to the optimal IB curve during training.
Revised: The IB principle was recently linked to DNNs by Tishby & Zaslavsky (2015), which hypothesized that DNN layers converge to the optimal IB curve during training.

Original: The subsequent work by ShwartzZiv & Tishby (2017) attempted to experimentally verify this by analyzing DNNs trained with the cross-entropy loss (with no regularization).
Revised: The subsequent work by Shwartz- Ziv & Tishby (2017) attempted to experimentally verify this by analyzing DNNs trained with the cross-entropy loss (with no regularization).

Original: In this work, the MI estimation was made tractable by binning (discretization) of the latent representations L i , which works for “toy” examples but does not scale to real-world scenarios.
Revised: In this work, the MI estimation was made tractable by binning (discretization) of the latent representations L i , which works for “toy” examples but does not scale to real-world scenarios.

Original: Among several claims, the authors report that two training phases emerge: a “ﬁtting” phase followed by a “compression” phase, which to their understanding is linked to increased generalization.
Revised: Among several claims, the authors report that two training phases emerge: a “ﬁtting” phase followed by a “compression” phase, which to their understanding is linked to increased generalization.

Original: Yet Saxe et al.
Revised: Yet Saxe et al.

Original: (2017) contradict these claims, by linking the compression phase to the activation type and discretization strategy, and questioning the connection between this compression phase and generalization.
Revised: (2018) contradict these claims, by linking the compression phase to the activation type and discretization strategy, and questioning the connection between this compression phase and generalization.

Original: Moreover, these authors as well as Amjad &
Revised: Moreover, these authors as well as Amjad &

Original: Geiger (2018), also recognize that the term I ( X ; L ) in the IB functional is theoretically inﬁnite for deterministic DNNs with a continuous input X , and thus the attempt to measure it is meaningless.
Revised: Geiger (2018), also recognize that the term I ( X ; L ) in the IB functional is theoretically inﬁnite for deterministic DNNs with a continuous input X , and thus the attempt to measure it is meaningless.

Original: Both works propose to remedy this by adding noise, which ensures this term is ﬁnite.
Revised: Both works propose to remedy this by adding noise, which ensures this term is ﬁnite.

Original: Despite these obstacles, DNNs were in fact trained on real high-dimensional data for classiﬁcation tasks with the IB functional.
Revised: Despite these obstacles, DNNs were in fact trained on real high-dimensional data for classiﬁcation tasks with the IB functional.

Original: The difﬁculties arising from the intractable and inﬁnite term I ( X ; L ) were overcome by (i) training stochastic DNNs which ensure it is ﬁnite, and (ii) using a variational approximation of the MI which makes it tractable (Alemi et al., 2018;
Revised: The difﬁculties arising from the intractable and inﬁnite term I ( X ; L ) were overcome by (i) training stochastic DNNs which ensure it is ﬁnite, and (ii) using a variational approximation of the MI which makes it tractable (Alemi et al., 2018;

Original: Kolchinsky et al., 2017; Chalk et al., 2016; Achille & Soatto, 2018; Belghazi et al., 2018).
Revised: Kolchinsky et al., 2017; Chalk et al., 2016; Achille & Soatto, 2018; Belghazi et al., 2018).

Original: However, in all, the IB objectivewas enforced only on a single “bottleneck layer”, which is the last encoder layer in a variational autoencoder type architecture (Kingma & Welling, 2014). This scheme deviates from the original theory, in which every layer is to approach the optimal IB curve.
Revised: These schemes all rely on some form of injected stochasticity, and in fact, most enforce the IB objective only on a single “bottleneck layer”.

Original: An attempt to optimize the IB functional  in a layer-wise fashion, as in the original theory, has yet to appear due to these practical difﬁculties.
Revised: An attempt to optimize the IB functional for deterministic DNNs in a layer-wise fashion, as in the original theory, has yet to appear due to these practical difﬁculties.

Original: Here, we report on such a training scheme and present its results in the following sections.
Revised: Here, we report on such a training scheme and present its results in the following sections.

Revised: We note that layer-wise training with a related information-theoretic objective has been studied by Ver Steeg & Galstyan (2015) and Gao et al.

Revised: (2018), which have shown an effective and practical method to compose and analyze hierarchical representations.

Revised: These works consider an unsupervised setting, whereas we analyze supervised DNN training (speciﬁcally) with the IB functional.

Original: Estimating mutual information
Revised: Estimating mutual information

Original: Quantifying the MI between distributions is inherently difﬁcult (Paninski, 2003), and is tractable only in discrete settings or for a limited family of problems.
Revised: Quantifying the MI between distributions is inherently difﬁcult (Paninski, 2003), and is tractable only in discrete settings or for a limited family of problems.

Original: In other more general settings the exact computation is impossible, and known approximations do not scale well with dimension and sample size (Gao et al., 2015).
Revised: In other more general settings the exact computation is impossible, and known approximations do not scale well with dimension and sample size (Gao et al., 2015).

Original: Most recently, Belghazi et al.
Revised: Most recently, Belghazi et al.

Original: (2018) proposed the MI Neural Estimator (MINE) for approximating the MI between continuous high-dimensional random variables via back-prop over a DNN.
Revised: (2018) proposed the MI Neural Estimator (MINE) for approximating the MI between continuous high-dimensional random variables via back-prop over a DNN.

Original: The core idea is to estimate the
Revised: The core idea is to estimate the

Original: Kullback-Leibler (KL) divergence that is used to deﬁne MI, through the maximization of the dual representation of Donsker & Varadhan (1983).
Revised: Kullback-Leibler (KL) divergence that is used to deﬁne MI, through the maximization of the dual representation of Donsker & Varadhan (1983).

Original: Minimizing the MI between the input X and hidden layer L i of a DNN using this neural MI estimator can be efﬁciently accomplished, by formulating a minmax objective between the examined
Revised: Minimizing the MI between the input X and hidden layer L i of a DNN using this neural MI estimator can be efﬁciently accomplished, by formulating a minmax objective between the examined

Original: DNN and the estimator (an auxiliary net), similar to adversarial training (Goodfellow et al., 2014)
Revised: DNN and the estimator (an auxiliary net), similar to adversarial training (Goodfellow et al., 2014)

Original: (see Belghazi et al.
Revised: (see Belghazi et al.

Original: (2018) for details).

Original: We use this strategy to enforce the IB objective on DNN layers during training.
Revised: We use this strategy to enforce the IB objective on DNN layers during training.

Original: Note that the original authors also demonstrate the IB principle with this estimator, however they only do so on a single “bottleneck layer”, by using the cross-entropy loss as an approximation for the MI with the desired output Y , and in an end-to-end manner (as described above).
Revised: Note that the original authors also demonstrate the IB principle with this estimator, however they only do so on a single “bottleneck layer”, by using the cross-entropy loss as an approximation for the MI with the desired output Y , and in an end-to-end manner (as described above).

Original:  T HE NOISE - REGULARIZED MUTUAL INFORMATION PENALTY AND ITS RELATION TO WEIGHT DECAY
Revised:  T HE NOISE - REGULARIZED MUTUAL INFORMATION PENALTY AND ITS RELATION TO WEIGHT DECAY

Original: The intuitive goal of the penalty I ( X ; L i ) in the IB principle, is to induce a “compressed” latent representation which does not contain information irrelevant for predicting the labels Y .
Revised: The intuitive goal of the penalty I ( X ; L i ) in the IB principle, is to induce a “compressed” latent representation which does not contain information irrelevant for predicting the labels Y .

Original: However, for deterministic DNNs, if X is a continuous random variable, then the MI between X and any DNN layer L i is inﬁnite and therefore meaningless.
Revised: However, for deterministic DNNs, if X is continuous, then the MI between X and any DNN layer L i is inﬁnite and therefore meaningless.

Original: We seek a related measure of representation compactness which is more effective.
Revised: We seek a related measure of representation compactness which is more effective.

Original: One such alternative is to penalize for the MI between the layersonly up to an additive Gaussian noise ε , i.e. I ( X ; L i + ε ) .
Revised: One such alternative is to penalize for the MI between the layers up to an additive Gaussian noise ε , i.e. I ( X ; L i + ε ) .

Original: Note that noise is added only for the sake of quantifying complexity, and is not part of the net.
Revised: Note that noise is added only for the sake of quantifying complexity, and is not part of the net.

Original: Regularization by injection of noise is common practice in neural net training (Srivastava et al., 2014; Kingma et al., 2015; Wan et al., 2013; Poole et al., 2014), and was also analytically studied for the Gaussian IB case (Chechik et al., 2005).
Revised: Regularization by injection of noise is common practice in neural net training (Srivastava et al., 2014; Kingma et al., 2015; Wan et al., 2013; Poole et al., 2014), and was also analytically studied for the Gaussian IB case (Chechik et al., 2005).

Original: To understand the effect of replacing the penalty I ( X ; L i ) by I ( X ; L i + ε ) , we can write
Revised: To understand the effect of replacing the penalty I ( X ; L i ) by I ( X ; L i + ε ) , we can write

Original: [Equation]
Revised: [Equation]

Original: where H denotes  entropy, and we wrote L i ( X ) to emphasize that the latent representation is a deterministic function of X .
Revised: where h denotes differential entropy, and we wrote L i ( X ) to emphasize that the latent representation is a deterministic function of X .

Original: First, notice that at zero noise level the second term becomes minus inﬁnity (since p L i ( X ) | X becomes a delta function), which highlights again the inadequacy of the penalty
Revised: First, notice that at zero noise level the second term becomes minus inﬁnity (since p L i ( X ) | X becomes a delta function), which highlights again the inadequacy of the penalty

Original: I
Revised: I

Original: ( X ; L i ) for deterministic DNNs.
Revised: ( X ; L i ) for deterministic DNNs.

Original: Second, notice that the second term in (2) can be further simpliﬁed (for Gaussian noise) as
Revised: Second, notice that the second term in (2) can be further simpliﬁed (for Gaussian noise) as

Original: [Equation]
Revised: [Equation]

Original: where Σ is the noise covariance matrix.
Revised: where Σ is the noise covariance matrix.

Original: This term is independent of the DNN parameters .
Revised: This term is independent of the DNN parameters .

Original: This shows that the MI penalty I ( X ; L i ( X ) + ε ) is in fact a penalty only on the entropy H
Revised: This shows that the MI penalty I ( X ; L i ( X ) + ε ) is in fact a penalty only on the entropy h

Original: ( L i ( X ) + ε ) of the representation L i (up to additive noise), which is a typical measure of compactness.
Revised: ( L i ( X ) + ε ) of the representation L i (up to additive noise), which is a typical measure of compactness.

Original: To develop further intuition, it is instructive to examine the simple case in which both the input X and the additive noise ε are scalar (independent) Gaussian variables and the transformation is linear, i.e. L i ( X ) =
Revised: To develop further intuition, it is instructive to examine the simple case in which both the input X and the additive noise ε are scalar (independent) Gaussian variables and the transformation is linear, i.e. L i ( X ) =

Original: aX + b .
Revised: aX + b .

Original: In this setting, simple calculation shows that
Revised: In this setting, simple calculation shows that

Original: [Equation]
Revised: [Equation]

Original: where σ 2 X and σ 2 ε are the variances of the input and noise, respectively.
Revised: where σ 2 X and σ 2 ε are the variances of the input and noise, respectively.

Original: This function is monotonically increasing in | a | and attains its minimum at a = 0 , in which case X and L i become independent.
Revised: This function is monotonically increasing in | a | and attains its minimum at a = 0 , in which case X and L i become independent.

Original: This highlights the fact that this term, in essence, induces a type of weight decay regularization, as popularly employed in deep net training.
Revised: This highlights the fact that this term, in essence, induces a type of weight decay regularization, as popularly employed in deep net training.

Original: Clearly, the same intuition is valid also for non-linear high-dimensional settings, since weight decay (at the extreme) drives X and L i to become independent and thus to have zero MI.
Revised: Clearly, the same intuition is valid also for non-linear high-dimensional settings, since weight decay (at the extreme) drives X and L i to become independent and thus to have zero MI.

Original: Yet the precise form of the penalty is generally different between the MI regularizer and hand-crafted ones.
Revised: Yet the precise form of the penalty is generally different between the MI regularizer and hand-crafted ones.

Original: Also notice that in our case, the noise level σ ε controls the strength of the regularizer, as we experimentally validate in Section 4.
Revised: Also notice that in our case, the noise level σ ε controls the strength of the regularizer, as we experimentally validate in Section 4.

Original: In light of the above discussion, our goal is to explore the validity of the noise-regularized version of the IB Langragian,
Revised: In light of the above discussion, our goal is to explore the validity of the noise-regularized version of the IB Langragian,

Original: [Equation]
Revised: [Equation]

Original: where ε is white Gaussian noise.
Revised: where ε is white Gaussian noise.

Original:  A CCURATE ESTIMATION OF MUTUAL INFORMATION
Revised:  A CCURATE ESTIMATION OF MUTUAL INFORMATION

Original: Given a DNN, to estimate the two MI terms in (5), we rely on the recently proposed MI neural estimator (MINE) of Belghazi et al.
Revised: Given a DNN, to estimate the two MI terms in (5), we rely on the recently proposed MI neural estimator (MINE) of Belghazi et al.

Original: This method uses the fact that I ( U ; V ) =
Revised: This method uses the fact that I ( U ; V ) =

Original: KL ( P U,V
Revised: KL ( P U,V

Original: || P
Revised: || P

Original: U ⊗ P V ) and exploits the Donsker-Vardan representation of the KL divergence,
Revised: U ⊗ P V ) and exploits the dual representation of Donsker & Varadhan (1983),

Original: [Equation]
Revised: [Equation]

Original: where the supremum is over all functions D : U × V → R for which the second expectation is ﬁnite.
Revised: where the supremum is over all functions D : U × V → R for which the second expectation is ﬁnite.

Original: To approximate this value, the expectations are replaced by sample means over a training set, and the optimization is performed over a smaller family of functions - those implementable by a deep net of predeﬁned architecture.
Revised: To approximate this value, the expectations are replaced by sample means over a training set, and the optimization is performed over a smaller family of functions - those implementable by a deep net of predeﬁned architecture.

Original: That is, D is taken to be a DNN by itself, whose parameters are optimized via back-propagation with stochastic gradient descent.
Revised: That is, D is taken to be a DNN by itself, whose parameters are optimized via back-propagation with stochastic gradient descent.

Original: Note that to optimize Eq.
Revised: Note that to optimize Eq.

Original: D must learn to discriminate between (i) pairs ( u, v ) drawn from the joint distribution P U,V , and (ii) pairs ( u, v ) drawn independently from the marginal distributions P U , P V .
Revised: D must learn to discriminate between (i) pairs ( u, v ) drawn from the joint distribution P U,V , and (ii) pairs ( u, v ) drawn independently from the marginal distributions P U , P V .

Original: This is similar to the principle underlying adversarial training (Goodfellow et al., 2014), where to estimate the JensonShannon divergence between “real” and “fake” distributions, a critic DNN must learn to discriminate between “real” and “fake” data instances.
Revised: This is similar to the principle underlying adversarial training (Goodfellow et al., 2014), where to estimate the JensonShannon divergence between “real” and “fake” distributions, a critic DNN must learn to discriminate between “real” and “fake” data instances.

Original: In our setting, we estimate the term
Revised: In our setting, we estimate the term

Original: I
Revised: I

Original: ( Y ; L i ) with a MINE, as described above.
Revised: ( Y ; L i ) with a MINE, as described above.

Original: However, estimating the term I ( X ; L i + ε ) is typically far more demanding, due to its higher dimensionality and the continuous nature of both arguments.
Revised: However, estimating the term I ( X ; L i + ε ) is typically far more demanding, due to its higher dimensionality and the continuous nature of both arguments.

Original: Experimentally, we observed inconsistent and non-convergent behaviors for this estimate.
Revised: Experimentally, we observed inconsistent and non-convergent behaviors for this estimate.

Original: In particular, as illustrated in Fig. 2, the estimate does not grow indefinitely as the noise variance is taken to 0 , despite the fact that the true MI becomes inﬁnite in this case (see Sec. 3).
Revised: In particular, as illustrated in Fig. 2, the estimate does not grow indefinitely as the noise variance is taken to 0 , despite the fact that the true MI becomes inﬁnite in this case (see Sec. 3).

Original: To improve the estimation accuracy, we exploit the strong prior knowledge we have in our setting, which is that the latent representation L i is a known deterministic function of the input X .
Revised: To improve the estimation accuracy, we exploit the strong prior knowledge we have in our setting, which is that the latent representation L i is a known deterministic function of the input X .

Original: Speciﬁcally, L i =
Revised: Speciﬁcally, L i =

Original: F i ( X ) , where F i is the function implemented by the ﬁrst i layers of the net.
Revised: F i ( X ) , where F i is the function implemented by the ﬁrst i layers of the net.

Original: This implies that D is actually trying to discriminate between (i) pairs ( x 1 , F i ( x 1 ) + ε ) where x 1 is drawn from P X , and (ii) pairs ( x 1 , F i ( x 2 ) + ε ) where x 1 , x 2 are independently drawn from 
Revised: This implies that D is actually trying to discriminate between (i) pairs ( x 1 , F i ( x 1 ) + ε ) where x 1 is drawn from P X , and (ii) pairs ( x 1 , F i ( x 2 ) + ε ) where x 1 , x 2 are independently drawn from P

Original: P X .
Revised: X .

Original: Intuitively, the easiest way to achieve this is by applying the function
Revised: Intuitively, the easiest way to achieve this is by applying the function

Original: F i on the ﬁrst argument, and checking whether the result is close to the second argument, up to noise.
Revised: F i on the ﬁrst argument, and checking whether the result is close to the second argument, up to noise.

Original: However, unless the discriminator has the capacity to implement the function F i internally, it will fail to converge to such a rule.
Revised: However, unless the discriminator has the capacity to implement the function F i internally, it will fail to converge to such a rule.

Original: And even if its capacity sufﬁces, convergence may be very difﬁcult.
Revised: And even if its capacity sufﬁces, convergence may be very difﬁcult.

Original: Nevertheless, we can make the discriminator’s task far easier, simply by informing it of the function F i .
Revised: Nevertheless, we can make the discriminator’s task far easier, simply by informing it of the function F i .

Original: We do this by implementing a copy of the sub-net F i within the discriminator and allowing it to pass its ﬁrst input through this net (see Fig. 1).
Revised: We do this by implementing a copy of the sub-net F i within the discriminator and allowing it to pass its ﬁrst input through this net (see Fig. 1).

Original: Since now the task becomes much simpler, we follow this by only a few fully-connected layers (see Appendix C), which sufﬁce for obtaining accurate MI estimates and fast convergence, as can be seen in Fig.
Revised: Since now the task becomes much simpler, we follow this by only a few fully-connected layers (see Appendix C), which sufﬁce for obtaining accurate MI estimates and fast convergence, as can be seen in Fig.

Original: Appendix A provides a detailed description of our AA-MINE training.
Revised: Appendix A provides a detailed description of our AA-MINE training.

Original: Notice that for the limit case of zero noise level, the discriminator need only check whether the two inputs are identical, which can be perfectly accomplished with a very simple architecture.
Revised: Notice that for the limit case of zero noise level, the discriminator need only check whether the two inputs are identical, which can be perfectly accomplished with a very simple architecture.

Original: This drives the optimization of the objective in (6) to inﬁnity, which aligns with the theory.
Revised: This drives the optimization of the objective in (6) to inﬁnity, which aligns with the theory.

Original: We coin this MI estimator an architecture aware MINE (AA-MINE).
Revised: We coin this MI estimator an architecture aware MINE (AA-MINE).

Original: Before moving on to our primary goal, which is to train DNNs according to the IB objective,at this point we can use our approach for the simpler task of just measuring the MI of each layer with the input/output while training with the common cross-entropy loss.
Revised: Before moving on to our primary goal, which is to train DNNs according to the IB objective, we can use our approach for the simpler task of just measuring the MI of each layer with the input/output while training with the common cross-entropy loss.

Original: To illustrate this, we trained a three-layer MLP with dimensions 784 − 512 − 512 − 10 and ReLU activations on the (unmodiﬁed) MNIST dataset (LeCun et al., 1998) using the cross entropy loss, and measured the two MI terms in (5) for each layer during training (see Appendix C forfull details).
Revised: To illustrate this, we trained a three-layer MLP with dimensions 784 − 512 − 512 − 10 and ReLU activations on the (unmodiﬁed) MNIST dataset (LeCun et al., 1998) using the cross entropy loss, and measured the two MI terms in (5) for each layer during training (see Appendix C for details).

Original: As can be seen in Fig. 3(a), both terms tend to increase throughout the optimization process, and there is no apparent compression phase, in which the MI with the input suddenly begins to decrease.
Revised: As can be seen in Fig. 3(a), both terms tend to increase throughout the optimization process, and there is no apparent compression phase, in which the MI with the input suddenly begins to decrease.

Original: This phenomenon has also been observed by Saxe et al.
Revised: This phenomenon has also been observed by Saxe et al.

Original: (2017), and is in contradiction with the observations in (Shwartz-Ziv & Tishby, 2017), which seem to be associated with the binning they used to measure MI (Saxe et al., 2017).
Revised: (2018), and is in contradiction with the observations in (Shwartz-Ziv & Tishby, 2017), which seem to be associated with the binning they used to measure MI (Saxe et al., 2018).

Original: However, interestingly, when we add weight decay as a regularizer, the two-phase dynamics emerge (see Fig. 3(b)).
Revised: However, interestingly, when we add weight decay as a regularizer, the two-phase dynamics emerge (see Fig. 3(b)).

Original: As discussed in Sec. 3, weight decayindeed induces a penalty on the MI with the input, and our experiments suggest that this penalty induces a compression phase which begins only after the MI with the output has reached high values.
Revised: As discussed in Sec. 3, weight decay induces a penalty on the MI with the input, and our experiments suggest that this penalty induces a compression phase which begins only after the MI with the output has reached high values.

Original: As weight decay is known to increase generalization (Krogh & Hertz, 1992), we conclude that the compression phase is indeed linked to generalization.
Revised: As weight decay is known to increase generalization (Krogh & Hertz, 1992), we conclude that the compression phase is indeed linked to generalization.

Original:  L AYER - BY - LAYER TRAINING WITH THE IB LOSS
Revised:  L AYER - BY - LAYER TRAINING WITH THE IB LOSS

Original: The stage is now set for training a DNN layer-by-layer with the IB functional.
Revised: The stage is now set for training a DNN layer-by-layer with the IB functional.

Original: Starting with the ﬁrst hidden layer, our goal is to train each layer independently until convergence, freeze its weights, and then move on th the next layer.
Revised: Starting with the ﬁrst hidden layer, our goal is to train each layer independently until convergence, freeze its weights, and then move on th the next layer.

Original: As our MI estimators are based on DNNs, we can achieve this goal by training them simultaneously with the layer of interest, by using back-propagation.
Revised: As our MI estimators are based on DNNs, we can achieve this goal by training them simultaneously with the layer of interest, by using back-propagation.

Original: Speciﬁcally, as in adversarial training and in the original MINE work (Goodfellow et al., 2014; Belghazi et al., 2018), to update the i th layer, we alternate between updating the discriminators D x , D y accordingto (6) and updating the layer’s parameters θ i according to the noise-regularized IB objective (5) with
Revised: Speciﬁcally, as in adversarial training and in the original MINE work (Goodfellow et al., 2014; Belghazi et al., 2018), to update the i th layer, we alternate between updating the discriminators D x , D y according

Original: [Figure]
Revised: [Figure]

Original:  the information terms replaced by the  discriminator nets, namely
Revised: to (6) and updating the layer’s parameters θ i according to the noise-regularized IB objective (5) with the information terms replaced by the dual representation with the discriminator nets, namely

Original: [Equation]
Revised: [Equation]

Original: Note that mutual information is invariant to (smooth) invertible transformations.
Revised: Note that mutual information is invariant to (smooth) invertible transformations.

Original: Thus applying any invertible transformation on the net’s output ˆ Y , will not change its MI with Y .
Revised: Thus applying any invertible transformation on the net’s output ˆ Y , will not change its MI with Y .

Original: To simplify the classiﬁcation task, it is convenient to seek an invertible transformation that brings ˆ Y closest to (the 1-hot representation of) Y .
Revised: To simplify the classiﬁcation task, it is convenient to seek an invertible transformation that brings ˆ Y closest to (the 1-hot representation of) Y .

Original: To achieve this, we freeze the DNN parameters after training is concluded, and post-train one additional fully-connected linear layer using a cross entropy loss.
Revised: To achieve this, we freeze the DNN parameters after training is concluded, and post-train one additional fully-connected linear layer using a cross entropy loss.

Original: This determines the best linear transformation to bring ˆ Y closest to Y , without changing the MI.

Original: [Table]
Revised: [Table]

Original:  Note that this does not expand the net capacity, since we use no non-linear activation betweenthe last layer of the net and the post-trained fully-connected layer, so that they can be combined into a single linear layer after this post-training step.
Revised: This determines the best linear transformation to bring ˆ Y closest to Y , without changing the MI. Note that this does not expand the net capacity since we use no non-linear activation between net post fully-connected layer, so that they can be combined post

Original: Figure 4 depicts the information plane dynamics for IB-based layer-by-layer training of the same net of Fig. 3.
Revised: Figure 4 depicts the information plane dynamics for IB-based layer-by-layer training of the same net of Fig. 3.

Original: Interestingly, in this experiment, we observe very different dynamics.
Revised: Interestingly, in this experiment, we observe very different dynamics.

Original: First, the MI with the input does not increase at any stage during training.
Revised: First, the MI with the input does not increase at any stage during training.

Original: Second, here we have a clear two-phase process.
Revised: Second, here we have a clear two-phase process.

Original: Namely, each layer starts by moving upwards in the information plane to increase its MI with the desired output Y [Table] and then turns to compress its latent representation by decreasing its MI with X , i.e. by moving leftwards.
Revised: Namely, each layer starts by moving upwards in the information plane to increase its MI with the desired output Y , and then turns to compress its latent representation by decreasing its MI with X , i.e. by moving leftwards.

Original: The resulting classiﬁcation accuracies are reported in Table 1.
Revised: The resulting classiﬁcation accuracies are reported in Table 1.

Original: Both when using 100% of the training set and when using only 1%, the performance of this training scheme is comparable to the baseline of training with the cross-entropy loss, and even slightly better.
Revised: Both when using 100% of the training set and when using only 1%, the performance of this training scheme is comparable to the baseline of training with the cross-entropy loss, and even slightly better.

Original: Also, notice that the penalty I ( X ; L i + ε ) is essential for generalization, especially when using only 1% of the training set, and without out it, there is a drop in test accuracy.
Revised: Also, notice that the penalty I ( X ; L i + ε ) is essential for generalization, especially when using only 1% of the training set, and without out it, there is a drop in test accuracy.

Original: This is thus the ﬁrst experiment to fully directly validate the effectiveness of the IB principle for DNNs.
Revised: This is thus the ﬁrst experiment to fully directly validate the effectiveness of the IB principle for DNNs.

Original: We also test the IB principle with a conv-net for classifying the higher-dimensional CIFARdataset (Krizhevsky & Hinton, 2009).
Revised: We also test the IB principle with a conv-net for classifying the higher-dimensional CIFARdataset (Krizhevsky & Hinton, 2009).

Original: Here we train layer-by-layer with the IB functional a net with three conv-layers (16 ﬁlters, ReLU activations, max-pooling after each), which are followed by three fully-connected layers (512-512-10, ReLU activations; see Appendix C for full details).
Revised: Here we train layer-by-layer with the IB functional a net with three conv-layers (16 ﬁlters, ReLU activations, max-pooling after each), which are followed by three fully-connected layers (512-512-10, ReLU activations; see Appendix C for full details).

Original: The resulting classiﬁcation accuracies are reported in Table 2.
Revised: The resulting classiﬁcation accuracies are reported in Table 2.

Original: In this more challenging scenario, the effectiveness of the IB principle is more pronounced.
Revised: In this more challenging scenario, the effectiveness of the IB principle is more pronounced.

Original: Training with the full IB functional leads to better accuracy compared to the cross-entropy baseline, and again, the regularization term which promotes a “compressed” latent representation proves quite advantageous.
Revised: Training with the full IB functional leads to better accuracy compared to the cross-entropy baseline, and again, the regularization term which promotes a “compressed” latent representation proves quite advantageous.

Original: [Table]
Revised: [Table]

Original: It is interesting to try to decipher how layer-by-layer training with the IB loss affects the internal representations.
Revised: It is interesting to try to decipher how layer-by-layer training with the IB loss affects the internal representations.

Original: In Fig.
Revised: In Fig.

Original: we plot the t-SNE embeddings (Maaten & Hinton, 2008) of the latent representations
Revised: we plot the t-SNE embeddings (Maaten & Hinton, 2008) of the latent representations

Original: L i of the MLP trained layer-by-layer on MNIST (Fig. 5(a),(b) with 100% of the train set, Fig. 5(c),(d) with 1% ).
Revised: L i of the MLP trained layer-by-layer on MNIST (Fig. 5(a),(b) with 100% of the train set, Fig. 5(c),(d) with 1% ).

Original: Recall that the regularization term I ( X ; L i + ε ) enforces reduced entropy of the latent representation L i (see Section 3).
Revised: Recall that the regularization term I ( X ; L i + ε ) enforces reduced entropy of the latent representation L i (see Section 3).

Original: The left side of Fig.
Revised: The left side of Fig.

Original: 5 shows the effect of increasing/decreasing the inﬂuence of this term on the ﬁrst hidden representation, L 1 .
Revised: 5 shows the effect of increasing/decreasing the inﬂuence of this term on the ﬁrst hidden representation, L 1 .

Original: As can be seen, for large β values, the representation becomes quantized, which is a form of compression and indeed has reduced entropy.
Revised: As can be seen, for large β values, the representation becomes quantized, which is a form of compression and indeed reduces the differential entropy by decreasing the effective support of the representation (differential entropy tends to −∞ as the support of the distribution grows smaller).

Original: On the right side of Fig.
Revised: On the right side of Fig.

Original: 5, we show the difference between the representation L 3 at the output of the third layer between (c) training end-to-end with a cross-entropy loss and (d) training layer-by-layer with the IB principle.
Revised: 5, we show the difference between the representation L 3 at the output of the third layer when (c) training end-to-end with a cross-entropy loss and (d) training layer-by-layer with the IB principle.

Original: In the latter, we obtain better separated and concentrated clusters, which illustrates the advantage of directly enforcing the IB on each layer.
Revised: In the latter, we obtain better separated and concentrated clusters, which illustrates the advantage of directly enforcing the IB on each layer.

Original: To test this quantitatively, we compute the mean distance between clusters divided by the mean standard deviation of clusters, where higher values indicate better separation and concentration.
Revised: To test this quantitatively, we compute the mean distance between clusters divided by the mean standard deviation of clusters, where higher values indicate better separation and concentration.

Original: The obtained value for (c) is 3 . 9 , and for (d) is 5 . 2 , conﬁrming the advantageous outcome.
Revised: The obtained value for (c) is 3 . 9 , and for (d) is 5 . 2 , conﬁrming the advantageous outcome.

Original:  S UMMARY AND DISCUSSION
Revised:  S UMMARY AND DISCUSSION

Original: Our experiments demonstrate that training DNNs explicitly with the IB functional (and without a cross-entropy loss) leads to competitive and even enhanced prediction performance.
Revised: Our experiments demonstrate that training DNNs explicitly with the IB functional (and without a cross-entropy loss) leads to competitive and even enhanced prediction performance.

Original: This provides, for the ﬁrst time, strong and direct empirical evidence for the validity of the IB theory of deep learning.
Revised: This provides, for the ﬁrst time, strong and direct empirical evidence for the validity of the IB theory of deep learning.

Original: Our training scheme was made possible by two key changes to prior attempts for training with the IB principle.
Revised: Our training scheme was made possible by two key changes to prior attempts for training with the IB principle.

Original: First, as previously commented by Saxe et al.
Revised: First, as previously commented by Saxe et al.

Original: (2017), we used a noise-regularized version of the IB functional, which removes the theoretical difﬁculty of the MI between layers being inﬁnite, while still being consistent with the intuition of a “complexity” penalty.
Revised: (2018), we used a noise-regularized version of the IB functional, which removes the theoretical difﬁculty of the MI between layers being inﬁnite, while still being consistent with the intuition of a “complexity” penalty.

Original: As we showed, the resulting term acts as a form of weight-decay regularization.
Revised: As we showed, the resulting term acts as a form of weight-decay regularization.

Original: Second, we derived an MI estimation scheme speciﬁcally tailored for MI estimation between deterministic DNN layers.
Revised: Second, we derived an MI estimation scheme speciﬁcally tailored for MI estimation between deterministic DNN layers.

Original: Our scheme is capable of accurate estimation in a scenario where this estimation has been shown to be generally difﬁcult and usually intractable.
Revised: Our scheme is capable of accurate estimation in a scenario where this estimation has been shown to be generally difﬁcult and usually intractable.

Original: We note that while the IB principle proved beneﬁcial for layer-bylayer training, our alternating optimization procedure is computationally intensive.
Revised: We note that while the IB principle proved beneﬁcial for layer-bylayer training, our alternating optimization procedure is computationally intensive.

Original: However, as we show in Appendix B, end-to-end training by enforcing the IB objective only on the last layer of the
Revised: However, as we show in Appendix B, end-to-end training by enforcing the IB objective only on the last layer of the

Original: DNN, also leads to improved results, and is signiﬁcantly more efﬁcient.
Revised: DNN, also leads to improved results, and is signiﬁcantly more efﬁcient.

Original: A A LGORITHM FOR ESTIMATING NOISE - REGULARIZED MI
Revised: A A LGORITHM FOR ESTIMATING NOISE - REGULARIZED MI

Original: B T RAINING E ND - TO -E ND
Revised: B T RAINING E ND - TO -E ND

Original: Despite originally using the IB functional to train layer-by-layer, we can also use it to train the whole network end-to-end by taking into account only the MI of the classiﬁcation layer with the target and input space.
Revised: Despite originally using the IB functional to train layer-by-layer, we can also use it to train the whole network end-to-end by taking into account only the MI of the classiﬁcation layer with the target and input space.

Original: Adding the IB functional as a regularizer to the standard cross-entropy (CE) loss, we can perhaps prevent over-ﬁtting in large scale DNNs.
Revised: Adding the IB functional as a regularizer to the standard cross-entropy (CE) loss, we can perhaps prevent over-ﬁtting in large scale DNNs.

Original: We therefore propose to minimize the objective
Revised: We therefore propose to minimize the objective

Original: [Equation]
Revised: [Equation]

Original: where CE is the standard cross-entropy loss.
Revised: where CE is the standard cross-entropy loss.

Original: Training DNNs end-to-end with the IB functional as regularizer has been proposed in several variations before (Belghazi et al., 2018;
Revised: Training DNNs end-to-end with the IB functional as regularizer has been proposed in several variations before (Belghazi et al., 2018;

Original: Alemi et al., 2018).
Revised: Alemi et al., 2018).

Original: Both of these works train stochastic encoderdecoder like networks to maximize cross entropy of decodings ˆ Y with the labels Y , which is a lower bound on MI
Revised: Both of these works train stochastic encoderdecoder like networks to maximize cross entropy of decodings ˆ Y with the labels Y , which is a lower bound on MI

Original: I ( ˆ Y , Y ) , while minimizing MI of stochastic encoding Z with input space X .
Revised: I ( ˆ Y , Y ) , while minimizing MI of stochastic encoding Z with input space X .

Original: Alemi et al.
Revised: Alemi et al.

Original: (2018) estimate I ( Z, X ) by using a Gaussian encoder and approximating X by a Spherical
Revised: (2018) estimate I ( Z, X ) by using a Gaussian encoder and approximating X by a Spherical

Original: Gaussian distribution of the same dimension as Z , while Belghazi et al.
Revised: Gaussian distribution of the same dimension as Z , while Belghazi et al.

Original: (2018) use a standard MINE and do not assume Gaussian distribution of neither Z nor X .
Revised: (2018) use a standard MINE and do not assume Gaussian distribution of neither Z nor X .

Original: For deterministic DNNs, I ( Z, X ) is inﬁnite, making both techniques untranslatable to this more popular setting.
Revised: For deterministic DNNs, I ( Z, X ) is inﬁnite, making both techniques untranslatable to this more popular setting.

Original: Moreover, neither of these works is capable of dealing with high dimensional data for classiﬁcation without pre-trained models, and so far have only shown results on MNIST.
Revised: Moreover, neither of these works is capable of dealing with high dimensional data for classiﬁcation without pre-trained models, and so far have only shown results on MNIST.

Original: Diverging from these works, we used our AA-MINE to estimate the MI
Revised: Diverging from these works, we used our AA-MINE to estimate the MI

Original: I
Revised: I

Original: ( X, ˆ Y + ε ) and a standard
Revised: ( X, ˆ Y + ε ) and a standard

Original: MINE to estimate the I ( Y, ˆ Y ) , in addition to the standard cross-entropy loss.
Revised: MINE to estimate the I ( Y, ˆ Y ) , in addition to the standard cross-entropy loss.

Original: Since stochasticity is used only for measuring MI, and the AA-MINE needs only the classiﬁcation layers representation of X , we can regularize deterministic DNNs trained on complex data and show the importance of explicitly measuring MI with target space, rather the only using a lower bound such as cross-entropy.
Revised: Since stochasticity is used only for measuring MI, and the AA-MINE needs only the classiﬁcation layers representation of X , we can regularize deterministic DNNs trained on complex data and show the importance of explicitly measuring MI with target space, rather the only using a lower bound such as cross-entropy.

Original: Furthermore, this method can also be used with with stochastic methods to regularize DNNs, such as DropOut (Srivastava et al., 2014) or DropConnect (Wan et al., 2013).
Revised: Furthermore, this method can also be used with with stochastic methods to regularize DNNs, such as DropOut (Srivastava et al., 2014) or DropConnect (Wan et al., 2013).

Original: The results for MNIST and
Revised: The results for MNIST and

Original: CIFAR-10 classiﬁcation are reported in Table 3, and convergence curves are plotted in Fig.
Revised: CIFAR-10 classiﬁcation are reported in Table 3, and convergence curves are plotted in Fig.

Original: See
Revised: See

Original: Appendix C for training details.
Revised: Appendix C for training details.

Original: [Table]
Revised: [Table]

Original: As can be seen, while the IB loss alone already leads to slight improvement, its combination with the CE loss provides a substantial boost.
Revised: As can be seen, while the IB loss alone already leads to slight improvement, its combination with the CE loss provides a substantial boost.

Original: This is especially evident in cases where the training set is small w.r.t.
Revised: This is especially evident in cases where the training set is small w.r.t.

Original: the complexity of the problem.
Revised: the complexity of the problem.

Original: C T RAINING DETAILS FOR THE MNIST AND CIFAR-10 EXPERIMENTS IN SECTIONS 4, 5, A PPENDIX . B
Revised: C T RAINING DETAILS FOR THE MNIST AND CIFAR-10 EXPERIMENTS IN SECTIONS 4, 5, A PPENDIX . B

Original: In these experiments, we started by training both MINE discriminators D x , D Y separately until convergence.
Revised: In these experiments, we started by training both MINE discriminators D x , D Y separately until convergence.

Original: Then, layer-by-layer training was performed with a total of 5000 iterations for each layer.
Revised: Then, layer-by-layer training was performed with a total of 5000 iterations for each layer.

Original: In each of these iterations, we alternate between 1 step for updating the trained layer, andsteps for updating each of the MINE discriminators.
Revised: In each of these iterations, we alternate between 1 step for updating the trained layer, andsteps for updating each of the MINE discriminators.

Original: For both MNIST and CIFAR-10, the variance of the Gaussian noise ε was σ 2 ε = 2 , and the learning rate was 10 − 4 for the MINE discriminators and 10 − 3 for the the DNN, respectively.
Revised: For both MNIST and CIFAR-10, the variance of the Gaussian noise ε was σ 2 ε = 2 , and the learning rate was 10 − 4 for the MINE discriminators and 10 − 3 for the the DNN, respectively.

Original: For MNIST, the bottleneck parameter was β = 10 − 3 for all the layers (Table 1).
Revised: For MNIST, the bottleneck parameter was β = 10 − 3 for all the layers (Table 1).

Original: For CIFAR-10, the bottleneck parameter was β = 0 for the convolutional layers and β = 10 − 3 for the fully-connected layers (Table 2).
Revised: For CIFAR-10, the bottleneck parameter was β = 0 for the convolutional layers and β = 10 − 3 for the fully-connected layers (Table 2).

Original: For the end-to-end training experiment (Table 3), the MNIST architecture was a MLP ( 784 − 600 − 400 − 10 ) with ReLU activations, and the CIFAR-10 architecture was a CNN (CONV48-CONV48CONV96-CONV96-CONV192-CONV192-FC512-FC512-FC10) with 2 × 2 max-pooling every second convolutional layer with 0 .
Revised: For the end-to-end training experiment (Table 3), the MNIST architecture was a MLP ( 784 − 600 − 400 − 10 ) with ReLU activations, and the CIFAR-10 architecture was a CNN (CONV48-CONV48CONV96-CONV96-CONV192-CONV192-FC512-FC512-FC10) with 2 × 2 max-pooling every second convolutional layer with 0 .

Original: 5 drop-out probability on convolutional/fully-connected layers, respectively.
Revised: 5 drop-out probability on convolutional/fully-connected layers, respectively.

Original: The architecture of the MINE discriminator was an MLP with two hidden layers of 1500 neurons and Leaky-ReLU activations, and a ﬁnal linear layer to a single neuron.
Revised: The architecture of the MINE discriminator was an MLP with two hidden layers of 1500 neurons and Leaky-ReLU activations, and a ﬁnal linear layer to a single neuron.

Original: The input was always taken to be a concatenation of the two variables between which MI is measured.
Revised: The input was always taken to be a concatenation of the two variables between which MI is measured.

Original: For AA-MINE, we used the same architecture, only after passing the ﬁrst input through a copy of layers 1 . . .
Revised: For AA-MINE, we used the same architecture, only after passing the ﬁrst input through a copy of layers 1 . . .

Original: i of the primary DNN.
Revised: i of the primary DNN.

Original: Therefore, this architecture was always applied to F i ( X ) and F i ( X ) + (cid:15) in the case of AA-MINE.
Revised: Therefore, this architecture was always applied to F i ( X ) and F i ( X ) + (cid:15) in the case of AA-MINE.

Original: In all experiments, training was performed with the Adam optimizer (Kingma & Ba, 2015) via python-TensorFlow (Abadi et al., 2015).
Revised: In all experiments, training was performed with the Adam optimizer (Kingma & Ba, 2015) via python-TensorFlow (Abadi et al., 2015).