Original: K NIFE : K ERNELIZED -N EURAL D IFFERENTIAL E NTROPY E STIMATION
Revised: K NIFE : K ERNELIZED -N EURAL D IFFERENTIAL E NTROPY E STIMATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Estimation of (differential) entropy and the related mutual information has been pursued with signiﬁcant efforts by the machine learning community.
Revised: Estimation of (differential) entropy and the related mutual information has been pursued with signiﬁcant efforts by the machine learning community.

Original: To address shortcomings in previously proposed estimators for differential entropy, here we introduce K NIFE , a fully parameterized, differentiable kernel-based estimator of differential entropy.
Revised: To address shortcomings in previously proposed estimators for differential entropy, here we introduce K NIFE , a fully parameterized, differentiable kernel-based estimator of differential entropy.

Original: The ﬂexibility of our approach also allows us to construct
Revised: The ﬂexibility of our approach also allows us to construct

Original: K NIFE -based estimators for conditional (on either discrete or continuous variables) differential entropy, as well as mutual information.
Revised: K NIFE -based estimators for conditional (on either discrete or continuous variables) differential entropy, as well as mutual information.

Original: We empirically validate our method on high-dimensional synthetic data and further apply it to guide the training of neural networks for real-world tasks.
Revised: We empirically validate our method on high-dimensional synthetic data and further apply it to guide the training of neural networks for real-world tasks.

Original: Our experiments on a large variety of tasks, including visual domain adaptation, textual fair classiﬁcation, and textual ﬁne-tuning demonstrate the effectiveness of K NIFE -based estimation.
Revised: Our experiments on a large variety of tasks, including visual domain adaptation, textual fair classiﬁcation, and textual ﬁne-tuning demonstrate the effectiveness of K NIFE -based estimation.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Learning tasks requires information (Principe et al., 2006) in the form of training data.
Revised: Learning tasks requires information (Principe et al., 2006) in the form of training data.

Original: Thus, information measures (Shannon, 1948) ( e.g. entropy, conditional entropy and mutual information) have been a source of inspiration for the design of learning objectives in modern machine learning (ML) models (Linsker, 1989; Torkkola, 2006).
Revised: Thus, information measures (Shannon, 1948) ( e.g. entropy, conditional entropy and mutual information) have been a source of inspiration for the design of learning objectives in modern machine learning (ML) models (Linsker, 1989; Torkkola, 2006).

Original: Over the years, a plethora of estimators have been introduced to estimate the value of the aforementioned measures of information and they have been applied to many different problems, including information and coding theory, limiting distributions, model selection, design of experiment and optimal prior distribution, data disclosure, and relative importance of predictors (Ebrahimi et al., 2010).
Revised: Over the years, a plethora of estimators have been introduced to estimate the value of the aforementioned measures of information and they have been applied to many different problems, including information and coding theory, limiting distributions, model selection, design of experiment and optimal prior distribution, data disclosure, and relative importance of predictors (Ebrahimi et al., 2010).

Original: In these applications, traditional research focused on both developing new estimators and obtaining provable guarantees on the asymptotic behavior of these estimators (Liu et al., 2012; Verdú, 2019).
Revised: In these applications, traditional research focused on both developing new estimators and obtaining provable guarantees on the asymptotic behavior of these estimators (Liu et al., 2012; Verdú, 2019).

Original: However, when used for training deep neural networks, additional requirements need to be satisﬁed.
Revised: However, when used for training deep neural networks, additional requirements need to be satisﬁed.

Original: In particular, the estimator needs to be differentiable w.r.t.
Revised: In particular, the estimator needs to be differentiable w.r.t.

Original: the data distribution (R1) , computationally tractable (R2) , and rapidly adapt to changes in the underlying distribution (R3) .
Revised: the data distribution (R1) , computationally tractable (R2) , and rapidly adapt to changes in the underlying distribution (R3) .

Original: For instance,
Revised: For instance,

Original: Mutual Information (MI), a fundamental measure of dependence between variables, only became a popular (standalone or regularizing) learning objective for DNNs once estimators satisfying the above requirements were proposed (Poole et al., 2019; Barber & Agakov, 2003).
Revised: Mutual Information (MI), a fundamental measure of dependence between variables, only became a popular (standalone or regularizing) learning objective for DNNs once estimators satisfying the above requirements were proposed (Poole et al., 2019; Barber & Agakov, 2003).

Original: Although MI is notoriously difﬁcult to estimate in high dimensions (Kraskov et al., 2004; Pichler et al., 2020;
Revised: Although MI is notoriously difﬁcult to estimate in high dimensions (Kraskov et al., 2004; Pichler et al., 2020;

Original: McAllester & Stratos, 2020), these estimators have demonstrated promising empirical results in unsupervised representation learning (Krause et al., 2010; Bridle et al., 1992; Hjelm et al., 2019;
Revised: McAllester & Stratos, 2020), these estimators have demonstrated promising empirical results in unsupervised representation learning (Krause et al., 2010; Bridle et al., 1992; Hjelm et al., 2019;

Original: Tschannen et al., 2020), discrete/invariant representations (Hu et al., 2017; Ji et al., 2019), generative modelling (Chen et al., 2016;
Revised: Tschannen et al., 2020), discrete/invariant representations (Hu et al., 2017; Ji et al., 2019), generative modelling (Chen et al., 2016;

Original: Zhao et al., 2017), textual disentangling (Cheng et al., 2020b; Colombo et al., 2021), and applications of the Information Bottleneck (IB) method (Mahabadi et al., 2021;
Revised: Zhao et al., 2017), textual disentangling (Cheng et al., 2020b; Colombo et al., 2021), and applications of the Information Bottleneck (IB) method (Mahabadi et al., 2021;

Original: Devlin et al., 2018;
Revised: Devlin et al., 2018;

Original: Alemi et al., 2016) among others.
Revised: Alemi et al., 2016) among others.

Original: Compared to MI, Differential Entropy (DE) has received less attention from the ML community while also having interesting applications.
Revised: Compared to MI, Differential Entropy (DE) has received less attention from the ML community while also having interesting applications.

Original: In this paper, we focus on the problem of DE estimation as this quantity naturally appears in many applications ( e.g. reinforcement learning (Shyam et al., 2019; Hazan et al., 2019;
Revised: In this paper, we focus on the problem of DE estimation as this quantity naturally appears in many applications ( e.g. reinforcement learning (Shyam et al., 2019; Hazan et al., 2019;

Original: Ahmed et al., 2019; Kim et al., 2019), IB (Alemi et al., 2016), mode collapse (Belghazi et al., 2018)).
Revised: Ahmed et al., 2019; Kim et al., 2019), IB (Alemi et al., 2016), mode collapse (Belghazi et al., 2018)).

Original: Traditional estimators of DE often violate at least one of the requirements (R1) – (R3) listed above ( e.g. knearest neighbor based estimators violate (R1) ).
Revised: Traditional estimators of DE often violate at least one of the requirements (R1) – (R3) listed above ( e.g. knearest neighbor based estimators violate (R1) ).

Original: As a consequence, the absence of DE estimator for arbitrary data distributions forces deep learning researchers to either restrict themselves to special cases where closed-form expressions for DE are available (Shyam et al., 2019) or use MI as a proxy
Revised: As a consequence, the absence of DE estimator for arbitrary data distributions forces deep learning researchers to either restrict themselves to special cases where closed-form expressions for DE are available (Shyam et al., 2019) or use MI as a proxy

Original: (Belghazi et al., 2018).
Revised: (Belghazi et al., 2018).

Original: In this work, we introduce a Kernelized Neural dIFferential Entropy (K NIFE ) estimator, that satisﬁes the aforementioned requirements and addresses limitations of existing DE estimators (Schraudolph, 2004; McAllester & Stratos, 2020).
Revised: In this work, we introduce a Kernelized Neural dIFferential Entropy (K NIFE ) estimator, that satisﬁes the aforementioned requirements and addresses limitations of existing DE estimators (Schraudolph, 2004; McAllester & Stratos, 2020).

Original: Stemming from recent theoretical insights (McAllester & Stratos, 2020) that justify the use of DE estimators as building blocks to better estimate MI, we further apply K NIFE to MI estimation.
Revised: Stemming from recent theoretical insights (McAllester & Stratos, 2020) that justify the use of DE estimators as building blocks to better estimate MI, we further apply K NIFE to MI estimation.

Original: In the context of deep neural networks with high dimensional data ( e.g. image, text), K NIFE achieves competitive empirical results in applications where DE or MI is required.
Revised: In the context of deep neural networks with high dimensional data ( e.g. image, text), K NIFE achieves competitive empirical results in applications where DE or MI is required.

Original: 1. C ONTRIBUTIONS
Revised: 1. C ONTRIBUTIONS

Original: Our work advances methods in DE and MI estimation in several ways.
Revised: Our work advances methods in DE and MI estimation in several ways.

Original: We showcase limitation of the existing DE estimators proposed in Schraudolph (2004); McAllester & Stratos (2020) with respect to desirable properties required for training deep neural networks.
Revised: We showcase limitation of the existing DE estimators proposed in Schraudolph (2004); McAllester & Stratos (2020) with respect to desirable properties required for training deep neural networks.

Original: To address these shortcomings, we introduce K NIFE , a fully learnable kernel-based estimator of DE.
Revised: To address these shortcomings, we introduce K NIFE , a fully learnable kernel-based estimator of DE.

Original: The ﬂexibility of K NIFE allows us to construct K NIFE -based estimators for conditional DE, conditioning on either a discrete or continuous random variable.
Revised: The ﬂexibility of K NIFE allows us to construct K NIFE -based estimators for conditional DE, conditioning on either a discrete or continuous random variable.

Original: We prove learnability under natural conditions on the underlying probability distribution.
Revised: We prove learnability under natural conditions on the underlying probability distribution.

Original: By requiring a ﬁxed Lipschitz condition and bounded support we are not only able to provide an asymptotic result, but also a conﬁdence bound in the case of a ﬁnite training set.
Revised: By requiring a ﬁxed Lipschitz condition and bounded support we are not only able to provide an asymptotic result, but also a conﬁdence bound in the case of a ﬁnite training set.

Original: This extends the consistency result by Ahmad & Lin (1976).
Revised: This extends the consistency result by Ahmad & Lin (1976).

Original: We validate on synthetic datasets (including multi-modal, non-Gaussian distributions), that K NIFE addresses the identiﬁed limitations and outperforms existing methods on both DE and MI estimation.
Revised: We validate on synthetic datasets (including multi-modal, non-Gaussian distributions), that K NIFE addresses the identiﬁed limitations and outperforms existing methods on both DE and MI estimation.

Original: In particular, K NIFE more rapidly adapts to changes in the underlying data distribution.
Revised: In particular, K NIFE more rapidly adapts to changes in the underlying data distribution.

Original: We conduct extensive experiments on natural datasets (including text and images) to compare K NIFE -based
Revised: We conduct extensive experiments on natural datasets (including text and images) to compare K NIFE -based

Original: MI estimators to most recent MI estimators.
Revised: MI estimators to most recent MI estimators.

Original: First, we apply K NIFE in the IB principle to ﬁne-tune a pretrained language
Revised: First, we apply K NIFE in the IB principle to ﬁne-tune a pretrained language

Original: model.
Revised: model.

Original: Using K NIFE , we leverage a closed-form expression of a part of the training objective and achieve the best scores among competing MI estimators.
Revised: Using K NIFE , we leverage a closed-form expression of a part of the training objective and achieve the best scores among competing MI estimators.

Original: Second, on fair textual classiﬁcation, the K NIFE -based MI estimator achieves near perfect disentanglement (with respect to the private, discrete label) at virtually no degradation of accuracy in the main task.
Revised: Second, on fair textual classiﬁcation, the K NIFE -based MI estimator achieves near perfect disentanglement (with respect to the private, discrete label) at virtually no degradation of accuracy in the main task.

Original: Lastly, in the challenging scenario of visual domain adaptation, where both variables are continuous,
Revised: Lastly, in the challenging scenario of visual domain adaptation, where both variables are continuous,

Original: K NIFE -based MI estimation also achieves superior results.
Revised: K NIFE -based MI estimation also achieves superior results.

Original: 1. E XISTENT M ETHODS AND R ELATED W ORKS
Revised: 1. E XISTENT M ETHODS AND R ELATED W ORKS

Original: DE estimation.
Revised: DE estimation.

Original: Existing methods for estimating DE ﬁt into one of three categories (Beirlant et al., 1997; Hlavá ˇ cková-Schindler et al., 2007; Verdú, 2019): plug-in estimates (Ahmad & Lin, 1976; Györﬁ & Van der Meulen, 1987), estimates based on sample-spacings (Tarasenko, 1968), and estimates based on nearest neighbor distances (Kozachenko & Leonenko, 1987;
Revised: Existing methods for estimating DE ﬁt into one of three categories (Beirlant et al., 1997; Hlavá ˇ cková-Schindler et al., 2007; Verdú, 2019): plug-in estimates (Ahmad & Lin, 1976; Györﬁ & Van der Meulen, 1987), estimates based on sample-spacings (Tarasenko, 1968), and estimates based on nearest neighbor distances (Kozachenko & Leonenko, 1987;

Original: Tsybakov &
Revised: Tsybakov &

Original: Van der Meulen, 1996); (Berrett et al., 2019).
Revised: Van der Meulen, 1996); (Berrett et al., 2019).

Original: Our proposed estimator falls into the ﬁrst category and we will thus focus here on previous work using that methodology.
Revised: Our proposed estimator falls into the ﬁrst category and we will thus focus here on previous work using that methodology.

Original: Excellent summaries of all the available methods can be found in the works (Beirlant et al., 1997; Hlavá ˇ cková-Schindler et al., 2007; Wang et al., 2009; Verdú, 2019).
Revised: Excellent summaries of all the available methods can be found in the works (Beirlant et al., 1997; Hlavá ˇ cková-Schindler et al., 2007; Wang et al., 2009; Verdú, 2019).

Original: In Ahmad & Lin (1976), a ﬁrst nonparametric estimator of
Revised: In Ahmad & Lin (1976), a ﬁrst nonparametric estimator of

Original: DE was suggested and theoretically analyzed.
Revised: DE was suggested and theoretically analyzed.

Original: It builds on the idea of kernel density estimation using
Revised: It builds on the idea of kernel density estimation using

Original: Parzen-Rosenblatt windowing (Rosenblatt, 1956; Parzen, 1962).
Revised: Parzen-Rosenblatt windowing (Rosenblatt, 1956; Parzen, 1962).

Original: More detailed analysis followed (Joe, 1989; Hall & Morton, 1993) but the estimator remained essentially unchanged.
Revised: More detailed analysis followed (Joe, 1989; Hall & Morton, 1993) but the estimator remained essentially unchanged.

Original: Unfortunately, this classical literature is mostly concerned with appropriate regularity conditions that guarantee asymptotic properties of estimators, such as (asymptotic) unbiasedness and consistency.
Revised: Unfortunately, this classical literature is mostly concerned with appropriate regularity conditions that guarantee asymptotic properties of estimators, such as (asymptotic) unbiasedness and consistency.

Original: Machine learning applications, however, usually deal with a ﬁxed—often very limited—number of samples.
Revised: Machine learning applications, however, usually deal with a ﬁxed—often very limited—number of samples.

Original: Differentiable DE estimation.
Revised: Differentiable DE estimation.

Original: A ﬁrst estimator that employed a differential learning rule was introduced in Viola et al.
Revised: A ﬁrst estimator that employed a differential learning rule was introduced in Viola et al.

Original: Indeed, the estimator proposed therein is optimized using stochastic optimization, it only used a single kernel with a low number of parameters.
Revised: Indeed, the estimator proposed therein is optimized using stochastic optimization, it only used a single kernel with a low number of parameters.

Original: An extension that uses a heteroscedastic kernel density estimate, i.e., using different kernels at different positions, has been proposed in Schraudolph (2004).
Revised: An extension that uses a heteroscedastic kernel density estimate, i.e., using different kernels at different positions, has been proposed in Schraudolph (2004).

Original: Still the number of parameters was quite low and varying means in the kernels or variable weights were not considered.
Revised: Still the number of parameters was quite low and varying means in the kernels or variable weights were not considered.

Original: Although the estimation of
Revised: Although the estimation of

Original: DE remained a topic of major interest as illustrated by recent works focusing on special classes of distributions (Kolchinsky & Tracey, 2017; Chaubey & Vu, 2021) and nonparametric estimators (Sricharan et al., 2013; Kandasamy et al., 2015; Moon et al., 2021), the estimator introduced in
Revised: DE remained a topic of major interest as illustrated by recent works focusing on special classes of distributions (Kolchinsky & Tracey, 2017; Chaubey & Vu, 2021) and nonparametric estimators (Sricharan et al., 2013; Kandasamy et al., 2015; Moon et al., 2021), the estimator introduced in

Original: Schraudolph (2004) was not further reﬁned and hardly explored in recent works.
Revised: Schraudolph (2004) was not further reﬁned and hardly explored in recent works.

Original: Differentiable MI estimation.
Revised: Differentiable MI estimation.

Original: In contrast, there has been a recent surge on new methods for the estimation of the closely related MI between two random variables.
Revised: In contrast, there has been a recent surge on new methods for the estimation of the closely related MI between two random variables.

Original: The most prominent examples include unnormalized energy-based variational lower bounds (Poole et al., 2019), the lower bounds developed in Nguyen et al.
Revised: The most prominent examples include unnormalized energy-based variational lower bounds (Poole et al., 2019), the lower bounds developed in Nguyen et al.

Original: (2010) using variational characterization of f-divergence, the MINEestimator developed in Belghazi et al.
Revised: (2010) using variational characterization of f-divergence, the MINEestimator developed in Belghazi et al.

Original: (2018) from the Donsker-Varadhan representation of MI which can be also interpreted as an improvement of the plug-in estimator of Suzuki et al.
Revised: (2018) from the Donsker-Varadhan representation of MI which can be also interpreted as an improvement of the plug-in estimator of Suzuki et al.

Original: (2008), the noise-contrastive based bound developed in van den Oord et al.
Revised: (2008), the noise-contrastive based bound developed in van den Oord et al.

Original: (2018) and ﬁnally a contrastive upper bound (Cheng et al., 2020a).
Revised: (2018) and ﬁnally a contrastive upper bound (Cheng et al., 2020a).

Original: McAllester & Stratos (2020) point out shortcomings in other estimation strategies and introduce their own Differences of Entropies (D O E) method.
Revised: McAllester & Stratos (2020) point out shortcomings in other estimation strategies and introduce their own Differences of Entropies (D O E) method.

Original:  K NIFE
Revised:  K NIFE

Original: In this section we identify limitations of existing entropy estimators introduced in Schraudolph (2004);
Revised: In this section we identify limitations of existing entropy estimators introduced in Schraudolph (2004);

Original: McAllester & Stratos (2020).
Revised: McAllester & Stratos (2020).

Original: Subsequently, we present K NIFE , which addresses these shortcomings.
Revised: Subsequently, we present K NIFE , which addresses these shortcomings.

Original: 2. L IMITATIONS OF E XISTING D IFFERENTIAL E NTROPY E STIMATORS
Revised: 2. L IMITATIONS OF E XISTING D IFFERENTIAL E NTROPY E STIMATORS

Original: Consider a continuous random variable X ∼ p .
Revised: Consider a continuous random variable X ∼ p .

Original: Our goal is to estimate the DE h( X ) :
Revised: Our goal is to estimate the DE h( X ) :

Original: = − (cid:82) p ( x ) log p ( x ) d x. Given the intractability of this integral, we will rely on a Monte-Carlo estimate of h( X ) , using N i.i.d.
Revised: = − (cid:82) p ( x ) log p ( x ) d x. Given the intractability of this integral, we will rely on a Monte-Carlo estimate of h( X ) , using N i.i.d.

Original: samples D x = { x n } Nn =1 to obtain
Revised: samples D x = { x n } Nn =1 to obtain

Original: [Equation]
Revised: [Equation]

Original: Unfortunately, assuming access to the true density p is often unrealistic, and we will thus construct an estimate ˆ p that can then be plugged into (1) instead of p .
Revised: Unfortunately, assuming access to the true density p is often unrealistic, and we will thus construct an estimate ˆ p that can then be plugged into (1) instead of p .

Original: If ˆ p is smooth, the resulting plug-in estimator of DE is differentiable (R1) .
Revised: If ˆ p is smooth, the resulting plug-in estimator of DE is differentiable (R1) .

Original: Assuming access to an additional—ideally independent—set of M i.i.d.
Revised: Assuming access to an additional—ideally independent—set of M i.i.d.

Original: samples E = { x (cid:48) m } Mm =1 , we build upon the Parzen-Rosenblatt estimator (Rosenblatt, 1956; Parzen, 1962)
Revised: samples E = { x (cid:48) m } Mm =1 , we build upon the Parzen-Rosenblatt estimator (Rosenblatt, 1956; Parzen, 1962)

Original: [Equation]
Revised: [Equation]

Original: where w > 0 denotes the bandwidth and κ is a kernel density.
Revised: where w > 0 denotes the bandwidth and κ is a kernel density.

Original: The resulting entropy estimator when replacing p in (1) by (2) was analyzed in Ahmad & Lin (1976).
Revised: The resulting entropy estimator when replacing p in (1) by (2) was analyzed in Ahmad & Lin (1976).

Original: In Schraudolph (2004), this approach was extended using the kernel estimator
Revised: In Schraudolph (2004), this approach was extended using the kernel estimator

Original: [Equation]
Revised: [Equation]

Original: where A :=
Revised: where A :=

Original: ( A 1 , . . . , A M ) are (distinct, diagonal) covariance matrices and κ A ( x ) =
Revised: ( A 1 , . . . , A M ) are (distinct, diagonal) covariance matrices and κ A ( x ) =

Original: N ( x ; 0 , A ) is a centered Gaussian density with covariance matrix A .
Revised: N ( x ; 0 , A ) is a centered Gaussian density with covariance matrix A .

Original: The D O E method of McAllester & Stratos (2020) is a MI estimator that separately estimates a DE and a conditional DE.
Revised: The D O E method of McAllester & Stratos (2020) is a MI estimator that separately estimates a DE and a conditional DE.

Original: For DE, a simple Gaussian density estimate
Revised: For DE, a simple Gaussian density estimate

Original: ˆ p D O E ( x ; θ ) =
Revised: ˆ p D O E ( x ; θ ) =

Original: κ A ( x − µ ) is used, where θ =
Revised: κ A ( x − µ ) is used, where θ =

Original: ( A, µ ) are the training parameters, the diagonal covariance matrix A and the mean µ .
Revised: ( A, µ ) are the training parameters, the diagonal covariance matrix A and the mean µ .

Original: While both S CHRAU .
Revised: While both S CHRAU .

Original: and D O E yield differentiable plug-in estimators for DE, they each have a major disadvantage.
Revised: and D O E yield differentiable plug-in estimators for DE, they each have a major disadvantage.

Original: The strategy of Schraudolph (2004) ﬁxes the kernel mean values at E , which implies that the method cannot adapt to a shifting input distribution (R3) .
Revised: The strategy of Schraudolph (2004) ﬁxes the kernel mean values at E , which implies that the method cannot adapt to a shifting input distribution (R3) .

Original: On the other hand, D O E allows for rapid adaptation, but its simple structure makes it inadequate for the DE estimation of multi-modal densities.
Revised: On the other hand, D O E allows for rapid adaptation, but its simple structure makes it inadequate for the DE estimation of multi-modal densities.

Original: We illustrate these limitations in Section 3.1.
Revised: We illustrate these limitations in Section 3.1.

Original: 2. K NIFE E STIMATOR
Revised: 2. K NIFE E STIMATOR

Original: In K NIFE , the kernel density estimate is given by
Revised: In K NIFE , the kernel density estimate is given by

Original: [Equation]
Revised: [Equation]

Original: where θ :=
Revised: where θ :=

Original: ( A , a , u ) and the additional parameters 0 ≤ u = ( u 1 , u 2 , . . .
Revised: ( A , a , u ) and the additional parameters 0 ≤ u = ( u 1 , u 2 , . . .

Original: , u M ) with 1 · u = 1 and a = ( a 1 , . . .
Revised: , u M ) with 1 · u = 1 and a = ( a 1 , . . .

Original: , a M ) are introduced.
Revised: , a M ) are introduced.

Original: Note that ˆ p K NIFE ( x ; θ ) is a smooth function of θ , and so is our proposed plug-in estimator
Revised: Note that ˆ p K NIFE ( x ; θ ) is a smooth function of θ , and so is our proposed plug-in estimator

Original: [Equation]
Revised: [Equation]

Original: K NIFE combines the ideas of Schraudolph (2004); McAllester & Stratos (2020).
Revised: K NIFE combines the ideas of Schraudolph (2004); McAllester & Stratos (2020).

Original: It is differentiable and able to adapt to shifting input distributions, while capable of matching multi-modal distributions.
Revised: It is differentiable and able to adapt to shifting input distributions, while capable of matching multi-modal distributions.

Original: Thus, as we will see in synthetic experiments, incorporating u m and shifts a m in the optimization enables the use of K NIFE in a non-stationary settings, where the distribution of X evolves over time.
Revised: Thus, as we will see in synthetic experiments, incorporating u m and shifts a m in the optimization enables the use of K NIFE in a non-stationary settings, where the distribution of X evolves over time.

Original: Learning step:
Revised: Learning step:

Original: Stemming from the observation that, by the Law of Large Numbers (LLN),
Revised: Stemming from the observation that, by the Law of Large Numbers (LLN),

Original: [Equation]
Revised: [Equation]

Original: we propose to learn the parameters θ by minimizing (cid:98) h K NIFE , where E may be used to initialize a .
Revised: we propose to learn the parameters θ by minimizing (cid:98) h K NIFE , where E may be used to initialize a .

Original: Although not strictly equivalent due to the Monte-Carlo approximation, minimizing (cid:98)
Revised: Although not strictly equivalent due to the Monte-Carlo approximation, minimizing (cid:98)

Original: h K NIFE can be understood as minimizing the Kullback-Leibler (KL) divergence in (6), effectively minimizing the gap between (cid:98) h K NIFE and h( X ) .
Revised: h K NIFE can be understood as minimizing the Kullback-Leibler (KL) divergence in (6), effectively minimizing the gap between (cid:98) h K NIFE and h( X ) .

Original: In fact, (cid:98) h K NIFE can also be interpreted as the standard maximum likelihood objective, widely used in modern machine learning.
Revised: In fact, (cid:98) h K NIFE can also be interpreted as the standard maximum likelihood objective, widely used in modern machine learning.

Original: It is worth to mention that the K NIFE estimator is fully differentiable with respect to θ and the optimization can be tackled by any gradient-based method (e.g., Adam (Kingma & Ba, 2014) or AdamW (Loshchilov & Hutter, 2017)).
Revised: It is worth to mention that the K NIFE estimator is fully differentiable with respect to θ and the optimization can be tackled by any gradient-based method (e.g., Adam (Kingma & Ba, 2014) or AdamW (Loshchilov & Hutter, 2017)).

Original: 2. C ONVERGENCE A NALYSIS
Revised: 2. C ONVERGENCE A NALYSIS

Original: Note that the classical Parzen-Rosenblatt estimator (cid:98) h( D x ; w ) , where (2) is plugged into (1), is a special case of K NIFE .
Revised: Note that the classical Parzen-Rosenblatt estimator (cid:98) h( D x ; w ) , where (2) is plugged into (1), is a special case of K NIFE .

Original: Thus, the convergence analysis provided in (Ahmad & Lin, 1976, Theorem 1) also applies and yields sufﬁcient conditions for (cid:98)
Revised: Thus, the convergence analysis provided in (Ahmad & Lin, 1976, Theorem 1) also applies and yields sufﬁcient conditions for (cid:98)

Original: h K NIFE ( D x , θ )
Revised: h K NIFE ( D x , θ )

Original: → h( X ) .
Revised: → h( X ) .

Original: In Appendix C, we extend this result and, assuming that the underlying distribution p is compactly supported and L -Lipschitz continuous, the following theorem is proved.
Revised: In Appendix C, we extend this result and, assuming that the underlying distribution p is compactly supported and L -Lipschitz continuous, the following theorem is proved.

Original: Theorem 1.
Revised: Theorem 1.

Original: For any δ > 0 , there exists a function ε ( N, M, w ) such that, with probability at least 1 − δ ,
Revised: For any δ > 0 , there exists a function ε ( N, M, w ) such that, with probability at least 1 − δ ,

Original: (cid:12)(cid:12)(cid:98) h
Revised: (cid:12)(cid:12)(cid:98) h

Original: ( D x ; w ) − h( X ) (cid:12)(cid:12) ≤ ε ( N, M, w ) .
Revised: ( D x ; w ) − h( X ) (cid:12)(cid:12) ≤ ε ( N, M, w ) .

Original: Additionally, ε ( N, M, w ) → 0 as M,N → ∞ and w →provided that
Revised: Additionally, ε ( N, M, w ) → 0 as M,N → ∞ and w →provided that

Original: Nw →
Revised: Nw →

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: where M and N denote the number of samples in E and D x , respectively.
Revised: where M and N denote the number of samples in E and D x , respectively.

Original: An explicit formula for ε ( N, M, w ) is also given in Theorem 2 in Appendix C. For instance, Theorem 1 provides a bound on the speed of convergence for the consistency analysis in (Ahmad & Lin, 1976, Theorem 1).
Revised: An explicit formula for ε ( N, M, w ) is also given in Theorem 2 in Appendix C. For instance, Theorem 1 provides a bound on the speed of convergence for the consistency analysis in (Ahmad & Lin, 1976, Theorem 1).

Original: 2. E STIMATING C ONDITIONAL D IFFERENTIAL E NTROPY AND M UTUAL I NFORMATION
Revised: 2. E STIMATING C ONDITIONAL D IFFERENTIAL E NTROPY AND M UTUAL I NFORMATION

Original: Similar to (McAllester & Stratos, 2020), the proposed DE estimator can be used to estimate other information measures.
Revised: Similar to (McAllester & Stratos, 2020), the proposed DE estimator can be used to estimate other information measures.

Original: In particular, we can use K NIFE to construct estimators of conditional DE and
Revised: In particular, we can use K NIFE to construct estimators of conditional DE and

Original: MI.
Revised: MI.

Original: When estimating the conditional DE and MI for a pair of random variables ( X, Y ) ∼ p , we not only use D x = { x n } Nn =1 , but also the according i.i.d.
Revised: When estimating the conditional DE and MI for a pair of random variables ( X, Y ) ∼ p , we not only use D x = { x n } Nn =1 , but also the according i.i.d.

Original: samples D y = { y n } Nn =1 , where ( x n , y n ) are drawn according to p .
Revised: samples D y = { y n } Nn =1 , where ( x n , y n ) are drawn according to p .

Original: Conditional Differential Entropy.
Revised: Conditional Differential Entropy.

Original: We estimate conditional DE h( X | Y ) by considering θ to be a parameterized function Θ ( y ) of y .
Revised: We estimate conditional DE h( X | Y ) by considering θ to be a parameterized function Θ ( y ) of y .

Original: Then all relations previously established naturally generalize and
Revised: Then all relations previously established naturally generalize and

Original: [Equation]
Revised: [Equation]

Original: Naturally, minimization of (6) is now performed over the parameters of Θ .
Revised: Naturally, minimization of (6) is now performed over the parameters of Θ .

Original: If Y is a continuous random variable, we use an artiﬁcial neural network Θ ( y ) , taking y as its input.
Revised: If Y is a continuous random variable, we use an artiﬁcial neural network Θ ( y ) , taking y as its input.

Original: On the other hand, if
Revised: On the other hand, if

Original: Y ∈ Y is a discrete random variable, we have one parameter θ for each y ∈ Y , i.e., Θ = { θ y } y ∈Y and ˆ p K NIFE ( x | y ; Θ )
Revised: Y ∈ Y is a discrete random variable, we have one parameter θ for each y ∈ Y , i.e., Θ = { θ y } y ∈Y and ˆ p K NIFE ( x | y ; Θ )

Original: =
Revised: =

Original: ˆ p K NIFE ( x ; Θ ( y )) =
Revised: ˆ p K NIFE ( x ; Θ ( y )) =

Original: ˆ p K NIFE ( x ; θ y ) .
Revised: ˆ p K NIFE ( x ; θ y ) .

Original: Mutual Information.
Revised: Mutual Information.

Original: To estimate the MI between random variables X and Y (either discrete or continuous), recall that MI can be written as I( X ; Y )
Revised: To estimate the MI between random variables X and Y (either discrete or continuous), recall that MI can be written as I( X ; Y )

Original: =
Revised: =

Original: h( X ) − h
Revised: h( X ) − h

Original: ( X | Y ) .
Revised: ( X | Y ) .

Original: Therefore, we use the marginal and conditional DE estimators (5) and (8) to build a K NIFE -based MI estimator (cid:98)
Revised: Therefore, we use the marginal and conditional DE estimators (5) and (8) to build a K NIFE -based MI estimator (cid:98)

Original: I K NIFE
Revised: I K NIFE

Original: ( D x , D y ; θ , Θ ) :
Revised: ( D x , D y ; θ , Θ ) :

Original: = (cid:98)
Revised: = (cid:98)

Original: [Equation]
Revised: [Equation]

Original: K NIFE ( D
Revised: K NIFE ( D

Original: [Equation]
Revised: [Equation]

Original: − (cid:98)
Revised: − (cid:98)

Original: [Equation]
Revised: [Equation]

Original: K NIFE ( D
Revised: K NIFE ( D

Original: [Equation]
Revised: [Equation]

Original: |D y ; Θ ) .
Revised: |D y ; Θ ) .

Original:  E XPERIMENTS USING S YNTHETIC D ATA 3. D IFFERENTIAL E NTROPY E STIMATION
Revised:  E XPERIMENTS USING S YNTHETIC D ATA 3. D IFFERENTIAL E NTROPY E STIMATION

Original: In this section we apply K NIFE for DE estimation, comparing it to (3), the method introduced in Schraudolph (2004), subsequently labeled “S CHRAU .”.
Revised: In this section we apply K NIFE for DE estimation, comparing it to (3), the method introduced in Schraudolph (2004), subsequently labeled “S CHRAU .”.

Original: It is worth to mention that we did not perform the Expectation Maximization algorithm, as suggested in (Schraudolph, 2004), but instead opted to use the same optimization technique as for K NIFE to facilitate a fair comparison.
Revised: It is worth to mention that we did not perform the Expectation Maximization algorithm, as suggested in (Schraudolph, 2004), but instead opted to use the same optimization technique as for K NIFE to facilitate a fair comparison.

Original: 3.1. G AUSSIAN D ISTRIBUTION
Revised: 3.1. G AUSSIAN D ISTRIBUTION

Original: As a sanity check, we test K NIFE on multivariate normal data in moderately high dimensions, comparing it to S CHRAU .
Revised: As a sanity check, we test K NIFE on multivariate normal data in moderately high dimensions, comparing it to S CHRAU .

Original: and D O E, which we trained with the exact same parameters.
Revised: and D O E, which we trained with the exact same parameters.

Original: We performed these experiments with d = 10 and d = 64 dimensional data.
Revised: We performed these experiments with d = 10 and d = 64 dimensional data.

Original: K NIFE yielded the lowest bias and variance in both cases, despite D O E being perfectly adapted to matching a multivariate Gaussian distribution.
Revised: K NIFE yielded the lowest bias and variance in both cases, despite D O E being perfectly adapted to matching a multivariate Gaussian distribution.

Original: Additional details can be found in Appendix A.1.
Revised: Additional details can be found in Appendix A.1.

Original: In order to use a DE estimation primitive in a machine learning system, it must be able to adapt to a changing input distribution during training (R3) .
Revised: In order to use a DE estimation primitive in a machine learning system, it must be able to adapt to a changing input distribution during training (R3) .

Original: As already pointed out in Section 2.1, this is a severe limitation of S CHRAU .
Revised: As already pointed out in Section 2.1, this is a severe limitation of S CHRAU .

Original: , as re-drawing the kernel support E can be either impractical or at the very least requires a complete re-training of the entropy estimator.
Revised: , as re-drawing the kernel support E can be either impractical or at the very least requires a complete re-training of the entropy estimator.

Original: Whereas in (4), the kernel support a is trainable and it can thus adapt to a change of the input distribution.
Revised: Whereas in (4), the kernel support a is trainable and it can thus adapt to a change of the input distribution.

Original: In order to showcase this ability, we utilize the approach of Cheng et al.
Revised: In order to showcase this ability, we utilize the approach of Cheng et al.

Original: (2020a)
Revised: (2020a)

Original: and successively decrease the entropy, observing how the estimator adapts.
Revised: and successively decrease the entropy, observing how the estimator adapts.

Original: We perform this experiment with data of dimension d = 64 and repeatedly multiply the covariance matrix of the training vectors with a factor of a = 12 .
Revised: We perform this experiment with data of dimension d = 64 and repeatedly multiply the covariance matrix of the training vectors with a factor of a = 12 .

Original: The resulting entropy estimation is depicted in Figure 1.
Revised: The resulting entropy estimation is depicted in Figure 1.

Original: It is apparent that S CHRAU .
Revised: It is apparent that S CHRAU .

Original: suffers from a varying bias.
Revised: suffers from a varying bias.

Original: The bias increases with decreasing variance, as the kernel support is ﬁxed and cannot adapt as the variance of D x shrinks.
Revised: The bias increases with decreasing variance, as the kernel support is ﬁxed and cannot adapt as the variance of D x shrinks.

Original: D O E is perfectly adapted to a single Gaussian distribution and performs similar to K NIFE .
Revised: D O E is perfectly adapted to a single Gaussian distribution and performs similar to K NIFE .

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 3.1. T RIANGLE M IXTURE
Revised: 3.1. T RIANGLE M IXTURE

Original: K NIFE is able to cope with distributions that have multiple modes.
Revised: K NIFE is able to cope with distributions that have multiple modes.

Original: While (3) is also capable of matching multi-modal distributions, D O E is unable to do so, as it approximates any distribution with a multivariate Gaussian.
Revised: While (3) is also capable of matching multi-modal distributions, D O E is unable to do so, as it approximates any distribution with a multivariate Gaussian.

Original: We illustrate this by matching a mixture of randomly drawn triangle distributions.
Revised: We illustrate this by matching a mixture of randomly drawn triangle distributions.

Original: The resulting estimated PDFs as well as the ground truth when estimating the entropy of a 1 -dimensional mixture of triangles with 10 components can be observed in Figure 2 (left).
Revised: The resulting estimated PDFs as well as the ground truth when estimating the entropy of a 1 -dimensional mixture of triangles with 10 components can be observed in Figure 2 (left).

Original: With increasing dimension the difﬁculty of this estimation rises quickly as in d dimensions, the resulting
Revised: With increasing dimension the difﬁculty of this estimation rises quickly as in d dimensions, the resulting

Original: PDF of independent c -component triangle mixtures has c d modes.
Revised: PDF of independent c -component triangle mixtures has c d modes.

Original: To showcase the performance of
Revised: To showcase the performance of

Original: K NIFE in this challenging task, we ran 10 training runs for DE estimation of 2 -component triangle mixtures in 8 dimensions.
Revised: K NIFE in this challenging task, we ran 10 training runs for DE estimation of 2 -component triangle mixtures in 8 dimensions.

Original: An example training run is depicted in Figure 2 (right).
Revised: An example training run is depicted in Figure 2 (right).

Original: [Figure]
Revised: [Figure]

Original: KNIFE
Revised: KNIFE

Original: 3. M UTUAL I NFORMATION E STIMATION
Revised: 3. M UTUAL I NFORMATION E STIMATION

Original: Multivariate Gauss We repeat the experiments in (Cheng et al., 2020a), stepping up the MI
Revised: Multivariate Gauss We repeat the experiments in (Cheng et al., 2020a), stepping up the MI

Original: I( X d ; Y d ) between d i.i
Revised: I( X d ; Y d ) between d i.i

Original: [Figure]
Revised: [Figure]

Original: d. copies of joint normal random variables ( X, Y ) by increasing their correlation coefﬁcient, i.e., ( X, Y ) are multivariate Gaussian with correlation coefﬁcient ρ i in the i -th epoch.
Revised: d. copies of joint normal random variables ( X, Y ) by increasing their correlation coefﬁcient, i.e., ( X, Y ) are multivariate Gaussian with correlation coefﬁcient ρ i in the i -th epoch.

Original: A training run is depicted in the top of Figure 3.
Revised: A training run is depicted in the top of Figure 3.

Original: As in (Cheng et al., 2020a), we also repeat the experiment, applying a cubic transformation to Y .
Revised: As in (Cheng et al., 2020a), we also repeat the experiment, applying a cubic transformation to Y .

Original: The estimation of MI between d i.i.d.
Revised: The estimation of MI between d i.i.d.

Original: copies of X and Y 3 can be observed in the middle row of Figure 3.
Revised: copies of X and Y 3 can be observed in the middle row of Figure 3.

Original: The MI is unaffected by this bijective transformation.
Revised: The MI is unaffected by this bijective transformation.

Original: In Appendix A.3, the bias and variance are depicted separately.
Revised: In Appendix A.3, the bias and variance are depicted separately.

Original: Sum of Uniformly Distributed Variables In order to test the ability of K NIFE to adapt to distributions substantially different from the Gaussian kernel shape, we apply it in MI estimation of I( X d ; Y d ) with uniformly distributed data.
Revised: Sum of Uniformly Distributed Variables In order to test the ability of K NIFE to adapt to distributions substantially different from the Gaussian kernel shape, we apply it in MI estimation of I( X d ; Y d ) with uniformly distributed data.

Original: To this end, let X and E be centered, uniformly distributed random variables with E [ X 2 ]
Revised: To this end, let X and E be centered, uniformly distributed random variables with E [ X 2 ]

Original: =
Revised: =

Original: E [ E 2 ] = 1 and deﬁne Y =
Revised: E [ E 2 ] = 1 and deﬁne Y =

Original: ρ i X
Revised: ρ i X

Original: + (cid:112) 1 − ρ 2 i E in the i -th epoch.
Revised: + (cid:112) 1 − ρ 2 i E in the i -th epoch.

Original: One training run with d = 20 is shown in Figure 3 (bottom).
Revised: One training run with d = 20 is shown in Figure 3 (bottom).

Original: Details about the source distribution as well as details of the experiments can be found in Appendix A.3.
Revised: Details about the source distribution as well as details of the experiments can be found in Appendix A.3.

Original:  E XPERIMENTS ON N ATURAL D ATA
Revised:  E XPERIMENTS ON N ATURAL D ATA

Original: In this section, we benchmark our proposed K NIFE -based MI estimator on three practical applications, spanning textual and visual data.
Revised: In this section, we benchmark our proposed K NIFE -based MI estimator on three practical applications, spanning textual and visual data.

Original: We reproduce and compare our method to the most recent MI estimators including MINE (Belghazi et al., 2018), NWJ (Nguyen et al., 2010), InfoNCE (van den
Revised: We reproduce and compare our method to the most recent MI estimators including MINE (Belghazi et al., 2018), NWJ (Nguyen et al., 2010), InfoNCE (van den

Original: Oord et al., 2018), CLUB (Cheng et al., 2020a), and D O E (McAllester & Stratos, 2020).
Revised: Oord et al., 2018), CLUB (Cheng et al., 2020a), and D O E (McAllester & Stratos, 2020).

Original: We do not explicitly include the SMILE estimator Song & Ermon (2019) in our comparison as it has the same gradient as NWJ.
Revised: We do not explicitly include the SMILE estimator Song & Ermon (2019) in our comparison as it has the same gradient as NWJ.

Original: Common notation :
Revised: Common notation :

Original: In all following applications, we will use Φ ψ : X → Z to denote an encoder, where X is the raw input space (i.e., texts or images), and Z denotes a lower dimensional continuous feature space.
Revised: In all following applications, we will use Φ ψ : X → Z to denote an encoder, where X is the raw input space (i.e., texts or images), and Z denotes a lower dimensional continuous feature space.

Original: Additionally, we will use C ψ : Z → Y to denote a shallow classiﬁer from the latent space Z to a discrete or continuous target space Y for classiﬁcation or regression, respectively.
Revised: Additionally, we will use C ψ : Z → Y to denote a shallow classiﬁer from the latent space Z to a discrete or continuous target space Y for classiﬁcation or regression, respectively.

Original: We will use ψ to denote the parameters of both models, Φ ψ and C ψ .
Revised: We will use ψ to denote the parameters of both models, Φ ψ and C ψ .

Original: CE denotes the cross entropy loss.
Revised: CE denotes the cross entropy loss.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: I NFORMATION B OTTLENECK FOR L ANGUAGE M ODEL F INETUNING
Revised: I NFORMATION B OTTLENECK FOR L ANGUAGE M ODEL F INETUNING

Original: IB has recently been applied to ﬁne-tune large-scale pretrained models (Mahabadi et al., 2021) such as
Revised: IB has recently been applied to ﬁne-tune large-scale pretrained models (Mahabadi et al., 2021) such as

Original: BERT (Devlin et al., 2018) and aims at suppressing irrelevant features in order to reduce overﬁtting.
Revised: BERT (Devlin et al., 2018) and aims at suppressing irrelevant features in order to reduce overﬁtting.

Original: Problem statement.
Revised: Problem statement.

Original: Given a textual input X ∈ X and a target label Y ∈ Y , the goal is to learn the encoder Φ ψ and classiﬁer C ψ , such that Φ ψ ( X ) retains little information about X , while still producing discriminative features, allowing the prediction of Y .
Revised: Given a textual input X ∈ X and a target label Y ∈ Y , the goal is to learn the encoder Φ ψ and classiﬁer C ψ , such that Φ ψ ( X ) retains little information about X , while still producing discriminative features, allowing the prediction of Y .

Original: Thus, the loss of interest is:
Revised: Thus, the loss of interest is:

Original: [Equation]
Revised: [Equation]

Original: where λ controls the trade-off between the downstream and the compression terms.
Revised: where λ controls the trade-off between the downstream and the compression terms.

Original: Setup.
Revised: Setup.

Original: Following Mahabadi et al.
Revised: Following Mahabadi et al.

Original: (2021) (relying on VUB), we work with the VIBERT model, which uses a Gaussian distribution as prior.
Revised: (2021) (relying on VUB), we work with the VIBERT model, which uses a Gaussian distribution as prior.

Original: Φ ψ is implemented as a stochastic encoder Φ ψ ( X ) =
Revised: Φ ψ is implemented as a stochastic encoder Φ ψ ( X ) =

Original: Z ∼ N ( µ ψ ( X ) , Σ ψ ( X )) .
Revised: Z ∼ N ( µ ψ ( X ) , Σ ψ ( X )) .

Original: Details on the architecture of µ ψ and Σ ψ can be found in Appendix B.
Revised: Details on the architecture of µ ψ and Σ ψ can be found in Appendix B.

Original: The classiﬁer C ψ is composed of dense layers.
Revised: The classiﬁer C ψ is composed of dense layers.

Original: To minimize L , the second part of the objective (10) is bounded using the variational bound from Barber & Agakov (2003).
Revised: To minimize L , the second part of the objective (10) is bounded using the variational bound from Barber & Agakov (2003).

Original: Since we use a Gaussian prior, h( Z | X ) can be expressed in closed form.
Revised: Since we use a Gaussian prior, h( Z | X ) can be expressed in closed form.

Original: Thus, when using K NIFE , I( X ; Z ) =
Revised: Thus, when using K NIFE , I( X ; Z ) =

Original: h( Z ) − h
Revised: h( Z ) − h

Original: ( Z | X ) can be estimated by using (cid:98) h K NIFE to estimate h( Z ) .
Revised: ( Z | X ) can be estimated by using (cid:98) h K NIFE to estimate h( Z ) .

Original: We compare this K NIFE -based MI estimator with aforementioned MI estimators and the variational upper bound (VUB).
Revised: We compare this K NIFE -based MI estimator with aforementioned MI estimators and the variational upper bound (VUB).

Original: For completeness, we also compare against a BERT model trained by direct minimization of a CE loss.
Revised: For completeness, we also compare against a BERT model trained by direct minimization of a CE loss.

Original: We closely follow the protocol of (Mahabadi et al., 2021) and work on the GLUE benchmark (Wang et al., 2018) originally composed of 5 datasets.
Revised: We closely follow the protocol of (Mahabadi et al., 2021) and work on the GLUE benchmark (Wang et al., 2018) originally composed of 5 datasets.

Original: However, following (Mahabadi et al., 2021), we choose to ﬁnetune neither on WNLI (Morgenstern & Ortiz, 2015) nor on CoLA (Warstadt et al., 2019) due to reported ﬂaws in these datasets.
Revised: However, following (Mahabadi et al., 2021), we choose to ﬁnetune neither on WNLI (Morgenstern & Ortiz, 2015) nor on CoLA (Warstadt et al., 2019) due to reported ﬂaws in these datasets.

Original: The evaluation is carried out on the standard validation splits as the test splits are not available.
Revised: The evaluation is carried out on the standard validation splits as the test splits are not available.

Original: Following standard practice (Liu et al., 2019; Yang et al., 2019), we report the accuracy and the F1 for MRPC, the accuracy for RTE and the Pearson and
Revised: Following standard practice (Liu et al., 2019; Yang et al., 2019), we report the accuracy and the F1 for MRPC, the accuracy for RTE and the Pearson and

Original: Spearman correlation coefﬁcient for STS-B.
Revised: Spearman correlation coefﬁcient for STS-B.

Original: Results.
Revised: Results.

Original: Table 1 reports our results on the GLUE benchmark.
Revised: Table 1 reports our results on the GLUE benchmark.

Original: We observe that K NIFE obtains the best results on all three datasets and the lowest variance on MRPC and STS-
Revised: We observe that K NIFE obtains the best results on all three datasets and the lowest variance on MRPC and STS-

Original: B.
Revised: B.

Original: The use of a
Revised: The use of a

Original: Gaussian prior in the stochastic encoder Φ ψ could explain the observed improvement of K NIFE -based estimation over MI-estimators such as CLUB, InfoNCE, MINE, D O E, or NWJ.
Revised: Gaussian prior in the stochastic encoder Φ ψ could explain the observed improvement of K NIFE -based estimation over MI-estimators such as CLUB, InfoNCE, MINE, D O E, or NWJ.

Original: 4. F AIR T EXTUAL C LASSIFICATION
Revised: 4. F AIR T EXTUAL C LASSIFICATION

Original: In fair classiﬁcation, we would like the model to take its decision without utilizing private information such as gender, age, or race.
Revised: In fair classiﬁcation, we would like the model to take its decision without utilizing private information such as gender, age, or race.

Original: For this task, MI can be minimized to disentangle the output of the encoder Z and a private label S ∈ S (e.g., gender, age, or race).
Revised: For this task, MI can be minimized to disentangle the output of the encoder Z and a private label S ∈ S (e.g., gender, age, or race).

Original: [Figure]
Revised: [Figure]

Original: Problem Statement.
Revised: Problem Statement.

Original: Given an input text X
Revised: Given an input text X

Original: [Figure]
Revised: [Figure]

Original: a discrete target label Y and a private label S , the loss is given by
Revised: a discrete target label Y and a private label S , the loss is given by

Original: [Equation]
Revised: [Equation]

Original: downstream task
Revised: downstream task

Original: [Equation]
Revised: [Equation]

Original: where λ controls the trade-off between minimizing MI and CE loss.
Revised: where λ controls the trade-off between minimizing MI and CE loss.

Original: In this framework, a classiﬁer is said to be fair or to achieve perfect privacy if no statistical information about S can be extracted from Φ ψ ( X ) by an adversarial classiﬁer.
Revised: In this framework, a classiﬁer is said to be fair or to achieve perfect privacy if no statistical information about S can be extracted from Φ ψ ( X ) by an adversarial classiﬁer.

Original: Overall, a good model should achieve high accuracy on the main task (i.e., prediction of Y ) while removing information about the protected attribute S .
Revised: Overall, a good model should achieve high accuracy on the main task (i.e., prediction of Y ) while removing information about the protected attribute S .

Original: This information is measured by training an ofﬂine classiﬁer to recover the protected attribute S from Φ ψ ( X ) .
Revised: This information is measured by training an ofﬂine classiﬁer to recover the protected attribute S from Φ ψ ( X ) .

Original: Setup.
Revised: Setup.

Original: We compute the second term of (11) with competing MI estimators, as well as the model from Elazar & Goldberg (2018), which will be referred to as “Adv”, as it utilizes an adversary to recover the private label from the latent representation Z .
Revised: We compute the second term of (11) with competing MI estimators, as well as the model from Elazar & Goldberg (2018), which will be referred to as “Adv”, as it utilizes an adversary to recover the private label from the latent representation Z .

Original: For K NIFE -based MI estimation, we use two DE estimators (as S is a binary label), following the approach outlined in Section 2.4.
Revised: For K NIFE -based MI estimation, we use two DE estimators (as S is a binary label), following the approach outlined in Section 2.4.

Original: All derivations are detailed in Appendix B.
Revised: All derivations are detailed in Appendix B.

Original: We follow the experimental setting from Elazar & Goldberg (2018); Barrett et al.
Revised: We follow the experimental setting from Elazar & Goldberg (2018); Barrett et al.

Original: and use two datasets from the DIAL corpus (Blodgett et al., 2016)
Revised: and use two datasets from the DIAL corpus (Blodgett et al., 2016)

Original: (over 50 million tweets) where the protected attribute S is the race and the main labels are sentiment or mention labels.
Revised: (over 50 million tweets) where the protected attribute S is the race and the main labels are sentiment or mention labels.

Original: The mention label indicates whether a tweet is conversational or not.
Revised: The mention label indicates whether a tweet is conversational or not.

Original: We follow the ofﬁcial split using 160 000 tweets for training and two additional sets composed of 10 000 tweets each for development and testing.
Revised: We follow the ofﬁcial split using 160 000 tweets for training and two additional sets composed of 10 000 tweets each for development and testing.

Original: In all cases, the labels S and Y are binary and balanced, thus a random guess corresponds to 50% accuracy.
Revised: In all cases, the labels S and Y are binary and balanced, thus a random guess corresponds to 50% accuracy.

Original: Results.
Revised: Results.

Original: Figure 4 gathers results on the fair classiﬁcation task.
Revised: Figure 4 gathers results on the fair classiﬁcation task.

Original: The upper dashed lines represent the (private and main) task accuracies when training a model with only the CE loss (case λ =in (11)).
Revised: The upper dashed lines represent the (private and main) task accuracies when training a model with only the CE loss (case λ =in (11)).

Original: This shows that the learned encoding Φ ψ ( X ) contains information about the protected attribute, when training is only performed for the main task.
Revised: This shows that the learned encoding Φ ψ ( X ) contains information about the protected attribute, when training is only performed for the main task.

Original: On both the sentiment and mention task, we observe that a K NIFE -based estimator can achieve perfect privacy (see Figures 4b and 4d) with nearly no accuracy loss in the main task (see Figures 4a and 4c).
Revised: On both the sentiment and mention task, we observe that a K NIFE -based estimator can achieve perfect privacy (see Figures 4b and 4d) with nearly no accuracy loss in the main task (see Figures 4a and 4c).

Original: The other MI estimators exhibit different behavior.
Revised: The other MI estimators exhibit different behavior.

Original: For sentiment labels, most MI estimators fail to reach perfect privacy (CLUB,
Revised: For sentiment labels, most MI estimators fail to reach perfect privacy (CLUB,

Original: NWJ, D O E, and Adv) while others (InfoNCE) achieve perfect privacy while degrading the main task accuracy (10% loss on main accuracy).
Revised: NWJ, D O E, and Adv) while others (InfoNCE) achieve perfect privacy while degrading the main task accuracy (10% loss on main accuracy).

Original: For mention labels, CLUB can also reach perfect privacy with almost no degradation of the accuracy of the main task.
Revised: For mention labels, CLUB can also reach perfect privacy with almost no degradation of the accuracy of the main task.

Original: Overall, it is worth noting that K NIFE -based
Revised: Overall, it is worth noting that K NIFE -based

Original: MI estimation enables better control of the degree of disentanglement than the reported baselines.
Revised: MI estimation enables better control of the degree of disentanglement than the reported baselines.

Original: 4. U NSUPERVISED D OMAIN A DAPTATION
Revised: 4. U NSUPERVISED D OMAIN A DAPTATION

Original: In unsupervised domain adaptation, the goal is to transfer knowledge from the source domain ( S ) with a potentially large number of labeled examples to a target domain ( T ), where only unlabeled examples are available.
Revised: In unsupervised domain adaptation, the goal is to transfer knowledge from the source domain ( S ) with a potentially large number of labeled examples to a target domain ( T ), where only unlabeled examples are available.

Original: Problem Statement.
Revised: Problem Statement.

Original: The learner is given access to labeled images from a source domain ( x s , y ) ∼ ( X S , Y ) ∈
Revised: The learner is given access to labeled images from a source domain ( x s , y ) ∼ ( X S , Y ) ∈

Original: X S × Y and unlabeled images from a target domain x t ∼ X T ∈ X T .
Revised: X S × Y and unlabeled images from a target domain x t ∼ X T ∈ X T .

Original: The goal is to
Revised: The goal is to

Original: [Table]
Revised: [Table]

Original: a classiﬁcation model { Φ ψ , C ψ } that generalizes well to the target domain.
Revised: a classiﬁcation model { Φ ψ , C ψ } that generalizes well to the target domain.

Original: Training models on the supervised source data only results in domain-speciﬁc latent representations Φ ψ ( X ) leading to poor generalization (when X is chosen randomly from { X S , X T } ).
Revised: Training models on the supervised source data only results in domain-speciﬁc latent representations Φ ψ ( X ) leading to poor generalization (when X is chosen randomly from { X S , X T } ).

Original: In order to make the latent representations as domain-agnostic as possible, we follow the information-theoretic method proposed by Gholami et al.
Revised: In order to make the latent representations as domain-agnostic as possible, we follow the information-theoretic method proposed by Gholami et al.

Original: (2020), and used in Cheng et al.
Revised: (2020), and used in Cheng et al.

Original: (2020a).
Revised: (2020a).

Original: The idea is to learn an additional binary model { Φ d ν , C d ν } , whose goal it is to guess the domain D ∈ { 0 , 1 } of X .
Revised: The idea is to learn an additional binary model { Φ d ν , C d ν } , whose goal it is to guess the domain D ∈ { 0 , 1 } of X .

Original: The latent representation learned by Φ d ν will therefore contain all the domain-speciﬁc information that we would like the main encoder Φ ψ to discard.
Revised: The latent representation learned by Φ d ν will therefore contain all the domain-speciﬁc information that we would like the main encoder Φ ψ to discard.

Original: In other words, we would like Φ ψ ( X ) and Φ d ν ( X ) to be completely disentangled, which naturally corresponds to the minimization of I(Φ ψ ( X ); Φ d ν ( X )) .
Revised: In other words, we would like Φ ψ ( X ) and Φ d ν ( X ) to be completely disentangled, which naturally corresponds to the minimization of I(Φ ψ ( X ); Φ d ν ( X )) .

Original: Concretely, the domain classiﬁer is trained to minimize the CE between domain labels D and its own predictions, whereas the main classiﬁer is trained to properly classify support samples while minimizing the MI between Φ ψ ( X ) and Φ d ν ( X ) .
Revised: Concretely, the domain classiﬁer is trained to minimize the CE between domain labels D and its own predictions, whereas the main classiﬁer is trained to properly classify support samples while minimizing the MI between Φ ψ ( X ) and Φ d ν ( X ) .

Original: Using f d ν :=
Revised: Using f d ν :=

Original: C d ν ◦ Φ d ν and f ψ :=
Revised: C d ν ◦ Φ d ν and f ψ :=

Original: C ψ ◦ Φ ψ , the objectives are min ν
Revised: C ψ ◦ Φ ψ , the objectives are min ν

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Setup.
Revised: Setup.

Original: The different MI estimators are compared based on their ability to guide training by estimating I(Φ ψ ( X ); Φ d ν ( X )) in (12).
Revised: The different MI estimators are compared based on their ability to guide training by estimating I(Φ ψ ( X ); Φ d ν ( X )) in (12).

Original: We follow the setup of Cheng et al.
Revised: We follow the setup of Cheng et al.

Original: (2020a) as closely as possible, and consider a total of 6 source/target scenarios formed with MNIST (LeCun & Cortes, 2010),
Revised: (2020a) as closely as possible, and consider a total of 6 source/target scenarios formed with MNIST (LeCun & Cortes, 2010),

Original: MNIST-M (Ganin et al., 2016), SVHN (Netzer et al., 2011), CIFAR-10 (Krizhevsky et al., 2009), and STL-10 (Coates et al., 2011) datasets.
Revised: MNIST-M (Ganin et al., 2016), SVHN (Netzer et al., 2011), CIFAR-10 (Krizhevsky et al., 2009), and STL-10 (Coates et al., 2011) datasets.

Original: We reproduce all methods and allocate the same budget for hyper-parameter tuning to every method.
Revised: We reproduce all methods and allocate the same budget for hyper-parameter tuning to every method.

Original: The exhaustive list of hyper-parameters can be found in Appendix B.
Revised: The exhaustive list of hyper-parameters can be found in Appendix B.

Original: Results.
Revised: Results.

Original: Results are presented in Table 2.
Revised: Results are presented in Table 2.

Original: The K NIFE -based estimator is able to outperform MI estimators in this challenging scenario where both Φ ψ ( X ) and Φ d ν ( X ) are continuous.
Revised: The K NIFE -based estimator is able to outperform MI estimators in this challenging scenario where both Φ ψ ( X ) and Φ d ν ( X ) are continuous.

Original:  C ONCLUDING R EMARKS
Revised:  C ONCLUDING R EMARKS

Original: We introduced K NIFE , a fully learnable, differentiable kernel-based estimator of differential entropy, designed for deep learning applications.
Revised: We introduced K NIFE , a fully learnable, differentiable kernel-based estimator of differential entropy, designed for deep learning applications.

Original: We constructed a mutual information estimator based on
Revised: We constructed a mutual information estimator based on

Original: K NIFE and showcased several applications.
Revised: K NIFE and showcased several applications.

Original: K NIFE is a general purpose estimator and does not require any special properties of the learning problem.
Revised: K NIFE is a general purpose estimator and does not require any special properties of the learning problem.

Original: It can thus be incorporated as part of any training objective, where differential entropy or mutual information estimation is desired.
Revised: It can thus be incorporated as part of any training objective, where differential entropy or mutual information estimation is desired.

Original: In the case of mutual information, one random variable may even be discrete.
Revised: In the case of mutual information, one random variable may even be discrete.

Original: Despite the fundamental challenges in the problem of differential entropy estimation, beyond limitations arising from the use of a ﬁnite number of samples, K NIFE has demonstrated promising empirical results in various representation learning tasks.
Revised: Despite the fundamental challenges in the problem of differential entropy estimation, beyond limitations arising from the use of a ﬁnite number of samples, K NIFE has demonstrated promising empirical results in various representation learning tasks.

Original: Future work will focus on improving the conﬁdence bounds given in Theorem 1.
Revised: Future work will focus on improving the conﬁdence bounds given in Theorem 1.

Original: In particular, tailoring them towards K NIFE using tools from (Birge & Massart, 1995; Singh & Poczos, 2014).
Revised: In particular, tailoring them towards K NIFE using tools from (Birge & Massart, 1995; Singh & Poczos, 2014).

Original: Another potential extension is direct estimation of the gradient of entropy, when ˆ p K NIFE ( x ; θ ) has been learned (Mohamed et al., 2020; Song et al., 2020).
Revised: Another potential extension is direct estimation of the gradient of entropy, when ˆ p K NIFE ( x ; θ ) has been learned (Mohamed et al., 2020; Song et al., 2020).

Original: This could be applied after the learning phase of K NIFE and is left for future work.
Revised: This could be applied after the learning phase of K NIFE and is left for future work.

Original: A PPENDIX A E XPERIMENTAL D ETAILS OF E XPERIMENTS WITH S YNTHETIC D ATA
Revised: A PPENDIX A E XPERIMENTAL D ETAILS OF E XPERIMENTS WITH S YNTHETIC D ATA

Original: Implementation of K NIFE in PyTorch (Paszke et al., 2019) is rather straightforward.
Revised: Implementation of K NIFE in PyTorch (Paszke et al., 2019) is rather straightforward.

Original: The constraint on the weights u can be satisﬁed by applying a softmax transformation.
Revised: The constraint on the weights u can be satisﬁed by applying a softmax transformation.

Original: The covariance matrices were parameterized by the lower-triangular factor in the Cholesky decomposition of the precision matrices, guaranteeing the deﬁniteness constraint to be satisﬁed.
Revised: The covariance matrices were parameterized by the lower-triangular factor in the Cholesky decomposition of the precision matrices, guaranteeing the deﬁniteness constraint to be satisﬁed.

Original: A. D IFFERENTIAL E NTROPY E STIMATION OF G AUSSIAN D ATA
Revised: A. D IFFERENTIAL E NTROPY E STIMATION OF G AUSSIAN D ATA

Original: In Section 3.1.1, the estimation of the entropy h( X ) =
Revised: In Section 3.1.1, the estimation of the entropy h( X ) =

Original: d 2 log 2 πe for X ∼ N ( 0 , I d ) was performed with the hyperparameters given in Table 3.
Revised: d 2 log 2 πe for X ∼ N ( 0 , I d ) was performed with the hyperparameters given in Table 3.

Original: The mean error and its empirical standard deviation are reported in Table 5 over 20 runs, where an independently drawn evaluation set with the same size as the training set is used.
Revised: The mean error and its empirical standard deviation are reported in Table 5 over 20 runs, where an independently drawn evaluation set with the same size as the training set is used.

Original: At d = 10 we have the entropy h
Revised: At d = 10 we have the entropy h

Original: =
Revised: =

Original: d 2 log 2 πe = 14 . 19 , while for the higher dimension, d = 64 we ﬁnd h = 90 .
Revised: d 2 log 2 πe = 14 . 19 , while for the higher dimension, d = 64 we ﬁnd h = 90 .

Original: In the experiment depicted in Figure 1, entropy is decreased after every epoch by letting X i ∼
Revised: In the experiment depicted in Figure 1, entropy is decreased after every epoch by letting X i ∼

Original: N ( 0 , a i I d ) , where i = 0 , . . .
Revised: N ( 0 , a i I d ) , where i = 0 , . . .

Original: , 4 is the epoch index.
Revised: , 4 is the epoch index.

Original: That is, X i = √
Revised: That is, X i = √

Original: a i G d , where G is a standard normal random variable, resulting in an decrease of the DE by ∆ = − d 2 log a ≈ 22 .
Revised: a i G d , where G is a standard normal random variable, resulting in an decrease of the DE by ∆ = − d 2 log a ≈ 22 .

Original: 18 for a =with every epoch.
Revised: 18 for a =with every epoch.

Original: We start at h( X 0 ) =
Revised: We start at h( X 0 ) =

Original: d 2 log 2 πe ≈ 90 . 81 and successively decrease until h( X 4 ) =
Revised: d 2 log 2 πe ≈ 90 . 81 and successively decrease until h( X 4 ) =

Original: h( X 0 ) + 4∆ ≈ 2 . 1 .
Revised: h( X 0 ) + 4∆ ≈ 2 . 1 .

Original: Additional parameters can be found in Table 4.
Revised: Additional parameters can be found in Table 4.

Original: Computational Resources.
Revised: Computational Resources.

Original: Training was performed on an NVidia V100 GPU.
Revised: Training was performed on an NVidia V100 GPU.

Original: Taken together, training for the ﬁrst experiments of entropy estimation in dimensions d = 10 , 64 , as well as the experiment depicted in Figure 1 used GPU time of less than 5 minutes.
Revised: Taken together, training for the ﬁrst experiments of entropy estimation in dimensions d = 10 , 64 , as well as the experiment depicted in Figure 1 used GPU time of less than 5 minutes.

Original: A. D IFFERENTIAL E NTROPY E STIMATION OF T RIANGLE M IXTURES
Revised: A. D IFFERENTIAL E NTROPY E STIMATION OF T RIANGLE M IXTURES

Original: In Section 3.1.2, we perform an estimation of the entropy of c -component triangle mixture distributions.
Revised: In Section 3.1.2, we perform an estimation of the entropy of c -component triangle mixture distributions.

Original: The PDF of such a c -component triangle-mixture, is given by
Revised: The PDF of such a c -component triangle-mixture, is given by

Original: [Equation]
Revised: [Equation]

Original: where Λ s ( x ) :
Revised: where Λ s ( x ) :

Original: = 1 s max { 0 , 2 − 4 s | x |} is a centered triangle PDF with width s > 0 .
Revised: = 1 s max { 0 , 2 − 4 s | x |} is a centered triangle PDF with width s > 0 .

Original: The scales s =
Revised: The scales s =

Original: ( s 1 , . . .
Revised: ( s 1 , . . .

Original: , s c ) and weights w = ( w 1 , . . .
Revised: , s c ) and weights w = ( w 1 , . . .

Original: , w c ) satisfy 0 < s i , w i < 1 and (cid:80) ci =
Revised: , w c ) satisfy 0 < s i , w i < 1 and (cid:80) ci =

Original: 1 w i = 1 .
Revised: 1 w i = 1 .

Original: Before the experiment, we choose w uniformly at random from the c -probability simplex and the scales are chosen uniformly at random in [0 . 1 , 1 . 0] .
Revised: Before the experiment, we choose w uniformly at random from the c -probability simplex and the scales are chosen uniformly at random in [0 . 1 , 1 . 0] .

Original: An example for c = 10 is the true PDF depicted in Figure
Revised: An example for c = 10 is the true PDF depicted in Figure

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: (left).
Revised: (left).

Original: For d > 1 , we perform the estimation on d i.i.d.
Revised: For d > 1 , we perform the estimation on d i.i.d.

Original: copies.
Revised: copies.

Original: Note that the triangle mixture with c components in d -dimensional space has c d modes, i
Revised: Note that the triangle mixture with c components in d -dimensional space has c d modes, i

Original: [Table]
Revised: [Table]

Original: e., the support can be partitioned into c d disjoint components.
Revised: e., the support can be partitioned into c d disjoint components.

Original: The parameters of the experiment yielding Figure 2 (left) are given in Table
Revised: The parameters of the experiment yielding Figure 2 (left) are given in Table

Original: [Table]
Revised: [Table]

Original: while the details of the experiment depicted in Figure 2 (right) can be found in Table 7.
Revised: while the details of the experiment depicted in Figure 2 (right) can be found in Table 7.

Original: In the latter experiment, over ten runs, entropy was estimated to an accuracy of 1 . 6563 ± 0 . 8528 by K NIFE , accurate to 2 .
Revised: In the latter experiment, over ten runs, entropy was estimated to an accuracy of 1 . 6563 ± 0 . 8528 by K NIFE , accurate to 2 .

Original: 4445 ± 0 .using (3) and with an accuracy of 7 . 1070 ± 2 . 

Revised: 4445 ± 0 .using (3) and with an accuracy of 7 . 1070 ± 2 .

Original: by D O E. This is the mean absolute error and its empirical standard deviation over all 10 runs, where the evaluation set was drawn independently from the training set and has the same size as the training set.
Revised: by D O E. This is the mean absolute error and its empirical standard deviation over all 10 runs, where the evaluation set was drawn independently from the training set and has the same size as the training set.

Original: Computational Resources.
Revised: Computational Resources.

Original: Training was performed on an NVidia V100 GPU.
Revised: Training was performed on an NVidia V100 GPU.

Original: Training in d =dimension, that resulted in Figure 2 (left) can be performed in seconds, while all training required for producing Figure 2 (right) used approximately 1 .
Revised: Training in d =dimension, that resulted in Figure 2 (left) can be performed in seconds, while all training required for producing Figure 2 (right) used approximately 1 .

Original: 5 hours of GPU time.
Revised: 5 hours of GPU time.

Original: A. M UTUAL I NFORMATION E STIMATION
Revised: A. M UTUAL I NFORMATION E STIMATION

Original: In Section 3.2, we estimate I( X d ; Y d ) and I( X d ; ( Y 3 ) d ) where ( X, Y ) are multivariate correlated
Revised: In Section 3.2, we estimate I( X d ; Y d ) and I( X d ; ( Y 3 ) d ) where ( X, Y ) are multivariate correlated

Original: Gaussian distributions with correlation coefﬁcient ρ i in the i -th epoch.
Revised: Gaussian distributions with correlation coefﬁcient ρ i in the i -th epoch.

Original: Subsequently, we estimate
Revised: Subsequently, we estimate

Original: I( X d ; Y d ) where X, E ∼ U [ −√ 3 , √ 3] are independent and Y is given by Y =
Revised: I( X d ; Y d ) where X, E ∼ U [ −√ 3 , √ 3] are independent and Y is given by Y =

Original: ρ i X
Revised: ρ i X

Original: + (cid:112) 1 − ρ 2 i E .
Revised: + (cid:112) 1 − ρ 2 i E .

Original: In both cases, ρ i is chosen such that I( X d ; Y d ) =
Revised: In both cases, ρ i is chosen such that I( X d ; Y d ) =

Original: 2 i in the i -th epoch.
Revised: 2 i in the i -th epoch.

Original: All neural networks are randomly initialized.
Revised: All neural networks are randomly initialized.

Original: The bias, variance, and MSE during training as a function of the MI, can be observed in Figure 5.
Revised: The bias, variance, and MSE during training as a function of the MI, can be observed in Figure 5.

Original: The estimation is performed in 10 runs, randomly choosing the training meta-parameters as proposed by McAllester & Stratos (2020).
Revised: The estimation is performed in 10 runs, randomly choosing the training meta-parameters as proposed by McAllester & Stratos (2020).

Original: In Figure 3 (bottom), we present the best run for each method, selected by distance from the true MI at the end of training.
Revised: In Figure 3 (bottom), we present the best run for each method, selected by distance from the true MI at the end of training.

Original: The bias, variance, and MSE during training, as a function of the MI, can be observed in Figure 6.
Revised: The bias, variance, and MSE during training, as a function of the MI, can be observed in Figure 6.

Original: Details about the source distribution as well as details of the experiments can be found in Table 8.
Revised: Details about the source distribution as well as details of the experiments can be found in Table 8.

Original: During experimentation it turned out to be
Revised: During experimentation it turned out to be

Original: [Figure]
Revised: [Figure]

Original: MI
Revised: MI

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: beneﬁcial to train the parameters Θ and θ in (9) separately and substantially increase the learning rate for the training of θ .
Revised: beneﬁcial to train the parameters Θ and θ in (9) separately and substantially increase the learning rate for the training of θ .

Original: Thus, we increase the learning rate for the training of θ by a factor of 10 3 .
Revised: Thus, we increase the learning rate for the training of θ by a factor of 10 3 .

Original: Model Architecture for Θ .
Revised: Model Architecture for Θ .

Original: We utilize the feed-forward architecture, also used in McAllester & Stratos (2020).
Revised: We utilize the feed-forward architecture, also used in McAllester & Stratos (2020).

Original: It is a simple architecture with two linear layers, one hidden layer using tanh activation, immediately followed by an output layer.
Revised: It is a simple architecture with two linear layers, one hidden layer using tanh activation, immediately followed by an output layer.

Original: The number of neurons in the hidden layer is a meta-parameter selected randomly from { 64 , 128 , 256 } for each training run.
Revised: The number of neurons in the hidden layer is a meta-parameter selected randomly from { 64 , 128 , 256 } for each training run.

Original: Three models with this architecture are used for the three parameters ( A , a , u ) , as described by (4), where only the output dimension is changed to ﬁt the parameter dimension.
Revised: Three models with this architecture are used for the three parameters ( A , a , u ) , as described by (4), where only the output dimension is changed to ﬁt the parameter dimension.

Original: Computational Resources.
Revised: Computational Resources.

Original: Training was performed, using about 6 hours of GPU time on an
Revised: Training was performed, using about 6 hours of GPU time on an

Original: NVidia V100 GPU to carry out the experiment depicted in Figure 3 (bottom).
Revised: NVidia V100 GPU to carry out the experiment depicted in Figure 3 (bottom).

Original: B E XPERIMENTAL D ETAILS OF E XPERIMENTS ON N ATURAL D ATA B. O N THE PARAMETER UPDATE
Revised: B E XPERIMENTAL D ETAILS OF E XPERIMENTS ON N ATURAL D ATA B. O N THE PARAMETER UPDATE

Original: In Section 4, we rely on two different types of models: pretrained (e.g., ﬁne tuning with VIBERT) and randomly initialized (e.g., in fair classiﬁcation and domain adaptation).
Revised: In Section 4, we rely on two different types of models: pretrained (e.g., ﬁne tuning with VIBERT) and randomly initialized (e.g., in fair classiﬁcation and domain adaptation).

Original: When working with randomly initialized networks the parameters are updated.
Revised: When working with randomly initialized networks the parameters are updated.

Original: However, it is worth noting that in the litterature the pre-trained model parameters ( i.e. ψ ) are not always updated (see Ravfogel et al.
Revised: However, it is worth noting that in the litterature the pre-trained model parameters ( i.e. ψ ) are not always updated (see Ravfogel et al.

Original: In our experiments: (i) We always update the parameters (even for pretrained models), and (ii) we did not change the way the parameters were updated in concurrent works (to ensure fair comparison).
Revised: In our experiments: (i) We always update the parameters (even for pretrained models), and (ii) we did not change the way the parameters were updated in concurrent works (to ensure fair comparison).

Original: Speciﬁcally,
Revised: Speciﬁcally,

Original: • for language model ﬁnetuning (Appendix B.2), we followed Mahabadi et al.
Revised: • for language model ﬁnetuning (Appendix B.2), we followed Mahabadi et al.

Original: and did a joint update; • for the fair classiﬁcation task (Appendix B.3), we followed common practice and used the algorithm described in Algorithm 1 which rely on an alternated update; • for the domain adaptation task (Appendix B.4), we followed common practice and used a joint method.
Revised: and did a joint update; • for the fair classiﬁcation task (Appendix B.3), we followed common practice and used the algorithm described in Algorithm 1 which rely on an alternated update; • for the domain adaptation task (Appendix B.4), we followed common practice and used a joint method.

Original: B. I NFORMATION B OTTLENECK FOR L ANGUAGE M ODEL F INETUNING
Revised: B. I NFORMATION B OTTLENECK FOR L ANGUAGE M ODEL F INETUNING

Original: For this experiment we follow the experimental setting introduced in Mahabadi et al.
Revised: For this experiment we follow the experimental setting introduced in Mahabadi et al.

Original: and work with the GLUE data 2 .
Revised: and work with the GLUE data 2 .

Original: Model Architecture.
Revised: Model Architecture.

Original: We report in Table 9, the multilayer perceptron (MLP) used to compute the compressed sentence representations produced by BERT.
Revised: We report in Table 9, the multilayer perceptron (MLP) used to compute the compressed sentence representations produced by BERT.

Original: Variance and Mean MLP networks are composed of fully connected layers.
Revised: Variance and Mean MLP networks are composed of fully connected layers.

Original: B i a s     

Revised: B i a s

Original: V a r i a n c e
Revised: V a r i a n c e

Original: MI Values
Revised: MI Values

Original: [Figure]
Revised: [Figure]

Original: Algorithm 1 Disentanglement using a MI-based regularizer 1: I NPUT Labelled training s j
Revised: Algorithm 1 Disentanglement using a MI-based regularizer 1: I NPUT Labelled training s j

Original: N
Revised: N

Original: } ; independent set of samples E ; θ parameters K NIFE ψ parameters of network.
Revised: } ; independent set of samples E ; θ parameters K NIFE ψ parameters of network.

Original: 2: I do 5:
Revised: 2: I do 5:

Original: 8: end for 9:
Revised: 8: end for 9:

Original: 11: end while 12: O
Revised: 11: end while 12: O

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Model Training.
Revised: Model Training.

Original: For model training, all models are trained for 6 epochs and we use early stopping (best model is selected on validation set error).
Revised: For model training, all models are trained for 6 epochs and we use early stopping (best model is selected on validation set error).

Original: For IB, λ is selected in { 10 − 4 , 10 − 5 , 10 − 6 } and K is selected in { 144 , 192 , 288 , 384 } .
Revised: For IB, λ is selected in { 10 − 4 , 10 − 5 , 10 − 6 } and K is selected in { 144 , 192 , 288 , 384 } .

Original: We follow (Alemi et al., 2016) where the posterior is averaged over 5 samples and a linear annealing schedule is used for λ .
Revised: We follow (Alemi et al., 2016) where the posterior is averaged over 5 samples and a linear annealing schedule is used for λ .

Original: Additional hyper-parameters are reported in
Revised: Additional hyper-parameters are reported in

Original: Table 10.
Revised: Table 10.

Original: Dataset Statistics.
Revised: Dataset Statistics.

Original: Table 11 reports the statistics of the dataset used in our ﬁnetuning experiment.
Revised: Table 11 reports the statistics of the dataset used in our ﬁnetuning experiment.

Original: Computational Resources.
Revised: Computational Resources.

Original: For all these experiments we rely on NVidia-P100 with 16GB of RAM.
Revised: For all these experiments we rely on NVidia-P100 with 16GB of RAM.

Original: To complete the full grid-search on 10 seeds and on the three datasets
Revised: To complete the full grid-search on 10 seeds and on the three datasets

Original: [Table]
Revised: [Table]

Original: approximately 1.5k hours are required.
Revised: approximately 1.5k hours are required.

Original: B. F AIR T EXTUAL C LASSIFICATION
Revised: B. F AIR T EXTUAL C LASSIFICATION

Original: In this section
Revised: In this section

Original: [Table]
Revised: [Table]

Original: we gather the experimental details for the textual fair classiﬁcation task.
Revised: we gather the experimental details for the textual fair classiﬁcation task.

Original: B.3. D ETAILS OF THE K NIFE - BASED E STIMATOR
Revised: B.3. D ETAILS OF THE K NIFE - BASED E STIMATOR

Original: In this experiment
Revised: In this experiment

Original: [Table]
Revised: [Table]

Original: we estimate the MI between a continuous random variable, namely Z =
Revised: we estimate the MI between a continuous random variable, namely Z =

Original: Φ ψ ( X ) , and a discrete variable, denoted by S ∈ S
Revised: Φ ψ ( X ) , and a discrete variable, denoted by S ∈ S

Original: = { 1 , 2 , . .
Revised: = { 1 , 2 , . .

Original: . , |S|} .
Revised: . , |S|} .

Original: We follow the strategy outlined in
Revised: We follow the strategy outlined in

Original: Section 2.4 for estimating the conditional DE h( Z | S ) .
Revised: Section 2.4 for estimating the conditional DE h( Z | S ) .

Original: However, we will reuse the estimate of the conditional PDF ˆ p ( z | s ; Θ ) to compute an estimate of the DE as
Revised: However, we will reuse the estimate of the conditional PDF ˆ p ( z | s ; Θ ) to compute an estimate of the DE as

Original: [Equation]
Revised: [Equation]

Original: where ˆ p ( s ) = 1 N |{ n : s n = s }| is used to indicate the empirical distribution of S in the training set
Revised: where ˆ p ( s ) = 1 N |{ n : s n = s }| is used to indicate the empirical distribution of S in the training set

Original: D s .
Revised: D s .

Original: In our experiments, with |S| = 2 , we found that estimating the DE h( Z ) based on the K NIFE estimator learnt for h( Z | S ) increases the stability of training.
Revised: In our experiments, with |S| = 2 , we found that estimating the DE h( Z ) based on the K NIFE estimator learnt for h( Z | S ) increases the stability of training.

Original: We adopted the same strategy for D O E.
Revised: We adopted the same strategy for D O E.

Original: B.3. E XPERIMENTAL D ETAILS
Revised: B.3. E XPERIMENTAL D ETAILS

Original: Model Architecture.
Revised: Model Architecture.

Original: For the encoder, we use a bidirectionnal GRU with two layers with hidden and input dimension set to 128.
Revised: For the encoder, we use a bidirectionnal GRU with two layers with hidden and input dimension set to 128.

Original: We use LeakyReLU as the activation function.
Revised: We use LeakyReLU as the activation function.

Original: The classiﬁcation head is composed of fully connected layers of input dimension 256.
Revised: The classiﬁcation head is composed of fully connected layers of input dimension 256.

Original: We use a learning rate of 0 .for
Revised: We use a learning rate of 0 .for

Original: AdamW.
Revised: AdamW.

Original: The dropout rate is set to 0.2.
Revised: The dropout rate is set to 0.2.

Original: The number of warmup steps is set to 1000.
Revised: The number of warmup steps is set to 1000.

Original: Computational Resources.
Revised: Computational Resources.

Original: For all these experiments, we rely on NVIDIA-P100 with 16GB of
Revised: For all these experiments, we rely on NVIDIA-P100 with 16GB of

Original: RAM.
Revised: RAM.

Original: Each model is trained for 30k steps.
Revised: Each model is trained for 30k steps.

Original: The model with the lowest MI is selected.
Revised: The model with the lowest MI is selected.

Original: The training of a single network takes around 3 hours.
Revised: The training of a single network takes around 3 hours.

Original: B. U NSUPERVISED D OMAIN A DAPTATION
Revised: B. U NSUPERVISED D OMAIN A DAPTATION

Original: We follow the experimental setup given in Cheng et al.
Revised: We follow the experimental setup given in Cheng et al.

Original: (2020a) as closely as possible, i.e., we pick hyperparameters given in the paper, or if not provided, those set in the code:
Revised: (2020a) as closely as possible, i.e., we pick hyperparameters given in the paper, or if not provided, those set in the code:

Original: Model Training.
Revised: Model Training.

Original: We use Adam optimizer for all modules with a learning rate of 0.001.
Revised: We use Adam optimizer for all modules with a learning rate of 0.001.

Original: Batch size is set to 128.
Revised: Batch size is set to 128.

Original: We set the weighting parameter λ = 0 .
Revised: We set the weighting parameter λ = 0 .

Original: The original code of Cheng et al.
Revised: The original code of Cheng et al.

Original: (2020a) uses 15 000 training iterations, but we found most methods had not properly converged at this stage, and hence use 25 000 iterations instead.
Revised: (2020a) uses 15 000 training iterations, but we found most methods had not properly converged at this stage, and hence use 25 000 iterations instead.

Original: Similar to other experiments, we set the kernel size M = 128 .
Revised: Similar to other experiments, we set the kernel size M = 128 .

Original: Model Architecture.
Revised: Model Architecture.

Original: Table 12 summarizes the architectures used for the different modules.
Revised: Table 12 summarizes the architectures used for the different modules.

Original: For the MI network of each method, the best conﬁguration, based on the validation set of the ﬁrst task
Revised: For the MI network of each method, the best conﬁguration, based on the validation set of the ﬁrst task

Original: MNIST → MNIST-M, is chosen among 4 conﬁgurations: with or without LayerNorm and with ReLU or tanh activation.
Revised: MNIST → MNIST-M, is chosen among 4 conﬁgurations: with or without LayerNorm and with ReLU or tanh activation.

Original: Computational Resources.
Revised: Computational Resources.

Original: For these experiments, we used a cluster of NVIDIA-V100 with 16GB of RAM.
Revised: For these experiments, we used a cluster of NVIDIA-V100 with 16GB of RAM.

Original: Each training (i.e., 25k iterations) on a single task requires on average 2 hours.
Revised: Each training (i.e., 25k iterations) on a single task requires on average 2 hours.

Original: Given that we have 6 tasks, and repeat the training for 3 different seeds, on average 36 hours computation time is required for each method.
Revised: Given that we have 6 tasks, and repeat the training for 3 different seeds, on average 36 hours computation time is required for each method.

Original: C B OUNDING THE E RROR
Revised: C B OUNDING THE E RROR

Original: In the following, ﬁx L > 0 and let P L be the set of L -Lipschitz PDFs supported 5 on X :=
Revised: In the following, ﬁx L > 0 and let P L be the set of L -Lipschitz PDFs supported 5 on X :=

Original: [0 , 1] d , i.e., (cid:82) X p ( x ) d x = 1 , and
Revised: [0 , 1] d , i.e., (cid:82) X p ( x ) d x = 1 , and

Original: [Equation]
Revised: [Equation]

Original: (15) for p ∈ P L , where 6 (cid:107)
Revised: (15) for p ∈ P L , where 6 (cid:107)

Original: [Equation]
Revised: [Equation]

Original: Assume p ∈ P L and let κ be a PDF supported on X .
Revised: Assume p ∈ P L and let κ be a PDF supported on X .

Original: In order to show that estimation of h( X ) is achievable, we use a standard Parzen-Rosenblatt estimator ˆ p ( x ; w ) :
Revised: In order to show that estimation of h( X ) is achievable, we use a standard Parzen-Rosenblatt estimator ˆ p ( x ; w ) :

Original: = 1 Nw (cid:80) Mm =1 κ (cid:0) x − X (cid:48) m w (cid:1) , as in (2).
Revised: = 1 Mw (cid:80) Mm = 1 κ (cid:0) x − X (cid:48) m w (cid:1) , as in (2).

Original: The entropy estimate is then deﬁned by the empirical average
Revised: The entropy estimate is then deﬁned by the empirical average

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Further, deﬁne the following quantities, which are assumed to be ﬁnite:
Revised: Further, deﬁne the following quantities, which are assumed to be ﬁnite:

Original: [Equation]
Revised: [Equation]

Original: Note that it is easily seen that p max ≤ L 2 and C 1 ≤ max (cid:8) p max log 2 p max , 4 e − 2 (cid:9) by our assumptions.
Revised: Note that it is easily seen that p max ≤ L 2 and C 1 ≤ max (cid:8) p max log 2 p max , 4 e − 2 (cid:9) by our assumptions.

Original: The requirement C 2 , K max < ∞ represents a mild condition on the kernel function κ .
Revised: The requirement C 2 , K max < ∞ represents a mild condition on the kernel function κ .

Original: We can now show the following.
Revised: We can now show the following.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: (64, H, W) (64, H // 2, W //
Revised: (64, H, W) (64, H // 2, W //

Original: [Table]
Revised: [Table]

Original: (64, H // 2, W // 2) (64, H // 2, W //
Revised: (64, H // 2, W // 2) (64, H // 2, W //

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: H //
Revised: H //

Original: [Table]
Revised: [Table]

Original: W //
Revised: W //

Original: [Table] [Table]
Revised: [Table] [Table]

Original: H //
Revised: H //

Original: [Table]
Revised: [Table]

Original: W /
Revised: W /

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: W
Revised: W

Original: [Table]
Revised: [Table]

Original: (3, H, W) (64, H, W) 3x
Revised: (3, H, W) (64, H, W) 3x

Original: [Table]
Revised: [Table]

Original: 64 channels, Stride=1, Padding=
Revised: 64 channels, Stride=1, Padding=

Original: [Table]
Revised: [Table]

Original: (3, H, W) (64, H, W)
Revised: (3, H, W) (64, H, W)

Original: [Table]
Revised: [Table]

Original: (3, H, W) (64, H, W)
Revised: (3, H, W) (64, H, W)

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: (64, H, W) (64, H, W) 3x3, 64 channels, Stride=1, Padding=
Revised: (64, H, W) (64, H, W) 3x3, 64 channels, Stride=1, Padding=

Original: [Table]
Revised: [Table]

Original: (64, H, W) (64, H, W) -
Revised: (64, H, W) (64, H, W) -

Original: Activation (64, H, W) (64, H, W)
Revised: Activation (64, H, W) (64, H, W)

Original: LeakyRelu 0.
Revised: LeakyRelu 0.

Original: 2D convolution (64, H, W) (64, H, W) 3x3, 64 channels, Stride=1, Padding= 2D BatchNorm (64, H, W) (64, H, W) -
Revised: 2D convolution (64, H, W) (64, H, W) 3x3, 64 channels, Stride=1, Padding= 2D BatchNorm (64, H, W) (64, H, W) -

Original: Activation (64, H, W) (64, H, W)
Revised: Activation (64, H, W) (64, H, W)

Original: LeakyRelu 0.
Revised: LeakyRelu 0.

Original: Noisy downsampling
Revised: Noisy downsampling

Original: Layer type
Revised: Layer type

Original: Input shape
Revised: Input shape

Original: Output shape
Revised: Output shape

Original: Parameters
Revised: Parameters

Original: MaxPool (64, H, W) (64, H // 2, H // 2) 2x2, Stride=
Revised: MaxPool (64, H, W) (64, H // 2, H // 2) 2x2, Stride=

Original: Dropout (64, H // 2, W // 2) (64, H // 2, H // 2) p=0.
Revised: Dropout (64, H // 2, W // 2) (64, H // 2, H // 2) p=0.

Original: Noise (64, H // 2, W // 2) (64, H // 2, H // 2)
Revised: Noise (64, H // 2, W // 2) (64, H // 2, H // 2)

Original: Gaussian with σ =
Revised: Gaussian with σ =

Original: MI network
Revised: MI network

Original: Layer type
Revised: Layer type

Original: Input shape
Revised: Input shape

Original: Output shape
Revised: Output shape

Original: Details
Revised: Details

Original: LayerNorm ( C in ,) ( C in ,)
Revised: LayerNorm ( C in ,) ( C in ,)

Original: Optional
Revised: Optional

Original: Fully connected ( C in ,) (64,)
Revised: Fully connected ( C in ,) (64,)

Original: Activation = [ReLU, tanh ]
Revised: Activation = [ReLU, tanh ]

Original: LayerNorm ( C in ,) (64,)
Revised: LayerNorm ( C in ,) (64,)

Original: Optional
Revised: Optional

Original: Fully connected (64,) ( C out ,)
Revised: Fully connected (64,) ( C out ,)

Original: Optional
Revised: Optional

Original: Theorem 2.
Revised: Theorem 2.

Original: With probability greater than 1 − δ we have
Revised: With probability greater than 1 − δ we have

Original: [Equation]
Revised: [Equation]

Original: δ

Original: [Equation] Nδ [Equation] (21) if the expression in the logarithm is positive.
Revised: 21) if the expression in the logarithm is positive.

Original: In particular, the estimation error approaches zero as N → ∞ if w = https : //www.overleaf.com/ project/ 609 d 4 ea 5 edff 1 a 4 d 656 d 35 fbw ( N ) → 0 , M = M ( N ) → ∞ are chosen such that
Revised: In particular, the estimation error approaches zero as N → ∞ if w = w ( N ) → 0 , M = M ( N ) → ∞ are chosen such that

Original: [Equation]
Revised: [Equation]

Original: We prove Theorem 2 in several Lemmas.
Revised: We prove Theorem 2 in several Lemmas.

Original: Lemma 3.
Revised: Lemma 3.

Original: Fix δ > 0 and x 0 ∈ X .
Revised: Fix δ > 0 and x 0 ∈ X .

Original: Then, with probability greater than 1 − δ ,
Revised: Then, with probability greater than 1 − δ ,

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: First, we can show that
Revised: First, we can show that

Original: [Equation]
Revised: [Equation]

Original: as κ is a kernel of order (cid:96) = 0 and p ∈ P (1 , L ) , using the notation of (Tsybakov, 2008, Sec. 1.2.1) and the (cid:96) 1 norm.
Revised: as κ is a kernel of order (cid:96) = 0 and p ∈ P (1 , L ) , using the notation of (Tsybakov, 2008, Sec. 1.2.1) and the (cid:96) 1 norm.

Original: We thus obtain (25) by (Tsybakov, 2008, Prop. 1.2).
Revised: We thus obtain (25) by (Tsybakov, 2008, Prop. 1.2).

Original: Next, note that
Revised: Next, note that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: holds as the requirements of McDiarmid’s inequality (Paninski, 2003, Sec. 3) are satisﬁed with c j = K max Mw and thus P {| E [ˆ p ( x 0 )] − ˆ p ( x 0 ) | ≥ ε } ≤ δ with 
Revised: holds as the requirements of McDiarmid’s inequality (Paninski, 2003, Sec. 3) are satisﬁed with c j = K max Mw and thus P {| E [ˆ p ( x 0 )] − ˆ p ( x 0 ) | ≥ ε } ≤ δ with ε = K max

Original: [Equation]
Revised: [Equation]

Original: Combining (25) and (26) gives (24).
Revised: Combining (25) and (26) gives (24).

Original: Lemma 4.
Revised: Lemma 4.

Original: For any continuous random variable X supported on X and a ≥ 0 , we have
Revised: For any continuous random variable X supported on X and a ≥ 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We apply Markov’s inequality to the random variable Y = 1 p ( X ) and observe that
Revised: We apply Markov’s inequality to the random variable Y = 1 p ( X ) and observe that

Original: [Equation]
Revised: [Equation]

Original: Lemma 5.
Revised: Lemma 5.

Original: If x > 0 , y ≥ a > 0 , 0 < a < 1 , and | x − y | ≤ δ < a , then
Revised: If x > 0 , y ≥ a > 0 , 0 < a < 1 , and | x − y | ≤ δ < a , then

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Proof.
Revised: Proof.

Original: Case x ≥ y .
Revised: Case x ≥ y .

Original: We can write y = a + b and x = y + c = a + b + c for b ≥ 0 and 0 ≤ c ≤ δ < a .
Revised: We can write y = a + b and x = y + c = a + b + c for b ≥ 0 and 0 ≤ c ≤ δ < a .

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Furthermore, 
Revised: Furthermore, log

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Case x < y .
Revised: Case x < y .

Original: Here, we can write y = a + b and x = y − c
Revised: Here, we can write y = a + b and x = y − c

Original: = a + b − c for b ≥ 0 and 0 ≤ c ≤ δ < a .
Revised: = a + b − c for b ≥ 0 and 0 ≤ c ≤ δ < a .

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 2.
Revised: Proof of Theorem 2.

Original: We apply Lemma 3 N times and use the union bound to show that with probability greater than 1 − δ 3 we have for every n ∈ [ N ]
Revised: We apply Lemma 3 N times and use the union bound to show that with probability greater than 1 − δ 3 we have for every n ∈ [ N ]

Original: [Equation]
Revised: [Equation]

Original: Similarly, by Lemma 4, we have with probability greater than 1 − δ 3 that
Revised: Similarly, by Lemma 4, we have with probability greater than 1 − δ 3 that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: for all n ∈ [ N ] .
Revised: for all n ∈ [ N ] .

Original: Again by the union bound, we have that with probability greater than 1 − 2 δ 3 both (40) and (41) hold, and in turn, by Lemma 5, we have
Revised: Again by the union bound, we have that with probability greater than 1 − 2 δ 3 both (40) and (41) hold, and in turn, by Lemma 5, we have

Original: [Equation]
Revised: [Equation]

Original: provided the argument in the logarithm is positive.
Revised: provided the argument in the logarithm is positive.

Original: Finally, we have the upper bound on the variance
Revised: Finally, we have the upper bound on the variance

Original: [Equation]
Revised: [Equation]

Original: and apply Chebychev’s inequality, showing that with probability greater than 1 − δ 3 ,
Revised: and apply Chebychev’s inequality, showing that with probability greater than 1 − δ 3 ,

Original: [Equation]
Revised: [Equation]

Original: The union bound and the triangle inequality applied to (44) and (48) yields the desired result.
Revised: The union bound and the triangle inequality applied to (44) and (48) yields the desired result.

Original: D L IBRARIES U SED
Revised: D L IBRARIES U SED

Original: For our experiments, we built upon code from the following sources.
Revised: For our experiments, we built upon code from the following sources.

Original: • VIBERT (Mahabadi et al., 2021) at github.com/rabeehk/vibert .
Revised: • VIBERT (Mahabadi et al., 2021) at github.com/rabeehk/vibert .

Original: • TRANSFORMERS
Revised: • TRANSFORMERS

Original: (Wolf et al., 2019) at github.com/huggingface/transformers .
Revised: (Wolf et al., 2019) at github.com/huggingface/transformers .

Original: • D O E
Revised: • D O E

Original: McAllester & Stratos, 2020) at github.com/karlstratos/doe .
Revised: McAllester & Stratos, 2020) at github.com/karlstratos/doe .

Original: • SMILE (Song & Ermon, 2019) at github.com/ermongroup/smile-mi-estimator .
Revised: • SMILE (Song & Ermon, 2019) at github.com/ermongroup/smile-mi-estimator .

Original: • InfoNCE, MINE, NWJ, CLUB (Cheng et al., 2020a) at github.com/Linear95/CLUB .
Revised: • InfoNCE, MINE, NWJ, CLUB (Cheng et al., 2020a) at github.com/Linear95/CLUB .