U NDERSTANDING THE L IMITATIONS OF V ARIATIONAL M UTUAL I NFORMATION E STIMATORS
A BSTRACT
Variational approaches based on neural networks are showing promise for estimating mutual information (MI) between high dimensional variables.
However, they can be difﬁcult to use in practice due to poorly understood bias/variance tradeoffs.
We theoretically show that, under some conditions, estimators such as MINE exhibit variance that could grow exponentially with the true amount of underlying
MI.
We also empirically demonstrate that existing estimators fail to satisfy basic self-consistency properties of MI, such as data processing and additivity under independence.
Based on a uniﬁed perspective of variational approaches, we develop a new estimator that focuses on variance reduction.
Empirical results on standard benchmark tasks demonstrate that our proposed estimator exhibits improved biasvariance trade-offs on standard benchmark tasks.
I NTRODUCTION
Mutual information (MI) estimation and optimization are crucial to many important problems in machine learning, such as representation learning (Chen et al., 2016;
Zhao et al., 2018b; Tishby &
Zaslavsky, 2015; Higgins et al., 2018) and reinforcement learning (Pathak et al., 2017; van den Oord et al., 2018).
However, estimating mutual information from samples is challenging (McAllester &
Statos, 2018) and traditional parametric and non-parametric approaches (Nemenman et al., 2004;
Gao et al., 2015; 2017) struggle to scale up to modern machine learning problems, such as estimating the MI between images and learned representations.
Recently, there has been a surge of interest in MI estimation with variational approaches (Barber & Agakov, 2003; Nguyen et al., 2010; Donsker & Varadhan, 1975), which can be naturally combined with deep learning methods (Alemi et al., 2016; van den Oord et al., 2018; Poole et al., 2019).
Despite their empirical effectiveness in downstream tasks such as representation learning (Hjelm et al., 2018; Velickovi ´ c et al., 2018), their effectiveness for MI estimation remains unclear.
In particular, higher estimated MI between observations and learned representations do not seem to indicate improved predictive performance when the representations are used for downstream supervised learning tasks (Tschannen et al., 2019).
In this paper, we discuss two limitations of variational approaches to MI estimation.
First, we theoretically demonstrate that the variance of certain estimators, such as MINE (Belghazi et al., 2018), could grow exponentially with the ground truth MI, leading to poor bias-variance trade-offs.
Second, we propose a set of self-consistency tests over basic properties of MI, and empirically demonstrate that all considered variational estimators fail to satisfy critical properties of MI, such as data processing and additivity under independence.
These limitations challenge the effectiveness of these methods for estimating or optimizing MI.
To mitigate these issues, we propose a uniﬁed perspective over variational estimators treating variational MI estimation as an optimization problem over (valid) density ratios.
This view highlights the role of partition functions estimation, which is the culprit of high variance issues in MINE.
To address this issue, we propose to improve MI estimation via variance reduction techniques for partition function estimation.
Empirical results demonstrate that our estimators have much better bias-variance trade-off compared to existing methods on standard benchmark tasks.
B ACKGROUND AND R ELATED W ORK 2. N OTATIONS
We use uppercase letters to denote a probability measure (e.g., P , Q ) and corresponding lowercase letters to denote its density 1 functions (e.g., p , q ) unless speciﬁed otherwise.
We use X, Y to denote random variables with separable sample spaces denoted as X and Y respectively, and P ( X ) (or
P ( Y ) ) to denote the set of all probability measures over the Borel σ -algebra on X (or Y ).
Under Q ∈ P ( X ) , the p -norm of a function r : X → R is deﬁned as (cid:107) r (cid:107) p :=
( (cid:82) | r | p d Q ) 1 /p with (cid:107) r (cid:107) ∞ =
lim p →∞ (cid:107) r (cid:107) p .
The set of locally p -integrable functions is deﬁned as L p ( Q ) :
= { r :
X → R | (cid:107) r (cid:107) p < ∞} .
The space of probability measures wrt.
Q is deﬁned as ∆( Q ) :
= { r ∈
L 1 ( Q ) | (cid:107) r (cid:107) 1 = 1 , r ≥ 0 } ; we also call this the space of “valid density ratios” wrt.
Q .
We use
P (cid:28) Q to denote that P is absolutely continuous with respect to Q .
We use ˆ I E to denote an estimator for I E where we replace expectations with sample averages.
2. V ARIATIONAL M UTUAL I NFORAMTION E STIMATION
The mutual information between two random variables X and Y is the KL divergence between the joint and the product of marginals:
[Equation]
1) which we wish to estimate using samples from P ( X, Y ) ; in certain cases we may know the density of marginals (e.g. P ( X ) ).
There are a wide range of variational approaches to variational MI estimation.
Variational information maximization uses the following result (Barber & Agakov, 2003):
Lemma 1 (Barber-Agakov (BA)) .
For two random variables X and Y :
[Equation]
where q φ : Y → P ( X ) is a valid conditional distribution over X given y ∈ Y and p ( x ) is the probability density function of the marginal distribution P ( X ) .
Another family of approaches perform MI estimation through variational lower bounds to KL divergences.
For example, the Mutual Information Neural Estimator (MINE, Belghazi et al.
(2018)) applies the following lower bound to KL divergences (Donsker & Varadhan, 1975).
Lemma 2 (Donsker-Varadahn (DV)) .
∀ P, Q ∈ P ( X ) such that P (cid:28) Q ,
[Equation]
One could set P =
P
( X, Y ) and Q = P ( X ) P ( Y ) , T as a parametrized neural network (e.g.
T θ ( x , y ) parametrized by θ ), and obtain the estimate by optimizing the above objective via stochastic gradient descent over mini-batches.
However, the corresponding estimator ˆ I MINE (where we replace the expectations in Eq.
(3) with sample averages) is biased, leading to biased gradient estimates;
Belghazi et al.
(2018) propose to reduce bias via estimating the partition function E Q [ e T ] with exponential moving averages of mini-batches.
The variational f -divergence estimation approach (Nguyen et al., 2010; Nowozin et al., 2016) considers lower bounds on f -divergences which can be specialize to KL divergence, and subsequently to mutual information estimation:
Lemma 3 (Nyugen et al.
(NWJ)) .
∀ P, Q ∈ P ( X ) such that P (cid:28) Q ,
[Equation]
and D KL ( P (cid:107) Q ) =
I NWJ ( T )
when T = log(d P/ d Q ) + 1 .
The supremum over T is a invertible function of the density ratio d P/ d Q , so one could use this approach to estimate density ratios by inverting the function (Nguyen et al., 2010; Nowozin et al., 2016; Grover & Ermon, 2017).
The corresponding mini-batch estimator (denoted as ˆ I NWJ ) is unbiased, so unlike MINE, this approach does not require special care to reduce bias in gradients.
Contrastive Predictive Coding (CPC, van den
Oord et al.
(2018)) considers the following objective:
[Equation]
where f θ : X × Y → R ≥ 0 is a neural network parametrized by θ and P n ( X, Y ) denotes the joint pdf for n i.i.d.
random variables sampled from P ( X, Y ) .
CPC generally has less variance but is more biased because its estimate does not exceed log n , where n is the batch size (van den Oord et al., 2018; Poole et al., 2019).
While one can further reduce the bias with larger n , the number of evaluations needed for estimating each batch with f θ is n 2 , which scales poorly.
To address the high-bias issue of CPC, Poole et al.
proposed an interpolation between I CPC and I NWJ to obtain more ﬁne-grained bias-variance trade-offs.
V ARIATIONAL MUTUAL INFORMATION ESTIMATION AS OPTIMIZATION OVER DENSITY RATIOS
In this section, we unify several existing methods for variational mutual information estimation.
We ﬁrst show that variational mutual information estimation can be formulated as a constrained optimization problem, where the feasible set is ∆( Q ) , i.e. the valid density ratios with respect to Q .
Theorem 1.
∀ P, Q ∈ P ( X ) such that P (cid:28) Q we have
[Equation]
where the supremum is achived when r = d P/ d Q .
We defer the proof in Appendix A.
The above argument works for KL divergence between general distributions, but in this paper we focus on the special case of mutual information estimation.
For the remainder of the paper, we use P to represent the short-hand notation for the joint distribution
P ( X, Y ) and use Q to represent the short-hand notation for the product of marginals P ( X ) P ( Y ) .
3. A SUMMARY OF EXISTING VARIATIONAL METHODS
From Theorem 1, we can describe a general approach to variational MI estimation:
Obtain a density ratio estimate – denote the solution as r ; 2.
Project r to be close to ∆( Q ) – in practice we only have samples from Q , so we denote the solution as Γ( r ; Q n ) , where Q n is the empirical distribution of n i.i
d. samples from Q ; 3.
Estimate mutual information with E P [log Γ( r ; Q n )] .
We illustrate two examples of variational mutual information estimation that can be summarized with this approach.
In the case of Barber-Agakov, the proposed density ratio estimate is r BA = q φ ( x | y ) /p
( x ) (assuming that p ( x ) is known), which is guaranteed to be in ∆( Q ) because
[Equation]
for all conditional distributions q φ .
In the case of MINE / Donsker-Varadahn, the logarithm of the density ratio is estimated with T θ ( x , y ) ; the corresponding density ratio might not be normalized, so one could apply the following normalization for n samples:
[Equation]
] (8) where E Q n [ e T θ ] (the sample average) is an unbiased estimate of the partition function E Q [ e T θ ] ; Γ DV ( e T θ , Q n ) ∈ ∆( Q ) is only true when n → ∞ .
Similarly, we show I CPC is a lower bound to MI in Corollary 2, Appendix A, providing an alternative proof to the one in Poole et al.
These examples demonstrate that different mutual information estimators can be obtained in a procedural manner by implementing the above steps, and one could involve different objectives at each step.
For example, one could estimate density ratio via logistic regression (Hjelm et al., 2018;
Poole et al., 2019; Mukherjee et al., 2019) while using I NWJ or I MINE to estimate MI.
While logistic regression does not optimize for a lower bound for KL divergence, it provides density ratio estimates between P and Q which could be used for subsequent steps.
[Table]
3. G ENERATIVE AND DISCRIMINATIVE APPROACHES TO MI ESTIMATION
The above discussed variational mutual information methods can be summarized into two broad categories based on how the density ratio is obtained.
• The discriminative approach estimates the density ratio d P/ d Q directly; examples include the MINE, NWJ and CPC estimators.
• The generative approach estimates the densities of P and Q separately; examples include the BA estimator where a conditional generative model is learned.
In addition, we describe a generative approach that explicitly learns generative models (GM) for P ( X, Y ) , P ( X ) and P ( Y ) :
[Equation]
where p θ , p φ , p ψ are maximum likelihood estimates of P ( X, Y ) , P ( X ) and P ( Y ) respectively.
We can learn the three distributions with generative models, such as VAE (Kingma & Welling, 2013) or Normalizing ﬂows (Dinh et al., 2016), from samples.
We summarize various generative and discriminative variational estimators in Table 1.
Differences between two approaches While both generative and discriminative approaches can be summarized with the procedure in Section 3.1, they imply different choices in modeling, estimation and optimization.
• On the modeling side, the generative approaches might require more stringent assumptions on the architectures (e.g. likelihood or evidence lower bound is tractable), whereas the discriminative approaches do not have such restrictions.
• On the estimation side, generative approaches do not need to consider samples from the product of marginals P ( X ) P ( Y ) (since it can model P ( X, Y ) , P ( X ) , P ( Y ) separately), yet the discriminative approaches require samples from P ( X ) P ( Y ) ; if we consider a minibatch of size n , the number of evaluations for generative approaches is Ω( n ) whereas that for discriminative approaches it could be Ω( n 2 ) .
• On the optimization side, discriminative approaches may need additional projection steps to be close to ∆( Q ) (such as I MINE ), while generative approaches might not need to perform
this step (such as I BA ).
L IMITATIONS OF EXISTING VARIATIONAL ESTIMATORS 4. G OOD DISCRIMINATIVE ESTIMATORS REQUIRE EXPONENTIALLY LARGE BATCHES
In the ˆ I NWJ and ˆ I MINE estimators, one needs to estimate the “partition function” E Q [ r ] for some density ratio estimator r ; for example, ˆ I MINE needs this in order to perform the projection step Γ MINE ( r, Q n ) in Eq (8).
Note that the I NWJ
and I MINE lower bounds are maximized when r takes the optimal value r (cid:63) =
d P/ d Q .
However, the sample averages ˆ I MINE and ˆ I NWJ of E Q [ r (cid:63) ] could have a variance that scales exponentially with the ground-truth MI; we show this in Theorem 2.
Theorem 2.
Assume that the ground truth density ratio r (cid:63) =
d P/ d Q and Var Q [ r (cid:63) ] exist.
Let Q n denote the empirical distribution of n i.i.d.
samples from Q and let E Q n denote the sample average over Q n .
Then under the randomness of the sampling procedure, we have:
[Equation]
We defer the proof to Appendix A. Note that in the theorem above, we assume the ground truth density ratio r (cid:63) is already obtained, which is the optimal ratio for NWJ and MINE estimators.
As a natural consequence, the NWJ and MINE estimators under the optimal solution could exhibit variances that grow exponentially with the ground truth MI (recall that in our context MI is a KL divergence).
One could achieve smaller variances with some r (cid:54) =
r (cid:63) , but this guarantees looser bounds and higher bias.
Corollary 1.
Assume that the assumptions in Theorem 2 hold.
Let P m and Q n be the empirical distributions of m i.i.d.
samples from P and n i.i.d.
samples from Q , respectively.
Deﬁne
[Equation]
where r (cid:63)
=
d P/ d Q .
Then under the randomness of the sampling procedure, we have ∀ m ∈ N :
[Equation]
This high variance phenomenon has been empirically observed in Poole et al.
(Figure 3) for ˆ
I NWJ under various batch sizes, where the log-variance scales linearly with MI.
We also demonstrate this in Figure 2 (Section 6.1).
In order to keep the variance of ˆ I MINE and ˆ I NWJ relatively constant with growing MI, one would need a batch size of n =
Θ( e D KL ( P (cid:107) Q ) ) .
ˆ I CPC has small variance, but it would need n ≥ e D KL ( P (cid:107) Q ) to have small bias, as its estimations are bounded by log n .
4. S ELF - CONSISTENCY ISSUES FOR MUTUAL INFORMATION ESTIMATORS
If we consider X , Y to be high-dimensional, estimation of mutual information becomes more difﬁcult.
The density ratio between P ( X, Y ) and P ( X ) P ( Y ) could be very difﬁcult to estimate from ﬁnite samples without proper parametric assumptions (McAllester & Statos, 2018;
Zhao et al., 2018a).
Additionally, the exact value of mutual information is dependent on the deﬁnition of the sample space; given ﬁnite samples, whether the underlying random variable is assumed to be discrete or continuous will lead to different measurements of mutual information (corresponding to entropy and differential entropy, respectively).
In machine learning applications, however, we are often more interested in maximizing or minimizing mutual information (estimates), rather than estimating its exact value.
For example, if an estimator is off by a constant factor, it would still be useful for downstream applications, even though it can be highly biased.
To this end, we propose a set of self-consistency tests for any MI estimator ˆ
I , based on properties of mutual information:
(Independence) if X and Y are independent, then ˆ I ( X ; Y )
= 0 ;
(Data processing) for all functions g, h , ˆ I ( X ; Y ) ≥
ˆ I ( g ( X ); h ( Y )) and ˆ I ( X ; Y )
≈ ˆ I ([ X, g ( X )]; [ Y, h ( Y )]) where [ · , · ] denotes concatenation.
(Additivity) denote X 1 ,
X 2 as independent random variables that have the same distribution as X (similarly deﬁne Y 1 , Y 2 ), then ˆ I ([ X 1 , X 2 ]; [ Y 1 , Y 2 ]) ≈
· ˆ I ( X, Y ) .
These properties holds under both entropy and differential entropy, so they do not depend on the choice of the sample space.
While these conditions are necessary but obviously not sufﬁcient for accurate mutual information estimation, we argue that satisfying them is highly desirable for applications such as representation learning (Chen et al., 2016) and information bottleneck (Tishby &
Zaslavsky, 2015).
Unfortunately, none of the MI estimators we considered above pass all the selfconsistency tests when X, Y are images, as we demonstrate below in Section 6.2.
In particular, the generative approaches perform poorly when MI is low (failing in independence and data processing), whereas discriminative approaches perform poorly when MI is high (failing in additivity).
I MPROVED MI ESTIMATION VIA CLIPPED DENSITY RATIOS
To address the high-variance issue in the I NWJ and I MINE estimators, we propose to clip the density ratios when estimating the partition function.
We deﬁne the following clip function:
[Equation]
For an empirical distribution of n samples Q n , instead of estimating the partition function via E
Q n [ r ] , we instead consider E Q n [clip( r, e − τ , e τ )] where τ ≥ 0 is a hyperparameter; this is equivalent to clipping the log density ratio estimator between − τ and τ .
We can then obtain a following estimator with smoothed partition function estimates:
[Equation]
where T θ is a neural network that estimates the log-density ratio (similar to the role of T θ in ˆ I MINE ).
We term this the Smoothed Mutual Information “Lower-bound” Estimator (SMILE) with hyperparameter τ ; I SMILE converges to I MINE when τ → ∞ .
In our experiments, we consider learning the density ratio with logistic regression, similar to the procedure in Deep InfoMax (Hjelm et al., 2018).
The selection of τ affects the bias-variance trade-off when estimating the partition function; with a smaller τ , variance is reduced at the cost of (potentially) increasing bias.
In the following theorems, we analyze the bias and variance in the worst case for density ratio estimators whose actual partition function is S for some S ∈ (0 , ∞ ) .
Theorem 3.
Let r ( x ) :
X → R ≥ 0 be any non-negative measurable function such that (cid:82) r d Q = S ,
S ∈ (0 , ∞ ) and r ( x ) ∈ [0 , e K ] .
Deﬁne r τ ( x ) =
clip( r ( x ) , e τ , e − τ ) for ﬁnite, non-negative τ .
If τ < K , then the bias for using r τ to estimate the partition function of r satisﬁes:
[Equation]
if τ ≥ K , then
[Equation]
Theorem 4.
The variance of the estimator E Q n [ r τ ] (using n samples from Q ) satisﬁes:
[Equation]
We defer the proofs to Appendix A. Theorems 3 and 4 suggest that as we decrease τ , variance is decreased at the cost of potentially increasing bias.
However, if S is close to 1, then we could use small τ values to obtain estimators where both variance and bias are small.
We further discuss the bias-variance trade-off for a ﬁxed r over changes of τ in Theorem 3 and Corollary 3.
E XPERIMENTS 6. B ENCHMARKING ON MULTIVARIATE G AUSSIANS
First, we evaluate the performance of MI bounds on two toy tasks detailed in (Poole et al., 2019;
Belghazi et al., 2018), where the ground truth MI is tractable.
The ﬁrst task ( Gaussian ) is where ( x , y ) are drawn from a 20-d Gaussian distribution with correlation ρ , and the second task ( Cubic ) is the same as Gaussian
but we apply the transformation y
(cid:55)→ y 3 .
We consider three discriminative approaches ( I CPC , I NWJ , I SMILE ) and one generative approach ( I GM ).
For the discriminative approaches, we consider the joint critic in (Belghazi et al., 2018) and the separate critic in (van den
Oord et al., 2018).
For I GM we consider invertible ﬂow models (Dinh et al., 2016).
We train all models for 20k iterations, with the ground truth mutual information increasing by 2 per 4k iterations.
More training details are included in Appendix B 2 .
[Figure]
Figure 1 shows the estimated mutual information over the number of iterations.
In both tasks,
I CPC has high bias and I NWJ has high variance when the ground truth MI is high, whereas I SMILE has relatively low bias and low variance across different architectures and tasks.
Decreasing τ in the SMILE estimator decreases variances consistently but has different effects over bias; for example, under the joint critic bias is higher for τ = 5 . 0 in Gaussian but lower in Cubic .
I GM with ﬂow models has the best performance on Gaussian , yet performs poorly on Cubic , illustrating the importance of model parametrization in the generative approaches
[Figure]
In Figure
[Figure]
we compare the bias, variance and mean squared error (MSE) of the discriminative methods .
We observe that the variance of I NWJ increases exponentially with mutual information, which is consistent with our theory in Corollary 1.
On the other hand, the SMILE estimator is able to achieve much lower variances with small τ values; in comparison the variance of SMILE when τ = ∞ is similar to that of I NWJ in Cubic .
In Table 2, we show that I SMILE can have nearly two orders of magnitude smaller variance than I NWJ while having similar bias.
Therefore I SMILE enjoys lower MSE in this benchmark MI estimation task compared to I NWJ and I CPC .
6. S ELF - CONSISTENCY TESTS ON I MAGES
[Figure]
Next, we perform our proposed self-consistency tests on high-dimensional images (MNIST and
CIFAR10) under three settings, where the ground truth MI is difﬁcult to obtain (if not impossible).
These settings are illustrated in Figure 3.
The ﬁrst setting is where X is an image and Y is the same image where we mask the bottom rows, leaving the top t rows from X ( t is selected before evaluation).
The rationale behind this choice of Y is twofold: 1) I ( X ; Y ) should be non-decreasing with t ; 2) it is easier (compared to low-d representations) to gain intuition about the amount of information remaining in Y .
In the second setting, X corresponds to two identical images, and Y to the top t 1 , t 2 rows of the two images ( t 1 ≥ t 2 ); this considers the “data-processing” property.
In the third setting, X corresponds to two independent images, and Y to the top t rows of both; this considers the “additivity” property.
We compare four approaches: I CPC , I MINE , I SMILE and I GM .
We use the same CNN architecture for I CPC , I MINE and I SMILE , and use VAEs (Kingma & Welling, 2013) for I GM .
We include more experimental details and alternative image processing approaches in Appendix B.
[Figure]
Baselines We evaluate the ﬁrst setting with Y having varying number of rows t in Figure 4, where the estimations are normalized by the estimated ˆ I ( X ; X ) .
Most methods (except for I GM ) predicts zero MI when X and Y are independent, passing the ﬁrst self-consistency test.
Moreover, the estimated MI is non-decreasing with increasing t , but with different slopes.
As a reference, we show the validation accuracy of predicting the label where only the top t rows are considered.
Data-processing
In the second setting we set t 2 = t 1 − 3 .
Ideally, the estimator should satisfy ˆ
I ([ X, X ]; [ Y, h ( Y )]) /
ˆ I ( X, Y )
≈ 1 , as additional processing should not increase information.
We show the above ratio in Figure 5 under varying t 1 values.
All methods except for I MINE and I GM performs well in both datasets; I GM performs poorly in CIFAR10 (possibly due to limited capacity of VAE), whereas I MINE performs poorly in MNIST (possibly due to numerical stability issues).
[Figure]
[Figure]
Additivity In the third setting, the estimator should double its value compared to the baseline with the same t , i.e. ˆ I ([ X 1 , X 2 ]; [ Y 1 , Y 2 ]) /
ˆ I ( X, Y )
≈ 2 .
Figure 6 shows the above ratio under different values of t .
None of the discriminative approaches worked well in this case except when t is very small, when t is large this ratio converges to 1 (possibly due to initialization and saturation of the training objective).
I GM however, performs near perfectly on this test for all values of t .
D ISCUSSION
In this work, we discuss generative and discriminative approaches to variational mutual information estimation and demonstrate their limitations.
We show that estimators based on I NWJ and I MINE are prone to high variances when estimating with mini-batches, inspiring our I SMILE estimator that improves performances on benchmark tasks.
However, none of the approaches are good enough to pass the self-consistency tests.
The generative approaches perform poorly when MI is small (failing independence and data-processing tests) while the discriminative approaches perform poorly when
MI is large (failing additivity tests).
These empirical evidences suggest that optimization over these variational estimators are not necessarily related to optimizing MI, so the empirical successes with these estimators might have little connections to optimizing mutual information.
Therefore, it would be helpful to acknowledge these limitations and consider alternative measurements of information that are more suited for modern machine learning applications (Ozair et al., 2019; Tschannen et al., 2019).
A CKNOWLEDGEMENTS
This research was supported by AFOSR (FA9550-19-1-0024), NSF (#1651565, #1522054, #1733686), ONR, and FLI.
The authors would like to thank Shengjia Zhao, Yilun Xu and Lantao
Yu for helpful discussions.
A P ROOFS A. P ROOFS IN S ECTION
Theorem 1.
∀ P, Q ∈ P ( X ) such that P (cid:28) Q we have
[Equation]
where the supremum is achived when r = d P/ d Q .
Proof.
For every T ∈ L ∞ ( Q ) , deﬁne r T =
e T E Q [ e T ] , then r T ∈ ∆( Q ) and from the DonskerVaradhan inequality (Donsker & Varadhan, 1975)
[Equation]
Moreover, we have:
[Equation]
which completes the proof.
Corollary 2.
∀ P, Q ∈ P ( X ) such that P (cid:28) Q , ∀ f θ : X → R ≥
we have
[Equation]
Proof.
nI
[Equation] [Equation]
Since
[Equation] [Equation]
we can apply Theorem 1 to obtain:
[Equation]
where Y n 1 denotes the concatenation of n independent random variables ( Y 1 , . .
, Y n ) and
P
[Equation]
X i
[Equation]
Y
[Equation]
P
[Equation]
i
[Equation]
Y
[Equation]
P
[Equation]
) is the joint distribution of P ( X i , Y n 1 ) .
A. P ROOFS IN S ECTION
Theorem 2.
Assume that the ground truth density ratio r (cid:63) =
d P/ d Q and Var Q [ r (cid:63) ] exist.
Let Q n denote the empirical distribution of n i.i.d.
samples from Q and let E Q n denote the sample average over Q n .
Then under the randomness of the sampling procedure, we have:
[Equation]
Proof.
Consider the variance of r (cid:63) ( x ) when x ∼ Q :
[Equation]
where (29) uses the deﬁnition of variance, (30) uses the deﬁnition of Radon-Nikodym derivative to change measures, (31) uses Jensen’s inequality over log , and (32) uses the deﬁnition of KL divergences.
The variance of the mean of n i.i.d.
random variables then gives us:
[Equation]
which is the ﬁrst part of the theorem.
As n → ∞ , Var Q [ E Q n [ r ]] → 0 , so we can apply the delta method:
[Equation]
Applying f
=
log
and E [ X ] = 1 gives us the second part of the theorem:
[Equation]
which describes the variance in the asymptotic sense.
Corollary 1.
Assume that the assumptions in Theorem 2 hold.
Let P m and Q n be the empirical distributions of m i.i.d.
samples from P and n i.i.d.
samples from Q , respectively.
Deﬁne
[Equation]
where r (cid:63)
=
d P/ d Q .
Then under the randomness of the sampling procedure, we have ∀ m ∈ N :
[Equation]
Proof.
Since P m and Q n are independent, we have
[Equation]
[Equation]
) and
[Equation]
(38) which completes the proof.
A. P ROOFS IN S ECTION
Theorem 3.
Let r ( x ) :
X → R ≥ 0 be any non-negative measurable function such that (cid:82) r d Q = S ,
S ∈ (0 , ∞ ) and r ( x ) ∈ [0 , e K ] .
Deﬁne r τ ( x ) =
clip( r ( x ) , e τ , e − τ ) for ﬁnite, non-negative τ .
If τ < K , then the bias for using r τ to estimate the partition function of r satisﬁes:
[Equation]
if τ ≥ K , then
[Equation]
Proof.
We establish the upper bounds by ﬁnding a worst case r to ﬁnd the largest | E Q [ r ] − E Q [ r τ ] | .
First, without loss of generality, we may assume that r ( x ) ∈ ( −∞ , e − τ ] ∪ [ e τ , ∞ ) for all x ∈ X .
Otherwise, denote X τ ( r ) = { x ∈ X : e − τ < r ( x ) <
e τ } as the (measurable) set where the r ( x ) values are between e − τ and e τ .
Let
[Equation] [Equation]
be the integral of r over X τ ( r ) .
We can transform r ( x ) for all x ∈ X τ ( r ) to have values only in { e − τ , e τ } and still integrate to V τ ( r ) , so the expectation under Q is not changed.
Then we show that we can rescale all the values above e τ and below e τ to the same value without changing the expected value under Q .
We denote
[Equation]
where e K 1 and e K 2 represents the mean of r ( x ) for all r ( x ) ≤ e − τ and r ( x
) ≥ e τ respectively.
We then have:
[Equation]
so we can parametrize E Q [ r ] via K 1 and K 2 .
Since E Q [ r ] = S by assumption, we have:
[Equation]
and from the deﬁnition of r τ ( x ) :
[Equation]
We can obtain an upper bound once we ﬁnd max g ( K 1 , K 2 ) and min g ( K 1 , K 2 ) .
First, we have:
[Equation]
Therefore, g ( K 1 , K 2 ) is largest when K 1 → ∞ , K 2 = τ and smallest when K 1 = τ, K 2 → ∞ .
[Equation]
Therefore,
[Equation]
The proof for Theorem 3 simply follows the above analysis for ﬁxed K .
When τ < K , we consider the case when K 1 → ∞ , K 2 = τ and K
= τ, K 2 =
K ; when τ > K only the smaller values will be clipped, so the increased value is no larger than the case where K 1 → ∞ , K
[Equation]
K :
[Equation]
where e K ≥ S from the fact that (cid:82) r d Q = S .
Theorem 4.
The variance of the estimator E Q n [ r τ ] (using n samples from Q ) satisﬁes:
[Equation] [Equation]
Proof.
Since r τ ( x ) is bounded between e τ and e − τ , we have
[Equation]
[Equation]
Taking the mean of n independent random variables gives us the result.
Combining Theorem 3 and 4 with the bias-variance trade-off argument, we have the following:
Corollary 3.
Let r ( x ) :
X → R ≥ 0 be any non-negative measurable function such that (cid:82) r d Q = S ,
S ∈ (0 , ∞ ) and r ( x ) ∈ [0 , e K ] .
Deﬁne r τ ( x ) =
clip( r ( x ) , e τ , e − τ ) for ﬁnite, non-negative τ and E
Q n as the sample average of n i.i.d.
samples from Q .
If τ < K , then
E Q [( r − E Q n [ r τ ])
[Equation]
max
[Equation]
If τ ≥ K , then:
[Equation]
B A DDITIONAL E XPERIMENTAL D ETAILS B. B ENCHMARK T ASKS
Tasks We sample each dimension of ( x , y ) independently from a correlated Gaussian with meanand correlation of ρ , where X =
Y = R 20 .
The true mutual information is computed as:
[Equation]
The initial mutual information is 2 , and we increase the mutual information by 2 every 4 k iterations, so the total training iterations is 20 k .
Architecture and training procedure For all the discriminative methods, we consider two types of architectures – joint and separable .
The joint architecture concatenates the inputs x , y , and then passes through a two layer MLP with 256 neurons in each layer with ReLU activations at each layer.
The separaable architecture learns two separate neural networks for x and y (denoted as g ( x ) and h ( y ) ) and predicts g ( x ) (cid:62) h ( y ) ; g and h are two neural networks, each is a two layer MLP withneurons in each layer with ReLU activations at each layer; the output of g and h are 32 dimensions.
For the generative method, we consider the invertible ﬂow architecture described in (Dinh et al., 2014; 2016).
p θ , p φ , p ψ are ﬂow models with 5 coupling layers (with scaling), where each layer contains a neural network with 2 layers of 100 neurons and ReLU activation.
For all the cases, we use with the Adam optimizer (Kingma & Ba, 2014) with learning rate 5 × 10 − 4 and β 1 = 0 .
9 , β 2 = 0 .
999 and train for 20 k iterations with a batch size of 64 , following the setup in Poole et al.
Additional results We show the bias, variance and mean squared error of the discriminative approaches in Table 2.
We include additional results for I SMILE with τ = 10 . 0 .
[Table]
We show the bias, variance and MSE results in Figure 7.
We also evaluate the variance of estimating E
Q n [ r τ ] (partition function with clipped ratios) for different values of τ in the SMILE estimator in
Figure 8b.
With smaller τ we see a visible decrease in terms of variance in this term; this is consistent with the variance estimates in Figure 7, as there the variance of E P n [log r ] is also considered.
[Figure]
[Figure] [Figure]
S ELF - CONSISTENCY E XPERIMENTS
Tasks We consider three tasks with the mutual information estimator
ˆ I : 1.
ˆ I ( X ; Y ) where X is an image from MNIST (LeCun et al., 1998) or CIFAR
(Krizhevsky et al., 2012) and Y is the top t rows of X .
To simplify
architecture designs, we simply mask out the bottom rows to be zero, see Figure 3.
ˆ I ([ X, X ]; [ Y ; h ( Y )]) where X is an image, Y is the top t rows of X , h ( Y ) is the top ( t − 3) rows of Y and [ · , · ] denotes concatenation.
Ideally, the prediction should be close to ˆ I ( X ; Y ) .
ˆ I ([ X 1 , X 2 ] , [ Y 1 , Y 2 ]) where X 1 and X 2 are independent images from MNIST or CIFAR10, Y 1 and Y 2 are the top t rows of X 1 and X 2 respectively.
Ideally, this prediction should be close to 2 · ˆ I ( X ; Y ) .
Architecture and training procedure We consider the same architecture for all the discriminative approaches.
The ﬁrst layer is a convolutional layer with 64 output channels, kernel size of 5 , stride of 2 and padding of 2 ; the second layer is a convolutional layer with 128 output channels, kernel size of 5 , stride of 2 and padding of 2 .
This is followed another fully connected layer with 1024 neurons and ﬁnally a linear layer that produces an output of 1 .
All the layers (except the last one) use ReLU activations.
We stack variables over the channel dimension to perform concatenation.
For the generative approach, we consider the following VAE architectures.
The encoder architecture is identical to the discriminative approach except the last layer has 20 outputs that predict the mean and standard deviations of 10 Gaussians respectively.
The decoder for MNIST is a two layer
MLP with 400 neurons each; the decoder for CIFAR10 is the corresponding transposed convolution network for the encoder.
All the layers (except the last layers for encoder and decoder) use ReLU activations.
For concatenation we stack variables over the channel dimension.
For all the cases, we use with the Adam optimizer (Kingma & Ba, 2014) with learning rate 10 − 4 and β 1 = 0 .
9 , β 2 = 0 .
For I GM we train for 10 epochs, and for the discriminative methods, we train for 2 epochs, due to numerical stability issues of I MINE .
Additional experiments on scaling, rotation and translation We consider additional benchmark experiments on MNIST where instead of removing rows, we apply alternative transformations such as random scaling, rotation and translations.
For random scaling, we upscale the image randomly by 1x to 1.2x; for random rotation, we randomly rotate the image between ± 20 degrees; for random translation, we shift the image randomly by no more than 3 pixels horizontally and vertically.
We consider evaluating the data processing and additivity properties, where the ideal value for the former is no more than 1, and the ideal value for the latter is 2.
From the results in Table 3, none of the considered approaches achieve good results in all cases.
[Table]