M UTUAL I NFORMATION G RADIENT E STIMATION FOR R EPRESENTATION L EARNING
A BSTRACT
Mutual Information (MI) plays an important role in representation learning.
However, MI is unfortunately intractable in continuous and high-dimensional settings.
Recent advances establish tractable and scalable MI estimators to discover useful representation.
However, most of the existing methods are not capable of providing an accurate estimation of MI with low-variance when the MI is large.
We argue that directly estimating the gradients of MI is more appealing for representation learning than estimating MI in itself.
To this end, we propose the Mutual
Information Gradient Estimator (MIGE) for representation learning based on the score estimation of implicit distributions.
MIGE exhibits a tight and smooth gradient estimation of MI in the high-dimensional and large-MI settings.
We expand the applications of MIGE in both unsupervised learning of deep representations based on InfoMax and the Information Bottleneck method.
Experimental results have indicated the signiﬁcant performance improvement in learning useful representation.
I NTRODUCTION
Mutual information (MI) is an appealing metric widely used in information theory and machine learning to quantify the amount of shared information between a pair of random variables.
Speciﬁcally, given a pair of random variables x , y , the MI, denoted by I ( x ; y ) , is deﬁned as
[Equation]
where E is the expectation over the given distribution.
Since MI is invariant to invertible and smooth transformations, it can capture non-linear statistical dependencies between variables (Kinney & Atwal, 2014).
These appealing properties make it act as a fundamental measure of true dependence.
Therefore, MI has found applications in a wide range of machine learning tasks, including feature selection (Kwak & Choi, 2002; Fleuret, 2004; Peng et al., 2005), clustering (M¨uller et al., 2012;
Ver Steeg & Galstyan, 2015), and causality (Butte & Kohane, 1999).
It has also been pervasively used in science, such as biomedical sciences (Maes et al., 1997), computational biology (Krishnaswamy et al., 2014), and computational neuroscience (Palmer et al., 2015).
Recently, there has been a revival of methods in unsupervised representation learning based on MI.
A seminal work is the InfoMax principle (Linsker, 1988), where given an input instance x , the goal of the InfoMax principle is to learn a representation E ψ ( x ) by maximizing the MI between the input and its representation.
A growing set of recent works have demonstrated promising empirical performance in unsupervised representation learning via MI maximization (Krause et al., 2010;
Hu et al., 2017; Alemi et al., 2018b;
Oord et al., 2018; Hjelm et al., 2019).
Another closely related work is the Information Bottleneck method (Tishby et al., 2000; Alemi et al., 2017), where MI is used to limit the contents of representations.
Speciﬁcally, the representations are learned by extracting taskrelated information from the original data while being constrained to discard parts that are irrelevant to the task.
Several recent works have also suggested that by controlling the amount of information between learned representations and the original data, one can tune desired characteristics of trained models such as generalization error (Tishby & Zaslavsky, 2015; Vera et al., 2018), robustness (Alemi et al., 2017), and detection of out-of-distribution data (Alemi et al., 2018a).
Despite playing a pivotal role across a variety of domains, MI is notoriously intractable.
Exact computation is only tractable for discrete variables, or for a limited family of problems where the probability distributions are known.
For more general problems, MI is challenging to analytically compute or estimate from samples.
A variety of MI estimators have been developed over the years, including likelihood-ratio estimators (Suzuki et al., 2008), binning (Fraser & Swinney, 1986; Darbellay & Vajda, 1999; Shwartz-Ziv & Tishby, 2017), k-nearest neighbors (Kozachenko & Leonenko, 1987; Kraskov et al., 2004; P´erez-Cruz, 2008; Singh & P´oczos, 2016), and kernel density estimators (Moon et al., 1995; Kwak & Choi, 2002; Kandasamy et al., 2015).
However, few of these mutual information estimators scale well with dimension and sample size in machine learning problems (Gao et al., 2015).
In order to overcome the intractability of MI in the continuous and high-dimensional settings, Alemi et al.
(2017) combines variational bounds of Barber & Agakov (2003) with neural networks for the estimation.
However, the tractable density for the approximate distribution is required due to variational approximation.
This limits its application to the general-purpose estimation, since the underlying distributions are often unknown.
Alternatively, the Mutual Information Neural Estimation (MINE, Belghazi et al.
and the Jensen-Shannon MI estimator (JSD, Hjelm et al.
(2019)) enable differentiable and tractable estimation of MI by training a discriminator to distinguish samples coming from the joint distribution or the product of the marginals.
In detail, MINE employs a lower-bound to the MI based on the Donsker-Varadhan representation of the KL-divergence, and
JSD follows the formulation of f-GAN KL-divergence.
In general, these estimators are often noisy and can lead to unstable training due to their dependence on the discriminator used to estimate the bounds of mutual information.
As pointed out by Poole et al.
(2019), these unnormalized critic estimators of MI exhibit high variance and are challenging to tune for estimation.
An alternative low-variance choice of MI estimator is Information Noise-Contrastive Estimation (InfoNCE, Oord et al.
(2018)), which introduces the Noise-Contrastive Estimation with ﬂexible critics parameterized by neural networks as a bound to approximate MI.
Nonetheless, its estimation saturates at log of the batch size and suffers from high bias.
Despite their modeling power, none of the estimators are capable of providing accurate estimation of MI with low variance when the MI is large and the batch size is small (Poole et al., 2019).
As supported by the theoretical ﬁndings in McAllester &
Statos (2018), any distribution-free high-conﬁdence lower bound on entropy requires a sample size exponential in the size of the bound.
More discussions about the bounds of MI and their relationship can be referred to Poole et al.
In summary, existing estimators ﬁrst approximate MI and then use these approximations to optimize the associated parameters.
For estimating MI based on any ﬁnite number of samples, there exists an inﬁnite number of functions, with arbitrarily diverse gradients, that can perfectly approximate the true MI at these samples.
However, these approximate functions can lead to unstable training and poor performance in optimization due to gradients discrepancy between approximate estimation and true MI.
Estimating gradients of MI rather than estimating MI may be a better approach for MI optimization.
To this end, to the best of our knowledge, we ﬁrstly propose the Mutual Information
Gradient Estimator (MIGE) in representation learning.
In detail, we estimate the score function of an implicit distribution, ∇ x log q ( x ) , to achieve a general-purpose MI gradient estimation for representation learning.
In particular, to deal with high-dimensional inputs, such as text, images and videos, score function estimation via Spectral Stein Gradient Estimator (SSGE) (Shi et al., 2018) is computationally expensive and complex.
We thus propose an efﬁcient high-dimensional score function estimator to make SSGE scalable.
To this end, we derive a new reparameterization trick for the representation distribution based on the lower-variance reparameterization trick proposed by Roeder et al.
We summarize the contributions of this paper as follows:
M u t u a l I n f o r m a t
i
o n d = [Figure] MINE [Figure]
G r a d i e n t
[Figure]
d =0.
[Figure]
d =
[Figure]
• We propose the Mutual Information Gradient Estimator (MIGE) for representation learning based on the score function estimation of implicit distributions.
Compared with MINE and MINE- f , MIGE provides a tighter and smoother gradient estimation of MI in a highdimensional and large-MI setting, as shown in Figure 1 of Section 4.
• We propose the Scalable SSGE to alleviate the exorbitant computational cost of SSGE in high-dimensional settings.
• To learn meaningful representations, we apply SSGE as gradient estimators for both InfoMax and Information Bottlenck, and have achieved improved performance than their corresponding competitors.
S CALABLE S PECTRAL S TEIN G RADIENT E STIMATOR
Score estimation of implicit distributions has been widely explored in the past few years (Song et al., 2019; Li & Turner, 2017; Shi et al., 2018).
A promising method of score estimation is the Stein gradient estimator (Li & Turner, 2017; Shi et al., 2018), which is proposed for implicit distributions.
It is inspired by generalized Steins identity (Gorham & Mackey, 2015; Liu & Wang, 2016) as follows.
Steins identity.
Let q ( x ) be a continuously differentiable (also called smooth) density supported on X ⊆ R d , and h ( x ) =
[ h 1 ( x ) , h 2 ( x ) , . . .
, h d (cid:48)
( x )]
T is a smooth vector function.
Further, the boundary conditions on h is
[Equation]
, or [Equation] Under this condition, the following identity can be easily checked using integration by parts, assuming mild zero boundary conditions on h ,
[Equation]
Here h is called the Stein class of q ( x ) if Steins identity Eq.
(3) holds.
Monte Carlo estimation of the expectation in Eq.
(3) builds the connection between ∇ x log q ( x ) and the samples from q ( x ) in
Steins identity.
For modeling implicit distributions, Motivated by Steins identity, Shi et al.
(2018) proposed Spectral Stein Gradient Estimator (SSGE) for implicit distributions based on Stein’s identity and a spectral decomposition of kernel operators where the eigenfunctions are approximated by the Nystr ¨ o m method.
Below we brieﬂy review SSGE.
More details refer to Shi et al.
Specifically, we denote the target gradient function to estimate by g : X → R d : g ( x ) = ∇ x log q ( x ) .
The i th component of the gradient is g i ( x ) = ∇ x i log q ( x ) .
We assume g 1 , . .
. , g d ∈ L 2 ( X , q ) .
{ ψ j } j ≥ 1 denotes an orthonormal basis of L 2 ( X , q ) .
We can expand g i ( x ) into the spectral series, i.e., g i ( x ) =
(cid:80) ∞
j =1 β ij ψ j ( x ) .
The value of the j th eigenfunction ψ j at x can be approximated by the Nystr ¨ o m method
(Xu et al., 2015).
Due to the orthonormality of eigenfunctions { ψ j } j ≥ 1 , there is a constraint under the probability measure q(.): (cid:82) ψ
i ( x ) ψ j ( x ) q ( x ) d x = δ ij , where δ ij = 1 [ i = j ] .
Based on this constraint, we can obtain the following equation for { ψ j } j ≥ 1 :
[Equation]
where k ( . ) is a kernel function.
The left side of the above equation can be approximated by the
Monte Carlo estimate using i.i.d.
samples x 1 , ..., x M from q ( . ) :
1 M K ψ ≈ µψ , where K is
the
Gram Matrix and ψ = (cid:2) ψ (cid:0) x 1 (cid:1) , . .
, ψ (cid:0) x M (cid:1)(cid:3) (cid:62) .
We can solve this eigenvalue problem by choose the J largest eigenvalues λ 1 ≥ · · · ≥ λ J for K .
u
j denotes the eigenvector of the Gram matrix.
The approximation for { ψ j } j ≥ 1 can be obtained combined with Eq.
(4) as following: ψ j ( x [Equation]
≈ ˆ ψ j [Equation] x
[Equation]
Furthermore, based on the orthonormality of { ψ j } j ≥ 1 , we can easily obtain β ij = − E q ∇ x i ψ j ( x ) .
By taking derivative both sides of Eq.
(4), we can show that:
[Equation]
Then we can estimate as following:
[Equation]
Finally, by truncating the expansion to the ﬁrst J terms and plugging in the Nystr ¨ o m approximations of { ψ j } j ≥ 1 , we can get the score estimator:
[Equation]
In general, representation learning for large-scale datasets is usually costly in terms of storage and computation.
For instance, the dimension of images in the STL-10 dataset is 96 × 96 × 3 (i.e., the vector length is 27648).
This makes it almost impossible to directly estimate the gradient of MI between the input and representation.
To alleviate this problem, we introduce random projection (RP) (Bingham & Mannila, 2001) to reduce the dimension of x .
We brieﬂy review RP.
More details refer to Bingham & Mannila (2001).
RP projects the original d -dimensional data into a k -dimensional ( k << d ) subspace.
Concretely, let matrix X d × N denotes the original set of N d -dimensional data, the projection of the original data X RPk × N is obtained by introducing a random matrix R k × d whose columns have unit length, as follows (Bingham & Mannila, 2001), X RPk × N = R k ×
d X d × N .
After the RP, distance between two original data vectors can be approximated by scaled Euclidean distance of these vectors in reduced spaces:
[Equation]
(8) where x 1 and x 2 denote the two data vectors in the original large dimensional space.
Based on the principle of RP, we can derive a Salable Spectral Stein Gradient Estimator, which is an efﬁcient high-dimensional score function estimator.
One can show that the RBF kernel satisﬁes
Steins identity (Liu & Wang, 2016).
Shi et al.
(2018) also shows that it is a promising choice for
SSGE with a lower error bound.
To reduce the computation of the kernel similarities of SSGE in high-dimensional settings, we replace the input of SSGE with a projections obtained by RP according to the approximation of Eq.
(8) for the computation of the RBF kernel.
M UTUAL I NFORMATION G RADIENT E STIMATOR
As gradient estimation is a straightforward and effective method in optimization, we propose a gradient estimator for MI based on score estimation of implicit distributions, which is called Mutual
Information Gradient estimator (MIGE).
In this section, we focus on three most general cases of MI gradient estimation for representation learning, and derive the corresponding MI gradient estimator for these circumstances.
We outline the general setting of training an encoder to learn a representation.
Let X and Z be the domain, and E ψ : X → Z with parameters ψ denotes a continuous and (almost everywhere) differentiable parametric function, which is usually a neural network, namely an encoder.
p ( x ) denotes the empirical distribution given the input data x ∈ X .
We can obtain the representation of the input data through the encoder, z = E ψ ( x ) .
q ψ ( z ) is deﬁned as the marginal distribution induced by pushing samples from p ( x ) through encoder E ψ ( . )
We also deﬁne q ψ ( x , z ) as the joint distribution with x and z , which is determined by encoder E ψ ( . ) .
Circumstance I .
Given that the encoder E ψ ( . ) is deterministic, our goal is to estimate the gradient of MI between input x and encoder output z w.r.t.
the encoder parameters ψ .
There is a close relationship between mutual information and entropy, which is as following: I ψ ( x ; z ) =
H ( x ) +
H ψ ( z ) − H ψ ( x , z ) .
Here H ( x ) is data entropy and not relevant to ψ .
The optimization of I ψ ( x , z ) with parameters ψ can neglect the entry H ( x ) .
We decompose the gradient of the entropy of q ψ ( z ) and q ψ ( x , z ) as (see Appendix A):
[Equation]
Hence, we can represent the gradient of MI between input x and encoder output z w.r.t.
encoder parameters ψ as following:
[Equation]
However, this equation is intractable since an expectation w.r.t q ψ ( z ) is directly not differentiable w.r.t ψ .
Roeder et al.
(2017) proposed a general variant of the standard reparameterization trick for the variational evidence lower bound, which demonstrates lower-variance.
To address above problem, we adapt this trick for MI gradient estimator in representation learning.
Speciﬁcally, we can obtain the samples from the marginal distribution of z by pushing samples from the data empirical distribution p ( x ) through E ψ ( . ) for representation learning.
Hence we can reparameterize the representations variable z ∼ q ψ ( z ) using a differentiable transformation:
z = E ψ ( x ) with x ∼ [Equation] where the data empirical distribution p ( x ) is independent of encoder parameters ψ .
This reparameterization can rewrite an expectation w.r.t q ψ ( z ) and q ψ ( x , z ) such that the Monte Carlo estimate of the expectation is differentiable w.r.t ψ .
Relying on this reparameterization trick, we can represent the gradient of MI w.r.t.
encoder parameters ψ in Eq.
10 as follows:
[Equation]
where the score function ∇ z log q ψ ( E ψ ( x )) can be estimated based on i.i.d.
samples from an implicit density q ψ ( E ψ ( x ))
(Shi et al., 2018; Song et al., 2019).
The samples form the joint distribution q ψ ( x , z ) are produced as following: we sample observations from empirical distribution p ( x ) ; then the corresponding samples of z is obtained through E ψ ( . ) .
Hence we can also estimate ∇ ( x , z ) log q ( x , E ψ ( x )) based on i.i.d.
samples from q ψ ( x , E ψ ( x )) .
∇ ψ E ψ ( x ) and ∇ ψ ( x , E ψ ( x )) are directly computed with x .
Circumstance II .
Assume that we encode the input to latent data space
h
= C ψ ( x ) that reﬂects useful structure in the data.
Next, we summarize this latent variable mapping into ﬁnal representations, E ψ ( x ) =
f ψ ◦ C ψ ( x ) .
The gradient estimator of MI between h and z is represented by the data reparameterization trick as follows:
[Equation]
Circumstance III .
Consider stochastic encoder function E ψ ( ., (cid:15) ) where (cid:15) is an auxiliary variable with independent marginal p ( (cid:15) ) .
By utilizing data reparameterization trick and reparameterization trick, we can represent the gradient of the conditional entropy H ψ ( z | x ) as following (see Appendix
A):
[Equation]
where the term ∇ ( z | x ) log q ( E ψ ( x , (cid:15) )
| x ) can be easily estimated by score estimation.
Based on the condition entropy gradient estimation in Eq.
(14), the gradient estimator of MI between input and encoder output can be represented as following:
[Equation]
In practical MI optimization, we can construct MIGE of the full dataset based on mini-batch Monte
Carlo estimates.
We have provided an algorithm description for MIGE in Appendix B.
T OY E XPERIMENT
Recently, MINE and MINE-
f enable effective computation of MI in the continuous and highdimensional settings.
To compare with MINE and MINE- f , we evaluate MIGE in the correlated
Gaussian problem taken from (Belghazi et al., 2018).
Experimental Settings .
We consider two random variables x and y ( x , y ∈ R d ), coming from a 2 d -dimension multivariate Gaussian distribution.
The component-wise correlation of x and y is deﬁned as follows: corr ( x i , y i ) =
δ ij ρ, ρ ∈ ( − 1 , 1) , where δ ij is Kronecker’s delta and ρ is the correlation coefﬁcient.
Since MI is invariant to smooth transformations of x , y , we only consider standardized Gaussian for marginal distribution p ( x ) and p ( y ) .
The gradient of MI w.r.t ρ has the analytical solution: ∇ ρ I ( x ; y )
=
ρd 1 − ρ 2 .
We apply MINE and MINE- f to estimate MI of x , y by sampling from the correlated Gaussian distribution and its marginal distributions, and the corresponding gradient of MI w.r.t ρ can be computed by backpropagation implemented in Pytorch.
Results .
Fig.1 presents our experimental results in different dimensions d
= { 5 , 10 , 20 } .
In the case of low-dimensional ( d = 5) , all the estimators give promising estimation of MI and its gradient.
However, the MI estimation of MINE and MINE-
f are unstable due to its relying on a discriminator to produce estimation of the bound on MI.
Hence, as showed in Fig.1, corresponding estimation of
MI and its gradient is unsmoothed.
As the dimension d and the absolute value of correlation coefﬁcient | ρ | increase, MINE and MINE- f are apparently hard to reach the True MI, and their gradient estimation of MI is thus high biased.
This phenomenon would be more signiﬁcant in the case of high-dimensional or large MI.
Contrastively, MIGE demonstrates the signiﬁcant improvement over
MINE and MINE-
f when estimating MI gradient between twenty-dimensional random variables x , y .
In this experiment, we compare our method with two baselines on an analyzable problem and ﬁnd that the gradient curve estimated by our method is far superior to other methods in terms of smoothness and tightness in a high-dimensional and large-MI setting compared with MINE and
MINE- f .
A PPLICATIONS
To demonstrate the performance in downstream tasks, we deploy MIGE to Deep InfoMax (Hjelm et al., 2019) and Information Bottleneck (Tishby et al., 2000) respectively, namely replacing the [Table] original MI estimators with MIGE.
We ﬁnd that MIGE achieves higher and more stable classiﬁcation accuracy, which indicating its good gradient estimation performance in practical applications.
5. D EEP I NFO M AX
Discovering useful representations from unlabeled data is one core problem for deep learning.
Recently, a growing set of methods is explored to train deep neural network encoders by maximizing the mutual information between its input and output.
A number of methods based on tractable variational lower bounds, such as JSD and infoNCE, have been proposed to improve the estimation of
MI between high dimensional input/output pairs of deep neural networks (Hjelm et al., 2019).
To compare with JSD and infoNCE, we expand the application of MIGE in unsupervised learning of deep representations based on the InfoMax principle.
Experimental Settings. For consistent comparison, we follow the experiments of Deep InfoMax(DIM) 1 to set the experimental setup as in Hjelm et al.
We test DIM on image datasets
CIFAR-10, CIFAR-100 and STL-10 to evaluate our MIGE.
For the high-dimensional images in STL-10, directly applying SSGE is almost impossible since it results in exorbitant computational cost.
Our proposed Scalable SSGE is applied, to reduce the dimension of images and achieve reasonable computational cost.
As mentioned in Hjelm et al.
(2019), non-linear classiﬁer is chosen to evaluate our representation, After learning representation, we freeze the parameters of the encoder and train a non-linear classiﬁer using the representation as the input.
The same classiﬁers are used for all methods.
Our baseline results are directly copied from Hjelm et al.
(2019) or by running the code of author.
Results. As shown in Table 1, MIGE outperforms all the competitive models in DIM experiments on CIFAR-10 and CIFAR-100.
Besides the numerical improvements, it is notable that our model have the less accuracy decrease across layers than that of DIM(JSD) and DIM(infoNCE).
The results indicate that, compared to variational lower bound methods, MIGE gives more favorable gradient direction, and demonstrates more power in controlling information ﬂows without signiﬁcant loss.
With the aid of Random Projection, we could evaluate on bigger datasets, e.g., STL-10. Tableshows the result of DIM experiments on STL-10.
We can observe signiﬁcant improvement over the baselines when RP to 512d.
Note that our proposed gradient estimator can also be extended to the multi-view setting(i.e.
, with local and global features) of DIM, it is beyond the scope of this paper.
More discussions refer to Appendix C.
Ablation Study. To verify the effect of different dimensions of Random Projection on classiﬁcation accuracy in DIM experiments, we conduct an ablation study on STL-10 with the above experimental settings. Varying RP dimension k ∈ { 16 , 32 , 64 , 128 , 256 , 512 , 1024 } , we measure the classiﬁcation accuracy of Y(64) which is shown in Fig.2. We ﬁnd that the classiﬁcation accuracy increases with
RP dimension from 16 to 128.
After that, the approximation in Equ.(8) with the further increase of the RP dimension reaches saturation, while bringing extra computational costs.
[Table] 5. I NFORMATION B OTTLENECK Information Bottleneck (IB) has been widely applied to a variety of application domains, such as classiﬁcation (Tishby & Zaslavsky, 2015;
Alemi et al., 2017; Chalk et al., 2016; Kolchinsky et al., 2017), clustering (Slonim & Tishby, 2000), and coding theory and quantization (Zeitler et al., 2008;
Courtade & Wesel, 2011).
In particular, given the input variable x and the target variable y , the goal of the IB is to learn a representation of x (denoted by the variable z ) that satisﬁes the following characteristics:
1) z is sufﬁcient for the target y , that is, all information about target y contained in x should also be contained in z .
In optimization, it should be 2) z is minimal.
In order not to contain irrelevant information that is not related to y , z is required to contain the smallest information among all sufﬁcient representations.
The objective function of IB is written as follows:
[Equation]
Equivalently, by introducing a Lagrangian multiplier β , the IB method can maximize the following objective function: G IB =
I ( z ; y ) − βI ( z ; x ) .
Further, it is generally acknowledged that I ( z ; y ) =
H ( y ) − H ( y | z ) , and H ( y ) is constant.
Hence we can also minimize the objective function of the following form:
[Equation]
where β ≥ 0 plays a role in trading off the sufﬁciency and minimality.
Note that the above formulas omit the parameters for simplicity.
To overcome the intractability of MI in the continuous and high-dimension setting, Alemi et al.
(2017) presents a variational approximation to IB, which adopts deep neural network encoder to produce a conditional multivariate normal distribution, called Deep Variational Bottleneck (DVB).
Rencently, DVB is exploited to restrict the capacity of discriminators in GANs (Peng et al., 2019).
However, a tractable density is required for the approximate posterior in DVB due to their reliance on a variational approximation while MIGE does not.
To evaluate our method, we compare MIGE-IB with DVB and MINE-IB in IB application.
We demonstrate an implementation of the IB objective on permutation invariant MNIST using MIGE.
Experiments. For consistent comparison, we adopt the same architecture and empirical settings used in Alemi et al.
(2017) except that the initial learning rate of 2e-4 is set for Adam optimizer, and exponential decay with decaying rate by a factor of 0.96 was set for every 2 epochs.
The implementation of DVB is available from its authors 2 .
Under these experimental settings, we use our MI Gradient Estimator to replace the MI estimator in DVB experiment.
The threshold of score function’s Stein gradient estimator is set as 0 .
The threshold is the hyper-parameter of Spectral
Stein Gradient Estimator (SSGE), and it is used to set the kernel bandwidth of RBF kernel.
Our results can be seen in Table 3 and it manifests that our proposed MIGE [Table]
C ONCLUSION
In this paper, we present a gradient estimator, called Mutual Information Gradient Estimator (MIGE), to avoid the various problems met in direct mutual information estimation.
We manifest the effectiveness of gradient estimation of MI over direct MI estimation by applying it in unsupervised or supervised representation learning.
Experimental results have indicated the remarkable improvement over MI estimation in the Deep InfoMax method and the Information Bottleneck method.
A CCKNOWLEDGEMENT
This work was partially funded by the National Key R&D Program of China (No. 2018YFB & No. 2018YFB1005104).
A D ERIVATION OF G RADIENT E STIMATES FOR E NTROPY
Unconditional Entropy Given that the encoder E ψ ( . ) is deterministic, our goal is to optimize the entropy H( q ) =
− E q log q , where q is short for the distribution q ψ ( z ) of the representation z w.r [Equation]
t. its parameters ψ .
We can decompose the gradient of the entropy of q ψ ( z ) as:
[Equation]
The second term on the right side of the equation can be calculated:
[Equation]
Therefore [Equation] the gradient of the entropy of q ψ ( z ) becomes
[Equation]
Conditional Entropy Consider nondeterministic encoder function E ψ ( ., (cid:15) ) where (cid:15) is an auxiliary variable with independent marginal p ( (cid:15) ) .
The distribution q ψ ( z | x ) is determined by (cid:15) and the encoder parameters ψ .
The auxiliary variable (cid:15) introduces randomness to the encoder.
First, we decompose the gradients of Conditional Entropy as following:
[Equation]
Note that z = E ψ ( x , (cid:15) ) , such that we can apply reparameterization trick to the gradient estimator of conditional entropy in Eq.
[Equation]
B MIGE A LGORITHM D ESCRIPTION
The algorithm description of our proposed MIGE is stated in Algorithm
[Equation]
Algorithm 1 MIGE (Circumstance I)
Sampling :
Draw n samples from the data distribution p ( x ) , n denotes mini-batch size, then compute the corresponding output of the encoder ( x (1) , z (1) ) , · · · , ( x ( n ) , z ( n ) ) ∼ q ψ ( x
[Equation]
z ) z ( [Equation] [Equation] z ( n ) ∼ q ψ ( z )
Estimate the score function : ∇ z log q ψ ( z ( i ) ) ← SSGE( z (1) , · · · , z ( n ) ) ∇ ( x , z ) log q ψ ( x ( i ) , z ( i ) ) ← SSGE(( x (1) , z (1) ) , · · · , ( x ( n ) , z ( n ) )) 3. Estimate the entropy gradient : ∇ ψ H ( z ) ← − 1 n (cid:80) ni
=1 (cid:2) [Equation] 4. Estimate the MI gradient :
[Equation]
DIM(L) (Hjelm et al., 2019) is the state-of-the-art unsupervised model for representaion learning, which maximizes the average MI between the high-level representation and local patches of the image, and achieve an even higher classiﬁcation accuracy than supervised learning.
As shown in
Table 4, we apply MIGE into DIM(L) and surprisingly ﬁnd there is a signiﬁcant performance gap to
DIM(L).
To our knowledge, the principle of DIM(L) is still unclear.
Tschannen et al.
(2019) argues that maximizing tighter bounds in DIM(L) can lead to worse results, and the success of these methods cannot be attributed to the properties of MI alone, and they strongly depend on the inductive bias in both the choice of feature extractor architectures and the parameterization of the employed MI estimators.
For MIGE, we are investigating the behind reasons, e.g., to investigate the distributions of the patches. [Table]