D ECOMPOSING M UTUAL I NFORMATION FOR R EPRESENTATION L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Many self-supervised representation learning methods maximize mutual information (MI) across views.
In this paper, we transform each view into a set of subviews and then decompose the original MI bound into a sum of bounds involving conditional MI between the subviews.
E.g., given two views x and y of the same input example, we can split x into two subviews, x (cid:48) and x (cid:48)(cid:48) , which depend only on x but are otherwise unconstrained.
The following holds: I ( x ; y ) ≥ I ( x (cid:48)(cid:48) ; y )
+ I ( x (cid:48)
; y | x (cid:48)(cid:48) ) , due to the chain rule and information processing inequality.
By maximizing both terms in the decomposition, our approach explicitly rewards the encoder for any information about y which it extracts from x (cid:48)(cid:48) , and for information about y extracted from x (cid:48) in excess of the information from x (cid:48)(cid:48) .
We provide a novel contrastive lower bound on conditional MI, that relies on sampling contrast sets from p ( y | x (cid:48)(cid:48) ) .
By decomposing the original MI into a sum of increasingly challenging MI bounds between sets of increasingly informed views, our representations can capture more of the total information shared between the original views.
We empirically test the method in a vision domain and for dialogue generation.
I NTRODUCTION
The ability to extract actionable information from data in the absence of explicit supervision seems to be a core prerequisite for building systems that can, for instance, learn from few data points or quickly make analogies and transfer to other tasks.
Approaches to this problem include generative models (Hinton, 2012; Kingma & Welling, 2014) and self-supervised representation learning approaches, in which the objective is not to maximize likelihood, but to formulate a series of (label-agnostic) tasks that the model needs to solve through its representations (Noroozi & Favaro, 2016;
Devlin et al., 2019; Gidaris et al., 2018; Hjelm et al., 2019).
Self-supervised learning includes successful models leveraging contrastive learning, which have recently attained comparable performance to their fully-supervised counterparts (Bachman et al., 2019; Chen et al., 2020a).
Many self-supervised learning methods train an encoder such that the representations of a pair of views x and y derived from the same input example are more similar to each other than to representations of views sampled from a contrastive negative sample distribution, which is usually the marginal distribution of the data.
For images, different views can be built using random ﬂipping, color jittering and cropping (Bachman et al., 2019; Chen et al., 2020a).
For sequential data such as conversational text, the views can be past and future utterances in a given dialogue.
It can be shown that these methods maximize a lower bound on mutual information (MI) between the views, I ( x ; y ) , w.r.t.
the encoder, i.e. the InfoNCE bound (Oord et al., 2018).
One signiﬁcant shortcoming of this approach is the large number of contrastive samples required, which directly impacts the total amount of information which the bound can measure (McAllester & Stratos, 2018; Poole et al., 2019).
In this paper, we consider creating subviews of x by removing information from it in various ways, e.g. by masking some pixels.
Then, we use representations from less informed subviews as a source of hard contrastive samples for representations from more informed subviews.
For example, in Fig. 1, one can mask a pixel region in x (cid:48) to obtain x (cid:48)(cid:48) and ask (the representation of) x
(cid:48)(cid:48) to be closer to y than to random images of the corpus, and for x (cid:48) to be closer to y than to samples from p ( y | x (cid:48)(cid:48) ) .
This corresponds to decomposing the MI between x and y into I ( x ; y ) ≥ I ( x (cid:48)(cid:48) ; y )
+ I ( x (cid:48)
; y
| x (cid:48)(cid:48) ) .
The conditional MI measures the information about y that the model has gained by looking at x (cid:48) beyond the information already contained in x (cid:48)(cid:48) .
In Fig.
1 ( left ), standard contrastive approaches
[Figure]
could focus on the overall “shape” of the object and would need many negative samples to capture other discriminative features.
In our approach, the model is more directly encouraged to capture these additional features, e g. the embossed detailing.
In the context of predictive coding on sequential data such as dialogue, by setting x (cid:48)(cid:48) to be the most recent utterance (Fig. 1, right ), the encoder is directly encouraged to capture long-term dependencies that cannot be explained by x (cid:48)(cid:48) .
We formally show that, by such decomposition, our representations can potentially capture more of the total information shared between the original views x and y .
Maximizing MI between multiple views can be related to recent efforts in representation learning, amongst them AMDIM (Bachman et al., 2019), CMC (Tian et al., 2019) and SwAV (Caron et al., 2020).
However, these models maximize the sum of MIs between views I ( { x (cid:48) , x (cid:48)(cid:48) } ; y )
=
I ( x (cid:48)(cid:48) ; y )
+
I ( x (cid:48) ; y ) .
E.g., in Bachman et al.
, x (cid:48) and x (cid:48)(cid:48) could be global and local representations of an image, and in Caron et al.
(2020), x (cid:48) and x (cid:48)(cid:48) could be the views resulting from standard cropping and the aggressive multi-crop strategy.
This equality is only valid when the views x (cid:48) and x (cid:48)(cid:48) are statistically independent, which usually does not hold.
Instead, we argue that a better decomposition is I ( { x (cid:48) , x (cid:48)(cid:48) } ; y ) =
I ( x (cid:48)(cid:48) ; y )
+ I ( x (cid:48)
; y | x (cid:48)(cid:48) ) , which always holds.
Most importantly, the conditional
MI term encourages the encoder to capture more non-redundant information across views.
To maximize our proposed decomposition, we present a novel lower bound on conditional MI in
Section 3.
For the conditional MI maximization, we give a computationally tractable approximation that adds minimal overhead.
In Section 4, we ﬁrst show in a synthetic setting that decomposing MI and using the proposed conditional MI bound leads to capturing more of the ground-truth MI.
Finally, we present evidence of the effectiveness of the method in vision and in dialogue generation.
P ROBLEM S ETTING
The maximum MI predictive coding framework (McAllester, 2018; Oord et al., 2018; Hjelm et al., 2019) prescribes learning representations of input data such that they maximize MI.
Estimating MI is generally a hard problem that has received a lot of attention in the community (Kraskov et al., 2004;
Barber & Agakov, 2003).
Let x
and y be two random variables which can generally describe input data from various domains, e.g. text, images or sound.
We can learn representations of x and y by maximizing the MI of the respective features produced by encoders f, g : X → R d , which by the data processing inequality, is bounded by I ( x ; y ) :
[Equation]
We assume that the encoders can be shared, i.e. f = g .
The optimization in Eq.
1 is challenging but can be lower-bounded.
Our starting point is the recently proposed InfoNCE lower bound on MI (Oord et al., 2018) and its application to self-supervised learning for visual representations (Bachman et al., 2019; Chen et al., 2020a).
In this setting, x and y are paired input images, or independentlyaugmented copies of the same image.
These are encoded using a neural network encoder which is trained such that the representations of the two image copies are closer to each other in the embedding space than to other images drawn from the marginal distribution of the corpus.
This can be viewed as a contrastive estimation of the MI (Oord et al., 2018).
We present the InfoNCE bound next.
2. I NFO NCE B OUND
InfoNCE (Oord et al., 2018) is a lower-bound on I ( x ; y ) obtained by comparing pairs sampled from the joint distribution x, y 1 ∼ p ( x, y ) to a set of negative samples, y 2: K ∼ p ( y 2: K ) =
(cid:81)
Kk =2 p ( y k ) , also called contrastive , independently sampled from the marginal:
[Equation]
where E is a critic assigning a real valued score to x, y pairs.
We provide an exact derivation for this bound in the Appendix 1 .
For this bound, the optimal critic is the log-odds between the conditional distribution p ( y | x ) and the marginal distribution of y , E ∗ ( x, y )
= log p ( y | x ) p ( y )
+ c ( x ) (Oord et al., 2018; Poole et al., 2019).
The InfoNCE bound is loose if the true mutual information I ( x ; y ) is larger than log K .
In order to overcome this difﬁculty, recent methods either train with large batch sizes (Chen et al., 2020a) or exploit an external memory of negative samples in order to reduce memory requirements (Chen et al., 2020b; Tian et al., 2020).
These methods rely on uniform sampling from the training set in order to form the contrastive sets.
For further discussion of the limits of variational bounds of MI, see McAllester & Stratos (2018).
D ECOMPOSING M UTUAL I NFORMATION
By the data processing inequality: I ( x ; y ) ≥ I ( { x 1 , . . .
, x N } ; y ) , where { x 1 , . . .
, x N } are different subviews of x – i.e., views derived from x without adding any exogenous information.
For example, { x 1 , . . .
, x N } can represent exchanges in a longer dialog x , sentences in a document x , or different augmentations of the same image x .
Equality is obtained when the set of subviews retains all information about x , e.g. if x is in the set.
Without loss of generality, we consider the case N = 2 , I ( x ; y ) ≥ I ( { x (cid:48) , x (cid:48)(cid:48) } ; y ) , where { x (cid:48) , x (cid:48)(cid:48) } indicates two subviews derived from the original x .
We can apply the chain rule for MI:
[Equation]
where the equality is obtained if and only if I ( x ; y |{ x (cid:48) , x (cid:48)(cid:48) } )
= 0 , i.e. x doesn’t give any information about y in excess to { x (cid:48) , x (cid:48)(cid:48) } 2 .
This suggests that we can maximize I ( x ; y ) by maximizing each of the MI terms in the sum.
The conditional MI term can be written as: [Equation]
,x (cid:48)(cid:48) ,y ) (cid:20) log p (
[Equation]
|
[Equation]
(cid:48)
[Equation]
x (cid:48)(cid:48) ) p ( y | x (cid:48)(cid:48) ) (cid:21) . This conditional MI is different from the unconditional MI, I ( x (cid:48) ; y ) , insofar it measures the amount of information shared between x (cid:48) and y which cannot be explained by x (cid:48)(cid:48) .
Note that the decomposition holds for arbitrary partitions of x (cid:48) , x (cid:48)(cid:48) , e.g. I ( { x (cid:48) , x (cid:48)(cid:48) } ; y )
=
I ( x (cid:48) ; y )
+ I ( x (cid:48)(cid:48)
; y
| x (cid:48) ) .
When X is high-dimensional, the amount of mutual information between x and y will potentially be larger than the amount of MI that I N CE can measure given computational constraints associated with large K and the poor log scaling properties of the bound.
The idea that we put forward is to split the total MI into a sum of MI terms of smaller magnitude, thus for which I NCE would have less bias for any given K , and estimate each of those terms in turn.
The resulting decomposed bound can be written into a sum of unconditional and conditional MI terms:
[Equation] =
I ( x ; y |{ x (cid:48) , x (cid:48)(cid:48) } ) + I ( { x (cid:48) , x (cid:48)(cid:48) } ; y ) , given that I ( { x, x (cid:48) , x (cid:48)(cid:48) } ; y )
=
I ( x ; y ) , equality is obtained iff I ( x ; y |{ x (cid:48) , x (cid:48)(cid:48) } )
= 0 .
where I CNCE is a lower-bound on conditional MI and will be presented in the next section. Both conditional (Eq. 6) and unconditional bounds on the MI (Eq. 14) can capture at most log K nats of
MI.
Therefore, the bound that arises from the decomposition of the MI in Eq.
5 potentially allows to capture up to N log K nats of MI in total, where N is the number of subviews used to describe x .
This shows that measuring mutual information by decomposing it in a sequence of estimation problems potentially allows to capture more nats of MI than with the standard I NCE , which is [-usually a key assumption in self-supervised learning approaches and allows to compute the InfoNCE bound in Eq. 
 (Oord et al., 2018).-]bounded by log K .  C ONTRASTIVE B OUNDS ON C ONDITIONAL M UTUAL I NFORMATION
One of the difﬁculties in computing the decomposed bound is measuring the conditional mutual information.
In this section, we provide bounds and approximations of this quantity.
First, we show that we can readily extend InfoNCE.
Proposition 1 ( Conditional InfoNCE ) .
The following is a lower-bound on the conditional mutual information I ( x (cid:48) ; y | x (cid:48)(cid:48) ) and veriﬁes the properties below: [Equation]
I CNCE ≤
I ( x (cid:48) ; y | x (cid:48)(cid:48) ) .
E ∗ =
arg sup E
I CNCE = log
[Equation]
y
[Equation]
x
[Equation]
x
[Equation]
When K → ∞ and E = E ∗ , we recover the true conditional MI:
[Equation]
The proof can be found in Sec.
A.2 and follows closely the derivation of the InfoNCE bound by applying a result from Barber & Agakov (2003) and setting the proposal distribution of the variational approximation to p ( y | x (cid:48)(cid:48) ) . An alternative derivation of this bound was also presented in parallel in Foster et al.
(2020) for optimal experiment design. Eq. 6 shows that a lower bound on the conditional MI can be obtained by sampling contrastive sets from the proposal distribution p ( y | x (cid:48)(cid:48) ) .
Indeed, since we want to estimate the MI conditioned on x (cid:48)(cid:48) , we should allow our contrastive distribution to condition on x (cid:48)(cid:48) .
Note that E is now a function of three variables.
Computing Eq.
6 requires access to a large number of samples from p ( y | x (cid:48)(cid:48) ) , which is unknown and usually challenging to obtain.
In order to overcome this, we propose two solutions. 4. V ARIATIONAL A PPROXIMATION The next proposition shows that it is possible to obtain a bound on the conditional MI by approximating the unknown conditional distribution p ( y | x (cid:48)(cid:48) ) with a variational distribution
τ ( y | x (cid:48)(cid:48) ) .
Proposition 2 ( Variational I CNCE ) . For any variational approximation τ ( y | x (cid:48)(cid:48) ) in lieu of p ( y | x (cid:48)(cid:48) ) , [Equation]
(cid:48)(cid:48) , E, τ, K ) = E p ( x (cid:48) ,x (cid:48)(cid:48) ,y 1 ) τ ( y 2: K | x (cid:48)(cid:48) ) (cid:20) log e E ( x (cid:48)(cid:48) ,x (cid:48) ,y 1 ) 1 K (cid:80)
[Equation]
[Equation]
E ( x (cid:48)(cid:48)
[Equation]
x (cid:48)
[Equation]
y k ) (cid:21) (7) − E
[Equation]
x (cid:48)(cid:48)
[Equation]
(cid:20)
[Equation]
(cid:48)(cid:48)
[Equation]
(cid:107)
[Equation]
( y | x (cid:48)(cid:48) )) (cid:21) , with p ( ·| x (cid:48)(cid:48) )
<< τ ( ·| x (cid:48)(cid:48) ) for any x (cid:48)(cid:48) , we have the following properties:
I V AR ≤ I ( x (cid:48) ;
y | x (cid:48)(cid:48) ) .
If τ ( y | x (cid:48)(cid:48) )
=
p ( y | x (cid:48)(cid:48) ) , I V AR =
I CNCE .
lim K →∞ sup E
I V AR ( x (cid:48) ;
y | x
(cid:48)(cid:48) , E, τ, K ) =
I ( x (cid:48) ; y | x (cid:48)(cid:48) ) .
See Sec.
A.3 for the proof.
This bound side-steps the problem of requiring access to an arbitrary number of contrastive samples from the unknown p ( y | x (cid:48)(cid:48) ) by i.i.d. sampling from the known and tractable τ ( y | x (cid:48)(cid:48) ) . We prove that as the number of examples goes to ∞ , optimizing the bound w.r.t. E converges to the true conditional MI.
Interestingly, this holds true for any value of τ , though the choice of τ will most likely impact the convergence rate of the estimator. Eq. 3 is superﬁcially similar to the ELBO (evidence lower bound) objective used to train
VAEs (Kingma & Welling, 2014), where τ plays the role of the approximate posterior (although the KL direction in the ELBO is inverted). This parallel suggests that τ ∗ ( y | x (cid:48)(cid:48) ) =
p ( y | x (cid:48)(cid:48) ) may not be the optimal solution for some values of K and E . However, we see trivially that if we ignore the dependency of the ﬁrst expectation term on τ and only optimize τ to minimize the KL term, then it is guaranteed that τ ∗ ( y | x ) = p ( y | x (cid:48)(cid:48) ) , for any K and E . Thus, by the second property in Proposition 2, optimizing I V AR ( E, τ ∗ , K ) w.r.t E will correspond to optimizing I CNCE .
In practice, the latter observation signiﬁcantly simpliﬁes the estimation problem as one can minimize a Monte-Carlo approximation of the KL divergence w.r.t τ by standard supervised learning: we can efﬁciently approximate the KL by taking samples from p ( y | x (cid:48)(cid:48) ) . Those can be directly obtained by using the joint samples from p ( x, y ) included in the training set and computing x (cid:48)(cid:48) from x . 4. I MPORTANCE S AMPLING A PPROXIMATION Maximizing I V AR can still be challenging as it requires estimating a distribution over potentially high-dimensional inputs.
In this section, we provide an importance sampling approximation of
I CNCE that bypasses this issue.
We start by observing that the optimal critic for I NCE ( x (cid:48)(cid:48) ; y | E, K ) is ¯ E ( x (cid:48)(cid:48) , y )
= log p [Equation] y | x (cid:48)(cid:48) ) p
[Equation] y ) + c ( x (cid:48)(cid:48) ) , for any c .
Assuming we have appropriately estimated ¯ E ( x (cid:48)(cid:48) , y ) , it is possible to use importance sampling to produce approximate samples from p ( y | x (cid:48)(cid:48) ) . This is achieved by ﬁrst sampling y (cid:48) 1: M ∼ p ( y ) and resampling K ≤ M ( K > 0 ) examples i.i.d. from the normalized importance distribution q
[Equation]
=
[Equation]
[Equation] ) , where w k = [Equation] This process is also called “sampling importance resampling” (SIR).
As M/K → ∞ , it is guaranteed to produce samples from p ( y | x (cid:48)(cid:48) )
(Rubin, 1987).
The SIR estimator is written as: [Equation] where we note the dependence of q SIR on w k and hence ¯ E .
SIR is known to increase the variance of the estimator (Skare et al., 2003) and is wasteful given that only a smaller set of K examples are actually used for MI estimation. Hereafter, we provide a cheap approximation of the SIR estimator. The key idea is to rewrite the contribution of the negative samples in the denominator of Eq.
8 as an average ( K − 1) (cid:80) Kk = 1 K − 1 e E ( x (cid:48)(cid:48) ,x (cid:48) ,y k ) and use the normalized importance weights w k to estimate that term under the resampling distribution. We hypothesize that this variant has less variance as it does not require the additional resampling step.
The following proposition shows that as the number of negative examples goes to inﬁnity, the proposed approximation converges to the true value of the conditional MI.
Proposition 3 ( Importance Sampling I CNCE ) .
The following approximation of I SIR :
[Equation]
(9) where w k = [Equation] and ¯ E = arg sup E I NCE ( x
[Equation]
, y | E, K ) , veriﬁes:
lim K →∞ sup E
I IS ( x (cid:48) ; y | x
(cid:48)(cid:48) , E, K ) =
I ( x (cid:48) ; y | x (cid:48)(cid:48) ) , 2.
lim K →∞ arg sup E
I IS =
log p ( y | x (cid:48)(cid:48) ,x (cid:48) ) p ( y | x (cid:48)(cid:48) ) + c ( x (cid:48) , x (cid:48)(cid:48) ) . [Figure]
The proof can be found in Sec. A.4. This objective up-weights the negative contribution to the normalization term of examples that have high probability under the resampling distribution. This approximation is cheap to compute given that the negative samples still initially come from the marginal distribution p ( y ) and avoids the need for resampling.
The proposition shows that in the limit of K → ∞ , optimizing I IS w.r.t. E converges to the conditional MI and the optimal E converges to the optimal I CNCE solution. We also note that we suppose E The I IS approximation provides a general, grounded way of sampling “harder” negatives by ﬁltering samples from the easily-sampled marginal p ( y ) .
E XPERIMENTS
We start by investigating whether maximizing the decomposed MI using our conditional MI bound leads to a better estimate of the ground-truth MI in a synthetic experiment.
Then, we experiment on a self-supervised image representation learning domain.
Finally, we explore an application to natural language generation in a sequential setting, such as conversational dialogue.
5. S YNTHETIC D ATA
We extend Poole et al.
(2019)’s two variable setup to three variables.
We posit that { x (cid:48) , x (cid:48)(cid:48) , y } are three Gaussian co-variates, x (cid:48) , x (cid:48)(cid:48) , y ∼ N (0 , Σ) and we choose Σ such that we can control the total mutual information
I ( { x (cid:48) , x (cid:48)(cid:48) } ; y ) such that I = { 5 , 10 , 15 , 20 } (see Appendix for pseudo-code and details of the setup).
We aim to estimate the total MI I ( { x (cid:48) , x (cid:48)(cid:48) } ; y ) and compare the performance of our approximators in doing so.
For more details of this particular experimental setting, see App.
B.
In Figure 2, we compare the estimate of the MI obtained by:
InfoNCE , which computes I NCE ( { x (cid:48) , x (cid:48)(cid:48) } , y | E, K )
and will serve as
our baseline; 2.
InfoNCEs , which probes the effectiveness of decomposing the total MI into a sum of smaller terms and computes I NCE ( x (cid:48)(cid:48) , y | E, K/ 2)+ I CNCE
( x (cid:48) , y | x (cid:48)(cid:48) , E, K/ 2) , where K/samples are obtained from p ( y ) and K/ 2 are sampled from p ( y | x (cid:48)(cid:48) ) ; 3.
InfoNCEs IS , the decomposed bound using our importance sampling approximation to the
conditional MI I IS , i.e. I NCE ( x (cid:48)(cid:48) ,
y | E, K )
+ I IS ( x (cid:48) , y | x (cid:48)(cid:48) , E, K ) .
This does not require access to samples from p ( y | x (cid:48)(cid:48) ) and aims to test the validity of our approximation in an empirical setting.
Both terms reuse the same number of samples K .
For 2., we use only half as many samples as InfoNCE to estimate each term in the MI decomposition ( K /2), so that the total number of negative samples is comparable to InfoNCE.
Note that we use K samples in “InfoNCE IS”, because those are reused for the conditional MI computation. All critics E are parametrized by MLPs as explained in Sec.
B. Our results in Figure 2 show that, for larger amounts of true MI, decomposing MI as we proposed can capture more nats than InfoNCE with an order magnitude less examples. We also note that the importance sampling estimator seems to [Table] Additional Losses / More Epochs
[Table] estimate MI reliably. Its empirical behavior for MI = { 5 , 10 } could indicate that InfoNCEs IS is a valid lower bound on MI, although we couldn’t prove it formally. 5. V ISION
Imagenet We study self-supervised learning of image representations using 224x224 images from
ImageNet.
The evaluation is performed by ﬁtting a linear classiﬁer to the task labels using the pre-trained representations only, that is, we ﬁx the weights of the pre-trained image encoder f .
Each input image is independently augmented into two views x and y using a stochastically applied transformation.
For the base model hyper-parameters and augmentations, we follow the “InfoMin
Aug.” setup (Tian et al., 2020).
This uses random resized crop, color jittering, gaussian blur, rand augment, color dropping, and jigsaw as augmentations and uses a momentum-contrastive memory buffer of K = 65536 examples (Chen et al., 2020b).
We fork x into two sub-views { x (cid:48) , x (cid:48)(cid:48) } : we set x (cid:48)
(cid:44) x and x (cid:48)(cid:48) to be an information-restricted view of x .
We found beneﬁcial to maximize both decompositions of the MI: I ( x (cid:48) ; y )
+ I ( x (cid:48)(cid:48)
; y
| x (cid:48) ) =
I ( x (cid:48)(cid:48) ; y )
+ I ( x (cid:48)
; y
| x (cid:48)(cid:48) ) .
By noting that I ( x (cid:48)(cid:48) ; y | x (cid:48) ) is likely zero given that the information of x (cid:48)(cid:48) is contained in x (cid:48) , our encoder f is trained to maximize:
[Equation]
Note that if x (cid:48)(cid:48)
=
x , then our decomposition boils down to maximizing the standard InfoNCE bound.
Therefore, InfoMin Aug. is recovered by ﬁxing λ = 1 or by setting x (cid:48)(cid:48)
=
x .
The computation of the conditional MI term does not add computational cost as it can be computed by caching the logits used in the two unconditional MI terms (see Sec. B).
We experiment with two ways of obtaining restricted information views x (cid:48)(cid:48) : cut , which applies cutout to x , and crop which is inspired by Caron et al.
and consists in cropping the image aggressively and resizing the resulting crops to 96x96.
To do so, we use the RandomResizedCrop from the torchvision .
transforms module with parameters: s = ( 0 . 05 , 0 . 14 ) .
Results are reported in
Table 1.
Augmenting the InfoMin Aug. base model with our conditional contrastive loss leads to 0.8% gains on top-1 accuracy and 0.6% on top-5 accuracy.
We notice that the crop strategy seems to perform slightly better than the cut strategy.
One reason could be that cutout introduces image patches that do not follow the pixel statistics in the corpus.
More generally, we think there could be information restricted views that are better suited than others.
In order to isolate the impact on performance due to integrating an additional view x (cid:48)(cid:48) , i.e. the I NCE ( x (cid:48)(cid:48) ; y | f, K ) term in the optimization, we set the conditional mutual information term to zero in the line “without cond.
MI”.
We see that this does not improve over the baseline InfoMin Aug., and its performance is 1% lower than our method, pointing to the fact that maximizing conditional MI across views provides the observed gains.
We also include the very recent results of SwAV (Caron et al., 2020) and ByOL (Grill et al., 2020) which use a larger number of views (SwAV) and different loss functions (SwAV, ByOL) and thus we think are orthogonal to our approach.
We think our approach is general and could be integrated in those solutions as well.
CIFAR-10 We also experiment on CIFAR-10 building upon SimCLR (Chen et al., 2020b), which uses a standard ResNet-50 architecture by replacing the ﬁrst 7x7 Conv of stride 2 with 3x3 Conv of stride 1 and also remove the max pooling operation.
In order to generate the views, we use Inception crop (ﬂip and resize to 32x32) and color distortion.
We train with learning rate 0.5, batch-size 800, momentum coefﬁcient of 0.9 and cosine annealing schedule.
Our energy function is the cosine similarity between representations scaled by a temperature of 0.5 (Chen et al., 2020b).
We obtain a top-1 accuracy of 94.7% using a linear classiﬁer compared to 94.0% as reported in Chen et al.
(2020b) and 95.1% for a supervised baseline with same architecture.
5. D IALOGUE
For dialogue language modeling, we adopt the predictive coding framework (Elias, 1955; McAllester & Stratos, 2018) and consider past and future in a dialogue as views of the same conversation.
Given L utterances x = ( x 1 , . . .
, x L ) , we maximize I NCS ( x ≤ k ; x >k | f, K ) , where past x ≤ k = ( x 1 , . . .
, x k ) and future x >k = ( x k +1 , . . .
, x L ) are obtained by choosing a split point 1 < k < L .
We obtain f ( x ≤ k ) , f ( x >k ) by computing a forward pass of the ﬁne-tuned “small” GPT2 model (Radford et al., 2019) on past and future tokens, respectively, and obtaining the state corresponding to the last token in the last layer.
We evaluate our introduced models against different baselines.
GPT2 is a basic small pre-trained model ﬁne-tuned on the dialogue corpus.
TransferTransfo
(Wolf et al., 2019) augments the standard next-word prediction loss in GPT2 with the next-sentence prediction loss similar to Devlin et al.
Our baseline GPT2+InfoNCE maximizes I NCE ( x ≤ k ; x >k | f, K ) in addition to standard nextword prediction loss.
In GPT2+InfoNCE S , we further set x (cid:48)
= x ≤ k and x (cid:48)(cid:48)
= x k , the recent past, and maximize I NCES ( x ≤ k , x >k ) .
To maximize the conditional MI bound, we sample contrastive futures from p ( x >k | x k ; θ GPT2 ) , using GPT2 itself as the variational approximation 4 .
We ﬁne-tune all models on the Wizard of Wikipedia (WoW) dataset (Dinan et al., 2018) with early stopping on validation perplexity.
We evaluate our models using automated metrics and human evaluation: we report perplexity ( ppl ), BLEU (Papineni et al., 2002), and word-repetition-based metrics from Welleck et al.
(2019), speciﬁcally: seq-rep- n measures the portion of duplicate n-grams and seq-rep-avg averages over n ∈ { 2 , 3 , 4 , 5 , 6 } .
We measure diversity via dist- n (Li et al., 2016), the number of unique n -grams, normalized by the total number of n -grams.
Table 4 shows results on the validation set.
For the test set results, please refer to the Appendix.
Incorporating InfoNCE yields improvements in all metrics 5 .
Please refer to the Appendix for sample dialogue exchanges.
We also perform human evaluation on randomly sampled 1000 WoW dialogue contexts.
We present the annotators with a pair of candidate responses consisting of GPT2+InfoNCE S responses and baseline responses.
They were asked to compare the pairs regarding interestingness, relevance and humanness, using a 3-point Likert scale (Zhang et al., 2019).
Table 4 lists the difference between fraction of wins for GPT2+InfoNCE S and other models as H-rel , H-hum , and H-int .
Overall, GPT2+InfoNCE S was strongly preferred over GPT2, TransferTransfo and GPT2+InfoNCE, but not the gold response.
Bootstrap conﬁdence intervals and p-values (t-test) indicate all improvements except for GPT2+InfoNCE on the relevance criterion are signiﬁcant at α =0 .
D ISCUSSION
The result in Eq. 5 is reminiscent of conditional noise-contrastive estimation (CNCE)
(Ceylan &
Gutmann, 2018) which proposes a framework for data-conditional noise distributions for noise contrastive estimation (Gutmann & Hyvärinen, 2012).
Here, we provide an alternative interpretation in terms of a bound on conditional mutual information.
In CNCE, the proposal distribution is obtained by noising the conditional proposal distribution.
It would be interesting to investigate whether it [Table] is possible to form information-restricted views by similar noise injection
[Table]
A D ERIVATIONS A. D ERIVATION OF I NFO NCE, I NCE
We start from Barber and Agakov’s variational lower bound on MI (Barber & Agakov, 2003).
I ( x ; y ) can be bounded as follows:
[Equation]
where q is an arbitrary distribution.
We show that the InfoNCE bound (Oord et al., 2018) corresponds to a particular choice for the variational distribution q followed by the application of the Jensen inequality.
Speciﬁcally, q ( y | x ) is deﬁned by independently sampling a set of examples { y 1 , . . .
, y K } from a proposal distribution π ( y ) and then choosing y from { y 1 , . . .
, y K } in proportion to the importance weights w y = e E ( x,y ) (cid:80) k e E ( x,yk ) , where E is a function that takes x and y and outputs a scalar. In the context of representation learning, E is usually a dot product between some representations of x and y , e.g. f ( x ) T f ( y )
(Oord et al., 2018).
The unnormalized density of y given a speciﬁc set of samples y 2: K = { y 2 , . . .
, y K } and x is: [Equation] where we introduce a factor K which provides “normalization in expectation”. By normalization in expectation, we mean that taking the expectation of q ( y | x, y 2: K ) with respect to resampling of the alternatives y 2: K from π ( y ) produces a normalized density (see Sec.
A.1.1 for a derivation):
[Equation]
where π ( y 2: K ) = (cid:81) Kk =2 π ( y k ) . The InfoNCE bound (Oord et al., 2018) is then obtained by setting the proposal distribution as the marginal distribution, π ( y ) ≡ p ( y ) and applying Jensen’s inequality, giving:
[Equation]
where the second inequality has been obtained using Jensen’s inequality
[Equation] A.1. D ERIVATION OF NORMALIZED DISTRIBUTION
We follow Cremer et al.
(2017) to show that q ( y | x ) =
E y 2:
K ∼ π ( y )
[ q ( y | x, y 2: K )] is a normalized distribution: [Equation] y )   e E ( x,y ) 1 K (cid:16)(cid:80) Kk =2 e E ( x
[Equation]
y k )
+ e E ( x
[Equation]
y ) (cid:17) 
 dy =
E π ( y ) E π ( y 2: K )   e E ( x,y ) 1 K (cid:16)(cid:80)
Kk =2 e E ( x,y k )
+ e E ( x,y ) (cid:17) 
 =
E π ( y 1: K ) (cid:32)
e E ( x
[Equation]
y ) 1 K (cid:80)
[Equation]
[Equation]
( x
[Equation]
y k ) (cid:33) =
[Equation]
[Equation]
[Equation]
) (cid:33) =
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
) =
[Equation]
) (cid:80)
[Equation]
) (cid:80)
[Equation]
A. P ROOFS FOR I CNCE
Proposition 1 ( Conditional InfoNCE ) .
The following is a lower-bound on the conditional mutual information
I ( x (cid:48) ; y | x (cid:48)(cid:48) ) and veriﬁes the properties below:
[Equation]
[Equation]
When K → ∞ and E = E ∗ , we recover the true conditional MI:
[Equation]
Proof.
We begin with 1., the derivation is as follows:
[Equation]
21) where we used in Eq.
16 the Jensen’s inequality following Barber and Agakov’s bound (Barber & Agakov, 2003) and used p ( y | x (cid:48)(cid:48) ) as our proposal distribution for the variational approximation ¯ q ( y | x (cid:48)(cid:48) , x (cid:48) ) .
For 2., we rewrite I CNCE by grouping the expectation w.r.t x (cid:48)(cid:48) :
[Equation]
[Equation]
Given that both distributions in the inner-most expectation condition on the same x (cid:48)(cid:48) , this term has the same form as I NCE and therefore the optimal solution is E ∗ x
(cid:48)(cid:48) =
[Equation]
( x
[Equation]
) (Ma & Collins, 2018).
The optimal E for I CNCE is thus obtained by choosing E ( x (cid:48)(cid:48) , x (cid:48) , y ) =
E ∗ x (cid:48)(cid:48) for each x (cid:48)(cid:48) , giving E ∗ = log p ( y | x (cid:48) ,x (cid:48)(cid:48) )
p ( y | x (cid:48)(cid:48)
[Equation]
For proving 3., we substitute the optimal critic and take the limit K → ∞ .
We have:
[Equation]
From the Strong Law of Large Numbers
[Equation]
we know that as
[Equation]
= 1 , as K → ∞ a
[Equation]
s., therefore (relabeling y = y 1 ):
[Equation] [Equation] A. P ROOFS FOR I V AR
Proposition 2 ( Variational I CNCE ) .
For any variational approximation τ ( y | x (cid:48)(cid:48) ) in lieu of p ( y | x (cid:48)(cid:48) ) ,
I V AR ( x (cid:48) , y | x (cid:48)(cid:48) , E, τ, K ) =
E p ( x (cid:48) ,x (cid:48)(cid:48) ,y 1 ) τ ( y 2: K | x (cid:48)(cid:48) )
(cid:20) log
e E ( x (cid:48)(cid:48) ,x (cid:48) ,y 1 ) 1 K (cid:80)
Kk =1 e E ( x (cid:48)(cid:48) ,x (cid:48) ,y k ) (cid:21) (7) − E p ( x (cid:48)(cid:48) )
(cid:20) KL (cid:0) p ( y | x (cid:48)(cid:48) )
(cid:107) τ
( y | x (cid:48)(cid:48) ) (cid:1) (cid:21) , with p ( ·| x (cid:48)(cid:48) ) <
[Equation]
τ ( ·| x (cid:48)(cid:48) ) for any x (cid:48)(cid:48) , we have the following properties:
I V AR ≤ I ( x (cid:48) ;
y | x (cid:48)(cid:48) ) .
If τ ( y | x (cid:48)(cid:48) )
=
p ( y | x (cid:48)(cid:48) ) , I V AR =
I CNCE .
lim K →∞ sup E
I V AR
[Equation]
x
[Equation]
; y | x (cid:48)(cid:48) , E, τ, K )
=
I ( x (cid:48) ; y | x (cid:48)(cid:48) ) .
Proof.
For 1., we proceed as follows:
[Equation]
where the last step has been obtained as in Eq.
Proving 2. is straightforward by noting that if τ
[Equation]
KL
[Equation]
{+) = 
+}
and the ﬁrst term corresponds to I CNCE .
Proving 3. goes as follows:
[Equation]
This is obtained by noting that (1) for any K and τ , arg sup E I V AR =
p
( y | x (cid:48)(cid:48) ,x (cid:48) )
τ ( y | x (cid:48) )
(because the KL doesn’t depend on E ) and (2) the second term in the last line goes to 0 for K → ∞ (a straightforward applica- tion of the Strong Law of Large Numbers shows that for samples y 2: K drawn from τ ( y 2: K | x (cid:48)(cid:48) ) , we have:
[Equation]
A. P ROOFS FOR I IS
We will be using the following lemma.
Lemma 1.
For any x (cid:48)(cid:48) , x (cid:48) and y , and any sequence E K such that || E K − E || ∞
→ K →∞ 0 :
[Equation]
[Equation]
where w k =
[Equation]
Proof.
We see that almost surely, for y 2: K ∼ p ( · ) :
[Equation]
where we applied the Strong Law of Large Numbers to the denominator.
For the numerator, we write:
[Equation]
and note that the ﬁrst term is the standard IS estimator using p ( y k ) as proposal distribution and tends to
E p ( y | x
[Equation]
) e E ( x (cid:48)(cid:48) ,x (cid:48) ,y ) from the Strong Law of Large Numbers, while the second term goes to 0 as E K tends to E uniformly.
This gives
[Equation]
Following the same logic, without the importance-sampling demonstrates that:
[Equation]
which concludes the proof.
Proposition 3 ( Importance Sampling I CNCE ) .
The following approximation of I SIR :
I IS
[Equation]
where w k =
[Equation]
lim K →∞ sup E
I IS ( x
[Equation]
; y | x (cid:48)(cid:48) , E, K )
=
I
[Equation]
x
[Equation]
; y | x (cid:48)(cid:48) ) , 2.
lim K →∞ arg sup E
I IS =
log p
[Equation]
y | x
[Equation]
,x
[Equation]
) p ( y | x (cid:48)(cid:48) )
[Equation]
c ( x (cid:48) , x (cid:48)(cid:48) ) .
Proof.
By applying Lemma 1 with E K = E
[Equation]
we know that for any E :
[Equation]
In particular, the RHS of the equality corresponds to lim K →∞
I CNCE ( x (cid:48) , y | x (cid:48)(cid:48) , E, K ) .
That quantity is smaller than I ( x (cid:48) , y | x (cid:48)(cid:48) ) , with equality for E = E ∗ .
This guarantees that:
[Equation]
We now prove the reverse inequality.
We let 2 (cid:15) =
lim K →∞ sup E
I IS ( x (cid:48) ; y | x
(cid:48)(cid:48) , E, ¯ E, K ) − I ( x (cid:48) , y | x (cid:48)(cid:48) ) , and assume toward a contradiction that (cid:15) > 0 .
We know that:
[Equation]
Now, ∀ K ≥ K 0 , let E K be such that:
[Equation]
and thus: ∀ K ≥ K 0 , I IS ( x (cid:48) ;
y | x
(cid:48)(cid:48) , E K , ¯ E, K ) ≥
I ( x (cid:48) , y
| x (cid:48)(cid:48) )
+ (cid:15) 2 .
Since E K ∈ R |X|×|X|×|Y| , { E K } K ≥ K 0 contains a subsequence that converges to a certain E ∞ ∈ ¯
R |X|×|X|×|Y| .
Without loss of generality, we assume that ∀ K, ∀ x (cid:48)(cid:48) , ∀ x (cid:48) , E p ( y )
[ E K ( x (cid:48)(cid:48) , x (cid:48) , y )]
= 0 which implies that E p ( y )
[ E ∞ ( x (cid:48)(cid:48) , x (cid:48) , y )]
= 0 (similarly to I NCE , I IS is invariant to constants added to E ).
In particular, this guarantees that || E ∞
|| ∞ < ∞ .
Otherwise, we would have E ∞ ( x (cid:48)(cid:48) , x (cid:48) , y ) =
−∞ for a given y , which would then imply I
IS ( x (cid:48) ; y | x (cid:48)(cid:48) , E ∞ , ¯ E, K ) =
−∞ and give a contradiction.
We can now apply Lemma 1 to { E K } and E ∞ to show that lim K →∞ I IS ( x (cid:48) ;
y | x
(cid:48)(cid:48) , E K , ¯ E, K ) =
lim
K →∞ I CNCE ( x (cid:48) , y | x (cid:48)(cid:48) , E ∞ , K ) , and get a contradiction: the ﬁrst term is larger than I ( x (cid:48) , y | x (cid:48)(cid:48) )
+ (cid:15)while the second is smaller than I ( x (cid:48) , y | x (cid:48)(cid:48) ) .
B P SEUDOCODE B. L OSS COMPUTATION
We provide a pseudo-code for the loss computation which uses MocoV2 backbone comprising a memory of contrastive examples obtained using a momentum-averaged encoder (Chen et al., 2020b). def compute_loss(xp, xpp, y, f, f_ema, memory, lam=0.5): """
Args:xpp: info-restricted view xp: a view y: a view f: standard encoder f_ema: momentum averaged encoder memory: memory bank of representations
Returns:lam* mi(xp; y)
+ (1 - lam) * (mi(xpp; y)
+ mi(xp; y | xpp)) """#encode xp and xpp with standard encoder, (1, dim) q_xp, q_xpp =
f( [Table] [Table] [Table]
[Table]
# encode y with momentum-averaged encoder, ( [Table] , dim) [Table]
[Table]
[Table]
[Table]
and
[Table]
.sum
[Table]
[Equation]
[Table]
[Table]
on the
[Table]
mutual information
[Equation]
# compose final loss
[Equation]
loss
[Equation]
return loss B. S YNTHETIC E XPERIMENTS
Here, we provide details for Sec. 5.1.
In this experiment, each x (cid:48) , x (cid:48)(cid:48) and y are 20-dimensional.
For each dimension, we sampled ( x (cid:48)
i , x (cid:48)(cid:48)
i , y i ) from a correlated Gaussian with mean 0 and covariance matrix cov i .
For a given value of MI, mi = { 5 , 10 , 15 , 20 } , we sample covariance matrices cov i = sample_cov(mi i ) , such that (cid:80) i mi i =
mi , mi i chosen at random.
We optimize the bounds by stochastic gradient descent (Adam, learning rate 5 · 10 − 4 ).
All encoders f are multi-layer perceptrons with a single hidden layer and ReLU activation.
Both hidden and output layer have size 100. InfoNCE computes:
[Equation] where the proposal is the marginal distribution p ( y ) , E is chosen to be a dot product between representations, E p denotes expectation w.r.t. the known joint distribution p ( InfoNCEs computes:
[Equation]
[Equation]
log e f
[Equation]
x (cid:48) ]) T f ( y ) e
[Equation]
[Equation]
k
[Equation]
(cid:35) [Equation] K where f ( x ) is just f ([ x, 0 ]) in order to re-use MLP parameters for the two terms. The negative samples of the conditional MI term come from the conditional distribution p ( y | x (cid:48)(cid:48) ) , which is assumed to be known in this controlled setting.
We maximize both lower bounds with respect to the encoder f .
We report pseudo-code for sample _ cov , used to generate 3 × 3 covariance matrices for a ﬁxed mi
= I ( { x (cid:48) , x (cid:48)(cid:48) } ; y ) and uniformly sampled α
= I ( x (cid:48)(cid:48) ; y ) /I ( { x
[Equation] , x [Equation] } ; y ) : def sample_cov(mi): alpha = random.uniform(0.1, 0.9) params =
random.normal(0, I 6 )
# use black box optimizer (Nealder-Mead) to determine opt_params opt_param =
arg min x residual ( params , mi , α ) return project_posdef(opt_params) def project_posdef(x): # project x ∈ R 6 to a positive definite 3x3 matrix cov =
zeros(3, 3) cov[tril_indices(3)]
= x cov /=
column_norm(cov) return dot(cov, cov.
T) def analytical_mi(cov): # compute analytical MI of 3 covariate Gaussian variables cov_01 = cov[:2, :2] cov_2 = cov[2:3, 2:3] mi_xp_xpp_y = 0.5 * (log(det(cov_01)) + log(det(cov_2)) - log(det(cov)))
cov_1 = cov[1:2, 1:2] cov_23 = cov[1:, 1:] mi_xp_y
= 0.5 * (log(det(cov_1))
+ log(det(cov_2)) - log(det(cov_23))
) return mi_xp_xpp_y, mi_xp_y def residual(x, mi, α ): # penalize difference between analytical mi and target mi, α mi cov = project_posdef(x) mi_xp_y, mi_xp_y = analytical_mi(cov) return (mi_xp_xpp_y - mi) ** 2 + (mi_xp_y - α * mi) **
C E XPERIMENTS ON D IALOGUE C. I NFO NCE D ETAILS
For all InfoNCE terms, given the past, the model is trained to pick the ground-truth future among a set of N future candidates.
This candidate set includes the ground-truth future and N − 1 negative futures drawn from different proposal distributions.
To compute InfoNCE ( f ( x ≤ k ); f ( x >k )) , we consider the ground truth future of each sample as a negative candidate for the other samples in the batch.
Using this approach, the number of candidates N is equated to the batch size.
This ensures that negative samples are sampled from the marginal distribution p ( x >k ) .
To compute the conditional information bound InfoNCE S , we sample negative futures p ( y | x k ) by leveraging the GPT2 model itself, by conditioning the model only on the most recent utterance x k in the past.
C. E XPERIMENTAL S ETUP
Given memory constraints, the proposed models are trained with a batch size of 5 per GPU over 10 epochs, considering up to three utterances for the future and ﬁve utterances in the past.
All the models are trained on 2 NVIDIA V100s.
The models early-stop in the 4th epoch.
We use the Adam optimizer with a learning rate of 6.25 × 10 −5 , which we linearly decay to zero during training.
Dropout is set to 10% on all layers.
InfoNCE/InfoNCE S terms are weighted with a factor 0.1 in the loss function.
[Table]
A : I like parachuting or skydiving .
B : I’ve never done either
but they sound terrifying, not a fan of heights.
A :
But it is interesting game.
This ﬁrst parachute jump in history was made by Andre Jacques.
B : Oh really ?
Sounds like a french name, what year did he do it ?
A :
It done in October 22 1797.
They tested his contraption by leaping from a hydrogen balloon.
B : Was he successful or did he kick the bucket off that stunt?
A : I think its a success.
The military developed parachuting tech.
[Table]
|
[Table]
k ) B gt
Yeah nowadays they are a lot more stable and well made.
[Table]
That is great.
I’ve been skydiving for days now .
How is it ?
B 2 : Oh I have never ﬂown but I’m glad to know.
B 3 : I’ve been dying for it since I was a kid.
B 4 : Yes, that is why NASA had an advanced mechanics tech for months. B [Table]
: I went parachuting last Sunday and enjoyed it.
[Table]
science ﬁction is an amazing genre for
[Table]
without
[Table]
course
[Table]
That
[Table]
the reason
[Table]
picked it
[Table] [Table]
[Table]
think
[Table] [Table]
anything about Archery ?
[Table]
is a sport /
[Table]
of using a bow to propel arrows.
A [Table]
a great sport it is . Do you know where archery originated from ?
[Table]
arose
[Table]
the Mesolithic and [Table] era. [Table]
to
[Table]
arrows.
[Table]
know
[Table]
French
[Table]
it archer’s art. A
[Table]
What is your motivation?
B: That’s a tough question. I ’d say my own progress. I like seeing how much I improve with something.
You ?
A: I am retired now.
Are you familiar with the work of Mehr and Meyer, well known psychologists ?
B: They said that " Motivation is a word that is part of the popular culture as few other psychological concepts are . "
Do you agree with what they said ?
[Table]
You should check out Mark Twain’s famous motivational book !
There are theories that point out how important it is to meditate.
GPT2+InfoNCE :
They said that in psychology re- search, theories and evidence, participants who enjoyed continuous experi- ence could be more likely to be successful.
TransferTransfo :
Absolutely, I disagree.
Are you an author or a commentator?
GPT2 : That’s awesome.
So , what types of differences exist between “good " motivation and bad?