Original: L EARNING R OBUST R EPRESENTATIONS VIA M ULTIV IEW I NFORMATION B OTTLENECK
Revised: L EARNING R OBUST R EPRESENTATIONS VIA M ULTIV IEW I NFORMATION B OTTLENECK

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The information bottleneck method of Tishby et al.
Revised: The information bottleneck method of Tishby et al.

Original: (2000) provides an information-theoretic view of representation learning. The original formulation, however, can only be applied in the supervised setting where task-speciﬁc labels are available at learning time.
Revised: (2000) provides an information-theoretic method for representation learning, by training an encoder to retain all information which is relevant for predicting the label, while minimizing the amount of other, superﬂuous information in the representation. The original formulation, however, requires labeled data in order to identify which information is superﬂuous.

Original: We extend this method to the  unsupervised setting, by taking advantage of multi-view data, whichprovides two views of the same underlying entity.
Revised: In this work, we extend this ability to the multi-view unsupervised setting, in which two views of the same underlying entity are provided but the label in unknown. This enables us to identify superﬂuous information as that which is not shared by both views.

Original: A theoretical analysis leads to the deﬁnition of a new multiview model which produces state-of-the-art results on two standard multi-view datasets, Sketchy  and MIR-Flickr.
Revised: A theoretical analysis leads to the definition of a new multi-view model which produces state-of-the-art results on the Sketchy dataset and on label-limited versions of the MIR-Flickr dataset.

Original: We also extend our theory to the single-view setting by taking advantage of standard data augmentation techniques, empirically showing better generalization capabilities when compared to traditional unsupervised approaches.
Revised: We also extend our theory to the single-view setting by taking advantage of standard data augmentation techniques, empirically showing better generalization capabilities when compared to traditional unsupervised approaches for representation learning.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The goal of deep representation learning (LeCun et al., 2015) is to transform a raw observational input, x , into a representation, z , to extract useful information.
Revised: The goal of deep representation learning (LeCun et al., 2015) is to transform a raw observational input, x , into a representation, z , to extract useful information.

Original: Signiﬁcant progress has been made in deep learning via supervised representation learning, where the labels, y , for the downstream task are known while p ( y | x ) is learned directly (Sutskever et al., 2012; Hinton et al., 2012).
Revised: Signiﬁcant progress has been made in deep learning via supervised representation learning, where the labels, y , for the downstream task are known while p ( y | x ) is learned directly (Sutskever et al., 2012; Hinton et al., 2012).

Original: Due to the cost of acquiring large labeled datasets, a recently renewed focus on unsupervised representation learning seeks to generate representations, z , that allow learning of (a priori unknown) target supervised tasks more efﬁciently, i.e. with fewer labels (Devlin et al., 2018;
Revised: Due to the cost of acquiring large labeled datasets, a recently renewed focus on unsupervised representation learning seeks to generate representations, z , that allow learning of (a priori unknown) target supervised tasks more efﬁciently, i.e. with fewer labels (Devlin et al., 2018;

Original: Radford et al., 2019).
Revised: Radford et al., 2019).

Original: Our work is based on the information bottleneck principle (Tishby et al., 2000) stating that whenever a data representation discards information from the input which is not useful for a given task, it becomes less affected by nuisances, resulting in increased robustness for downstream tasks.
Revised: Our work is based on the information bottleneck principle (Tishby et al., 2000) stating that whenever a data representation discards information from the input which is not useful for a given task, it becomes less affected by nuisances, resulting in increased robustness for downstream tasks.

Original: In the supervised setting one can directly apply the information bottleneck method by minimizing the mutual information between z and x while simultaneously maximizing the mutual information between z and y (Alemi et al., 2017).
Revised: In the supervised setting, one can directly apply the information bottleneck method by minimizing the mutual information between z and x while simultaneously maximizing the mutual information between z and y (Alemi et al., 2017).

Original: In the unsupervised setting, discarding only superﬂuous information is more challenging as without labels one cannot directly identify the relevant information.
Revised: In the unsupervised setting, discarding only superﬂuous information is more challenging as without labels one cannot directly identify the relevant information.

Original: In this setting recent literature (Devon Hjelm et al., 2019; van den Oord et al., 2018) has instead focused on the InfoMax objective maximizing the mutual information between x and z , I ( x , z ) , instead of minimizing it.
Revised: Recent literature (Devon Hjelm et al., 2019; van den Oord et al., 2018) has instead focused on the InfoMax objective maximizing the mutual information between x and z , I ( x , z ) , instead of minimizing it.

Original: In this paper, we extend the information bottleneck method to the unsupervised multi-view setting.
Revised: In this paper, we extend the information bottleneck method to the unsupervised multi-view setting.

Original: To do this, we rely on a basic assumption of the multi-view literature – that each view provides the same task relevant information
Revised: To do this, we rely on a basic assumption of the multi-view literature – that each view provides the same task relevant information

Original: (Zhao et al., 2017).
Revised: (Zhao et al., 2017).

Original: Hence, one can improve generalization by discarding from the representation all information which is not shared by both views.
Revised: Hence, one can improve generalization by discarding from the representation all information which is not shared by both views.

Original: We do this through an objective which maximizes the mutual information between the representations of the two views (Multi-View InfoMax) while at the same time reducing the mutual information between each view and its corresponding representation (as with the information bottleneck).
Revised: We do this through an objective which maximizes the mutual information between the representations of the two views (Multi-View InfoMax) while at the same time reducing the mutual information between each view and its corresponding representation (as with the information bottleneck).

Original: The resulting representation contains only the information shared by both views, eliminating the effect of independent factors of variations.
Revised: The resulting representation contains only the information shared by both views, eliminating the effect of independent factors of variations.

Original: Our contributions are three-fold: (1) We extend the information bottleneck principle to the unsupervised multi-view setting and provide a rigorous theoretical analysis of it’s application.
Revised: Our contributions are three-fold: (1) We extend the information bottleneck principle to the unsupervised multi-view setting and provide a rigorous theoretical analysis of its application.

Original: (2) We deﬁne a new model that empirically leads to state-of-the-art results  on two standard multi-view datasets, Sketchy and MIR-Flickr.
Revised: (2) We deﬁne a new model that empirically leads to state-of-the-art results in the low-data setting on two standard multi-view datasets, Sketchy and MIR-Flickr.

Original: (3) We further extend our theory to the single-view case, i.e. multiple views are obtained by standard data augmentation techniques rather than from paired data, and empirically show  therobustness of our representations compared to popular unsupervised approaches.
Revised: (3) By exploiting standard data augmentation techniques, we empirically show that the representations obtained with our model in single-view settings are more robust than other popular unsupervised approaches for representation learning, connecting our theory to the choice of augmentation function class.

Original:  P RELIMINARIES AND F RAMEWORK
Revised:  P RELIMINARIES AND F RAMEWORK

Original: The challenge of representation learning can be formulated as ﬁnding a distribution p ( z | x ) that maps data observations x ∈ X into a code space z ∈ Z .
Revised: The challenge of representation learning can be formulated as ﬁnding a distribution p ( z | x ) that maps data observations x ∈ X into a code space z ∈ Z .

Original: Whenever the end goal involves predicting a label y , we consider only the z that are discriminative enough to identify the label.
Revised: Whenever the end goal involves predicting a label y , we consider only the z that are discriminative enough to identify the label.

Original: This requirement can be quantiﬁed by considering the amount of target information that remains accessible after encoding the data, and is known in literature as sufﬁciency of z for y (Achille & Soatto, 2018):
Revised: This requirement can be quantiﬁed by considering the amount of target information that remains accessible after encoding the data, and is known in literature as sufﬁciency of z for y (Achille & Soatto, 2018):

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: Sufﬁciency:
Revised: Sufﬁciency:

Original: A representation z of x is sufﬁcient for y
Revised: A representation z of x is sufﬁcient for y

Original: if and only if I ( x ; y | z ) =
Revised: if and only if I ( x ; y | z ) =

Original: Any model that has access to a sufﬁcient representation z must be able to predict y at least as accurately as if it has access to the original data x instead.
Revised: Any model that has access to a sufﬁcient representation z must be able to predict y at least as accurately as if it has access to the original data x instead.

Original: In fact, z is sufﬁcient for y if and only if the amount of information regarding the task is unchanged by the encoding procedure (see
Revised: In fact, z is sufﬁcient for y if and only if the amount of information regarding the task is unchanged by the encoding procedure (see

Original: Proposition B.1 in the Appendix):
Revised: Proposition B.1 in the Appendix):

Original: [Equation]
Revised: [Equation]

Original: Among sufﬁcient representations, the ones that result in better generalization for unlabeled data instances are particularly appealing.
Revised: Among sufﬁcient representations, the ones that result in better generalization for unlabeled data instances are particularly appealing.

Original: When x has higher information content than y , some of the information in x must be irrelevant for the prediction task.
Revised: When x has higher information content than y , some of the information in x must be irrelevant for the prediction task.

Original: This can be better understood by subdividing
Revised: This can be better understood by subdividing

Original: I ( x ; z ) into three components by using the chain rule of mutual information (see Appendix A): 
Revised: I ( x ; z ) into three components by using the chain rule of mutual information (see Appendix A): I

Original: [Equation]
Revised: [Equation]

Original: (cid:124) (cid:123)(cid:122) (cid:125) predictive information not in z .

Original: Conditional mutual information I ( x ; z | y ) represents the information in z that is not predictive of y , i.e. superﬂuous information .
Revised: Conditional mutual information I ( x ; z | y ) represents the information in z that is not predictive of y , i.e. superﬂuous information .

Original: While I ( x ; y ) is a constant determined by how much label information is accessible from the raw observations; the last term I ( x ; y | z ) represents the amount of information regarding y that is lost by encoding x into z .
Revised: While I ( x ; y ) is a constant determined by how much label information is accessible from the raw observations; the last term I ( x ; y | z ) represents the amount of information regarding y that is lost by encoding x into z .

Original: Note that this last term is zero whenever z is sufﬁcient for x .
Revised: Note that this last term is zero whenever z is sufﬁcient for x .

Original: Since the amount of predictive information I ( x ; y ) is ﬁxed,
Revised: Since the amount of predictive information I ( x ; y ) is ﬁxed,

Original: Proposition 2.1.
Revised: Proposition 2.1.

Original: A sufﬁcient representation z of x for y is minimal whenever I ( x ; z | y ) is minimal.
Revised: A sufﬁcient representation z of x for y is minimal whenever I ( x ; z | y ) is minimal.

Original: Minimizing the amount of superﬂuous information can be done directly only in supervised settings.
Revised: Minimizing the amount of superﬂuous information can be done directly only in supervised settings.

Original: In fact, reducing I ( x ; z ) without violating the sufﬁciency constraint necessarily requires making some additional assumptions on the predictive task (see Theorem B.1 in the Appendix).
Revised: In fact, reducing I ( x ; z ) without violating the sufﬁciency constraint necessarily requires making some additional assumptions on the predictive task (see Theorem B.1 in the Appendix).

Original: In Sectionwe describe a strategy to safely reduce the information content of a representation even when the label y is not observed, by exploiting redundant information in the form of an additional view on the data.
Revised: In Sectionwe describe a strategy to safely reduce the information content of a representation even when the label y is not observed, by exploiting redundant information in the form of an additional view on the data.

Original:  M ULTI -V IEW I NFORMATION B OTTLENECK
Revised:  M ULTI -V IEW I NFORMATION B OTTLENECK

Original: Let v 1 and v 2 be two images of the same object from different view-points and y be its label.
Revised: Let v 1 and v 2 be two images of the same object from different view-points and y be its label.

Original: Assuming that the object is clearly distinguishable from both v 1 and v 2 , any representation z with all the information that is accessible from both views would also contain the necessary label information.
Revised: Assuming that the object is clearly distinguishable from both v 1 and v 2 , any representation z with all the information that is accessible from both views would also contain the necessary label information.

Original: Furthermore, if z captures only the details that are visible from both pictures, it would reduce the total information content, discarding the view-speciﬁc details and reducing the sensitivity of the representation to view-changes.
Revised: Furthermore, if z captures only the details that are visible from both pictures, it would reduce the total information content, discarding the view-speciﬁc details and reducing the sensitivity of the representation to view-changes.

Original: The theory to support this intuition is described in the following where v 1 and v 2 are jointly observed and referred to as data-views.
Revised: The theory to support this intuition is described in the following where v 1 and v 2 are jointly observed and referred to as data-views.

Original: 3. S UFFICIENCY AND R OBUSTNESS IN THE M ULTI - VIEW SETTING
Revised: 3. S UFFICIENCY AND R OBUSTNESS IN THE M ULTI - VIEW SETTING

Original: In this section we extend our analysis of sufﬁciency and robustness to the multi-view setting.
Revised: In this section we extend our analysis of sufﬁciency and robustness to the multi-view setting.

Original: Intuitively, we can guarantee that z is sufﬁcient for predicting y even without knowing y by ensuring that z maintains all information which is shared by v 1 and v 2 .
Revised: Intuitively, we can guarantee that z is sufﬁcient for predicting y even without knowing y by ensuring that z maintains all information which is shared by v 1 and v 2 .

Original: This intuition relies on a basic assumption of the multi-view environment – that each view provides the same task relevant information.
Revised: This intuition relies on a basic assumption of the multi-view environment – that each view provides the same task relevant information.

Original: To formalize this we deﬁne redundancy .
Revised: To formalize this we deﬁne redundancy .

Original: [Figure]

Original: [Figure]

Original: Single-View vvp θ ( z 1 | v 1 ) p ψ ( z 2 | v 2 ) ∼ ∼ zz

Original: D SKL ˆ I ξ ( z 1 ; z 2 ) x vvp θ ( z 1 | v 1 ) p θ ( z 2 | v 2 ) ∼ ∼ zz

Original: D SKL ˆ I ξ ( z 1 ; z 2 ) tt

Original: Algorithm

Original: MIB θ, if Multi-View then { ( v ( i ) 1 , v ( i ) 2 ) } Bi

Original: =1 ∼ p ( v 1 , v 2 ) ; else { x ( i ) }

Original: Bi =1 ∼ p ( x ) ; { ( t ( i ) 1 , t ( i ) 2 ) } Bi =1 ∼ p 2 ( t ) ; for i ← 1 to B do v ( i ) 1 , v ( i ) 2 ← t ( i ) 1 ( x ( i ) ) , t ( i ) 2 ( x ( i ) ) ; end for end if for i ← 1 to B do z ( i ) 1 , z ( i ) 2 ∼ p θ ( z 1 | v ( i ) 1 ) , p ψ ( z 2 | v ( i ) 2 ) ;

Original: L ( i ) m ← D SKL ( p θ ( z 1 | v ( i ) 1 ) ||

Original: p ψ ( z 2 | v ( i ) 2 )) ; end for return βB

Original: (cid:80) Bi

Original: =1 L ( i ) m

Original: − ˆ I ξ ( { ( z ( i ) 1 , z

Original: ( i ) 2 ) }

Original: Bi =1 )

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: Redundancy: v 1 is redundant with respect to v 2 for y
Revised: Redundancy: v 1 is redundant with respect to v 2 for y

Original: if and only if I ( y ; v 1 | v 2 ) =
Revised: if and only if I ( y ; v 1 | v 2 ) =

Original: Intuitively, a view v 1 is redundant for a task whenever it is irrelevant for the prediction of y when v 2 is already observed.
Revised: Intuitively, a view v 1 is redundant for a task whenever it is irrelevant for the prediction of y when v 2 is already observed.

Original: Whenever v 1 and v 2 are mutually redundant ( v 1 is redundant with respect to v 2  , and vice-versa), we can show the following:
Revised: Whenever v 1 and v 2 are mutually redundant ( v 1 is redundant with respect to v 2 for y , and vice-versa), we can show the following:

Original: Corollary 1.
Revised: Corollary 1.

Original: Let v 1 and v 2 be two mutually redundant views for a target y and let z 1 be a representation of v 1 .
Revised: Let v 1 and v 2 be two mutually redundant views for a target y and let z 1 be a representation of v 1 .

Original: If z 1 is sufﬁcient for v 

Revised: If z 1 is sufﬁcient for v

Original: ( I ( v 1 ; v 2 | z 1 ) = 0 ) then z 1 is as predictive for y as the joint observation of the two views ( I ( v 1 v 2 ; y )
Revised: ( I ( v 1 ; v 2 | z 1 ) = 0 ) then z 1 is as predictive for y as the joint observation of the two views ( I ( v 1 v 2 ; y )

Original: =
Revised: =

Original: I ( y ; z 1 ) ).
Revised: I ( y ; z 1 ) ).

Original: In other words, whenever it is possible to assume mutual redundancy, any representation which contains all the information shared by both views (the redundant information) is as useful as both views for predicting the label y .
Revised: In other words, whenever it is possible to assume mutual redundancy, any representation which contains all the information shared by both views (the redundant information) is as useful as their joint observation for predicting the label y .

Original: By factorizing the mutual information between v 1 and z 1 analogously to Equation 2, we can identify 3 components:
Revised: By factorizing the mutual information between v 1 and z 1 analogously to Equation 2, we can identify 3 components:

Original: [Equation]
Revised: [Equation]

Original: Since I ( v 1 ; v 2 ) is a constant that depends only on the two views and I ( v 1 ; v 2 | z 1 ) must be zero if we want the representation to be sufﬁcient for the label, we conclude that I ( v 1 ; z 1 ) can be reduced by minimizing I ( v 1 ; z 1 | v 2 ) .
Revised: Since I ( v 1 ; v 2 ) is a constant that depends only on the two views and I ( v 1 ; v 2 | z 1 ) must be zero if we want the representation to be sufﬁcient for the label, we conclude that I ( v 1 ; z 1 ) can be reduced by minimizing I ( v 1 ; z 1 | v 2 ) .

Original: This term intuitively represents the information z 1 contains which is unique to v 1 and not shared by v 2 .
Revised: This term intuitively represents the information z 1 contains which is unique to v 1 and not shared by v 2 .

Original: Since we assumed mutual redundancy between the two views, this information must be irrelevant for the predictive task and, therefore, it can be safely discarded.
Revised: Since we assumed mutual redundancy between the two views, this information must be irrelevant for the predictive task and, therefore, it can be safely discarded.

Original: The proofs and formal assertions for the above statements and Corollary 1 can be found in Appendix
Revised: The proofs and formal assertions for the above statements and Corollary 1 can be found in Appendix

Original: B.
Revised: B.

Original: The less the two views have in common, the more I ( z 1 ; v 1 ) can be reduced without violating sufﬁciency for the label, the more robust the resulting representation.
Revised: The less the two views have in common, the more I ( z 1 ; v 1 ) can be reduced without violating sufﬁciency for the label, the more robust the resulting representation.

Original: At the extreme, v 1 and vshare only label information, in which case we can show that z 1 is minimal for y and our method is identical to the supervised information bottleneck method without needing to access the labels.
Revised: At the extreme, v 1 and vshare only label information, in which case we can show that z 1 is minimal for y and our method is identical to the supervised information bottleneck method without needing to access the labels.

Original: Conversely, if v 1 and v 2 are identical, then our method degenerates to the InfoMax principle since no information can be safely discarded (see Appendix C).
Revised: Conversely, if v 1 and v 2 are identical, then our method degenerates to the InfoMax principle since no information can be safely discarded (see Appendix E).

Original: 3. T HE M ULTI -V IEW I NFORMATION B OTTLENECK L OSS F UNCTION
Revised: 3. T HE M ULTI -V IEW I NFORMATION B OTTLENECK L OSS F UNCTION

Original: Given v 1 and v 2 that satisfy the mutual redundancy condition for a label y , we would like to deﬁne an objective function for the representation z 1 of v 1 that discards as much information as possible without losing any label information.
Revised: Given v 1 and v 2 that satisfy the mutual redundancy condition for a label y , we would like to deﬁne an objective function for the representation z 1 of v 1 that discards as much information as possible without losing any label information.

Original: In Section 3.1 we showed that we can maintain sufﬁciency for y by ensuring that I ( v 1 ; v 2 | z 1 ) = 0 , and that decreasing I ( z 1 ; v 1 | v 2 ) will increase the robustness of the representation by discarding irrelevant information.
Revised: In Section 3.1 we showed that we can maintain sufﬁciency for y by ensuring that I ( v 1 ; v 2 | z 1 ) = 0 , and that decreasing I ( z 1 ; v 1 | v 2 ) will increase the robustness of the representation by discarding irrelevant information.

Original: So if we combine these two terms using a relaxed Lagrangian objective, then we obtain:
Revised: So if we combine these two terms using a relaxed Lagrangian objective, then we obtain:

Original: [Equation]
Revised: [Equation]

Original: where θ denotes the dependency on the parameters of the encoder p θ ( z 1 | v 1 ) , and λ 1 represents the Lagrangian multiplier introduced by the constrained optimization.
Revised: where θ denotes the dependency on the parameters of the encoder p θ ( z 1 | v 1 ) , and λ 1 represents the Lagrangian multiplier introduced by the constrained optimization.

Original: Symmetrically, we deﬁne a loss L 2 to optimize the parameters ψ of a conditional distribution p ψ ( z 2 | v 2 ) that deﬁnes a robust sufﬁcient representation z 2 of the second view v 2 :
Revised: Symmetrically, we deﬁne a loss L 2 to optimize the parameters ψ of a conditional distribution p ψ ( z 2 | v 2 ) that deﬁnes a robust sufﬁcient representation z 2 of the second view v 2 :

Original: [Equation]
Revised: [Equation]

Original: Although L 1 and L 2 can not be computed directly, by deﬁning z 1 and z 2 on the same domain Z and re-parametrizing the Lagrangian multipliers, their sum can be upper bounded as follows: 
Revised: Although L 1 and L 2 can not be computed directly, by deﬁning z 1 and z 2 on the same domain Z and re-parametrizing the Lagrangian multipliers, their sum can be upper bounded as follows: [Figure] ∼ [Figure]

Revised: [Figure]

Revised: if Multi-View then

Original: [Equation]
Revised: [Figure]

Revised: else

Revised: [Figure]

Revised: [Figure]

Revised: for i ← 1 to B do

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: ←

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: end for end if for i ← 1 to B do

Revised: [Figure]

Revised: end for

Revised: [Figure]

Revised: [Figure]

Revised: [Equation]

Revised: [Figure]

Revised: [Equation]

Revised: [Figure]

Revised: [Equation]

Revised: [Figure]

Revised: [Equation]

Revised: [Figure]

Revised: [Equation]

Original:  where D SKL is the symmetrized KL divergence obtained by averaging D KL ( p θ ( z 1 | v 1 )
Revised: [Figure] [Table] [Equation] where D SKL is the symmetrized KL divergence obtained by averaging D KL ( p θ ( z 1 | v 1 )

Original: ||
Revised: ||

Original: p ψ ( z 2 | v 2 )) and D KL ( p ψ ( z 2 | v 2 )
Revised: p ψ ( z 2 | v 2 )) and D KL ( p ψ ( z 2 | v 2 )

Original: ||
Revised: ||

Original: p θ ( z 1 | v 1 )) , while the coefﬁcient β deﬁnes the trade-off between sufﬁciency and robustness of the representation, which is a hyper-parameter in this work.
Revised: p θ ( z 1 | v 1 )) , while the coefﬁcient β deﬁnes the trade-off between sufﬁciency and robustness of the representation, which is a hyper-parameter in this work.

Original: The resulting MultiView Infomation Bottleneck (MIB) model (Equation 5) is visualized in Figure 1, while the batchbased computation of the loss function is summarized in Algorithm 1.
Revised: The resulting MultiView Infomation Bottleneck (MIB) model (Equation 5) is visualized in Figure 1, while the batchbased computation of the loss function is summarized in Algorithm 1.

Original: The symmetrized KL divergence D SKL ( p θ ( z 1 | v 1 ) || p ψ ( z 2 | v 2 )) can be computed directly whenever p θ ( z 1 | v 1 ) and p ψ ( z 2 | v 2 ) have a known density, while the mutual information between the two representations I θψ ( z 1 ; z 2 ) can be maximized by using any sample-based differentiable mutual information lower bound.
Revised: The symmetrized KL divergence D SKL ( p θ ( z 1 | v 1 ) || p ψ ( z 2 | v 2 )) can be computed directly whenever p θ ( z 1 | v 1 ) and p ψ ( z 2 | v 2 ) have a known density, while the mutual information between the two representations I θψ ( z 1 ; z 2 ) can be maximized by using any sample-based differentiable mutual information lower bound.

Original: Both the Jensen-Shannon I JSD (Devon Hjelm et al., 2019; Poole et al., 2019) and the InfoNCE I NCE (van den Oord et al., 2018) estimators used in this work require introducing an auxiliary parameteric model C ξ ( z 1 , z 2 ) , which is jointly optimized during the training procedure.
Revised: Both the Jensen-Shannon I JS (Devon Hjelm et al., 2019; Poole et al., 2019) and the InfoNCE I NCE (van den Oord et al., 2018) estimators used in this work require introducing an auxiliary parameteric model C ξ ( z 1 , z 2 ) , which is jointly optimized during the training procedure.

Original: The full derivation for the MIB loss function can be found in Appendix D.
Revised: The full derivation for the MIB loss function can be found in Appendix F.

Original: 3. S ELF - SUPERVISION AND I NVARIANCE
Revised: 3. S ELF - SUPERVISION AND I NVARIANCE

Original: In this section, we introduce a methodology to build mutually redundant views starting from single observations x with domain X by exploiting known symmetries of the task.
Revised: In this section, we introduce a methodology to build mutually redundant views starting from single observations x with domain X by exploiting known symmetries of the task.

Original: By picking a class T of functions t : X → W that do not affect label information, it is possible to artiﬁcially build views that satisfy mutual redundancy for y with a procedure similar to dataaugmentation.
Revised: By picking a class T of functions t : X → W that do not affect label information, it is possible to artiﬁcially build views that satisfy mutual redundancy for y with a procedure similar to dataaugmentation.

Original: Let t 1 and t 2 be two random variables over T , then v 1 :=
Revised: Let t 1 and t 2 be two random variables over T , then v 1 :=

Original: t 1 ( x ) and v 2 := t 2 ( x ) must be mutually redundant for y .
Revised: t 1 ( x ) and v 2 := t 2 ( x ) must be mutually redundant for y .

Original: Since no function in T affects label information ( I ( v 1 ; y ) =
Revised: Since no function in T affects label information ( I ( v 1 ; y ) =

Original: I ( v 2 ; y )
Revised: I ( v 2 ; y )

Original: =
Revised: =

Original: I ( x ; y ) ), a representation z 1 of v 1 that is sufﬁcient to v 2 must contain same amount of predictive information as x .
Revised: I ( x ; y ) ), a representation z 1 of v 1 that is sufﬁcient for v 2 must contain same amount of predictive information as x .

Original: Formal proofs can be found in Appendix B.4.
Revised: Formal proofs can be found in Appendix B.4.

Original: Whenever the two transformations for the same observations are independent ( I ( t 1 ; t 2 | x ) = 0 ), they introduce uncorrelated variations in the two views.
Revised: Whenever the two transformations for the same observations are independent ( I ( t 1 ; t 2 | x ) = 0 ), they introduce uncorrelated variations in the two views.

Original: As an example, if T represents a set of small translations, the two views will differ by a small shift.
Revised: As an example, if T represents a set of small translations, the two views will differ by a small shift.

Original: Since this information is not shared, z 1 that contains only common information between v 1 and v 2 will discard ﬁne-grained details regarding the position.
Revised: Since this information is not shared, z 1 that contains only common information between v 1 and v 2 will discard ﬁne-grained details regarding the position.

Original: For single-view datasets, we generate the two views v 1 and v 2 by independently sampling two functions from the same function class T with uniform probability.
Revised: For single-view datasets, we generate the two views v 1 and v 2 by independently sampling two functions from the same function class T with uniform probability.

Original: Since the resulting t 1 and t have the same distribution, the two generated views will also have the same marginals.
Revised: Since the resulting t 1 and t have the same distribution, the two generated views will also have the same marginals.

Original: For this reason, the two conditional distributions p θ ( z 1 | v 1 ) and p ψ ( z 2 | v 2 ) can share their parameters and only one encoder can be used.
Revised: For this reason, the two conditional distributions p θ ( z 1 | v 1 ) and p ψ ( z 2 | v 2 ) can share their parameters and only one encoder can be used.

Original: Full (or partial) parameter sharing can be also applied in the multiview settings whenever the two views have the same (or similar) marginal distributions.
Revised: Full (or partial) parameter sharing can be also applied in the multiview settings whenever the two views have the same (or similar) marginal distributions.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The space of all the possible representations z of x for a predictive task y can be represented as a region in the Information Plane (Tishby et al., 2000).
Revised: The space of all the possible representations z of x for a predictive task y can be represented as a region in the Information Plane (Tishby et al., 2000).

Original: Each representation is characterised by the amount of information regarding the raw observation I ( x ; z ) and the corresponding measure of accessible predictive information I ( y ; z ) ( x and y axis respectively on Figure 2).
Revised: Each representation is characterised by the amount of information regarding the raw observation I ( x ; z ) and the corresponding measure of accessible predictive information I ( y ; z ) ( x and y axis respectively on Figure 2).

Original: Ideally, a good representation would be maximally informative about the label while retaining a minimal amount of information from the observations (top left corner of the parallelogram).
Revised: Ideally, a good representation would be maximally informative about the label while retaining a minimal amount of information from the observations (top left corner of the parallelogram).

Original: Further details on the
Revised: Further details on the

Original: Information Plane and the bounds visualized in Figure 2 are described in Appendix E.
Revised: Information Plane and the bounds visualized in Figure 2 are described in Appendix C.

Original: [Figure]
Revised: [Figure]

Original: Thanks to recent progress in mutual information estimation (Nguyen et al., 2008; Ishmael
Revised: Thanks to recent progress in mutual information estimation (Nguyen et al., 2008; Ishmael

Original: Belghazi et al., 2018; Poole et al., 2019), the
Revised: Belghazi et al., 2018; Poole et al., 2019), the

Original: InfoMax principle (Linsker, 1988) has gained attention for unsupervised representation learning (Devon Hjelm et al., 2019; van den Oord et al., 2018).
Revised: InfoMax principle (Linsker, 1988) has gained attention for unsupervised representation learning (Devon Hjelm et al., 2019; van den Oord et al., 2018).

Original: Since the InfoMax objective involves maximizing I ( x ; z ) , the resulting representation aims to preserve all the information regarding the raw observations (top right corner in Figure 2).
Revised: Since the InfoMax objective involves maximizing I ( x ; z ) , the resulting representation aims to preserve all the information regarding the raw observations (top right corner in Figure 2).

Original: Despite their success, Tschannen et al.
Revised: Despite their success, Tschannen et al.

Original: (2019) has shown that the effectiveness of the InfoMax models is due to inductive biases introduced by the architecture and estimators rather than the training objective itself, since the InfoMax objective can be trivially maximized by using invertible encoders.
Revised: (2019) has shown that the effectiveness of the InfoMax models is due to inductive biases introduced by the architecture and estimators rather than the training objective itself, since the InfoMax objective can be trivially maximized by using invertible encoders.

Original: On the other hand, Variational Autoencoders (VAEs) (Kingma & Welling, 2014) deﬁne a training objective that balances compression and reconstruction error (Alemi et al., 2018) through an hyper-parameter β .
Revised: On the other hand, Variational Autoencoders (VAEs) (Kingma & Welling, 2014) deﬁne a training objective that balances compression and reconstruction error (Alemi et al., 2018) through an hyper-parameter β .

Original: Whenever β is close to 0, the VAE objective aims for a lossless representation, approaching the same region of the Information Plane as the one targeted by InfoMax (Barber &
Revised: Whenever β is close to 0, the VAE objective aims for a lossless representation, approaching the same region of the Information Plane as the one targeted by InfoMax (Barber &

Original: Agakov, 2003).
Revised: Agakov, 2003).

Original: When β approaches large values, the representation becomes more compressed, showing increased generalization and disentanglement (Higgins et al., 2017; Burgess et al., 2018), and, as β approaches inﬁnity, I ( z ; x ) goes to zero.
Revised: When β approaches large values, the representation becomes more compressed, showing increased generalization and disentanglement (Higgins et al., 2017; Burgess et al., 2018), and, as β approaches inﬁnity, I ( z ; x ) goes to zero.

Original: During this transition from low to high β , however, there are no guarantees that VAEs will retain label information (Theorem B.1 in the Appendix).
Revised: During this transition from low to high β , however, there are no guarantees that VAEs will retain label information (Theorem B.1 in the Appendix).

Original: The path between the two regimes depends on how well the label information aligns with the inductive bias introduced by encoder (Jimenez Rezende & Mohamed, 2015; Kingma et al., 2016), prior (Tomczak & Welling, 2018) and decoder architectures (Gulrajani et al., 2017; Chen et al., 2017).
Revised: The path between the two regimes depends on how well the label information aligns with the inductive bias introduced by encoder (Jimenez Rezende & Mohamed, 2015; Kingma et al., 2016), prior (Tomczak & Welling, 2018) and decoder architectures (Gulrajani et al., 2017; Chen et al., 2017).

Original: Concurrent work applies the InfoMax principle in Multi-View settings (Ji et al., 2019; H´enaff et al., 2019; Tian et al., 2019; Bachman et al., 2019), aiming to maximize mutual information between the representation z of a ﬁrst data-view x and a second one v 2 .
Revised: Concurrent work applies the InfoMax principle in Multi-View settings (Ji et al., 2019; H´enaff et al., 2019; Tian et al., 2019; Bachman et al., 2019), aiming to maximize mutual information between the representation z of a ﬁrst data-view x and a second one v 2 .

Original: The target representation for the MultiView InfoMax (MV-InfoMax) models should contain at least the amount of information in x that is predictive for v 2 , targeting the region I ( z ; x ) ≥ I ( x ; v 2 ) on the Information Plane.
Revised: The target representation for the MultiView InfoMax (MV-InfoMax) models should contain at least the amount of information in x that is predictive for v 2 , targeting the region I ( z ; x ) ≥ I ( x ; v 2 ) on the Information Plane.

Original: Whenever x is redundant with respect to v 2 for y , the representation must be also sufﬁcient for y (Corollary 1).
Revised: Whenever x is redundant with respect to v 2 for y , the representation must be also sufﬁcient for y (Corollary 1).

Original: Since z has no incentive in discarding any information regarding x , a representation that is optimal according to the InfoMax principle is also optimal for MV-InfoMax.
Revised: Since z has no incentive in discarding any information regarding x , a representation that is optimal according to the InfoMax principle is also optimal for MV-InfoMax.

Original: Our model with β =(Equation 5) belong to this family of objectives since the minimality term is discarded.
Revised: Our model with β =(Equation 5) belong to this family of objectives since the minimality term is discarded.

Original: In contrast to all of the above, our work is the ﬁrst to explicitly identify and discard superﬂuous information from the representation in the unsupervised multi-view setting.
Revised: In contrast to all of the above, our work is the ﬁrst to explicitly identify and discard superﬂuous information from the representation in the unsupervised multi-view setting.

Original: The idea of discardingonly irrelevant information was introduced in Tishby et al.
Revised: The idea of discarding irrelevant information was introduced in Tishby et al.

Original: (2000) and identiﬁed as one of the possible reasons behind the generalization capabilities of deep neural networks by Tishby & Zaslavsky (2015) and Achille & Soatto (2018),
Revised: (2000) and identiﬁed as one of the possible reasons behind the generalization capabilities of deep neural networks by Tishby & Zaslavsky (2015) and Achille & Soatto (2018).

Original: but so far has been utilized only in supervised settings (Alemi et al., 2017).
Revised: The direct removal of superﬂuous information has, so far, been done only in supervised settings (Alemi et al., 2017).

Original: Conversely, β -VAE models remove informationindiscriminately without explicitly
Revised: Conversely, β -VAE models remove information

Original: [Table]
Revised: [Table]

Original:  identifying which information is superﬂuous, and the InfoMax and Multiview-InfoMax methods do not explicitly try to remove superﬂuous information at all.
Revised: indiscriminately without identifying which part is superﬂuous, and the InfoMax and Multi-View InfoMax methods do not explicitly try to remove superﬂuous information at all.

Original: In fact, among the representations that are optimal according to Multi-View InfoMax (purple dotted line in Figure 2), the MIB objective results in thethe representation with the least superﬂuous information,
Revised: In fact, among the representations that are optimal according to Multi-View InfoMax (purple dotted line in Figure 2), the MIB objective results in the representation with the least superﬂuous information,

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: e. the most robust.
Revised: e. the most robust.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section we demonstrate the effectiveness of our model against state-of-the-art baselines in both the Multi-View and Single-View setting.
Revised: In this section we demonstrate the effectiveness of our model against state-of-the-art baselines in both the multi-view and single-view setting.

Original: In the Single-View setting, we also estimate the coordinates on the Information Plane for each of the baseline methods as well as our method to validate the theory in Sec. 3.
Revised: In the single-view setting, we also estimate the coordinates on the Information Plane for each of the baseline methods as well as our method to validate the theory in Section 3.

Revised: The results reported in the following sections are obtained using the Jensen-Shannon I JS (Devon

Revised: Hjelm et al., 2019; Poole et al., 2019) estimator, which resulted in better performance for MIB and the other InfoMax-based models (Table 2 in the supplementary material).

Revised: In order to facilitate the comparison between the effect of the different loss functions, the same estimator is used across the different models.

Original: 5. M ULTI -V IEW T ASKS
Revised: 5. M ULTI -V IEW T ASKS

Original: We compare MIB on the sketch-based image retrieval (Sangkloy et al., 2016) and Flickr multiclass image classiﬁcation  tasks with domain speciﬁc and prior Multi-View learning methods.
Revised: We compare MIB on the sketch-based image retrieval (Sangkloy et al., 2016) and Flickr multiclass image classiﬁcation (Huiskes & Lew, 2008) tasks with domain speciﬁc and prior multi-view learning methods.

Original: Sketchy The Sketchy dataset (Sangkloy et al., 2016) consists of 12,500 images and 75,471 handdrawn sketches of objects from 125 classes.
Revised: Sketchy The Sketchy dataset (Sangkloy et al., 2016) consists of 12,500 images and 75,471 handdrawn sketches of objects from 125 classes.

Original: As in Liu et al.
Revised: As in Liu et al.

Original: (2017), we also include another 60, images from the ImageNet (Deng et al., 2009) from the same classes, which results in total 73, natural object images.
Revised: (2017), we also include another 60, images from the ImageNet (Deng et al., 2009) from the same classes, which results in total 73, natural object images.

Original: As per the experimental protocol of Zhang et al.
Revised: As per the experimental protocol of Zhang et al.

Original: (2018), a total of 6, sketches (50 sketches per category) are randomly selected and removed from the training set for testing purpose, which leaves 69,221 sketches for training the model.
Revised: (2018), a total of 6, sketches (50 sketches per category) are randomly selected and removed from the training set for testing purpose, which leaves 69,221 sketches for training the model.

Original: The sketch-based image retrieval task is a ranking of 73,002 natural images according to the unseen test (query) sketch.
Revised: The sketch-based image retrieval task is a ranking of 73,002 natural images according to the unseen test (query) sketch.

Original: Retrieval is done for our model by generating representations for the query sketch as well as all natural images, and ranking the image by the euclidean distance of their representation from the sketch representation.
Revised: Retrieval is done for our model by generating representations for the query sketch as well as all natural images, and ranking the image by the euclidean distance of their representation from the sketch representation.

Original: The baselines use various domain speciﬁc ranking methodologies.
Revised: The baselines use various domain speciﬁc ranking methodologies.

Original: Model performance is computed based on the class of the ranked pictures corresponding to the query.
Revised: Model performance is computed based on the class of the ranked pictures corresponding to the query.

Original: The training set consists of pairs of image v 1 and sketch v 2 randomly selected from the same class, to ensure that both views contain the equivalent label information (mutual redundancy).
Revised: The training set consists of pairs of image v 1 and sketch v 2 randomly selected from the same class, to ensure that both views contain the equivalent label information (mutual redundancy).

Original: Following recent prior works (Zhang et al., 2018; Dutta & Akata, 2019), we use features extracted from images and sketches by a VGG (Simonyan & Zisserman, 2014) architecture trained for classiﬁcation on the TU-Berlin dataset (Eitz et al., 2012).
Revised: Following recent prior works (Zhang et al., 2018; Dutta & Akata, 2019), we use features extracted from images and sketches by a VGG (Simonyan & Zisserman, 2014) architecture trained for classiﬁcation on the TU-Berlin dataset (Eitz et al., 2012).

Original: The resulting ﬂattened 4096-dimensional feature vectors are fed to our image and sketch encoders to produce a 64-dimensional representation. 
Revised: The resulting ﬂattened 4096-dimensional feature vectors are fed to our image and sketch encoders to produce a 64-dimensional representation. Both Method [Figure]

Original: Both encoders consist of neural networks with hidden layers of 2048 and 1024 units respectively.
Revised: [Table] [Table] [Table] [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: Examples

Revised: [Figure]

Revised: [Figure]

Revised: encoders consist of neural networks with hidden layers of 2048 and 1024 units respectively.

Original: Size of the representation and regularization strength β are tuned on a validation sub-split.
Revised: Size of the representation and regularization strength β are tuned on a validation sub-split.

Original: We evaluate
Revised: We evaluate

Original: MIB on ﬁve different train/test splits 1 and report mean and standard deviation in Table 5.1.
Revised: MIB on ﬁve different train/test splits 1 and report mean and standard deviation in Table 5.1.

Original: Further details on our training procedure and architecture are in Appendix F.
Revised: Further details on our training procedure and architecture are in Appendix G.

Original: Table 5.1 shows that the our model achieves strong performance for both mean average precision (mAP@all) and precision at 200 (Prec@200), suggesting that the representation is able to capture [Figure] Contrastive † (Hermann & [Table]
Revised: Table 5.1 shows that the our model achieves strong performance for both mean average precision (mAP@all) and precision at 200 (Prec@200), suggesting that the representation is able to capture the common class information between the paired pictures and sketches.

Original: [Figure]

Original: [Figure]

Original: et al

Original: [Table]

Original: [Figure]

Original: [Figure]

Original: (Ngiam

Original: [Figure]

Original: al

Original: [Table]

Original: [Figure]

Original: [Figure]

Original: [Table]

Original: Wang et al.,

Original: [Figure]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: the common class information between the paired pictures and sketches.

Original: The effectiveness of MIB on the retrieval task can be mostly imputed to the regularization introduced with the symmetrized
Revised: The effectiveness of MIB on the retrieval task can be mostly imputed to the regularization introduced with the symmetrized

Original: KL divergence between the two encoded views.
Revised: KL divergence between the two encoded views.

Original: Other than discarding view-private information, this term actively aligns the representations of v 1 and v 2 , making the MIB model especially suitable for retrieval tasks
Revised: Other than discarding view-private information, this term actively aligns the representations of v 1 and v 2 , making the MIB model especially suitable for retrieval tasks

Original: MIR-Flickr
Revised: MIR-Flickr

Original: The MIR-Flickr dataset (Huiskes & Lew, 2008) consists of 1M images annotated with 800K distinct user tags.
Revised: The MIR-Flickr dataset (Huiskes & Lew, 2008) consists of 1M images annotated with 800K distinct user tags.

Original: Each image is represented by a vector of 3,857 hand-crafted image features ( v 1 ), while the 2,000 most frequent tags are used to produce a 2000-dimensional multihot encoding ( v 2 ) for each picture.
Revised: Each image is represented by a vector of 3,857 hand-crafted image features ( v 1 ), while the 2,000 most frequent tags are used to produce a 2000-dimensional multihot encoding ( v 2 ) for each picture.

Original: The dataset is divided into labeled and unlabeled sets that respectively contain 975K and 25K images, where the labeled set also contains 38 distinct topic classes together with the user tags.
Revised: The dataset is divided into labeled and unlabeled sets that respectively contain 975K and 25K images, where the labeled set also contains 38 distinct topic classes together with the user tags.

Original: Training images with less than two tags are removed, which reduces the total number of training samples to 749,647 pairs (Sohn et al., 2014;
Revised: Training images with less than two tags are removed, which reduces the total number of training samples to 749,647 pairs (Sohn et al., 2014;

Original: Wang et al., 2016).
Revised: Wang et al., 2016).

Original: The labeled set contains 5 different splits of train, validation and test sets of size 10K/5K/10K respectively.
Revised: The labeled set contains 5 different splits of train, validation and test sets of size 10K/5K/10K respectively.

Original: Following a standard procedure in literature (Srivastava & Salakhutdinov, 2014; Wang et al., 2016), we train our model on the unlabeled pairs of images and tags.
Revised: Following a standard procedure in literature (Srivastava & Salakhutdinov, 2014; Wang et al., 2016), we train our model on the unlabeled pairs of images and tags.

Original: Then a multi-label logistic classiﬁer is trained from the representation of 10K labeled train images to the corresponding macro-categories.
Revised: Then a multi-label logistic classiﬁer is trained from the representation of 10K labeled train images to the corresponding macro-categories.

Original: The quality of the representation is assessed based on the performance of the trained logistic classiﬁer on the labeled test set.
Revised: The quality of the representation is assessed based on the performance of the trained logistic classiﬁer on the labeled test set.

Original: Each encoder consists of a multi-layer perceptron of 4 hidden layers with ReLU activations learning two 1024-dimensional representations z 1 and z 2 for images v 1 and tags v 2 respectively.
Revised: Each encoder consists of a multi-layer perceptron of 4 hidden layers with ReLU activations learning two 1024-dimensional representations z 1 and z 2 for images v 1 and tags v 2 respectively.

Original: Examples of the two views, labels, and further details on the training procedure are in Appendix F.
Revised: Examples of the two views, labels, and further details on the training procedure are in Appendix G.

Original: Our MIB model is compared with other popular Multi-View learning models in Figure 3 for β =(Multi-View InfoMax), β = 1 and β = 10 − 3 (where β = 10 − 3 was the result of tuning β onour validation set).
Revised: Our MIB model is compared with other popular multi-view learning models in Figure 3 for β =(Multi-View InfoMax), β = 1 and β = 10 − 3 (best on validation set).

Original: Although the tuned MIB performs similarly to Multi-View InfoMax with a large number of labels, it outperforms it when fewer labels are available.
Revised: Although the tuned MIB performs similarly to Multi-View InfoMax with a large number of labels, it outperforms it when fewer labels are available.

Original: Furthermore, by choosing a larger β the accuracy of our model drastically increases in scarce label regimes (see right side of Figure 3). A possible reason for the effectiveness of MIB against some of the other baselines may be our ability to use mutual information estimators that do not require reconstruction. Both Multi-View
Revised: Furthermore, by choosing a larger β the accuracy of our model drastically increases in scarce label regimes, while slightly reducing the accuracy when all the labels are observed

Original: VAE (MVAE) and Deep Variational CCA (VCCA) rely on a reconstruction term to capture crossmodal information, which can introduce bias that decreases performance.

Original: 14 I ( x ; z ) 0.
Revised: 14 I ( x ; z ) 0.

Original: I ( z ; y ) =
Revised: I ( z ; y ) =

Original: 2 4 =2 1 [-=2=2=
 [Figure] [Figure]-] Examplesper Label 0.
Revised: 2 4 =2 1 [-=2=2=
 [Figure] [Figure]-]=2=2=21010101010 5 Number of Labeled Examples 0.

Original: A cc u r [Figure] c y
Revised: A cc u r a c y

Original: Model VAE ( =0)
Revised: Model VAE ( =0)

Original: VAE ( =4)
Revised: VAE ( =4)

Original: VAE ( =8)
Revised: VAE ( =8)

Original: InfoMax MV-InfoMax
Revised: InfoMax MV-InfoMax

Original: MIB ( =1)
Revised: MIB ( =1)

Revised: (see right side of Figure 3).

Revised: This effect is likely due to a violation of the mutual redundancy constraint (see Figure 6 in the supplementary material) which can be compensated with smaller values of β for less aggressive compression.

Revised: A possible reason for the effectiveness of MIB against some of the other baselines may be our ability to use mutual information estimators that do not require reconstruction.

Revised: Both Multi-View

Revised: VAE (MVAE) and Deep Variational CCA (VCCA) rely on a reconstruction term to capture crossmodal information, which can introduce bias that decreases performance.

Original: 5. S ELF - SUPERVISED S INGLE -V IEW T ASK
Revised: 5. S ELF - SUPERVISED S INGLE -V IEW T ASK

Original: In this section, we compare the performance of different unsupervised learning models by measuring their data efﬁciency and empirically estimating the coordinates of their representation on the Information Plane.
Revised: In this section, we compare the performance of different unsupervised learning models by measuring their data efﬁciency and empirically estimating the coordinates of their representation on the Information Plane.

Original: Since accurate estimation of mutual information is extremely expensive (McAllester & Stratos, 2018), we focus on relatively small experiments that aim to uncover the difference between popular approaches for representation learning.
Revised: Since accurate estimation of mutual information is extremely expensive (McAllester & Stratos, 2018), we focus on relatively small experiments that aim to uncover the difference between popular approaches for representation learning.

Original: The dataset is generated from MNIST by creating the two views, v 1 and v 2 , via the application of data augmentation consisting of small afﬁne transformations and independently pixel corruption to each image.
Revised: The dataset is generated from MNIST by creating the two views, v 1 and v 2 , via the application of data augmentation consisting of small afﬁne transformations and independent pixel corruption to each image.

Original: These are kept small enough to ensure that label information is not effected.
Revised: These are kept small enough to ensure that label information is not effected.

Original: Each pair of views is generated from the same underlying image, so no label information is used in this process.
Revised: Each pair of views is generated from the same underlying image, so no label information is used in this process (details in Appendix G).

Original: Further details on the data augmentation are in Appendix F. To evaluate we model, we train the representation model on the unlabeled multi-view dataset just described, and then ﬁx the representation model.
Revised: To evaluate, we train the encoders using the unlabeled multi-view dataset just described, and then ﬁx the representation model.

Original: A logistic regression model is trained using these representations along with some set of labels for the training set, and we report the accuracy of this model on a disjoint test set as is standard for the multi-view literature(Tschannen et al., 2019).
Revised: A logistic regression model is trained using the resulting representations along with a subset of labels for the training set, and we report the accuracy of this model on a disjoint test set as is standard for the unsupervised representation learning literature (Tschannen et al., 2019; Tian et al., 2019; van den Oord et al., 2018).

Original: We estimate I ( x ; z ) and I ( y ; z ) using mutual information estimation networks trained from scratch on the ﬁnal representations using batches of joint samples { ( x ( i ) , y ( i ) , z ( i ) ) }
Revised: We estimate I ( x ; z ) and I ( y ; z ) using mutual information estimation networks trained from scratch on the ﬁnal representations using batches of joint samples { ( x ( i ) , y ( i ) , z ( i ) ) }

Original: Bi =1 ∼ p ( x , y ) p
Revised: Bi =1 ∼ p ( x , y ) p

Original: θ ( z | x ) .
Revised: θ ( z | x ) .

Original: All models are trained using the same encoder architecture consisting of 2 layers of 1024 hidden units with ReLU activations, resulting in 64-dimensional representations.
Revised: All models are trained using the same encoder architecture consisting of 2 layers of 1024 hidden units with ReLU activations, resulting in 64-dimensional representations.

Original: The  data augmentation procedure was also applied for single-view architectures and models were trained for 1 million iterations with batch size B = 64 .
Revised: The same data augmentation procedure was also applied for single-view architectures and models were trained for 1 million iterations with batch size B = 64 .

Original: Figure 4 summarizes the results.
Revised: Figure 4 summarizes the results.

Original: The empirical measurements of mutual information reported on the Information Plane are consistent with the theoretical analysis reported in Sec. 4: models that retain less information about the data while maintaining the maximal amount of predictive information, result in better classiﬁcation performance at low-label regimes, conﬁrming the hypothesis that discarding irrelevant information yields robustness and more data-efﬁcient representations.
Revised: The empirical measurements of mutual information reported on the Information Plane are consistent with the theoretical analysis reported in Section 4: models that retain less information about the data while maintaining the maximal amount of predictive information, result in better classiﬁcation performance at low-label regimes, conﬁrming the hypothesis that discarding irrelevant information yields robustness and more data-efﬁcient representations.

Original: Notably, the MIB model  retains almost exclusively label information, hardly decreasing the classiﬁcation performance when only one label is used for each data point.
Revised: Notably, the MIB model with β = 1 retains almost exclusively label information, hardly decreasing the classiﬁcation performance when only one label is used for each data point.

Original:  C ONCLUSIONS AND F UTURE WORK
Revised:  C ONCLUSIONS AND F UTURE WORK

Original: In this work, we introduce  a novel method that relies on multiple data-views to produce robust representation for downstream task.
Revised: In this work, we introduce Multi-View Information Bottleneck, a novel method that relies on multiple data-views to produce robust representation for downstream tasks.

Revised: Most of the multi-view literature operates under the assumption that each view is individually sufﬁcient for determining the label (Zhao et al., 2017), while our method only requires the weaker mutual redundancy condition outlined in Section 3, enabling it to be applied to any traditional multi-view task.

Original: Our model is compared  empirically against other approaches in the literature of sketch-based image retrieval, Multi-View and unsupervised  learning.
Revised: In our experiments, we compared MIB empirically against other approaches in the literature on three such tasks: sketch-based image retrieval, multi-view and unsupervised representation learning.

Original: The strong performances obtained in the different areas show that Multi-View Information Bottleneck can be practically applied to various tasks for which the paired observations are either available or are artiﬁcially produced.
Revised: The strong performance obtained in the different areas show that Multi-View Information Bottleneck can be practically applied to various tasks for which the paired observations are either available or are artiﬁcially produced.

Original: Future work will address optimization strategy, ﬂexibility of the encoding distribution and the role of different choices for data augmentation.
Revised: Furthermore, the positive results on the MIR-Flickr dataset show that our model can work well in practice even when mutual redundancy holds only approximately. There are multiple extensions that we would like to explore in future work. One interesting direction would be considering more than two views. In Appendix D we discuss why the mutual redundancy condition cannot be trivially extended to more than two views, but we still believe such an extension is possible.

Original: We believe that this direction of exploration would be able to bridge the  Information Bottleneck principle and  invariant neural networks, which are able to exploit known symmetries and structure of the data to remove superﬂuous information.
Revised: Secondly, we believe that exploring the role played by different choices of data augmentation could bridge the gap between the Information Bottleneck principle and with the literature on invariant neural networks (Bloem-Reddy & Whye Teh, 2019), which are able to exploit known symmetries and structure of the data to remove superﬂuous information.

Original: A P ROPERTIES OF M UTUAL I NFORMATION AND E NTROPY
Revised: A P ROPERTIES OF M UTUAL I NFORMATION AND E NTROPY

Original: In this section we enumerate some of the properties of mutual information that are used to prove the theorems reported in this work.
Revised: In this section we enumerate some of the properties of mutual information that are used to prove the theorems reported in this work.

Original: For any random variables w , x , y and z :
Revised: For any random variables w , x , y and z :

Original: [Equation]
Revised: [Equation]

Original: Positivity
Revised: Positivity

Original: [Equation]
Revised: [Equation]

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: B T HEOREMS AND P ROOFS B. O N S UFFICIENCY
Revised: B T HEOREMS AND P ROOFS B. O N S UFFICIENCY

Original: Proposition B.1.
Revised: Proposition B.1.

Original: Let x
Revised: Let x

Original: and y be random variables with joint distribution p ( x , y ) .
Revised: and y be random variables with joint distribution p ( x , y ) .

Original: Let z be a representation of x , then z is sufﬁcient for y if and only
Revised: Let z be a representation of x , then z is sufﬁcient for y if and only

Original: if I ( x ; y )
Revised: if I ( x ; y )

Original: =
Revised: =

Original: I ( y ; z )
Revised: I ( y ; z )

Original: Hypothesis:
Revised: Hypothesis:

Original: [Equation]
Revised: [Equation]

Original: Thesis:
Revised: Thesis:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Since both I ( x ; y ) and I ( y ; z ) are non-negative ( P 1 ) , I ( x ; y | z ) = 

Revised: Since both I ( x ; y ) and I ( y ; z ) are non-negative ( P 1 ) , I ( x ; y | z ) =

Original: ⇐⇒ I
Revised: ⇐⇒ I

Original: ( y ; z )
Revised: ( y ; z )

Original: =
Revised: =

Original: I ( x ; y )
Revised: I ( x ; y )

Original: B.
Revised: B.

Original: N O F
Revised: N O F

Original: REE
Revised: REE

Original: G
Revised: G

Original: ENERALIZATION
Revised: ENERALIZATION

Original: Theorem B.1.
Revised: Theorem B.1.

Original: Let x , z
Revised: Let x , z

Original: and y be random variables with joint distribution p ( x , y , z ) .
Revised: and y be random variables with joint distribution p ( x , y , z ) .

Original: Let z (cid:48) be a representation of x that satisﬁes
Revised: Let z (cid:48) be a representation of x that satisﬁes

Original: I ( x ; z ) > I ( x ; z (cid:48) ) , then it is always possible to ﬁnd a label y for which z (cid:48) is not predictive for y while z is.
Revised: I ( x ; z ) > I ( x ; z (cid:48) ) , then it is always possible to ﬁnd a label y for which z (cid:48) is not predictive for y while z is.

Original: Hypothesis:
Revised: Hypothesis:

Original: [Equation]
Revised: [Equation]

Original: Thesis:
Revised: Thesis:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By construction.
Revised: By construction.

Original: We ﬁrst factorize x as a function of two independent random variables (Proposition 2.
Revised: We ﬁrst factorize x as a function of two independent random variables (Proposition 2.

Original: Achille & Soatto (2018))
Revised: Achille & Soatto (2018))

Original: by picking y such that
Revised: by picking y such that

Original: : ( C 1 ) I ( y ; z (cid:48) ) =( C 2 ) x
Revised: : ( C 1 ) I ( y ; z (cid:48) ) =( C 2 ) x

Original: = f ( z (cid:48) , y ) for some deterministic function f .
Revised: = f ( z (cid:48) , y ) for some deterministic function f .

Original: Note that such y always exists.
Revised: Note that such y always exists.

Original: Since x is a function of y and z (cid:48) :
Revised: Since x is a function of y and z (cid:48) :

Original: [Equation]
Revised: [Equation]

Original: ( x ; z | yz
Revised: ( x ; z | yz

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: Considering I ( y ; z ) :
Revised: Considering I ( y ; z ) :

Original: I ( y ; z ) ( P 3 ) =
Revised: I ( y ; z ) ( P 3 ) =

Original: I ( y ; z | x )
Revised: I ( y ; z | x )

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ; y ; z )
Revised: ; y ; z )

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: z (cid:48) )
Revised: z (cid:48) )

Original: [Equation]
Revised: [Equation]

Original: z (cid:48) )
Revised: z (cid:48) )

Original: [Equation]
Revised: [Equation]

Original: ( y
Revised: ( y

Original: [Equation]
Revised: [Equation]

Original: z (cid:48)
Revised: z (cid:48)

Original: [Equation]
Revised: [Equation]

Original: x ) ( P 1 ) ≥ I ( x
Revised: x ) ( P 1 ) ≥ I ( x

Original: [Equation]
Revised: [Equation]

Original: z ) − I ( x
Revised: z ) − I ( x

Original: [Equation]
Revised: [Equation]

Original: Whenever I ( x ; z ) >
Revised: Whenever I ( x ; z ) >

Original: I ( x ; z (cid:48) ) , I ( y ; z ) must be strictly positive, while I ( y ; z (cid:48) )
Revised: I ( x ; z (cid:48) ) , I ( y ; z ) must be strictly positive, while I ( y ; z (cid:48) )

Original: = 0 by construction.
Revised: = 0 by construction.

Original: Therefore such y exists.
Revised: Therefore such y exists.

Original: Corollary B.1.1.
Revised: Corollary B.1.1.

Original: Let z (cid:48) be a representation of x that discards observational information.
Revised: Let z (cid:48) be a representation of x that discards observational information.

Original: There is always a label y for which a z (cid:48) is not predictive, while the original observations are.
Revised: There is always a label y for which a z (cid:48) is not predictive, while the original observations are.

Original: Hypothesis:
Revised: Hypothesis:

Original: ( H 1 ) x is discrete (empirical distribution)
Revised: ( H 1 ) x is discrete (empirical distribution)

Original: ( H 2 ) I ( z (cid:48) ; x ) < H ( x )
Revised: ( H 2 ) I ( z (cid:48) ; x ) < H ( x )

Original: Thesis:
Revised: Thesis:

Original: ( T 1 ) ∃ y .I
Revised: ( T 1 ) ∃ y .I

Original: ( y ; x ) >
Revised: ( y ; x ) >

Original: I ( y ; z (cid:48) ) =
Revised: I ( y ; z (cid:48) ) =

Original: Proof.
Revised: Proof.

Original: By construction using Theorem B.1.
Revised: By construction using Theorem B.1.

Original: Set z
Revised: Set z

Original: = x : ( C 1 ) I ( x ; z ) ( P 5 )
Revised: = x : ( C 1 ) I ( x ; z ) ( P 5 )

Original: =
Revised: =

Original: H ( x ) − H ( x | z ) ( H 1 ) =
Revised: H ( x ) − H ( x | z ) ( H 1 ) =

Original: H ( x )
Revised: H ( x )

Original: [Equation]
Revised: [Equation]

Original: Since the hypothesis are met
Revised: Since the hypothesis are met

Original: [Equation]
Revised: [Equation]

Original: we conclude that there exist y
Revised: we conclude that there exist y

Original: such that
Revised: such that

Original: I ( y ; x ) > I ( y ; z (cid:48) )
Revised: I ( y ; x ) > I ( y ; z (cid:48) )

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: B. M ULTI -V IEW B.3. M ULTI -V IEW R EDUNDANCY AND S UFFICIENCY
Revised: B. M ULTI -V IEW B.3. M ULTI -V IEW R EDUNDANCY AND S UFFICIENCY

Original: Proposition B.2.
Revised: Proposition B.2.

Original: Let v 1 , v 2 , y be random variables with joint distribution p ( v 1 , v 2 , y ) .
Revised: Let v 1 , v 2 , y be random variables with joint distribution p ( v 1 , v 2 , y ) .

Original: Let z 1 be a representation of v 1 , then:
Revised: Let z 1 be a representation of v 1 , then:

Original: [Equation]
Revised: [Equation]

Original: Hypothesis:
Revised: Hypothesis:

Original: [Equation]
Revised: [Equation]

Original: Thesis:
Revised: Thesis:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Since z 1 is a representation of v 1 :
Revised: Since z 1 is a representation of v 1 :

Original: [Equation]
Revised: [Equation]

Original: Therefore:
Revised: Therefore:

Original: [Equation]
Revised: [Equation]

Original: Proposition B.3.
Revised: Proposition B.3.

Original: Let v 1 be a redundant view with respect to v 2 for y .
Revised: Let v 1 be a redundant view with respect to v 2 for y .

Original: Any representation z 1 of vthat is sufﬁcient for v 2 is also sufﬁcient for y .
Revised: Any representation z 1 of vthat is sufﬁcient for v 2 is also sufﬁcient for y .

Original: Hypothesis:
Revised: Hypothesis:

Original: [Equation]
Revised: [Equation]

Original: Thesis:
Revised: Thesis:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Using the results from Theorem B.2:
Revised: Using the results from Theorem B.2:

Original: [Equation]
Revised: [Equation]

Original: Therefore I ( v 1 ; v 2 | z 1 ) = 

Revised: Therefore I ( v 1 ; v 2 | z 1 ) =

Original: = ⇒
Revised: = ⇒

Original: I ( v 1 ; y | z 1 ) =
Revised: I ( v 1 ; y | z 1 ) =

Original: Theorem B
Revised: Theorem B

Original: [Equation]
Revised: [Equation]

Original: Let v 1 , v 2 and y be random variables with distribution p ( v 1 , v 2 , y ) .
Revised: Let v 1 , v 2 and y be random variables with distribution p ( v 1 , v 2 , y ) .

Original: Let z be a representation of v 1 , then
Revised: Let z be a representation of v 1 , then

Original: [Equation]
Revised: [Equation]

Original: Hypothesis:
Revised: Hypothesis:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Corollary B.2.1.
Revised: Corollary B.2.1.

Original: Let v 1 and v 2 be mutually redundant views for y .
Revised: Let v 1 and v 2 be mutually redundant views for y .

Original: Let z 1 be a representation of v 1 that is sufﬁcient for v 2 .
Revised: Let z 1 be a representation of v 1 that is sufﬁcient for v 2 .

Original: Then:
Revised: Then:

Original: [Equation]
Revised: [Equation]

Original: Hypothesis:
Revised: Hypothesis:

Original: [Equation]
Revised: [Equation]

Original: Thesis:
Revised: Thesis:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Using Theorem B.
Revised: Using Theorem B.

Original: [Equation]
Revised: [Equation]

Original: Since I ( y ; z 1 ) ≤ I ( y ; v 1 v 2 ) is a consequence of the data processing inequality, we conclude that
Revised: Since I ( y ; z 1 ) ≤ I ( y ; v 1 v 2 ) is a consequence of the data processing inequality, we conclude that

Original: I ( y ; z 1 ) =
Revised: I ( y ; z 1 ) =

Original: I ( y ; v 1 v 2 )
Revised: I ( y ; v 1 v 2 )

Original: B. S UFFICIENCY AND A UGMENTATION
Revised: B. S UFFICIENCY AND A UGMENTATION

Original: Let x
Revised: Let x

Original: and y be random variables with domain X and Y respectively.
Revised: and y be random variables with domain X and Y respectively.

Original: Let T be a class of functions t : X → W and let t 1 and t 2 be a random variables over T that depends only on x .
Revised: Let T be a class of functions t : X → W and let t 1 and t 2 be a random variables over T that depends only on x .

Original: For the theorems and corollaries discussed in this section, we are going to consider the independence assumption that can be derived from the graphical model G reported in Figure 5.
Revised: For the theorems and corollaries discussed in this section, we are going to consider the independence assumption that can be derived from the graphical model G reported in Figure 5.

Original: [Figure]
Revised: [Figure]

Original: Proposition B.4.
Revised: Proposition B.4.

Original: Whenever I ( t 1 ( x ); y ) =
Revised: Whenever I ( t 1 ( x ); y ) =

Original: I ( t 2 ( x ); y )
Revised: I ( t 2 ( x ); y )

Original: =
Revised: =

Original: I ( x ; y ) the two views t 1 ( x ) and t 2 ( x ) must be mutually redundant for y .
Revised: I ( x ; y ) the two views t 1 ( x ) and t 2 ( x ) must be mutually redundant for y .

Original: Hypothesis: ( H 1 ) Independence relations determined by G
Revised: Hypothesis: ( H 1 ) Independence relations determined by G

Original: Thesis:
Revised: Thesis:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Considering G we have:
Revised: Considering G we have:

Original: [Equation]
Revised: [Equation]

Original: Since t 2 ( x ) is uniquely determined by x and t 2 :
Revised: Since t 2 ( x ) is uniquely determined by x and t 2 :

Original: [Equation]
Revised: [Equation]

Original: Consider I ( t 1 ( x );
Revised: Consider I ( t 1 ( x );

Original: y | t 2 ( x ))
Revised: y | t 2 ( x ))

Original: I ( t 1 ( x ); y | t 2 ( x ))
Revised: I ( t 1 ( x ); y | t 2 ( x ))

Original: ( P 3 )
Revised: ( P 3 )

Original: =
Revised: =

Original: I ( t 1 ( x ); y | xt 2 ( x ))
Revised: I ( t 1 ( x ); y | xt 2 ( x ))

Original: + I ( t 1 ( x );
Revised: + I ( t 1 ( x );

Original: y ; x
Revised: y ; x

Original: | t 2 ( x ))
Revised: | t 2 ( x ))

Original: ( C 1 )
Revised: ( C 1 )

Original: =
Revised: =

Original: I ( t 1 ( x ); y ; x | t 2 ( x ))
Revised: I ( t 1 ( x ); y ; x | t 2 ( x ))

Original: ( P 3 )
Revised: ( P 3 )

Original: =
Revised: =

Original: I ( y ; x | t 2 ( x )) − I ( y ; x | t 1 ( x ) t 2 ( x ))

Original: ( P 1 ) ≤ I ( y ; x | t 2 ( x )) ( P 3 ) = I ( y ; x  ) 
Revised: I ( y ; x | t 2 ( x )) − I ( y ; x | t 1 ( x ) t 2 ( x ))

Original: −

Original: I ( y ;

Original: x

Original: ; t 2 ( x )) (

Original: P = I ( y x
Revised: ( P

Original: −

Original: I ( y ;

Original: t

Original: 2 ( x ))

Original: +

Original:  I ( y ;t 2 ( x) |  x ) (
Revised: ) ≤ I ( y ; x | t x ))

Original: P

Original: = I y ;

Original: x )

Original: − I y ; t

Original: 2 ( x ))

Original: + I y ;

Original: t 2 ( x ) | t 2 x )
Revised: ( P 3 )

Original: +

Original: I ( y

Original: ; t

Original: 2 ( x )

Original: t 2 | x ) (

Original: C

Original: =
Revised: =

Original: I (

Original: y ;

Original:  x ) − I  y ;  t 2 ( x ))+ I y ;
Revised: I ( y ; x ) − I ( y ; x ; t 2 ( x ))

Original: t

Original: (x ; t | x ( P 
Revised: ( P 3 )

Original: =
Revised: =

Original: I ( y ; x ) − I ( y ; t 2 ( x ))
Revised: I ( y ; x ) − I ( y ; t 2 ( x ))

Original: + I ( y ; t 2 ( x ) | x )  − I ( y ; t 2 ( x  ) | t  x ) 
Revised: + I ( y ; t 2 ( x ) | x ) ( P 3 ) = I ( y ; x ) − I ( y ; t 2 ( x )) + I ( y ; t 2 ( x ) | t 2 x ) + I ( y ; t 2 ( x ); t 2 | x ) ( C 3 )

Original: P 1 ) ≥ I ( y ; x ) − I ( y ; t  x )) 
Revised: = I ( y ; x ) − I ( y ; t 2 ( x )) + I ( y ; t 2 ( x ); t 2 | x ) ( P 3 )

Original:  + I ( y ; t  x ) |  x ) 
Revised: = I ( y ; x ) − I ( y ; t 2 ( x )) + I ( y ; t 2 ( x ) | x ) − I ( y ; t 2 ( x ) | t 2 x ) ( P 1 ) ≥ I ( y ; x ) − I ( y ; t 2 ( x ))

Original:  C 2 ) ≥ I ( y ; x ) − I ( y ; t 2 ( x 
Revised: + I ( y ; t 2 ( x ) | x ) ( C 2 ) ≥ I ( y ; x ) − I ( y ; t 2 ( x ))

Original: Therefore I ( y ; x ) =
Revised: Therefore I ( y ; x ) =

Original: I ( y ; t 2 ( x )) =
Revised: I ( y ; t 2 ( x )) =

Original: ⇒ I ( t 1 ( x )
Revised: ⇒ I ( t 1 ( x )

Original: ; y | t 2 ( x )) =
Revised: ; y | t 2 ( x )) =

Original: The proof for I ( y ; x ) =
Revised: The proof for I ( y ; x ) =

Original: I ( y ; t 1 ( x )) =
Revised: I ( y ; t 1 ( x )) =

Original: ⇒ I ( t 2 ( x ); y | t 1 ( x ))
Revised: ⇒ I ( t 2 ( x ); y | t 1 ( x ))

Original: = 0 is symmetric, therefore we conclude I ( t 1 ( x ); y )
Revised: = 0 is symmetric, therefore we conclude I ( t 1 ( x ); y )

Original: =
Revised: =

Original: I ( t 2 ( x ); y )
Revised: I ( t 2 ( x ); y )

Original: =
Revised: =

Original: I ( x ; y ) =
Revised: I ( x ; y ) =

Original: ⇒ I ( t 1 ( x )
Revised: ⇒ I ( t 1 ( x )

Original: ; y | t 2 ( x ))
Revised: ; y | t 2 ( x ))

Original: + I ( t 2 ( x ); y | t 1 ( x )) =
Revised: + I ( t 2 ( x ); y | t 1 ( x )) =

Original: Theorem B.3.
Revised: Theorem B.3.

Original: Let I ( t 1 ( x ); y )
Revised: Let I ( t 1 ( x ); y )

Original: =
Revised: =

Original: I ( t 2 ( x ); y )
Revised: I ( t 2 ( x ); y )

Original: =
Revised: =

Original: I ( x ; y ) .
Revised: I ( x ; y ) .

Original: Let z 1 be a representation of t 1 ( x ) .
Revised: Let z 1 be a representation of t 1 ( x ) .

Original: If z 1 is sufﬁcient for t 2 ( x ) then I ( x ; y )
Revised: If z 1 is sufﬁcient for t 2 ( x ) then I ( x ; y )

Original: =
Revised: =

Original: I ( y ; z 1 ) .
Revised: I ( y ; z 1 ) .

Original: Hypothesis:
Revised: Hypothesis:

Original: ( H 1 ) Independence relations determined by G ( H 2 )
Revised: ( H 1 ) Independence relations determined by G ( H 2 )

Original: I ( t 1 ( x ); y )
Revised: I ( t 1 ( x ); y )

Original: =
Revised: =

Original: I ( t 2 ( x ); y )
Revised: I ( t 2 ( x ); y )

Original: =
Revised: =

Original: I ( x ; y )
Revised: I ( x ; y )

Original: Thesis: 
Revised: Thesis: ( T 1 )

Original: ( T 1 ) I ( t 1 ( x ); t 2 ( x ) | z 1 )
Revised: I ( t 1 ( x ); t 2 ( x ) | z 1 )

Original: = 

Revised: =

Original: = ⇒
Revised: = ⇒

Original: I ( x ; y )
Revised: I ( x ; y )

Original: =
Revised: =

Original: I ( y ; z 1 )
Revised: I ( y ; z 1 )

Original: Proof.
Revised: Proof.

Original: Since t 1 ( x ) is redundant for t 2 ( x ) (Proposition B.4)
Revised: Since t 1 ( x ) is redundant for t 2 ( x ) (Proposition B.4)

Original: any representation z 1 of t 1 ( x ) that is sufﬁcient for t 2 ( x ) must also be sufﬁcient for y (Theorem B.2).
Revised: any representation z 1 of t 1 ( x ) that is sufﬁcient for t 2 ( x ) must also be sufﬁcient for y (Theorem B.2).

Original: Using Proposition B.1 we have
Revised: Using Proposition B.1 we have

Original: I ( y ; z 1 ) =
Revised: I ( y ; z 1 ) =

Original: I ( y ; t 1 ( x )) .
Revised: I ( y ; t 1 ( x )) .

Original: Since I ( y ; t 1 ( x )) =
Revised: Since I ( y ; t 1 ( x )) =

Original: I ( y ; x ) by hypothesis, we conclude I ( x ; y ) =
Revised: I ( y ; x ) by hypothesis, we conclude I ( x ; y ) =

Original: I ( y ; z 1 )
Revised: I ( y ; z 1 )

Revised: C I NFORMATION P LANE

Revised: Every representation z of x must satisfy the following constraints:

Revised: • 0 ≤ I ( y ; z ) ≤ I ( x ; y ) :

Revised: The amount of label information ranges from 0 to the total predictive information accessible from the raw observations I ( x ; y ) .

Revised: • I ( y ; z )

Original: C E QUIVALENCES OF DIFFERENT OBJECTIVES
Revised: ≤ I ( x ; z ) ≤ I ( y ; z )+ H ( x | y ) : The representation must contain more information about the observations than about the label. When x is discrete, the amount of discarded label information I ( x ; y ) − I ( y ; z ) must be smaller than the amount of discarded observational information H ( x ) − I ( x ; z ) , which implies I ( x ; z ) ≤ I ( y ; z ) + H ( x | y ) .

Revised: Proof.

Revised: Since z is a representation of x :

Revised: [Equation]

Revised: Considering the four bounds separately:

Revised: I ( y ; z ) ≥ 0 : Follows from P2.

Revised: I ( x ; z ) ≥ I ( y ; z ) : Follows from ( T 1 ) in Lemma B. 3.

Revised: I ( y ; z ) ≤

Revised: I ( y ; x ) : Data processing inequality

Revised: [Equation]

Revised: Note that ( H 2 ) is needed only to prove bound 4 .

Revised: For continuous x bounds 1 , 2 and 3 still hold.

Revised: D N ON - TRANSITIVITY OF M UTUAL R EDUNDANCY

Revised: The mutual redundancy condition between two views v 1 and v 2 for a label y can not be trivially extended to an arbitrary number of views, as the relation is not transitive because of some higher order interaction between the different views and the label.

Revised: This can be shown with a simple example.

Revised: Given three views v 1 , v 2 and v 3 and a task y such that:

Revised: • v 1 and v 2 are mutually redundant for y • v 2 and v 3 are mutually redundant for y

Revised: Then, v 1 is not necessarily mutually redundant with respect to v 3 for y .

Revised: We can show this with a simple example, Let v 1 , v 2 and v 3 be fair and independent binary random variables.

Revised: Deﬁning y as the exclusive or of v 1 and v 3 ( y := v 1 XOR v 3 ), we have that I ( v 1 ; y ) =

Revised: I ( v 3 ; y ) = 0 .

Revised: In this settings, v 1 and v 2 are mutually redundant for y :

Revised: [Equation]

Revised: Analogously, v 2 and v 3 are also mutually redundant for y as the three random variables are not predictive for each other.

Revised: Nevertheless, v 1 and v 3 and not mutually redundant for y :

Revised: [Equation]

Revised: Where H ( v 1 | v 3 y )

Revised: =

Revised: H ( v 3 | v 1 y ) = 0 follows from v 1 = v 3 XOR y and v 3 = v 1 XOR y , while

Revised: H ( v 1 ) =

Revised: H ( v 3 )

Revised: = 1 holds by construction.

Revised: This counter-intuitive higher order interaction between multiple views makes our theory non-trivial to generalize to more than two views, requiring an extension of our theory to ensure sufﬁciency for the label.

Revised: E E QUIVALENCES OF DIFFERENT OBJECTIVES

Original: Different objectives in literature can be seen as a special case of the Multi-View Information Bottleneck principle.
Revised: Different objectives in literature can be seen as a special case of the Multi-View Information Bottleneck principle.

Original: In this section we show that the supervised version of Information Bottleneck is equivalent to the corresponding Multi-View version whenever the two redundant views have only label information in common.
Revised: In this section we show that the supervised version of Information Bottleneck is equivalent to the corresponding Multi-View version whenever the two redundant views have only label information in common.

Original: A second subsection show equivalence between InfoMax and MultiView Information Bottleneck whenever the two views are identical.
Revised: A second subsection show equivalence between InfoMax and MultiView Information Bottleneck whenever the two views are identical.

Original: C. M ULTI -V IEW I NFORMATION B OTTLENECK AND S UPERVISED I NFORMATION B OTTLENECK
Revised: E. M ULTI -V IEW I NFORMATION B OTTLENECK AND S UPERVISED I NFORMATION B OTTLENECK

Original: Whenever the two mutually redundant views v 1 and v 2 have only label information in common (or when one of the two views is the label itself) the Multi-View Information Bottleneck objective is equivalent to the respective supervised version.
Revised: Whenever the two mutually redundant views v 1 and v 2 have only label information in common (or when one of the two views is the label itself) the Multi-View Information Bottleneck objective is equivalent to the respective supervised version.

Original: This can be shown by proving that I ( v 1 ; z 1 | v 2 ) =
Revised: This can be shown by proving that I ( v 1 ; z 1 | v 2 ) =

Original: I ( v 1 ; z 1 | y ) , i.e. a representation z 1 of v 1 that is sufﬁcient and minimal for v 2 is also sufﬁcient and minimal for y .
Revised: I ( v 1 ; z 1 | y ) , i.e. a representation z 1 of v 1 that is sufﬁcient and minimal for v 2 is also sufﬁcient and minimal for y .

Original: Proposition C.1.
Revised: Proposition E.1.

Original: Let v 1 and v 2 be mutually redundant views for a label y that share only label information.
Revised: Let v 1 and v 2 be mutually redundant views for a label y that share only label information.

Original: Then a sufﬁcient representation z 1 of v 1 for v 2 that is minimal for v 2 is also a minimal representation for y .
Revised: Then a sufﬁcient representation z 1 of v 1 for v 2 that is minimal for v 2 is also a minimal representation for y .

Original: Hypothesis:
Revised: Hypothesis:

Original: [Equation]
Revised: [Equation]

Original: Thesis:
Revised: Thesis:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Consider I ( v 1 ; z ) :
Revised: Consider I ( v 1 ; z ) :

Original: [Equation]
Revised: [Equation]

Original: Using Corollary 1, from ( H 2 ) and ( H 3 ) follows I ( v 1 ; y | z 1 ) =3.
Revised: Using Corollary 1, from ( H 2 ) and ( H 3 ) follows I ( v 1 ; y | z 1 ) =

Original: I ( v 1 ; z ) can be alternatively expressed as:I ( v 1 ; z 1 ) ( P 3 ) = I ( v 1 ; z 1 | y ) + I ( v 1 ; y ; z 1 ) ( P 3 )
Revised: I ( v 1 ; z ) can be alternatively expressed as:

Original: =

Original: I ( v 1 ; z 1 | y )

Original: + I ( v 1 ; y ) − I ( v 1 ; y | z 1 ) ( Cor 1 ) = I ( v 1 ; z 1 | y ) + I ( v 1 ; y )
Revised: [Equation]

Original: Equating 1 and 3, we conclude I ( v 1 ; z 1 | v 2 )
Revised: Equating 1 and 3, we conclude I ( v 1 ; z 1 | v 2 )

Original: =
Revised: =

Original: I ( v 1 ; z 1 | y ) .
Revised: I ( v 1 ; z 1 | y ) .

Original: C. M ULTI -V IEW I NFORMATION B OTTLENECK AND I NFOMAX
Revised: E. M ULTI -V IEW I NFORMATION B OTTLENECK AND I NFOMAX

Original: Whenever v 1 = v 2 , a representation z 1 of v 1 that is sufﬁcient for v 1 must contain all the original information.
Revised: Whenever v 1 = v 2 , a representation z 1 of v 1 that is sufﬁcient for v 1 must contain all the original information.

Original: Furthermore since I ( v 1 ; z 1 | v 1 ) = 

Revised: Furthermore since I ( v 1 ; z 1 | v 1 ) =

Original: for every representation, no superﬂuous information can be identiﬁed and removed.
Revised: for every representation, no superﬂuous information can be identiﬁed and removed.

Original: As a consequence, a minimal sufﬁcient representation z 1 of vfor v 1 is any representation for which mutual information is maximal, hence InfoMax.
Revised: As a consequence, a minimal sufﬁcient representation z 1 of vfor v 1 is any representation for which mutual information is maximal, hence InfoMax.

Original: D L OSS C OMPUTATION
Revised: F L OSS C OMPUTATION

Original: Starting from Equation [Equation] we consider the sum of the losses L 1 ( θ ; λ 1 ) and L 2 ( ψ ; λ 2 ) that aim to create the minimal sufﬁcient representations z 1 and z 2 respectively:
Revised: Starting from Equation 3, we consider the sum of the losses L 1 ( θ ; λ 1 ) and L 2 ( ψ ; λ 2 ) that aim to create the minimal sufﬁcient representations z 1 and z 2 respectively:

Original: [Equation]
Revised: [Equation]

Original: Considering z 1 and z 2 on the same domain Z [Equation] I θ ( v 1 ; z 1 | v 2 ) can be expressed as:
Revised: Considering z 1 and z 2 on the same domain Z , I θ ( v 1 ; z 1 | v 2 ) can be expressed as:

Original: [Equation]
Revised: [Equation]

Original: Note that the bound is tight whenever p ψ ( z 2 | v 2 ) coincides with p θ ( z 1 | v 2 ) .
Revised: Note that the bound is tight whenever p ψ ( z 2 | v 2 ) coincides with p θ ( z 1 | v 2 ) .

Original: This happens whenever z 1 and z 2 produce a consistent encoding.
Revised: This happens whenever z 1 and z 2 produce a consistent encoding.

Original: Analogously I ψ ( v 2 ; z 2 | v 1 ) is upper bounded by
Revised: Analogously I ψ ( v 2 ; z 2 | v 1 ) is upper bounded by

Original: D KL ( p ψ ( z 2 | v 2 )
Revised: D KL ( p ψ ( z 2 | v 2 )

Original: ||
Revised: ||

Original: p θ ( z 1 | v 1 )) .
Revised: p θ ( z 1 | v 1 )) .

Original: I θ ( v 1 ; v 2 | z 1 ) can be rephrased as:
Revised: I θ ( v 1 ; v 2 | z 1 ) can be rephrased as:

Original: [Equation]
Revised: [Equation]

Original: Where ∗ follows from z 2 representation of v 2 .
Revised: Where ∗ follows from z 2 representation of v 2 .

Original: The bound reported in this equation is tight whenever z 2 is sufﬁcient for z 1 .
Revised: The bound reported in this equation is tight whenever z 2 is sufﬁcient for z 1 .

Original: This happens whenever z 2 contains all the information regarding z 1 (and therefore v 1 ).
Revised: This happens whenever z 2 contains all the information regarding z 1 (and therefore v 1 ).

Original: Once again, the same bound can symmetrically be used to deﬁne I θ ( v 1 ; v 2 | z 2 ) ≤
Revised: Once again, the same bound can symmetrically be used to deﬁne I θ ( v 1 ; v 2 | z 2 ) ≤

Original: I ( v 1 ; v 2 ) − I θψ ( z 1 ; z 2 ) .
Revised: I ( v 1 ; v 2 ) − I θψ ( z 1 ; z 2 ) .

Original: Since I ( v 1 ; v 2 ) is constant in θ and ψ , the loss function in Equation 6 can be upper-bounded with;
Revised: Since I ( v 1 ; v 2 ) is constant in θ and ψ , the loss function in Equation 6 can be upper-bounded with;

Original: [Equation]
Revised: [Equation]

Original: Where:
Revised: Where:

Original: [Equation]
Revised: [Equation]

Original: Lastly, multiplying both terms with β := 2 λ 1 + λ 2 and re-parametrizing the objective, we obtain:
Revised: Lastly, multiplying both terms with β := 2 λ 1 + λ 2 and re-parametrizing the objective, we obtain:

Original: [Equation]
Revised: [Equation]

Original: E I NFORMATION P LANE

Original: Every representation z of x must satisfy the following constraints:

Original: • 0 ≤ I ( y ; z ) ≤ I ( x ; y ) :

Original: The amount of label information ranges from 0 to the total predictive information accessible from the raw observations I ( x ; y ) .

Original: • I ( y ; z )

Original: ≤ I ( x ; z ) ≤ I ( y ; z )+ H ( x | y ) :

Original: The representation must contain more information about the observations than about the label.

Original: When x is discrete, the amount of discarded label information I ( x ; y ) − I ( y ; z ) must be smaller than the amount of discarded observational information H ( x ) − I ( x ; z ) , which implies I ( x ; z ) ≤ I ( y ; z )

Original: + H ( x | y )

Original: Proof.

Original: Since z is a representation of x :

Original: [Equation]

Original: [Equation]

Original: Considering the four bounds separately:

Original: I ( y ; z ) ≥ 0 : Follows from P2.

Original: I ( x ; z ) ≥ I ( y ; z ) : Follows from ( T 1 ) in Lemma B. 3.

Original: I ( y ; z ) ≤

Original: I ( y ; x ) : Data processing inequality

Original: [Equation]

Original: I ( x ; z ) ≤ I ( y ; z ) + H ( x |

Original: [Equation]

Original: Note that ( H 2 ) is needed only to prove bound 4 .

Original: For continuous x bounds 1 , 2 and 3 still hold.

Original: F E XPERIMENTAL PROCEDURE AND DETAILS F. M ODELING
Revised: G E XPERIMENTAL PROCEDURE AND DETAILS G. M ODELING

Original: The two stochastic encoders p θ ( z 1 | v 1 ) and p ψ ( z 2 | v 2 ) are modeled by Normal distributions parametrized with neural networks ( µ θ , σ 2 θ ) and ( µ ψ , σ 2 ψ ) respectively:
Revised: The two stochastic encoders p θ ( z 1 | v 1 ) and p ψ ( z 2 | v 2 ) are modeled by Normal distributions parametrized with neural networks ( µ θ , σ 2 θ ) and ( µ ψ , σ 2 ψ ) respectively:

Original: [Equation]
Revised: [Equation]

Original: Since the density of the two encoders can be evaluated, the symmetrized KL-divergence in equation 4 can be directly computed.
Revised: Since the density of the two encoders can be evaluated, the symmetrized KL-divergence in equation 4 can be directly computed.

Original: On the other hand, I θψ ( z 1 ; z 2 ) requires the use of a mutual information estimator.
Revised: On the other hand, I θψ ( z 1 ; z 2 ) requires the use of a mutual information estimator.

Original: To facilitate the optimization, the hyper-parameter β is slowly increased during training, starting from a small value ≈ 10 − 4 to its ﬁnal value with an exponential schedule.
Revised: To facilitate the optimization, the hyper-parameter β is slowly increased during training, starting from a small value ≈ 10 − 4 to its ﬁnal value with an exponential schedule.

Original: This is because the mutual information estimator is trained together with the other architectures and, since it starts from a random initialization, it requires an initial warm-up.
Revised: This is because the mutual information estimator is trained together with the other architectures and, since it starts from a random initialization, it requires an initial warm-up.

Original: Starting with bigger β results in the encoder collapsing into a ﬁxed representation.
Revised: Starting with bigger β results in the encoder collapsing into a ﬁxed representation.

Original: The update policy for the hyper-parameter during training has not shown strong inﬂuence on the representation, as long as the mutual information estimator network has reached full capacity.
Revised: The update policy for the hyper-parameter during training has not shown strong inﬂuence on the representation, as long as the mutual information estimator network has reached full capacity.

Original: All the experiments have been performed using the Adam optimizer with a learning rate of 10 −for both encoders and the estimation network.
Revised: All the experiments have been performed using the Adam optimizer with a learning rate of 10 −for both encoders and the estimation network.

Original: Higher learning rate can result in instabilities in the training procedure.
Revised: Higher learning rate can result in instabilities in the training procedure.

Original: The results reported in the main text relied on the Jensen-Shannon mutual information estimator (Devon Hjelm et al., 2019) since the InfoNCE counterpart (van den Oord et al., 2018) generally resulted in worse performances that could be explained by the effect of the factorization of the critic network (Poole et al., 2019).[Figure]
Revised: The results reported in the main text relied on the Jensen-Shannon mutual information estimator (Devon Hjelm et al., 2019) since the InfoNCE counterpart (van den Oord et al., 2018) generally resulted in worse performance that could be explained by the effect of the factorization of the critic network (Poole et al., 2019).

Original:  S KETCHY E XPERIMENTS
Revised: G. S KETCHY E XPERIMENTS

Original: • Input: The two views for the sketch-based classiﬁcation task consist of 4096 dimensional sketch and image features extracted from two distinct VGG-16 network models which were pre-trained on images and sketches from the TU-Berlin dataset Eitz et al.
Revised: • Input: The two views for the sketch-based classiﬁcation task consist of 4096 dimensional sketch and image features extracted from two distinct VGG-16 network models which were pre-trained on images and sketches from the TU-Berlin dataset Eitz et al.

Original: (2012) for endto-end classiﬁcation.
Revised: (2012) for endto-end classiﬁcation.

Original: The feature extractors are frozen during the training procedure of for the two representations.
Revised: The feature extractors are frozen during the training procedure of for the two representations.

Original: Each training iteration used batches of size B = 128 .
Revised: Each training iteration used batches of size B = 128 .

Original: • Encoder and Critic architectures: Both sketch and image encoders consist of multi-layer perceptrons of 2 hidden ReLU units of size 2,048 and 1,024 respectively with an output of size 2x64 that parametrizes mean and variance for the two Gaussian posteriors.
Revised: • Encoder and Critic architectures: Both sketch and image encoders consist of multi-layer perceptrons of 2 hidden ReLU units of size 2,048 and 1,024 respectively with an output of size 2x64 that parametrizes mean and variance for the two Gaussian posteriors.

Original: The critic architecture also consists of a multi layer perceptron of 2 hidden ReLU units of size 512.
Revised: The critic architecture also consists of a multi layer perceptron of 2 hidden ReLU units of size 512.

Original: • β update policy: The initial value of β is set to 10 − 4 .
Revised: • β update policy: The initial value of β is set to 10 − 4 .

Original: Starting from the 10,000 th training iteration, the value of β is exponentially increased up to 1.0 during the following 250, training iterations.
Revised: Starting from the 10,000 th training iteration, the value of β is exponentially increased up to 1.0 during the following 250, training iterations.

Original: The value of β is then kept ﬁxed to one until the end of the training procedure (500,000 iterations).
Revised: The value of β is then kept ﬁxed to one until the end of the training procedure (500,000 iterations).

Original: • Evaluation: All natural images are used as both training sets and retrieval galleries.
Revised: • Evaluation: All natural images are used as both training sets and retrieval galleries.

Original: The 64 dimensional real outputs of sketch and image representation are compared using Euclidean distance.
Revised: The 64 dimensional real outputs of sketch and image representation are compared using Euclidean distance.

Original: For having a fair comparison other methods that rely on binary hashing (Liu et al., 2017; Zhang et al., 2018), we used Hamming distance on a binarized representation (obtained by applying iterative quantization Gong et al.
Revised: For having a fair comparison other methods that rely on binary hashing (Liu et al., 2017; Zhang et al., 2018), we used Hamming distance on a binarized representation (obtained by applying iterative quantization Gong et al.

Original: on our real valued representation).
Revised: on our real valued representation).

Original: We report the mean average precision (mAP@all) and precision at toprank 200 (Prec@200) Su et al.
Revised: We report the mean average precision (mAP@all) and precision at toprank 200 (Prec@200) Su et al.

Original: (2015) on both the real and binary representation to evaluate our method and compare it with prior works.
Revised: (2015) on both the real and binary representation to evaluate our method and compare it with prior works.

Original: [Figure] Salakhutdinov, 2014) [Table]
Revised: G. MIR-F LICKR E XPERIMENTS

Original: • Input: Whitening is applied to the handcrafted image features.
Revised: • Input: Whitening is applied to the handcrafted image features.

Original: Batches of size B =are used for each update step.
Revised: Batches of size B =are used for each update step.

Original: • Encoders and Critic architectures:
Revised: • Encoders and Critic architectures:

Original: The two encoders consists of a multi layer perceptron of 4 hidden ReLU units of size 1,024, which exactly resemble the architecture used in  Wang et al.
Revised: The two encoders consists of a multi layer perceptron of 4 hidden ReLU units of size 1,024, which exactly resemble the architecture used in [Figure] Wang et al.

Original: Both representations z 

Revised: Both representations z

Original: and z 2 have a size of 1,024, therefore the two architecture output a total of 2x1,024 parameters that deﬁne mean and variance of the respective factorized Gaussian posterior.
Revised: and z 2 have a size of 1,024, therefore the two architecture output a total of 2x1,024 parameters that deﬁne mean and variance of the respective factorized Gaussian posterior.

Original: Similarly to the Sketchy experiments, the critic is consists of a multi-layer perceptron of 2 hidden ReLU units of size512.
Revised: Similarly to the Sketchy experiments, the critic is consists of a multi-layer perceptron of 2 hidden ReLU units of size

Original: • β update policy: The initial value of β is set to 10 − 8 .
Revised: • β update policy: The initial value of β is set to 10 − 8 .

Original: Starting from 150000 th iteration, β is set to exponentially increase up to 1.0 (and 10 − 3 ) during the following 150,000 iterations.
Revised: Starting from 150000 th iteration, β is set to exponentially increase up to 1.0 (and 10 − 3 ) during the following 150,000 iterations.

Original: • Evaluation: Once the models are trained on the unlabeled set, the representation of the 25,000 labeled images is computed.
Revised: • Evaluation: Once the models are trained on the unlabeled set, the representation of the 25,000 labeled images is computed.

Original: The resulting vectors are used for training and evaluating a multi-label logistic regression classiﬁer on the respective splits.
Revised: The resulting vectors are used for training and evaluating a multi-label logistic regression classiﬁer on the respective splits.

Original: The optimal parameters (such as β ) for our model are chosen based on the performance on the validation set.
Revised: The optimal parameters (such as β ) for our model are chosen based on the performance on the validation set.

Original: In Table 3, we report the aggregated mean of the 5 test splits as the ﬁnal value mean average precision value.
Revised: In Table 3, we report the aggregated mean of the 5 test splits as the ﬁnal value mean average precision value.

Original: F. MNIST E XPERIMENTS
Revised: G. MNIST E XPERIMENTS

Original: • Input: The two views v 1 and v 2 for the MNIST dataset are generated by applying small translation ([0-10]%), rotation ([-15,15] degrees), scale ([90,110]%), shear ([-15,15] degrees) and pixel corruption (20%).
Revised: • Input: The two views v 1 and v 2 for the MNIST dataset are generated by applying small translation ([0-10]%), rotation ([-15,15] degrees), scale ([90,110]%), shear ([-15,15] degrees) and pixel corruption (20%).

Original: Batches of size B = 64 samples are used during training.
Revised: Batches of size B = 64 samples are used during training.

Original: • Encoders, Decoders and Critic architectures:
Revised: • Encoders, Decoders and Critic architectures:

Original: All the encoders used for the MNIST experiments consist of neural networks with two hidden layers of 1,024 units and ReLU activations, producing a 2x64-dimensional parameter vector that is used to parameterize mean and variance for the Gaussian posteriors.
Revised: All the encoders used for the MNIST experiments consist of neural networks with two hidden layers of 1,024 units and ReLU activations, producing a 2x64-dimensional parameter vector that is used to parameterize mean and variance for the Gaussian posteriors.

Original: The decoders used for the VAE experiments also consist of the networks of the same size.
Revised: The decoders used for the VAE experiments also consist of the networks of the same size.

Original: Similarly, the critic architecture used for mutual information estimation consists of two hidden layers of 1,204 units each and ReLU activations.
Revised: Similarly, the critic architecture used for mutual information estimation consists of two hidden layers of 1,204 units each and ReLU activations.

Original: • β update policy: The initial value of β is set to 10 − 3 , which is increased with an exponential schedule starting from the 50,000 th until 1the 50,000 th iteration.
Revised: • β update policy: The initial value of β is set to 10 − 3 , which is increased with an exponential schedule starting from the 50,000 th until 1the 50,000 th iteration.

Original: The value of β is then kept constant until the 1,000,000 th iteration.
Revised: The value of β is then kept constant until the 1,000,000 th iteration.

Original: The same annealing policy is used to trained the different β -VAEs reported in this work.
Revised: The same annealing policy is used to trained the different β -VAEs reported in this work.

Original: • Evaluation: The trained representation are evaluated following the well-known protocol described in Tschannen et al.
Revised: • Evaluation: The trained representation are evaluated following the well-known protocol described in Tschannen et al.

Original: ; Tian et al.
Revised: ; Tian et al.

Original: ; Bachman et al.
Revised: ; Bachman et al.

Original: ; van den Oord et al.
Revised: ; van den Oord et al.

Original: Each logistic regression is trained 5 different balanced splits of the training set for different percentages of training examples, ranging from 1 example per label to the whole training set.
Revised: Each logistic regression is trained 5 different balanced splits of the training set for different percentages of training examples, ranging from 1 example per label to the whole training set.

Original: The accuracy reported in this work has been computed on the disjoint test set.
Revised: The accuracy reported in this work has been computed on the disjoint test set.

Original: Mean and standard deviation are computed according to the 5 different subsets used for training the logistic regression.
Revised: Mean and standard deviation are computed according to the 5 different subsets used for training the logistic regression.

Original: Mean and variance for the mutual information estimation reported on the Information Plane (Figure 4) are computed by training two estimation networks from scratch on the ﬁnal representation of the non-augmented train set.
Revised: Mean and variance for the mutual information estimation reported on the Information Plane (Figure 4) are computed by training two estimation networks from scratch on the ﬁnal representation of the non-augmented train set.

Original: The two estimation architectures consist of 2 hidden layers of 2048 and 1024 units each, and have been trained with batches of size B = 256 for a total of approximately 25,000 iterations.
Revised: The two estimation architectures consist of 2 hidden layers of 2048 and 1024 units each, and have been trained with batches of size B = 256 for a total of approximately 25,000 iterations.

Original: The Jensen-Shannon mutual information lower bound is maximized during training, while the numerical estimation are computed using an energy-based bound (Poole et al., 2019; Devon Hjelm et al., 2019).
Revised: The Jensen-Shannon mutual information lower bound is maximized during training, while the numerical estimation are computed using an energy-based bound (Poole et al., 2019; Devon Hjelm et al., 2019).

Original: The ﬁnal values for I ( x ; z ) and I ( y ; z ) are computed by averaging the mutual information estimation on the whole dataset.
Revised: The ﬁnal values for I ( x ; z ) and I ( y ; z ) are computed by averaging the mutual information estimation on the whole dataset.

Original: In order to reduce the variance of the estimator, the lowest and highest 5% are removed before averaging.
Revised: In order to reduce the variance of the estimator, the lowest and highest 5% are removed before averaging.

Original: This practical detail makes the estimation more consistent and less susceptible to numerical instabilities.
Revised: This practical detail makes the estimation more consistent and less susceptible to numerical instabilities.

Original: G A BLATION STUDIES G. D IFFERENT RANGES OF DATA AUGMENTATION
Revised: G.4. R ESULTS AND V ISUALIZATION

Revised: In this section we include additional quantitative results and visualizations which refer to the singleview MNIST experiments reported in section 5.2. Table 2 reports the quantitative results used for to produce the visualizations reported in Figure 4, including the comparison between the performance resulting from different mutual information estimators. As the Jensen-Shannon estimator generally resulted in better performance for the InfoMax,

Revised: MV-InfoMax and MIB models, all the experiments reported on the main text make use of this estimator.

Revised: Note that the InfoMax model with the I JS estimator is equivalent to the global model reported in Devon Hjelm et al.

Revised: while MV-InfoMax with the I NCE estimator results in a similar architecture to the one introduced in Tian et al.

Revised: [Table]

Revised: Figure 7 reports the linear projection of the embedding obtained using the MIB model.

Revised: The latent space appears to roughly consists of ten clusters which corresponds to the different digits.

Revised: This observation is consistent with the empirical measurement of input and label information I ( x ; z )

Revised: ≈

Revised: I ( z ; y )

Revised: ≈ log 10 , and the performance of the linear classiﬁer in scarce label regimes.

Revised: As the cluster are distinct and concentrated around the respective centroids, 10 labeled examples are sufﬁcient to align the centroid coordinates with the digit labels.

Revised: H A BLATION STUDIES H. D IFFERENT RANGES OF DATA AUGMENTATION

Original: Figure 7 visualizes the effect of different ranges of corruption probabily as data augmentation strategy to produce the two views v 1 and v 2 .
Revised: Figure 8 visualizes the effect of different ranges of corruption probabily as data augmentation strategy to produce the two views v 1 and v 2 .

Original: The MV-InfoMax Model does not seem to get any advantage from the use increasing amount of corruption, and it representation remains approximately in the same region of the information plane.
Revised: The MV-InfoMax Model does not seem to get any advantage from the use increasing amount of corruption, and it representation remains approximately in the same region of the information plane.

Original: On the other hand, the models trained with the MIB objective are able to take advantage of the augmentation to remove irrelevant data information and the representation transitions from the top right corner of the Information Plane (no-augmentation) to the top-left.
Revised: On the other hand, the models trained with the MIB objective are able to take advantage of the augmentation to remove irrelevant data information and the representation transitions from the top right corner of the Information Plane (no-augmentation) to the top-left.

Original: When the amount of corruption approaches 100%, the mutual redundancy assumption is clearly violated, and the performances of MIB deteriorate.
Revised: When the amount of corruption approaches 100%, the mutual redundancy assumption is clearly violated, and the performances of MIB deteriorate.

Original: In the initial part of the transitions between the two regimes (which corresponds to extremely low probability of corruption) the
Revised: In the initial part of the transitions between the two regimes (which corresponds to extremely low probability of corruption) the

Original: MIB models drops some label information that is quickly re-gained when pixel corruption becomes more frequent.
Revised: MIB models drops some label information that is quickly re-gained when pixel corruption becomes more frequent.

Original: We hypothesize that this behavior is due to a problem with the optimization procedure, since the corruption are extremely unlikely, the Monte-Carlo estimation for the symmetrized
Revised: We hypothesize that this behavior is due to a problem with the optimization procedure, since the corruption are extremely unlikely, the Monte-Carlo estimation for the symmetrized

Original: Kullback-Leibler divergence is more biased.
Revised: Kullback-Leibler divergence is more biased.

Original: Using more examples of views produced from the same data-point within the same batch could mitigate this issue.
Revised: Using more examples of views produced from the same data-point within the same batch could mitigate this issue

Original: [Figure]
Revised: [Figure]

Original: G. E FFECT OF β
Revised: H. E FFECT OF β

Original: The hyper-parameter β (Equation 5) determines the trade-off between sufﬁciency and minimality of the representation for the second data view.
Revised: The hyper-parameter β (Equation 5) determines the trade-off between sufﬁciency and minimality of the representation for the second data view.

Original: When β is zero, the training objective of MIB is equivalent to the Multi-View InfoMax target, since the representation has no incentive to discard any information.
Revised: When β is zero, the training objective of MIB is equivalent to the Multi-View InfoMax target, since the representation has no incentive to discard any information.

Original: When 0 < β ≤ 1 the sufﬁciency constrain is enforced, while the superﬂuous information is gradually removed from the representation.
Revised: When 0 < β ≤ 1 the sufﬁciency constrain is enforced, while the superﬂuous information is gradually removed from the representation.

Original: Values of β > 1 can result in representations that violate the sufﬁciency constraint, since the minimization of I ( x ; z | v 2 ) is prioritized.
Revised: Values of β > 1 can result in representations that violate the sufﬁciency constraint, since the minimization of I ( x ; z | v 2 ) is prioritized.

Original: The trade-off resulting from the choice of different β is visualized in Figure 8 and compared against β -VAE.
Revised: The trade-off resulting from the choice of different β is visualized in Figure 9 and compared against β -VAE.

Original: Note that in each point of the pareto-front the MIB model results in a better trade-off between I ( x ; z ) and I ( y ; z ) when compared to β -VAE.
Revised: Note that in each point of the pareto-front the MIB model results in a better trade-off between I ( x ; z ) and I ( y ; z ) when compared to β -VAE.

Original: The effectiveness of the Multi-View Information Bottleneck model is also justiﬁed by the corresponding values of predictive 
Revised: The effectiveness of the Multi-View Information Bottleneck model is also justiﬁed by the corresponding values of predictive accuracy.

Original: [Figure]
Revised: [Figure]