Original: DICE D IVERSITY IN  E NSEMBLE WITH C ONDITIONAL E NTROPY B OTTLENECK THROUGH M UTUAL I NFORMATION N EURAL E STIMATION
Revised: DICE: D IVERSITY IN D EEP E NSEMBLES VIA C ONDITIONAL R EDUNDANCY A DVERSARIAL E STIMATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep ensembles perform better than a single network thanks to the diversity among their members.
Revised: Deep ensembles perform better than a single network thanks to the diversity among their members.

Original: Recent approaches regularize predictions to increase diversity; however, they also drastically decrease individual members performances.
Revised: Recent approaches regularize predictions to increase diversity; however, they also drastically decrease individual members’ performances.

Original: In this paper, we argue that learning strategies for  ensembles need to tackle the trade-off between ensemble diversity and individual accuracies .
Revised: In this paper, we argue that learning strategies for deep ensembles need to tackle the trade-off between ensemble diversity and individual accuracies .

Original: Motivated by  information theoryarguments, we introduce a novel training criterion, DICE, that enforces diversity among members’ features . Leveraging variational approximations and  recent advances in neural estimation of conditional mutual information, DICE deletes spurious correlations by adversarial conditional featuresalignment .
Revised: Motivated by arguments from information theory and leveraging recent advances in neural estimation of conditional mutual information, we introduce a novel training criterion called DICE: it increases diversity by reducing spurious correlations among features .

Original: Features extracted from pairs of members should only share information useful for target class prediction without being conditionally redundant: thus, besides the classiﬁcation loss with information bottleneck, we adversarially prevent features from being  predictable from each other.
Revised: The main idea is that features extracted from pairs of members should only share information useful for target class prediction without being conditionally redundant. Therefore, besides the classiﬁcation loss with information bottleneck, we adversarially prevent features from being conditionally predictable from each other.

Original: It reduces simultaneous errors while protecting class information.
Revised: We manage to reduce simultaneous errors while protecting class information.

Original: We obtain state-of-the-art accuracy results on CIFAR10/100: e.g., an ensemble of 5 networks trained with DICE matches an ensemble of 7  trained independently.
Revised: We obtain state-of-the-art accuracy results on CIFAR-10/100: for example, an ensemble of 5 networks trained with DICE matches an ensemble of 7 networks trained independently.

Original: We improve calibration, uncertainty estimation in- and out- domain, out-of-distribution detection, before and after temperature scaling.
Revised: We further analyze the consequences on calibration, uncertainty estimation, out-of-distribution detection and online co-distillation.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Averaging the predictions of several models can signiﬁcantly improve the generalization ability of a predictive system.
Revised: Averaging the predictions of several models can signiﬁcantly improve the generalization ability of a predictive system.

Original: Due to its effectiveness, ensemble has been a popular research topic (Nilsson, 1965; Hansen & Salamon, 1990; Wolpert, 1992; Krogh & Vedelsby, 1995; Breiman, 1996; Dietterich, 2000; Zhou et al., 2002; Kuncheva & Whitaker, 2003; Rokach, 2010) as a simple alternative of bayesian methods (Blundell et al., 2015; Gal & Ghahramani, 2016).
Revised: Due to its effectiveness, ensembling has been a popular research topic (Nilsson, 1965; Hansen & Salamon, 1990; Wolpert, 1992; Krogh & Vedelsby, 1995; Breiman, 1996; Dietterich, 2000; Zhou et al., 2002; Kuncheva & Whitaker, 2003; Rokach, 2010) as a simple alternative to fully Bayesian methods (Blundell et al., 2015; Gal & Ghahramani, 2016).

Original: It is currently the de facto solution for many machine learning applications and  competitions such as Kaggle.
Revised: It is currently the de facto solution for many machine learning applications and Kaggle competitions (Hin, 2020).

Original: Intuitively, M identical members do not contain more information than a single one: the signiﬁcant accuracy boost is therefore due to the predictions’ diversity. The reduction factor of  the variance in the Bias-Variance Decomposition (Kohavi et al., 1996) equals to M only when errors are uncorrelated (Krogh & Vedelsby, 1995; Kuncheva)
Revised: Ensembling reduces the variance of estimators (see Appendix E.1) thanks to the diversity in predictions. This reduction is most effective when errors are uncorrelated and members are diverse, i.e., when they do not simultaneously fail on the same examples. Conversely, an ensemble of M identical networks is no better than a single one.

Original: (see Appendix E.1).

Original: In deep ensembles (Lakshminarayanan et al., 2016), the weights are traditionally trained independently: diversity among members only relies on the randomness of the initialization and of the learning procedure.
Revised: In deep ensembles (Lakshminarayanan et al., 2017), the weights are traditionally trained independently : diversity among members only relies on the randomness of the initialization and of the learning procedure.

Original: Figureshows that it does not not fully leverage additional members and that beneﬁts get quickly plateaued.
Revised: Figure 1 shows that the performance of this procedure quickly plateaus with additional members.

Original: To obtain more diverse ensembles, we could adapt the training samples through bagging (Breiman, 1996) and bootstrapping (Efron & Tibshirani, 1994), but a reduction of training samples has a negative impact on members with multiple local minima (Lee et al., 2015).
Revised: To obtain more diverse ensembles, we could adapt the training samples through bagging (Breiman, 1996) and bootstrapping (Efron & Tibshirani, 1994), but a reduction of training samples has a negative impact on members with multiple local minima (Lee et al., 2015).

Original: Sequential boosting does not scale well for time-consuming deep learners that overﬁt their training dataset.
Revised: Sequential boosting does not scale well for time-consuming deep learners that overﬁt their training dataset.

Original: Liu & Yao (1999a;b);
Revised: Liu & Yao (1999a;b);

Original: Brown et al.
Revised: Brown et al.

Original: (2005b)tried to explicitly quantify the diversity and regularizedthe members into having negatively correlated errors.
Revised: (2005b) explicitly quantiﬁed the diversity and regularized members into having negatively correlated errors.

Original: However, the ensemble accuracyremains almost unchanged when applied with deep learners (Shui et al., 2018; Pang et al., 2019): while members should predict the same target, they force disagreements  and therefore increase members’ bias.
Revised: However, these ideas have not signiﬁcantly improved accuracy when applied to deep learning (Shui et al., 2018; Pang et al., 2019): while members should predict the same target, they force disagreements among strong learners and therefore increase their bias.

Original: It highlightsour main challenge: the  trade-off between ensemble diversity and individual accuracies .ber: ResNet-32. See details in Table 7. DICE adversarially prevents features from being predictable from each other conditionally features extracted by
Revised: It highlights the main objective and challenge of our paper: ﬁnding a training strategy to reach an improved trade-off between ensemble diversity and individual accuracies .

Original: ( , ) should not share more information that when features are from two inputs from the same class ( , ).

Original: We took an opposite line as we believe that all members should predict the same thing, butnot for the same reasons.
Revised: Our core approach is to encourage all members to predict the same thing, but for different reasons .

Original: Therefore, to protect the ability of each member to predict correctly, the diversity should be enforced in the features  and not in the predictions.
Revised: Therefore the diversity is enforced in the features space and not on predictions.

Original: Intuitively, to maximize the impact of a new member, extracted features should bring information about the target that isyet absent therefore unpredictable from other members’ features.
Revised: Intuitively, to maximize the impact of a new member, extracted features should bring information about the target that is absent at this time so unpredictable from other members’ features.

Original: They should ﬁt different modes and have uncorrelated bias to have non-simultaneous errors: they will not fail on the same hard examples.
Revised: It would remove spurious correlations , e g. information redundantly shared among features extracted by different members but useless for class prediction. This redundancy may be caused by a detail in the image background and therefore will not be found in features extracted from other images belonging to the same class. This could make members predict badly simultaneously, as shown in Figure 2.

Original: We draw inspiration from Information Bottleneck (IB) principles, (Tishby et al., 2000;
Revised: Our new learning framework, called DICE, is driven by Information Bottleneck (IB) (Tishby, 1999;

Original: Alemi et al., 2017)  that force learned representation to be concise by forgetting the task-irrelevant factors: it limits overﬁtting while increasing robustness.
Revised: Alemi et al., 2017) principles, that force features to be concise by forgetting the task-irrelevant factors and increase robustness.

Original: The Minimum Necessary Information(MNI) criterion (Fischer, 2020) for deep ensembles leads to reducing the mutual information (MI) betweenthe features andthe inputs, but alsothe conditional mutual information (CMI) between features conditioned on the target.
Revised: Speciﬁcally, DICE leverages the Minimum Necessary Information criterion (Fischer, 2020) for deep ensembles, and aims at reducing the mutual information (MI) between features and inputs, but also information shared between features.

Original: We make our features conditionally independent to reduce the covariance in predictions, by aligning the conditional features distributions.
Revised: We prevent extracted features from being redundant.

Original: As MI can detect arbitrary dependencies between random variables (not just linear ones, as seen in Figure 2), reducing CMI increases the distance between pairs of members: it promotes diversity without decreasing the capacity of extracted features to predict the target.
Revised: As mutual information can detect arbitrary dependencies between random variables (such as symmetry, see Figure 2), we increase the distance between pairs of members: it promotes diversity by reducing predictions’ covariance. Most importantly, our DICE strategy protects features’ informativeness about the target by conditioning mutual information upon the target.

Original: We build upon recent neural approaches (Belghazi et al., 2018) for MI estimationfrom samples, based on the Donsker-Varadhan representation of the KL.
Revised: We build upon recent neural approaches (Belghazi et al., 2018) for DICE estimation based on the Donsker-Varadhan representation of the KL.

Original: We summarize our contributions as follows:
Revised: We summarize our contributions as follows:

Original: • We introduce DICE, a new  framework to explicitly increasethe diversity between members by minimizing the conditional mutual information (CMI) betweenmembers’ features.
Revised: • We introduce DICE, a new adversarial learning framework to explicitly increase diversity in ensemble by minimizing the conditional redundancy between features.

Original: • We justify our training objective with arguments from information theory.
Revised: • We rationalize our training objective by arguments from information theory.

Original: • We make our criterion tractable throughadversarial neural estimation of CMI.
Revised: • We propose an efﬁcient implementation of our objective through neural estimation of conditional redundancy.

Original: We empirically demonstrate consistent improvements in accuracyand robustness on CIFAR-10/ (Krizhevsky et al., 2009).
Revised: We consistently improve accuracy on CIFAR-10/100 as summarized in Figure 1, with better uncertainty estimation and calibration.

Original: We analyze  the two components of our loss and their relative impact on the accuracy-diversity trade-off.
Revised: We analyze how the two components of our loss modify the accuracy-diversity trade-off.

Original: We give hints about our adversarial discriminator’ behaviour while improving out-of-distribution detection.
Revised: We investigate the behaviour of our adversarial discriminator while improving out-of-distribution detection.

Original: Finally, we extend DICE for online co-distillation.
Revised: Finally, we extend DICE for online co-distillation.

Original:  DICE MODEL
Revised:  DICE M ODEL

Original: Notations Given an input distribution X , a network θ is trained to extract the best possible dense features Z to model the distribution p θ ( Y | X ) over the targets, that should be close to the  true dirac.
Revised: Notations Given an input distribution X , a network θ is trained to extract the best possible dense features Z to model the distribution p θ ( Y | X ) over the targets, which should be close to the Dirac on the true label.

Original: Our approach is designed for ensembles with M members θ m , m ∈ { 1 , . .
Revised: Our approach is designed for ensembles with M members θ m , m ∈ { 1 , . .

Original: , M } that extract Z m .
Revised: , M } extracting Z m .

Original: Inthe branch-based setup, members share low-level weights to reducethe computation cost.
Revised: In branch-based setup, members share low-level weights to reduce computation cost.

Original: We average members predictions in inference.
Revised: We average the M predictions in inference.

Original: We initially consider an ensemble of M = 2 members.
Revised: We initially consider an ensemble of M = 2 members.

Original: Summary First, we train each member  for classiﬁcation with conditional entropy bottleneck .
Revised: Quick overview First, we train each member separately for classiﬁcation with information bottleneck.

Original: Second, the discriminator learns to distinguish features sampled from one image versus features sampled from two images belonging to the same class, conditioned on the class Y . Simultaneously, both members adversarially delete spurious  correlations found only when features are extracted from the same image, which would otherwise simplify the discriminator task.
Revised: Second, we train members together to remove spurious redundant correlations while training adversarially a discriminator.

Original: In conclusion,our members learn to classify while having conditionally uncorrelated features for increased diversity and accuracy.
Revised: In conclusion, members learn to classify with conditionally uncorrelated features for increased diversity.

Original: Our procedure, summarized in Figure 3, is motivated by the following theoretical ﬁndings.
Revised: Our procedure is driven by the following theoretical ﬁndings.

Original: 2. M INIMUM N ECESSARY I NFORMATION 2.1. M UTUAL INFORMATION FORMULATION
Revised: 2.A D ERIVING T RAINING O BJECTIVE 2.A. B ASELINE : N ON -C ONDITIONAL O BJECTIVE

Original: The Minimum Necessary Information (MNI) criterion from (Fischer, 2020) aims at ﬁnding minimal statistics.
Revised: The Minimum Necessary Information (MNI) criterion from (Fischer, 2020) aims at ﬁnding minimal statistics.

Original: In deep ensembles, Z 1 and Z 2 should capture only minimal information from X , while preserving the necessary information about the task Y .
Revised: In deep ensembles, Z 1 and Z 2 should capture only minimal information from X , while preserving the necessary information about the task Y .

Original: As entropy measures information, we derive:

Original: [Equation] We ﬁrst consider separately the two Markov chains Z 1 ← X ↔ Y and Z 2 ← X ↔ Y .
Revised: First, we consider separately the two Markov chains Z 1 ← X ↔ Y and Z 2 ← X ↔ Y .

Original: The entropy of Z 1 and Z 2 not related to Y should be minimized.
Revised: As entropy measures information, entropy of Z 1 and Z 2 not related to Y should be minimized.

Original: We recoverthe IBobjectives (Alemi et al., 2017): IB β ib ( Z 1 , Z 2 ) ≡ IB β ib ( Z 1 ) + IB β ib ( Z 2 )
Revised: We recover IB (Alemi et al., 2017) in deep ensembles:

Original: = 1 β ib [ I ( X ; Z 1 ) + I ( X ; Z 2 )] − [ I ( Y ; Z 1 ) +
Revised: [Equation]

Original: I ( Y ; Z 2 )] . Let’s additionally consider I ( Z 1 ; Z 2 ) that measures the shared information between both features. We minimize it to respect the minimal criterion of (Fischer, 2020).
Revised: Second, let’s consider I ( Z 1 ; Z 2 ) : we minimize it following the minimality constraint of the MNI.

Original: [Equation]
Revised: [Equation]

Revised: 3: Venn Information Diagram .

Revised: DICE minimizes conditional redundancy

Original: The information from X is compressed into two independent & disentangled views Z 1 and Z 2that both try to predict Y , in a relevancy-redundancy trade-off depending on  β ib and δ mi .
Revised: I ( Z 1 ; Z 2 | Y ) (green vertical stripes ) with no overlap with relevancy (red stripes). Analysis In this baseline criterion, relevancy encourages Zand Z 2 to capture information about Y . Compression & redundancy (R) split the information from X into two compressed & independent views. The relevancy-compressionredundancy trade-off depends on the values of β ib & δ r .

Original: 2.1. C ONDITIONAL MUTUAL INFORMATION FORMULATION
Revised: 2.A. DICE: C ONDITIONAL O BJECTIVE

Original: IBMI ignores Y in the regularization terms which causes loss of necessary information: it is impossible to have Z 1 and Zfully independent while training them to predict the same Y .
Revised: The problem is that IBR ignores Y in the compression & redundancy terms which cause loss of necessary information: it is detrimental to have Z 1 and Z 2 fully disentangled while training them to predict the same Y .

Original: As shown on Figure 4, redundancy regions (blue horizontal stripes ) overlap with relevancy regions (red stripes).
Revised: As shown on Figure 3, redundancy regions (blue horizontal stripes ) overlap with relevancy regions (red stripes).

Original: What the MNI  really entails are these conditional equalities:
Revised: Indeed, the true constraints that the MNI criterion really entails are the following conditional equalities given Y :

Original: [Equation]
Revised: [Equation]

Original: Mutual information being non-negative, we transform them into the following objective:
Revised: Mutual information being non-negative and following Lagrange method, we transform them into our main DICE objective:

Original: [Equation]
Revised: [Equation]

Original: | Y )]

Original: [Equation]
Revised: [Equation]

Original: Conditional Compression
Revised: where we recover two conditional entropy bottleneck (CEB) (Fischer, 2020) components,

Original: [Equation]
Revised: CEB β ceb ( Z m ) =

Original: Conditional Redundancy . 
Revised: 1 β ceb I ( X ; Z m | Y ) − I ( Y ; Z m ) , with β ceb > 0 and δ cr > 0 . Analysis The relevancy terms force features to be informative about the task Y . But contrary to

Original: Analyze DICE  only minimizes irrelevant information to the task.
Revised: IBR, DICE bottleneck constraints only minimize irrelevant information to Y .

Original: To speciﬁcally remove the spurious correlations between Z 1 and Z 2 , we directly target the conditional redundancy.
Revised: First, the conditional compression removes in Z 1 (or Z 2 ) information from X not relevant to Y .

Revised: Second, the conditional redundancy (CR) reduces spurious correlations between members and only forces them to have independent bias, but deﬁnitely not independent features .

Original: It encourages diversityby forcing members to have uncorrelated bias without affecting their individual precision as it protects information related to the target class.
Revised: It encourages diversity without affecting members’ individual precision as it protects information related to the target class in Z 1 and Z 2 .

Original: Useless information from X to predict Y should  not be simultaneously in Z 1 and Z 2  as it would correlate bias causing double-faults.
Revised: Useless information from X to predict Y should certainly not be in Z 1 or Z 2 , but it is even worse if they are in Z 1 and Z 2 simultaneously as it would cause simultaneous errors.

Original: Note that reducing I ( Z 1 , X | Y ) and I ( Z 2 , X | Y ) , the information Z 1 and Z 2 share with X given Y , also indirectly reduces I (  Z 1 , Z 2 | Y ) , but less efﬁciently.
Revised: Even if for m ∈ { 1 , 2 } , reducing I ( Z m , X | Y ) indirectly controls I ( Z 1 , Z 2 | Y ) (as I ( Z 1 ; Z 2 | Y ) ≤ I ( X ; Z m | Y ) , see Appendix E.6), it is more efﬁcient to directly target this intersection region through the CR term. In a ﬁnal word, DICE is to IBR for deep ensembles as CEB is to IB for a single network.

Original:  DICE beneﬁts from both components (see Appendix E.7).
Revised: We now approximate the two CEB and the CR components in DICE objective from equation 1.

Original: 2. T RANSFORMING DICE INTO A TRACTABLE LOSS 2.2. V ARIATIONAL APPROXIMATION OF CONDITIONAL ENTROPY BOTTLENECK
Revised: 2.B A PPROXIMATING DICE INTO A T RACTABLE L OSS 2.B. V ARIATIONAL A PPROXIMATION OF C ONDITIONAL E NTROPY B OTTLENECK

Original: We leverage Markov assumptions in Z  ← X ↔ Y and empirically estimate on the classiﬁcation training datasetD consisting of N i.i.d.
Revised: We leverage Markov assumptions in Z m ← X ↔ Y, m ∈ { 1 , 2 } and empirically estimate on the classiﬁcation training dataset of N i.i.d.

Original: data points D = { x n , y n } Nn =1 , y n ∈ { 1 , . .
Revised: points D = { x n , y n } Nn =1 , y n ∈ { 1 , . .

Original: . , K } .
Revised: . , K } .

Original: Following (Fischer, 2020), CEB β ceb ( Z  ) =[-
-]
Revised: Following Fischer (2020), CEB β ceb ( Z m ) =[-
-]

Original:  β ceb
Revised: 1 β ceb

Original: I ( X ; Z  | Y ) − I ( Y ; Z  ) is variationally upper bounded by: 
Revised: I ( X ; Z m | Y ) − I ( Y ; Z m ) is variationally upper bounded by: [Equation] [Equation] N

Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Seedetailed explanation in Appendix E.4.
Revised: See explanation in Appendix E.4.

Original: e  ( z | x ) is the true features distribution generated by the encoder, c
Revised: e m ( z | x ) is the true features distribution generated by the encoder ,

Original:  ( y | z ) is a variational approximation of p ( y | z ) by the classiﬁer, and b  ( z | y ) is a variational approximation of p ( z | y ) by the backward encoder, all with parameters to be learned.
Revised: c m ( y | z ) is a variational approximation of p ( y | z ) by the classiﬁer , and b m ( z | y ) is a variational approximation of p ( z | y ) by the backward encoder, all with parameters to be learned.

Original: We parametrize our distributions with gaussians: it enables (1) the exact computation of the KL divergence (see Appendix E.3) and (2) the reparameterization trick that gives a valid sample from the encoding distribution to the classiﬁer, z =
Revised: We parameterize distributions with gaussians to enable (1) the exact computation of the KL divergence (see Appendix E.3) and (2) the reparameterization trick for sampling from the encoder to the classiﬁer:

Original:  e  ( x, (cid:15) ) =
Revised: z = e m ( x, (cid:15) ) =

Original: e µ ( x )+ (cid:15)e σ ( x ) where (cid:15) ∼ N (0 , 1) .
Revised: e µm ( x ) + (cid:15)e σm ( x ) w. (cid:15) ∼ N (0 , 1) .

Original: Theonly advantage of VCEB versus VIB (Alemi et al., 2017) is the class conditional b  ( z | y ) versus unconditional b  ( z )variational marginal: VCEB protects class information while constraining the information shared between X and Z .
Revised: The advantage of VCEB versus VIB (Alemi et al., 2017) is the class conditional b m ( z | y ) versus non-conditional b m ( z ) marginal: VCEB protects class information.

Original: We apply this loss  on each individual member, as done in step 1 from Figure 3.
Revised: The loss is applied separately on each member θ m = { e m , b m , c m } : we learn distributions (means and covariances) rather than deterministic points in the features space.

Original: 2.2. N EURAL CONDITIONAL MUTUAL INFORMATION ESTIMATION
Revised: 2.B. A DVERSARIAL E STIMATION OF C ONDITIONAL R EDUNDANCY

Original: Theoretical problem We now focus on estimating I ( Z 1 ; Z 2 | Y ) , with no such Markov properties.
Revised: Theoretical Problem We now focus on estimating I ( Z 1 ; Z 2 | Y ) , with no such Markov properties.

Original: Despite being a pivotal measure, mutual information estimation historically relied on nearest neighbors (Singh et al., 2003; Kraskov et al., 2004; Gao et al., 2018) or density kernels (Kandasamy et al., 2015) and do not scale well in high dimensions.
Revised: Despite being a pivotal measure, mutual information estimation historically relied on nearest neighbors (Singh et al., 2003; Kraskov et al., 2004; Gao et al., 2018) or density kernels (Kandasamy et al., 2015) that do not scale well in high dimensions.

Original: We beneﬁt from recent advances inmutual information neural estimation  (Belghazi et al., 2018), built on optimizing Donsker & Varadhan (1975) dual representations of the KL divergence.
Revised: We beneﬁt from recent advances in neural estimation of mutual information (Belghazi et al., 2018), built on optimizing Donsker & Varadhan (1975) dual representations of the KL divergence.

Original: See (Poole et al., 2019; Song & Ermon, 2020) for a review.

Original: (Mukherjee et al., 2019; Molavipour et al., 2020) extended this formulation for CMI estimation.
Revised: Mukherjee et al. (2020) extended this formulation for conditional mutual information estimation.

Original: [Equation]
Revised: [Equation]

Original: where f ∗ computes the point-wise log-likelihood ratio, i.e f ∗ ( z 1 , z 2 , y )
Revised: where f ∗ computes the pointwise likelihood ratio, i.e. , f ∗ ( [Equation] 1 , z [Equation]

Original: =
Revised: =

Revised: [Equation]

Revised: Empirical Neural Estimation

Original:  p ( z 
Revised: We estimate CR (1) using the empirical data distribution and (2) replacing f ∗ = w ∗ 1 − w ∗ by the output of a discriminator w , trained to imitate the optimal w ∗ . Let B j be a batch sampled from the observed joint distribution p ( z 1 , z 2 , y

Original: [Equation]
Revised: [Equation]

Revised: p

Original: [Equation]
Revised: [Equation]

Revised: e

Revised: [Equation]

Original: p ( z
Revised: | x )

Original: [Equation]
Revised: [Equation]

Revised: z | x ) , y ) ; we select the features extracted by the two members from one input.

Revised: Let B p be sampled from the product distribution p ( z 1 , y )

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: Empirical neural estimation We estimate conditional mutual information (1) using the empirical data distribution and (2) substituting the optimal f ∗ by the output of a trained neural-based classiﬁer w . Let B j and B p be two random batches sampled respectively from the observed joint distribution p ( z 1 , z 2 , y ) = p ( e 1 ( z | x ) , e 2 ( z | x ) , y ) and the product distribution p ( z 1 , y ) p ( z 2 | y ) =
Revised: z 2 | y [Equation]

Original:  p ( e1 ( z | x ) , y ) p ( z 2 | y ) .
Revised: = p [Equation] e

Original: To sample from p ( z 1 , y ) p ( z 2 | y ) , we select the features extracted by the two networks from two different inputs that share the same class, from the same batch when possible, or from previous batches otherwise.
Revised: [Equation] z | x ) , y ) p ( z 2 | y ) ; we select the features extracted by the two members from two different inputs that share the same class.

Original: We trainw , a multi-layer network with sigmoid as ﬁnal activation, on the binary task of distinguishing these two distributions with the standard cross-entropy loss:L ce ( w ) =
Revised: We train a multi-layer network w on the binary task of distinguishing these two distributions with the standard cross-entropy loss:

Original: [Equation]
Revised: [Equation]

Original: At the end of training, if w is calibrated (see Appendix B.4) and by the ”Universal Function Approximation Theorem” (Cybenko, 1989), w will be close to the optimal w ∗ that satisﬁes: f ∗ ( z 1 , z 2 , y ) =

Original: w ∗ ( z 1 , z 2 , y ) 1 − w ∗ ( z 1 , z 2 , y )

Original: ≈ w ( z 1 , z 2 , y )

Original: 1 − w ( z 1 , z 2 , y ) = f ( z 1 , z 2 , y ) . Therefore a consistent  estimate of the CMI (see theorem 1 of Mukherjee et al. (2019)) is:
Revised: If w is calibrated (see Appendix B.4), a consistent (Mukherjee et al., 2020) estimate of CR is:

Original: [Equation]
Revised: [Equation]

Original: Intuition By training our members to minimize ˆ I CMIDV , we force triples from the true joint distribution to become indistinguishable from triples from the false product distribution.
Revised: Intuition By training our members to minimize ˆ I CRDV , we force triples from the true joint distribution to be indistinguishable from triples from the false product distribution.

Original: Let’s imagine that two features are conditionally correlated, some spurious information is shared between features only when they are from the same input and not from two inputs (from the same class).
Revised: Let’s imagine that two features are conditionally correlated, some spurious information is shared between features only when they are from the same input and not from two inputs (from the same class).

Revised: This correlation can be informative about a detail in the background, an unexpected shape in the image, that is rarely found in samples from this input’s class.

Original: Therefore the joint and the product distribution are easily distinguishable by the discriminator.
Revised: In that case, the product and joint distributions are easily distinguishable by the discriminator.

Original: The ﬁrst adversarial component will force the extracted features to reduce the correlation, and increase diversity.
Revised: The ﬁrst adversarial component will force the extracted features to reduce the correlation, and ideally one of the two features loses this information: it reduces redundancy and increases diversity.

Original: The second term would create fake correlations between features from different inputs and hinder the discriminator training.
Revised: The second term would create fake correlations between features from different inputs and hinder the discriminator.

Original: As we seek increased diversity and not  a precise CMI estimation, we get rid of this second term, which adds complexity and empirically destabilizes the training.
Revised: As we are not interested in a precise estimation of the CR, we get rid of this second term that, empirically, did not increase diversity.

Original: [Equation]
Revised: [Equation]

Revised: Summary First, we train each member for classiﬁcation with VCEB from equation 2, as shown in Step 1 from Figure 4.

Original: Summary It gives proper differentiable signals for adversarial (Goodfellow et al., 2014; Arjovsky et al., 2017) conditional features distributions alignment, as shown in step 2 from Figure 3. Features are class-conditionally independent while sharing class information.
Revised: Second, as shown in Step 2 from Figure 4, the discriminator, conditioned on the class Y , learns to distinguish features sampled from one image versus features sampled from two images belonging to Y . Simultaneously, both members adversarially (Goodfellow et al., 2014) delete spurious correlations to reduce CR estimation from equation 4 with differentiable signals: it conditionally aligns features. We provide a pseudo-code in B.2.

Original: Our global loss is therefore:
Revised: While we derive similar losses for

Revised: IBR and CEBR in Appendix E.5, the full DICE loss is ﬁnally:

Original: [Equation]
Revised: [Equation]

Original: We derive similar losses for IBMI and CEBMI in Appendix E.5 and provide a pseudo code in B.2.

Original:  F ULL PROCEDURE WITH M MEMBERS
Revised: 2.C F ULL P ROCEDURE WITH M M EMBERS

Original: We expand our objective for an ensemble with M > 2 members.
Revised: We expand our objective for an ensemble with M > 2 members.

Original: We only consider pairwise interactions for simplicity to keep quadratic rather than exponential growth in number of components and leave the analysis of higher order interactions, such as I ( Z i ; Z j , Z k | Y ) , for future work.[Equation]
Revised: We only consider pairwise interactions for simplicity to keep quadratic rather than exponential growth in number of components and leave the analysis of higher order interactions, e.g. I ( Z i ; Z j , Z k | Y ) , for future work.

Original: Justiﬁed by previous variationallower bounds and neural estimations, wejointly train θ m = { e m , b m , c m } , m ∈ { 1 , . . . 
Revised: Driven by previous variational and neural estimations, we train θ m = { e m , b m , c m } , m ∈ { 1 , . . . , M } on: L DICE ( [Equation]

Original: , M } on L DICE ( θ 1: M ) (see Appendix E.6) while training adversarially w on L ce .
Revised: [Equation] M ) = [Equation] ) while training adversarially w on L ce .

Original: B j(resp. B p ) samples from the concatenation of p ( z m , z m (cid:48) , y ) (resp. p ( z m , y ) p ( z m (cid:48) | y )) where m (cid:54) = m (cid:48) .
Revised: B j samples from the concatenation of p ( z m , z m (cid:48) , y ) where m (cid:54) = m (cid:48) , while B p samples from p ( z m , y ) p ( z m (cid:48) | y ) .

Original: We use the same discriminator w for (cid:0) M 2 (cid:1) estimates for simplicity.
Revised: We use the same discriminator w for (cid:0) M 2 (cid:1) estimates to improve scalability by reducing the number of parameters to be learned.

Original:  R ELATED WORK
Revised:  R ELATED W ORK

Original: To reduce the training cost of deep ensembles (Hansen & Salamon, 1990; Lakshminarayanan et al., 2016), Huang et al.
Revised: To reduce the training cost of deep ensembles (Hansen & Salamon, 1990; Lakshminarayanan et al., 2017), Huang et al.

Original: (2017) collect snapshots on training trajectories.
Revised: (2017) collect snapshots on training trajectories.

Original: One stage end-to-end codistillation (Song & Chai, 2018; Lan et al., 2018; Chen et al., 2019) share low-level features among members in branch-based ensemble while forcing each member to mimic a dynamic weighted combination of the predictions to increase individual accuracy.
Revised: One stage end-to-end codistillation (Song & Chai, 2018; Lan et al., 2018; Chen et al., 2020b) share low-level features among members in branch-based ensemble while forcing each member to mimic a dynamic weighted combination of the predictions to increase individual accuracy.

Original: However both methods correlate errors among members, homogenize predictions and fail to ﬁt the different modes of the data which overall reduce diversity.
Revised: However both methods correlate errors among members, homogenize predictions and fail to ﬁt the different modes of the data which overall reduce diversity.

Original: Beyond random initializations (Kolen & Pollack, 1991), some implicitly introduced stochasticity into the training, by providing subsets of data to learners with bagging (Breiman, 1996) or by backpropagating subsets of gradients (Lee et al., 2016): however, the reduction of training samples hurts performance for sufﬁciently complex models that overﬁt their training dataset (Nakkiran et al., 2019).
Revised: Beyond random initializations (Kolen & Pollack, 1991), authors implicitly introduced stochasticity into the training, by providing subsets of data to learners with bagging (Breiman, 1996) or by backpropagating subsets of gradients (Lee et al., 2016) ; however, the reduction of training samples hurts performance for sufﬁciently complex models that overﬁt their training dataset (Nakkiran et al., 2019).

Original: Boosting with sequential training is not suitable for deep members (Lakshminarayanan et al., 2016).
Revised: Boosting with sequential training is not suitable for deep members (Lakshminarayanan et al., 2017).

Original: Some approaches applied different data augmentations (Dvornik et al., 2019;
Revised: Some approaches applied different data augmentations (Dvornik et al., 2019;

Original: Stickland & Murray, 2020), used different networks or hyper-parameters (Singh et al., 2016;
Revised: Stickland & Murray, 2020), used different networks or hyperparameters (Singh et al., 2016;

Original: Ruiz & Verbeek, 2020; Yang & Soatto, 2020), but are not general-purpose and depend on speciﬁc engineering choices.
Revised: Ruiz & Verbeek, 2020; Yang & Soatto, 2020), but are not general-purpose and depend on speciﬁc engineering choices.

Original: Others explicitly encourage orthogonality of the gradients (Ross et al., 2019;
Revised: Others explicitly encourage orthogonality of the gradients (Ross et al., 2020;

Original: Kariyappa & Qureshi, 2019; Dabouei et al., 2020) or of the predictions , by boosting (Freund &
Revised: Kariyappa & Qureshi, 2019; Dabouei et al., 2020) or of the predictions , by boosting (Freund &

Original: Schapire, 1999; Margineantu & Dietterich) or with a negative correlation regularization (Shui et al., 2018), but they reduce members accuracy.
Revised: Schapire, 1999; Margineantu & Dietterich) or with a negative correlation regularization (Shui et al., 2018), but they reduce members accuracy.

Original: As far as we know, adaptive diversity promoting (ADP) (Pang et al., 2019) is the unique approach that performs better than the independent baseline by decorrelating the non-maximal predictions.
Revised: As far as we know, adaptive diversity promoting (ADP) (Pang et al., 2019) is the unique approach that performs better than the independent baseline by decorrelating the non-maximal predictions.

Original: The limited success of these logits approaches suggests that we seek diversity in features .
Revised: The limited success of these logits approaches suggests that we seek diversity in features .

Original: Empirically we found that the increase of ( L 1 , L 2 , − cos ) distances between features (Kim et al., 2018b) reduce performance: they are not invariant to variables’ translation.
Revised: Empirically we found that the increase of ( L 1 , L 2 , − cos ) distances between features (Kim et al., 2018) reduce performance: they are not invariant to variables’ symmetry.

Original: Simultaneously to our ﬁndings, Sinha et al.
Revised: Simultaneously to our ﬁndings, Sinha et al.

Original: (2020) is somehow equivalent to our IBMI objective (see Appendix C.2) but without information bottleneck motivations for the diversity loss.
Revised: (2020) is somehow equivalent to our IBR objective (see Appendix C.2) but without information bottleneck motivations for the diversity loss.

Original: The uniqueness of mutual information (see Appendix E.2) as a distance measure between variables has been applied in countless machine learning projects, such as reinforcement learning (Kim et al., 2018a), metric learning (Kemertas et al., 2020), or evolutionary algorithms (Aguirre & Coello, 2004).
Revised: The uniqueness of mutual information (see Appendix E.2) as a distance measure between variables has been applied in countless machine learning projects, such as reinforcement learning (Kim et al., 2019a), metric learning (Kemertas et al., 2020), or evolutionary algorithms (Aguirre & Coello, 2004).

Original: Objectives are often a trade-off between (1) informativeness and (2) compression.
Revised: Objectives are often a trade-off between (1) informativeness and (2) compression.

Original: In computer vision, unsupervised deep representation learning (Hjelm et al.; Oord et al., 2018; Tian et al., 2019; Bachman et al., 2019) maximizes correlation between features and inputs following Infomax (Linsker, 1988; Bell & Sejnowski, 1995), while discarding information not shared among different views (Bhardwaj et al., 2020), or penalizing predictability of one latent dimension given the others for disentanglement (Comon, 1994; Schmidhuber, 1992;  Kingma & Welling, 2014;Higgins et al., 2016; Kim & Mnih, 2018; Blot et al., 2018).
Revised: In computer vision, unsupervised deep representation learning (Hjelm et al., 2019; van den Oord et al., 2018; Tian et al., 2020a; Bachman et al., 2019) maximizes correlation between features and inputs following Infomax (Linsker, 1988; Bell & Sejnowski, 1995), while discarding information not shared among different views (Bhardwaj et al., 2020), or penalizing predictability of one latent dimension given the others for disentanglement (Schmidhuber, 1992; Comon, 1994; Kingma & Welling, 2014; Kim & Mnih, 2018; Blot et al., 2018).

Original: The ideal level of compression is task dependent (Soatto & Chiuso, 2014).
Revised: The ideal level of compression is task dependent (Soatto & Chiuso, 2014).

Original: As a selection criterion, features should not be redundant (Battiti, 1994; Peng et al., 2005) but relevant and complementary given the task (Novovi ˇ cov ´ a et al., 2007; Brown, 2009; Brown et al., 2012).
Revised: As a selection criterion, features should not be redundant (Battiti, 1994; Peng et al., 2005) but relevant and complementary given the task (Novoviˇcov´a et al., 2007; Brown, 2009).

Original: As a learning criteria, correlations between features and inputs are minimized according tothe Information Bottleneck principle (Tishby et al., 2000; Alemi et al., 2017;Fischer, 2020; Kirsch et al., 2020; Saporta et al., 2019), while those between features and targets are maximized (LeCun et al., 2006; Qin & Kim, 2019).
Revised: As a learning criteria, correlations between features and inputs are minimized according to Information Bottleneck (Tishby, 1999; Alemi et al., 2017; Kirsch et al., 2020; Saporta et al., 2019), while those between features and targets are maximized (LeCun et al., 2006; Qin & Kim, 2019).

Original: It forces the features to ignore task-irrelevant factors (Zhao et al., 2019), to reduce overﬁtting (Alemi et al., 2018) while protecting needed information (Tian et al., 2020).
Revised: It forces the features to ignore task-irrelevant factors (Zhao et al., 2020), to reduce overﬁtting (Alemi et al., 2018) while protecting needed information (Tian et al., 2020b).

Original: Fischer & Alemi (2020) concludes in the superiority of conditionaldistributions alignment to reach the MNI with CMI regularization (Steinke & Zakynthinou, 2020).
Revised: Fischer & Alemi (2020) concludes in the superiority of conditional alignment to reach the MNI point.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we present our experimental results on the CIFAR-10 and CIFAR-100 (Krizhevsky et al., 2009)classiﬁcation datasets.
Revised: In this section, we present our experimental results on the CIFAR-10 and CIFAR-100 (Krizhevsky et al., 2009) datasets.

Original: We detail our implementation in Appendix B. We took most hyper-parameters from Chen et al.
Revised: We detail our implementation in Appendix B. We took most hyperparameter values from Chen et al.

Revised: (2020b).

Original: Hyper-parameters for adversarial training and information bottleneck were ﬁne-tuned on a validation dataset made of 5% of the training dataset, see Appendix D.1.
Revised: Hyperparameters for adversarial training and information bottleneck were ﬁne-tuned on a validation dataset made of 5% of the training dataset, see Appendix D.1.

Original: Bold highlights best score.
Revised: Bold highlights best score.

Original: We improve accuracy, calibration, uncertainty estimates in-domain and out-domain, out-of-distribution detection and co-distillation performances.
Revised: First, we show gain in accuracy. Then, we further analyze our strategy’s impacts on calibration, uncertainty estimation, out-of-distribution detection and co-distillation.

Original: 4. C OMPARISON OF CLASSIFICATION ACCURACY
Revised: 4.A C OMPARISON OF C LASSIFICATION A CCURACY

Original: [Table]
Revised: [Table]

Original: Table 1 reports the Top-1 classiﬁcation accuracy averaged over 3 runs with standard deviation.
Revised: Table 1 reports the Top-1 classiﬁcation accuracy averaged over 3 runs with standard deviation.

Original: DICE surpasses concurrent approaches (summarized in Appendix C), including the baseline Ind. that refers to the training of  independent deterministic members without interactions between them, for ResNet and Wide-ResNet architectures, in network-based and even more in the branch-based setup. Branch-based (Song & Chai, 2018) ensemble is attractive as it reduces bias by gradient diffusion (shared layers receive various gradients), at only a slight cost in diversity which makes our approach even more valuable.
Revised: { 3,4,5 } - { branch,net } refers to the training of { 3,4,5 } members { with,without } low-level weights sharing. Ind. refers to independent deterministic deep ensembles without interactions between members (except optionally the low-level weights sharing).

Original: The more networks the better, but accuracy quickly reaches a plateau. The saturation worsens with co-distillation methods (ONE(Lan et al., 2018), OKDDip (Chen et al., 2019)) that often perform worst than Ind.: they sacriﬁce diversity for individual accuracy. On the contrary, diversity approaches more deeply beneﬁt from additional members, as shown in Figure 1 and Table 7.
Revised: DICE surpasses concurrent approaches (summarized in Appendix C) for ResNet and Wide-ResNet architectures, in network and even more in branch setup.

Original: We bring  systematic improvement to the current state-of-the-art ADP (Pang et al., 2019): e.g. on CIFAR-100, { +0 . 52 , +0 . 30 , +0 . 41 } for { 3,4,5 } -branches ResNet-32, { +0 . 94 , +0 . 53 } for { 3,4 } -branches ResNet-110 and ﬁnally +0 .
Revised: We bring signiﬁcant and systematic improvements to the current state-of-the-art ADP (Pang et al., 2019): e.g., { +0 . 52 , +0 . 30 , +0 . 41 } for { 3,4,5 } -branches ResNet-32, { +0 . 94 , +0 . 53 } for { 3,4 } -branches ResNet-110 and ﬁnally +0 .

Original: 34 for 3-nets WRN-28-2.
Revised: 34 for 3-networks WRN-28-2.

Revised: Diversity approaches better leverage size, as shown on the main Figure 1, which is detailed in Table 7: on CIFAR-100,

Revised: DICE outperforms Ind. by { +0 . 60 , +0 . 73 , +0 . 84 } for { 3,4,5 } -branches ResNet-32.

Revised: Finally, learning only the redundancy loss without compression yields unstable results: CEB learns a distribution (at almost no extra cost) that stabilizes adversarial training (see Appendix E.6) through sampling, with lower standard deviation in results than IB ( β ib can hinder the learnability (Wu et al., 2019b)).

Original: 4. A BLATION STUDY
Revised: 4.B A BLATION S TUDY

Revised: Branch-based is attractive: it reduces bias by gradient diffusion among shared layers, at only a slight cost in diversity which makes our approach even more valuable.

Original: We  study the 4-branches ResNet-32 on CIFAR-100 in following experiments.
Revised: We therefore study the 4-branches ResNet-32 on CIFAR-100 in following experiments.

Original: We ablate the two components of DICE: (1) deterministic, with VIB(Alemi et al., 2017) or VCEB (Fischer, 2020) (2) no adversarial loss,  with MI or CMI loss.
Revised: We ablate the two components of DICE: (1) deterministic, with VIB or VCEB, and (2) no adversarial loss, or with redundancy, conditionally or not.

Original: We measure diversity by the ratio-error (Aksela, 2003), r = N single N shared , which computes the ratio between the number of single errors N single and of shared errors N shared .
Revised: We measure diversity by the ratio-error (Aksela, 2003), r = N single N shared , which computes the ratio between the number of single errors N single and of shared errors N shared .

Original: A higher average over the (cid:0) M 2 (cid:1) pairs means higher diversity as members are less likely to err on the same inputs.
Revised: A higher average over the (cid:0) M 2 (cid:1) pairs means higher diversity as members are less likely to err on the same inputs.

Original: Our analysis remains valid for non-pairwise  measures, see Appendix A.5.
Revised: Our analysis remains valid for non-pairwise diversity measures, analyzed in Appendix A.5.

Original: First, CEB has  higher diversity than the deterministic Ind. (see Figure 5): reducing what features share with the input automatically reduces what they share between them (see Appendix E.7). It also beneﬁts from additional randomness brought by the reparameterization trick. Moreover CEB has lower standard deviation than IB (Fischer, 2020) and is less dependent on the hyper-parameter values: β ib can hinder the learnability (Wu et al., 2019b).
Revised: In Figure 5, CEB has slightly higher diversity than Ind.: it beneﬁts from compression.

Original: ADP reaches higher diversitythan CEB but reduces individual accuracy.
Revised: ADP reaches higher diversity but sacriﬁces individual accuracies.

Revised: On the contrary, co-distillation OKDDip sacriﬁces diversity for individual accuracies.

Original: Finally, DICE performs best.
Revised: DICE curve is above all others, and notably δ cr = 0 .

Original: While CEBMI reaches same diversity, it is more damaging to individual accuracies as it forces features to lose some categorical information.
Revised: 2 induces an optimal trade-off between ensemble diversity and individual accuracies on validation. CEBR reaches same diversity with lower individual accuracies: information about Y is removed.

Original: In Figure 6, we see that starting from random initializations, diversity begins small and increases quickly. Once the ramp up is over at step 80, our approach minimizes the estimated CMI among branches andtherefore increases diversity  compared to independent learners.
Revised: Figure 6 shows that starting from random initializations, diversity begins small: DICE minimizes the estimated CR in features and increases diversity in predictions compared to CEB ( δ cr = 0 . 0 ).

Original: The effect is correlated withthe diversity coefﬁcient δ cmi : anegative value ( − 0 . 025 ) has the opposite effect while a high value ( 0 . 6 ) creates too much diversity. We chose δ cmi = 0 . 
Revised: The effect is correlated with δ cr : a high value ( 0 . 6 ) creates too much diversity. On the contrary, a negative value ( − 0 . 025 ) can decrease diversity.

Original: 2 that induces an optimal trade-off between ensemble diversity and individual accuracies on the validation dataset (see Figure 8).
Revised: Figure 8 highlights opposing dynamics in accuracies. CEBR) is learned with different δ cr (r. δ r )

Original: 4. I N - DOMAIN AND OUT - DOMAIN UNCERTAINTY ESTIMATION AND CALIBRATION
Revised: 4.C F URTHER A NALYSIS : U NCERTAINTY ESTIMATION AND C ALIBRATION

Original: Table 2 compares in-domain (CIFAR-100) and out-domain (CIFAR-100-C, with corrupted images at ﬁve different levels) performance before and after temperature scaling (procedure in Appendix

Original: D.2): indeed, Ashukha et al.
Revised: Ashukha et al.

Original: (2020) showed that “comparison of [...] ensembling methods without temperature scaling might not provide a fair ranking”.
Revised: (2019) showed that “comparison of [...] ensembling methods without temperature scaling might not provide a fair ranking”.

Revised: Therefore, Table 2 compares performances before and after temperature scaling: the procedure is explained in Appendix D.2.

Original: We recover that ensemble and IB improve robustness.
Revised: We recover that ensemble and IB improve performances.

Original: ADP performs well on out-domain because its entropy maximisation loss causes a low internal T ( ≈ 0 . 64 ). Members’s disagreement in DICE increases uncertainty estimation and calibration (see Figure 8 inAppendix A.4), notably for harder examples from CIFAR-100-C.
Revised: DICE performs best. Members’ disagreements decrease internal temperature and increase uncertainty estimation, even after temperature scaling. Calibration criteria (ECE and TACE) beneﬁt from diversity but highly depend on hyperparameters (see Figure 8 in

Revised: Appendix A.4): they do “not provide a consistent ranking” (Ashukha et al., 2019).

Original: [Table]
Revised: [Table]

Original: CIFAR-100c Acc. ↑ % - 43. 49.34 NLL ↓ 10 −(cid:55) / (cid:51) 32.7/25.9 27.8/23.9 29.7/25.
Revised: .D

Original: [-27.3/24.8 26.21/24.38 25.78/24.76 25.6/ 22.77 BS ↓ 10 −(cid:55) / (cid:51) 8.10/7.
 6.93/ 6.63 ECE ↓ 10 −(cid:55) / (cid:51) 27.9/13.8 23.2/14.0 24.83/15.29 7.59 /16.
 21.21/14.68 21.9/14.55 20.14/14.54 18.04/14.91 18.08/ 13.47 TACE ↓ 10 −(cid:55) / (cid:51) 4.94/3.60 4.42/3.
-]
Revised: [-27.3/24.8 26.21/24.38 25.78/24.76 25.6/ 22.77 BS ↓ 10 −(cid:55) / (cid:51) 8.10/7.
 6.93/ 6.63 ECE ↓ 10 −(cid:55) / (cid:51) 27.9/13.8 23.2/14.0 24.83/15.29 7.59 /16.
 21.21/14.68 21.9/14.55 20.14/14.54 18.04/14.91 18.08/ 13.47 TACE ↓ 10 −(cid:55) / (cid:51) 4.94/3.60 4.42/3.
-][Table]

Original: 4. O UT - OF - DISTRIBUTION DETECTION
Revised: A NALYSIS : D ISCRIMINATOR B EHAVIOUR THROUGH OOD D ETECTION

Original: [Figure]
Revised: [Figure]

Original:  like an ”input-dependant temperature”.
Revised: [Figure] (AUROC ↑ ). DICE × w uses the discriminator to scale its conﬁdence: 1 − w ’s predictions behave like an ”input-dependant temperature”.

Original: To measure the ability of our ensemble to distinguish in- and out-of-distribution (OOD) images, we consider other datasets at test time following (Hendrycks & Gimpel, 2016)
Revised: To measure the ability of our ensemble to distinguish in- and out-of-distribution (OOD) images, we consider other datasets at test time following (Hendrycks & Gimpel, 2017)

Original: (see Appendix D.3).
Revised: (see Appendix D.3).

Original: The conﬁdence score is estimated with the maximum softmax value: the conﬁdence for OOD images should ideally be lower than for
Revised: The conﬁdence score is estimated with the maximum softmax value: the conﬁdence for OOD images should ideally be lower than for

Original: CIFAR-100 test images.
Revised: CIFAR-100 test images.

Original: Results are detailed in Table 5 (resp. Table 6), before (resp. after) temperature scaling (TS), and are illustrated in Figure 7.
Revised: Temperature scaling (TS, results in Table 6) reﬁnes performances (results without TS in Table 5).

Original: DICE beats Ind. and CEB.
Revised: DICE beats Ind. and CEB in both cases.

Original: We suspected that features were more correlated for OOD images as they would share  artifacts.DICE × w multiplies the classiﬁcation logits by 1 − [Equation] m
Revised: Moreover, we suspected that features were more correlated for OOD images: they may share redundant artifacts.

Original: [Equation] w ( z m , z m [Equation] , ˆ y ) ,  with predicted ˆ y (as the true y is not available at test time).
Revised: DICE × w multiplies the classiﬁcation logits by the mean over all pairs of 1 − w ( z m , z m (cid:48) , ˆ y ) , m (cid:54) = m (cid:48) , with predicted ˆ y (as the true y is not available at test time).

Original: It performs  better than traditional TS, at the cost of additional operations.
Revised: DICE × w performs even better than DICE+TS, but at the cost of additional operations.

Original: It shows the ability of w to detect conditional residual correlations, adversarially deleted only when found in training.
Revised: It shows that w can detect spurious correlations, adversarially deleted only when found in training.

Original: 4. D IVERSE TEACHER FOR IMPROVED CO - DISTILLATION
Revised: 4.E F URTHER A NALYSIS : D IVERSE T EACHER FOR I MPROVED C O - DISTILLATION

Revised: [Figure] [Table] [Figure]

Original: [Table]
Revised: [Table]

Original: The inference time in network-based ensemble grows linearly with M, limiting applicability. Sharing early-features is one solution.
Revised: The inference time in network-ensembles grows linearly with M. Sharing early-features is one solution.

Original: We experiment another solution by using only the M-th branch at test time.
Revised: We experiment another one by using only the M-th branch at test time.

Original: We combine DICE with OKDDip (Chen et al., 2019): the M-th branch  learns to mimic the soft predictions from the M-1 ﬁrst branches, among which we enforce diversity.
Revised: We combine DICE with OKDDip (Chen et al., 2020b): the M-th branch (= the student) learns to mimic the soft predictions from the M-1 ﬁrst branches (= the teacher), among which we enforce diversity.

Original: Our teacher has lower internal temperature (as shown in experiment 4.3): DICE performs best when soft predictions are generated with lowerco-distillation T .
Revised: Our teacher has lower internal temperature (as shown in Experiment 4.C): DICE performs best when soft predictions are generated with lower T .

Original: We improve sota by { +0 . 42 , +0 . 53 } for { 3,4 } -branches.
Revised: We improve state-of-the-art by { +0 . 42 , +0 . 53 } for { 3,4 } -branches.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper we addressed the task of improving deep ensembles by increased diversity.
Revised: In this paper, we addressed the task of improving deep ensembles’ learning strategies.

Original: Motivated by  information theoretic arguments, we derive a novel adversarial  loss, based on conditional mutual information estimation.
Revised: Motivated by arguments from information theory, we derive a novel adversarial diversity loss, based on conditional mutual information.

Original: We highlight the trade-off between individual accuracy and ensemble diversity  and surpass state-of-the-art performances on standard image classiﬁcation benchmarks.
Revised: We tackle the trade-off between individual accuracies and ensemble diversity by deleting spurious and redundant correlations. We reach state-of-the-art performance on standard image classiﬁcation benchmarks.

Original: As the success of many real-world systems in production depends on the robustness oftheir deep learning systems, we hope to pave the way towards general-purpose methods for diverse & better ensemble.
Revised: Finally, the success of many real-world systems in production depends on the robustness of deep ensembles: we hope to pave the way towards general-purpose strategies that go beyond independent learning.

Original: Appendices
Revised: Appendices

Original: The structure of the Appendix is as follows.
Revised: The structure of the Appendix is as follows.

Original: Appendix A shows additional experiments.
Revised: Appendix A shows additional experiments.

Original: Then in
Revised: Then in

Original: Appendix B, we give a detailed description of our implementation to facilitate reproduction.
Revised: Appendix B, we give a detailed description of our implementation to facilitate reproduction.

Original: In Appendix C, we summarize the concurrent approaches (see Table 8).
Revised: In Appendix C, we summarize the concurrent approaches (see Table 8).

Original: In Appendix D, we describe the datasets and the metrics used in our experiments.
Revised: In Appendix D, we describe the datasets and the metrics used in our experiments.

Original: Appendix E clariﬁes certain theoretical formulations.
Revised: Appendix E clariﬁes certain theoretical formulations.

Original: We conclude with a sociological analogy in Appendix F.
Revised: We conclude with a sociological analogy in Appendix F.

Original: A A DDITIONAL E XPERIMENTS A. C OMPARISONS WITH CO - DISTILLATION AND SNAPSHOT - BASED APPROACHES
Revised: A A DDITIONAL E XPERIMENTS A. C OMPARISONS WITH C O - DISTILLATION AND S NAPSHOT - BASED A PPROACHES

Original: al

Original: [Table]
Revised: [Table]

Original: A. O UT - OF - DISTRIBUTION D ETECTION
Revised: A. O UT -O F -D ISTRIBUTION D ETECTION

Original: Table 5 summarizes our OOD experiments in the 4-branches ResNet-32 setup.
Revised: Table 5 summarizes our OOD experiments in the 4-branches ResNet-32 setup.

Original: We recover that
Revised: We recover that

Original: IB improves OOD detection (Alemi et al., 2018).
Revised: IB improves OOD detection (Alemi et al., 2018).

Original: Moreover, we empirically validate our intuition: features for in-distribution images are in average less predictive from each other compared to pairs of features of OOD images.
Revised: Moreover, we empirically validate our intuition: features from in-distribution images are in average less predictive from each other compared to pairs of features from OOD images.

Original: w can perform alone as a OOD-detector [Table]
Revised: w can perform alone as a OOD-detector, but is best used in complement to DICE.

Original: but is best used in complement to DICE: in DICE × w , logitsfrom DICE are multiplied by the sigmoid output of w averaged over all pairs.
Revised: In DICE × w , logits are multiplied by the sigmoid output of w averaged over all pairs.

Original: Table 6 shows that temperature scaling improves all approaches but do not modify the ranking.
Revised: Table 6 shows that temperature scaling improves all approaches without modifying ranking.

Original: Finally [Table] DICE × w  without temperature scaling is better than DICE with temperature scaling.
Revised: Finally, DICE × w , even without temperature scaling, is better than DICE, even with temperature scaling.

Original: [Figure]
Revised: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table][Figure] [Table] [Figure]
Revised: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: A. A CCURACY VERSUS S IZE
Revised: A. A CCURACY V ERSUS S IZE

Original: We recover from Table 7 the Memory Split Advantage (MSA) from Chirkova et al.
Revised: We recover from Table 7 the Memory Split Advantage (MSA) from Chirkova et al.

Original: (2020): splitting the memory budget between three branches of ResNet-32 results in better performance than spending twice the budget on one ResNet-110.
Revised: (2020): splitting the memory budget between three branches of ResNet-32 results in better performance than spending twice the budget on one ResNet-110.

Original: DICE further improves this advantage.
Revised: DICE further improves this advantage.

Original: Our framework is particularly effective in the branch-based setting, as it reduces the computational overhead (especially in terms of FLOPS) at a slight cost in diversity.
Revised: Our framework is particularly effective in the branch-based setting, as it reduces the computational overhead (especially in terms of FLOPS) at a slight cost in diversity.

Original: A 4-branches DICE ensemble has the same accuracy in average than a classical 7-branches ensemble.
Revised: A 4-branches DICE ensemble has the same accuracy in average as a classical 7-branches ensemble.

Original:  [Table][Figure] [Figure]
Revised: phenomenon [Table]

Original:  T RAINING DYNAMICS IN TERMS OF ACCURACY , UNCERTAINTY ESTIMATION AND CALIBRATION
Revised: A. T RAINING D YNAMICS IN TERMS OF A CCURACY , U NCERTAINTY E STIMATION AND C ALIBRATION

Original: We measured diversity in 4.2 with the ratio error (Aksela, 2003).
Revised: We measured diversity in 4.B with the ratio error (Aksela, 2003).

Original: But as stated by Kuncheva &
Revised: But as stated by Kuncheva &

Original: Whitaker (2003), diversity can be measured in numerous ways.
Revised: Whitaker (2003), diversity can be measured in numerous ways.

Original: For pairwise measures, we averaged over the (cid:0) M 2 (cid:1) pairs: the Q-statistics is positive when classiﬁers recognize the same object, the agreement score measures the frequency that both classiﬁers predict the same class.
Revised: For pairwise measures, we averaged over the (cid:0) M 2 (cid:1) pairs: the Q-statistics is positive when classiﬁers recognize the same object, the agreement score measures the frequency that both classiﬁers predict the same class.

Original: Note that even if we only apply pairwise constraints, we also increase non-pairwise measures: for example, the
Revised: Note that even if we only apply pairwise constraints, we also increase non-pairwise measures: for example, the

Original: Kohavi-Wolpert variance (Kohavi et al., 1996) which measures the variability of the predicted class, and the entropy diversity which measures overall disagreement.
Revised: Kohavi-Wolpert variance (Kohavi et al., 1996) which measures the variability of the predicted class, and the entropy diversity which measures overall disagreement.

Original: B T RAINING D ETAILS B. G ENERAL O PTIMIZATION
Revised: B T RAINING D ETAILS B. G ENERAL O PTIMIZATION

Original: Experiments We conducted all experiments in Pytorch 1.4 (Paszke et al., 2019) on an NVIDIA Tesla V100 SXM2 32 GB. Our code was forked from https://github.com/valeoai/ ConfidNet (Corbi ` ere et al., 2019) and will be released upon acceptance.
Revised: Experiments Our code, in Pytorch 1.4 (Paszke et al., 2017), was forked from https:// github.com/valeoai/ConfidNet (Corbi`ere et al., 2019) and will be released upon acceptance.

Original: Classical hyperparameters were taken from (Chen et al., 2019) for conducting fair comparisons.
Revised: Classical hyperparameters were taken from (Chen et al., 2020b) for conducting fair comparisons.

Original: Newly added hyper-parameters were ﬁne-tuned on a validation dataset made of 5% of the training dataset.
Revised: Newly added hyperparameters were ﬁne-tuned on a validation dataset made of 5% of the training dataset.

Original: Architecture We implemented the proposed method with ResNet (He et al.) and Wide-ResNet (Zagoruyko & Komodakis, 2016) architectures.
Revised: Architecture We implemented the proposed method with ResNet (He et al., 2016) and WideResNet (Zagoruyko & Komodakis, 2016) architectures.

Original: Following standard practices, we average the log- its of our predictions uniformly.
Revised: Following standard practices, we average the logits of our predictions uniformly.

Original: For branch-based ensemble, we separate the last block and the classiﬁer of each member from the weights sharing while the other low-level layers were shared.
Revised: For branch-based ensemble, we separate the last block and the classiﬁer of each member from the weights sharing while the other low-level layers were shared.

Original: Learning Following (Chen et al., 2019), we used SGD with Nesterov with momentum of 0.9, mini-batch size of 128, weight decay of 5e-4, 300 epochs, a standard learning rate scheduler that sets values { 0 . 1 , 0 .
Revised: Learning Following (Chen et al., 2020b), we used SGD with Nesterov with momentum of 0.9, mini-batch size of 128, weight decay of 5e-4, 300 epochs, a standard learning rate scheduler that sets values { 0 . 1 , 0 .

Original: 001 , 0 . 0001 } at steps { 0 , 150 , 225 } for CIFAR-10/100.
Revised: 001 , 0 . 0001 } at steps { 0 , 150 , 225 } for CIFAR-10/100.

Original: In CIFAR-100, we additionally set the learning rate at 0.00001 at step 250.
Revised: In CIFAR-100, we additionally set the learning rate at 0.00001 at step 250.

Original: We used  basic data augmentation that consists of horizontal ﬂips and a random crop of 32 pixels with a padding of 4 pixels 1 , but other augmentations would improve overall performance.
Revised: We used traditional basic data augmentation that consists of horizontal ﬂips and a random crop of 32 pixels with a padding of 4 pixels.

Original: The learning curve is shown on Figure 8.
Revised: The learning curve is shown on Figure 8.

Original: B. P SEUDO CODE
Revised: B. P SEUDO -C ODE

Original: Algorithm

Original: /* Setup */
Revised: /* Setup */

Original: Parameters: θ 1 = { e 1 , b 1 , c 1 } , θ 2 = { e 2 , b 2 , c 2 } and discriminator w , randomly initialized
Revised: Parameters: θ 1 = { e 1 , b 1 , c 1 } , θ 2 = { e 2 , b 2 , c 2 } and discriminator w , randomly initialized

Original: Input: Observations { x n , y n } Nn =1 , coefﬁcients β ceb and δ cmi , schedulings sche ceb and rampup endstepstartstep , clipping threshold τ , batch size b , optimisers g θ 1 , 2 and g w , number of discriminators step nstep d , number of samples num s , ratio of positive/negative sample ratio negpos /* Training procedure */ 1 for s ← 1 to 300 do β sceb ← sche ceb ( startvalue= 0 , endvalue= β ceb , step= s ) δ scmi ← rampup 800 ( startvalue= 0 , endvalue= δ cmi , step= s )
Revised: Input: Observations { x n , y n } Nn =1 , coefﬁcients β ceb and δ cr , schedulings sche ceb and rampup endstepstartstep , clipping threshold τ , batch size b , optimisers g θ 1 , 2 and g w , number of discriminators step nstep d , number of samples num s , ratio of positive/negative sample ratio negpos /* Training Procedure */ 1 for s ← 1 to 300 do β sceb ← sche ceb ( startvalue= 0 , endvalue= β ceb , step= s ) δ scr ← rampup 800 ( startvalue= 0 , endvalue= δ cr , step= s )

Original: Randomly select batch { ( x n , y n ) } n ∈B of size b // Batch sampling /
Revised: Randomly select batch { ( x n , y n ) } n ∈B of size b // Batch Sampling /

Original: * Step 1: classification loss */ for i ← 1 to 2 do z ni 
Revised: * Step 1: Classification Loss with CEB */ for i ← 1 to 2 do z ni ← e µi ( z | x n ) + (cid:15)e σi ( z | x n ) , ∀ n ∈ B

Original: [Table]
Revised: [Table]

Original: z ni } /*
Revised: ∼ N (

Original: [Table]
Revised: [Table]

Original: diversity loss */
Revised: VCEB ← 1 b (cid:80) n ∈B { 1 β sceb D KL ( e i ( z | x n ) (cid:107) b i ( z | y n )) − log c i ( y n | z ni } /*

Original: [Table]
Revised: [Table]

Original: i
Revised: Diversity

Original: [Table]
Revised: [Table]

Original: do
Revised: Conditional Redundancy */

Original: [Table] [Table]
Revised: [Table] [Table]

Original: do

Original: [Table]
Revised: [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: num

Original: [Table]
Revised: [Table]

Original:  // Joint distrib.
Revised: num s } // Joint Distrib.

Original: [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table] [Table] [Table]
Revised: // Backprop Ensemble /

Original: [Table]
Revised: [Table]

Original: [Table] [Table] [Table] [Table]
Revised: Training */ s } //

Original: // Joint distrib.
Revised: Joint Distrib.

Original: [Table] [Table] [Table] [Table]
Revised: ∀ k ∈ { 1 , . .

Original: num s } , k (cid:48) ∈ { 1 , . .
Revised: num s } , k (cid:48) ∈ { 1 , . .

Original: . , ratio negpos } with n (cid:

Revised: . , ratio negpos } with n (cid:

Original: ) ∈ B , y n = y n (cid:48) , n (cid:54) = n (cid:48) //
Revised: ) ∈ B , y n = y n (cid:48) , n (cid:54) = n (cid:48) //

Original: Product distributionw ← g w ( ∇ w L ce ( w )) // Back. adv. Sample new z ni,k /* Test procedure */
Revised: Product distribution

Original: Data: Inputs { x n } Tn = //
Revised: Data: Inputs { x n } Tn = //

Original: Test data
Revised: Test Data

Original: Output: arg max k ∈{ 1 ,...,K } ( 1 2 [ c 1 ( e µ 1 ( z | x n ))
Revised: Output: arg max k ∈{ 1 ,...,K } ( 12 [ c 1 ( e µ 1 ( z | x n ))

Original: + c 2 ( e µ 2 ( z | x n ))]) , ∀ n ∈ { 1 , . .
Revised: + c 2 ( e µ 2 ( z | x n ))]) , ∀ n ∈ { 1 , . .

Original: , T }
Revised: , T }

Original: B. I NFORMATION B OTTLENECK DETAILS
Revised: B. I NFORMATION B OTTLENECK I MPLEMENTATION

Original: Architecture We use the last feature map just before the dense layer since deeper layers are more semantics, of size d = { 64 , 128 , 256 } for { ResNet-32, WRN-28-2, ResNet-110 } .
Revised: Architecture Features are extracted just before the dense layer since deeper layers are more semantics, of size d = { 64 , 128 , 256 } for { ResNet-32, WRN-28-2, ResNet-110 } .

Original: Our encoder does not provide a deterministic point in the features space but a feature distribution encoded bya mean anda diagonal covariance matrix.
Revised: Our encoder does not provide a deterministic point in the features space but a feature distribution encoded by mean and diagonal covariance matrix.

Original: The covariance is predicted after a Softplus activation function with an additional dense layer, taking as input the features mean,therefore with d ( d + 1) trainable weights.
Revised: The covariance is predicted after a Softplus activation function with one additional dense layer, taking as input the features mean, with d ( d + 1) trainable weights.

Original: In training we sample once from this features’ distribution with the reparameterization trick.
Revised: In training we sample once from this features distribution with the reparameterization trick.

Original: In inference, we predict from the distribution’s mean (and therefore only once).
Revised: In inference, we predict from the distribution’s mean (and therefore only once).

Original: We parameterized b ( z | y ) ∼ N ( b µ ( y ) , 1 ) witha trainable mean andan unit diagonal covariance, with d additional trainable weights per class.
Revised: We parameterized b ( z | y ) ∼ N ( b µ ( y ) , 1 ) with trainable mean and unit diagonal covariance, with d additional trainable weights per class.

Original: As noticed in (Fischer & Alemi, 2020), this can be represented as a single embedding layer mapping one-hot classes to d -dimensional tensors.
Revised: As noticed in (Fischer & Alemi, 2020), this can be represented as a single embedding layer mapping one-hot classes to d -dimensional tensors.

Original: Therefore in total we only add d ( d + 1 + K ) trainable weights, that all can be discarded during inference.
Revised: Therefore in total we only add d ( d + 1 + K ) trainable weights, that all can be discarded during inference.

Original: For VIB, the embedding b µ is shared among classes: in total it adds d ( d +2) trainable weights.
Revised: For VIB, the embedding b µ is shared among classes: in total it adds d ( d + 2) trainable weights.

Original: Contrary to recent IB approaches (Wu et al., 2019b; Wu & Fischer, 2020; Fischer & Alemi, 2020), we only have one dense layer to predict logits after the features bottleneck, and we did not change the batch normalization, for fair comparisons with traditional ensemble methods.
Revised: Contrary to recent IB approaches (Wu et al., 2019b; Wu & Fischer, 2020; Fischer & Alemi, 2020), we only have one dense layer to predict logits after the features bottleneck, and we did not change the batch normalization, for fair comparisons with traditional ensemble methods.

Original: Scheduling We employ the jump-start method that facilitates the learning of bottleneck-inspired models (Wu et al., 2019b; Wu & Fischer, 2020; Fischer & Alemi, 2020): we progressively anneal the value of β ceb .
Revised: Scheduling We employ the jump-start method that facilitates the learning of bottleneck-inspired models (Wu et al., 2019b; Wu & Fischer, 2020; Fischer & Alemi, 2020): we progressively anneal the value of β ceb .

Original: For CIFAR-10, log( β ceb ) reaches values { 100 , 10 , 2 } at steps { 0 , 5 , 100 } .
Revised: For CIFAR-10, log( β ceb ) reaches values { 100 , 10 , 2 } at steps { 0 , 5 , 100 } .

Original: For
Revised: For

Original: CIFAR-100, log( β ceb ) reaches values { 100 , 10 , 2 , 1 . 5 , 1 } at steps { 0 , 8 , 175 , 250 , 300 } .
Revised: CIFAR-100, log( β ceb ) reaches values { 100 , 10 , 2 , 1 . 5 , 1 } at steps { 0 , 8 , 175 , 250 , 300 } .

Revised: This slow scheduling increases progressively the covariance predictions e σ ( x ) and facilitates learning.

Original: For VIB, we scheduled similarly using the equivalence from (Fischer, 2020): log( β ib )=
Revised: For VIB, we scheduled similarly using the equivalence from (Fischer, 2020): log( β ib )

Original:  log( β ceb − 1) .
Revised: = log( β ceb − 1) .

Original: This slow ramp up is critical to progressively increase the covariance predictions e σ ( x ) , facilitate learning and reduce performances’ variance.

Original: Otherwise e σ ( x ) would early reach 1 and the high variance in the reparameterization trick would cause underﬁtting. We found VCEB to be less dependent on the hyper-parameters choices than VIB.
Revised: We found VCEB to be less dependent on the hyperparameter choices than VIB and have lower standard deviation in performances: β ib can hinder the learnability (Wu et al., 2019b).

Original: B. D ISCRIMINATOR DETAILS
Revised: B. A DVERSARIAL T RAINING I MPLEMENTATION

Original: Mutual Information Following standard adversarial learning practises, our discriminator for MI estimation is a MLP with 4 layers of size { 256, 256, 100, 1 } , with leaky-ReLus of slope 0.2, optimized by RMSProp with learning rate { 0 . 003 , 0 . 005 } for CIFAR- { 10, 100 } .
Revised: Redundancy Following standard adversarial learning practices, our discriminator for redundancy estimation is a MLP with 4 layers of size { 256, 256, 100, 1 } , with leaky-ReLus of slope 0.2, optimized by RMSProp with learning rate { 0 . 003 , 0 . 005 } for CIFAR- { 10, 100 } .

Original: We empirically found that four steps for the discriminator for one step of the classiﬁer increases stability.
Revised: We empirically found that four steps for the discriminator for one step of the classiﬁer increase stability.

Original: Speciﬁcally, it takes as input the concatenation of the two hidden representations of size d , sampled with a reparameterization trick.
Revised: Speciﬁcally, it takes as input the concatenation of the two hidden representations of size d , sampled with a reparameterization trick.

Original: Gradients are not back-propagated in the layer that predicts the covariance, as it would artiﬁcially increase the variance to reduce the  informationshared among branches.
Revised: Gradients are not backpropagated in the layer that predicts the covariance, as it would artiﬁcially increase the covariance to reduce the mutual information among branches.

Original: The output, followed by a sigmoid activation function, should be close to 1 (resp. 0) if the sample comes from the joint (resp. product) distribution.
Revised: The output, followed by a sigmoid activation function, should be close to 1 (resp. 0) if the sample comes from the joint (resp. product) distribution.

Original: Conditional Mutual Information In order to take into account the category information, the discriminator for CMI estimation needs to take into account he class information.
Revised: Conditional Redundancy The discriminator for CR estimation needs to take into account the target class Y .

Original: It ﬁrst embeds Y in an embedding layer of size 64, which is concatenated at the inputs of the ﬁrst and second layers.
Revised: It ﬁrst embeds Y in an embedding layer of size 64, which is concatenated at the inputs of the ﬁrst and second layers.

Original: Improved features merging method could be applied, such as (Ben-Younes et al., 2019).
Revised: Improved features merging method could be applied, such as Ben-Younes et al.

Original: The output has size K , and we select the index associated with the sample’s class.
Revised: The output has size K , and we select the index associated with the Y .

Original: We note in Figure 11 that our discriminator remains calibrated.
Revised: We note in Figure 11 that our discriminator remains calibrated.

Original: Ensemble with M Models In the general case, we only consider pairwise interactions, therefore we need to estimate (cid:0) M 2 (cid:1) values.
Revised: Ensemble with M Models In the general case, we only consider pairwise interactions, therefore we need to estimate (cid:0) M 2 (cid:1) values.

Original: To reduce the number of parameters, we use only one discriminator w .
Revised: To reduce the number of parameters, we use only one discriminator w .

Original: We ﬁll with zeros the features associated with z k  when we sample from p ( z i , z j , y ) or from p ( z i , y ) p ( z j | y ) , where i, j, k ∈ { 1 , . . .
Revised: Features z k with zeros when we sample from p ( z i , z j , y ) or from p ( z i , y ) p ( z j | y ) , where i, j, k ∈ { 1 , . . , M } , k (cid:54) =

Original: , M } .
Revised: and

Original: Adversarial Training We follow the traditional ramp-up (Lan et al., 2018; Kim et al., 2019; Chen et al., 2019) scheduling up to step 80 before reaching the ﬁnal value of δ cmi .
Revised: (cid:54) =

Original: The optimal valuedepends on the diversity-accuracy trade-off. For branch-based and network-based CIFAR-100, we set δ cmi at { 0 . 1 , 0 . 15 , 0 . 2 , 0 . 22 , 0 .25 } for { 2, 3, 4, 5, 6 } members.
Revised: j δ cr value For branch-based and network-based CIFAR- , we found δ cr at { 0 . 1 , 0 . 15 , 0 . 2 , 0 . 22 , 0 . for { 2, 3, 4, 5, 6 } members to perform best on the validation dataset when training on 95% on the classical training dataset.

Original: For CIFAR10, { 0 . 1 } for 4 members.
Revised: For CIFAR-10, { 0 . 1 } for 4 members.

Original: We found that lower values of δ mi were necessary for our baselines IBMI and CEBMI.
Revised: We found that lower values of δ r were necessary for our baselines IBR and CEBR.

Revised: Scheduling For fair comparison, we apply the traditional rampup scheduling up to step 80 from the co-distillation literature (Lan et al., 2018;

Revised: Kim et al., 2019b; Chen et al., 2020b) to all concurrent approaches and to our redundancy training.

Original: Sampling To sample from p ( z 1 , z 2 , y ) , we select features extracted from one image.
Revised: Sampling To sample from p ( z 1 , z 2 , y ) , we select features extracted from one image.

Original: To sample from p ( z 1 , y ) p ( z 2 | y ) , we select features extracted from two different inputs, that share the same class.
Revised: To sample from p ( z 1 , y ) p ( z 2 | y ) , we select features extracted from two different inputs, that share the same class y .

Original: In practise, we keep a memory from previous batches as the batch size is 128 whereas we have 100 classes in CIFAR-100.
Revised: In practise, we keep a memory from previous batches as the batch size is 128 whereas we have 100 classes in CIFAR-100.

Original: Our sampling is a special case of k -NN sampling (Molavipour et al., 2020): as we sample from a discrete categorical variable, the closest neighbour has exactly the same discrete value.
Revised: Our sampling is a special case of k -NN sampling (Molavipour et al., 2020): as we sample from a discrete categorical variable, the closest neighbour has exactly the same discrete value.

Original: The training can be unstable as it minimises the divergence between two distributions.
Revised: The training can be unstable as it minimises the divergence between two distributions.

Original: To make them overlap over the features space, we sample num sample = { 4 } times from the gaussian distribution of Z 1 and Z 2 with the reparameterization trick.
Revised: To make them overlap over the features space, we sample num sample = { 4 } times from the gaussian distribution of Z 1 and Z 2 with the reparameterization trick.

Original: This procedure is similar to instance noise (Sønderby et al., 2016) and it allows us to safely optimise w at each iteration.
Revised: This procedure is similar to instance noise (Sønderby et al., 2016) and it allows us to safely optimise w at each iteration.

Original: It gives better robustness than just giving the gaussian mean.
Revised: It gives better robustness than just giving the gaussian mean.

Original: Moreover, we progressively ease the discriminator task by scheduling the variance through time with a linear rampup.
Revised: Moreover, we progressively ease the discriminator task by scheduling the covariance through time with a linear ramp-up.

Original: First the variance is set to 1 until epoch 100, then it linearly reduces to the predicted variance e σ ( x ) until step 250.
Revised: First the covariance is set to 1 until epoch 100, then it linearly reduces to the predicted covariance e σ ( x ) until step 250.

Original: We sample a ratio ratio negpos of one positive pair for { 2 , 4 } negative pairs on CIFAR- { 10 , 100 } 
Revised: We sample a ratio ratio negpos of one positive pair for { 2 , 4 } negative pairs on CIFAR- { 10 , 100 } .

Original: Clipping Following Bachman et al.
Revised: Clipping Following Bachman et al.

Original: (2019), we clip the density ratios ( tanhclip ) by computing the non linearity exp[ τ tanh log[ f ( z 1 ,z 2 ,y )] τ ] .
Revised: (2019), we clip the density ratios ( tanhclip ) by computing the non linearity exp[ τ tanh log[ f ( z 1 ,z 2 ,y )] τ ] .

Original: A lower τ reduces the variance of the estimation and stabilizes the training even with a strong discriminator, at the cost of increasing bias.
Revised: A lower τ reduces the variance of the estimation and stabilizes the training even with a strong discriminator, at the cost of additional bias.

Original: The clipping threshold τ was set to 10 like in Song & Ermon (2020).
Revised: The clipping threshold τ was set to 10 as in Song & Ermon (2020).

Original: Empirical Limitations Our approach relies on very recent works in neural network estimation of mutual information, that still suffer fromhigh variance and loose approximations.
Revised: Our approach relies on very recent works in neural network estimation of mutual information, that still suffer from loose approximations.

Original: Improvements in this area woulddeeply facilitate our learning procedure.
Revised: Improvements in this area would facilitate our learning procedure.

Original: Our approach increases the number of operations because of the adversarial procedure, but only during training: the inference time remainsexactly the same.
Revised: Our approach increases the number of operations because of the adversarial procedure, but only during training: the inference time remains the same.

Original: C C ONCURRENT APPROACHES
Revised: C C ONCURRENT A PPROACHES

Original: [Table]

Original: Both

Original: [Table]

Original: Both ?

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table] [Figure] [Table] Concurrent approaches can be divided in two general patterns: they promote either individual accuracy by co-distillation either ensemble diversity
Revised: Concurrent approaches can be divided in two general patterns: they promote either individual accuracy by co-distillation either ensemble diversity.

Original: C. C O - DISTILLATION APPROACHES
Revised: C. C O - DISTILLATION A PPROACHES

Original: Contrary to the traditional distillation (Hinton et al., 2015) that aligns the soft prediction between a static pre-trained strong teacher towards a smaller student, online co-distillation performs teaching in an end-to-end one-stage procedure: the teacher and the student are trained simultaneously.
Revised: Contrary to the traditional distillation (Hinton et al., 2015) that aligns the soft prediction between a static pre-trained strong teacher towards a smaller student, online co-distillation performs teaching in an end-to-end one-stage procedure: the teacher and the student are trained simultaneously.

Original: Distillation in logits The seminal ”Deep Mutual Learning” (DML) (Zhang et al., 2018) introduced the main idea: multiple networks learn to mimic each other by reducing KL-losses between pairs of predictions. ” 
Revised: Distillation in Logits The seminal ”Deep Mutual Learning” (DML) (Zhang et al., 2018) introduced the main idea: multiple networks learn to mimic each other by reducing KL-losses between pairs of predictions. ” Collaborative learning for deep neural networks” (CL-ILR) (Song & Chai, [Table]

Original: Collaborative learning for deep neural networks” (CL-ILR) (Song & Chai, 2018) used the branch-based architecture by sharing low-level layers to reduce the training complexity, and ”Knowledge Distillation by On-the-Fly Native Ensemble” (ONE) (Lan et al., 2018) used a weighted combination of logits as teacher hence providing better information to each network. ”
Revised: 2018) used the branch-based architecture by sharing low-level layers to reduce the training complexity [Table] and ”Knowledge Distillation by On-the-Fly Native Ensemble” (ONE) (Lan et al., 2018) used a weighted combination of logits as teacher hence providing better information to each network. ”

Original: Online Knowledge Distillation via Collaborative Learning” (KDCL)
Revised: Online Knowledge Distillation via Collaborative Learning” (KDCL)

Original: (Guo et al.) computed the optimum weight on an held-out validation dataset. ”
Revised: (Guo et al.) computed the optimum weight on an held-out validation dataset. ”

Original: Feature Fusion for Online Mutual Knowledge Distillation” (FFL) (Kim et al., 2019) introduced a feature fusion module.
Revised: Feature Fusion for Online Mutual Knowledge Distillation” (FFL) (Kim et al., 2019b) introduced a feature fusion module.

Original: These approaches improve individual performance at the cost of increased homogenization. ”
Revised: These approaches improve individual performance at the cost of increased homogenization. ”

Original: Online Knowledge Distillation with Diverse Peers” (OKDDip) (Chen et al., 2019) slightly alleviates this problem with an asymmetric distillation and a self-attention mechanism. ”
Revised: Online Knowledge Distillation with Diverse Peers” (OKDDip) (Chen et al., 2020b) slightly alleviates this problem with an asymmetric distillation and a self-attention mechanism. ”

Original: Peer Collaborative Learning for Online
Revised: Peer Collaborative Learning for Online

Original: Knowledge Distillation” (PCL) (Wu & Gong, 2020) beneﬁted from the mean-teacher paradigm with temporal ensembling and from diverse data augmentation, at the cost of multiple inferences through the shared backbone.
Revised: Knowledge Distillation” (PCL) (Wu & Gong, 2020) beneﬁted from the mean-teacher paradigm with temporal ensembling and from diverse data augmentation, at the cost of multiple inferences through the shared backbone.

Original: Distillation in features Whereas all previous approaches only apply distillation on the logits, the recent ”Feature-map-level Online Adversarial Knowledge Distillation” (AFD) (Chung et al., 2020) aligned features distributions by adversarial training.
Revised: Distillation in Features Whereas all previous approaches only apply distillation on the logits, the recent ”Feature-map-level Online Adversarial Knowledge Distillation” (AFD) (Chung et al., 2020) aligned features distributions by adversarial training.

Original: Note that this is not opposite to our approach, as they force distributions to be similar while we force them to be uncorrelated.
Revised: Note that this is not opposite to our approach, as they force distributions to be similar while we force them to be uncorrelated.

Original: C. D IVERSITY APPROACHES
Revised: C. D IVERSITY A PPROACHES

Original: On the other hands, some recent papers in computer vision explicitly encourage diversity among the members with regularization losses.
Revised: On the other hands, some recent papers in computer vision explicitly encourage diversity among the members with regularization losses.

Original: Diversity in logits ”Diversity Regularization in Deep Ensembles” (Shui et al., 2018) applied negative correlation (Liu & Yao, 1999a) to regularize the training for improved calibration, with no impact on accuracy. ”
Revised: Diversity in Logits ”Diversity Regularization in Deep Ensembles” (Shui et al., 2018) applied negative correlation (Liu & Yao, 1999a) to regularize the training for improved calibration, with no impact on accuracy. ”

Original: Adaptive Diversity Promoting” (ADP) (Pang et al., 2019) decorrelates only the non-maximal predictions to maintain the individual accuracies, while promoting ensemble entropy.
Revised: Adaptive Diversity Promoting” (ADP) (Pang et al., 2019) decorrelates only the non-maximal predictions to maintain the individual accuracies, while promoting ensemble entropy.

Original: It forces different members to have different ranking of predictions among non maximal predictions.
Revised: It forces different members to have different ranking of predictions among non maximal predictions.

Original: However, Liang et al.
Revised: However, Liang et al.

Original: (2017) has shown that ranking of outputs are critical: for example, non maximal logits tend to be more separated from each other for in-domain inputs compared to out-of-domain inputs.
Revised: (2018) has shown that ranking of outputs are critical: for example, non maximal logits tend to be more separated from each other for in-domain inputs compared to out-of-domain inputs.

Original: Therefore individual accuracy are decreased.
Revised: Therefore individual accuracies are decreased.

Original: We apply the same scheduling for weighting the diversity-inducing loss in ADP and found it to stabilize their results and increase accuracy.

Original: Coefﬁcients α and β are respectively set to 2 and 0.5, as in the original paper.
Revised: Coefﬁcients α and β are respectively set to 2 and 0.5, as in the original paper.

Original: Diversity in features One could think about increasing classical distances among features like Lin (Kim et al., 2018b), but in our experiments it reduces overall accuracy: it is not even invariant to linear transformations such as translation. ”
Revised: Diversity in Features One could think about increasing classical distances among features like L 2 in (Kim et al., 2018), but in our experiments it reduces overall accuracy: it is not even invariant to linear transformations such as translation. ”

Original: Diversity inducing Information Bottleneck in Model
Revised: Diversity inducing Information Bottleneck in Model

Original: Ensembles” from Sinha et al.
Revised: Ensembles” from Sinha et al.

Original: (2020) trains a multi-branch network and applies VIB on individual branch, by encoding p ( z | y ) ∼ N (0 , 1) , which was shown to be hard to learn (Wu & Fischer, 2020).
Revised: (2020) trains a multi-branch network and applies VIB on individual branch, by encoding p ( z | y ) ∼ N (0 , 1) , which was shown to be hard to learn (Wu & Fischer, 2020).

Original: Moreover, we notice that their diversity-inducing adversarial loss is an estimation of the JSdivergence between pairs of features, built on the dual f -divergence representation (Nowozin et al., 2016): similar idea was recently used for saliency detection (Chen et al.).
Revised: Moreover, we notice that their diversity-inducing adversarial loss is an estimation of the JS-divergence between pairs of features, built on the dual f -divergence representation (Nowozin et al., 2016): similar idea was recently used for saliency detection (Chen et al., 2020a).

Original: As the JS-divergence is a symmetrical formulation of the KL, we argue that DIBS and IBMI share the sametheoretical motivations anddiverge only on few optimization tricks: the adversarial terms  with both terms sampled from the same branch and both terms sampled from the same prior.
Revised: As the JS-divergence is a symmetrical formulation of the KL, we argue that DIBS and IBR share the same motivations and only have minor discrepancies: the adversarial terms in DIBS loss with both terms sampled from the same branch and both terms sampled from the same prior.

Original: In our experiments, these differences reduce overall performance.
Revised: In our experiments, these differences reduce overall performance.

Original: We will include their scores when they publish measurable results on CIFAR datasets or when they release their code.
Revised: We will include their scores when they publish measurable results on CIFAR datasets or when they release their code.

Original: Diversity in gradients ”Improving adversarial robustness of ensembles with diversity training.”
Revised: Diversity in Gradients ” Improving adversarial robustness of ensembles with diversity training.”

Original: (GAL) (Kariyappa & Qureshi, 2019) enforced diversity in the gradients with a gradient alignment loss. ”
Revised: (GAL) (Kariyappa & Qureshi, 2019) enforced diversity in the gradients with a gradient alignment loss. ”

Original: Exploiting Joint Robustness to Adversarial Perturbations” (Dabouei et al., 2020) considered the optimal bound for the similarity of gradients.
Revised: Exploiting Joint Robustness to Adversarial Perturbations” (Dabouei et al., 2020) considered the optimal bound for the similarity of gradients.

Original: However, as stated in the latter, “promoting diversity of gradient directions slightly degrades the classiﬁcation performance on natural examples ...
Revised: However, as stated in the latter, “promoting diversity of gradient directions slightly degrades the classiﬁcation performance on natural examples ...

Original: [because] classiﬁers learn to discriminate input samples based on distinct sets of representative features”.
Revised: [because] classiﬁers learn to discriminate input samples based on distinct sets of representative features”.

Original: Therefore we do not consider them as concurrent work.
Revised: Therefore we do not consider them as concurrent work.

Original: D E XPERIMENTAL S ETUP D. T RAINING D ATASETS
Revised: D E XPERIMENTAL S ETUP D. T RAINING D ATASETS

Original: We train our procedure on two image classiﬁcation benchmarks, CIFAR-100 and CIFAR-10, (Krizhevsky et al., 2009).
Revised: We train our procedure on two image classiﬁcation benchmarks, CIFAR-100 and CIFAR-10, (Krizhevsky et al., 2009).

Original: They consist of 60k 32*32 natural and colored images in respectively 100 classes and 10 classes, with 50k training images and 10k test images.
Revised: They consist of 60k 32*32 natural and colored images in respectively 100 classes and 10 classes, with 50k training images and 10k test images.

Original: For hyper-parameters selection and ablation studies, we train on 95% of the training dataset, and analyze performances on the validation dataset made of the remaining 5%.
Revised: For hyperparameter selection and ablation studies, we train on 95% of the training dataset, and analyze performances on the validation dataset made of the remaining 5%.

Original: D. R OBUSTNESS
Revised: D. R OBUSTNESS

Original: Dataset For CIFAR-100, we evaluate results on shifted data from The Common Corruptions

Original: Benchmark (Hendrycks & Dietterich, 2019): CIFAR-100-C modiﬁes the CIFAR-100 test set withcorruptions (speckle noise, shot noise, impulse noise, defocus blur, gaussian blur, glass blur, motion blur, zoom blur, snow, fog, brightness, contrast, elastic transform, pixelate and jpeg compression) at ﬁve different levels.

Original: Note that none of these transformations are used in training.

Original: Metrics We follow the procedure from (Ashukha et al., 2020).
Revised: Metrics We follow the procedure from (Ashukha et al., 2019).

Original: To measure the calibration, i.e. how classiﬁcation conﬁdences match the observed prediction accuracy, we report the Expected Calibration Error (ECE) (Naeini et al., 2015) and the Thresholded Adaptive Calibration Error (TACE) (Nixon et al.) with 15 bins: while they both evaluate the calibration properties of our models, TACE resolve some pathologies found in ECE by thresholding and adaptive binning.
Revised: To measure the calibration, i.e., how classiﬁcation conﬁdences match the observed prediction accuracy, we report the ”Expected Calibration Error” (ECE) (Naeini et al., 2015) and the ”Thresholded Adaptive Calibration Error” (TACE) (Nixon et al.) with 15 bins: while they both evaluate the calibration properties of our models, TACE resolves some pathologies found in ECE by thresholding and adaptive binning.

Original: To evaluate the quality of the uncertainty estimates, we reported two complementary proper scoring rules (Gneiting & Raftery, 2007), the Negative Log-Likelihood (NLL) and the Brier Score (BS) (Brier, 1950).
Revised: To evaluate the quality of the uncertainty estimates, we reported two complementary proper scoring rules (Gneiting & Raftery, 2007), the ”Negative Log-Likelihood” (NLL) and the ”Brier Score” (BS) (Brier, 1950).

Original: Temperature scaling On CIFAR-100, we randomly divide the test set into two equal parts, then compute metrics for each half of the test set using the temperature optimized on another half.
Revised: Temperature Scaling On CIFAR-100, we randomly divide the test set into two equal parts, then compute metrics for each half of the test set using the temperature optimized on another half.

Original: For

Original: CIFAR-100c, we compute the temperature on the full CIFAR-100 test set.

Original: These 3 values have low variance and their mean is reported in Table 2 under the name ”Temp.
Revised: These 2 values have low variance and their mean is reported in Table 2 under the name ”Temp.

Original: avg value”.
Revised: avg value”.

Original: D. OOD
Revised: D. OOD

Original: Dataset We used the traditional out-of-distribution datasets for CIFAR-100, described in (Liang et al., 2017):
Revised: Dataset We used the traditional out-of-distribution datasets for CIFAR-100, described in (Liang et al., 2018):

Original: TinyImageNet (Deng et al., 2009), LSUN (Yu et al., 2015), iSUN(Xu et al., 2015), and CIFAR-10.
Revised: TinyImageNet (Deng et al., 2009), LSUN (Yu et al., 2015), iSUN(Xu et al., 2015), and CIFAR-10.

Original: We run all evaluations with the  code from https://github.com/ uoguelph-mlrg/ confidence_estimation
Revised: We borrowed the evaluation code from https://github.com/ uoguelph-mlrg/confidence_estimation

Original: (DeVries & Taylor, 2018).
Revised: (DeVries & Taylor, 2018).

Original: Metrics We reported the standard metrics for binary classiﬁcation: FPR at 95 % TPR, Detection error, AUROC (Area Under the Receiver Operating Characteristic curve) and AUPR(Area under the
Revised: Metrics We reported the standard metrics for binary classiﬁcation: FPR at 95 % TPR, Detection error, AUROC (Area Under the Receiver Operating Characteristic curve) and AUPR (Area under the

Original: Precision-Recall curve, -in or -out depending on which dataset is speciﬁed as positive).
Revised: Precision-Recall curve, -in or -out depending on which dataset is speciﬁed as positive).

Original: See (Liang et al., 2017) for deﬁnitions and interpretations of these metrics
Revised: See Liang et al. (2018) for deﬁnitions and interpretations of these metrics.

Original: E A DDITIONAL THEORETIC COMPONENTS E. B IAS VARIANCE COVARIANCE DECOMPOSITION
Revised: E A DDITIONAL T HEORETICAL E LEMENTS E. B IAS V ARIANCE C OVARIANCE D ECOMPOSITION

Original: The Bias-Variance-Covariance Decomposition (Ueda & Nakano, 1996) generalizes the BiasVariance Decomposition (Kohavi et al., 1996) by treating the ensemble of M members as a single learning unit.
Revised: The Bias-Variance-Covariance Decomposition (Ueda & Nakano, 1996) generalizes the BiasVariance Decomposition (Kohavi et al., 1996) by treating the ensemble of M members as a single learning unit.

Original: [Equation]
Revised: [Equation]

Original: with
Revised: with

Original: [Equation]
Revised: [Equation]

Original: The estimation improves when the covariance between members is zero, i.e when the  errors areunbiased and uncorrelated.
Revised: The estimation improves when the covariance between members is zero: the reduction factor of the variance component equals to M when errors are uncorrelated.

Original: Compared to the Bias-Variance Decomposition (Kohavi et al., 1996), it leads to a variance reduction of 1 M .
Revised: Compared to the Bias-Variance Decomposition (Kohavi et al., 1996), it leads to a variance reduction of 1 M .

Original: Brown et al.
Revised: Brown et al.

Original: (2005a;b) summarized it this way: “in addition to the bias and variance of the individual estimators, the generalisation error of an ensemble also depends on the covariance between the individuals.
Revised: (2005a;b) summarized it this way: “in addition to the bias and variance of the individual estimators, the generalisation error of an ensemble also depends on the covariance between the individuals.

Original: This raises the interesting issue of why we should ever train ensemble members separately; why shouldn’t we try to ﬁnd some way to capture the effect of the covariance in the error function?”.We could not agree more.
Revised: This raises the interesting issue of why we should ever train ensemble members separately; why shouldn’t we try to ﬁnd some way to capture the effect of the covariance in the error function?”.

Original: E. M UTUAL I NFORMATION
Revised: E. M UTUAL I NFORMATION

Original: Nobody knows what entropy really is.
Revised: Nobody knows what entropy really is.

Original: John Van Neumann to Claude Shannon
Revised: John Van Neumann to Claude Shannon

Original: At the cornerstone ofthe Shannon’s information theory in 1948 (Shannon, 1948), Mutual Information (MI) is the difference between the sum of individual entropies and the entropy of the variables considered jointly.
Revised: At the cornerstone of Shannon’s information theory in 1948 (Shannon, 1948), mutual information is the difference between the sum of individual entropies and the entropy of the variables considered jointly.

Original: Stated otherwise, it is the reduction in the uncertainty of one variable due to the knowledge of the other variable (Cover, 1999).
Revised: Stated otherwise, it is the reduction in the uncertainty of one variable due to the knowledge of the other variable (Cover, 1999).

Original: Entropy owed its name to the thermodynamic measure of uncertainty introduced by Rudolf Clausius and developed by Ludwig Boltzmann.
Revised: Entropy owed its name to the thermodynamic measure of uncertainty introduced by Rudolf Clausius and developed by Ludwig Boltzmann.

Original: [Equation]
Revised: [Equation]

Original: The conditional mutual information(CMI) generalizes MI when a third variable is given:
Revised: The conditional mutual information generalizes mutual information when a third variable is given:

Original: [Equation]
Revised: [Equation]

Original: E. KL BETWEEN G AUSSIANS
Revised: E. KL BETWEEN G AUSSIANS

Original: The Kullback-Leibler divergence (Kullback, 1959) between two gaussian distributions takes a particularly simple form:
Revised: The Kullback-Leibler divergence (Kullback, 1959) between two gaussian distributions takes a particularly simple form:

Original: [Equation]
Revised: [Equation]

Original: The variance component forces the predicted variance e σ ( x ) to be close to b σ ( y )
Revised: The variance component forces the predicted variance e σ ( x ) to be close to b σ ( y )

Original: = 1 .
Revised: = 1 .

Original: The mean component forces the class-embedding b µ ( y ) to converge to the average of the different elements in its class.
Revised: The mean component forces the class-embedding b µ ( y ) to converge to the average of the different elements in its class.

Original: These class-embeddings are similar to class-prototypes, highlighting a theoretical links between CEB (Fischer, 2020; Fischer & Alemi, 2020) and prototype based learning methods (Liu & Nakagawa, 2001).
Revised: These class-embeddings are similar to class-prototypes, highlighting a theoretical link between CEB (Fischer, 2020; Fischer & Alemi, 2020) and prototype based learning methods (Liu & Nakagawa, 2001).

Original: E. D IFFERENCE  VCEB AND VIB
Revised: E. D IFFERENCE BETWEEN VCEB AND VIB

Original: In Fischer (2020)
Revised: In Fischer (2020)

Original: [Equation]
Revised: [Equation]

Original: CEB is variationally upper bounded by VCEB.
Revised: CEB is variationally upper bounded by VCEB.

Original: We detail the computations:
Revised: We detail the computations:

Original: [Equation]
Revised: [Equation]

Original: (Reparameterization trick)

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: As a reminder, Alemi et al.
Revised: As a reminder, Alemi et al.

Original: (2017) upper bounded: IB β ib ( Z ) = 1 β ib I ( X ; Z ) − I
Revised: (2017) upper bounded: IB β ib ( Z ) = 1 β ib I ( X ; Z ) − I

Original: ( Y ; Z )
Revised: ( Y ; Z )

Original: by :
Revised: by :

Original: [Equation]
Revised: [Equation]

Original: In VIB, all features distribution e ( z | x ) are moved towards the same class-agnostic distribution b ( z ) ∼ N ( µ, σ ) , independently of y .
Revised: In VIB, all features distribution e ( z | x ) are moved towards the same class-agnostic distribution b ( z ) ∼ N ( µ, σ ) , independently of y .

Original: In VCEB,all e ( z | x ) are moved towards the class conditional marginal b µ ( y ) ∼ N ( b µ ( y ) , b σ ( y )) .
Revised: In VCEB, e ( z | x ) are moved towards the class conditional marginal b µ ( y ) ∼ N ( b µ ( y ) , b σ ( y )) .

Original: This is the unique difference between VIB and VCEB :
Revised: This is the unique difference between VIB and VCEB .

Original: VIB leads to a looser approximation with more bias than VCEB.
Revised: VIB leads to a looser approximation with more bias than VCEB.

Original: E. T RANSFORMING IBMI AND CEBMI INTO TRACTABLE LOSSES
Revised: E. T RANSFORMING IBR AND CEBR INTO T RACTABLE L OSSES

Original: In this section we derive the variational approximation of the IBMI criterion, deﬁned by:
Revised: In this section we derive the variational approximation of the IBR criterion, deﬁned by:

Original: [Equation]
Revised: [Equation]

Original: Mutual Information Estimation To estimate the MI component, we apply the same procedure than for CMI but without the categorical constraint, as in the seminal work of Belghazi et al. 
Revised: Redundancy Estimation To estimate the redundancy component, we apply the same procedure as for conditional redundancy but without the categorical constraint, as in the seminal work of Belghazi et al. (2018) for mutual information estimation.

Original: Let B j and B p be two random batches sampled respectively from the observed joint distribution p ( z 1 , z 2 )
Revised: Let B j and B p be two random batches sampled respectively from the observed joint distribution p ( z 1 , z 2 )

Original: = p ( e 1 ( z | x ) , e 2 ( z | x )) and the product distribution 
Revised: = p ( e 1 ( z | x ) , e 2 ( z | x )) and the product distribution p ( z 1 ) p ( z 2 ) =

Original: [Equation] where x, x (cid:48) are two images.
Revised: p ( e 1 ( z | x )) p ( e 2 ( z | x (cid:48) )) , where x, x (cid:48) are two inputs that may not belong to the same class.

Original: We similarly train a network w that tries to discriminate these two distributions.
Revised: We similarly train a network w that tries to discriminate these two distributions.

Original: With f = w 1 − w , the MI estimation is:
Revised: With f = w 1 − w , the redundancy estimation is:

Original: [Equation]
Revised: [Equation]

Original: and the ﬁnal loss:
Revised: and the ﬁnal loss:

Original: [Equation]
Revised: [Equation]

Original: IBMI Finally we train θ 1 = { e 1 , b 1 , c 1 } and θ 2 = { e 2 , b 2 , c 2 } jointly by minimizing:
Revised: IBR Finally we train θ 1 = { e 1 , b 1 , c 1 } and θ 2 = { e 2 , b 2 , c 2 } jointly by minimizing:

Original: [Equation]
Revised: [Equation]

Original: CEBMI For ablation study, we also consider a model that would beneﬁt from CEB tight approximation while having MI regularization:[Equation]  DICE FOR M MEMBERS
Revised: CEBR For ablation study, we also consider a criterion that would beneﬁt from CEB’s tight approximation but with non-conditional redundancy regularization:

Original: We trivially expand our framework for M > 2 using the same variational lower bounds and neural estimations.

Original: The training loss is:

Original: [Equation]
Revised: [Equation]

Original: E. C AN WE LEARN FEATURES INDEPENDENCE WITHOUT INFORMATION BOTTLENECK ?
Revised: E. C AN WE LEARN FEATURES INDEPENDENCE WITHOUT COMPRESSION ?

Original: In the Markov chain Z 1 ← X → Z 2 , the chain rules provides:
Revised: In the Markov chain Z 1 ← X → Z 2 , the chain rules provides:

Original: [Equation]
Revised: [Equation]

Original: therefore I ( Z 1 ; Z 2 | Y ) ≤
Revised: therefore I ( Z 1 ; Z 2 | Y ) ≤

Original: I ( X ; Z 2 | Y ) .
Revised: I ( X ; Z 2 | Y ) .

Original: By symmetry we have I ( Z 1 ; Z 2 | Y ) ≤
Revised: By symmetry we have I ( Z 1 ; Z 2 | Y ) ≤

Original: I ( X ; Z 1 | Y ) .
Revised: I ( X ; Z 1 | Y ) .

Original: In ﬁrst order, reducing I ( Z 1 , X | Y ) and I ( Z 2 , X | Y ) has an impact on I ( Z 1 , Z 2 | Y ) .
Revised: In ﬁrst order, reducing I ( Z 1 , X | Y ) and I ( Z 2 , X | Y ) has an impact on I ( Z 1 , Z 2 | Y ) .

Original: That’s the reason why CEB has higher diversity than a deterministic baseline.
Revised: That is the reason why CEB has higher diversity than a deterministic baseline.

Revised: Note that it also beneﬁts from additional randomness brought by the reparameterization trick.

Original: Our adversarial loss is a second order improvement, that cannot leverage a parameterized variational approximation.
Revised: Our redundancy loss is critical but more complex to optimize as it cannot leverage a parameterized variational approximation.

Original: Moreover,  without the IB component, there is no constraint on the predicted variance.
Revised: Moreover, compression losses learn distributions rather than deterministic points in the features space: without the compression component, there is no constraint on the predicted variance.

Original: We tried different approaches, with variance set to 1 , with and without the reparametrization  trick in classiﬁcation and in adversarial training: we experimentally verify that learning only the adversarial loss without bottleneck losses (VIB or VCEB) yields unstable results.
Revised: We tried different approaches; deterministic; with variance set to 1 ; with and without the reparametrization trick. They all underperformed. Notably, reducing conditional redundancy without reparametrization trick destabilizes overall adversarial training as the discriminator’s task becomes too easy.

Original: In conclusion,  DICE beneﬁts from both components. E. I S THE THEORETIC FORMULATION STILL VALID IN THE BRANCH - BASED SETTING ?
Revised: In conclusion, we have empirically veriﬁed that DICE beneﬁts from both components: learning only the redundancy loss without compression losses yielded unstable results.

Original: Yes!
Revised: And most importantly,

Original: It exactly remains the same ... except for the last assumption that ” Z 1 ← X → Z 2 is a Markov chain”. In this case, we just need to consider that X is the latest shared features among all members.
Revised: CEB comes at almost no extra cost.

Original: F S OCIOLOGICAL A NALOGY
Revised: F S OCIOLOGICAL A NALOGY

Original: We showed that increasing diversity in features while encouraging the different learners to agree improves performance: the optimal diversity-accuracy trade-off was obtained with a large diversity in neural networks.
Revised: We showed that increasing diversity in features while encouraging the different learners to agree improves performance for neural networks: the optimal diversity-accuracy trade-off was obtained with a large diversity.

Original: To ﬁnish, we make a short discussion about the impact of diversity in our society.
Revised: To ﬁnish, we make a short analogy with the importance of diversity in our society.

Original: Decision making in group is better than individual decision as long as the members do not belong to the same cluster.
Revised: Decision-making in group is better than individual decision as long as the members do not belong to the same cluster.

Original: Homogenization of the decision makers increases vulnerability to failures, whereas diversity among backgrounds sparks new discoveries (Muldoon, 2016): ideas should be shared and debated among members reﬂecting the diversity of the society’s various components.
Revised: Homogenization of the decision makers increases vulnerability to failures, whereas diversity of backgrounds sparks new discoveries (Muldoon, 2016): ideas should be shared and debated among members reﬂecting the diversity of the society’s various components.

Original: Academia especially needs this diversity to promotethe trust in research (Sierra-Mercado & L´azaroMu˜noz, 2018), to improvethe quality of the ﬁndings (Swartz et al., 2019), productivity of the teams (Vasilescu et al., 2015) and even schooling’s impact (Bowman, 2013).
Revised: Academia especially needs this diversity to promote trust in research (Sierra-Mercado & L ´ azaroMu ˜ noz, 2018), to improve quality of the ﬁndings (Swartz et al., 2019), productivity of the teams (Vasilescu et al., 2015) and even schooling’s impact (Bowman, 2013).