Original: FiLM-Ensemble: Probabilistic Deep Learning via Feature-wise Linear Modulation
Revised: FiLM-Ensemble: Probabilistic Deep Learning via Feature-wise Linear Modulation

Original: Abstract
Revised: Abstract

Original: The ability to estimate epistemic uncertainty is often crucial when deploying machine learning in the real world, but modern methods often produce overconfident, uncalibrated uncertainty predictions.
Revised: The ability to estimate epistemic uncertainty is often crucial when deploying machine learning in the real world, but modern methods often produce overconfident, uncalibrated uncertainty predictions.

Original: A common approach to quantify epistemic uncertainty, usable across a wide class of prediction models, is to train a model ensemble .
Revised: A common approach to quantify epistemic uncertainty, usable across a wide class of prediction models, is to train a model ensemble .

Original: In a naïve implementation, the ensemble approach has high computational cost and high memory demand.
Revised: In a naïve implementation, the ensemble approach has high computational cost and high memory demand.

Original: This challenges in particular modern deep learning, where even a single deep network is already demanding in terms of compute and memory, and has given rise to a number of attempts to emulate the model ensemble without actually instantiating separate ensemble members.
Revised: This challenges in particular modern deep learning, where even a single deep network is already demanding in terms of compute and memory, and has given rise to a number of attempts to emulate the model ensemble without actually instantiating separate ensemble members.

Original: We introduce FiLM-Ensemble, a deep, implicit ensemble method based on the concept of Feature-wise Linear Modulation (FiLM).
Revised: We introduce FiLM-Ensemble, a deep, implicit ensemble method based on the concept of Feature-wise Linear Modulation (FiLM).

Original: That technique was originally developed for multi-task learning, with the aim of decoupling different tasks.
Revised: That technique was originally developed for multi-task learning, with the aim of decoupling different tasks.

Original: We show that the idea can be extended to uncertainty quantification: by modulating the network activations of a single deep network with FiLM, one obtains a model ensemble with high diversity, and consequently well-calibrated estimates of epistemic uncertainty, with low computational overhead in comparison.
Revised: We show that the idea can be extended to uncertainty quantification: by modulating the network activations of a single deep network with FiLM, one obtains a model ensemble with high diversity, and consequently well-calibrated estimates of epistemic uncertainty, with low computational overhead in comparison.

Original: Empirically,
Revised: Empirically,

Original: FiLM-Ensemble outperforms other implicit ensemble methods, and it comes very close to the upper bound of an explicit ensemble of networks (sometimes even beating it), at a fraction of the memory cost.
Revised: FiLM-Ensemble outperforms other implicit ensemble methods, and it comes very close to the upper bound of an explicit ensemble of networks (sometimes even beating it), at a fraction of the memory cost.

Original:  Introduction
Revised:  Introduction

Original: A key component for reliable and trustworthy machine learning are algorithms that output not only accurate predictions of the target variables, but also well-calibrated estimates of their uncertainty Gal and Ghahramani [2016].
Revised: A key component for reliable and trustworthy machine learning are algorithms that output not only accurate predictions of the target variables, but also well-calibrated estimates of their uncertainty [Gal and Ghahramani, 2016].

Original: The overall uncertainty of a predictor is usually decomposed into two parts [Der Kiureghian and Ditlevsen, 2009].
Revised: The overall uncertainty of a predictor is usually decomposed into two parts [Der Kiureghian and Ditlevsen, 2009].

Original: Aleatoric uncertainty is inherent in the data, for instance due to class overlap or sensor noise.
Revised: Aleatoric uncertainty is inherent in the data, for instance due to class overlap or sensor noise.

Original: On the contrary, epistemic uncertainty characterises the uncertainty of the model weights, due to a lack of knowledge about parts of the input space that are insufficiently represented in the training set.
Revised: On the contrary, epistemic uncertainty characterises the uncertainty of the model weights, due to a lack of knowledge about parts of the input space that are insufficiently represented in the training set.

Original: Uncertainty caused by distributional shifts between the training and test data is sometimes conceived as a third source of uncertainty [Malinin and Gales, 2018], but in practice often modelled as part of epistemic uncertainty.
Revised: Uncertainty caused by distributional shifts between the training and test data is sometimes conceived as a third source of uncertainty [Malinin and Gales, 2018], but in practice often modelled as part of epistemic uncertainty.

Original: Measuring epistemic uncertainty for complex models such as deep neural networks is not trivial: by definition one cannot derive it from the training data, since it concerns the behavior of the model in regions of the input space that are not represented in the training set.
Revised: Measuring epistemic uncertainty for complex models such as deep neural networks is not trivial: by definition one cannot derive it from the training data, since it concerns the behavior of the model in regions of the input space that are not represented in the training set.

Original: Different methods have been explored [Welling and Teh, 2011, Graves, 2011, Blundell et al., 2015, Lakshminarayanan et al., 2017a, Huang and Belongie, 2017, Gal and Ghahramani, 2016], however the de facto standard remain deep ensemble models [Lakshminarayanan et al., 2017a].
Revised: Different methods have been explored [Welling and Teh, 2011, Graves, 2011, Blundell et al., 2015, Lakshminarayanan et al., 2017a, Huang and Belongie, 2017, Gal and Ghahramani, 2016], however the de facto standard remain deep ensemble models [Lakshminarayanan et al., 2017a].

Original: In its basic form, such a deep ensemble is simply a collection of independently trained networks that can be regarded as Monte Carlo samples from the model space.
Revised: In its basic form, such a deep ensemble is simply a collection of independently trained networks that can be regarded as Monte Carlo samples from the model space.

Original: To obtain ensemble members with reasonably low correlation, one can exploit the stochastic nature of the optimisation procedure, with different (random) weight initialisation and different (random) batches during stochastic gradient descent.
Revised: To obtain ensemble members with reasonably low correlation, one can exploit the stochastic nature of the optimisation procedure, with different (random) weight initialisation and different (random) batches during stochastic gradient descent.

Original: The expectation is that, once trained, the ensemble members will agree for inputs near the training samples, since the loss function favours similar outputs at those locations.
Revised: The expectation is that, once trained, the ensemble members will agree for inputs near the training samples, since the loss function favours similar outputs at those locations.

Original: Whereas they may disagree in unseen regions of the data space.
Revised: Whereas they may disagree in unseen regions of the data space.

Original: Thus, the spread of their predictions is a measure of epistemic uncertainty.
Revised: Thus, the spread of their predictions is a measure of epistemic uncertainty.

Original: The ensemble idea is conceptually very simple, but nevertheless yields uncertainties that are well calibrated, i.e., they are in line with the actual prediction errors (in expectation).
Revised: The ensemble idea is conceptually very simple, but nevertheless yields uncertainties that are well calibrated, i.e., they are in line with the actual prediction errors (in expectation).

Original: The drawback of deep ensembles is their large computational cost.
Revised: The drawback of deep ensembles is their large computational cost.

Original: Both computation and memory consumption grow directly proportionally with the number of ensemble members, during training as well as during inference.
Revised: Both computation and memory consumption grow directly proportionally with the number of ensemble members, during training as well as during inference.

Original: This makes them impractical in hardware-constrained settings, and leads to a widening gap as models continue to grow in size, while at the same time applications like mobile robotics, augmented reality and smart sensor networks increase the need for mobile and embedded computing.
Revised: This makes them impractical in hardware-constrained settings, and leads to a widening gap as models continue to grow in size, while at the same time applications like mobile robotics, augmented reality and smart sensor networks increase the need for mobile and embedded computing.

Original: To improve efficiency, several researchers have explored ways to mimic deep ensembles without explicitly duplicating the underlying network.
Revised: To improve efficiency, several researchers have explored ways to mimic deep ensembles without explicitly duplicating the underlying network.

Original: Possible strategies include the reuse and recombination of network modules [e.g., Wen et al., 2020, Havasi et al., 2021], injection of noise at inference time [e.g., Gal and Ghahramani, 2016], as well as hybrid variants [e.g., Durasov et al., 2021].
Revised: Possible strategies include the reuse and recombination of network modules [e.g., Wen et al., 2020, Havasi et al., 2021], injection of noise at inference time [e.g., Gal and Ghahramani, 2016], as well as hybrid variants [e.g., Durasov et al., 2021].

Original: Empirically, these models do speed up training and/or inference, but still exhibit a significant performance gap compared to the naïve, explicit ensemble, both w.r.t.
Revised: Empirically, these models do speed up training and/or inference, but still exhibit a significant performance gap compared to the naïve, explicit ensemble, both w.r.t.

Original: prediction quality and w.r.t.
Revised: prediction quality and w.r.t.

Original: the calibration of the predicted (epistemic) uncertainties.
Revised: the calibration of the predicted (epistemic) uncertainties.

Original: The lottery ticket hypothesis [Frankle and Carbin, 2018] and other network pruning studies, e.g., 
Revised: The lottery ticket hypothesis [Frankle and Carbin, 2018] and other network pruning studies, e.g., by

Original: Han et al.
Revised: Han et al.

Original: [2015], Lee et al.
Revised: [2015], Lee et al.

Original: [2019], Mallya and Lazebnik [2018], underline that neural networks are heavily over-parameterized.
Revised: [2019], Mallya and Lazebnik [2018], underline that neural networks are heavily over-parameterized.

Original: Their parameters are used inefficiently, and they can be pruned significantly without large performance drops.
Revised: Their parameters are used inefficiently, and they can be pruned significantly without large performance drops.

Original: In lifelong learning and multi-task learning it is essential to use the network efficiently, in order to limit computational overhead when introducing new tasks.
Revised: In lifelong learning and multi-task learning it is essential to use the network efficiently, in order to limit computational overhead when introducing new tasks.

Original: There are recent works that achieve good performance in these tasks by introducing modulation (respectively, adaptation) strategies.
Revised: There are recent works that achieve good performance in these tasks by introducing modulation (respectively, adaptation) strategies.

Original: In particular, Li et al.
Revised: In particular, Li et al.

Original: [2018], Takeda et al.
Revised: [2018], Takeda et al.

Original: [2021] propose an efficient lifelong learning / domain adaptation method for multi-task learning, utilizing single feature-wise linear modulation (FiLM).
Revised: [2021] propose an efficient lifelong learning / domain adaptation method for multi-task learning, utilizing single feature-wise linear modulation (FiLM).

Original: Inspired by that line of work, we propose a new, efficient ensemble method, FiLM-Ensemble.
Revised: Inspired by that line of work, we propose a new, efficient ensemble method, FiLM-Ensemble.

Original: Our method adapts feature-wise linear modulation as an alternative way to construct an ensemble for (epistemic) uncertainty estimation.
Revised: Our method adapts feature-wise linear modulation as an alternative way to construct an ensemble for (epistemic) uncertainty estimation.

Original: FiLM-Ensemble greatly reduces the computational overhead compared to the naïve ensemble approach, while performing almost on par with it, sometimes even better.
Revised: FiLM-Ensemble greatly reduces the computational overhead compared to the naïve ensemble approach, while performing almost on par with it, sometimes even better.

Original: In a nutshell, FiLM-Ensemble can be described as an implicit model ensemble in which each individual member is defined via its own set of linear modulation parameters for the activations, whereas all other network parameters are shared among ensemble members – and therefore only need to be stored and trained once.
Revised: In a nutshell, FiLM-Ensemble can be described as an implicit model ensemble in which each individual member is defined via its own set of linear modulation parameters for the activations, whereas all other network parameters are shared among ensemble members – and therefore only need to be stored and trained once.

Original: Thanks to this design, our method requires only a very small number of additional parameters on top of the base network, e.g., converting a single ResNet-18 model to an ensemble of 16 models increases the parameter count by 1.3%, compared to an increase by 1500% when setting up a naïve ensemble, see Table 1.
Revised: Thanks to this design, our method requires only a very small number of additional parameters on top of the base network, e.g., converting a single ResNet-18 model to an ensemble of 16 models increases the parameter count by 1.3%, compared to an increase by 1500% when setting up a naïve ensemble, see Table 1.

Original: We further show that FiLM-Ensemble results in more diverse ensemble members compared to other efficient ensemble methods.
Revised: We further show that FiLM-Ensemble results in more diverse ensemble members compared to other efficient ensemble methods.

Original: For instance, on the Cifar-10 benchmark it achieves diversity scores > 6 .
Revised: For instance, on the Cifar-10 benchmark it achieves diversity scores > 6 .

Original: 9% and up to 9 .
Revised: 9% and up to 9 .

Original: 2% (depending on ensemble size), against 6 . 8% for a naive ensemble, see Fig.
Revised: 2% (depending on ensemble size), against 6 . 8% for a naive ensemble, see Fig.

Original: Our contributions can be summarized as follows
Revised: Our contributions can be summarized as follows

Original: • We propose FiLM-Ensemble, a novel parameter- and time-efficient deep ensemble method.
Revised: • We propose FiLM-Ensemble, a novel parameter- and time-efficient deep ensemble method.

Original: • FiLM-Ensemble is designed in such a way that it can be readily combined with many popular deep learning models, by simply replacing batch normalisation (batchnorm) layers with conditional batchnorm layers.
Revised: • FiLM-Ensemble is designed in such a way that it can be readily combined with many popular deep learning models, by simply replacing batch normalisation (batchnorm) layers with conditional batchnorm layers.

Original: • We show that FiLM-Ensemble provides an excellent trade-off between accuracy and calibration performance, improving over existing ensemble methods.
Revised: • We show that FiLM-Ensemble provides an excellent trade-off between accuracy and calibration performance, improving over existing ensemble methods.

Original:  FiLM-Ensemble
Revised:  FiLM-Ensemble

Original: In [Perez et al., 2018] the linear feature modulationwas achieved by varying the affine parameters of the batch normalization layers γ and β .
Revised: Perez et al. [2018] achieved the linear feature modulation by varying the affine parameters of the batch normalization layers γ and β .

Original: For each batchnorm layer n in the network, the parameters are predicted according to some conditioning input z (for instance, different prediction tasks):
Revised: For each batchnorm layer n in the network, the parameters are predicted according to some conditioning input z (for instance, different prediction tasks):

Original: [Equation]
Revised: [Equation]

Original: The size of γ n and β n is R D n where D n is the feature dimension at layer n .
Revised: The size of γ n and β n is R D n where D n is the feature dimension at layer n .

Original: These parameters are then used to linearly modulate the activations at the n -th layer F n :
Revised: These parameters are then used to linearly modulate the activations at the n -th layer F n :

Original: [Equation]
Revised: [Equation]

Original: with ◦ being the Hadamard (element-wise) product taken w.r.t.
Revised: with ◦ being the Hadamard (element-wise) product taken w.r.t.

Original: the feature dimension.
Revised: the feature dimension.

Original: In our scenario we aim to use the affine parameters to instantiate different ensemble members, i.e., the conditioning variable z is simply an index m ∈ { 1 , ..., M } that identifies the ensemble member.
Revised: In our scenario we aim to use the affine parameters to instantiate different ensemble members, i.e., the conditioning variable z is simply an index m ∈ { 1 , ..., M } that identifies the ensemble member.

Original: The functions g n ( z ) and h n ( z ) degenerate to look-up tables, so we can dispose of them and simply learn M different sets of affine parameters for every batchnorm layer F n :
Revised: The functions g n ( z ) and h n ( z ) degenerate to look-up tables, so we can dispose of them and simply learn M different sets of affine parameters for every batchnorm layer F n :

Original: [Equation]
Revised: [Equation]

Original: To achieve ensemble members with diverse parameters, we resort to Xavier initialization, i.e., all γ m n and β mn are sampled from a uniform distribution bounded between:
Revised: To achieve ensemble members with diverse parameters, we resort to Xavier initialization, i.e., all γ mn and β mn are sampled from a uniform distribution bounded between:

Original: [Equation]
Revised: [Equation]

Original: where D n is the number of features (channels) in the n -th layer, and ρ is an initialization gain factor.
Revised: where D n is the number of features (channels) in the n -th layer, and ρ is an initialization gain factor.

Original: In our setting ρ is a tunable hyperparameter that allows one to control the trade-off between predictive accuracy and calibration of the model, see Section 3.5.
Revised: In our setting ρ is a tunable hyperparameter that allows one to control the trade-off between predictive accuracy and calibration of the model, see Section 3.5.

Original: In general, increasing ρ leads to more diverse ensemble members and thus favours calibration.
Revised: In general, increasing ρ leads to more diverse ensemble members and thus favours calibration.

Original: Note that as ρ → 0 , all ensemble members start from similar initial values for γ mn and β mn and the FiLM-Ensemble gradually collapses to a single model.
Revised: Note that as ρ → 0 , all ensemble members start from similar initial values for γ mn and β mn and the FiLM-Ensemble gradually collapses to a single model.

Original: During training, we feed each input sample x to every ensemble member and obtain predictions y m =
Revised: During training, we feed each input sample x to every ensemble member and obtain predictions y m =

Original: f θ , γ m , β m ( x ) , which depend both on the member-specific parameters ( β m , γ m ) and on the shared parameters θ .
Revised: f θ , γ m , β m ( x ) , which depend both on the member-specific parameters ( β m , γ m ) and on the shared parameters θ .

Original: All those parameters are optimized together to minimise the chosen loss function.
Revised: All those parameters are optimized together to minimise the chosen loss function.

Original: Here we focus on classification and use a standard cross-entropy (CE) loss.
Revised: Here we focus on classification and use a standard cross-entropy (CE) loss.

Original: At inference time the final prediction ˆ y is obtained by averaging the M predictions of the ensemble:
Revised: At inference time the final prediction ˆ y is obtained by averaging the M predictions of the ensemble:

Original: [Equation]
Revised: [Equation]

Original: 2.1 Implementation Details
Revised: 2.1 Implementation Details

Original: We implemented 1-dimensional and 2-dimensional FiLM-Ensemble layers, to be used in combination with 1-dimensional and 2-dimensional convolution operations, respectively.
Revised: We implemented 1-dimensional and 2-dimensional FiLM-Ensemble layers, to be used in combination with 1-dimensional and 2-dimensional convolution operations, respectively.

Original: The forward passes through different ensemble members can easily be parallelized by replicating both the input tensor and the FiLM parameters γ , β along the batch dimension and applying the affine transformation Eq.
Revised: The forward passes through different ensemble members can easily be parallelized by replicating both the input tensor and the FiLM parameters γ , β along the batch dimension and applying the affine transformation Eq.

Original: 3 (the same holds for the BatchEnsemble method).
Revised: 3 (the same holds for the BatchEnsemble method).

Original: In this way all ensemble members run simultaneously on a single device, thus optimally utilizing modern tensor computing hardware without having to load several instances of the classification network into memory.
Revised: In this way all ensemble members run simultaneously on a single device, thus optimally utilizing modern tensor computing hardware without having to load several instances of the classification network into memory.

Original: We have implemented FiLM-Ensemble in PyTorch and release the source code, including implementations of the baselines used in the experiments, together with trained model checkpoints.
Revised: We have implemented FiLM-Ensemble in PyTorch and release the source code.

Original: For all experiments, we optimize the model parameters with standard stochastic gradient descent, with momentum µ = 0 .
Revised: For all experiments, we optimize the model parameters with standard stochastic gradient descent, with momentum µ =0 .

Original: 9 and weight decay λ = 0 .
Revised: 9 and weight decay λ =0 .

Original: 0005 for regularization.
Revised: 0005 for regularization.

Original: We train forepochs with batch size 128.
Revised: We train for 200 epochs with batch size 128.

Original: The learning rate is initially set to 0 .
Revised: The learning rate is initially set to 0 .

Original: 1 and decays according to a cosine annealing schedule [Loshchilov and Hutter, 2017].
Revised: 1 and decays according to a cosine annealing schedule [Loshchilov and Hutter, 2017].

Original: The initialisation gain is set to ρ = 2 for all experiments, except for genome sequences (see Table 3), where ρ ∈ { 4 , 8 , 16 , 32 } .
Revised: The initialisation gain is set to ρ = 2 for all experiments, except for genome sequences (see Table 3), where ρ ∈ { 4 , 8 , 16 , 32 } .

Original: Please refer to the supplementary material for additional, dataset-specific details.
Revised: Please refer to the supplementary material for additional, dataset-specific details.

Original:  Experiments
Revised:  Experiments

Original: In the following, we first empirically examine the ability of FiLM-Ensemble to learn independent ensemble members, and compare it to deep ensembles.
Revised: In the following, we first empirically examine the ability of FiLM-Ensemble to learn independent ensemble members, and compare it to deep ensembles.

Original: Then we analyze the predictive accuracy and uncertainty calibration of our method, as well as its computational cost and memory consumption.
Revised: Then we analyze the predictive accuracy and uncertainty calibration of our method, as well as its computational cost and memory consumption.

Original: We compare its performance against several baseline methods and state-of-the-art alternatives on a variety of different datasets, and using different backbone architectures.
Revised: We compare its performance against several baseline methods and state-of-the-art alternatives on a variety of different datasets, and using different backbone architectures.

Original: All experimental results are averaged over three runs with different random seeds.
Revised: All experimental results are averaged over three runs with different random seeds.

Original: We evaluate our proposed method on a diverse set of classification tasks, including popular image classification benchmarks, image-based medical diagnosis, and genome sequence analysis.
Revised: We evaluate our proposed method on a diverse set of classification tasks, including popular image classification benchmarks, image-based medical diagnosis, and genome sequence analysis.

Original: CIFAR10 and CIFAR-100 [Krizhevsky, 2009] are widely used testbeds for image classification, and deep learning in general.
Revised: CIFAR10 and CIFAR-100 [Krizhevsky, 2009] are widely used testbeds for image classification, and deep learning in general.

Original: They consist of clean images of objects from 10, respectively 100, different semantic classes.
Revised: They consist of clean images of objects from 10, respectively 100, different semantic classes.

Original: Each of the two datasets contains 60,000 images, of which 10,000 are reserved for testing.
Revised: Each of the two datasets contains 60,000 images, of which 10,000 are reserved for testing.

Original: The semantic classes are uniformly distributed in the datasets and stratified across the train/test spit.
Revised: The semantic classes are uniformly distributed in the datasets and stratified across the train/test spit.

Original: Retina Glaucoma Detection [Diaz-Pinto et al., 2019] is a real-world clinical dataset that includes microscopic retina images from 956 patients with the neuropathic disease Glaucoma, and from 1401 subjects with normal (healthy) retinas.
Revised: Retina Glaucoma Detection [Diaz-Pinto et al., 2019] is a real-world clinical dataset that includes microscopic retina images from 956 patients with the neuropathic disease Glaucoma, and from 1401 subjects with normal (healthy) retinas.

Original: Each input sample is a single RGB image; we resize all images to 128 × 128 .
Revised: Each input sample is a single RGB image; we resize all images to 128 × 128 .

Original: The image augmentation applies by combination of crop, horizontal flip, and color jitter.
Revised: The image augmentation applies by combination of crop, horizontal flip, and color jitter.

Original: REFUGE 

Revised: REFUGE

Original: [Orlando et al., 2020] was a challenge at the MICCAI-2020 conference, aimed at retinal Glaucoma diagnosis.
Revised: [Orlando et al., 2020] was a challenge at the MICCAI-2020 conference, aimed at retinal Glaucoma diagnosis.

Original: The dataset consists of 800 microscopic retina images with size 1411 × 1411 pixels, collected from different clinics.
Revised: The dataset consists of 800 microscopic retina images with size 1411 × 1411 pixels, collected from different clinics.

Original: We use this dataset in conjunction with the models trained on the previous dataset [Diaz-Pinto et al., 2019], to evaluate the ability to detect out-of-distribution samples with the help of the predicted uncertainties (Table 4).
Revised: We use this dataset in conjunction with the models trained on the previous dataset [Diaz-Pinto et al., 2019], to evaluate the ability to detect out-of-distribution samples with the help of the predicted uncertainties (Table 4).

Original: 6mA Identification [Li et al., 2021], is a 1-dimensional sequential genome dataset.
Revised: 6mA Identification [Li et al., 2021], is a 1-dimensional sequential genome dataset.

Original: It consists of DNA sequences of rice plants, along with binary labels that indicate whether the sequence is a N6-methyladenine (6mA) site.
Revised: It consists of DNA sequences of rice plants, along with binary labels that indicate whether the sequence is a N6-methyladenine (6mA) site.

Original: 6mA is an important DNA modification associated with several biological processes, such as regulating gene transcription, DNA replication and DNA repair [Campbell and Kleckner, 1990,
Revised: 6mA is an important DNA modification associated with several biological processes, such as regulating gene transcription, DNA replication and DNA repair [Campbell and Kleckner, 1990,

Original: Cheng et al., 2016, Pukkila et al., 1983].
Revised: Cheng et al., 2016, Pukkila et al., 1983].

Original: Each sequence consists of 41 nucleotides.
Revised: Each sequence consists of 41 nucleotides.

Original: As there aredifferent types of nucleotides, each one-hot encoded sequence is represented by a 41 × 4 matrix.
Revised: As there aredifferent types of nucleotides, each one-hot encoded sequence is represented by a 41 × 4 matrix.

Original: In total there are 269,500 training samples and 38,500 test samples.
Revised: In total there are 269,500 training samples and 38,500 test samples.

Original: We compare FiLM-Ensemble against (i) a single model without any ensembling, as the elementary baseline, (ii) a naïve deep ensemble
Revised: We compare FiLM-Ensemble against (i) a single model without any ensembling, as the elementary baseline, (ii) a naïve deep ensemble

Original: [Lakshminarayanan et al., 2017b], and (iii) MC-Dropout [Gal and Ghahramani, 2016].
Revised: [Lakshminarayanan et al., 2017b], and (iii) MC-Dropout [Gal and Ghahramani, 2016].

Original: Furthermore, we compare with other state-of-the-art methods, including: (iv) Masksemble [Durasov et al., 2021] which can be seen as an extension of MC-Dropout, (v) MIMO ("multi-input multi-output")
Revised: Furthermore, we compare with other state-of-the-art methods, including: (iv) Masksemble [Durasov et al., 2021] which can be seen as an extension of MC-Dropout, (v) MIMO ("multi-input multi-output")

Original: [Havasi et al., 2021] which defines a multi-head architecture where each head acts as one ensemble member, and
Revised: [Havasi et al., 2021] which defines a multi-head architecture where each head acts as one ensemble member, and

Original: (vi) BatchEnsemble [Wen et al., 2020] which creates an efficient ensemble by expanding the layer weights using low rank matrices.
Revised: (vi) BatchEnsemble [Wen et al., 2020] which creates an efficient ensemble by expanding the layer weights using low rank matrices.

Original: Arguably, this layer-wise modification of an underlying, common representation is the approach that comes closest to our work.
Revised: Arguably, this layer-wise modification of an underlying, common representation is the approach that comes closest to our work.

Original: 3.1 Diversity Analysis
Revised: 3.1 Diversity Analysis

Original: Diversity is the key to constructing powerful ensembles: nothing is to be gained from highly correlated ensemble members that return similar outputs for (almost) any input [Zhang and Ma, 2012].
Revised: Diversity is the key to constructing powerful ensembles: nothing is to be gained from highly correlated ensemble members that return similar outputs for (almost) any input [Zhang and Ma, 2012].

Original: In order to analyze the diversity (respectively, the degree of independence) among members, we compute two distance metrics between the members’ predictive distributions.
Revised: In order to analyze the diversity (respectively, the degree of independence) among members, we compute two distance metrics between the members’ predictive distributions.

Original: Let f i and f j denote two different ensemble members for a classification task.
Revised: Let f i and f j denote two different ensemble members for a classification task.

Original: We first measure the disagreement score D , defined as the fraction of all s test samples for which the two members return different answers, averaged over all possible pairs of members:
Revised: We first measure the disagreement score D , defined as the fraction of all s test samples for which the two members return different answers, averaged over all possible pairs of members:

Original: [Equation]
Revised: [Equation]

Original: with [ · ] being the Iverson bracket.
Revised: with [ · ] being the Iverson bracket.

Original: Second, we also measure the Kullback–Leibler (KL) divergence between the two predictive distributions p ( f i ) and p ( f j ) , again averaged over all pairs:
Revised: Second, we also measure the Kullback–Leibler (KL) divergence between the two predictive distributions p ( f i ) and p ( f j ) , again averaged over all pairs:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: For both metrics
Revised: For both metrics

Original: [Table]
Revised: [Table]

Original: higher numbers correspond to higher diversity.
Revised: higher numbers correspond to higher diversity.

Original: See Fig.
Revised: See Fig.

Original: We observe that the average diversity increases with the number of ensemble members.
Revised: We observe that the average diversity increases with the number of ensemble members.

Original: In both metrics, FiLM-Ensemble achieves higher scores than the naïve, explicit deep ensemble.
Revised: In both metrics, FiLM-Ensemble achieves higher scores than the naïve, explicit deep ensemble.

Original: Meaning that the predictions of FiLMEnsemble are less correlated than those of an equivalent number of networks trained independently with different random seeds.
Revised: Meaning that the predictions of FiLMEnsemble are less correlated than those of an equivalent number of networks trained independently with different random seeds.

Original: Also, note that with the increasing number of members, improvements in diversity metrics for the naïve explicit deep ensemble are negligible
Revised: Also, note that with the increasing number of members, improvements in diversity metrics for the naïve explicit deep ensemble are negligible

Original: [Table]
Revised: [Table]

Original: diverse representations.
Revised: representations.

Original: See Section 3.1.
Revised: See Section 3.1.

Original: 3.2 Computational Cost
Revised: 3.2 Computational Cost

Original: We go on to measure the efficiency of our proposed method, both in terms of parameter count and in terms of resources needed for a forward pass.
Revised: We go on to measure the efficiency of our proposed method, both in terms of parameter count and in terms of resources needed for a forward pass.

Original: Table 1 (left section) reports the numbers of additional numbers of parameters compared to a single network (i.e., without uncertainty calibration).
Revised: Table 1 (left section) reports the numbers of additional numbers of parameters compared to a single network (i.e., without uncertainty calibration).

Original: Due to their design, MC-Dropout and Masksemble do not require any additional parameters.
Revised: Due to their design, MC-Dropout and Masksemble do not require any additional parameters.

Original: Among the remaining methods, FiLM-Ensemble has the lowest (VGG-11) or second-lowest (Resnet-18) overhead in terms of parameter count, while performing significantly better, as we will see below.
Revised: Among the remaining methods, FiLM-Ensemble has the lowest (VGG-11) or second-lowest (Resnet-18) overhead in terms of parameter count, while performing significantly better, as we will see below.

Original: In terms of inference complexity (Tab. 1, center and right), FiLM-Ensemble turns out to be very competitive.
Revised: In terms of inference complexity (Tab. 1, center and right), FiLM-Ensemble turns out to be very competitive.

Original: Only MIMO with a ResNet-18 backbone is significantly faster and lighter, however this comes at a considerable price in terms of accuracy and calibration, see below.
Revised: Only MIMO with a ResNet-18 backbone is significantly faster and lighter, however this comes at a considerable price in terms of accuracy and calibration, see below.

Original: 3.3 CIFAR-10 / CIFAR- 
Revised: 3.3 CIFAR-10 / CIFAR-

Original: We perform several experiments on widely used benchmarks in computer vision, CIFAR-10 and
Revised: We perform several experiments on widely used benchmarks in computer vision, CIFAR-10 and

Original: CIFAR-100.
Revised: CIFAR-100.

Original: As performance metrics, we plot the test set accuracy and the expected calibration error
Revised: As performance metrics, we plot the test set accuracy and the expected calibration error

Original: [Table]
Revised: [Table]

Original: (ECE) against the ensemble size M in Fig.
Revised: (ECE) against the ensemble size M in Fig.

Original: 2, for all compared methods.
Revised: 2, for all compared methods.

Original: In terms of accuracy (left subfigure)
Revised: In terms of accuracy (left subfigure)

Original: [Table]
Revised: [Table]

Original: FiLM-Ensemble is outperformed only by the explicit deep ensemble, across all tested values of M ∈ { 2 , 4 , 8 , 16 } .
Revised: FiLM-Ensemble is outperformed only by the explicit deep ensemble, across all tested values of M ∈ { 2 , 4 , 8 , 16 } .

Original: These two surpass all other methods by a clear margin.
Revised: These two surpass all other methods by a clear margin.

Original: Surprisingly, we observe that BatchEnsemble generally exhibits a negative correlation between ensemble size and test set accuracy, with M = 4 performing best among all settings.
Revised: Surprisingly, we observe that BatchEnsemble generally exhibits a negative correlation between ensemble size and test set accuracy, with M = 4 performing best among all settings.

Original: MIMO shows very poor performance in this experimental setting in terms of accuracy (and also in ECE).
Revised: MIMO shows very poor performance in this experimental setting in terms of accuracy (and also in ECE).

Original: We speculate that its shared backbone probably has a tendency to fragment into largely independent ensemble members of low channel depth, which lack the necessary capacity when using comparatively small networks like VGG-11 or Resnet-18.
Revised: We speculate that its shared backbone probably has a tendency to fragment into largely independent ensemble members of low channel depth, which lack the necessary capacity when using comparatively small networks like VGG-11 or Resnet-18.

Original: With regard to ECE (right subfigure), we see all methods improving with growing ensemble size M .
Revised: With regard to ECE (right subfigure), we see all methods improving with growing ensemble size M .

Original: FiLM-Ensemble achieves better calibration then the widely used deep ensemble and MC-dropout methods, with significant margins at large ensemble sizes.
Revised: FiLM-Ensemble achieves better calibration then the widely used deep ensemble and MC-dropout methods, with significant margins at large ensemble sizes.

Original: Masksemble and BatchEnsemble achieve very good calibration for sizes M ∈ [4 . . .
Revised: Masksemble and BatchEnsemble achieve very good calibration for sizes M ∈ [4 . . .

Original: 16] , but at the cost of significant drops in classification performance.
Revised: 16] , but at the cost of significant drops in classification performance.

Original: An attractive feature of FiLM-Ensemble is that it offers a simple mechanism for trading off accuracy against calibration, by tuning the gain factor ρ .
Revised: An attractive feature of FiLM-Ensemble is that it offers a simple mechanism for trading off accuracy against calibration, by tuning the gain factor ρ .

Original: See Section 3.5.
Revised: See Section 3.5.

Original: In Tab. 2 we quantitatively compare all tested methods on the CIFAR-100 dataset, both with a standard Resnet-18 backbone and with a larger Resnet-34.
Revised: In Tab. 2 we quantitatively compare all tested methods on the CIFAR-100 dataset, both with a standard Resnet-18 backbone and with a larger Resnet-34.

Original: We observe that with both architectures, and in terms of both predictive accuracy and calibration, FiLM-Ensemble always either ranks first, or it ranks second behind the inefficient, explicit deep ensemble.
Revised: We observe that with both architectures, and in terms of both predictive accuracy and calibration, FiLM-Ensemble always either ranks first, or it ranks second behind the inefficient, explicit deep ensemble.

Original: Also, note that when the capacity of the network is increased (Resnet-18 → Resnet-34),  some of theother implicit methods except MIMO see an improvement in calibration performance but at the cost of a reduction in accuracy.
Revised: Also, note that when the capacity of the network is increased (Resnet-18 → Resnet-34), calibration performance improves for some of the implicit methods, but at the cost of reduced accuracy.

Original: Overall, the experiments with both variants of CIFAR and all three model architectures confirm that our method presents an excellent trade-off between predictive accuracy, uncertainty calibration and computational efficiency.
Revised: Overall, the experiments with both variants of CIFAR and all three model architectures confirm that our method presents an excellent trade-off between predictive accuracy, uncertainty calibration and computational efficiency.

Original: [Table]
Revised: [Table]

Original: 3.4 Retinal Glaucoma Detection
Revised: 3.4 Retinal Glaucoma Detection

Original: Glaucoma is currently the leading reason of irreversible blindness in the world.
Revised: Glaucoma is currently the leading reason of irreversible blindness in the world.

Original: Detection of glaucomatous structural damages and changes is a challenging task in the field of ophthalmology.
Revised: Detection of glaucomatous structural damages and changes is a challenging task in the field of ophthalmology.

Original: We evaluate our proposed FiLM-Ensemble, as well as the baselines described above, on the ask of diagnosing Glaucoma and quantifying the uncertainty associated with the prediction, see Table 3.
Revised: We evaluate our proposed FiLM-Ensemble, as well as the baselines described above, on the ask of diagnosing Glaucoma and quantifying the uncertainty associated with the prediction, see Table 3.

Original: The proposed FiLM-Ensemble achieves the best classification result across all ensemble sizes, and also the best overall result, with M = 16 .
Revised: The proposed FiLM-Ensemble achieves the best classification result across all ensemble sizes, and also the best overall result, with M =16 .

Original: Whereas there is no clear trend with respect to uncertainty calibration.
Revised: Whereas there is no clear trend with respect to uncertainty calibration.

Original: 3.5 6mA Identification
Revised: 3.5 6mA Identification

Original: With the 6mA identification task we show that FiLM-Ensemble can also be readily combined with existing models for 1-dimensional sequential genome data.
Revised: With the 6mA identification task we show that FiLM-Ensemble can also be readily combined with existing models for 1-dimensional sequential genome data.

Original: We use the 6mA-rice-Lv dataset and a 1D-CNN architecture whose hyper-parameters have already been optimized for this dataset [Li et al., 2021].
Revised: We use the 6mA-rice-Lv dataset and a 1D-CNN architecture whose hyper-parameters have already been optimized for this dataset [Li et al., 2021].

Original: FiLM-Ensemble improves the accuracy and the calibration of that model, see Fig.3.
Revised: FiLM-Ensemble improves the accuracy and the calibration of that model, see Fig.3.

Original: Our method performs on par with the explicit deep ensemble and better than a single instance of the model tuned for the specific task.
Revised: Our method performs on par with the explicit deep ensemble and better than a single instance of the model tuned for the specific task.

Original: More importantly, one can reach a significantly better calibration (lower ECE) by increasing the gain ρ , with only a minimal accuracy drop by < 0 .
Revised: More importantly, one can reach a significantly better calibration (lower ECE) by increasing the gain ρ , with only a minimal accuracy drop by < 0 .

Original: 25 percent points.
Revised: 25 percent points.

Original: Please refer to the Section B.2 for more results.
Revised: Please refer to the Section B.2 for more results.

Original: 3.6 Out-of-Distribution Detection
Revised: 3.6 Out-of-Distribution Detection

Original: Domain shift often occurs in medical datasets; thus, detecting out-of-distribution (OOD) samples is an important task in clinical diagnosis.
Revised: Domain shift often occurs in medical datasets; thus, detecting out-of-distribution (OOD) samples is an important task in clinical diagnosis.

Original: Model uncertainty can be used for OOD detection.
Revised: Model uncertainty can be used for OOD detection.

Original: For our experiment, we regard the retinal Glaucoma dataset [Diaz-Pinto et al., 2019] as the withindistribution samples and perform OOD detection with the REFUGE dataset [Orlando et al., 2020].
Revised: For our experiment, we regard the retinal Glaucoma dataset [Diaz-Pinto et al., 2019] as the withindistribution samples and perform OOD detection with the REFUGE dataset [Orlando et al., 2020].

Original: As performance metric, we report AUROC (Area Under the Receiver Operating Characteristic) scores,
Revised: As performance metric, we report AUROC (Area Under the Receiver Operating Characteristic) scores,

Original: [Table]
Revised: [Table]

Original: efficient ensemble methods, standard state-of-the-art OOD detection methods
Revised: efficient ensemble methods, standard state-of-the-art OOD detection methods

Original: [Table]
Revised: [Table]

Original: and even the explicit deep ensemble.
Revised: and even the explicit deep ensemble.

Original: This suggests that for challenging test samples, which are not adequately represented in the training data, the uncertainties estimated with FiLM-Ensemble are better calibrated
Revised: This suggests that for challenging test samples, which are not adequately represented in the training data, the uncertainties estimated with FiLM-Ensemble are better calibrated

Original: [Table]
Revised: [Table]

Original:  Related Work 4.1 Epistemic Uncertainty Quantification
Revised:  Related Work 4.1 Epistemic Uncertainty Quantification

Original: A large corpus of related work addresses the estimation of epistemic (model) uncertainty in neural networks.
Revised: A large corpus of related work addresses the estimation of epistemic (model) uncertainty in neural networks.

Original: At the heart of such modeling is often the concept of marginalization instead of optimization , i.e., integrating out a (possibly uncountably infinite) set of models weighted by their posterior probability, instead of committing to a point estimate of that distribution.
Revised: At the heart of such modeling is often the concept of marginalization instead of optimization , i.e., integrating out a (possibly uncountably infinite) set of models weighted by their posterior probability, instead of committing to a point estimate of that distribution.

Original: A multitude of methods have been proposed to implement approximate Bayesian inference w.r.t.
Revised: A multitude of methods have been proposed to implement approximate Bayesian inference w.r.t.

Original: the model weights, given the training data and an appropriate prior, a process that is not analytically tractable in general [Kendall and Gal, 2017].
Revised: the model weights, given the training data and an appropriate prior, a process that is not analytically tractable in general [Kendall and Gal, 2017].

Original: For instance, methods based on variational inference [Graves, 2011, Ranganath et al., 2014, Blundell et al., 2015] seek to learn an approximate posterior distribution which is a member of a simpler family of variational distributions.
Revised: For instance, methods based on variational inference [Graves, 2011, Ranganath et al., 2014, Blundell et al., 2015] seek to learn an approximate posterior distribution which is a member of a simpler family of variational distributions.

Original: This variational distribution can often be learned using backpropagation [Blundell et al., 2015] and can then be sampled from or sometimes even used for exact inference.
Revised: This variational distribution can often be learned using backpropagation [Blundell et al., 2015] and can then be sampled from, or sometimes even used for exact inference.

Original: Markov chain Monte Carlo sampling (MCMC) approaches [Neal, 1996, Welling and Teh, 2011, Chen et al., 2014] construct a Markov chain which has the exact posterior distribution as its stationary distribution, that can then be employed for sampling.
Revised: Markov chain Monte Carlo sampling (MCMC) approaches [Neal, 1996, Welling and Teh, 2011, Chen et al., 2014] construct a Markov chain which has the exact posterior distribution as its stationary distribution, that can then be employed for sampling.

Original: However, in practise, these approaches often fail to sufficiently explore high-dimensional, multi-modal loss landscapes as they are common in deep learning [Gustafsson et al., 2020].
Revised: However, in practise, these approaches often fail to sufficiently explore high-dimensional, multi-modal loss landscapes as they are common in deep learning [Gustafsson et al., 2020].

Original: 4.2 Ensembles and Sub-networks
Revised: 4.2 Ensembles and Sub-networks

Original: In a method referred to as deep ensembles [Lakshminarayanan et al., 2017a], a set of M neural network models are randomly and independently initialized, and are subjected to stochastic mini-batch sampling during SGD training.
Revised: In a method referred to as deep ensembles [Lakshminarayanan et al., 2017a], a set of M neural network models are randomly and independently initialized, and are subjected to stochastic mini-batch sampling during SGD training.

Original: The models generally converge to different minima in the parameter space, and can be considered samples from an approximate posterior [Wilson and Izmailov, 2020,
Revised: The models generally converge to different minima in the parameter space, and can be considered samples from an approximate posterior [Wilson and Izmailov, 2020,

Original: Gustafsson et al., 2020, Izmailov et al., 2021].
Revised: Gustafsson et al., 2020, Izmailov et al., 2021].

Original: Deep ensembles often achieve the best calibration and predictive accuracy [Ovadia et al., 2019, Gustafsson et al., 2020, Ashukha et al., 2020], but suffer from high computational complexity as it requires training, storing, and running inference on several network models.
Revised: Deep ensembles often achieve the best calibration and predictive accuracy [Ovadia et al., 2019, Gustafsson et al., 2020, Ashukha et al., 2020], but suffer from high computational complexity as they require training, storing, and running inference on several full instances of the network.

Original: Many alternative methods have recently been proposed in an attempt of reducing either the computational cost or the storage cost of deep ensembles.
Revised: Many alternative methods have recently been proposed in an attempt to reduce either the computational cost or the storage cost of deep ensembles.

Original: Monte Carlo (MC) Dropout
Revised: Monte Carlo (MC) Dropout

Original: [Gal and Ghahramani, 2016] runs multiple forward passes on the dropout layers in order to obtain multiple predictions and obtain an uncertainty estimate.
Revised: [Gal and Ghahramani, 2016] runs multiple forward passes on the dropout layers in order to obtain multiple predictions and obtain an uncertainty estimate.

Original: Although, the method requires less computations compared to deep ensembles, it also leads to less accurate uncertainty estimates.
Revised: Although the method requires fewer computations compared to deep ensembles, it also leads to less accurate uncertainty estimates.

Original: Snapshot ensembles [Huang et al., 2017] uses a cyclic learning rate scheduling in order to find multiple local minima, they then store multiple copies of the network to create the ensemble.
Revised: Snapshot ensembles [Huang et al., 2017] use a cyclic learning rate schedule in order to find multiple local minima, they then store multiple copies of the network to create the ensemble.

Original: While this approach reduces the computation cost during training it does not alleviate the storage cost.
Revised: While this approach reduces the computation cost during training it does not alleviate the storage cost.

Original: BatchEnsemble [Wen et al., 2020] propose to use multiple low rank matrices, which can be stored efficiently, in order to modulate the parameters of a neural network and thus mimic an ensemble of network models.
Revised: BatchEnsembles [Wen et al., 2020] employ multiple low rank matrices, which can be stored efficiently, in order to modulate the parameters of a neural network and thus mimic an ensemble of network models.

Original: Masksembles [Durasov et al., 2021] aims to improve the performance of MC Dropout by carefully selecting which masks, used to drop certain features, leading toa better uncertainty quantification.
Revised: Masksembles [Durasov et al., 2021] aim to improve the performance of MC Dropout by carefully selecting the dropout masks, used to drop certain features, such that they lead to better uncertainty quantification.

Original: Another approach which uses multiple sub-networks is proposed in Havasi et al.
Revised: Another approach that uses multiple sub-networks is proposed in Havasi et al.

Original: In this case additional, independent, layers are added at the beginning and at the end of the network in order to obtain multiple prediction with an individual network model.
Revised: In this case additional, independent layers are added at the beginning and at the end of the network, in order to obtain multiple prediction with an single backbone model.

Original: Although such method seems to reduce the computational resource required at training and inference time, the gain is limited to larger backbones, such asuse Wide-ResNet (e.g., ResNet28-10), whereas forstandard widely used  architectures like VGG (e.g., -16), or ResNet (e.g., -34), the approach is not very effective as shown in our experiments.
Revised: Although such method seems to reduce the computational resources required at training and inference time, the gain is limited to larger backbones, such as Wide-ResNet (e.g., ResNet28-10), whereas for widely used standard architectures like VGG (e.g., -16), or ResNet (e.g., -34), the approach is not very effective, as shown in our experiments.

Original: Although many methods  that aim at reducing the computational cost of deep ensembles have been proposed, none of the methods appear to unquestionably outperform the others.
Revised: Although many methods have been proposed that aim at reducing the computational cost of deep ensembles, none of them appears to clearly outperform most others.

Original: As a result the questionof how to effectively model uncertaintyis still  open.
Revised: In summary, the question how to effectively model uncertainty still remains open.

Original: 4.3 Feature-wise Linear Modulation
Revised: 4.3 Feature-wise Linear Modulation

Original: The idea of controlling the batch normalization parameters to modulate a network’s function has been explored by many different authors to accomplish different tasks.
Revised: The idea of controlling the batch normalization parameters to modulate a network’s function has been explored by many different authors to accomplish different tasks.

Original: Conditional batch normalization (CBN) was proposed by de Vries et al. , who achieved good performance in VQA experiments by using an MLP to estimate residuals for normalization parameters used in a pre-trained ResNet
Revised: Conditional batch normalization (CBN) was proposed by de Vries et al. , who achieved good performance in VQA experiments by using an MLP to estimate residuals for normalization parameters used in a pre-trained ResNet

Original: [De
Revised: [De

Original: Vries et al., 2017b].
Revised: Vries et al., 2017b].

Original: Perez et al.
Revised: Perez et al.

Original: [2018] proposed FiLM also for solving VQA tasks.
Revised: [2018] proposed FiLM also for solving VQA tasks.

Original: Strub et al.
Revised: Strub et al.

Original: [2018] modify FiLM to produce normalization parameters in several stages instead of all at once.
Revised: [2018] modify FiLM to produce normalization parameters in several stages instead of all at once.

Original: This formulation is better able to handle longer conditioning information, such as dialogues instead of questions, and achieved excellent results on the GuessWhat?!
Revised: This formulation is better able to handle longer conditioning information, such as dialogues instead of questions, and achieved excellent results on the GuessWhat?!

Original: visual dialogue task [De Vries et al., 2017a].
Revised: visual dialogue task [De Vries et al., 2017a].

Original: Such modulation operations have also been used for image style transfer.
Revised: Such modulation operations have also been used for image style transfer.

Original: It has been observed that the statistics of a feature map, which are associated with and can be controlled by the parameters produced by FiLM, are directly related to the style of an image [Huang and Belongie, 2017].
Revised: It has been observed that the statistics of a feature map, which are associated with and can be controlled by the parameters produced by FiLM, are directly related to the style of an image [Huang and Belongie, 2017].

Original: Dumoulin et al.
Revised: Dumoulin et al.

Original: [2017] succeeded in capturing the styles of artistic paintings, as well as combining extracted styles to create new ones, by using conditional instance normalization, which can be seen as a variation of
Revised: [2017] succeeded in capturing the styles of artistic paintings, as well as combining extracted styles to create new ones, by using conditional instance normalization, which can be seen as a variation of

Original: FiLM.
Revised: FiLM.

Original: Ghiasi et al.
Revised: Ghiasi et al.

Original: [2017] expand on this work by jointly training a style prediction network and a style transfer network, which also operate based on conditional instance normalization.
Revised: [2017] expand on this work by jointly training a style prediction network and a style transfer network, which also operate based on conditional instance normalization.

Original: Finally,
Revised: Finally,

Original: Brock et al.
Revised: Brock et al.

Original: [2018] used FiLM for natural image synthesis using generative adversarial networks (GANs).
Revised: [2018] used FiLM for natural image synthesis using generative adversarial networks (GANs).

Original: They report that this allowed for a reduction in computation and memory costs, as well as a 37% increase in training speed.
Revised: They report that this allowed for a reduction in computation and memory costs, as well as a 37% increase in training speed.

Original: Yang et al.
Revised: Yang et al.

Original: [2018] have used FiLM to modulate the layers of a segmentation network to perform video object segmentation.
Revised: [2018] have used FiLM to modulate the layers of a segmentation network to perform video object segmentation.

Original: This made it possible to avoid the fine-tuning process that was used by competing methods, which resulted in aspeed up of 70 times while achieving similar accuracy.
Revised: This made it possible to avoid the fine-tuning process that was used by competing methods, which resulted in a 70 × speed-up while achieving similar accuracy.

Original: Feature modulation has also been applied for various other tasks.
Revised: Feature modulation has also been applied for various other tasks.

Original: Oreshkin et al.
Revised: Oreshkin et al.

Original: [2018] used FiLM for task-dependent metric scaling, which allowed them to achieve excellent results in few-shot classification.
Revised: [2018] used FiLM for taskdependent metric scaling, which allowed them to achieve excellent results in few-shot classification.

Original: Vuorio et al.
Revised: Vuorio et al.

Original: [2019] use FiLM for meta-learning via task-aware modulation.
Revised: [2019] use FiLM for meta-learning via task-aware modulation.

Original: The authors note that FiLM outperformsand is more stable than attention-based modulation in this context.
Revised: The authors note that FiLM outperforms attention-based modulation in this context, and is more stable.

Original: Finally, Vinyals et al.
Revised: Finally, Vinyals et al.

Original: [2019] used FiLM in their AlphaStar neural architecture for multiagent reinforcement learning.
Revised: [2019] used FiLM in their AlphaStar neural architecture for multi-agent reinforcement learning.

Original: To our knowledge, FiLM has so far not been used for (implicit) model ensembling or uncertainty quantification.
Revised: To our knowledge, FiLM has so far not been used for (implicit) model ensembling or uncertainty quantification.

Original:  Limitations & Future Work
Revised:  Limitations & Future Work

Original: The work presented in this paper gives rise to several questions that can be explored in furure work.
Revised: The work presented in this paper gives rise to several questions that can be explored in future work.

Original: Using pre-trained models is standard procedure in deep learning applications.
Revised: Using pre-trained models is standard procedure in deep learning applications.

Original: It would be useful to explore how FiLM-Ensemble performs when used in conjunction with pre-trained models.
Revised: It would be useful to explore how FiLM-Ensemble performs when used in conjunction with pre-trained models.

Original: This may not be straightforward since FiLM-Ensemble (also other implicit ensemble methods e.g., BatchEnsemble, MIMO) is based on variations between ensemble members, which may be reduced if all members are similarly initialized.
Revised: This may not be straightforward since FiLM-Ensemble (and also other implicit ensemble methods, e.g., BatchEnsemble, MIMO) is based on variations between ensemble members, which may be reduced if all members are similarly initialized.

Original: Furthermore, self-attention-based models e.g.,Transformers Vaswani et al.
Revised: Furthermore, self-attention-based models, e.g.,

Original: [2017] and Vision Transformers (ViTs) Dosovitskiy et al. [2021] have recently become very popular; therefore it is natural to adapt FilM-Ensemble to work with such models.
Revised: Transformers [Vaswani et al., 2017] and Vision Transformers [ViTs, Dosovitskiy et al., 2021] have recently become very popular; therefore it is natural to adapt FilM-Ensemble to work with such models.

Original: Note that layer normalization is standard in Transformers instead of batch normalization, which prevents  straightforward application of the presented method in this case.
Revised: Note that layer normalization is standard in Transformers instead of batch normalization, which prevents the straightforward application of the presented method in this case.

Original: Also, a number of measures designed to enhance implicit ensembles are orthogonal to our approach and could potentially be combined with FiLM-Ensemble to further improve its performance and uncertainty calibration, while minimally increasing the computational costs.
Revised: Also, a number of measures designed to enhance implicit ensembles are orthogonal to our approach and could potentially be combined with FiLM-Ensemble to further improve its performance and uncertainty calibration, while minimally increasing the computational costs.

Original: It appears straight-forward to add the selection of independent examples for each member during training [as in MIMO, Havasi et al., 2021], temperature scaling [Guo et al., 2017] or data augmentation strategies [such as, e.g., Ramé et al., 2021].
Revised: It appears straight-forward to add the selection of independent examples for each member during training [as in MIMO, Havasi et al., 2021], temperature scaling [Guo et al., 2017] or data augmentation strategies [such as, e.g., Ramé et al., 2021].

Original:  Conclusion
Revised:  Conclusion

Original: In this paper we present FiLM-Ensemble, a novel implicit deep ensembling method.
Revised: In this paper we present FiLM-Ensemble, a novel implicit deep ensembling method.

Original: We achieve high efficiency using a simple yet effective idea – feature-wise linear modulation – which has been shown to be effective in different domains, such as image style transfer, model-agnostic meta-learning, or multi-task learning.
Revised: We achieve high efficiency using a simple yet effective idea – feature-wise linear modulation – which has been shown to be effective in different domains, such as image style transfer, model-agnostic meta-learning, or multi-task learning.

Original: Our extensive evaluation shows that FiLM-Ensemble outperforms, or is on par with, state-of-the-art ensemble methods in many different experimental settings.
Revised: Our extensive evaluation shows that FiLM-Ensemble outperforms, or is on par with, state-of-the-art ensemble methods in many different experimental settings.

Original: Broader Impact
Revised: Broader Impact

Original: Machine learning has recently witnessed a steep increase in model sizes and associated computational costs, and as a consequence a rapid growth in energy consumption.
Revised: Machine learning has recently witnessed a steep increase in model sizes and associated computational costs, and as a consequence a rapid growth in energy consumption.

Original: For instance, training state-of-theart language models like GPT-3 would amount to at least 1400 MWh or 4.6 million $.
Revised: For instance, training state-of-theart language models like GPT-3 would amount to at least 1400 MWh, or 4.6 million $.

Original: Therefore, recently the term Green AI has been introduced, referring to AI research that yields novel results while taking into account the computational cost, encouraging a reduction in resources spent.
Revised: Therefore, recently the term Green AI has been introduced, referring to AI research that yields novel results while taking into account the computational cost, encouraging a reduction in resources spent.

Original: To this extent, we believe that the presented method can foster more efficient ensemble methods and be helpful towards a greener AI.
Revised: To this extent, we believe that the presented method can foster more efficient ensemble methods and be helpful towards a greener AI.

Original: Acknowledgements We would like to thank Liyuan Zhu for his invaluable contributions to experiments.
Revised: Acknowledgements We would like to thank Liyuan Zhu for his invaluable contributions to experiments.

Original: H.A.G. received funding from German Federal Ministry of Education and Research (BMBF,
Revised: H.A.G. received funding from German Federal Ministry of Education and Research (BMBF,

Original: Grant “GenomeNet” 031L0199B).
Revised: Grant “GenomeNet” 031L0199B).

Original: B. B. and M. R. were supported by the Bavarian Ministry of
Revised: B. B. and M. R. were supported by the Bavarian Ministry of

Original: Economic Affairs, Regional Development and Energy through the Center for Analytics - DataApplications (ADACenter) within the framework of BAYERN DIGITAL II (20-3410-2-9-8) and the
Revised: Economic Affairs, Regional Development and Energy through the Center for Analytics - DataApplications (ADACenter) within the framework of BAYERN DIGITAL II (20-3410-2-9-8) and the

Original: German Federal Ministry of Education and Research and the Bavarian State Ministry for Science and the Arts.
Revised: German Federal Ministry of Education and Research and the Bavarian State Ministry for Science and the Arts.

Original: A Training Details
Revised: A Training Details

Original: A.1 CIFAR-10 /
Revised: A.1 CIFAR-10 /

Original: CIFARPlease refer to Section 2.1.
Revised: CIFARPlease refer to Section 2.1.

Original: A.2 Retinal Glaucoma Detection
Revised: A.2 Retinal Glaucoma Detection

Original: We followed the same training procedure as for CIFAR-10/100, please refer to Section 2.1.
Revised: We followed the same training procedure as for CIFAR-10/100, please refer to Section 2.1.

Original: Resnetis used as a backbone.
Revised: Resnetis used as a backbone.

Original: Models are trained with 70% of the dataset for 150 epochs and tested on the test set (30% of the dataset).
Revised: Models are trained with 70% of the dataset for 150 epochs and tested on the test set (30% of the dataset).

Original: A.3 6mA Identification
Revised: A.3 6mA Identification

Original: A 1-dimensional CNN architcture is used, whose hyperparameters such as kernel size and the number of layers are optimized by Li et al.
Revised: A 1-dimensional CNN architcture is used, whose hyperparameters such as kernel size and the number of layers are optimized by Li et al.

Original: The CNN consists of 5 convolutional blocks, where each block contains a 1-dimensional convolution, ReLU activation, batch normalization, and dropout with a rate of 0.5.
Revised: The CNN consists of 5 convolutional blocks, where each block contains a 1-dimensional convolution, ReLU activation, batch normalization, and dropout with a rate of 0.5.

Original: The convolutional layers have a filter size of 256, kernel size of 10, a stride of 1, and the first convolutional layer have a padding of 5.
Revised: The convolutional layers have a filter size of 256, kernel size of 10, a stride of 1, and the first convolutional layer have a padding of 5.

Original: On top of the last convolutional block, there is a linear layer for predicting the binary labels.
Revised: On top of the last convolutional block, there is a linear layer for predicting the binary labels.

Original: Binary cross-entropy is used as a loss.
Revised: Binary cross-entropy is used as a loss.

Original: All models are trained for 20 epochs.
Revised: All models are trained for 20 epochs.

Original: The initial learning rate of 0.01 is used, as in Li et al.
Revised: The initial learning rate of 0.01 is used, as in Li et al.

Original: Cosine-annealing is used as a learning rate scheduler.
Revised: Cosine-annealing is used as a learning rate scheduler.

Original: B Additional Results B.1 EfficientNet as Backbone & More Calibration Metrics
Revised: B Additional Results B.1 EfficientNet as Backbone & More Calibration Metrics

Original: We run more experiments using more modern architecture:
Revised: We run more experiments using more modern architecture:

Original: EfficientNet whose proportion of the number of channels vs. the number of layers can vary drastically, compared to Resnets.
Revised: EfficientNet whose proportion of the number of channels vs. the number of layers can vary drastically, compared to Resnets.

Original: In this experiment we use two extra calibration metrics: (i) the Brier score Brier et al.
Revised: In this experiment we use two extra calibration metrics: (i) the Brier score Brier et al.

Original: [1950] and (ii) the Static Calibration Error (SCE) Nixon et al.
Revised: [1950] and (ii) the Static Calibration Error (SCE) Nixon et al.

Original: SCE can be considered an extension of
Revised: SCE can be considered an extension of

Original: ECE but more accurately account for calibration by considering all classes, instead of just the one with the highest confidence.
Revised: ECE but more accurately account for calibration by considering all classes, instead of just the one with the highest confidence.

Original: Table 5 shows that FiLM-Ensemble can also be effectively used in conjunction with EfficientNet architecture.
Revised: Table 5 shows that FiLM-Ensemble can also be effectively used in conjunction with EfficientNet architecture.

Original: In addition, other calibration metrics are also in favor of
Revised: In addition, other calibration metrics are also in favor of

Original: FilM-Ensemble.
Revised: FilM-Ensemble.

Original: [Table]
Revised: [Table]

Original: B.2 Calibration-Accuracy Trade-off
Revised: B.2 Calibration-Accuracy Trade-off

Original: As in Section 3.
Revised: As in Section 3.

Original: [Table]
Revised: [Table]

Original: we show that one can reach a significantly better calibration (lower ECE) by increasing the gain ρ
Revised: we show that one can reach a significantly better calibration (lower ECE) by increasing the gain ρ

Original: [Table]
Revised: [Table]

Original: with only a minimal accuracy drop.
Revised: with only a minimal accuracy drop.

Original: In this case
Revised: In this case

Original: [Table]
Revised: [Table]

Original: we use Resnet-34 withensemble members on Cifar-100 dataset.
Revised: we use Resnet-34 withensemble members on Cifar-100 dataset.

Original: See Fig.
Revised: See Fig.

Original: Also note Fig.
Revised: Also note Fig.

Original: 5 is an extension of Fig.
Revised: 5 is an extension of Fig.

Original: 3 with various number of ensemble members.
Revised: 3 with various number of ensemble members.