Original: FiLM-Ensemble: Probabilistic Deep Learning via Feature-wise Linear Modulation
Revised: FiLM-Ensemble: Probabilistic Deep Learning via Feature-wise Linear Modulation

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: The ability to estimate epistemic uncertainty is often crucial when deploying machine learning in the real world, but modern methods often produce overconfident,uncalibrated uncertainty predictions.
Revised: The ability to estimate epistemic uncertainty is often crucial when deploying machine learning in the real world, but modern methods often produce overconfident, uncalibrated uncertainty predictions.

Original: A common approach to quantify epistemicuncertainty, usable across a wide class of prediction models, is to train a modelensemble .
Revised: A common approach to quantify epistemic uncertainty, usable across a wide class of prediction models, is to train a model ensemble .

Original: In a naïve implementation, the ensemble approach has high computational cost and high memory demand.
Revised: In a naïve implementation, the ensemble approach has high computational cost and high memory demand.

Original: This challenges in particular moderndeep learning, where even a single deep network is already demanding in termsof compute and memory, and has given rise to a number of attempts to emulatethe model ensemble without actually instantiating separate ensemble members.
Revised: This challenges in particular modern deep learning, where even a single deep network is already demanding in terms of compute and memory, and has given rise to a number of attempts to emulate the model ensemble without actually instantiating separate ensemble members.

Original: We introduce FiLM-Ensemble, a deep, implicit ensemble method based on theconcept of Feature-wise Linear Modulation (FiLM).
Revised: We introduce FiLM-Ensemble, a deep, implicit ensemble method based on the concept of Feature-wise Linear Modulation (FiLM).

Original: That technique was originallydeveloped for multi-task learning, with the aim of decoupling different tasks.
Revised: That technique was originally developed for multi-task learning, with the aim of decoupling different tasks.

Original: Weshow that the idea can be extended to uncertainty quantification: by modulatingthe network activations of a single deep network with FiLM, one obtains a modelensemble with high diversity, and consequently well-calibrated estimates of epistemic uncertainty, with low computational overhead in comparison.
Revised: We show that the idea can be extended to uncertainty quantification: by modulating the network activations of a single deep network with FiLM, one obtains a model ensemble with high diversity, and consequently well-calibrated estimates of epistemic uncertainty, with low computational overhead in comparison.

Original: Empirically,
Revised: Empirically,

Original: FiLM-Ensemble outperforms other implicit ensemble methods, and it comes veryclose to the upper bound of an explicit ensemble of networks (sometimes evenbeating it), at a fraction of the memory cost.
Revised: FiLM-Ensemble outperforms other implicit ensemble methods, and it comes very close to the upper bound of an explicit ensemble of networks (sometimes even beating it), at a fraction of the memory cost.

Original:  Introduction
Revised:  Introduction

Original: A key component for reliable and trustworthy machine learning are algorithms that output not onlyaccurate predictions of the target variables, but also well-calibrated estimates of their uncertainty Galand Ghahramani [2016].
Revised: A key component for reliable and trustworthy machine learning are algorithms that output not only accurate predictions of the target variables, but also well-calibrated estimates of their uncertainty Gal and Ghahramani [2016].

Original: The overall uncertainty of a predictor is usually decomposed into two parts[Der Kiureghian and Ditlevsen, 2009].
Revised: The overall uncertainty of a predictor is usually decomposed into two parts [Der Kiureghian and Ditlevsen, 2009].

Original: Aleatoric uncertainty is inherent in the data, for instance dueto class overlap or sensor noise.
Revised: Aleatoric uncertainty is inherent in the data, for instance due to class overlap or sensor noise.

Original: On the contrary, epistemic uncertainty characterises the uncertaintyof the model weights, due to a lack of knowledge about parts of the input space that are insufficientlyrepresented in the training set.
Revised: On the contrary, epistemic uncertainty characterises the uncertainty of the model weights, due to a lack of knowledge about parts of the input space that are insufficiently represented in the training set.

Original: Uncertainty caused by distributional shifts between the training and test data is sometimes conceived as a third source of uncertainty [Malinin and Gales, 2018], but inpractice often modelled as part of epistemic uncertainty.
Revised: Uncertainty caused by distributional shifts between the training and test data is sometimes conceived as a third source of uncertainty [Malinin and Gales, 2018], but in practice often modelled as part of epistemic uncertainty.

Original: Measuring epistemic uncertainty for complex models such as deep neural networks is not trivial: bydefinition one cannot derive it from the training data, since it concerns the behavior of the modelin regions of the input space that are not represented in the training set.
Revised: Measuring epistemic uncertainty for complex models such as deep neural networks is not trivial: by definition one cannot derive it from the training data, since it concerns the behavior of the model in regions of the input space that are not represented in the training set.

Original: Different methods havebeen explored [Welling and Teh, 2011, Graves, 2011, Blundell et al., 2015, Lakshminarayanan et al. ,2017a, Huang and Belongie, 2017, Gal and Ghahramani, 2016], however the de facto standard remaindeep ensemble models [Lakshminarayanan et al., 2017a].
Revised: Different methods have been explored [Welling and Teh, 2011, Graves, 2011, Blundell et al., 2015, Lakshminarayanan et al., 2017a, Huang and Belongie, 2017, Gal and Ghahramani, 2016], however the de facto standard remain deep ensemble models [Lakshminarayanan et al., 2017a].

Original: In its basic form, such a deep ensemble is simply a collection of independently trained networks that can be regarded as Monte Carlo samples from the model space.
Revised: In its basic form, such a deep ensemble is simply a collection of independently trained networks that can be regarded as Monte Carlo samples from the model space.

Original: To obtain ensemble members with reasonably low correlation, one can exploit the stochastic nature of the optimisation procedure, with different (random) weight initialisation anddifferent (random) batches during stochastic gradient descent.
Revised: To obtain ensemble members with reasonably low correlation, one can exploit the stochastic nature of the optimisation procedure, with different (random) weight initialisation and different (random) batches during stochastic gradient descent.

Original: The expectation is that, once trained,the ensemble members will agree for inputs near the training samples, since the loss function favourssimilar outputs at those locations.
Revised: The expectation is that, once trained, the ensemble members will agree for inputs near the training samples, since the loss function favours similar outputs at those locations.

Original: Whereas they may disagree in unseen regions of the data space.
Revised: Whereas they may disagree in unseen regions of the data space.

Original: Thus, the spread of their predictions is a measure of epistemic uncertainty.
Revised: Thus, the spread of their predictions is a measure of epistemic uncertainty.

Original: The ensemble idea isconceptually very simple, but nevertheless yields uncertainties that are well calibrated, i.e., they arein line with the actual prediction errors (in expectation).
Revised: The ensemble idea is conceptually very simple, but nevertheless yields uncertainties that are well calibrated, i.e., they are in line with the actual prediction errors (in expectation).

Original: The drawback of deep ensembles is their large computational cost.
Revised: The drawback of deep ensembles is their large computational cost.

Original: Both computation and memoryconsumption grow directly proportionally with the number of ensemble members, during training aswell as during inference.
Revised: Both computation and memory consumption grow directly proportionally with the number of ensemble members, during training as well as during inference.

Original: This makes them impractical in hardware-constrained settings, and leads toa widening gap as models continue to grow in size, while at the same time applications like mobilerobotics, augmented reality and smart sensor networks increase the need for mobile and embeddedcomputing.
Revised: This makes them impractical in hardware-constrained settings, and leads to a widening gap as models continue to grow in size, while at the same time applications like mobile robotics, augmented reality and smart sensor networks increase the need for mobile and embedded computing.

Original: To improve efficiency, several researchers have explored ways to mimic deep ensembles withoutexplicitly duplicating the underlying network.
Revised: To improve efficiency, several researchers have explored ways to mimic deep ensembles without explicitly duplicating the underlying network.

Original: Possible strategies include the reuse and recombinationof network modules [e.g., Wen et al., 2020, Havasi et al., 2021], injection of noise at inferencetime [e.g., Gal and Ghahramani, 2016], as well as hybrid variants [e.g., Durasov et al., 2021].
Revised: Possible strategies include the reuse and recombination of network modules [e.g., Wen et al., 2020, Havasi et al., 2021], injection of noise at inference time [e.g., Gal and Ghahramani, 2016], as well as hybrid variants [e.g., Durasov et al., 2021].

Original: Empirically, these models do speed up training and/or inference, but still exhibit a significantperformance gap compared to the naïve, explicit ensemble, both w.r.t.
Revised: Empirically, these models do speed up training and/or inference, but still exhibit a significant performance gap compared to the naïve, explicit ensemble, both w.r.t.

Original: prediction quality and w.r.t.
Revised: prediction quality and w.r.t.

Original: thecalibration of the predicted (epistemic) uncertainties.
Revised: the calibration of the predicted (epistemic) uncertainties.

Original: The lottery ticket hypothesis [Frankle and Carbin, 2018] and other network pruning studies, e.g.,
Revised: The lottery ticket hypothesis [Frankle and Carbin, 2018] and other network pruning studies, e.g.,

Original: Han et al.
Revised: Han et al.

Original: [2015], Lee et al.
Revised: [2015], Lee et al.

Original: [2019], Mallya and Lazebnik [2018], underline that neural networksare heavily over-parameterized.
Revised: [2019], Mallya and Lazebnik [2018], underline that neural networks are heavily over-parameterized.

Original: Their parameters are used inefficiently, and they can be prunedsignificantly without large performance drops.
Revised: Their parameters are used inefficiently, and they can be pruned significantly without large performance drops.

Original: In lifelong learning and multi-task learning it isessential to use the network efficiently, in order to limit computational overhead when introducingnew tasks.
Revised: In lifelong learning and multi-task learning it is essential to use the network efficiently, in order to limit computational overhead when introducing new tasks.

Original: There are recent works that achieve good performance in these tasks by introducingmodulation (respectively, adaptation) strategies.
Revised: There are recent works that achieve good performance in these tasks by introducing modulation (respectively, adaptation) strategies.

Original: In particular, Li et al.
Revised: In particular, Li et al.

Original: [2018], Takeda et al.
Revised: [2018], Takeda et al.

Original: [2021]propose an efficient lifelong learning / domain adaptation method for multi-task learning, utilizingsingle feature-wise linear modulation (FiLM).
Revised: [2021] propose an efficient lifelong learning / domain adaptation method for multi-task learning, utilizing single feature-wise linear modulation (FiLM).

Original: Inspired by that line of work, we propose a new, efficient ensemble method, FiLM-Ensemble.
Revised: Inspired by that line of work, we propose a new, efficient ensemble method, FiLM-Ensemble.

Original: Ourmethod adapts feature-wise linear modulation as an alternative way to construct an ensemble for(epistemic) uncertainty estimation.
Revised: Our method adapts feature-wise linear modulation as an alternative way to construct an ensemble for (epistemic) uncertainty estimation.

Original: FiLM-Ensemble greatly reduces the computational overheadcompared to the naïve ensemble approach, while performing almost on par with it, sometimes evenbetter.
Revised: FiLM-Ensemble greatly reduces the computational overhead compared to the naïve ensemble approach, while performing almost on par with it, sometimes even better.

Original: In a nutshell, FiLM-Ensemble can be described as an implicit model ensemble in which eachindividual member is defined via its own set of linear modulation parameters for the activations,whereas all other network parameters are shared among ensemble members – and therefore only needto be stored and trained once.
Revised: In a nutshell, FiLM-Ensemble can be described as an implicit model ensemble in which each individual member is defined via its own set of linear modulation parameters for the activations, whereas all other network parameters are shared among ensemble members – and therefore only need to be stored and trained once.

Original: Thanks to this design, our method requires only a very small numberof additional parameters on top of the base network, e.g., converting a single ResNet-18 model to anensemble of 16 models increases the parameter count by 1.3%, compared to an increase by 1500%when setting up a naïve ensemble, see Table 1.
Revised: Thanks to this design, our method requires only a very small number of additional parameters on top of the base network, e.g., converting a single ResNet-18 model to an ensemble of 16 models increases the parameter count by 1.3%, compared to an increase by 1500% when setting up a naïve ensemble, see Table 1.

Original: We further show that FiLM-Ensemble results inmore diverse ensemble members compared to other efficient ensemble methods.
Revised: We further show that FiLM-Ensemble results in more diverse ensemble members compared to other efficient ensemble methods.

Original: For instance, onthe Cifar10 benchmark it achieves diversity scores > 6 .
Revised: For instance, on the Cifar-10 benchmark it achieves diversity scores > 6 .

Original: 9% and up to 9 .
Revised: 9% and up to 9 .

Original: 2% (depending on ensemblesize), against 6 . 8% for a naive ensemble, see Fig.
Revised: 2% (depending on ensemble size), against 6 . 8% for a naive ensemble, see Fig.

Original: Our contributions can be summarized as follows.
Revised: Our contributions can be summarized as follows

Original: • We propose FiLM-Ensemble, a novel parameter- and time-efficient deep ensemble method.
Revised: • We propose FiLM-Ensemble, a novel parameter- and time-efficient deep ensemble method.

Original: • FiLM-Ensemble is designed in such a way that it can be readily combined with many popular deep learning models, by simply replacing batch normalisation (batchnorm) layers with conditional batchnorm layers.
Revised: • FiLM-Ensemble is designed in such a way that it can be readily combined with many popular deep learning models, by simply replacing batch normalisation (batchnorm) layers with conditional batchnorm layers.

Original: • We show that FiLM-Ensemble provides an excellent trade-off between accuracy and calibra- tion performance, improving over existing ensemble methods.
Revised: • We show that FiLM-Ensemble provides an excellent trade-off between accuracy and calibration performance, improving over existing ensemble methods.

Original:  FiLM-Ensemble
Revised:  FiLM-Ensemble

Original: In [Perez et al., 2018] the linear feature modulation was achieved by varying the affine parameters ofthe batch normalization layers γ and β .
Revised: In [Perez et al., 2018] the linear feature modulation was achieved by varying the affine parameters of the batch normalization layers γ and β .

Original: For each batchnorm layer n in the network, the parameters are predicted according to some conditioning input z (for instance, different prediction tasks):
Revised: For each batchnorm layer n in the network, the parameters are predicted according to some conditioning input z (for instance, different prediction tasks):

Original: [Equation]
Revised: [Equation]

Original: The size of γ n and β n is R D n where D n is the feature dimension at layer n .
Revised: The size of γ n and β n is R D n where D n is the feature dimension at layer n .

Original: These parameters arethen used to linearly modulate the activations at the n -th layer F n :
Revised: These parameters are then used to linearly modulate the activations at the n -th layer F n :

Original: [Equation]
Revised: [Equation]

Original: with ◦ being the Hadamard (element-wise) product taken w.r.t.
Revised: with ◦ being the Hadamard (element-wise) product taken w.r.t.

Original: the feature dimension.
Revised: the feature dimension.

Original: In our scenario we aim to use the affine parameters to instantiate different ensemble members, i.e.,the conditioning variable z is simply an index m ∈ { 1 , ..., M } that identifies the ensemble member.
Revised: In our scenario we aim to use the affine parameters to instantiate different ensemble members, i.e., the conditioning variable z is simply an index m ∈ { 1 , ..., M } that identifies the ensemble member.

Original: The functions g n ( z ) and h n ( z ) degenerate to look-up tables, so we can dispose of them and simplylearn M different sets of affine parameters for every batchnorm layer F n :
Revised: The functions g n ( z ) and h n ( z ) degenerate to look-up tables, so we can dispose of them and simply learn M different sets of affine parameters for every batchnorm layer F n :

Original: [Equation]
Revised: [Equation]

Original: To achieve ensemble members with diverse parameters, we resort to Xavier initialization, i.e., all γ mnand β mn are sampled from a uniform distribution bounded between:
Revised: To achieve ensemble members with diverse parameters, we resort to Xavier initialization, i.e., all γ m n and β mn are sampled from a uniform distribution bounded between:

Original: [Equation]
Revised: [Equation]

Original: where D n is the number of features (channels) in the n -th layer, and ρ is an initialization gain factor.
Revised: where D n is the number of features (channels) in the n -th layer, and ρ is an initialization gain factor.

Original: In our setting ρ is a tunable hyperparameter that allows one to control the trade-off between predictiveaccuracy and calibration of the model, see Section 3.6.
Revised: In our setting ρ is a tunable hyperparameter that allows one to control the trade-off between predictive accuracy and calibration of the model, see Section 3.5.

Original: In general, increasing ρ leads to more diverseensemble members and thus favours calibration.
Revised: In general, increasing ρ leads to more diverse ensemble members and thus favours calibration.

Original: Note that as ρ → 0 , all ensemble members startfrom similar initial values for γ mn and β mn and the FiLM-Ensemble gradually collapses to a singlemodel.
Revised: Note that as ρ → 0 , all ensemble members start from similar initial values for γ mn and β mn and the FiLM-Ensemble gradually collapses to a single model.

Original: During training, we feed each input sample x to every ensemble member and obtain predictionsy m = f θ , γ m , β m ( x ) , which depend both on the member-specific parameters ( β m , γ m ) and on theshared parameters θ .
Revised: During training, we feed each input sample x to every ensemble member and obtain predictions y m = f θ , γ m , β m ( x ) , which depend both on the member-specific parameters ( β m , γ m ) and on the shared parameters θ .

Original: All those parameters are optimized together to minimise the chosen loss function.
Revised: All those parameters are optimized together to minimise the chosen loss function.

Original: Here we focus on classification and use a standard cross-entropy (CE) loss.
Revised: Here we focus on classification and use a standard cross-entropy (CE) loss.

Original: At inference time the final prediction ˆ y is obtained by averaging the M predictions of the ensemble:
Revised: At inference time the final prediction ˆ y is obtained by averaging the M predictions of the ensemble:

Original: [Equation]
Revised: [Equation]

Original: 2.1 Implementation Details
Revised: 2.1 Implementation Details

Original: We implemented 1-dimensional and 2-dimensional FiLM-Ensemble layers, to be used in combinationwith 1-dimensional and 2-dimensional convolution operations, respectively.
Revised: We implemented 1-dimensional and 2-dimensional FiLM-Ensemble layers, to be used in combination with 1-dimensional and 2-dimensional convolution operations, respectively.

Original: The forward passesthrough different ensemble members can easily be parallelized by replicating both the input tensor andthe FiLM parameters γ , β along the batch dimension and applying the affine transformation Eq.
Revised: The forward passes through different ensemble members can easily be parallelized by replicating both the input tensor and the FiLM parameters γ , β along the batch dimension and applying the affine transformation Eq.

Original: 3 (thesame holds for the BatchEnsemble method).
Revised: 3 (the same holds for the BatchEnsemble method).

Original: In this way all ensemble members run simultaneouslyon a single device, thus optimally utilizing modern tensor computing hardware without having to loadseveral instances of the classification network into memory.
Revised: In this way all ensemble members run simultaneously on a single device, thus optimally utilizing modern tensor computing hardware without having to load several instances of the classification network into memory.

Original: We have implemented FiLM-Ensemblein PyTorch andwill release the source code, including implementations of the baselines used in theexperiments, together with trained model checkpoints.
Revised: We have implemented FiLM-Ensemble in PyTorch and release the source code, including implementations of the baselines used in the experiments, together with trained model checkpoints.

Original: The source code is submitted as supplementarymaterial.

Original: For all experiments, we optimize the model parameters with standard stochastic gradient descent,with momentum µ = 0 .
Revised: For all experiments, we optimize the model parameters with standard stochastic gradient descent, with momentum µ = 0 .

Original: 9 and weight decay λ = 0 .
Revised: 9 and weight decay λ = 0 .

Original: 0005 for regularization.
Revised: 0005 for regularization.

Original: We train for 200epochs with batch size 128.
Revised: We train forepochs with batch size 128.

Original: The learning rate is initially set to 0 .
Revised: The learning rate is initially set to 0 .

Original: 1 and decays according to a cosineannealing schedule [Loshchilov and Hutter, 2017].
Revised: 1 and decays according to a cosine annealing schedule [Loshchilov and Hutter, 2017].

Original: The initialisation gain is set to ρ = 2 for allexperiments, except for genome sequences (see Table 3), where ρ ∈ { 4 , 8 , 16 , 32 } .
Revised: The initialisation gain is set to ρ = 2 for all experiments, except for genome sequences (see Table 3), where ρ ∈ { 4 , 8 , 16 , 32 } .

Original: Please refer tothe supplementary material for additional, dataset-specific details.
Revised: Please refer to the supplementary material for additional, dataset-specific details.

Original:  Experiments
Revised:  Experiments

Original: In the following, we first empirically examine the ability of FiLM-Ensemble to learn independentensemble members, and compare it to deep ensembles.
Revised: In the following, we first empirically examine the ability of FiLM-Ensemble to learn independent ensemble members, and compare it to deep ensembles.

Original: Then we analyze the predictions accuracy and uncertainty calibration of FiLM-Ensemble, as well as its computational cost and memory consumption.
Revised: Then we analyze the predictive accuracy and uncertainty calibration of our method, as well as its computational cost and memory consumption.

Original: We compare its performance against several baseline methods and state-of-the-art alternatives on avariety of different datasets, and using different backbone architectures.
Revised: We compare its performance against several baseline methods and state-of-the-art alternatives on a variety of different datasets, and using different backbone architectures.

Original: All experimental results areaveraged over three runs with different random seeds.
Revised: All experimental results are averaged over three runs with different random seeds.

Original: We evaluate our proposed method on a diverse set of classification tasks, including popular imageclassification benchmarks, image-based medical diagnosis, and genome sequence analysis.
Revised: We evaluate our proposed method on a diverse set of classification tasks, including popular image classification benchmarks, image-based medical diagnosis, and genome sequence analysis.

Original: CIFAR10 and CIFAR-100 [Krizhevsky, 2009] are widely used testbeds for image classification, and deeplearning in general.
Revised: CIFAR10 and CIFAR-100 [Krizhevsky, 2009] are widely used testbeds for image classification, and deep learning in general.

Original: They consist of clean images of objects from 10, respectively 100, differentsemantic classes.
Revised: They consist of clean images of objects from 10, respectively 100, different semantic classes.

Original: Each of the two datasets contains 60,000 images, of which 10,000 are reservedfor testing.
Revised: Each of the two datasets contains 60,000 images, of which 10,000 are reserved for testing.

Original: The semantic classes are uniformly distributed in the datasets and stratified across thetrain/test spit.
Revised: The semantic classes are uniformly distributed in the datasets and stratified across the train/test spit.

Original: Retina Glaucoma Detection [Diaz-Pinto et al., 2019] is a real-world clinical datasetthat includes microscopic retina images from 956 patients with the neuropathic disease Glaucoma,and from 1401 subjects with normal (healthy) retinas.
Revised: Retina Glaucoma Detection [Diaz-Pinto et al., 2019] is a real-world clinical dataset that includes microscopic retina images from 956 patients with the neuropathic disease Glaucoma, and from 1401 subjects with normal (healthy) retinas.

Original: Each input sample is a single RGB image;we resize all images to 128 × 128 .
Revised: Each input sample is a single RGB image; we resize all images to 128 × 128 .

Revised: The image augmentation applies by combination of crop, horizontal flip, and color jitter.

Original: REFUGE 

Revised: REFUGE

Original: [Orlando et al., 2020] was a challenge at the MICCAI-2020 conference,again aimed at retinal Glaucoma.
Revised: [Orlando et al., 2020] was a challenge at the MICCAI-2020 conference, aimed at retinal Glaucoma diagnosis.

Original: The dataset consists of 800 microscopicretina images with size 1411 × 1411 pixels, collected from different clinics.
Revised: The dataset consists of 800 microscopic retina images with size 1411 × 1411 pixels, collected from different clinics.

Original: We use this dataset inconjunction with the models trained on the previous dataset [Diaz-Pinto et al., 2019], to evaluate theability to detect out-of-distribution samples with the help of the predicted uncertainties (Table 4).6mA Identification [Li et al., 2021], is a 1-dimensional sequential genome dataset.
Revised: We use this dataset in conjunction with the models trained on the previous dataset [Diaz-Pinto et al., 2019], to evaluate the ability to detect out-of-distribution samples with the help of the predicted uncertainties (Table 4). 6mA Identification [Li et al., 2021], is a 1-dimensional sequential genome dataset.

Original: It consists of DNA sequences of rice plants, along with binary labels that indicate whether the sequence is a N6methyladenine (6mA) site.
Revised: It consists of DNA sequences of rice plants, along with binary labels that indicate whether the sequence is a N6-methyladenine (6mA) site.

Original: 6mA is an important DNA modification associated with several biologicalprocesses, such as regulating gene transcription, DNA replication and DNA repair [Campbell and 
Revised: 6mA is an important DNA modification associated with several biological processes, such as regulating gene transcription, DNA replication and DNA repair [Campbell and Kleckner, 1990,

Original: Kleckner, 1990, Cheng et al., 2016, Pukkila et al., 1983].
Revised: Cheng et al., 2016, Pukkila et al., 1983].

Original: Each sequence consists of 41 nucleotides.
Revised: Each sequence consists of 41 nucleotides.

Original: As there are 4 different types of nucleotides, each one-hot encoded sequence is represented by a41 × 4 matrix.
Revised: As there aredifferent types of nucleotides, each one-hot encoded sequence is represented by a 41 × 4 matrix.

Original: In total there are 269,500 training samples and 38,500 test samples.
Revised: In total there are 269,500 training samples and 38,500 test samples.

Original: We compare FiLM-Ensemble against (i) a single model without any ensembling, as the elementarybaseline, (ii) a naïve deep ensemble
Revised: We compare FiLM-Ensemble against (i) a single model without any ensembling, as the elementary baseline, (ii) a naïve deep ensemble

Original: [Lakshminarayanan et al., 2017b], and (iii) MC-Dropout [Galand Ghahramani, 2016].
Revised: [Lakshminarayanan et al., 2017b], and (iii) MC-Dropout [Gal and Ghahramani, 2016].

Original: Furthermore, we compare with other state-of-the-art methods, namely: (iv) Masksemble [Durasov et al., 2021] which can be seen as an extension of MC-Dropout, (v) MIMO (“multi-input multi-output")
Revised: Furthermore, we compare with other state-of-the-art methods, including: (iv) Masksemble [Durasov et al., 2021] which can be seen as an extension of MC-Dropout, (v) MIMO ("multi-input multi-output")

Original: [Havasi et al., 2021] which defines a multi-head architecture where eachhead acts as one ensemble member, and
Revised: [Havasi et al., 2021] which defines a multi-head architecture where each head acts as one ensemble member, and

Original: (vi) BatchEnsemble [Wen et al., 2020] which creates anefficient ensemble by expanding the layer weights using low rank matrices.
Revised: (vi) BatchEnsemble [Wen et al., 2020] which creates an efficient ensemble by expanding the layer weights using low rank matrices.

Original: Arguably, this layer-wisemodification of an underlying, common representation is the approach that comes closest to our work.
Revised: Arguably, this layer-wise modification of an underlying, common representation is the approach that comes closest to our work.

Original: 3.1 Diversity Analysis
Revised: 3.1 Diversity Analysis

Original: Diversity is the key to constructing powerful ensembles: nothing is to be gained from highly correlatedensemble members that return similar outputs for (almost) any input [Zhang and Ma, 2012].
Revised: Diversity is the key to constructing powerful ensembles: nothing is to be gained from highly correlated ensemble members that return similar outputs for (almost) any input [Zhang and Ma, 2012].

Original: In orderto analyze the diversity (respectively, the degree of independence) among members, we compute twodistance metrics between the members’ predictive distributions.
Revised: In order to analyze the diversity (respectively, the degree of independence) among members, we compute two distance metrics between the members’ predictive distributions.

Original: Let f i and f j denote two differentensemble members for a classification task.
Revised: Let f i and f j denote two different ensemble members for a classification task.

Original: We first measure the disagreement score D , defined asthe fraction of all s test samples for which the two members return different answers, averaged overall possible pairs of members:
Revised: We first measure the disagreement score D , defined as the fraction of all s test samples for which the two members return different answers, averaged over all possible pairs of members:

Original: [Equation]
Revised: [Equation]

Original: with [ · ] being the Iverson bracket.
Revised: with [ · ] being the Iverson bracket.

Original: Second, we also measure the Kullback–Leibler (KL) divergencebetween the two predictive distributions p ( f i ) and p ( f j ) , again averaged over all pairs:
Revised: Second, we also measure the Kullback–Leibler (KL) divergence between the two predictive distributions p ( f i ) and p ( f j ) , again averaged over all pairs:

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: For both metrics, higher numbers correspond to higher diversity.
Revised: For both metrics [Table] higher numbers correspond to higher diversity.

Original: See Fig.
Revised: See Fig.

Original: We observe that theaverage diversity increases with the number of ensemble members.
Revised: We observe that the average diversity increases with the number of ensemble members.

Original: In both metrics, FiLM-Ensembleachieves higher scores than the naïve, explicit deep ensemble.
Revised: In both metrics, FiLM-Ensemble achieves higher scores than the naïve, explicit deep ensemble.

Original: Meaning that the predictions of FiLMEnsemble are less correlated than those of an equivalent number of networks trained independentlywith different random seeds.
Revised: Meaning that the predictions of FiLMEnsemble are less correlated than those of an equivalent number of networks trained independently with different random seeds.

Revised: Also, note that with the increasing number of members, improvements in diversity metrics for the naïve explicit deep ensemble are negligible

Original: [Table]
Revised: [Table]

Revised: diverse representations.

Revised: See Section 3.1.

Original: 3.2 Computational Cost
Revised: 3.2 Computational Cost

Original: [Table]

Original: We go on to measure the efficiency of our proposed method, both in terms of parameter count and interms of resources needed for a forward pass.
Revised: We go on to measure the efficiency of our proposed method, both in terms of parameter count and in terms of resources needed for a forward pass.

Original: Table 1 (left section) reports the numbers of additionalnumbers of parameters compared to a single network (i [Table] e., without uncertainty calibration).
Revised: Table 1 (left section) reports the numbers of additional numbers of parameters compared to a single network (i.e., without uncertainty calibration).

Original: Due totheir design, MC-Dropout and Masksemble do not require any additional parameters.
Revised: Due to their design, MC-Dropout and Masksemble do not require any additional parameters.

Original: Among theremaining methods, FiLM-Ensemble has the lowest (VGG-11) or second-lowest (Resnet18) overheadin terms of parameter count, while performing significantly better, as we will see below.
Revised: Among the remaining methods, FiLM-Ensemble has the lowest (VGG-11) or second-lowest (Resnet-18) overhead in terms of parameter count, while performing significantly better, as we will see below.

Original: In terms ofinference complexity (Tab. 1, center and right), FiLM-Ensemble turns out to be very competitive.
Revised: In terms of inference complexity (Tab. 1, center and right), FiLM-Ensemble turns out to be very competitive.

Original: Only MIMO with a ResNet-18 backbone is significantly faster and lighter, however this comes at aconsiderable price in terms of accuracy and calibration, see below.
Revised: Only MIMO with a ResNet-18 backbone is significantly faster and lighter, however this comes at a considerable price in terms of accuracy and calibration, see below.

Original: 3.3 CIFAR10 / CIFAR100 
Revised: 3.3 CIFAR-10 / CIFAR-

Original: We start with widely usedstandard benchmarks in computer vision, CIFAR10 and CIFAR100.
Revised: We perform several experiments on widely used benchmarks in computer vision, CIFAR-10 and CIFAR-100.

Original: Asperformance metrics, we plot the test set accuracy and the expected calibration error  (ECE) againstthe ensemble size M in Fig.
Revised: As performance metrics, we plot the test set accuracy and the expected calibration error [Table] (ECE) against the ensemble size M in Fig.

Original: 2, for all compared methods.
Revised: 2, for all compared methods.

Original: In terms of accuracy (left subfigure),
Revised: In terms of accuracy (left subfigure)

Original:  FiLM-Ensemble is outperformed only by the explicit deep ensemble, across all tested values of M ∈ { 2 , 4 , 8 , 16 } .
Revised: [Table] FiLM-Ensemble is outperformed only by the explicit deep ensemble, across all tested values of M ∈ { 2 , 4 , 8 , 16 } .

Original: These two surpass all other methods by a clear margin.
Revised: These two surpass all other methods by a clear margin.

Original: Surprisingly, we observethat BatchEnsemble generally exhibits a negative correlation between ensemble size and test setaccuracy, with M = 4 performing best among all settings.
Revised: Surprisingly, we observe that BatchEnsemble generally exhibits a negative correlation between ensemble size and test set accuracy, with M = 4 performing best among all settings.

Original: MIMO shows very poor performancein this experimental setting in terms of accuracy (and also in ECE).
Revised: MIMO shows very poor performance in this experimental setting in terms of accuracy (and also in ECE).

Original: We speculate that its sharedbackbone probably has a tendency to fragment into largely independent ensemble members of lowchannel depth, which lack the necessary capacity when using comparatively small networks like VGG11 or Resnet18.
Revised: We speculate that its shared backbone probably has a tendency to fragment into largely independent ensemble members of low channel depth, which lack the necessary capacity when using comparatively small networks like VGG-11 or Resnet-18.

Original: With regard to ECE (right subfigure), we see all methods improving withgrowing ensemble size M .
Revised: With regard to ECE (right subfigure), we see all methods improving with growing ensemble size M .

Original: FiLM-Ensemble achieves better calibration then the widely used deep[Table] ensemble and MC-dropout methods, with significant margins at large ensemble sizes.
Revised: FiLM-Ensemble achieves better calibration then the widely used deep ensemble and MC-dropout methods, with significant margins at large ensemble sizes.

Original: Masksembleand BatchEnsemble achieve very good calibration for sizes M ∈ [4 . . .
Revised: Masksemble and BatchEnsemble achieve very good calibration for sizes M ∈ [4 . . .

Original: [Table] but at the cost ofsignificant drops in classification performance.
Revised: 16] , but at the cost of significant drops in classification performance.

Original: An attractive feature of FiLM-Ensemble is that itoffers a simple mechanism for trading off accuracy against calibration, by tuning the gain factor ρ .
Revised: An attractive feature of FiLM-Ensemble is that it offers a simple mechanism for trading off accuracy against calibration, by tuning the gain factor ρ .

Original: See Section 3.6.
Revised: See Section 3.5.

Original: In Tab. 2 we quantitatively compare all tested methods on the CIFAR-100 dataset, both with astandard Resnet-18 backbone and with a larger Resnet-34.
Revised: In Tab. 2 we quantitatively compare all tested methods on the CIFAR-100 dataset, both with a standard Resnet-18 backbone and with a larger Resnet-34.

Original: We observe that with both architectures,and in terms of both predictive accuracy and calibration, FiLM-Ensemble always either ranks first, orit ranks second behind the inefficient, explicit deep ensemble.
Revised: We observe that with both architectures, and in terms of both predictive accuracy and calibration, FiLM-Ensemble always either ranks first, or it ranks second behind the inefficient, explicit deep ensemble.

Revised: Also, note that when the capacity of the network is increased (Resnet-18 → Resnet-34), some of the other implicit methods except

Revised: MIMO see an improvement in calibration performance but at the cost of a reduction in accuracy.

Original: Overall, the experiments with bothvariants of CIFAR and all three model architectures confirm that our method presents an excellenttrade-off between predictive accuracy, uncertainty calibration and computational efficiency. 
Revised: Overall, the experiments with both variants of CIFAR and all three model architectures confirm that our method presents an excellent trade-off between predictive accuracy, uncertainty calibration and computational efficiency. [Table]

Original: 3.4 Retinal Glaucoma Detection
Revised: 3.4 Retinal Glaucoma Detection

Original: Glaucoma is currently the leading reason of irreversible blindness in the world.
Revised: Glaucoma is currently the leading reason of irreversible blindness in the world.

Original: Detection ofglaucomatous structural damages and changes is a challenging task in the field of ophthalmology.
Revised: Detection of glaucomatous structural damages and changes is a challenging task in the field of ophthalmology.

Original: We evaluate our proposed FiLM-Ensemble, as well as the baselines described above, on the ask ofdiagnosing Glaucoma and quantifying the uncertainty associated with the prediction, see Table 3.
Revised: We evaluate our proposed FiLM-Ensemble, as well as the baselines described above, on the ask of diagnosing Glaucoma and quantifying the uncertainty associated with the prediction, see Table 3.

Original: The proposed FiLM-Ensemble achieves the best classification result across all ensemble sizes, andalso the best overall result, with M = 16 .
Revised: The proposed FiLM-Ensemble achieves the best classification result across all ensemble sizes, and also the best overall result, with M = 16 .

Original: Whereas there is no clear trend with respect to uncertaintycalibration.
Revised: Whereas there is no clear trend with respect to uncertainty calibration.

Revised: 3.5 6mA Identification

Revised: With the 6mA identification task we show that FiLM-Ensemble can also be readily combined with existing models for 1-dimensional sequential genome data.

Revised: We use the 6mA-rice-Lv dataset and a 1D-CNN architecture whose hyper-parameters have already been optimized for this dataset [Li et al., 2021].

Original: 3.5 Out-of-Distribution Detection
Revised: FiLM-Ensemble improves the accuracy and the calibration of that model, see Fig.3. Our method performs on par with the explicit deep ensemble and better than a single instance of the model tuned for the specific task. More importantly, one can reach a significantly better calibration (lower ECE) by increasing the gain ρ , with only a minimal accuracy drop by < 0 . 25 percent points.

Revised: Please refer to the Section B.2 for more results.

Revised: 3.6 Out-of-Distribution Detection

Original: Domain shift often occurs in medical datasets; thus, detecting out-of-distribution (OOD) samplesis an important task in clinical diagnosis.
Revised: Domain shift often occurs in medical datasets; thus, detecting out-of-distribution (OOD) samples is an important task in clinical diagnosis.

Original: Model uncertainty can be used for OOD detection.
Revised: Model uncertainty can be used for OOD detection.

Original: Forour experiment, we regard the retinal Glaucoma dataset [Diaz-Pinto et al., 2019] as the within- [Table] [Table] distribution samples and perform OOD detection with the REFUGE dataset [Orlando et al., 2020].
Revised: For our experiment, we regard the retinal Glaucoma dataset [Diaz-Pinto et al., 2019] as the withindistribution samples and perform OOD detection with the REFUGE dataset [Orlando et al., 2020].

Original: Asperformance metric, we report AUROC (Area Under the Receiver Operating Characteristic) scores,
Revised: As performance metric, we report AUROC (Area Under the Receiver Operating Characteristic) scores,

Original: Table
Revised: [Table]

Original: FiLM-Ensemble can detect OOD test samples with significantly higher accuracy than otherefficient ensemble methods, standard state-of-the-art OOD detection methods
Revised: efficient ensemble methods, standard state-of-the-art OOD detection methods

Original: [Table]
Revised: [Table]

Original: and even the explicitdeep ensemble.
Revised: and even the explicit deep ensemble.

Original: This suggests that for challenging test samples, which are not adequately representedin the training data, the uncertainties estimated with FiLM-Ensemble are better calibrated.
Revised: This suggests that for challenging test samples, which are not adequately represented in the training data, the uncertainties estimated with FiLM-Ensemble are better calibrated

Original: 3.6 6mA Identification

Original: With the 6mA dataset we show that FiLM-Ensemble can also be readily combined with existingmodels for 1-dimensional sequential genome data. We use a 1D-CNN architecture whose hyperparameters have already been optimized for the dataset [Li et al., 2021]. FiLM-Ensemble improvesthe accuracy and the calibration of that model, see Fig.3. Our method performs on par with theexplicit deep ensemble and better than a single instance of the model tuned for the specific task.
Revised: [Table]

Original: Moreimportantly, one can reach a significantly better calibration (lower ECE) by increasing the gain ρ ,with only a minimal accuracy drop by < 0 .

Original: 25 percent points.

Original:  Related Work 
Revised:  Related Work 4.1 Epistemic Uncertainty Quantification

Original: 4.1 Epistemic Uncertainty Quantification

Original: A large corpus of related work addresses the estimation of epistemic (model) uncertainty in neuralnetworks.
Revised: A large corpus of related work addresses the estimation of epistemic (model) uncertainty in neural networks.

Original: At the heart of such modeling is often the concept of marginalization instead of optimization , i.e., integrating out a (possibly uncountably infinite) set of models weighted by their posteriorprobability, instead of committing to a point estimate of that distribution.
Revised: At the heart of such modeling is often the concept of marginalization instead of optimization , i.e., integrating out a (possibly uncountably infinite) set of models weighted by their posterior probability, instead of committing to a point estimate of that distribution.

Original: A multitude of methodshave been proposed to implement approximate Bayesian inference w.r.t.
Revised: A multitude of methods have been proposed to implement approximate Bayesian inference w.r.t.

Original: the model weights, given thetraining data and an appropriate prior, a process that is not analytically tractable in general [Kendalland Gal, 2017].
Revised: the model weights, given the training data and an appropriate prior, a process that is not analytically tractable in general [Kendall and Gal, 2017].

Original: For instance, methods based on variational inference [Graves, 2011, Ranganath et al., 2014, Blundellet al., 2015] seek to learn an approximate posterior distribution which is a member of a simpler familyof variational distributions.
Revised: For instance, methods based on variational inference [Graves, 2011, Ranganath et al., 2014, Blundell et al., 2015] seek to learn an approximate posterior distribution which is a member of a simpler family of variational distributions.

Original: This variational distribution can often be learned using backpropagation[Blundell et al., 2015] and can then be sampled from or sometimes even used for exact inference.
Revised: This variational distribution can often be learned using backpropagation [Blundell et al., 2015] and can then be sampled from or sometimes even used for exact inference.

Original: Markov chain Monte Carlo sampling (MCMC) approaches [Neal, 1996, Welling and Teh, 2011, Chenet al., 2014] construct a Markov chain which has the exact posterior distribution as its stationarydistribution, that can then be employed for sampling.
Revised: Markov chain Monte Carlo sampling (MCMC) approaches [Neal, 1996, Welling and Teh, 2011, Chen et al., 2014] construct a Markov chain which has the exact posterior distribution as its stationary distribution, that can then be employed for sampling.

Original: However, in practise, these approaches oftenfail to sufficiently explore high-dimensional, multi-modal loss landscapes as they are common indeep learning [Gustafsson et al., 2020].
Revised: However, in practise, these approaches often fail to sufficiently explore high-dimensional, multi-modal loss landscapes as they are common in deep learning [Gustafsson et al., 2020].

Original: 4.2 Ensembles and Sub-networks
Revised: 4.2 Ensembles and Sub-networks

Original: In a method referred to as deep ensembles [Lakshminarayanan et al., 2017a], a set of M neuralnetwork models are randomly and independently initialized, and are subjected to stochastic mini-batchsampling during SGD training.
Revised: In a method referred to as deep ensembles [Lakshminarayanan et al., 2017a], a set of M neural network models are randomly and independently initialized, and are subjected to stochastic mini-batch sampling during SGD training.

Original: The models generally converge to different minima in the parameterspace, and can be considered samples from an approximate posterior [Wilson and Izmailov, 2020,
Revised: The models generally converge to different minima in the parameter space, and can be considered samples from an approximate posterior [Wilson and Izmailov, 2020,

Original: Gustafsson et al., 2020, Izmailov et al., 2021].
Revised: Gustafsson et al., 2020, Izmailov et al., 2021].

Original: Deep ensembles often achieve the best calibration andpredictive accuracy [Ovadia et al., 2019, Gustafsson et al., 2020, Ashukha et al., 2020], but sufferfrom high computational complexity as it requires training, storing, and running inference on severalnetwork models.
Revised: Deep ensembles often achieve the best calibration and predictive accuracy [Ovadia et al., 2019, Gustafsson et al., 2020, Ashukha et al., 2020], but suffer from high computational complexity as it requires training, storing, and running inference on several network models.

Original: Many alternative methods have recently been proposed in an attempt of reducing either the computational cost or the storage cost of deep ensembles.
Revised: Many alternative methods have recently been proposed in an attempt of reducing either the computational cost or the storage cost of deep ensembles.

Original: Monte Carlo (MC) Dropout
Revised: Monte Carlo (MC) Dropout

Original: [Gal and Ghahramani,2016] runs multiple forward passes on the dropout layers in order to obtain multiple predictions andobtain an uncertainty estimate.
Revised: [Gal and Ghahramani, 2016] runs multiple forward passes on the dropout layers in order to obtain multiple predictions and obtain an uncertainty estimate.

Original: Although, the method requires less computations compared to deepensembles, it also leads to less accurate uncertainty estimates.
Revised: Although, the method requires less computations compared to deep ensembles, it also leads to less accurate uncertainty estimates.

Original: Snapshot ensembles [Huang et al.
Revised: Snapshot ensembles [Huang et al.,

Original: ,2017] uses a cyclic learning rate scheduling in order to find multiple local minima, they then storemultiple copies of the network to create the ensemble.
Revised: 2017] uses a cyclic learning rate scheduling in order to find multiple local minima, they then store multiple copies of the network to create the ensemble.

Original: While this approach reduces the computationcost during training it does not alleviate the storage cost.
Revised: While this approach reduces the computation cost during training it does not alleviate the storage cost.

Original: BatchEnsemble [Wen et al., 2020] propose to use multiple low rank matrices, which can be storedefficiently, in order to modulate the parameters of a neural network and thus mimic an ensembleof network models.
Revised: BatchEnsemble [Wen et al., 2020] propose to use multiple low rank matrices, which can be stored efficiently, in order to modulate the parameters of a neural network and thus mimic an ensemble of network models.

Original: Masksembles [Durasov et al., 2021] aims to improve the performance of MC
Revised: Masksembles [Durasov et al., 2021] aims to improve the performance of MC

Original: Dropout by carefully selecting which masks, used to drop certain features, leading to a betteruncertainty quantification.
Revised: Dropout by carefully selecting which masks, used to drop certain features, leading to a better uncertainty quantification.

Original: Another approach which uses multiple sub-networks is proposed in Havasiet al.
Revised: Another approach which uses multiple sub-networks is proposed in Havasi et al.

Original: In this case additional, independent, layers are added at the beginning and at the endof the network in order to obtain multiple prediction with an individual network model.
Revised: In this case additional, independent, layers are added at the beginning and at the end of the network in order to obtain multiple prediction with an individual network model.

Original: Althoughsuch method seems to reduce the computational resource required at training and inference time,the gain is limited to larger backbones, such as use Wide-ResNet (e.g., ResNet28-10), whereas forstandard widely used architectures like VGG (e.g., -16), or ResNet (e.g., -34), the approach is notvery effective as shown in our experiments.
Revised: Although such method seems to reduce the computational resource required at training and inference time, the gain is limited to larger backbones, such as use Wide-ResNet (e.g., ResNet28-10), whereas for standard widely used architectures like VGG (e.g., -16), or ResNet (e.g., -34), the approach is not very effective as shown in our experiments.

Original: Although many methods that aim at reducing the computational cost of deep ensembles have beenproposed, none of the methods appear to unquestionably outperform the others.
Revised: Although many methods that aim at reducing the computational cost of deep ensembles have been proposed, none of the methods appear to unquestionably outperform the others.

Original: As a result thequestion of how to effectively model uncertainty is still open.
Revised: As a result the question of how to effectively model uncertainty is still open.

Original: 4.3 Feature-wise Linear Modulation
Revised: 4.3 Feature-wise Linear Modulation

Original: The idea of controlling the batch normalization parameters to modulate a network’s function has beenexplored by many different authors to accomplish different tasks.
Revised: The idea of controlling the batch normalization parameters to modulate a network’s function has been explored by many different authors to accomplish different tasks.

Original: Conditional batch normalization(CBN) was proposed by de Vries et al. , who achieved good performance in VQA experiments byusing an MLP to estimate residuals for normalization parameters used in a pre-trained ResNet [De
Revised: Conditional batch normalization (CBN) was proposed by de Vries et al. , who achieved good performance in VQA experiments by using an MLP to estimate residuals for normalization parameters used in a pre-trained ResNet [De

Original: Vries et al., 2017b].
Revised: Vries et al., 2017b].

Original: Perez et al.
Revised: Perez et al.

Original: [2018] proposed FiLM also for solving VQA tasks.
Revised: [2018] proposed FiLM also for solving VQA tasks.

Original: Strub et al.[2018] modify FiLM to produce normalization parameters in several stages instead of all at once.
Revised: Strub et al. [2018] modify FiLM to produce normalization parameters in several stages instead of all at once.

Original: This formulation is better able to handle longer conditioning information, such as dialogues insteadof questions, and achieved excellent results on the GuessWhat?!
Revised: This formulation is better able to handle longer conditioning information, such as dialogues instead of questions, and achieved excellent results on the GuessWhat?!

Original: visual dialogue task [De Vries et al. ,2017a].
Revised: visual dialogue task [De Vries et al., 2017a].

Original: Such modulation operations have also been used for image style transfer.
Revised: Such modulation operations have also been used for image style transfer.

Original: It has been observed that thestatistics of a feature map, which are associated with and can be controlled by the parameters producedby FiLM, are directly related to the style of an image [Huang and Belongie, 2017].
Revised: It has been observed that the statistics of a feature map, which are associated with and can be controlled by the parameters produced by FiLM, are directly related to the style of an image [Huang and Belongie, 2017].

Original: Dumoulin et al.[2017] succeeded in capturing the styles of artistic paintings, as well as combining extracted styles tocreate new ones, by using conditional instance normalization, which can be seen as a variation of
Revised: Dumoulin et al. [2017] succeeded in capturing the styles of artistic paintings, as well as combining extracted styles to create new ones, by using conditional instance normalization, which can be seen as a variation of

Original: FiLM.
Revised: FiLM.

Original: Ghiasi et al.
Revised: Ghiasi et al.

Original: [2017] expand on this work by jointly training a style prediction network anda style transfer network, which also operate based on conditional instance normalization.
Revised: [2017] expand on this work by jointly training a style prediction network and a style transfer network, which also operate based on conditional instance normalization.

Original: Finally,
Revised: Finally,

Original: Brock et al.
Revised: Brock et al.

Original: [2018] used FiLM for natural image synthesis using generative adversarial networks(GANs).
Revised: [2018] used FiLM for natural image synthesis using generative adversarial networks (GANs).

Original: They report that this allowed for a reduction in computation and memory costs, as well as a37% increase in training speed.
Revised: They report that this allowed for a reduction in computation and memory costs, as well as a 37% increase in training speed.

Original: Yang et al.
Revised: Yang et al.

Original: [2018] have used FiLM to modulate the layers of a segmentation network to performvideo object segmentation.
Revised: [2018] have used FiLM to modulate the layers of a segmentation network to perform video object segmentation.

Original: This made it possible to avoid the fine-tuning process that was usedby competing methods, which resulted in a speed up of 70 times while achieving similar accuracy.
Revised: This made it possible to avoid the fine-tuning process that was used by competing methods, which resulted in a speed up of 70 times while achieving similar accuracy.

Original: Feature modulation has also been applied for various other tasks.
Revised: Feature modulation has also been applied for various other tasks.

Original: Oreshkin et al.
Revised: Oreshkin et al.

Original: [2018] used FiLMfor task-dependent metric scaling, which allowed them to achieve excellent results in few-shotclassification.
Revised: [2018] used FiLM for task-dependent metric scaling, which allowed them to achieve excellent results in few-shot classification.

Original: Vuorio et al.
Revised: Vuorio et al.

Original: [2019] use FiLM for meta-learning via task-aware modulation.
Revised: [2019] use FiLM for meta-learning via task-aware modulation.

Original: Theauthors note that FiLM outperforms and is more stable than attention-based modulation in thiscontext.
Revised: The authors note that FiLM outperforms and is more stable than attention-based modulation in this context.

Original: Finally, Vinyals et al.
Revised: Finally, Vinyals et al.

Original: [2019] used FiLM in their AlphaStar neural architecture for multiagent reinforcement learning.
Revised: [2019] used FiLM in their AlphaStar neural architecture for multiagent reinforcement learning.

Original: To our knowledge, FiLM has so far not been used for (implicit) modelensembling or uncertainty quantification.
Revised: To our knowledge, FiLM has so far not been used for (implicit) model ensembling or uncertainty quantification.

Original:  Conclusion & Future Work
Revised:  Limitations & Future Work

Original: In this paper we present FiLM-Ensemble, a novel implicit deep ensembling method.
Revised: The work presented in this paper gives rise to several questions that can be explored in furure work. Using pre-trained models is standard procedure in deep learning applications. It would be useful to explore how FiLM-Ensemble performs when used in conjunction with pre-trained models. This may not be straightforward since FiLM-Ensemble (also other implicit ensemble methods e.g., BatchEnsemble, MIMO) is based on variations between ensemble members, which may be reduced if all members are similarly initialized.

Revised: Furthermore, self-attention-based models e.g., Transformers Vaswani et al.

Original: We achieve highefficiency using a simple yet effective idea – feature-wise linear modulation – which has been shownto be effective in different domains, such as image style transfer, model-agnostic meta-learning,or multi-task learning. Our extensive evaluation shows that FiLM-Ensemble outperforms, or is onpar with, state-of-the-art ensemble methods in many different experimental settings.
Revised: [2017] and Vision Transformers (ViTs) Dosovitskiy et al. [2021] have recently become very popular; therefore it is natural to adapt FilM-Ensemble to work with such models. Note that layer normalization is standard in Transformers instead of batch normalization, which prevents straightforward application of the presented method in this case.

Original: A numberof measures designed to enhance implicit ensembles are orthogonal to our approach and couldpotentially be combined with FiLM-Ensemble to further improve its performance and uncertaintycalibration, while minimally increasing the computational costs.
Revised: Also, a number of measures designed to enhance implicit ensembles are orthogonal to our approach and could potentially be combined with FiLM-Ensemble to further improve its performance and uncertainty calibration, while minimally increasing the computational costs.

Original: It appears straight-forward to addthe selection of independent examples for each member during training [as in MIMO, Havasi et al. ,2021], temperature scaling [Guo et al., 2017] or data augmentation strategies [such as, e.g., Raméet al., 2021].  
Revised: It appears straight-forward to add the selection of independent examples for each member during training [as in MIMO, Havasi et al., 2021], temperature scaling [Guo et al., 2017] or data augmentation strategies [such as, e.g., Ramé et al., 2021].  Conclusion

Revised: In this paper we present FiLM-Ensemble, a novel implicit deep ensembling method.

Revised: We achieve high efficiency using a simple yet effective idea – feature-wise linear modulation – which has been shown to be effective in different domains, such as image style transfer, model-agnostic meta-learning, or multi-task learning.

Revised: Our extensive evaluation shows that FiLM-Ensemble outperforms, or is on par with, state-of-the-art ensemble methods in many different experimental settings.

Original: Broader Impact
Revised: Broader Impact

Original: Nowadays, Machine learning has resulted in rapidly growing computational cost and thus energycosts.
Revised: Machine learning has recently witnessed a steep increase in model sizes and associated computational costs, and as a consequence a rapid growth in energy consumption.

Original: For instance, training state-of-the-art language models like GPT-3 would amount to at least1400 MWh or 4.6 million $.
Revised: For instance, training state-of-theart language models like GPT-3 would amount to at least 1400 MWh or 4.6 million $.

Original: Therefore, recently the term Green AI has been introduced, referring to AI research that yields novel results while taking into account the computational cost, encouraging areduction in resources spent.
Revised: Therefore, recently the term Green AI has been introduced, referring to AI research that yields novel results while taking into account the computational cost, encouraging a reduction in resources spent.

Original: To this extent, we believe that the presented method can foster moreefficient ensemble methods and be helpful towards a greener AI.
Revised: To this extent, we believe that the presented method can foster more efficient ensemble methods and be helpful towards a greener AI.

Revised: Acknowledgements We would like to thank Liyuan Zhu for his invaluable contributions to experiments.

Original: Checklist
Revised: H.A.G. received funding from German Federal Ministry of Education and Research (BMBF, Grant “GenomeNet” 031L0199B). B. B. and M. R. were supported by the Bavarian Ministry of Economic Affairs, Regional Development and Energy through the Center for Analytics - DataApplications (ADACenter) within the framework of BAYERN DIGITAL II (20-3410-2-9-8) and the

Original: For all authors...
Revised: German Federal Ministry of Education and Research and the Bavarian State Ministry for Science and the Arts. A Training Details A.1 CIFAR-10 / CIFARPlease refer to Section 2.1.

Revised: A.2 Retinal Glaucoma Detection

Revised: We followed the same training procedure as for CIFAR-10/100, please refer to Section 2.1.

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: Resnetis used as a backbone. Models are trained with 70% of the dataset for 150 epochs and tested on the test set (30% of the dataset). A.3 6mA Identification A 1-dimensional CNN architcture is used, whose hyperparameters such as kernel size and the number of layers are optimized by Li et al.

Original: [Yes]
Revised: The CNN consists of 5 convolutional blocks, where each block contains a 1-dimensional convolution, ReLU activation, batch normalization, and dropout with a rate of 0.5. The convolutional layers have a filter size of 256, kernel size of 10, a stride of 1, and the first convolutional layer have a padding of 5. On top of the last convolutional block, there is a linear layer for predicting the binary labels. Binary cross-entropy is used as a loss.

Original: (b) Did you describe the limitations of your work?
Revised: All models are trained for 20 epochs.

Original: [Yes]
Revised: The initial learning rate of 0.01 is used, as in Li et al. Cosine-annealing is used as a learning rate scheduler.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: We run more experiments using more modern architecture:

Original: [No] Purely methodological contribution that, when added to a learning system, does not alterits potential societal impacts.
Revised: EfficientNet whose proportion of the number of channels vs. the number of layers can vary drastically, compared to Resnets.

Original: If anything, better uncertainty calibration reduces the risk of negative effects caused by undetected prediction errors.
Revised: In this experiment we use two extra calibration metrics: (i) the Brier score Brier et al.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [1950] and (ii) the Static Calibration Error (SCE) Nixon et al.

Original: [Yes] If you are including theoretical results...
Revised: SCE can be considered an extension of

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: ECE but more accurately account for calibration by considering all classes, instead of just the one with the highest confidence.

Original: [N/A] (b) Did you include complete proofs of all theoretical results? [N/A]
Revised: Table 5 shows that FiLM-Ensemble can also be effectively used in conjunction with EfficientNet architecture.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: In addition, other calibration metrics are also in favor of

Original: mental results (either in the supplemental material or as a URL)?
Revised: FilM-Ensemble.

Original: [Yes]
Revised: [Table]

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: B.2 Calibration-Accuracy Trade-off

Original: how they were chosen)?
Revised: As in Section 3.

Original: [Yes]
Revised: [Table]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: we show that one can reach a significantly better calibration (lower ECE) by increasing the gain ρ

Original: [Yes]
Revised: [Table]

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators? [Yes] (b) Did you mention the license of the assets?
Revised: with only a minimal accuracy drop.

Original: [No] license information for the used datasets is public. (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: In this case

Original: [N/A]
Revised: [Table]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: we use Resnet-34 withensemble members on Cifar-100 dataset.

Original: [N/A]
Revised: See Fig.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] If you used crowdsourcing or conducted research with human subjects...
Revised: Also note Fig.

Original: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable? [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: 5 is an extension of Fig.

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
Revised: 3 with various number of ensemble members.