Original: FiLM-Ensemble: Probabilistic Deep Learning via Feature-wise Linear Modulation
Revised: FiLM-Ensemble: Probabilistic Deep Learning via Feature-wise Linear Modulation

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: The ability to estimate epistemic uncertainty is often crucial when deploying machine learning in the real world, but modern methods often produce overconfident,uncalibrated uncertainty predictions.
Revised: The ability to estimate epistemic uncertainty is often crucial when deploying machine learning in the real world, but modern methods often produce overconfident,uncalibrated uncertainty predictions.

Original: A common approach to quantify epistemicuncertainty, usable across a wide class of prediction models, is to train a modelensemble .
Revised: A common approach to quantify epistemicuncertainty, usable across a wide class of prediction models, is to train a modelensemble .

Original: In a naïve implementation, the ensemble approach has high computational cost and high memory demand.
Revised: In a naïve implementation, the ensemble approach has high computational cost and high memory demand.

Original: This challenges in particular moderndeep learning, where even a single deep network is already demanding in termsof compute and memory, and has given rise to a number of attempts to emulatethe model ensemble without actually instantiating separate ensemble members.
Revised: This challenges in particular moderndeep learning, where even a single deep network is already demanding in termsof compute and memory, and has given rise to a number of attempts to emulatethe model ensemble without actually instantiating separate ensemble members.

Original: We introduce FiLM-Ensemble, a deep, implicit ensemble method based on theconcept of Feature-wise Linear Modulation (FiLM).
Revised: We introduce FiLM-Ensemble, a deep, implicit ensemble method based on theconcept of Feature-wise Linear Modulation (FiLM).

Original: That technique was originallydeveloped for multi-task learning, with the aim of decoupling different tasks.
Revised: That technique was originallydeveloped for multi-task learning, with the aim of decoupling different tasks.

Original: Weshow that the idea can be extended to uncertainty quantification: by modulatingthe network activations of a single deep network with FiLM, one obtains a modelensemble with high diversity, and consequently well-calibrated estimates of epistemic uncertainty, with low computational overhead in comparison.
Revised: Weshow that the idea can be extended to uncertainty quantification: by modulatingthe network activations of a single deep network with FiLM, one obtains a modelensemble with high diversity, and consequently well-calibrated estimates of epistemic uncertainty, with low computational overhead in comparison.

Original: Empirically,
Revised: Empirically,

Original: FiLM-Ensemble outperforms other implicit ensemble methods, and it and comesvery close to the upper bound of an explicit ensemble of networks (sometimes evenbeating it), at a fraction of the memory cost.
Revised: FiLM-Ensemble outperforms other implicit ensemble methods, and it comes veryclose to the upper bound of an explicit ensemble of networks (sometimes evenbeating it), at a fraction of the memory cost.

Original:  Introduction
Revised:  Introduction

Original: A key component for reliable and trustworthy machine learning are algorithms that output not onlyaccurate predictions of the target variables, but also well-calibrated estimates of their uncertainty Galand Ghahramani [2016].
Revised: A key component for reliable and trustworthy machine learning are algorithms that output not onlyaccurate predictions of the target variables, but also well-calibrated estimates of their uncertainty Galand Ghahramani [2016].

Original: The overall uncertainty of a predictor is usually decomposed into two parts[Der Kiureghian and Ditlevsen, 2009].
Revised: The overall uncertainty of a predictor is usually decomposed into two parts[Der Kiureghian and Ditlevsen, 2009].

Original: Aleatoric uncertainty is inherent in the data, for instance dueto class overlap or sensor noise.
Revised: Aleatoric uncertainty is inherent in the data, for instance dueto class overlap or sensor noise.

Original: On the contrary, epistemic uncertainty characterises the uncertaintyof the model weights, due to a lack of knowledge about parts of the input space that are insufficientlyrepresented in the training set.
Revised: On the contrary, epistemic uncertainty characterises the uncertaintyof the model weights, due to a lack of knowledge about parts of the input space that are insufficientlyrepresented in the training set.

Original: Uncertainty caused by distributional shifts between the training and test data is sometimes conceived as a third source of uncertainty [Malinin and Gales, 2018], but inpractice often modelled as part of epistemic uncertainty.
Revised: Uncertainty caused by distributional shifts between the training and test data is sometimes conceived as a third source of uncertainty [Malinin and Gales, 2018], but inpractice often modelled as part of epistemic uncertainty.

Original: Measuring epistemic uncertainty for complex models such as deep neural networks is not trivial: bydefinition one cannot derive it from the training data, since it concerns the behavior of the modelin regions of the input space that are not represented in the training set.
Revised: Measuring epistemic uncertainty for complex models such as deep neural networks is not trivial: bydefinition one cannot derive it from the training data, since it concerns the behavior of the modelin regions of the input space that are not represented in the training set.

Original: Different methods havebeen explored [Welling and Teh, 2011, Graves, 2011, Blundell et al., 2015, Lakshminarayanan et al.
Revised: Different methods havebeen explored [Welling and Teh, 2011, Graves, 2011, Blundell et al., 2015, Lakshminarayanan et al.

Original: ,2017a, Huang and Belongie, 2017, Gal and Ghahramani, 2016], however the de facto standard remaindeep ensemble models [Lakshminarayanan et al., 2017a].
Revised: ,2017a, Huang and Belongie, 2017, Gal and Ghahramani, 2016], however the de facto standard remaindeep ensemble models [Lakshminarayanan et al., 2017a].

Original: In its basic form, such a deep ensemble is simply a collection of independently trained networks that can be regarded as Monte Carlo samples from the model space.
Revised: In its basic form, such a deep ensemble is simply a collection of independently trained networks that can be regarded as Monte Carlo samples from the model space.

Original: To obtain ensemble members with reasonably low correlation, one can exploit the stochastic nature of the optimisation procedure, with different (random) weight initialisation anddifferent (random) batches during stochastic gradient descent.
Revised: To obtain ensemble members with reasonably low correlation, one can exploit the stochastic nature of the optimisation procedure, with different (random) weight initialisation anddifferent (random) batches during stochastic gradient descent.

Original: The expectation is that, once trained,the ensemble members will agree for inputs near the training samples, since the loss function favourssimilar outputs at those locations.
Revised: The expectation is that, once trained,the ensemble members will agree for inputs near the training samples, since the loss function favourssimilar outputs at those locations.

Original: Whereas they may disagree in unseen regions of the data space.
Revised: Whereas they may disagree in unseen regions of the data space.

Original: Thus, the spread of their predictions is a measure of epistemic uncertainty.
Revised: Thus, the spread of their predictions is a measure of epistemic uncertainty.

Original: The ensemble idea isconceptually very simple, but nevertheless yields uncertainties that are well calibrated, i.e., they arein line with the actual prediction errors (in expectation).
Revised: The ensemble idea isconceptually very simple, but nevertheless yields uncertainties that are well calibrated, i.e., they arein line with the actual prediction errors (in expectation).

Original: The drawback of deep ensembles is their large computational cost.
Revised: The drawback of deep ensembles is their large computational cost.

Original: Both computation and memoryconsumption grow directly proportionally with the number of ensemble members, during training aswell as during inference.
Revised: Both computation and memoryconsumption grow directly proportionally with the number of ensemble members, during training aswell as during inference.

Original: This makes them impractical in hardware-constrained settings, and leads toa widening gap as models continue to grow in size, while at the same time applications like mobilerobotics, augmented reality and smart sensor networks increase the need for mobile and embeddedcomputing.
Revised: This makes them impractical in hardware-constrained settings, and leads toa widening gap as models continue to grow in size, while at the same time applications like mobilerobotics, augmented reality and smart sensor networks increase the need for mobile and embeddedcomputing.

Original: To improve efficiency, several researchers have explored ways to mimic deep ensembles withoutexplicitly duplicating the underlying network.
Revised: To improve efficiency, several researchers have explored ways to mimic deep ensembles withoutexplicitly duplicating the underlying network.

Original: Possible strategies include the reuse and recombinationof network modules [e.g., Wen et al., 2020, Havasi et al., 2021], injection of noise at inferencetime [e.g., Gal and Ghahramani, 2016], as well as hybrid variants
Revised: Possible strategies include the reuse and recombinationof network modules [e.g., Wen et al., 2020, Havasi et al., 2021], injection of noise at inferencetime [e.g., Gal and Ghahramani, 2016], as well as hybrid variants

Original: [e.g., Durasov et al., 2021].
Revised: [e.g., Durasov et al., 2021].

Original: Empirically, these models do speed up training and/or inference, but still exhibit a significantperformance gap compared to the na ive, explicit ensemble, both w.r.t.
Revised: Empirically, these models do speed up training and/or inference, but still exhibit a significantperformance gap compared to the naïve, explicit ensemble, both w.r.t.

Original: prediction quality and w.r.t.the calibration of the predicted (epistemic) uncertainties.
Revised: prediction quality and w.r.t. thecalibration of the predicted (epistemic) uncertainties.

Original: The lottery ticket hypothesis [Frankle and Carbin, 2018] and other network pruning studies, e.g.,
Revised: The lottery ticket hypothesis [Frankle and Carbin, 2018] and other network pruning studies, e.g.,

Original: Han et al.
Revised: Han et al.

Original: [2015], Lee et al.
Revised: [2015], Lee et al.

Original: [2019], Mallya and Lazebnik [2018], underline that neural networksare heavily over-parameterized.
Revised: [2019], Mallya and Lazebnik [2018], underline that neural networksare heavily over-parameterized.

Original: Their parameters are used inefficiently, and they can be prunedsignificantly without large performance drops.
Revised: Their parameters are used inefficiently, and they can be prunedsignificantly without large performance drops.

Original: In lifelong learning and multi-task learning it isessential to use the network efficiently, in order to limit computational overhead when introducingnew tasks.
Revised: In lifelong learning and multi-task learning it isessential to use the network efficiently, in order to limit computational overhead when introducingnew tasks.

Original: There are recent works that achieve good performance in these tasks by introducingmodulation (respectively, adaptation) strategies.
Revised: There are recent works that achieve good performance in these tasks by introducingmodulation (respectively, adaptation) strategies.

Original: In particular, Li et al.
Revised: In particular, Li et al.

Original: [2018], Takeda et al.
Revised: [2018], Takeda et al.

Original: [2021]propose an efficient lifelong learning / domain adaptation method for multi-task learning, utilizingsingle feature-wise linear modulation (FiLM).
Revised: [2021]propose an efficient lifelong learning / domain adaptation method for multi-task learning, utilizingsingle feature-wise linear modulation (FiLM).

Original: Inspired by that line of work, we propose a new, efficient ensemble method, FiLM-Ensemble.
Revised: Inspired by that line of work, we propose a new, efficient ensemble method, FiLM-Ensemble.

Original: Ourmethod adapts feature-wise linear modulation as an alternative way to construct an ensemble for(epistemic) uncertainty estimation.
Revised: Ourmethod adapts feature-wise linear modulation as an alternative way to construct an ensemble for(epistemic) uncertainty estimation.

Original: FiLM-Ensemble greatly reduces the computational overheadcompared to the naïve ensemble approach, while performing almost on par with it, sometimes evenbetter.
Revised: FiLM-Ensemble greatly reduces the computational overheadcompared to the naïve ensemble approach, while performing almost on par with it, sometimes evenbetter.

Original: In a nutshell, FiLM-Ensemble can be described as an implicit model ensemble in which eachindividual member is defined via its own set of linear modulation parameters for the activations,whereas all other network parameters are shared among ensemble members – and therefore only needto be stored and trained once.
Revised: In a nutshell, FiLM-Ensemble can be described as an implicit model ensemble in which eachindividual member is defined via its own set of linear modulation parameters for the activations,whereas all other network parameters are shared among ensemble members – and therefore only needto be stored and trained once.

Original: Thanks to this design, our method requires only a very small numberof additional parameters on top of the base network, e.g., converting a single ResNet-18 model to anensemble of 16 models increases the parameter count by 1.3%, compared to an increase by 1500%when setting up a naïve ensemble, see Table 1.
Revised: Thanks to this design, our method requires only a very small numberof additional parameters on top of the base network, e.g., converting a single ResNet-18 model to anensemble of 16 models increases the parameter count by 1.3%, compared to an increase by 1500%when setting up a naïve ensemble, see Table 1.

Original: We further show that FiLM-Ensemble results inmore diverse ensemble members compared to other efficient ensemble methods.
Revised: We further show that FiLM-Ensemble results inmore diverse ensemble members compared to other efficient ensemble methods.

Original: For instance, onthe Cifar10 benchmark it achieves diversity scores > 6 .
Revised: For instance, onthe Cifar10 benchmark it achieves diversity scores > 6 .

Original: 9% and up to 9 .
Revised: 9% and up to 9 .

Original: 2% (depending on ensemblesize), against 6 . 8% for a naive ensemble, see Fig.
Revised: 2% (depending on ensemblesize), against 6 . 8% for a naive ensemble, see Fig.

Original: Our contributions can be summarized as follows.
Revised: Our contributions can be summarized as follows.

Original: • We propose FiLM-Ensemble, a novel parameter- and time-efficient deep ensemble method.
Revised: • We propose FiLM-Ensemble, a novel parameter- and time-efficient deep ensemble method.

Original: • FiLM-Ensemble is designed in such a way that it can be readily combined with many populardeep learning models, by simply replacing batch normalisation (batchnorm) layers withconditional batchnorm layers.
Revised: • FiLM-Ensemble is designed in such a way that it can be readily combined with many popular deep learning models, by simply replacing batch normalisation (batchnorm) layers with conditional batchnorm layers.

Original: • We show that FiLM-Ensemble provides an excellent trade-off between accuracy and calibra
Revised: • We show that FiLM-Ensemble provides an excellent trade-off between accuracy and calibra-

Original: -tion performance, improving over existing ensemble methods.
Revised: tion performance, improving over existing ensemble methods.

Original:  FiLM-Ensemble
Revised:  FiLM-Ensemble

Original: In [Perez et al., 2018] the linear feature modulation was achieved by varying the affine parameters ofthe batch normalization layers γ and β .
Revised: In [Perez et al., 2018] the linear feature modulation was achieved by varying the affine parameters ofthe batch normalization layers γ and β .

Original: For each batchnorm layer n in the network, the parameters are predicted according to some conditioning input z (for instance, different prediction tasks):
Revised: For each batchnorm layer n in the network, the parameters are predicted according to some conditioning input z (for instance, different prediction tasks):

Original: [Equation]
Revised: [Equation]

Original: The size of γ n and β n is R D n where D n is the feature dimension at layer n .
Revised: The size of γ n and β n is R D n where D n is the feature dimension at layer n .

Original: These parameters arethen used to linearly modulate the activations at the n -th layer F n :
Revised: These parameters arethen used to linearly modulate the activations at the n -th layer F n :

Original: [Equation]
Revised: [Equation]

Original: with ◦ being the Hadamard (element-wise) product taken w.r.t.
Revised: with ◦ being the Hadamard (element-wise) product taken w.r.t.

Original: the feature dimension.
Revised: the feature dimension.

Original: In our scenario we aim to use the affine parameters to instantiate different ensemble members, i.e.,the conditioning variable z is simply an index m ∈ { 1 , ..., M } that identifies the ensemble member.
Revised: In our scenario we aim to use the affine parameters to instantiate different ensemble members, i.e.,the conditioning variable z is simply an index m ∈ { 1 , ..., M } that identifies the ensemble member.

Original: The functions g n ( z ) and h n ( z ) degenerate to look-up tables, so we can dispose of them and simplylearn M different sets of affine parameters for every batchnorm layer F n :
Revised: The functions g n ( z ) and h n ( z ) degenerate to look-up tables, so we can dispose of them and simplylearn M different sets of affine parameters for every batchnorm layer F n :

Original: [Equation]
Revised: [Equation]

Original: To achieve ensemble members with diverse parameters, we resort to Xavier initialization, i.e., all γ mnand β mn are sampled from a uniform distribution bounded between:
Revised: To achieve ensemble members with diverse parameters, we resort to Xavier initialization, i.e., all γ mnand β mn are sampled from a uniform distribution bounded between:

Original: [Equation]
Revised: [Equation]

Original: where D n is the number of features (channels) in the n -th layer, and ρ is an initialization gain factor.
Revised: where D n is the number of features (channels) in the n -th layer, and ρ is an initialization gain factor.

Original: In our setting ρ is a tunable hyperparameter that allows one to control the trade-off between predictiveaccuracy and calibration of the model, see Section 3.6.
Revised: In our setting ρ is a tunable hyperparameter that allows one to control the trade-off between predictiveaccuracy and calibration of the model, see Section 3.6.

Original: In general, increasing ρ leads to more diverseensemble members and thus favours calibration.
Revised: In general, increasing ρ leads to more diverseensemble members and thus favours calibration.

Original: Note that as ρ → 0 , all ensemble members startfrom similar initial values for γ mn and β mn and the FiLM-Ensemble gradually collapses to a singlemodel.
Revised: Note that as ρ → 0 , all ensemble members startfrom similar initial values for γ mn and β mn and the FiLM-Ensemble gradually collapses to a singlemodel.

Original: During training, we feed each input sample x to every ensemble member and obtain predictionsy m = f θ , γ m , β m ( x ) , which depend bot on the member-specific parameters ( β m , γ m ) and on theshared parameters θ .
Revised: During training, we feed each input sample x to every ensemble member and obtain predictionsy m = f θ , γ m , β m ( x ) , which depend both on the member-specific parameters ( β m , γ m ) and on theshared parameters θ .

Original: All those parameters are optimized together to minimise the chosen loss function.
Revised: All those parameters are optimized together to minimise the chosen loss function.

Original: Here we focus on classification and use a standard cross-entropy (CE) loss.
Revised: Here we focus on classification and use a standard cross-entropy (CE) loss.

Original: At inference time the final prediction ˆ y is obtained by averaging the M predictions of the ensemble:
Revised: At inference time the final prediction ˆ y is obtained by averaging the M predictions of the ensemble:

Original: [Equation]
Revised: [Equation]

Original: 2.1 Implementation Details
Revised: 2.1 Implementation Details

Original: We implemented 1-dimensional and 2-dimensional FiLM-Ensemble layers, to be used in combinationwith 1-dimensional and 2-dimensional convolution operations, respectively.
Revised: We implemented 1-dimensional and 2-dimensional FiLM-Ensemble layers, to be used in combinationwith 1-dimensional and 2-dimensional convolution operations, respectively.

Original: The forward passesthrough different ensemble members can easily be parallelized by replicating both the input tensor andthe FiLM parameters γ , β along the batch dimension and applying the affine transformation Eq.
Revised: The forward passesthrough different ensemble members can easily be parallelized by replicating both the input tensor andthe FiLM parameters γ , β along the batch dimension and applying the affine transformation Eq.

Original: 3 (thesame holds for the BatchEnsemble method).
Revised: 3 (thesame holds for the BatchEnsemble method).

Original: In thid way all ensemble members run simultaneouslyon a single device, thus optimally utilizing modern tensor computing hardware without having toload several instances of the classification network into memory.
Revised: In this way all ensemble members run simultaneouslyon a single device, thus optimally utilizing modern tensor computing hardware without having to loadseveral instances of the classification network into memory.

Original: We have implemented FiLMEnsemble in PyTorch and  release the source code, including implementations of the baselines usedin the experiments, together with trained model checkpoints.
Revised: We have implemented FiLM-Ensemblein PyTorch and will release the source code, including implementations of the baselines used in theexperiments, together with trained model checkpoints.

Original: The source code is submitted assupplementary material.
Revised: The source code is submitted as supplementarymaterial.

Original: For all experiments, we optimize the model parameters with standard stochastic gradient descent,with momentum µ = 0 .
Revised: For all experiments, we optimize the model parameters with standard stochastic gradient descent,with momentum µ = 0 .

Original: 9 and weight decay λ = 0 .
Revised: 9 and weight decay λ = 0 .

Original: 0005 for regularization.
Revised: 0005 for regularization.

Original: We train for 200epochs with batch size 128.
Revised: We train for 200epochs with batch size 128.

Original: The learning rate is initially set to 0 .
Revised: The learning rate is initially set to 0 .

Original: 1 and decays according to a cosineannealing schedule [Loshchilov and Hutter, 2017].
Revised: 1 and decays according to a cosineannealing schedule [Loshchilov and Hutter, 2017].

Original: The initialisation gain is set to ρ = 2 for allexperiments, except for genome sequences (see Table 3), where ρ ∈ { 2 , 8 , 16 , 32 } .
Revised: The initialisation gain is set to ρ = 2 for allexperiments, except for genome sequences (see Table 3), where ρ ∈ { 2 , 8 , 16 , 32 } .

Original: Please refer tothe supplementary material for additional, dataset-specific details.
Revised: Please refer tothe supplementary material for additional, dataset-specific details.

Original:  Experiments
Revised:  Experiments

Original: In the following, we first empirically examine the ability of FiLM-Ensemble to learn independentensemble members, and compare it to deep ensembles.
Revised: In the following, we first empirically examine the ability of FiLM-Ensemble to learn independentensemble members, and compare it to deep ensembles.

Original: Then we analyze the predictions accuracy and uncertainty calibration of FiLM-Ensemble, as well as its computational cost and memory consumption.
Revised: Then we analyze the predictions accuracy and uncertainty calibration of FiLM-Ensemble, as well as its computational cost and memory consumption.

Original: We compare its performance against several baseline methods and state-of-the-art alternatives on avariety of different datasets, and using different backbone architectures.
Revised: We compare its performance against several baseline methods and state-of-the-art alternatives on avariety of different datasets, and using different backbone architectures.

Original: All experimental results areaveraged over three runs with different random seeds.
Revised: All experimental results areaveraged over three runs with different random seeds.

Original: We evaluate our proposed method on a diverse set of classification tasks, including popular imageclassification benchmarks, image-based medical diagnosis, and genome sequence analysis.
Revised: We evaluate our proposed method on a diverse set of classification tasks, including popular imageclassification benchmarks, image-based medical diagnosis, and genome sequence analysis.

Original: CIFAR10 and CIFAR-100 [Krizhevsky, 2009] are widely used testbeds for image classification, and deeplearning in general.
Revised: CIFAR10 and CIFAR-100 [Krizhevsky, 2009] are widely used testbeds for image classification, and deeplearning in general.

Original: They consist of clean images of objects from 10, respectively 100, differentsemantic classes.
Revised: They consist of clean images of objects from 10, respectively 100, differentsemantic classes.

Original: Each of the two datasets contains 60,000 images, of which 10,000 are reservedfor testing.
Revised: Each of the two datasets contains 60,000 images, of which 10,000 are reservedfor testing.

Original: The semantic classes are uniformly distributed in the datasets and stratified across thetrain/test spit.
Revised: The semantic classes are uniformly distributed in the datasets and stratified across thetrain/test spit.

Original: Retina Glaucoma Detection [Diaz-Pinto et al., 2019] is a real-world clinical datasetthat includes microscopic retina images from 956 patients with the neuropathic disease Glaucoma,and from 1401 subjects with normal (healthy) retinas.
Revised: Retina Glaucoma Detection [Diaz-Pinto et al., 2019] is a real-world clinical datasetthat includes microscopic retina images from 956 patients with the neuropathic disease Glaucoma,and from 1401 subjects with normal (healthy) retinas.

Original: Each input sample is a single RGB image;we resize all images to 128 × 128 .
Revised: Each input sample is a single RGB image;we resize all images to 128 × 128 .

Original: REFUGE 

Revised: REFUGE

Original: [Orlando et al., 2020] was a challenge at the
Revised: [Orlando et al., 2020] was a challenge at the

Original: MICCAI-2020 conference. The task consists again in detecting retinal Glaucoma, in a dataset  of 800microscopic retina images with size 1411 × 1411 pixels, collected from different clinics.
Revised: MICCAI-2020 conference, again aimed at retinal Glaucoma. The dataset consists of 800 microscopicretina images with size 1411 × 1411 pixels, collected from different clinics.

Original: We use thisdataset as a test set in conjunction with the model trained on the previous dataset [Diaz-Pinto et al. ,2019], to evaluate performance under distributional shift (Table 4). 6mA Identification [Li et al. ,2021], is a 1-dimensional sequential genome dataset.
Revised: We use this dataset inconjunction with the models trained on the previous dataset [Diaz-Pinto et al., 2019], to evaluate theability to detect out-of-distribution samples with the help of the predicted uncertainties (Table 4).6mA Identification [Li et al., 2021], is a 1-dimensional sequential genome dataset.

Original: It consists of DNA sequences of rice plants,along with binary labels that indicate whether the sequence is a N6-methyladenine (6mA) site.
Revised: It consists of DNA sequences of rice plants, along with binary labels that indicate whether the sequence is a N6methyladenine (6mA) site.

Original: 6mAis an important DNA modification associated with several biological processes, such as regulatinggene transcription, DNA replication and DNA repair [Campbell andKleckner, 1990, Cheng et al.
Revised: 6mA is an important DNA modification associated with several biologicalprocesses, such as regulating gene transcription, DNA replication and DNA repair [Campbell and

Original: ,2016, Pukkila et al., 1983].
Revised: Kleckner, 1990, Cheng et al., 2016, Pukkila et al., 1983].

Original: Each sequence consists of 41 nucleotides.
Revised: Each sequence consists of 41 nucleotides.

Original: As there are 4 different typesof nucleotides, each one-hot encoded sequence is represented by a 41 × 4 matrix.
Revised: As there are 4 different types of nucleotides, each one-hot encoded sequence is represented by a41 × 4 matrix.

Original: In total there are269,500 training samples and 38,500 test samples.
Revised: In total there are 269,500 training samples and 38,500 test samples.

Original: We compare FiLM-Ensemble against (i) a single model without any ensembling, as the elementarybaseline, (ii) a naïve deep ensemble
Revised: We compare FiLM-Ensemble against (i) a single model without any ensembling, as the elementarybaseline, (ii) a naïve deep ensemble

Original: [Lakshminarayanan et al., 2017b], and (iii) MC-Dropout [Galand Ghahramani, 2016].
Revised: [Lakshminarayanan et al., 2017b], and (iii) MC-Dropout [Galand Ghahramani, 2016].

Original: Furthermore, we compare with other state-of-the-art methods, namely: (iv)
Revised: Furthermore, we compare with other state-of-the-art methods, namely: (iv)

Original: Masksemble [Durasov et al., 2021] which can be seen as an extension of MC-Dropout, (v) MIMO (“multi-input multi-output")
Revised: Masksemble [Durasov et al., 2021] which can be seen as an extension of MC-Dropout, (v) MIMO (“multi-input multi-output")

Original: [Havasi et al., 2021] which defines a multi-head architecture where eachhead acts as one ensemble member, and
Revised: [Havasi et al., 2021] which defines a multi-head architecture where eachhead acts as one ensemble member, and

Original: (vi) BatchEnsemble [Wen et al., 2020] which creates anefficient ensemble by expanding the layer weights using low rank matrices.
Revised: (vi) BatchEnsemble [Wen et al., 2020] which creates anefficient ensemble by expanding the layer weights using low rank matrices.

Original: Arguably, this layer-wisemodification of an underlying, common representation is the approach that comes closest to our work.
Revised: Arguably, this layer-wisemodification of an underlying, common representation is the approach that comes closest to our work.

Original: 3.1 Diversity Analysis
Revised: 3.1 Diversity Analysis

Original: Diversity is the key to constructing powerful ensembles: nothing is to be gained from highly correlatedensemble members that return similar outputs for (almost) any input [Zhang and Ma, 2012].
Revised: Diversity is the key to constructing powerful ensembles: nothing is to be gained from highly correlatedensemble members that return similar outputs for (almost) any input [Zhang and Ma, 2012].

Original: In orderto analyze the diversity (respectively, the degree of independence) among members, we compute twodistance metrics between the members’ predictive distributions.
Revised: In orderto analyze the diversity (respectively, the degree of independence) among members, we compute twodistance metrics between the members’ predictive distributions.

Original: Let f i and f j denote two differentensemble members for a classification task.
Revised: Let f i and f j denote two differentensemble members for a classification task.

Original: We first measure the disagreement score D , defined asthe fraction of all s test samples for which the two members return different answers, averaged overall possible pairs of members:
Revised: We first measure the disagreement score D , defined asthe fraction of all s test samples for which the two members return different answers, averaged overall possible pairs of members:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: with [ · ] being the Iverson bracket.
Revised: with [ · ] being the Iverson bracket.

Original: Second, we also measure the Kullback–Leibler (KL) divergencebetween the two predictive distributions p ( f i ) and p ( f j ) , again averaged over all pairs:
Revised: Second, we also measure the Kullback–Leibler (KL) divergencebetween the two predictive distributions p ( f i ) and p ( f j ) , again averaged over all pairs:

Original: [Equation]
Revised: [Equation]

Original: For both metrics, higher numbers correspond to higher diversity.
Revised: For both metrics, higher numbers correspond to higher diversity.

Original: See Fig.
Revised: See Fig.

Original: We observe that theaverage diversity increases with the number of ensemble members.
Revised: We observe that theaverage diversity increases with the number of ensemble members.

Original: In both metrics, FiLM-Ensembleachieves higher scores than the naïve, explicit deep ensemble.
Revised: In both metrics, FiLM-Ensembleachieves higher scores than the naïve, explicit deep ensemble.

Original: Meaning that the predictions of FiLMEnsemble are less correlated than those of an equivalent number of networks trained independentlywith different random seeds.
Revised: Meaning that the predictions of FiLMEnsemble are less correlated than those of an equivalent number of networks trained independentlywith different random seeds.

Original: [Table]
Revised: [Table]

Original: 3.2 Computational Cost
Revised: 3.2 Computational Cost

Revised: [Table]

Original: We go on to measure the efficiency of our proposed method, both in terms of parameter count andin terms of time needed for a forward pass.
Revised: We go on to measure the efficiency of our proposed method, both in terms of parameter count and interms of resources needed for a forward pass.

Original: Table 1 (left section) reports the numbers of additionalnumbers of parameters compared to a single network (i
Revised: Table 1 (left section) reports the numbers of additionalnumbers of parameters compared to a single network (i

Original: [Table]
Revised: [Table]

Original: e., without uncertainty calibration).
Revised: e., without uncertainty calibration).

Original: Due totheir design, MC-Dropout and Masksemble do not require any additional parameters.
Revised: Due totheir design, MC-Dropout and Masksemble do not require any additional parameters.

Original: Among theremaining methods, FiLM-Ensemble has the lowest (VGG-11) or second-lowest (Resnet18) overheadin terms of parameter count, while performing significantly better, as we will see below.
Revised: Among theremaining methods, FiLM-Ensemble has the lowest (VGG-11) or second-lowest (Resnet18) overheadin terms of parameter count, while performing significantly better, as we will see below.

Original: In terms ofinference time (Tab. 1, right section), FiLM-Ensemble turns out to be very competitive.
Revised: In terms ofinference complexity (Tab. 1, center and right), FiLM-Ensemble turns out to be very competitive.

Original: Only MIMOwith Resnet18 backbone is significantly faster, however this comes at a considerable price in terms of accuracy and calibration, see below.
Revised: Only MIMO with a ResNet-18 backbone is significantly faster and lighter, however this comes at aconsiderable price in terms of accuracy and calibration, see below.

Original: 3.3 CIFAR10 / CIFAR100
Revised: 3.3 CIFAR10 / CIFAR100

Original: We start with widely used standard benchmarks in computer vision, CIFAR10 and CIFAR100.
Revised: We start with widely used standard benchmarks in computer vision, CIFAR10 and CIFAR100.

Original: Asperformance metrics, we plot the test set accuracy and the expected calibration error (ECE) againstthe ensemble size M in Fig.
Revised: Asperformance metrics, we plot the test set accuracy and the expected calibration error (ECE) againstthe ensemble size M in Fig.

Original: 2, for all compared methods.
Revised: 2, for all compared methods.

Original: In terms of accuracy (left subfigure),
Revised: In terms of accuracy (left subfigure),

Original: FiLM-Ensemble is outperformed only by the explicit deep ensemble, across all tested values of
Revised: FiLM-Ensemble is outperformed only by the explicit deep ensemble, across all tested values of

Original: M ∈ { 2 , 4 , 8 , 16 } .
Revised: M ∈ { 2 , 4 , 8 , 16 } .

Original: These two surpass all other methods by a clear margin.
Revised: These two surpass all other methods by a clear margin.

Original: Surprisingly, we observethat BatchEnsemble generally exhibits a negative correlation between ensemble size and test setaccuracy, with M = 4 performing best among all settings.
Revised: Surprisingly, we observethat BatchEnsemble generally exhibits a negative correlation between ensemble size and test setaccuracy, with M = 4 performing best among all settings.

Original: MIMO shows very poor performancein this experimental setting in terms of accuracy (and also in ECE).
Revised: MIMO shows very poor performancein this experimental setting in terms of accuracy (and also in ECE).

Original: We speculate that its sharedbackbone probably has a tendency to fragment into largely independent ensemble members of lowchannel depth, which lack the necessary capacity when using comparatively small networks like
Revised: We speculate that its sharedbackbone probably has a tendency to fragment into largely independent ensemble members of lowchannel depth, which lack the necessary capacity when using comparatively small networks like

Original: VGG11 or Resnet18.
Revised: VGG11 or Resnet18.

Original: With regard to ECE (right subfigure), we see all methods improving withgrowing ensemble size M .
Revised: With regard to ECE (right subfigure), we see all methods improving withgrowing ensemble size M .

Original: FiLM-Ensemble achieves better calibration then the widely used deep
Revised: FiLM-Ensemble achieves better calibration then the widely used deep

Original: [Table]
Revised: [Table]

Original: ensemble and MC-dropout methods, with significant margins at large ensemble sizes.
Revised: ensemble and MC-dropout methods, with significant margins at large ensemble sizes.

Original: Masksembleand BatchEnsemble achieve very good calibration for sizes M ∈ [4 . . .
Revised: Masksembleand BatchEnsemble achieve very good calibration for sizes M ∈ [4 . . .

Original: [Table]
Revised: [Table]

Original: but at the cost ofsignificant drops in classification performance.
Revised: but at the cost ofsignificant drops in classification performance.

Original: An attractive feature of FiLM-Ensemble is that itoffers a simple mechanism for trading off accuracy against calibration, by tuning the gain factor ρ .
Revised: An attractive feature of FiLM-Ensemble is that itoffers a simple mechanism for trading off accuracy against calibration, by tuning the gain factor ρ .

Original: See Section 3.6.
Revised: See Section 3.6.

Original: In Tab. 2 we quantitatively compare all tested methods on the CIFAR-100 dataset, both with astandard Resnet-18 backbone and with a larger Resnet-34.
Revised: In Tab. 2 we quantitatively compare all tested methods on the CIFAR-100 dataset, both with astandard Resnet-18 backbone and with a larger Resnet-34.

Original: We observe that with both architectures,and in terms of both predictive accuracy and calibration, FiLM-Ensemble always either ranks first, orit ranks second behind the inefficient, explicit deep ensemble.
Revised: We observe that with both architectures,and in terms of both predictive accuracy and calibration, FiLM-Ensemble always either ranks first, orit ranks second behind the inefficient, explicit deep ensemble.

Original: Overall, the experiments with bothvariants of CIFAR and all three model architectures confirm that our method presents an excellenttrade-off between predictive accuracy, uncertainty calibration and computational efficiency.
Revised: Overall, the experiments with bothvariants of CIFAR and all three model architectures confirm that our method presents an excellenttrade-off between predictive accuracy, uncertainty calibration and computational efficiency.

Original: 3.4 Retinal Glaucoma Detection
Revised: 3.4 Retinal Glaucoma Detection

Original: Glaucoma is currently the leading reason of irreversible blindness in the world.
Revised: Glaucoma is currently the leading reason of irreversible blindness in the world.

Original: Detection ofglaucomatous structural damages and changes is one of the most challenging tasks in the fieldof ophthalmology.
Revised: Detection ofglaucomatous structural damages and changes is a challenging task in the field of ophthalmology.

Original: We evaluateand test, our proposed FiLM-Ensemble and other state-of-the-artalgorithms for estimating uncertainty for diagnosis of Glaucoma disease. We show a comparisonof thetest performance by different uncertainty estimation algorithms in Table 3 and report theperformance in terms of accuracy and ECE.
Revised: We evaluate our proposed FiLM-Ensemble, as well as the baselines described above, on the ask ofdiagnosing Glaucoma and quantifying the uncertainty associated with the prediction, see Table 3.

Original: Based on Table 3, the best performance achieved by FiLM-Ensemble when our network was composed of 16 members.
Revised: The proposed FiLM-Ensemble achieves the best classification result across all ensemble sizes, andalso the best overall result, with M = 16 .

Revised: Whereas there is no clear trend with respect to uncertaintycalibration.

Original: 3.5 Out-of-Distribution Detection
Revised: 3.5 Out-of-Distribution Detection

Original: Domain shift often occurs in medical datasets; thus, detecting out-of-distribution (OOD) samplesis an important task in clinical diagnosis.
Revised: Domain shift often occurs in medical datasets; thus, detecting out-of-distribution (OOD) samplesis an important task in clinical diagnosis.

Original: Model uncertainty can be used for OOD detection.
Revised: Model uncertainty can be used for OOD detection.

Original: Weperform OOD detection on REFUGE dataset [Diaz-Pinto et al., 2019] using models trained on Retinal
Revised: Forour experiment, we regard the retinal Glaucoma dataset [Diaz-Pinto et al., 2019] as the within-

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  [Orlando et al., 2020] (in-distribution) and compare AUROC (Area Under the
Revised: distribution samples and perform OOD detection with the REFUGE dataset [Orlando et al., 2020].

Original:  Receiver Operating Characteristic) scores.
Revised: Asperformance metric, we report AUROC (Area Under the Receiver Operating Characteristic) scores,

Original: As shown in Table4, FiLM-Ensemble can detect OOD testsamples much more accurately compared to other methods that are based on prediction uncertaintywhich suggests that FiLM-Ensemble has better calibration than competitors.
Revised: Table FiLM-Ensemble can detect OOD test samples with significantly higher accuracy than otherefficient ensemble methods, standard state-of-the-art OOD detection methods [Table] and even the explicitdeep ensemble.

Revised: This suggests that for challenging test samples, which are not adequately representedin the training data, the uncertainties estimated with FiLM-Ensemble are better calibrated.

Original: 3.6 6mA Identification
Revised: 3.6 6mA Identification

Original: Using the 6mA dataset, we showed that FiLM-Ensemble can  be readily combined withthe existingmodelsthat were proposed for 1-dimensional sequential genome data.
Revised: With the 6mA dataset we show that FiLM-Ensemble can also be readily combined with existingmodels for 1-dimensional sequential genome data.

Original: We used the 1D-CNNarchitecture whose hyperparameters are already  optimizedby Li et al. [2021] for the 6mA dataset.
Revised: We use a 1D-CNN architecture whose hyperparameters have already been optimized for the dataset [Li et al., 2021].

Original: FiLM-Ensemble improves the accuracy and  calibration of the proposed model.
Revised: FiLM-Ensemble improvesthe accuracy and the calibration of that model, see Fig.3.

Original: As shown in Fig.3,our method performs on par with  deep ensemble andachieves betterresults than  the optimized singlemodel.
Revised: Our method performs on par with theexplicit deep ensemble and better than a single instance of the model tuned for the specific task.

Original: Besides that, at the expense of a minor accuracy drop, the calibration (ECE) can be improvedsignificantly by increasing theinitialization gain factor.
Revised: Moreimportantly, one can reach a significantly better calibration (lower ECE) by increasing the gain ρ ,with only a minimal accuracy drop by < 0 . 25 percent points.

Original:  Related Work
Revised:  Related Work

Original: 4.1 Epistemic Uncertainty Quantification
Revised: 4.1 Epistemic Uncertainty Quantification

Original: A large corpus of related work addresses the estimation of epistemic (model) uncertainty in neuralnetworks.
Revised: A large corpus of related work addresses the estimation of epistemic (model) uncertainty in neuralnetworks.

Original: At the heart of such modeling is often the concept of marginalization instead of optimization , i.e., integrating out a (possibly uncountably infinite) set of models weighted by their posteriorprobability, instead of committing to a point estimate of that distribution.
Revised: At the heart of such modeling is often the concept of marginalization instead of optimization , i.e., integrating out a (possibly uncountably infinite) set of models weighted by their posteriorprobability, instead of committing to a point estimate of that distribution.

Original: A multitude of methodshave been proposed to implement approximate Bayesian inference w.r.t.
Revised: A multitude of methodshave been proposed to implement approximate Bayesian inference w.r.t.

Original: the model weights, given thetraining data and an appropriate prior, a process that is not analytically tractable in general [Kendalland Gal, 2017].
Revised: the model weights, given thetraining data and an appropriate prior, a process that is not analytically tractable in general [Kendalland Gal, 2017].

Original: For instance, methods based on variational inference [Graves, 2011, Ranganath et al., 2014, Blundellet al., 2015] seek to learn an approximate posterior distribution which is a member of a simpler familyof variational distributions.
Revised: For instance, methods based on variational inference [Graves, 2011, Ranganath et al., 2014, Blundellet al., 2015] seek to learn an approximate posterior distribution which is a member of a simpler familyof variational distributions.

Original: This variational distribution can often be learned using backpropagation[Blundell et al., 2015] and can then be sampled from or sometimes even used for exact inference.
Revised: This variational distribution can often be learned using backpropagation[Blundell et al., 2015] and can then be sampled from or sometimes even used for exact inference.

Original: Markov chain Monte Carlo sampling (MCMC) approaches [Neal, 1996, Welling and Teh, 2011, Chenet al., 2014] construct a Markov chain which has the exact posterior distribution as its stationarydistribution, that can then be employed for sampling.
Revised: Markov chain Monte Carlo sampling (MCMC) approaches [Neal, 1996, Welling and Teh, 2011, Chenet al., 2014] construct a Markov chain which has the exact posterior distribution as its stationarydistribution, that can then be employed for sampling.

Original: However, in practise, these approaches oftenfail to sufficiently explore high-dimensional, multi-modal loss landscapes as they are common indeep learning [Gustafsson et al., 2020].
Revised: However, in practise, these approaches oftenfail to sufficiently explore high-dimensional, multi-modal loss landscapes as they are common indeep learning [Gustafsson et al., 2020].

Original: 4.2 Ensembles and Sub-networks
Revised: 4.2 Ensembles and Sub-networks

Original: In a method referred to as deep ensembles [Lakshminarayanan et al., 2017a], a set of M neuralnetwork models are randomly and independently initialized, and are subjected to stochastic mini-batchsampling during SGD training.
Revised: In a method referred to as deep ensembles [Lakshminarayanan et al., 2017a], a set of M neuralnetwork models are randomly and independently initialized, and are subjected to stochastic mini-batchsampling during SGD training.

Original: The models generally converge to different minima in the parameterspace, and can be considered samples from an approximate posterior [Wilson and Izmailov, 2020,
Revised: The models generally converge to different minima in the parameterspace, and can be considered samples from an approximate posterior [Wilson and Izmailov, 2020,

Original: Gustafsson et al., 2020, Izmailov et al., 2021].
Revised: Gustafsson et al., 2020, Izmailov et al., 2021].

Original: Deep ensembles often achieve the best calibration andpredictive accuracy [Ovadia et al., 2019, Gustafsson et al., 2020, Ashukha et al., 2020], but sufferfrom high computational complexity as it requires training, storing, and running inference on severalnetwork models.
Revised: Deep ensembles often achieve the best calibration andpredictive accuracy [Ovadia et al., 2019, Gustafsson et al., 2020, Ashukha et al., 2020], but sufferfrom high computational complexity as it requires training, storing, and running inference on severalnetwork models.

Original: Many alternative methods have recently been proposed in an attempt of reducing either the computational cost or the storage cost of deep ensembles.
Revised: Many alternative methods have recently been proposed in an attempt of reducing either the computational cost or the storage cost of deep ensembles.

Original: Monte Carlo (MC) Dropout
Revised: Monte Carlo (MC) Dropout

Original: [Gal and Ghahramani,2016] runs multiple forward passes on the dropout layers in order to obtain multiple predictions andobtain an uncertainty estimate.
Revised: [Gal and Ghahramani,2016] runs multiple forward passes on the dropout layers in order to obtain multiple predictions andobtain an uncertainty estimate.

Original: Although, the method requires less computations compared to deepensembles, it also leads to less accurate uncertainty estimates.
Revised: Although, the method requires less computations compared to deepensembles, it also leads to less accurate uncertainty estimates.

Original: Snapshot ensembles [Huang et al.
Revised: Snapshot ensembles [Huang et al.

Original: ,2017] uses a cyclic learning rate scheduling in order to find multiple local minima, they then storemultiple copies of the network to create the ensemble.
Revised: ,2017] uses a cyclic learning rate scheduling in order to find multiple local minima, they then storemultiple copies of the network to create the ensemble.

Original: While this approach reduces the computationcost during training it does not alleviate the storage cost.
Revised: While this approach reduces the computationcost during training it does not alleviate the storage cost.

Original: BatchEnsemble [Wen et al., 2020] propose to use multiple low rank matrices, which can be storedefficiently, in order to modulate the parameters of a neural network and thus mimic an ensembleof network models.
Revised: BatchEnsemble [Wen et al., 2020] propose to use multiple low rank matrices, which can be storedefficiently, in order to modulate the parameters of a neural network and thus mimic an ensembleof network models.

Original: Masksembles [Durasov et al., 2021] aims to improve the performance of MC
Revised: Masksembles [Durasov et al., 2021] aims to improve the performance of MC

Original: Dropout by carefully selecting which masks, used to drop certain features, leading to a betteruncertainty quantification.
Revised: Dropout by carefully selecting which masks, used to drop certain features, leading to a betteruncertainty quantification.

Original: Another approach which uses multiple sub-networks is proposed in Havasiet al.
Revised: Another approach which uses multiple sub-networks is proposed in Havasiet al.

Original: In this case additional, independent, layers are added at the beginning and at the endof the network in order to obtain multiple prediction with an individual network model.
Revised: In this case additional, independent, layers are added at the beginning and at the endof the network in order to obtain multiple prediction with an individual network model.

Original: Althoughsuch method seems to reduce the computational resource required at training and inference time,the gain is limited to larger backbones, such as use Wide-ResNet (e.g., ResNet28-10), whereas forstandard widely used architectures like VGG (e.g., -16), or ResNet (e.g., -34), the approach is notvery effective as shown in our experiments.
Revised: Althoughsuch method seems to reduce the computational resource required at training and inference time,the gain is limited to larger backbones, such as use Wide-ResNet (e.g., ResNet28-10), whereas forstandard widely used architectures like VGG (e.g., -16), or ResNet (e.g., -34), the approach is notvery effective as shown in our experiments.

Original: Although many methods that aim at reducing the computational cost of deep ensembles have beenproposed, none of the methods appear to unquestionably outperform the others.
Revised: Although many methods that aim at reducing the computational cost of deep ensembles have beenproposed, none of the methods appear to unquestionably outperform the others.

Original: As a result thequestion of how to effectively model uncertainty 
Revised: As a result thequestion of how to effectively model uncertainty is still open.

Original: 4.3 Feature-wise Linear Modulation
Revised: 4.3 Feature-wise Linear Modulation

Original: The idea of controlling the batch normalization parameters to modulate a network’s function has beenexplored by many different authors to accomplish different tasks.
Revised: The idea of controlling the batch normalization parameters to modulate a network’s function has beenexplored by many different authors to accomplish different tasks.

Original: Conditional batch normalization(CBN) was proposed by de Vries et al. , who achieved good performance in VQA experiments byusing an MLP to estimate residuals for normalization parameters used in a pre-trained ResNet [De
Revised: Conditional batch normalization(CBN) was proposed by de Vries et al. , who achieved good performance in VQA experiments byusing an MLP to estimate residuals for normalization parameters used in a pre-trained ResNet [De

Original: Vries et al., 2017b].
Revised: Vries et al., 2017b].

Original: Perez et al.
Revised: Perez et al.

Original: [2018] proposed FiLM also for solving VQA tasks.
Revised: [2018] proposed FiLM also for solving VQA tasks.

Original: Strub et al.[2018] modify FiLM to produce normalization parameters in several stages instead of all at once.
Revised: Strub et al.[2018] modify FiLM to produce normalization parameters in several stages instead of all at once.

Original: This formulation is better able to handle longer conditioning information, such as dialogues insteadof questions, and achieved excellent results on the GuessWhat?!
Revised: This formulation is better able to handle longer conditioning information, such as dialogues insteadof questions, and achieved excellent results on the GuessWhat?!

Original: visual dialogue task [De Vries et al.
Revised: visual dialogue task [De Vries et al.

Original: ,2017a].
Revised: ,2017a].

Original: Such modulation operations have also been used for image style transfer.
Revised: Such modulation operations have also been used for image style transfer.

Original: It has been observed that thestatistics of a feature map, which are associated with and can be controlled by the parameters producedby FiLM, are directly related to the style of an image [Huang and Belongie, 2017].
Revised: It has been observed that thestatistics of a feature map, which are associated with and can be controlled by the parameters producedby FiLM, are directly related to the style of an image [Huang and Belongie, 2017].

Original: Dumoulin et al.[2017] succeeded in capturing the styles of artistic paintings, as well as combining extracted styles tocreate new ones, by using conditional instance normalization, which can be seen as a variation of
Revised: Dumoulin et al.[2017] succeeded in capturing the styles of artistic paintings, as well as combining extracted styles tocreate new ones, by using conditional instance normalization, which can be seen as a variation of

Original: FiLM.
Revised: FiLM.

Original: Ghiasi et al.
Revised: Ghiasi et al.

Original: [2017] expand on this work by jointly training a style prediction network anda style transfer network, which also operate based on conditional instance normalization.
Revised: [2017] expand on this work by jointly training a style prediction network anda style transfer network, which also operate based on conditional instance normalization.

Original: Finally,
Revised: Finally,

Original: Brock et al.
Revised: Brock et al.

Original: [2018] used FiLM for natural image synthesis using generative adversarial networks(GANs).
Revised: [2018] used FiLM for natural image synthesis using generative adversarial networks(GANs).

Original: They report that this allowed for a reduction in computation and memory costs, as well as a37% increase in training speed.
Revised: They report that this allowed for a reduction in computation and memory costs, as well as a37% increase in training speed.

Original: Yang et al.
Revised: Yang et al.

Original: [2018] have used FiLM to modulate the layers of a segmentation network to performvideo object segmentation.
Revised: [2018] have used FiLM to modulate the layers of a segmentation network to performvideo object segmentation.

Original: This made it possible to avoid the fine-tuning process that was usedby competing methods, which resulted in a speed up of 70 times while achieving similar accuracy.
Revised: This made it possible to avoid the fine-tuning process that was usedby competing methods, which resulted in a speed up of 70 times while achieving similar accuracy.

Original: Feature modulation has also been applied for various other tasks.
Revised: Feature modulation has also been applied for various other tasks.

Original: Oreshkin et al.
Revised: Oreshkin et al.

Original: [2018] used FiLMfor task-dependent metric scaling, which allowed them to achieve excellent results in few-shotclassification.
Revised: [2018] used FiLMfor task-dependent metric scaling, which allowed them to achieve excellent results in few-shotclassification.

Original: Vuorio et al.
Revised: Vuorio et al.

Original: [2019] use FiLM for meta-learning via task-aware modulation.
Revised: [2019] use FiLM for meta-learning via task-aware modulation.

Original: Theauthors note that FiLM outperforms and is more stable than attention-based modulation in thiscontext.
Revised: Theauthors note that FiLM outperforms and is more stable than attention-based modulation in thiscontext.

Original: Finally, Vinyals et al.
Revised: Finally, Vinyals et al.

Original: [2019] used FiLM in their AlphaStar neural architecture for multiagent reinforcement learning.
Revised: [2019] used FiLM in their AlphaStar neural architecture for multiagent reinforcement learning.

Original: To our knowledge, FiLM has so far not been used for (implicit) modelensembling or uncertainty quantification.
Revised: To our knowledge, FiLM has so far not been used for (implicit) modelensembling or uncertainty quantification.

Original:  Conclusion & Future Work
Revised:  Conclusion & Future Work

Original: In this paper we present FiLM-Ensemble, a novel implicit deep ensembling method.
Revised: In this paper we present FiLM-Ensemble, a novel implicit deep ensembling method.

Original: We achieve highefficiency using a simple yet effective idea – feature-wise linear modulation – which has been shownto be effective in different domains, such as image style transfer, model-agnostic meta-learning,or multi-task learning.
Revised: We achieve highefficiency using a simple yet effective idea – feature-wise linear modulation – which has been shownto be effective in different domains, such as image style transfer, model-agnostic meta-learning,or multi-task learning.

Original: Our extensive evaluation shows that FiLM-Ensemble outperforms, or is onpar with, state-of-the-art ensemble methods in many different experimental settings.
Revised: Our extensive evaluation shows that FiLM-Ensemble outperforms, or is onpar with, state-of-the-art ensemble methods in many different experimental settings.

Original: A numberof measures designed to enhance implicit ensembles are orthogonal to our approach and couldpotentially be combined with FiLM-Ensemble to further improve its performance and uncertaintycalibration, while minimally increasing the computational costs.
Revised: A numberof measures designed to enhance implicit ensembles are orthogonal to our approach and couldpotentially be combined with FiLM-Ensemble to further improve its performance and uncertaintycalibration, while minimally increasing the computational costs.

Original: It appears straight-forward to addthe selection of independent examples for each member during training [as in MIMO, Havasi et al.
Revised: It appears straight-forward to addthe selection of independent examples for each member during training [as in MIMO, Havasi et al.

Original: ,2021], temperature scaling [Guo et al., 2017] or data augmentation strategies [such as, e.g., Raméet al., 2021].
Revised: ,2021], temperature scaling [Guo et al., 2017] or data augmentation strategies [such as, e.g., Raméet al., 2021].

Original:  Broader Impact
Revised:  Broader Impact

Original: Nowadays, Machine learning has resulted in rapidly growing computational cost and thus energycosts.
Revised: Nowadays, Machine learning has resulted in rapidly growing computational cost and thus energycosts.

Original: For instance, training state-of-the-art language models like GPT-3 would amount to at least1400 MWh or 4.6 million $.
Revised: For instance, training state-of-the-art language models like GPT-3 would amount to at least1400 MWh or 4.6 million $.

Original: Therefore, recently the term Green AI has been introduced, referring to
Revised: Therefore, recently the term Green AI has been introduced, referring to

Original: AI research that yields novel results while taking into account the computational cost, encouraging areduction in resources spent.
Revised: AI research that yields novel results while taking into account the computational cost, encouraging areduction in resources spent.

Original: To this extent, we believe that the presented method can foster moreefficient ensemble methods and be helpful towards a greener AI.
Revised: To this extent, we believe that the presented method can foster moreefficient ensemble methods and be helpful towards a greener AI.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references. Please read the checklist guidelines carefully for information onhow to answer these questions. For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
Revised: For all authors...

Original: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description. For example: • Did you include the license to the code and datasets?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes]See Section ?? .• Did you include the license to the code and datasets? [No] The code and the data areproprietary.•
Revised: [Yes]

Original:  Did you include the license to the code and datasets?
Revised: (b) Did you describe the limitations of your work?

Original: [N/A]
Revised: [Yes]

Original: Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope? [Yes](b) Did you describe the limitations of your work?
Revised: [No] Purely

Original: [Yes](c)
Revised: methodological contribution that, when added to a learning system, does not alterits potential societal impacts. If anything, better uncertainty calibration reduces

Original: Did you discuss any potential negative societal impacts of your work?
Revised: the risk of negative effects caused by undetected prediction errors.

Original: [No](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]2.
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A](b)
Revised: [N/A]

Original:  Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [No] 
Revised: [No] license information for the used datasets is public.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [Yes]
Revised: [Yes]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [N/A](b)
Revised: [N/A]

Original:  Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]