Original: U NCERTAINTY -A
Revised: U NCERTAINTY -A

Original: WARE
Revised: WARE

Original: D EEP
Revised: D EEP

Original: V IDEO
Revised: V IDEO

Original: C OMPRESSION
Revised: C OMPRESSION

Original: WITH E NSEMBLES
Revised: WITH E NSEMBLES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep learning-based video compression is a challenging task and many previous state-of-the-art learning-based video codecs use optical ﬂows to exploit the temporal correlation between successive frames and then compress the residual error.
Revised: Deep learning-based video compression is a challenging task and many previous state-of-the-art learning-based video codecs use optical ﬂows to exploit the temporal correlation between successive frames and then compress the residual error.

Original: Although these two-stage models are end-to-end optimized, errors in the intermediate errors are propagated to later stages and would harm the overall performance.
Revised: Although these two-stage models are end-to-end optimized, errors in the intermediate errors are propagated to later stages and would harm the overall performance.

Original: In this work we investigate the inherent uncertainty in these intermediate predictions and present an ensemble-based video compression model to capture the predictive uncertainty.
Revised: In this work we investigate the inherent uncertainty in these intermediate predictions and present an ensemble-based video compression model to capture the predictive uncertainty.

Original: We also propose an ensemble-aware loss to encourage the diversity between ensemble members and investigate the beneﬁt of incorporating adversarial training in the video compression task.
Revised: We also propose an ensemble-aware loss to encourage the diversity between ensemble members and investigate the beneﬁt of incorporating adversarial training in the video compression task.

Original: Experimental results on 1080p sequences show that our model can effectively save bits by more than 20% compared to DVC Pro (Lu et al., 2020b)
Revised: Experimental results on 1080p sequences show that our model can effectively save bits by more than 20% compared to DVC Pro (Lu et al., 2020b)

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Video contents are reported to account for 82% percent of all consumer Internet trafﬁc by 2021 and are growing rapidly with an increasing demand for high-resolution videos (e.g., 4K movies) and live streaming services (Cisco, 2020).
Revised: Video contents are reported to account for 82% percent of all consumer Internet trafﬁc by 2021 and are growing rapidly with an increasing demand for high-resolution videos (e.g., 4K movies) and live streaming services (Cisco, 2020).

Original: It is therefore critical for us to improve the video compression performance to transmit video with higher quality given limited Internet bandwidth.
Revised: It is therefore critical for us to improve the video compression performance to transmit video with higher quality given limited Internet bandwidth.

Original: In recent years, there has been a surge of deep learning-based video compression models (Rippel et al., 2019; Lu et al., 2020b; Agustsson et al., 2020) and some of them have achieved comparable or even better performance than previous traditional video codecs, such as x.264 (Tomar, 2006).
Revised: In recent years, there has been a surge of deep learning-based video compression models (Rippel et al., 2019; Lu et al., 2020b; Agustsson et al., 2020) and some of them have achieved comparable or even better performance than previous traditional video codecs, such as x.264 (Tomar, 2006).

Original: Despite previous deep learning-based video codecs have achieved improved performance on many challenging datasets, most of the state-of-the-art models estimate deterministic predictions for intermediate representations, such asthe optical ﬂows andthe residuals.
Revised: Despite previous deep learning-based video codecs have achieved improved performance on many challenging datasets, most of the state-of-the-art models estimate deterministic predictions for intermediate representations, such as optical ﬂows and residuals.

Original: These models fail to represent the aleatoric uncertainty inherent in the model inputs or the epistemic uncertainty in the model parameters and would blindly assume the predictions to be accurate, which is not always the case (Der Kiureghian & Ditlevsen, 2009; Kendall & Gal, 2017).
Revised: These models fail to represent the aleatoric uncertainty inherent in the model inputs or the epistemic uncertainty in the model parameters and would blindly assume the predictions to be accurate, which is not always the case (Der Kiureghian & Ditlevsen, 2009; Kendall & Gal, 2017).

Original: In terms of video compression, such models produce deterministic motion vectors (or optical ﬂows) and residuals for each pixel location, ignoring the fact that optical ﬂows may not be estimated accurately in occluded regions and around object boundaries, and the quantization operation for lossless entropy coding also introduces additional noises to the inputs of the decoders.
Revised: In terms of video compression, such models produce deterministic motion vectors (or optical ﬂows) and residuals for each pixel location, ignoring the fact that optical ﬂows may not be estimated accurately in occluded regions and around object boundaries, and the quantization operation for lossless entropy coding also introduces additional noises to the inputs of the decoders.

Original: Underlying errors in such overconﬁdent intermediate predictions are propagated to later stages of the P-frame model and even to subsequent frames for models built on temporal correlation, leading to suboptimal performance of the compression system.
Revised: Underlying errors in such overconﬁdent intermediate predictions are propagated to later stages of the P-frame model and even to subsequent frames for models built on temporal correlation, leading to suboptimal performance of the compression system.

Original: Predictive uncertainty is crucial for us to understand how certain the model is about the predictions, especially for out-of-distribution data.
Revised: Predictive uncertainty is crucial for us to understand how certain the model is about the predictions, especially for out-of-distribution data.

Original: However, most neural networks do not offer such information and tend to produce overconﬁdent predictions (Gal, 2016; Lakshminarayanan et al., 2017).
Revised: However, most neural networks do not offer such information and tend to produce overconﬁdent predictions (Gal, 2016; Lakshminarayanan et al., 2017).

Original: Bayesian neural networks (MacKay, 1992; Hinton & Neal, 1995) are widely used to quantify predictive uncertainty, but lack practicality due to greatly increased computation complexity and do not scale well to high dimensional data.
Revised: Bayesian neural networks (MacKay, 1992; Hinton & Neal, 1995) are widely used to quantify predictive uncertainty, but lack practicality due to greatly increased computation complexity and do not scale well to high dimensional data.

Original: Gal & Ghahramani (2016) proposed Monte Carlo dropout that performs test-time dropout.
Revised: Gal & Ghahramani (2016) proposed Monte Carlo dropout that performs test-time dropout.

Original: It is simple to implement but is not suitable for deep learning-based compression since it requires multiple decoding-time inferences and yields non-deterministic outputs.
Revised: It is simple to implement but is not suitable for deep learning-based compression since it requires multiple decoding-time inferences and yields non-deterministic outputs.

Original: In terms of deep learning-based video compression, two non-Bayesian approaches are considered to represent the predictive uncertainty: (1) modeling the uncertainty explicitly by regressing the empirical uncertainty of the model outputs (Nix & Weigend, 1994); and (2) using ensembles for predictive uncertainty estimation (Lakshminarayanan et al., 2017).
Revised: In terms of deep learning-based video compression, two non-Bayesian approaches are considered to represent the predictive uncertainty: (1) modeling the uncertainty explicitly by regressing the empirical uncertainty of the model outputs (Nix & Weigend, 1994); and (2) using ensembles for predictive uncertainty estimation (Lakshminarayanan et al., 2017).

Original: Agustsson et al.
Revised: Agustsson et al.

Original: (2020) took the ﬁrst approach and proposed scale-space ﬂow for motion compensation.
Revised: (2020) took the ﬁrst approach and proposed scale-space ﬂow for motion compensation.

Original: Speciﬁcally, their model predicts a scale ﬁeld besides the standard 2-dimension ﬂow ﬁeld, representing the uncertainty associated with each MV.
Revised: Speciﬁcally, their model predicts a scale ﬁeld besides the standard 2-dimension ﬂow ﬁeld, representing the uncertainty associated with each MV.

Original: Gaussian blurring is then applied to the reference frame and the scale parameter is used to control the scale of the Gaussian kernel.
Revised: Gaussian blurring is then applied to the reference frame and the scale parameter is used to control the scale of the Gaussian kernel.

Original: Experimental results show that the self-supervised scale parameter increases around the boundaries of the objects and in regions where the motion is large, which conﬁrms that the scale ﬁeld learns to estimate the uncertainty of the optical ﬂow prediction.
Revised: Experimental results show that the self-supervised scale parameter increases around the boundaries of the objects and in regions where the motion is large, which conﬁrms that the scale ﬁeld learns to estimate the uncertainty of the optical ﬂow prediction.

Original: In this work, we consider the second approach and represent the underlying uncertainty with an ensemble of outputs.
Revised: In this work, we consider the second approach and represent the underlying uncertainty with an ensemble of outputs.

Original: Ensembles of models have been shown to improve predictive performance (Dietterich, 2000), and ensembles of neural networks have also been used to boost model performance on many challenging benchmark datasets (Szegedy et al., 2015).
Revised: Ensembles of models have been shown to improve predictive performance (Dietterich, 2000), and ensembles of neural networks have also been used to boost model performance on many challenging benchmark datasets (Szegedy et al., 2015).

Original: Instead of producing a deterministic prediction, ensemble methods perform model combination and reﬂect the uncertainty for out-of-distribution data.
Revised: Instead of producing a deterministic prediction, ensemble methods perform model combination and reﬂect the uncertainty of out-of-distribution data.

Original: Here we propose the ensemble-based multi-head decoding module that can generate an ensemble of intermediate outputs, such as motion vectors and residuals, and implicitly represent the predictive uncertainty with the variance of the Gaussian mixture prediction.
Revised: Here we propose an ensemble-based multi-head decoding module that can generate an ensemble of intermediate outputs, such as motion vectors and residuals, and implicitly represent the predictive uncertainty with the variance of the Gaussian mixture prediction.

Original: We no longer assume that any individual intermediate prediction is accurate and explicitly consider multiple candidates of them.
Revised: We no longer assume that any individual intermediate prediction is accurate and explicitly consider multiple candidates of them.

Original: Such uncertainty is then propagated to later stages and all modules in our framework are optimized in an end-to-end fashion.
Revised: Such uncertainty is then propagated to later stages and all modules in our framework are optimized in an end-to-end fashion.

Original: To further improve the performance of our ensemble-based video compression model, we propose an ensemble-aware loss to encourage diversity between different branches and incorporate an adversarial training strategy, fast gradient sign method (FGSM)
Revised: To further improve the performance of our ensemble-based video compression model, we propose an ensemble-aware loss to encourage diversity between different branches and incorporate an adversarial training strategy, fast gradient sign method (FGSM)

Original: (Goodfellow et al., 2014), to smooth the intermediate latent representations.
Revised: (Goodfellow et al., 2014), to smooth the intermediate latent representations.

Original: Our experiments show that our model achieves an average bitrate saving of 15% when the distortion is measured in PSNR.
Revised: Our experiments show that our model achieves an average bitrate saving of 15% when the distortion is measured in PSNR.

Original: The contributions of this work can be summarized as follows:
Revised: The contributions of this work can be summarized as follows:

Original: • We are the ﬁrst to leverage deep ensembles in a video compression model and demonstrate that our approach can be used to capture the underlying uncertainty of intermediate representations and prevent the model from being overconﬁdent with intermediate predictions.
Revised: • We are the ﬁrst to leverage deep ensembles in a video compression model and demonstrate that our approach can be used to capture the underlying uncertainty of intermediate representations and prevent the model from being overconﬁdent with intermediate predictions.

Original: • We demonstrate that our approach can effectively improve the performance of deeplearning based video compression and can be widely applied to optical ﬂow-based video codecs with negligible complexity increase.
Revised: • We demonstrate that our approach can effectively improve the performance of deeplearning based video compression and can be widely applied to optical ﬂow-based video codecs with negligible complexity increase.

Original: • Our experiments show that our model outperforms previous state-of-the-art models such as DVC Pro (Lu et al., 2020b) and Lu et al.
Revised: • Our experiments show that our model outperforms previous state-of-the-art models such as DVC Pro (Lu et al., 2020b) and Lu et al.

Original: (2020a), and our ablation study proves the effectiveness of each module.
Revised: (2020a), and our ablation study proves the effectiveness of each module.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Learned video compression.
Revised: Learned video compression.

Original: Previous learning-based video compression methods can be categorized into two groups: (i) one-stage models, such as methods based on 3D autoencoders (Pessoa et al., 2020; Habibian et al., 2019); and (ii) two-stage models, which is adopted by most previous state-of-the-art methods, consist of predicted frame generation and residual coding.
Revised: Previous learning-based video compression methods can be categorized into two groups: (i) one-stage models, such as methods based on 3D autoencoders (Pessoa et al., 2020; Habibian et al., 2019); and (ii) two-stage models, which is adopted by most previous state-of-the-art methods, consist of predicted frame generation and residual coding.

Original: Lu et al.
Revised: Lu et al.

Original: (2019) proposed an end-to-end trainable video codec, DVC, that utilizes an optical-ﬂow network (Ranjan & Black, 2017) for motion compensation and then compresses the residuals.
Revised: (2019) proposed an end-to-end trainable video codec, DVC, that utilizes an optical-ﬂow network (Ranjan & Black, 2017) for motion compensation and then compresses the residuals.

Original: An extension of

Original:  DVC, known as DVC Pro (Lu et al., 2020b), improves the compression performance by introducing reﬁnement modules and auto-regressive entropy models.
Revised: An extension of DVC, known as DVC Pro (Lu et al., 2020b), improves the compression performance by introducing reﬁnement modules and auto-regressive entropy models.

Original: Feng et al.
Revised: Feng et al.

Original: (2020) considered residual coding in the feature domain and choose between pixel-level residuals and feature-level residuals with RDO at encoding time.
Revised: (2020) considered residual coding in the feature domain and choose between pixel-level residuals and feature-level residuals with RDO at encoding time.

Original: Agustsson et al.
Revised: Agustsson et al.

Original: (2020) proposed scale-space ﬂow to blur the intermediate reconstructions when motion vectors are not estimated well.
Revised: (2020) proposed scale-space ﬂow to blur the intermediate reconstructions when motion vectors are not estimated well.

Revised: Chen et al.

Revised: (2021) proposed to represent videos with neural networks and achieved promising results for video compression.

Original: Model uncertainty.
Revised: Model uncertainty.

Original: Predictive uncertainty can be grouped into aleatoric uncertainty and epistemic uncertainty (Der Kiureghian & Ditlevsen, 2009).
Revised: Predictive uncertainty can be grouped into aleatoric uncertainty and epistemic uncertainty (Der Kiureghian & Ditlevsen, 2009).

Original: Aleatoric uncertainty captures the noises inherent in the observations and cannot be explained away with more data, while epistemic uncertainty accounts for uncertainty in the model structure or parameters and can be reduced with more training data.
Revised: Aleatoric uncertainty captures the noises inherent in the observations and cannot be explained away with more data, while epistemic uncertainty accounts for uncertainty in the model structure or parameters and can be reduced with more training data.

Original: Bayesian neural networks (MacKay, 1992; Hinton & Neal, 1995) is a widely used approach for modeling predictive uncertainty that extends the traditional neural networks by learning a posterior distribution of model parameters from the observed data.
Revised: Bayesian neural networks (MacKay, 1992; Hinton & Neal, 1995) is a widely used approach for modeling predictive uncertainty that extends the traditional neural networks by learning a posterior distribution of model parameters from the observed data.

Original: Various non-Bayesian approacheshave also been proposed, such as utilizing the probabilities of softmax distributions (Hendrycks &
Revised: Various non-Bayesian approaches

Original: [Figure]
Revised: [Figure]

Revised: have also been proposed, such as utilizing the probabilities of softmax distributions (Hendrycks &

Original: Gimpel, 2016) and Specialists+1 Ensemble for representing the predictive uncertainty for adversarial samples (Abbasi & Gagn ´ e, 2017).
Revised: Gimpel, 2016) and Specialists+1 Ensemble for representing the predictive uncertainty for adversarial samples (Abbasi & Gagn ´ e, 2017).

Original: Gal & Ghahramani (2016) proposed Monte Carlo dropout by performing multiple inferences with Dropout at test time.
Revised: Gal & Ghahramani (2016) proposed Monte Carlo dropout by performing multiple inferences with Dropout at test time.

Original: Lakshminarayanan et al.
Revised: Lakshminarayanan et al.

Original: (2017) proposed to use an ensemble of neural networks for quantifying predictive uncertainty.
Revised: (2017) proposed to use an ensemble of neural networks for quantifying predictive uncertainty.

Original: Deep Ensembles.
Revised: Deep Ensembles.

Original: The neural networks community has been investigating ensembles of deep networks since the early 1990s (Hansen & Salamon, 1990; Wolpert, 1992; Perrone & Cooper, 1993).
Revised: The neural networks community has been investigating ensembles of deep networks since the early 1990s (Hansen & Salamon, 1990; Wolpert, 1992; Perrone & Cooper, 1993).

Original: Krogh & Vedelsby (1995) proved the bias-variance trade-off for ensemble models which suggested the importance of the diversity among ensemble members.
Revised: Krogh & Vedelsby (1995) proved the bias-variance trade-off for ensemble models which suggested the importance of the diversity among ensemble members.

Original: Lee et al.
Revised: Lee et al.

Original: (2015) investigated several training strategies to train an ensemble and proposed ensemble-aware oracle loss to encourage diversity.
Revised: (2015) investigated several training strategies to train an ensemble and proposed ensemble-aware oracle loss to encourage diversity.

Original: GoogLeNet (Szegedy et al., 2015), one of the best-performing models on ILSVRC 2014, is an ensembles of CNNs.
Revised: GoogLeNet (Szegedy et al., 2015), one of the best-performing models on ILSVRC 2014, is an ensembles of CNNs.

Original: Lakshminarayanan et al.
Revised: Lakshminarayanan et al.

Original: (2017) proposed to estimate predictive uncertainty by training multiple stand-alone neural networks.
Revised: (2017) proposed to estimate predictive uncertainty by training multiple stand-alone neural networks.

Original: Garipov et al.
Revised: Garipov et al.

Original: ; Fort et al.
Revised: ; Fort et al.

Original: (2020) showed that deep ensembles can learn different modes of function with ensemble members that only differ in initialization weights.
Revised: (2020) showed that deep ensembles can learn different modes of function with ensemble members that only differ in initialization weights.

Original:  U NCERTAINTY -A WARE D EEP V IDEO C OMPRESSION
Revised:  U NCERTAINTY -A WARE D EEP V IDEO C OMPRESSION

Original: This section presents our main contributions.
Revised: This section presents our main contributions.

Original: We introduce the theoretical background and the motivation of our proposed approach in Section 3.1.
Revised: We introduce the theoretical background and the motivation of our proposed approach in Section 3.1.

Original: Then we introduce the ensemble-based multihead structure to decode multiple candidates of motion vectors and residuals in Section 3.2.
Revised: Then we introduce the ensemble-based multihead structure to decode multiple candidates of motion vectors and residuals in Section 3.2.

Original: In order to encourage the diversity among the ensemble members and to improve the overall performance, we propose an ensemble-aware loss for multi-head decoders in Section 3.3.
Revised: In order to encourage the diversity among the ensemble members and to improve the overall performance, we propose an ensemble-aware loss for multi-head decoders in Section 3.3.

Original: Finally, we introduce an adversarial training strategy that we ﬁnd beneﬁcial for the learning-based video compression task in
Revised: Finally, we introduce an adversarial training strategy that we ﬁnd beneﬁcial for the learning-based video compression task in

Original: Section 3.4.
Revised: Section 3.4.

Original: 3. U NCERTAINTIES IN D EEP V IDEO C OMPRESSION
Revised: 3. U NCERTAINTIES IN D EEP V IDEO C OMPRESSION

Original: We consider the predictive coding-based framework, DVC, proposed by Lu et al.
Revised: We consider the predictive coding-based framework, DVC, proposed by Lu et al.

Original: (2020b) (Fig. 1a).
Revised: (2020b) (Fig. 1a).

Original: Let the current frame be x i and the reconstructed previous frame from the buffer be ˆ x i − 1 .
Revised: Let the current frame be x i and the reconstructed previous frame from the buffer be ˆ x i − 1 .

Original: We estimate a motion vector (MV) map f
Revised: We estimate a motion vector (MV) map f

Original: i with a motion estimation network.
Revised: i with a motion estimation network.

Original: The optical ﬂow is then sent to a motion auto-encoder for transform coding, yielding quantized bits ˆ a i and the reconstructed optical ﬂow ˆ f i .
Revised: The optical ﬂow is then sent to a motion auto-encoder for transform coding, yielding quantized bits ˆ a i and the reconstructed optical ﬂow ˆ f i .

Original: Bilinear warping is used for motion compensation  and a raw prediction ˜ F i with residual r i = 
Revised: Bilinear warping is used for motion compensation (MC) and a MC prediction ˜ x i with residual r i = x

Original: F i − ˜ F i is obtained.
Revised: i − ˜ x i is obtained.

Original: The residual r i is then compressed with a residual encoder and decoder, outputting quantized residual bit stream ˆ b i and the decoded residual
Revised: The residual r i is then compressed with a residual encoder and decoder, outputting quantized residual bit stream ˆ b i and the decoded residual

Original: ˆ r i .
Revised: ˆ r i .

Original: The reconstructed current frame isthen the sum of the motion compensation prediction and the decoded residual written as
Revised: The reconstructed current frame is the sum of the MC prediction and the decoded residual written as

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: found

Original: Aleatoric uncertainty.
Revised: Aleatoric uncertainty.

Original: Although at encoding time we have full information necessary to decode ˆ F i , forthe purpose of lossy compression at certain bit rates, we quantize the bit stream that is passed to the decoder, and inevitably introduce aleatoric uncertainty at decoding time.
Revised: Although at encoding time we have full information necessary to decode ˆ F i , for lossy compression at certain bit rates, we quantize the bit stream that is passed to the decoder, and inevitably introduce aleatoric uncertainty at decoding time.

Original: Since  aleatoric uncertainty cannot be reduced with more training data, a well-trained codec cannot reduce the quantization noise or fully recover the estimated MV f i .
Revised: Since the aleatoric uncertainty cannot be reduced with more training data, a well-trained codec cannot reduce the quantization noise or fully recover the estimated MV f i .

Original: Consider the MV auto-encoder in the DVC framework above.
Revised: Consider the MV auto-encoder in the DVC framework above.

Original: The lossy compression of the motion vectors can be summarized as
Revised: The lossy compression of the motion vectors can be summarized as

Original: [Equation]
Revised: [Equation]

Original: If the MV decoder is implemented with a linear model parameterized by w , the impact of the quantization noise η on the decoded MV is given by
Revised: If the MV decoder is implemented with a linear model parameterized by w , the impact of the quantization noise η on the decoded MV is given by

Original: [Equation]
Revised: [Equation]

Original: Since η is introduced by the quantization operation, we have (cid:107) η (cid:107) ∞ ≤ 1 / 2 = ε
Revised: Since η is introduced by the quantization operation, we have (cid:107) η (cid:107) ∞ ≤ 1 / 2 = ε

Original: and it follows that the upper-bound of the effects from the quantization operation is given by 
Revised: and it follows that the upper bound of the effects from the quantization operation is given by (cid:107)

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original:  While in practice the MV decoder is usually implemented with a stack of convolution layers and nonlinear activation layers, such as leaky ReLUs, the transformation of the MV decoder may be too linear to reject the quantization noise (Goodfellow et al., 2014).
Revised: [Table] [Equation] [Equation] While in practice the MV decoder is usually implemented with a stack of convolution layers and nonlinear activation layers, such as leaky ReLUs, the transformation of the MV decoder may be too linear to reject the quantization noise (Goodfellow et al., 2014).

Original: We visualize the aleatoric uncertainty by adding a small perturbation (cid:107) η 0 (cid:107)
Revised: We visualize the aleatoric uncertainty by adding a small perturbation (cid:107) η 0 (cid:107)

Original: ≈ 0 .
Revised: ≈ 0 .

Original: [Equation] η (cid:107) ∞ to the bit stream (see Appendix A.1) and the impact on the decoded MVs is depicted in Fig.
Revised: 2 · (cid:107) η (cid:107) ∞ to the bit stream (see Appendix A.2) and the impact on the decoded MVs is depicted in Fig.

Original: 9e.
Revised: 2d.

Original: It shows that there’s more aleatoric uncertainty in regions where the motion is complicated and the impact due to the aleatoric uncertainty is non-trivial.
Revised: It shows that there is more aleatoric uncertainty in regions where the motion is complicated and the impact due to the aleatoric uncertainty is nontrivial.

Original: Epistemic uncertainty.
Revised: Epistemic uncertainty.

Original: Due to limited observed data during training, epistemic uncertainty accounts for the uncertainty in the estimated motion vectors.
Revised: Due to limited observed data during training, epistemic uncertainty accounts for the uncertainty in the estimated motion vectors.

Original: Motion vectors near the object boundaries and occluded regions tend not to be estimated well and warpingsuch erroneous motion vectors would propagate errors to the residual coding.
Revised: Motion vectors near the object boundaries and occluded regions tend not to be estimated well, and warping erroneous motion vectors would propagate errors to the residual coding.

Original: We may roughly visualize such uncertainty by optimizing an motion estimation network with regards to the mean squared errors (MSE) between the 
Revised: We may roughly visualize such uncertainty by optimizing an motion estimation network with regards to the mean squared errors (MSE) between the current frame

Revised: x i and the warped frame

Original: [Equation]
Revised: [Equation]

Original: We depict the results in Fig. 9f.
Revised: We depict the results in Fig. 2e.

Original: Since the motion estimation is optimized to minimize theME MSE, regions where the MSE is large are likely to have a larger epistemic uncertainty and the corresponding motion vectors cannot be estimated well given the limited training data.
Revised: Since the motion estimation is optimized to minimize the MSE, regions where the MSE is large are likely to have a larger epistemic uncertainty and the corresponding motion vectors cannot be estimated well given the limited training data.

Original: Ideally, we could save MV bits by not encoding MVs that are not estimated well and save residual bits by not warping MVs that would not help to reduce residual.
Revised: Ideally, we could save MV bits by not encoding MVs that are not estimated well and save residual bits by not warping MVs that would not help to reduce residuals.

Original: This is often difﬁcult to implement as an end-to-end trained deep neural network.
Revised: This is often difﬁcult to implement as an end-to-end trained deep neural network.

Original: In the next section, we will show that with the help of multi-head decoders, the model could learn to exploit available information in the bit stream and handle those predictions with larger uncertainty.
Revised: In the next section, we will show that with the help of multi-head decoders, the model could learn to exploit available information in the bit stream and handle those predictions with larger uncertainty.

Original: 3. M ULTI -H EAD D ECODER
Revised: 3. M ULTI -H EAD D ECODER

Original: Our proposed multi-head decoder structure decodes multiple groups of motion vector (MV) for motion compensation and multiple groups of residual for ﬁnal reconstruction.
Revised: Our proposed multi-head decoder structure decodes multiple groups of motion vectors (MV) for motion compensation and multiple groups of residuals for ﬁnal reconstruction.

Original: The multi-head MV decoder and multi-head residual decoder are depicted in Fig.
Revised: The multi-head MV decoder and multi-head residual decoder are depicted in Fig.

Original: 1b.
Revised: 1b.

Original: Take the multi-head MV decoder as an example.
Revised: Take the multi-head MV decoder as an example.

Original: The MV decoder base ﬁrst decodes MV feature from the quantized MV bitstream ˆ a i .
Revised: The MV decoder base ﬁrst decodes MV feature from the quantized MV bitstream ˆ a i .

Original: Then h mv groups of MVs, denoted by { ˆ f ki | k = 1 , . .
Revised: Then h mv groups of MVs, denoted by { ˆ f ki | k = 1 , . .

Original: , h mv } , are decoded from the MV feature with respective MV decoder heads.
Revised: , h mv } , are decoded from the MV feature with respective MV decoder heads.

Original: We obtain h mv warped frames { ˜ F ki | k = 1 , . . 
Revised: We obtain h mv warped frames { ˜ x ki | k = 1 , . . .

Original: , h mv } by bilinearly warping each ˆ f ki on the reference frame ˆ F i − 1 .
Revised: , h mv } by bilinearly warping each ˆ f ki on the reference frame ˆ x i − 1 .

Original: The h mv warped frames are then concatenated for motion compensation and retained for ﬁnal reconstruction.
Revised: The h mv warped frames are then concatenated for motion compensation and retained for ﬁnal reconstruction.

Original: Many previous ensemble-based models train an ensemble of stand-alone neural networks (Szegedy et al., 2015; Abbasi & Gagn´e, 2017; Wang et al., 2020).
Revised: Many previous ensemble-based models train an ensemble of stand-alone neural networks (Szegedy et al., 2015; Abbasi & Gagn´e, 2017; Wang et al., 2020).

Original: While they can outperform the single-model baseline by a wide margin, the number of parameters are greatly increased, as well as the inference complexity.
Revised: While they can outperform the single-model baseline by a wide margin, the number of parameters are greatly increased, as well as the inference complexity.

Original: Lee et al.
Revised: Lee et al.

Original: (2015) proposed to share backbone parameters with TreeNets, but the models achieve the best performance when very few layers are shared.
Revised: (2015) proposed to share backbone parameters with TreeNets, but the models achieve the best performance when very few layers are shared.

Original: In our multi-head decoder structure, each decoder branch shares most of the convolution layers, making each decoder head light-weight.
Revised: In our multi-head decoder structure, each decoder branch shares most of the convolution layers, making each decoder head light weight.

Original: This design effectively improves the overall performance with negligible complexity increase (see
Revised: This design effectively improves the overall performance with negligible complexity increase (see

Original: Section 4.4).
Revised: Section 4.4).

Original: The ensemble of decoded MVs can be represented by an equally-weighted Gaussian mixture model given by
Revised: The ensemble of decoded MVs can be represented by an equally weighted Gaussian mixture model given by

Original: [Equation]
Revised: [Equation]

Original: where σ ki,x and σ ki,y are the variance in x and y directions respectively.
Revised: where σ ki,x and σ ki,y are the variance in x and y directions respectively.

Original: The mean and variance of the Gaussian mixture model are respectively (see Appendix A.2)
Revised: The mean and variance of the Gaussian mixture model are respectively (see Appendix A.3)

Original: [Equation]
Revised: [Equation]

Original: Empirically, we can visualize the predictive uncertainty represented by this ensemble model with the variance of the Gaussian mixture model by setting ( σ ki,x ) 2 =
Revised: Empirically, we can visualize the predictive uncertainty represented by this ensemble model with the variance of the Gaussian mixture model by setting ( σ ki,x ) [Equation] ( σ ki

Original: ( σ ki,y ) 2 = 1 , which gives
Revised: [Equation] y ) 2 = 1 , which gives

Original: [Equation]
Revised: [Equation]

Original: The predictive uncertainty for the ﬁrst two frames in the BasketballDrill sequence is depicted in Fig.
Revised: The predictive uncertainty for the ﬁrst two frames in the BasketballDrill sequence is depicted in Fig.

Original: 9f.
Revised: 2e.

Original: We can see that the predictive uncertainty properly capture both the aleatoric uncertainty and the epistemic uncertainty shown in Fig.
Revised: We can see that the predictive uncertainty properly capture both the aleatoric uncertainty and the epistemic uncertainty shown in Fig.

Original: 2: the basketball has large aleatoric uncertainty due to rapid motion and the object boundaries have large epistemic uncertainty.
Revised: 2: the basketball has large aleatoric uncertainty due to rapid motion and the object boundaries have large epistemic uncertainty.

Original: Relation to scale-space ﬂow.
Revised: Relation to scale-space ﬂow.

Original: Agustsson et al.
Revised: Agustsson et al.

Original: (2020) estimated a scale ﬁeld g z besides the 2dimensional optical ﬂow ( g x , g y ) .
Revised: (2020) estimated a scale ﬁeld g z besides the 2dimensional optical ﬂow ( g x , g y ) .

Original: We may represent the decoded MV with a multivariate Gaussian distribution given by
Revised: We may represent the decoded MV with a multivariate Gaussian distribution given by

Original: [Equation]
Revised: [Equation]

Original: and the scale-space warp gives a weighted mean of the warped value obtained from g mv .
Revised: and the scale-space warp gives a weighted mean of the warped value obtained from g mv .

Original: The
Revised: The

Original: Gaussian mixture prediction from our proposed multi-head decoder can represent a more diverse predictive uncertainty distribution than the single Gaussian distribution from the scale-space ﬂow.
Revised: Gaussian mixture prediction from our proposed multi-head decoder can represent a more diverse predictive uncertainty distribution than the single Gaussian distribution in the scale-space ﬂow.

Original: It has been shown in Garipov et al.
Revised: It has been shown in Garipov et al.

Original: ; Fort et al.
Revised: ; Fort et al.

Original: that ensemble models produce diverse results by learning different modes of the function, rather than interpolating around a given mean in the output space.
Revised: that ensemble models produce diverse results by learning different modes of the function, rather than interpolating around a given mean in the output space.

Original: It should be noted that while we only regress the mean of MV predictions in each ensemble branch and model the predictive uncertainty with the variance of the Gaussian mixture prediction, the multi-head decoder can easily be extended to an ensemble of scale-space ﬂows.
Revised: It should be noted that while we only regress the mean of MV predictions in each ensemble branch and model the predictive uncertainty with the variance of the Gaussian mixture prediction, the multi-head decoder can easily be extended to an ensemble of scale-space ﬂows.

Original: 3. E NSEMBLE -A WARE T RAINING
Revised: 3. E NSEMBLE -A WARE T RAINING

Original: Intuitively, diversity is a key factor for ensemble models.
Revised: Intuitively, diversity is a key factor for ensemble models.

Original: Ensemble members similar in the parameter space are unlikely to provide any more useful information than their single-model counterpart.
Revised: Ensemble members similar in the parameter space are unlikely to provide any more useful information than their single-model counterpart.

Original: Krogh & Vedelsby (1995) proved the bias-variance trade-off in ensemble, E = ¯ E − ¯ A , which suggested that the inherent variance is the key for the ensemble models to be effective and we should encourage the diversity among the ensemble members.
Revised: Krogh & Vedelsby (1995) proved the bias-variance trade-off in ensemble, E = ¯ E − ¯ A , which suggested that the inherent variance is the key for the ensemble models to be effective and we should encourage the diversity among the ensemble members.

Original: In the previous literature, multiple approaches are considered, including random initialization, bagging, and boosting.
Revised: In the previous literature, multiple approaches are considered, including random initialization, bagging, and boosting.

Original: Randomly initializing the model parameters is a simple but effective approach to induce randomness and is quite suitable for deep ensembles (Lee et al., 2015).
Revised: Randomly initializing the model parameters is a simple but effective approach to induce randomness and is quite suitable for deep ensembles (Lee et al., 2015).

Original: Bagging trains ensemble members on independently drawn examples with bootstrap sampling.
Revised: Bagging trains ensemble members on independently drawn examples with bootstrap sampling.

Original: Lee et al.
Revised: Lee et al.

Original: (2015) showed that bagging may harm the model performance since each model may see only 63% of the available data and would perform poorly when there is high correlation inherent in the data (Bartlett et al., 1998).
Revised: (2015) showed that bagging may harm the model performance since each model may see only 63% of the available data and would perform poorly when there is high correlation inherent in the data (Bartlett et al., 1998).

Original: Boosting generates the ensemble models sequentially and can be quite time-consuming in terms of neural networks.
Revised: Boosting generates the ensemble models sequentially and can be quite time-consuming in terms of neural networks.

Original: Wang et al.

Original: (2020) proposed a modiﬁed boosting that divide the dataset into clusters with a base model and all ensemble models can be trained in parallel.

Original: For deep compression models with our proposed multi-head decoder structure, there is a single model with multiple parallel ensemble branches.
Revised: For deep compression models with our proposed multi-head decoder structure, there is a single model with multiple parallel ensemble branches.

Original: Bagging or boosting wouldn’t work for such network design since the model is trained on randomly sampled data but different ensemble branches would see the same data.
Revised: Bagging or boosting would not work for such network design since the model is trained on randomly sampled data but different ensemble branches would see the same data.

Original: Therefore, we choose to randomly initialize the network parameters and initial experiments show the effectiveness.
Revised: Therefore, we choose to randomly initialize the network parameters and initial experiments show the effectiveness.

Original: In order to further encourage the diversity among the ensemble members, we propose an ensemble-aware lossdesigned for our multi-head structure.
Revised: To further encourage the diversity among the ensemble members, we propose an ensemble-aware loss for deep ensembles to induce additional randomness.

Original: Given h decoded outputs from the multi-head decoder, we obtain h predictions (motioncompensated predictions or ﬁnal reconstructions) ¯ F ji for j = 1 , . . .
Revised: Given h decoded outputs from the multi-head decoder, we obtain h predictions (motioncompensated predictions or ﬁnal reconstructions) ¯ x t for t = 1 , . . .

Original: , h .
Revised: , h .

Original: The ensemble-aware loss is given by
Revised: The ensemble-aware loss is given by

Original: [Equation]
Revised: [Equation]

Original: and the minimum can be relaxed to choosing the smallest k out of the h norms.
Revised: and the minimum can be relaxed to choosing the smallest k out of the h norms.

Original: It is similar to  theensemble-aware oracle  lossbuilt on multiple choice learning proposed in Lee et al.
Revised: Our ensembleaware loss encourage diversity between heads by relaxing the magnitude of the loss, and at the same time ensure performance of individual heads by updating every head at every iteration. Since the gradients propagated to each head depend on the head with the minimal MSE, rather than the MSE of the head itself, we can effectively induce randomness and therefore encourage diversity. Although the oracle set loss proposed in Lee et al.

Original: Although the oracle loss helps to encourage diversity and boost the oracle accuracy, it would signiﬁcantly harm the performance of individual ensemble members, especially for small k ’s. Experiments on learned video compression also showed that adopting the oracle  loss would decrease the overall performance, no matter the choice of k .
Revised: (2015) helps to encourage diversity and boost the oracle accuracy, it would signiﬁcantly harm the performance of individual ensemble members since each head only sees a small portion of all training data. Experiments on learned video compression also showed that adopting the oracle set loss will decrease the overall performance.

Original: Instead,in our ensemble-aware loss, each ensemble member is supervised but gets the same supervision as the best-performing one. The beneﬁt of this loss is that each individual member gets to see all the available data, yet the underlying disagreement among the ensemble members is allowed.
Revised: Instead, our ensemble-aware loss can effectively encourage diversity among ensemble members, and at the same time, each individual member sees all training data and achieves reliable performance.

Original: 3. A DVERSARIAL T RAINING WITH FGSM
Revised: 3. A DVERSARIAL T RAINING WITH FGSM

Original: Adversarial examples (Szegedy et al., 2013) are training samples with a small but non-random perturbation that are misclassiﬁed by neuralnetworks with high conﬁdence. Goodfellow et al. (2014) proposed the fast gradient sign method (FGSM) that applies linear but intentionally worst-case perturbation to the training samples, which is given by
Revised: Adversarial examples (Szegedy et al., 2013) are training samples with a small but non-random perturbation that are misclassiﬁed by neural proposed the fast gradient sign method (FGSM) that applies linear but intentionally worst-case perturbation to the training samples, which is given by

Original: [Equation]
Revised: [Equation]

Original: ) (11) where J ( θ, x, y ) is the cost function, and (cid:15) controls the norm of the perturbation.
Revised: where J ( θ, x, y ) is the cost function, and (cid:15) controls the norm of the perturbation.

Original: This adversarial training strategy has been shown to boost the image classiﬁcation performance, as well as improve the model’s robustness to adversarial examples.
Revised: This adversarial training strategy has been shown to boost the image classiﬁcation performance, as well as improve the model’s robustness to adversarial examples.

Original: Lakshminarayanan et al.
Revised: Lakshminarayanan et al.

Original: (2017) interpreted FGSM as an efﬁcient solution to smooth the predictive distributions by increasing the likelihood of the target around an (cid:15) -neighborhood of the observed training samples.
Revised: (2017) interpreted FGSM as an efﬁcient solution to smooth the predictive distributions by increasing the likelihood of the target around an (cid:15) -neighborhood of the observed training samples.

Original: We ﬁnd adversarial training with FGSM closely related tothe task of learned lossy compression and an effective approach to improve the performance of learned video compression models.
Revised: We ﬁnd adversarial training with FGSM closely related to learned lossy compression and an effective approach to improve the performance of learned video codecs.

Original: Interms of transform coding, we want the latent representation to be as smooth as possible, since after quantization, all latent representations in the (cid:15) -neighborhood, { ˆ a + η | (cid:107) η (cid:107) ∞ < (cid:15) } , corresponds to the same decoded output.
Revised: In transform coding, we want the latent representation to be as smooth as possible, since after quantization, all latent representations in the (cid:15) -neighborhood, { ˆ a + η | (cid:107) η (cid:107) ∞ < (cid:15) } , corresponds to the same decoded output.

Original: Learning a smooth latent representation would help to make the outputs more robust to quantization noise.
Revised: Learning a smooth latent representation would help to make the outputs more robust to quantization noise.

Original: Although this could be a natural result of an end-to-end optimized video codec,  experimental results show that FGSM can effectively improve the video compression performance.
Revised: Although this could be a natural result of an end-to-end optimized video codec, the experimental results show that FGSM can effectively improve the rate-distortion performance.

Original:  E XPERIMENTS 4. E XPERIMENTAL S ETUP
Revised:  E XPERIMENTS 4. E XPERIMENTAL S ETUP

Original: Model architecture.
Revised: Model architecture.

Original: Our base model architecture follows the design in Lu et al.
Revised: Our base model architecture follows the design in Lu et al.

Original: (2020b)
Revised: (2020b)

Original: and we use auto-regressive and hierarchical priors for both the motion vector and residual compression.
Revised: and we use auto-regressive and hierarchical priors for both the motion vector and residual compression.

Original: In order to optimize the model in an end-to-end manner, we need to relax the bits estimation since quantizing the latent bits would make the gradients zero almost everywhere.
Revised: In order to optimize the model in an end-to-end manner, we need to relax the bits estimation since quantizing the latent bits would make the gradients zero almost everywhere.

Original: Following Ball ´ e et al.
Revised: Following Ball ´ e et al.

Original: , we substitute the quantization operation with additive uniform noise during training and perform actual quantization during inference.
Revised: , we substitute the quantization operation with additive uniform noise during training and perform actual quantization during inference.

Original: For multi-head decoder modules,
Revised: The structure of multi-head decoder modules are depicted in Figure 1b and more technical details on the implementation are available in Appendix A.1.

Original: Training datasets.
Revised: Training datasets.

Original: Our model is trained on 64,612 video sequences from the training part in Vimeo90K settuplet dataset (Xue et al., 2019).
Revised: Our model is trained on 64,612 video sequences from the training part in Vimeo90K settuplet dataset (Xue et al., 2019).

Original: Each video clip has 7 frames with a resolution of 448 × 256 .
Revised: Each video clip has 7 frames with a resolution of 448 × 256 .

Original: During training, we randomly crop the video sequences into 256 × 256 pixels.
Revised: During training, we randomly crop the video sequences into 256 × 256 pixels.

Original: Given two successive frames from a random sequence, we treat the ﬁrst frame as the reference frame and our model is trained to minimize the rate-distortion cost of encoding and decoding the second frame.
Revised: Given two successive frames from a random sequence, we treat the ﬁrst frame as the reference frame and our model is trained to minimize the rate-distortion cost of encoding and decoding the second frame.

Original: Training details.
Revised: Training details.

Original: In our experiments, we adopt the progressive training strategy and warm up the inter-coding module for 150,000 steps with the ensemble-aware motion compensation loss in
Revised: In our experiments, we adopt the progressive training strategy and warm up the inter-coding module for 150,000 steps with the ensemble-aware motion compensation loss in

Original: Equation 10.
Revised: Equation 10.

Original: Then the model is end-to-end optimized with the rate-distortion loss given by
Revised: Then the model is end-to-end optimized with the rate-distortion loss given by

Original: [Equation]
Revised: [Equation]

Original: where R w ( ˆ w i ) and R z (ˆ z i ) represent the numbers of bits used to encode the motion vectors and the residual, D ( F i , ˆ F i ) measures the distortion in mean squared error MSE or multi-scale structural similarity MS-SSIM (Wang et al., 2003), and λ is the hyperparameter controlling the trade-off.
Revised: where R w ( ˆ w i ) and R z (ˆ z i ) represent the numbers of bits used to encode the motion vectors and the residual, D ( F i , ˆ F i ) measures the distortion in mean squared error MSE or multi-scale structural similarity MS-SSIM (Wang et al., 2003), and λ is the hyperparameter controlling the trade-off.

Original: Four models are trained with different quality rates by setting λ = 256 , 512 , 1024 , 2048 .
Revised: Four models are trained with different quality rates by setting λ = 256 , 512 , 1024 , 2048 .

Original: We use the
Revised: We use the

Original: AdamW optimizer (Loshchilov & Hutter, 2017) with an initial learning rate of 1 × 10 − 4 , which is then decreased to 1 × 10 − 5 .
Revised: AdamW optimizer (Loshchilov & Hutter, 2017) with an initial learning rate of 1 × 10 − 4 , which is then decreased to 1 × 10 − 5 .

Original: Each model is trained on one NVIDIA V100 GPU.
Revised: Each model is trained on one NVIDIA V100 GPU.

Original: 4. Q UANTITATIVE R ESULTS
Revised: 4. Q UANTITATIVE R ESULTS

Original: To show the effectiveness of the proposed uncertainty-awareensemble-based model, we test our model on the ﬁrst 100 frames from video sequences in HEVC (Sullivan et al., 2012) with GoP size 10, and the ﬁrst 120 frames from sequences in UVG (Mercat et al., 2020), MCL-JCV (Wang et al., 2016) with GoP size 12.
Revised: To show the effectiveness of our proposed uncertainty-aware model, we test our model on the ﬁrst 100 frames from video sequences in HEVC (Sullivan et al., 2012) with GoP size 10, and the ﬁrstframes from sequences in UVG (Mercat et al., 2020), MCL-JCV (Wang et al., 2016) with GoP size 12.

Original: To balance the trade-off between complexity and performance, we choose h mv
Revised: To balance the trade-off between complexity and performance, we choose h mv

Original: = h res = 4 .
Revised: = h res = 4 .

Original: In order to build the best learning-based video codec, we adopt the state-ofthe-art image compression model (Cheng et al., 2020) for intra frame coding.
Revised: In order to build the best learning-based video codec, we adopt the state-of-the-art image compression model (Cheng et al., 2020) for intra frame coding.

Original: In order to fairly compare the performance, we test the DVC and DVC Pro with the same same intra frame model, denoted as DVC (cheng2020) and DVC Pro (cheng2020).
Revised: In order to fairly compare the performance, we test the DVC and DVC Pro with the same same intra frame model, denoted as DVC (cheng2020) and DVC Pro (cheng2020).

Original: For some of the previous state-of-the-art deep video compression methods that are not available, such as HU ECCV20 (Hu et al., 2020) and LU ECCV20 (Lu et al., 2020a), we use the public results reported in Hu (2020). We calculate the{+ECCV
+}
Revised: For some of the previous state-of-the-art deep video compression methods that are not available, such as HU ECCV20 (Hu et al., 2020) and LU {+ECCV
+}

Original: BD-rate (Bjøntegaard, 2001) of different learning-based video compression models using x.265 as the anchor model and the results on HEVC, UVG, MCL-JCV are reported in Table 1. We also plot the RD curves of different codec models in Fig. More RD curve plots are available in Appendix A.3.
Revised: (Lu et al., 2020a),

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Agustsson et al., 2020) on sequences from HEVC Class B and UVG.
Revised: Agustsson et al., 2020), and NeRV (Chen et al., 2021) on sequences from HEVC Class B and UVG.

Original: Results on other HEVC sequences and MCL-JCV are reported in Appendix A.3.
Revised: Results on other HEVC sequences and MCL-JCV are reported in Appendix A.4.

Original: Best viewed in color.
Revised: Best viewed in color.

Revised: we use the public results reported in Hu (2020).

Revised: We calculate the BD-rate (Bjøntegaard, 2001) of different learning-based video compression models using x.265 as the anchor model and the results on HEVC, UVG, MCL-JCV are reported in Table 1.

Revised: We also plot the RD curves of different codec models in Fig.

Revised: More RD curve plots are available in Appendix A.4.

Original: From the results reported in Tab. 1 and Fig.
Revised: From the results reported in Table 1 and Fig.

Original: 3, we could see that our proposed model can effectively save bits compared to our strong baseline and outperform previous state-of-the-art learning-based video codecs by a wide margin in all testing datasets.
Revised: 3, we could see that our proposed model can effectively save bits compared to our strong baseline and outperform previous state-of-the-art learning-based video codecs by a wide margin in all testing datasets.

Original: 4. Q UALITATIVE R ESULTS
Revised: 4. Q UALITATIVE R ESULTS

Original: In Fig.
Revised: In Fig.

Original: 2 we visualize the aleatoric uncertainty and epistemic uncertainty in the ﬁrst two frames of the BasketballDrill sequence, as well as the predictive uncertainty represented by the multi-head
Revised: 2 we visualize the aleatoric uncertainty and epistemic uncertainty in the ﬁrst two frames of the BasketballDrill sequence, as well as the predictive uncertainty represented by the multi-head

Original: MV decoder.
Revised: MV decoder.

Original: As we can see, the predictive uncertainty is larger in regions where the motion cannot be estimated well or too complicated to encode.
Revised: As we can see, the predictive uncertainty is larger in regions where the motion cannot be estimated well or too complicated to encode.

Original: Our uncertainty-aware model learns to effectively represent the underlying uncertainty with an ensemble of decoded MVs and such uncertainty if retained until  ﬁnal reconstruction.
Revised: Our uncertainty-aware model learns to effectively represent the underlying uncertainty with an ensemble of decoded MVs and such uncertainty is retained until the ﬁnal reconstruction.

Original: Other qualitative results can be found in Appendix A.4.
Revised: More visualizations of the unsupervised predictive uncertainty can be found in Appendix A.6. [Figure]

Original: 4. A BLATION S TUDY
Revised: 4. A BLATION S TUDY

Original: Effectiveness of various proposed modules.
Revised: Effectiveness of various proposed modules.

Original: We evaluate the effectiveness of multi-head decoders, ensemble-aware loss, and adversarial training with FGSM by running ablation experiments on the ﬁrst 30 frames from all sequences in the HEVC dataset.
Revised: We evaluate the effectiveness of multi-head decoders, ensemble-aware loss, and adversarial training with FGSM by running ablation experiments on the ﬁrst 30 frames from all sequences in the HEVC dataset.

Original: We adopt the short training strategy for[Figure] fast experimentation.
Revised: We adopt the short training strategy for fast experimentation.

Original: The RD curves are presented in Fig.
Revised: The RD curves are presented in Fig.

Original: 4a and the BD-rates are reported in A.5.
Revised: 4a and the BD-rates are reported in A.8.

Original: MH-MV and MH-Res stand for multi-head MV decoder and multi-head residual decoder, and EA-L refers to training with ensemble-aware loss.
Revised: MH-MV and MH-Res stand for multi-head MV decoder and multi-head residual decoder, and EA-L refers to training with ensemble-aware loss.

Original: The results show the effectiveness of the multi-head decoder modules and the training strategies.
Revised: The results show the effectiveness of the multi-head decoder modules and the training strategies.

Original: Ablation study on the number of decoder heads.
Revised: Ablation study on the number of decoder heads.

Original: We investigate the model’s performance with different number of headsh mv in the multi-head decoder on HEVC sequences.
Revised: We investigate the model’s performance with different number of heads in the multi-head decoder on HEVC sequences.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 4b, we train eight models with h mv = 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 where h mv = 1 is the baseline without ensembling.
Revised: 4b, we train eight models with h mv = 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 where h mv = 1 is the baseline without ensembling.

Original: We see that the multi-head decoder module is effective even with only two decoder heads and the performance is improved with more decoder heads.
Revised: We see that the multi-head decoder module is effective even with only two decoder heads and the performance is improved with more decoder heads.

Original: Quantitative results are reported in A.5.
Revised: Quantitative results are reported in A.8.

Original: Complexity analysis.
Revised: Complexity analysis.

Original: Most previous deep ensembles train multiple stand-alone models (Szegedy et al., 2015; Abbasi & Gagn ´ e, 2017; Wang et al., 2020) or share very few shallow layers (Lee et al., 2015) for the model to be effective.
Revised: Most previous deep ensembles train multiple stand-alone models (Szegedy et al., 2015; Abbasi & Gagn ´ e, 2017; Wang et al., 2020) or share very few shallow layers (Lee et al., 2015) for the model to be effective.

Original: With an ensemble of 6 models, the inference complexity (in
Revised: With an ensemble of 6 models, the inference complexity (in

Original: MACs) and model size (in number of parameters)would easily increase by 500% .
Revised: MACs) and model size (in number of parameters) easily increase by 500% .

Original: However, with the help of multi-head decoders, our ensemble-based model share the backbone features and achieve superior results with limited complexity increase.
Revised: With the help of multihead decoders, our ensemble-based model share the backbone features and achieve superior results with limited complexity increase.

Original: For one extra MVdecoder and residual decoder added, the complexity increases by approximately 7% and just 1% in the model size.
Revised: For one extra MV and residual decoder added, the complexity increases by approximately 7% and just 1% in the model size.

Original: Even for the largest model we consider, where h mv = h res = 8 , there is only a 48% increase in complexity and 10% in model size.
Revised: For the largest model we consider, where h mv = h res = 8 , there is only a 48% increase in complexity and 10% in model size.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In this paper, we studied the aleatoric uncertainty and epistemic uncertainty in deep learning-based video compression and proposed to utilize an ensemble of intermediate predictions to represent the predictive uncertainty at decoding time.
Revised: In this paper, we studied the aleatoric uncertainty and epistemic uncertainty in deep learning-based video compression and proposed to utilize an ensemble of intermediate predictions to represent the predictive uncertainty at decoding time.

Original: With multi-head decoders, our model can properly model the uncertainties in the decoded MVs or residuals, which would effectively avoid propagating underlying errors in a single deterministic prediction.
Revised: With multi-head decoders, our model can properly model the uncertainties in the decoded MVs or residuals, which would effectively avoid propagating underlying errors in a single deterministic prediction.

Original: We investigated the performance of our uncertainty-aware decoding module and proposed a novel ensemble-aware loss to boost the diversity among the parallel ensemble branches in a single model.
Revised: We investigated the performance of our uncertainty-aware decoding module and proposed a novel ensemble-aware loss to boost the diversity among the parallel ensemble branches in a single model.

Original: We also proposed to incorporate adversarial training for learning-based video codecs.
Revised: We also proposed to incorporate adversarial training for learning-based video codecs.

Original: Experimental results show the effectiveness of our approach.
Revised: Experimental results show the effectiveness of our approach.

Original: Compared to one-stage learning-based video compression models, such as those based on 3D autoencoders (Pessoa et al., 2020; Habibian et al., 2019), two-stage motion compensation-based models can decode high-quality frames with low-latency.
Revised: Compared to one-stage learning-based video compression models, such as those based on 3D autoencoders (Pessoa et al., 2020; Habibian et al., 2019), two-stage motion compensation-based models can decode high-quality frames with low-latency.

Original: However, intermediate predictions in these two-stage pipelines are not always accurate and erroneous predictions could severely harm the performance of later stages, especially for out-of-distribution data.
Revised: However, intermediate predictions in these two-stage pipelines are not always accurate and erroneous predictions could severely harm the performance of later stages, especially for out-of-distribution data.

Original: Therefore, it is critical to represent the predictive uncertainty and our proposed multi-head decoder is a simple but very effective approach to capture such uncertainty.
Revised: Therefore, it is critical to represent the predictive uncertainty and our proposed multi-head decoder is a simple but very effective approach to capture such uncertainty.

Original: Future directions could involve modules on the encoder side to model and propagate the uncertainty to the decoders for an end-to-end uncertainty-awareness.
Revised: Future directions could involve modules on the encoder side to model and propagate the uncertainty to the decoders for an end-to-end uncertainty-awareness.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: Key results in this paper are reproducible and in this section, we will refer to details helpful to reproduce the results.
Revised: Key results in this paper are reproducible and in this section, we will refer to details helpful to reproduce the results.

Original: The implementation of our model are modiﬁed from the codes released by
Revised: The implementation of our model are modiﬁed from the codes released by

Original: B´egaint et al.
Revised: B´egaint et al.

Original: (2020) and Hu (2020).
Revised: (2020) and Hu (2020).

Original: We propose the multi-head decoders in Section 3.2 and can be easily implemented by following the ﬂow charts in Figure 1.
Revised: We propose the multi-head decoders in Section 3.2 and can be easily implemented by following the ﬂow charts in Figure 1b and technical details in Appendix A.1.

Original: In Section 4.1 we introduce the dataset we use to train the model.
Revised: In Section 4.1 we introduce the dataset we use to train the model.

Original: The Vimeo-90K settuplet dataset (Xue et al., 2019) is publicly available and we ﬁt our model on the training part of the dataset.
Revised: The Vimeo-90K settuplet dataset (Xue et al., 2019) is publicly available and we ﬁt our model on the training part of the dataset.

Original: Also in Section 4.1 we summarize our training strategy, which includes an ensemble-aware loss for model warm-up and then minimizing the rate-distortion cost until convergence.
Revised: Also in Section 4.1 we summarize our training strategy, which includes an ensemble-aware loss for model warm-up and then minimizing the rate-distortion cost until convergence.

Original: The intuition and details of the ensemble-aware loss can be found in Section 3.
Revised: The motivation and formality of the ensemble-aware loss can be found in Section 3.3.

Revised: This loss can be easily implemented in PyTorch.

Revised: Since directly clamping the loss gives zero gradients almost everywhere, we create a custom autograd function to forward the gradients before and after the clamp operation and properly update the weights of each head.

Original: A A PPENDIX A. V ISUALIZING THE A LEATORIC U NCERTAINTY
Revised: A A PPENDIX A. I MPLEMENTATION OF M ULTI -H EAD D ECODERS

Revised: As depicted in Figure 1b, multi-head decoders consist of a decoder backbone and multiple decoder heads.

Revised: The decoder heads are light-weight and include two convolution layers with one leaky ReLU in between.

Revised: For multi-head MV decoder, the decoder backbone ﬁrst decodes MV feature from the

Revised: MV bitstream ˆ a i , and then h MVs are decoded with respective MV decoder heads.

Revised: From the h decoded MVs and the previous decoded frame, we obtain h MC predictions with bilinear warping.

Revised: The h MC predictions are concatenated with the previous decoded frame and sent to the Prediction

Revised: Reﬁne Net, from which we get h reﬁned MC predictions.

Revised: For multi-head residual decoder, the decoder backbone ﬁrst decodes residual feature from the residual bitstream ˆ b i , and then h residuals are decoded with respective residual decoder heads.

Revised: From the h decoded residuals and the h reﬁned MC predictions, we obtain h reconstructions.

Revised: The h reconstructions are concatenated with h reﬁned MC predictions and sent to the Reconstruction Reﬁne Net, from which we get 1 reﬁned reconstruction as the ﬁnal decoded frame.

Revised: All modules in our model, including decoder backbone, decoder head, and reﬁne nets, are implemented with neural networks and optimized in an end-to-end manner.

Original:  To visualize the aleatoric uncertainty introduced from the quantization operation, we conduct preliminary experiments.
Revised: V ISUALIZING THE A LEATORIC U NCERTAINTY To visualize the aleatoric uncertainty introduced from the quantization operation, we conduct preliminary experiments.

Original: We add a small perturbation η 0 to the quantized bit stream and obtain ˆ w (cid:48)
Revised: We add a small perturbation η 0 to the quantized bit stream and obtain ˆ w (cid:48)

Original: i
Revised: i

Original: =
Revised: =

Original: ˆ w i + η 0 .
Revised: ˆ w i + η 0 .

Original: The perturbation η 0 is only added to positions where the quantization gap is at least 0 .
Revised: The perturbation η 0 is only added to positions where the quantization gap is at least 0 .

Original: 1 and corresponds to only 20% of the size of the perturbation gap.
Revised: 1 and corresponds to only 20% of the size of the perturbation gap.

Original: We model the aleatoric uncertainty with the L1 norm between the two optical ﬂows decoded from bit streams that differs in a small perturbation.
Revised: We model the aleatoric uncertainty with the L1 norm between the two optical ﬂows decoded from bit streams that differs in a small perturbation.

Original: Results on the ﬁrst two frames of the BasketballDrill sequence are shown in
Revised: Results on the ﬁrst two frames of the BasketballDrill sequence are shown in

Original: Fig. 9e.
Revised: Figure 2d.

Original: As we can see, the aleatoric uncertainty is not uniform across the whole image.
Revised: As we can see, the aleatoric uncertainty is not uniform across the whole image.

Original: Instead, there’re more aleatoric uncertainty around the object boundaries and regions where the motion is large.
Revised: Instead, there are more aleatoric uncertainty around the object boundaries and regions where the motion is large.

Original: While by deﬁnition, such aleatoric uncertainty cannot be reduced away, blindly assuming the optical ﬂows to be accurate would lead to larger intermediate residual errors and cost more bits in the residual coding.
Revised: While by deﬁnition, such aleatoric uncertainty cannot be reduced away, blindly assuming the optical ﬂows to be accurate would lead to larger intermediate residual errors and cost more bits in the residual coding.

Original: A. MV P REDICTION AS A G AUSSIAN M IXTURE M ODEL
Revised: A. MV P REDICTION AS A G AUSSIAN M IXTURE M ODEL

Original: The multi-head MV decoder predicts an ensemble of motion vectors { ˆ f ki | k = 1 , . .
Revised: The multi-head MV decoder predicts an ensemble of motion vectors { ˆ f ki | k = 1 , . .

Original: , h mv } from the MV bit stream
Revised: , h mv } from the MV bit stream

Original: and we can represent the prediction as an equally-weighted Gaussian mixture model given by
Revised: and we can represent the prediction as an equally-weighted Gaussian mixture model given by

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since the Gaussian distributions are equally-weighted, the mean of the mixture distribution is
Revised: Since the Gaussian distributions are equally-weighted, the mean of the mixture distribution is

Original: [Equation]
Revised: [Equation]

Original: (14) and the variance of the mixture distribution in each direction is
Revised: and the variance of the mixture distribution in each direction is

Original: [Equation]
Revised: [Equation]

Original: In order to visualize the predictive uncertainty represented by this ensemble model
Revised: In order to visualize the predictive uncertainty represented by this ensemble model, we set ( σ ki,x ) 2 = ( σ ki,y ) 2 = 1 and obtain

Original: [Equation]
Revised: [Equation]

Original: we set ( σ ki,x ) 2 = ( σ ki,y ) 2 = 1 and obtain
Revised: [Figure] [Figure]

Original: [Equation]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Revised: A. M ODEL P ERFORMANCE USING D IFFERENT I-F RAME M ODELS

Original: [Figure]
Revised: In this section we investigate the performance of our proposed model using different I-frame models on HEVC Class B and UVG. In Section 4. we report the experimental results of our proposed model using Cheng2020 (Cheng et al., 2020) as the I-frame model. Compared to DVC Pro, our proposed model with multi-head decoders can effectively reduce the bpp while maintaining the same reconstruction quality.

Original: [Figure]
Revised: As shown in Figure 3, this brings our model to a range of lower bitrates. In order to investigate our model performance in higher bitrates, we use VTM (VTM) as the I-frame model and the performance of our model and DVC Pro using VTM are reported in Figure 9. We could see that using VTM instead of cheng2020 as the I-frame model brings both our proposed model and DVC Pro to higher bitrates and similar conclusions can be made. Our proposed model can still outperform previous state-of-the-art models in higher bitrates and the multi-head decoders can bring signiﬁcant bitrate savings regardless of the I-frame model used or the range of bitrates

Original: [Figure] 
Revised: [Figure] A.

Revised: V ISUALIZATION OF U NSUPERVISED P REDICTIVE U NCERTAINTY

Original: [Figure]
Revised: [Figure]

Revised: A. Q UALITATIVE R ESULTS OF D ECODED F RAMES

Original: [Figure]
Revised: In this section, we show the qualitative examples of decoded frames for subjective comparison. We compare the performance of our proposed model with the baseline DVC Pro to show the strength and weakness of multi-head decoders. In sequence BQTerrace from HEVC B, our proposed model achieves a bitrate saving of 20.5% over DVC Pro. Sequence 20 from MCL-JCV is the only sequence where our model failed to outperform DVC Pro with a bitrate increase of 2.2%.

Original:  D ETAILED A BLATION S TUDY R ESULTS
Revised: A. D ETAILED A BLATION S TUDY R ESULTS

Original: To show the effectiveness of various proposed modules and to compare between models with different number of decoder heads, we train multiple models on the Vimeo-90K dataset with the fast training strategy and test the models on the ﬁrst 30 frames in each HEVC sequence.
Revised: To show the effectiveness of various proposed modules and to compare between models with different number of decoder heads we train multiple models on the Vimeo-90K dataset with the fast training strategy and test the models on the ﬁrst 30 frames in each HEVC sequence.

Original: We measure the model performance with BD rates and the quantitative results are reported in Table 2 and Table 3 respectively.
Revised: We measure the model performance with BD rates and the quantitative results are reported in Table 2 and Table 3 respectively.

Original: Setting [Table]
Revised: [Table] [Table]

Original: MH-MV + EA-L
Revised: h mv = h res =

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: h mv =

Original: MH-MV + MH-Res + EA-L
Revised: h res = [Table] [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: MH-MV + MH-Res + EA-L + FGSM
Revised: h mv = h res = [Table] [Table]

Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Revised: h mv = h res =-8.

Revised: h mv =

Revised: h res =-9.

Original: [Table]
Revised: h mv = h res =-10. h mv = h res =-10.