Original: Generative Video Compression as Hierarchical Variational Inference
Revised: Generative Video Compression as Hierarchical Variational Inference

Original: Anonymous Authors

Original: Anonymous Institution

Original: Abstract
Revised: Abstract

Original: Recent work by Marino et al.
Revised: Recent work by Marino et al.

Original: (2020) showed improved performance in sequential density estimation by combining masked autoregressive ﬂows with hierarchical latent variable models.
Revised: (2020) showed improved performance in sequential density estimation by combining masked autoregressive ﬂows with hierarchical latent variable models.

Original: We draw a connection between such autoregressive generative models and the task of lossy video compression.
Revised: We draw a connection between such autoregressive generative models and the task of lossy video compression.

Original: Speciﬁcally, we view recent neural video compression methods (Lu et al., 2019;
Revised: Speciﬁcally, we view recent neural video compression methods (Lu et al., 2019;

Original: Yang et al., 2020b; Agustsson et al., 2020) as instances of a generalized stochastic temporal autoregressive transform, and propose avenues for enhancement based on this insight.
Revised: Yang et al., 2020b; Agustsson et al., 2020) as instances of a generalized stochastic temporal autoregressive transform, and propose avenues for enhancement based on this insight.

Original: Comprehensive evaluations on large-scale video data show improved rate-distortion performance over both state-of-the-art neural and conventional video compression methods.
Revised: Comprehensive evaluations on large-scale video data show improved rate-distortion performance over both state-of-the-art neural and conventional video compression methods.

Original: 1. Introduction
Revised: 1. Introduction

Original: Recent advances in deep generative modeling have enabled a surge in applications, including learning-based compression.
Revised: Recent advances in deep generative modeling have enabled a surge in applications, including learning-based compression.

Original: Generative models have already demonstrated empirical improvements in image compression, outperforming classical codecs (Minnen et al., 2018;
Revised: Generative models have already demonstrated empirical improvements in image compression, outperforming classical codecs (Minnen et al., 2018;

Original: Yang et al., 2020d), such as BPG (Bellard, 2014).
Revised: Yang et al., 2020d), such as BPG (Bellard, 2014).

Original: In contrast, the less developed area of neural video compression remains challenging due to complex temporal dependencies operating at multiple scales.
Revised: In contrast, the less developed area of neural video compression remains challenging due to complex temporal dependencies operating at multiple scales.

Original: Nevertheless, recent neural video codecs have shown promising performance gains (Agustsson et al., 2020), in some cases on par with current hand-designed, classical codecs, e.g., HEVC.
Revised: Nevertheless, recent neural video codecs have shown promising performance gains (Agustsson et al., 2020), in some cases on par with current hand-designed, classical codecs, e.g., HEVC.

Original: Source compression fundamentally involves decorrelation, i.e., transforming input data into white noise distributions that can be easily modeled and entropy-coded.
Revised: Source compression fundamentally involves decorrelation, i.e., transforming input data into white noise distributions that can be easily modeled and entropy-coded.

Original: Thus, improving a model’s capability to decorrelate data automatically improves its compression performance.
Revised: Thus, improving a model’s capability to decorrelate data automatically improves its compression performance.

Original: Likewise, we can improve the associated entropy model (i.e., the model’s prior) to capture any remaining dependencies.
Revised: Likewise, we can improve the associated entropy model (i.e., the model’s prior) to capture any remaining dependencies.

Original: Just as compression techniques attempt to remove structure, generative models attempt to model structure.
Revised: Just as compression techniques attempt to remove structure, generative models attempt to model structure.

Original: One family of models, autoregressive ﬂows, maps between less structured distributions, e.g., uncorrelated noise, and more structured distributions, e.g., images or video (Dinh et al., 2014, 2016).
Revised: One family of models, autoregressive ﬂows, maps between less structured distributions, e.g., uncorrelated noise, and more structured distributions, e.g., images or video (Dinh et al., 2014, 2016).

Original: The inverse mapping can remove dependencies in the data, making it more amenable for compression.
Revised: The inverse mapping can remove dependencies in the data, making it more amenable for compression.

Original: This paper draws on recent insights in hierarchical sequential latent variable models with autoregressive ﬂows (Marino et al., 2020).
Revised: This paper draws on recent insights in hierarchical sequential latent variable models with autoregressive ﬂows (Marino et al., 2020).

Original: In particular, we identify connections between this family of models and recently proposed neural video codecs based on motion estimation (Lu et al., 2019;
Revised: In particular, we identify connections between this family of models and recently proposed neural video codecs based on motion estimation (Lu et al., 2019;

Original: Agustsson et al., 2020).
Revised: Agustsson et al., 2020).

Original: By interpreting this technique as instantiation of a type of autoregressive ﬂow transform, we propose various alternatives and improvements based on insights from generative modeling.
Revised: By interpreting this technique as instantiation of a type of autoregressive ﬂow transform, we propose various alternatives and improvements based on insights from generative modeling.

Original: Our main contributions are as follows:
Revised: Our main contributions are as follows:

Original: A new framework.
Revised: A new framework.

Original: We interpret existing video compression methods through the more general framework of generative modeling, variational inference, and autoregressive ﬂows, allowing us to readily investigate extensions and ablations.
Revised: We interpret existing video compression methods through the more general framework of generative modeling, variational inference, and autoregressive ﬂows, allowing us to readily investigate extensions and ablations.

Original: In particular, we compare fully data-driven approaches with motion-estimation-based neural compression schemes.
Revised: In particular, we compare fully data-driven approaches with motion-estimation-based neural compression schemes.

Original: This framework also provides directions for future work.
Revised: This framework also provides directions for future work.

Original: A new model.
Revised: A new model.

Original: Our main proposed model is an improved version of Scale-Space Flow (SSF) (Agustsson et al., 2020) model, by augmenting a shift transform by scalethen-shift inspired by performance gains in extending NICE (Dinh et al., 2014) to RealNVP (Dinh et al., 2016).
Revised: Our main proposed model is an improved version of Scale-Space Flow (SSF) (Agustsson et al., 2020) model, by augmenting a shift transform by scalethen-shift inspired by performance gains in extending NICE (Dinh et al., 2014) to RealNVP (Dinh et al., 2016).

Original: Structured priors can improve the performance further.
Revised: Structured priors can improve the performance further.

Original: A new dataset.
Revised: A new dataset.

Original: The neural video compression community is lacking large, highresolution benchmark datasets.
Revised: The neural video compression community is lacking large, highresolution benchmark datasets.

Original: While we performed extensive experiments on the publicly available Vimeo-90k dataset (Xue et al., 2019), we also collected and utilized a larger dataset, YouTube-NT 1 , which is used for our ablation study.
Revised: While we performed extensive experiments on the publicly available Vimeo-90k dataset (Xue et al., 2019), we also collected and utilized a larger dataset, YouTube-NT 1 , which is used for our ablation study.

Original: Details can be found in Appendix D.
Revised: Details can be found in Appendix D.

Original: 2. Video Compression through Deep Autoregressive Modeling
Revised: 2. Video Compression through Deep Autoregressive Modeling

Original: We identify commonalities between hierarchical autoregressive ﬂow models (Marino et al., 2020) and state-of-the-art neural video compression architectures (Agustsson et al., 2020), and will use this viewpoint to propose improvements on existing models.
Revised: We identify commonalities between hierarchical autoregressive ﬂow models (Marino et al., 2020) and state-of-the-art neural video compression architectures (Agustsson et al., 2020), and will use this viewpoint to propose improvements on existing models.

Original: 2.1. Background
Revised: 2.1. Background

Original: We ﬁrst review VAE-based compression schemes (Ball´e et al., 2017; Theis et al., 2017) and formulate existing video codecs in this framework; we then review autoregressive ﬂows.
Revised: We ﬁrst review VAE-based compression schemes (Ball´e et al., 2017; Theis et al., 2017) and formulate existing video codecs in this framework; we then review autoregressive ﬂows.

Original: Generative Modeling and Source Compression.
Revised: Generative Modeling and Source Compression.

Original: Let x 1: T ∈ R T × D be a sequence of video frames.
Revised: Let x 1: T ∈ R T × D be a sequence of video frames.

Original: Lossy compression seeks to ﬁnd a compact description of x 1: T that simultaneously minimizes the description length R and information loss D .
Revised: Lossy compression seeks to ﬁnd a compact description of x 1: T that simultaneously minimizes the description length R and information loss D .

Original: The distortion D measures how much reconstruction error accrues due to encoding x 1: T into a latent representation ¯ z 1: T and subsequently decoding it back to ˆx 1: T , while R measures the bit rate (ﬁle size).
Revised: The distortion D measures how much reconstruction error accrues due to encoding x 1: T into a latent representation ¯ z 1: T and subsequently decoding it back to ˆx 1: T , while R measures the bit rate (ﬁle size).

Original: In learned compression methods (Ball´e et al., 2017; Theis et al., 2017), the above process is parameterized by ﬂexible functions f (“encoder”) and g (“decoder”) that map between the input video and its latent representation: ¯z 1: T = f ( x 1: T ) , ˆx 1: T = g ( ¯z 1: T ), and minimize a rate-distortion loss with hyperparameter β > 0:
Revised: In learned compression methods (Ball´e et al., 2017; Theis et al., 2017), the above process is parameterized by ﬂexible functions f (“encoder”) and g (“decoder”) that map between the input video and its latent representation: ¯z 1: T = f ( x 1: T ) , ˆx 1: T = g ( ¯z 1: T ), and minimize a rate-distortion loss with hyperparameter β > 0:

Original: [Equation]
Revised: [Equation]

Original: We adopt the end-to-end compression approach of Ball´e et al.
Revised: We adopt the end-to-end compression approach of Ball´e et al.

Original: ; Lombardo et al.
Revised: ; Lombardo et al.

Original: (2019), which approximates the rounding operations ( (cid:98)·(cid:101) ) by uniform noise injection to enable gradient-based optimization during training.
Revised: (2019), which approximates the rounding operations ( (cid:98)·(cid:101) ) by uniform noise injection to enable gradient-based optimization during training.

Original: With an appropriate choice of probability model p ( z 1: T ), the approximated version of the above R-D (rate-distortion) objective then corresponds to the VAE objective:
Revised: With an appropriate choice of probability model p ( z 1: T ), the approximated version of the above R-D (rate-distortion) objective then corresponds to the VAE objective:

Original: [Equation]
Revised: [Equation]

Original: In this model, the likelihood p ( x 1: T | z 1: T ) follows a Gaussian distribution with mean ˆx 1: T = g ( z 1: T ) and diagonal covariance β 2log 2 I , the approximate posterior q is chosen to be a unitwidth uniform (thus zero-diﬀerential-entropy) distribution whose mean ¯z 1: T is predicted by an amortized inference network f .
Revised: In this model, the likelihood p ( x 1: T | z 1: T ) follows a Gaussian distribution with mean ˆx 1: T = g ( z 1: T ) and diagonal covariance β 2log 2 I , the approximate posterior q is chosen to be a unitwidth uniform (thus zero-diﬀerential-entropy) distribution whose mean ¯z 1: T is predicted by an amortized inference network f .

Original: The prior density p ( z 1: T ) interpolates its discretized version, so that it measures the code length of discretized ¯z 1: T after entropy-coding.
Revised: The prior density p ( z 1: T ) interpolates its discretized version, so that it measures the code length of discretized ¯z 1: T after entropy-coding.

Original: Masked Autoregressive Flow (MAF).
Revised: Masked Autoregressive Flow (MAF).

Original: As a ﬁnal component in neural sequence modeling, we discuss MAF (Papamakarios et al., 2017), which models the joint distribution of a sequence p ( x 1: T ) in terms of a simpler distribution of its underlying noise variables y 1: T through the following autoregressive transform and its inverse:
Revised: As a ﬁnal component in neural sequence modeling, we discuss MAF (Papamakarios et al., 2017), which models the joint distribution of a sequence p ( x 1: T ) in terms of a simpler distribution of its underlying noise variables y 1: T through the following autoregressive transform and its inverse:

Original: [Equation]
Revised: [Equation]

Original: The noise variable y t usually comes from a standard normal distribution.
Revised: The noise variable y t usually comes from a standard normal distribution.

Original: While the forward
Revised: While the forward

Original: MAF transforms a sequence of standard normal noises into a data sequence, the inverse ﬂow “whitens” the data sequence and removes temporal correlations.
Revised: MAF transforms a sequence of standard normal noises into a data sequence, the inverse ﬂow “whitens” the data sequence and removes temporal correlations.

Original: Due to its invertible nature, MAF allows for exact likelihood computations, but as we will explain in Section 2.3, we will not exploit this aspect in compression but rather draw on its expressiveness in modeling conditional likelihoods.
Revised: Due to its invertible nature, MAF allows for exact likelihood computations, but as we will explain in Section 2.3, we will not exploit this aspect in compression but rather draw on its expressiveness in modeling conditional likelihoods.

Original: 2.2. A General Framework for Generative Video Coding
Revised: 2.2. A General Framework for Generative Video Coding

Original: We now describe a general framework that captures several existing low-latency (see Appendix B) neural compression methods as speciﬁc instances and gives rise to the exploration of new models.
Revised: We now describe a general framework that captures several existing low-latency (see Appendix B) neural compression methods as speciﬁc instances and gives rise to the exploration of new models.

Original: To this end, we combine latent variable models with autoregressive ﬂows into a joint framework.
Revised: To this end, we combine latent variable models with autoregressive ﬂows into a joint framework.

Original: We consider a sequential decoding procedure of the following form:
Revised: We consider a sequential decoding procedure of the following form:

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: 3 resembles the deﬁnition of the MAF in Eq. 2, but augments this transform with two sets of latent variables w t , v t ∼ p ( w t , v t ).
Revised: 3 resembles the deﬁnition of the MAF in Eq. 2, but augments this transform with two sets of latent variables w t , v t ∼ p ( w t , v t ).

Original: Above, h µ and h σ are functions that transform the previous reconstructed data frame ˆx t − 1 along with w t into a shift and scale parameter, respectively.
Revised: Above, h µ and h σ are functions that transform the previous reconstructed data frame ˆx t − 1 along with w t into a shift and scale parameter, respectively.

Original: The function g v ( v t , w t ) converts these latent variables into a noise variable that encodes residuals with respect to the mean next-frame prediction h µ ( ˆx t − 1 , w t ).
Revised: The function g v ( v t , w t ) converts these latent variables into a noise variable that encodes residuals with respect to the mean next-frame prediction h µ ( ˆx t − 1 , w t ).

Original: Our approach is inspired by Marino et al.
Revised: Our approach is inspired by Marino et al.

Original: (2020) who analyzed a restricted version of the model in Eq. 3, aiming to hybridize autoregressive ﬂows and sequential latent variable models for video prediction.
Revised: (2020) who analyzed a restricted version of the model in Eq. 3, aiming to hybridize autoregressive ﬂows and sequential latent variable models for video prediction.

Original: In contrast to Eq. 3, their model involved deterministic transforms as well as residual noise that came from a sequential VAE.
Revised: In contrast to Eq. 3, their model involved deterministic transforms as well as residual noise that came from a sequential VAE.

Original: 2.3. Example Models and Extensions
Revised: 2.3. Example Models and Extensions

Original: Next, we show that the general framework expressed by Eq.
Revised: Next, we show that the general framework expressed by Eq.

Original: 3 captures a variety of stateof-the-art neural video compression schemes and gives rise to extensions and new models.
Revised: 3 captures a variety of stateof-the-art neural video compression schemes and gives rise to extensions and new models.

Original: Temporal Autoregressive Transform (TAT).
Revised: Temporal Autoregressive Transform (TAT).

Original: The ﬁrst special case among the class of models that are captured by Eq.
Revised: The ﬁrst special case among the class of models that are captured by Eq.

Original: 3 is the autoregressive neural video compression model by Yang et al.
Revised: 3 is the autoregressive neural video compression model by Yang et al.

Original: (2020b), which we denote as temporal autoregressive transform (TAT).
Revised: (2020b), which we denote as temporal autoregressive transform (TAT).

Original: Shown
Revised: Shown

Original: [Figure]
Revised: [Figure]

Original: in Figure 1(a), the decoder g implements a deterministic scale-shift autoregressive transform of decoded noise y t ,
Revised: in Figure 1(a), the decoder g implements a deterministic scale-shift autoregressive transform of decoded noise y t ,

Original: [Equation]
Revised: [Equation]

Original: The encoder f inverts the transform to decorrelate the input frame x t into ¯y t and encodes the result as ¯z t = f ( x t , ˆx t − 1 ) =
Revised: The encoder f inverts the transform to decorrelate the input frame x t into ¯y t and encodes the result as ¯z t = f ( x t , ˆx t − 1 ) =

Original: f z ( ¯y t ), where ¯y t = x t − h µ ( ˆx t − 1 ) h
Revised: f z ( ¯y t ), where ¯y t = x t − h µ ( ˆx t − 1 ) h

Original: σ ( ˆx t − 1 ) .
Revised: σ ( ˆx t − 1 ) .

Original: The shift h µ and scale h σ transforms are parameterized by neural networks, f z is a convolutional neural network (CNN), and g z is a deconvolutional neural network (DNN) that approximately inverts f z .
Revised: The shift h µ and scale h σ transforms are parameterized by neural networks, f z is a convolutional neural network (CNN), and g z is a deconvolutional neural network (DNN) that approximately inverts f z .

Original: DVC (Lu et al., 2019) and Scale-Space Flow (SSF, Agustsson et al.
Revised: DVC (Lu et al., 2019) and Scale-Space Flow (SSF, Agustsson et al.

Original: The second class of models captured by Eq.
Revised: The second class of models captured by Eq.

Original: 3 belong to the conventional video compression framework based on predictive coding (Cutler, 1952; Wiegand et al., 2003; Sullivan et al., 2012); both models make use of two sets of latent variables z 1: T = { w 1: T , v 1: T } to capture diﬀerent aspects of information being compressed, where w captures motion estimation used in warping prediction, and v helps capture residual error not predicted by warping.
Revised: 3 belong to the conventional video compression framework based on predictive coding (Cutler, 1952; Wiegand et al., 2003; Sullivan et al., 2012); both models make use of two sets of latent variables z 1: T = { w 1: T , v 1: T } to capture diﬀerent aspects of information being compressed, where w captures motion estimation used in warping prediction, and v helps capture residual error not predicted by warping.

Original: Like most classical approaches to video compression by predictive coding, the reconstruction transform in the above models has the form of a prediction shifted by residual error, without scaling factor h σ compared to the autoregressive transform in Eq.
Revised: Like most classical approaches to video compression by predictive coding, the reconstruction transform in the above models has the form of a prediction shifted by residual error, without scaling factor h σ compared to the autoregressive transform in Eq.

Original: [Equation]
Revised: [Equation]

Original: where g w and g v are DNNs, o t :
Revised: where g w and g v are DNNs, o t :

Original: = g w ( w t ) has the interpretation of an estimated optical ﬂow (motion) ﬁeld, h warp is the computer vision technique of warping, and the residual r t :=
Revised: = g w ( w t ) has the interpretation of an estimated optical ﬂow (motion) ﬁeld, h warp is the computer vision technique of warping, and the residual r t :=

Original: g v ( v t , w t ) =
Revised: g v ( v t , w t ) =

Original: ˆx t − h warp ( ˆx t − 1 , o t ) represents the prediction error unaccounted for by warping.
Revised: ˆx t − h warp ( ˆx t − 1 , o t ) represents the prediction error unaccounted for by warping.

Original: Proposed: models based on Stochastic Temporal Autoregressive Transform.
Revised: Proposed: models based on Stochastic Temporal Autoregressive Transform.

Original: Finally, we consider the most general models as described by the stochastic autoregressive transform in Eq. 3, shown in Figure 1(c).
Revised: Finally, we consider the most general models as described by the stochastic autoregressive transform in Eq. 3, shown in Figure 1(c).

Original: We study two main variants, categorized by how they implement h µ and h σ :
Revised: We study two main variants, categorized by how they implement h µ and h σ :

Original: STAT uses DNNs for h µ and h σ as in (Yang et al., 2020b), but complements it with the latent variable w t that characterizes the transform.
Revised: STAT uses DNNs for h µ and h σ as in (Yang et al., 2020b), but complements it with the latent variable w t that characterizes the transform.

Original: In principle, more ﬂexible transform functions h µ and h σ should lead to better rate-distortion results; however, we ﬁnd the following variant more performant and parameter-eﬃcient in practice: 2.
Revised: In principle, more ﬂexible transform functions h µ and h σ should lead to better rate-distortion results; however, we ﬁnd the following variant more performant and parameter-eﬃcient in practice: 2.

Original: STAT-SSF : a less data-driven variant of the above that still uses scale-space warping (Agustsson et al., 2020) in the shift transform,
Revised: STAT-SSF : a less data-driven variant of the above that still uses scale-space warping (Agustsson et al., 2020) in the shift transform,

Original: i
Revised: i

Original: e., h µ ( ˆx t − 1 , w t ) =
Revised: e., h µ ( ˆx t − 1 , w t ) =

Original: h warp ( ˆx t − 1 , g w ( w t )).
Revised: h warp ( ˆx t − 1 , g w ( w t )).

Original: This can also be seen as an extended version of the SSF model, whose shift transform h µ is preceded by a new scale transform h σ .
Revised: This can also be seen as an extended version of the SSF model, whose shift transform h µ is preceded by a new scale transform h σ .

Original: Besides increasing the ﬂexibility of the reconstruction transform (hence the likelihood model for x t ), we also consider improving the topmost generative hierarchy in the form of a more expressive latent prior p ( z 1: T ), corresponding to improving the entropy model for compression.
Revised: Besides increasing the ﬂexibility of the reconstruction transform (hence the likelihood model for x t ), we also consider improving the topmost generative hierarchy in the form of a more expressive latent prior p ( z 1: T ), corresponding to improving the entropy model for compression.

Original: Speciﬁcally, we note that the prior in the SSF model assumes the factorization p ( w t , v t ) =
Revised: Speciﬁcally, we note that the prior in the SSF model assumes the factorization p ( w t , v t ) =

Original: p ( w t ) p ( v t ), which can be restrictive.
Revised: p ( w t ) p ( v t ), which can be restrictive.

Original: We propose a structured prior by introducing conditional dependence between w t and v t , so that p ( w t , v t ) = p ( w t ) p ( v t | w t ).
Revised: We propose a structured prior by introducing conditional dependence between w t and v t , so that p ( w t , v t ) = p ( w t ) p ( v t | w t ).

Original: This results in variants of the above models, STAT-SP and STAT-SSF-SP , where the structured prior is applied on top of the proposed STAT and STAT-SSF models.
Revised: This results in variants of the above models, STAT-SP and STAT-SSF-SP , where the structured prior is applied on top of the proposed STAT and STAT-SSF models.

Original: 3. Experiments
Revised: 3. Experiments

Original: In this section, we present performance comparisons of our model with neural and classical codecs across multiple evaluation datasets.
Revised: In this section, we present performance comparisons of our model with neural and classical codecs across multiple evaluation datasets.

Original: Ablations study about our new dataset, diﬀerent architecture components and eﬃciency optimization is elucidated in Appendix D and H.
Revised: Ablations study about our new dataset, diﬀerent architecture components and eﬃciency optimization is elucidated in Appendix D and H.

Original: 3.1. Training Datasets
Revised: 3.1. Training Datasets

Original: Vimeo-90k (Xue et al., 2019) consists of 90,000 clips of 7 frames at 448x256 resolution collected from vimeo.com .
Revised: Vimeo-90k (Xue et al., 2019) consists of 90,000 clips of 7 frames at 448x256 resolution collected from vimeo.com .

Original: As this dataset has been used in previous works (Lu et al., 2019;
Revised: As this dataset has been used in previous works (Lu et al., 2019;

Original: Yang et al., 2020a; Liu et al., 2019), it provides a benchmark for comparing models.
Revised: Yang et al., 2020a; Liu et al., 2019), it provides a benchmark for comparing models.

Original: 3.2. Evaluation Scheme
Revised: 3.2. Evaluation Scheme

Original: We evaluate compression performance on UVG (Mercat et al., 2020) and MCL JCV (Wang et al., 2016) datasets, which both consist of YUV420 format RAW videos.
Revised: We evaluate compression performance on UVG (Mercat et al., 2020) and MCL JCV (Wang et al., 2016) datasets, which both consist of YUV420 format RAW videos.

Original: UVG is widely used for testing the HEVC codec and contains seven 1080p videos at 120fps with smooth and mild motions or stable camera moving.
Revised: UVG is widely used for testing the HEVC codec and contains seven 1080p videos at 120fps with smooth and mild motions or stable camera moving.

Original: MCL JCV contains thirty 1080p videos at 30fps, which are generally more diverse, with a higher degree of motion and camera movement.
Revised: MCL JCV contains thirty 1080p videos at 30fps, which are generally more diverse, with a higher degree of motion and camera movement.

Original: We compute the bit rate (bits-per-pixel, BPP) and the reconstruction quality as measured in PSNR, averaged across all frames.
Revised: We compute the bit rate (bits-per-pixel, BPP) and the reconstruction quality as measured in PSNR, averaged across all frames.

Original: We note that PSNR is a more challenging metric than MS-SSIM (Wang et al., 2003) for learned codecs (Lu et al., 2019;
Revised: We note that PSNR is a more challenging metric than MS-SSIM (Wang et al., 2003) for learned codecs (Lu et al., 2019;

Original: Agustsson et al., 2020; Habibian et al., 2019; Yang et al., 2020a,c).
Revised: Agustsson et al., 2020; Habibian et al., 2019; Yang et al., 2020a,c).

Original: [Figure]
Revised: [Figure]

Revised: STAT-SSF (proposed) achieves the best performance.

Original: 3.3. Baseline Analysis
Revised: 3.3. Baseline Analysis

Original: To provide a benchmark comparison with baseline models, listed in Table 1, we primarily focus on the Vimeo-90k (Xue et al., 2019) training dataset.
Revised: To provide a benchmark comparison with baseline models, listed in Table 1, we primarily focus on the Vimeo-90k (Xue et al., 2019) training dataset.

Original: In Figure 2( a ), we compare our proposed models (STAT-SSF, STAT-SSF-SP) with previous neural codecs and the classical codec, HEVC, on the UVG evaluation dataset.
Revised: In Figure 2( a ), we compare our proposed models (STAT-SSF, STAT-SSF-SP) with previous neural codecs and the classical codec, HEVC, on the UVG evaluation dataset.

Original: Our models provide superior performance at bitrates ≥ 0 .
Revised: Our models provide superior performance at bitrates ≥ 0 .

Original: 12 bits/pixel, outperforming state-of-the-art models SSF (Agustsson et al., 2020), as well as DVC (Lu et al., 2019).
Revised: 12 bits/pixel, outperforming state-of-the-art models SSF (Agustsson et al., 2020), as well as DVC (Lu et al., 2019).

Original: Finally, we note that, as expected, our proposed
Revised: Finally, we note that, as expected, our proposed

Original: STAT model improves over TAT (Yang et al., 2020b) by adding stochasticity to the autoregressive transform.
Revised: STAT model improves over TAT (Yang et al., 2020b) by adding stochasticity to the autoregressive transform.

Original: The diversity of the MCL JCV dataset provides a more challenging evaluation benchmark.
Revised: The diversity of the MCL JCV dataset provides a more challenging evaluation benchmark.

Original: However, our STAT-SSF model still maintains a competitive edge over other baseline methods.
Revised: However, our STAT-SSF model still maintains a competitive edge over other baseline methods.

Original: Our structured prior model, STAT-SSF-SP, provides further performance improvements over STAT-SSF.
Revised: Our structured prior model, STAT-SSF-SP, provides further performance improvements over STAT-SSF.

Original: This suggests that our structured prior may further help capture intensive motion and camera moving information.
Revised: This suggests that our structured prior may further help capture intensive motion and camera moving information.

Original: 4. Discussion
Revised: 4. Discussion

Original: We provide a unifying perspective on sequential video compression and temporal autoregressive ﬂows (Marino et al., 2020), and elucidate the relationship between the two in terms of their underlying generative hierarchy.
Revised: We provide a unifying perspective on sequential video compression and temporal autoregressive ﬂows (Marino et al., 2020), and elucidate the relationship between the two in terms of their underlying generative hierarchy.

Original: From this perspective, we consider several video compression methods, particularly a state-of-the-art method Scale-Space-Flow (Agustsson et al., 2020), as instantiations of a more general stochastic temporal autoregressive transform.
Revised: From this perspective, we consider several video compression methods, particularly a state-of-the-art method Scale-Space-Flow (Agustsson et al., 2020), as instantiations of a more general stochastic temporal autoregressive transform.

Original: Further, we provide a new high-resolution video dataset.
Revised: Further, we provide a new high-resolution video dataset.

Original: Together, we hope that this new perspective and dataset will drive further progress in the nascent yet highly impactful ﬁeld of learned video compression.
Revised: Together, we hope that this new perspective and dataset will drive further progress in the nascent yet highly impactful ﬁeld of learned video compression.

Original: Appendix A. Qualitative Result
Revised: Appendix A. Qualitative Result

Original: [Figure]
Revised: [Figure]

Original:  ) can be seen as a gating mechanism in the  reconstruction ˆx t
Revised: Two methods in comparison, STAT-SSF (proposed) and SSF (Agustsson et al., 2020), have comparable reconstruction quality to STAT-SSFSP but higher bit-rate; the (BPP, PSNR) for STAT-SSF-SP, STAT-SSF, and SSF are (0.046, 36.97), (0.053, 36.94), and (0.075, 36.97), respectively. In this example, the warping prediction ˆ µ t = h µ ( ˆx t − 1 , (cid:98) ¯w t (cid:101) ) incurs large error around the dog’s moving contour, but models the mostly static background well, with the residual latents (cid:98) ¯v t (cid:101) taking up an order of magnitude higher bit-rate than (cid:98) ¯w t (cid:101) in the three methods. The proposed scale parameter ˆ σ t gives the model extra ﬂexibility when combining the noise ˆy t (decoded from ( (cid:98) ¯v t (cid:101) , (cid:98) ¯w t (cid:101) )) with the warping prediction ˆ µ t (decoded from (cid:98) ¯w t (cid:101) only) to form the reconstruction ˆx t

Original: = ˆ µ t + ˆ σ t (cid:12) ˆy t , switching oﬀ the  contribution from the noise ˆy t  where it is costly (around the dog) and instead relying on the warping prediction ˆ µ t decoded from the cheaper (cid:98) ¯w t (cid:101) .
Revised: = ˆ µ t + ˆ σ t (cid:12) ˆy t : the scale ˆ σ t downweights contribution from the noise ˆy t in the foreground where it is very costly, and reduces the residual bit-rate R ( (cid:98) ¯v t (cid:101) ) (and thus the overall bit-rate) compared to STAT-SSF and SSF (with similar reconstruction quality), as illustrated in the third and fourth ﬁgures in the top row.

Original: Appendix B. Low-Latency Sequential Compression
Revised: Appendix B. Low-Latency Sequential Compression

Original: We specialize Eq.
Revised: We specialize Eq.

Original: 1 to make it suitable for low-latency video compression, widely used in both conventional and recent neural codecs (Rippel et al., 2019; Agustsson et al., 2020).
Revised: 1 to make it suitable for low-latency video compression, widely used in both conventional and recent neural codecs (Rippel et al., 2019; Agustsson et al., 2020).

Original: To this end, we encode and decode individual frames x t in sequence.
Revised: To this end, we encode and decode individual frames x t in sequence.

Original: Given the ground truth current frame x t and the previously reconstructed frames ˆx <t , the encoder is restricted to be  of the form ¯z t = f ( x t , ˆx <t ), and similarly the decoder computes reconstruction sequentially based on previous reconstructions and the current encoding, ˆx t 
Revised: Given the ground truth current frame x t and the previously reconstructed frames ˆx <t , the encoder is restricted to be [Figure] of the form ¯z t = f ( x t , ˆx <t ), and similarly the decoder computes reconstruction sequentially based on previous reconstructions and the current encoding, ˆx t =

Original: = g ( ˆx <t , (cid:98) ¯z t (cid:101) )).
Revised: g ( ˆx <t , (cid:98) ¯z t (cid:101) )).

Original: Existing codecs usually condition on a single reconstructed frame, substituting ˆx <t by ˆx t − 1 in favour of eﬃciency.
Revised: Existing codecs usually condition on a single reconstructed frame, substituting ˆx <t by ˆx t − 1 in favour of eﬃciency.

Original: In the language of variational inference, the sequential encoder corresponds to a variational posterior of the form q ( z t | x t , z <t ), i.e., ﬁltering, and the sequential decoder corresponds to the likelihood p ( x t | z ≤ t ) =N ( ˆx t ,
Revised: In the language of variational inference, the sequential encoder corresponds to a variational posterior of the form q ( z t | x t , z <t ), i.e., ﬁltering, and the sequential decoder corresponds to the likelihood p ( x t | z ≤ t ) =

Original: [Table] 2log 2 I ); in both distributions, the probabilistic conditioning on z <t is based on the observation that ˆx t − 1 is a deterministic function of z <t .
Revised: N ( ˆx t , β 2log 2 I ); in both distributions, the probabilistic conditioning on z <t is based on the observation that ˆx t − 1 is a deterministic function of z <t .

Original: As we show, all sequential compression approaches considered in this work follow this paradigm [Table] and the form of the reconstruction transform ˆx determines the lowest hierarchy of the corresponding generative process of video x .
Revised: As we show, all sequential compression approaches considered in this work follow this paradigm, and the form of the reconstruction transform ˆx determines the lowest hier- archy of the corresponding generative process of video x .

Original: We note that this scheme does not require future frames ˆx >t to help compress frame ˆx t and this is so-called low-latency , which is widely used in some real-time applications such as live streaming
Revised: We note that this scheme does not require future frames ˆx >t to help compress frame ˆx t and this is so-called low-latency , which is widely used in some real-time applications such as live streaming

Original: [Table]
Revised: [Table]

Original: Appendix C. Comparison Table
Revised: Appendix C. Comparison Table

Original: [Table]
Revised: [Table]

Original: Appendix D. New Dataset
Revised: Appendix D. New Dataset

Original: YouTube-NT .
Revised: YouTube-NT .

Original: This is our new dataset.
Revised: This is our new dataset.

Original: We collected 8,000 nature videos and movie/videogame trailers from youtube.com and processed them into 300k high-resolution (720p) clips, which we refer to as YouTube-NT.
Revised: We collected 8,000 nature videos and movie/videogame trailers from youtube.com and processed them into 300k high-resolution (720p) clips, which we refer to as YouTube-NT.

Original: In contrast to existing datasets (Carreira and Zisserman, 2017; Xue et al., 2019), we provide YouTube-NT in the form of customizable scripts to enable future compression research.
Revised: In contrast to existing datasets (Carreira and Zisserman, 2017; Xue et al., 2019), we provide YouTube-NT in the form of customizable scripts to enable future compression research.

Original: Table 2 compares the current version of YouTube-NT with
Revised: Table 2 compares the current version of YouTube-NT with

Original: Vimeo-90k (Xue et al., 2019) and with Google’s proprietary training dataset (Agustsson et al., 2020).
Revised: Vimeo-90k (Xue et al., 2019) and with Google’s proprietary training dataset (Agustsson et al., 2020).

Original: In Figure 4( b ), we display the evaluation performance of the SSF model architecture after training on each dataset.
Revised: In Figure 5( b ), we display the evaluation performance of the SSF model architecture after training on each dataset.

Original: [Table]
Revised: [Table]

Original: Appendix E. Ablation
Revised: Appendix E. Ablation

Original: Using the baseline SSF (Agustsson et al., 2020) model and YouTube-NT training dataset, we demonstrate the improvements of our proposed components, stochastic temporal autoregressive transform (STAT) and structured prior (SP), evaluated on UVG.
Revised: Using the baseline SSF (Agustsson et al., 2020) model and YouTube-NT training dataset, we demonstrate the improvements of our proposed components, stochastic temporal autoregressive transform (STAT) and structured prior (SP), evaluated on UVG.

Original: As shown in
Revised: As shown in

Original: Figure 4( a ), STAT improves performance to a greater degree than SP, consistent with the results in Section 3.3 Figure 2( a ).[Figure]
Revised: Figure 5( a ), STAT improves performance to a greater degree than SP, consistent with the results in Section 3.3 Figure 2( a ).

Original: To quantify the eﬀect of the training dataset on performance, we compare performance on UVG for the SSF model architecture after training on Vimeo-90k (Xue et al., 2019) and
Revised: To quantify the eﬀect of the training dataset on performance, we compare performance on UVG for the SSF model architecture after training on Vimeo-90k (Xue et al., 2019) and

Original: YouTube-NT.
Revised: YouTube-NT.

Original: We also compare with the reported results from Agustsson et al.
Revised: We also compare with the reported results from Agustsson et al.

Original: (2020), trained on a larger proprietary dataset.
Revised: [Figure]

Original: This is shown in Figure 4( b ), where we see that training on YouTube-NT improves evaluation performance over Vimeo-90k, in some cases bridging the gap with the performance from the larger proprietary training dataset of Agustsson et al.
Revised: This is shown in Figure 5( b ), where we see that training on YouTube-NT improves evaluation performance over Vimeo-90k, in some cases bridging the gap with the performance from the larger proprietary training dataset of Agustsson et al.

Original: At higher bitrate, the model optimized with Vimeo-90k(Xue et al., 2019) tends to have a similar performance with YouTube-NT.
Revised: At higher bitrate, the model optimized with Vimeo-90k(Xue et al., 2019) tends to have a similar performance with YouTube-NT.

Original: This is likely because
Revised: This is likely because

Original: YouTube-NT currently only covers 8000 videos, limiting the diversity of the short clips.
Revised: YouTube-NT currently only covers 8000 videos, limiting the diversity of the short clips.

Original: Appendix F. Command for HEVC codec
Revised: Appendix F. Command for HEVC codec

Original: To avoid FFmpeg package taking the advantage of the input ﬁle color format (YUV420), we ﬁrst need to dump the video.yuv ﬁle to a sequence of lossless png ﬁles:[Equation]
Revised: To avoid FFmpeg package taking the advantage of the input ﬁle color format (YUV420), we ﬁrst need to dump the video.yuv ﬁle to a sequence of lossless png ﬁles:

Original: video [Equation] d. [Equation]
Revised: ffmpeg -i video.yuv -vsync 0 video/%d.png

Original: Then we use the default low-latency setting in ffmpeg to compress the dumped png sequences:
Revised: Then we use the default low-latency setting in ffmpeg to compress the dumped png sequences:

Original: ffmpeg -y -i video/%d.png -c [Equation] v libx265 -preset medium \ -x265-params bframes=

Revised: ffmpeg -y -i video/%d.png -c:v libx265 -preset medium \ -x265-params bframes=

Original: -crf
Revised: -crf

Original: {crf} video.mkv
Revised: {crf} video.mkv

Original: where crf is the parameter for quality control.
Revised: where crf is the parameter for quality control.

Original: Appendix G. Training schedule
Revised: Appendix G. Training schedule

Original: Training time is about four days on an NVIDIA Titan RTX.
Revised: Training time is about four days on an NVIDIA Titan RTX.

Original: Similar to Agustsson et al.
Revised: Similar to Agustsson et al.

Original: (2020), we use the Adam optimizer (Kingma and Ba, 2014), training the models for 1,050, steps.
Revised: (2020), we use the Adam optimizer (Kingma and Ba, 2014), training the models for 1,050, steps.

Original: The initial learning rate of 1e-4 is decayed to 1e-5 after 900,000 steps, and we increase the crop size to 384x384 for the last 50,000 steps.
Revised: The initial learning rate of 1e-4 is decayed to 1e-5 after 900,000 steps, and we increase the crop size to 384x384 for the last 50,000 steps.

Original: All models are optimized using MSE loss.
Revised: All models are optimized using MSE loss.

Original: Appendix H. Eﬃcient scale-space-ﬂow
Revised: Appendix H. Eﬃcient scale-space-ﬂow

Original: Agustsson et al.
Revised: Agustsson et al.

Original: (2020) leverages a simple implementation of scale-space ﬂow by convolving the previous reconstructed frame ˆ x t − 1 with a sequence of Gaussian kernel σ 2 = { 0 , σ 20 , (2 σ 0 ) 2 , (4 σ 0 ) 2 , (8 σ 0 ) 2 , (16 σ 0 ) 2 } .
Revised: (2020) leverages a simple implementation of scale-space ﬂow by convolving the previous reconstructed frame ˆ x t − 1 with a sequence of Gaussian kernel σ 2 = { 0 , σ 20 , (2 σ 0 ) 2 , (4 σ 0 ) 2 , (8 σ 0 ) 2 , (16 σ 0 ) 2 } .

Original: However, this may lead to a large kernel size when σ is increasing and signiﬁcantly reduce the eﬃciency.
Revised: However, this may lead to a large kernel size when σ is increasing and signiﬁcantly reduce the eﬃciency.

Original: For example, Gaussian kernel with σ 2 = 256 usually requires kernel size 97x97 to avoid artifact (usually kernel size =
Revised: For example, Gaussian kernel with σ 2 = 256 usually requires kernel size 97x97 to avoid artifact (usually kernel size =

Original: (6 ∗ σ +1) 2 ).
Revised: (6 ∗ σ +1) 2 ).

Original: To alleviate the problem, we leverage an eﬃcient version of Gaussian scale-space by using Gaussian pyramid with upsampling.
Revised: To alleviate the problem, we leverage an eﬃcient version of Gaussian scale-space by using Gaussian pyramid with upsampling.

Original: In our implementation, we use σ 2 = { 0 , σ 20 , σ 20 +(2 σ 0 ) 2 , σ 20 + (2 σ 0 ) 

Revised: In our implementation, we use σ 2 = { 0 , σ 20 , σ 20 +(2 σ 0 ) 2 , σ 20 + (2 σ 0 )

Original: +(4 σ 0 ) 2 , σ 20 +(2 σ 0 ) 

Revised: +(4 σ 0 ) 2 , σ 20 +(2 σ 0 )

Original: +(4 σ 0 ) 

Revised: +(4 σ 0 )

Original: +(8 σ 0 ) 2 , σ 20 +(2 σ 0 ) 

Revised: +(8 σ 0 ) 2 , σ 20 +(2 σ 0 )

Original: +(4 σ 0 ) 

Revised: +(4 σ 0 )

Original: +(8 σ 0 ) 

Revised: +(8 σ 0 )

Original: +(16 σ 0 ) 2 } , because by using Gaussian pyramid, we can always use Gaussian kernel with σ = σ 0 to consecutively blur and downsample the image to build a pyramid .
Revised: +(16 σ 0 ) 2 } , because by using Gaussian pyramid, we can always use Gaussian kernel with σ = σ 0 to consecutively blur and downsample the image to build a pyramid .

Original: At the ﬁnal step, we only need to upsample all the downsampled images to the original size to approximate a scale-space 3D tensor.
Revised: At the ﬁnal step, we only need to upsample all the downsampled images to the original size to approximate a scale-space 3D tensor.

Original: Result: sst : Scale-space 3D tensor
Revised: Result: sst : Scale-space 3D tensor

Original: Input: input input ; σ 0 base scale; M scale  sst = input] kernel =
Revised: Input: input input image; σ 0 base scale; M scale depth sst = [input] kernel =

Original: Create Gaussian Kernel( σ 0 ) for i=0 to M-1 do input = GaussianBlur(input, kernel) if i =
Revised: Create Gaussian Kernel( σ 0 ) for i=0 to M-1 do input = GaussianBlur(input, kernel) if i =

Original: = 0 then sst else  end 
Revised: = 0 then sst.append(input) else tmp = input for j=0 to i-1 do tmp = UpSample2x(tmp); { step upsampling for smooth interpolation } end ssv.append(tmp) end input end return Concat

Original:  I. Architecture
Revised: Appendix I. Architecture

Original: Figure 5 illustrates the low-level encoder, decoder and hyper-en/decoder modules used in our proposed STAT-SSF and STAT-SSF-SP models, as well as in the baseline TAT and SSF models, based on Agustsson et al.
Revised: Figure 6 illustrates the low-level encoder, decoder and hyper-en/decoder modules used in our proposed STAT-SSF and STAT-SSF-SP models, as well as in the baseline TAT and SSF models, based on Agustsson et al.

Original: Figure 6 shows the encoder-decoder ﬂowchart for w t and v t separately, as well as their corresponding entropy models (priors), in the
Revised: Figure 7 shows the encoder-decoder ﬂowchart for w t and v t separately, as well as their corresponding entropy models (priors), in the

Original: STAT-SSF-SP model.
Revised: STAT-SSF-SP model.