H IERARCHICAL A UTOREGRESSIVE M ODELING FOR N EURAL V IDEO C OMPRESSION
A BSTRACT
Recent work by Marino et al.
(2020) showed improved performance in sequential density estimation by combining masked autoregressive ﬂows with hierarchical latent variable models.
We draw a connection between such autoregressive generative models and the task of lossy video compression.
Speciﬁcally, we view recent neural video compression methods (Lu et al., 2019;
Yang et al., 2020b; Agustsson et al., 2020) as instances of a generalized stochastic temporal autoregressive transform, and propose avenues for enhancement based on this insight.
Comprehensive evaluations on large-scale video data show improved rate-distortion performance over both state-of-the-art neural and conventional video compression methods.
I NTRODUCTION
With recent advances, deep generative modeling has seen a surge in applications, including neuralbased compression.
Generative models have already demonstrated empirical improvements in image compression, outperforming classical codecs, such as BPG (Bellard, 2014).
In contrast, the less developed area of neural-based video compression remains challenging due to complex temporal dependencies operating at multiple scales.
Nevertheless, recent neural video codecs have shown promising performance gains (Agustsson et al., 2020), in some cases on par with current handdesigned, classical codecs, e.g., HEVC.
Compared with hand-designed codecs, learnable codecs can be tuned to speciﬁc data modalities, providing a promising approach for streaming specialized content, such as sporting events or video chatting.
Therefore, improving neural-based video compression is a necessity for dealing with the ever-growing amount of video content being created.
Source compression fundamentally involves decorrelation, i.e., transforming input data into white noise distributions that can be easily modeled and entropy-coded.
Thus, improving a model’s capability to decorrelate data automatically improves its compression performance.
Likewise, we can improve the associated entropy model (i.e., the model’s prior) to capture any remaining dependencies.
Just as compression techniques attempt to remove structure, generative models attempt to model structure.
One family of models, autoregressive ﬂows, maps between less structured distributions, e.g., uncorrelated noise, and more structured distributions, e.g., images or video (Dinh et al., 2014; 2016).
The inverse mapping can remove dependencies in the data, making it more amenable for compression.
Thus, a natural question to ask is how autoregressive ﬂows can best be utilized in compression, and if mechanisms in existing compression schemes can be interpreted as ﬂows.
This paper draws on recent insights in hierarchical sequential latent variable models with autoregressive ﬂows (Marino et al., 2020).
In particular, we identify connections between this family of models and a recently proposed neural video codec based on scale-space warping (Agustsson et al., 2020).
By interpreting this technique as a speciﬁc instantiation of a ﬂow, we propose various alternatives and improvements, drawing on insights from generative modeling and autoregressive ﬂows.
In more detail, our main contributions are as follows:
A new framework.
We interpret existing video compression methods through the more general framework of generative modeling, variational inference, and autoregressive ﬂows, allowing us to readily investigate extensions and ablations.
In particular, we compare fully data-driven approaches with motion-estimation-based neural compression schemes, and consider a more expressive prior model for better entropy coding (described in the second bullet point below).
This framework also provides directions for future work.
A new model.
Our main proposed model extends the Scale-Space Flow (SSF) (Agustsson et al., 2020) model.
Following the classical predictive coding approach to video compression, this model uses motion estimation to predict the frame being compressed, and further compresses the residual obtained by subtraction.
We improve on SSF by: • Incorporating a learnable scaling transform, to allow for more expressive and accurate reconstruction.
Augmenting a shift transform by scale-then-shift is inspired by improvements from extending NICE (Dinh et al., 2014) to RealNVP (Dinh et al., 2016).
• Introducing a structured prior over the two sets of latent variables in the probabilistic generative model of SSF (based our generative framework), corresponding to jointly encoding the motion information and residual information.
As the two tend to be spatially correlated, encoding residual information conditioned on motion information results in a more informed prior model, and thus better entropy model, for the residual information, cutting down its bit-rate that typically dominates the overall bit-rate.
A new dataset.
The neural video compression community is lacking large, high-resolution benchmark datasets.
While we performed extensive experiments on the publicly available Vimeo-90k dataset (Xue et al., 2019), we also collected and utilized a larger dataset, YouTube-NT 1 , available through executable scripts.
Since no training data was publicly released for the previous state-of-the-art method (Agustsson et al., 2020), YouTube-NT will be a useful resource for making and comparing further progress in this ﬁeld.
R ELATED W ORK
We divide relevant related work into three categories: neural image compression, neural video compression and sequential generative models.
Neural Image Compression.
Considerable progress has been made by applying neural networks to image compression.
Early works proposed by Toderici et al.
(2017) and Johnston et al.
(2018) leveraged LSTMs to model spatial correlations of the pixels within an image.
Theis et al.
ﬁrst proposed an autoencoder architecture for image compression and used the straight-through estimator (Bengio et al., 2013) for learning a discrete latent representation.
The connection to probabilistic generative models was drawn by Ball ´ e et al.
(2017), who ﬁrstly applied variational autoencoders (VAEs) (Kingma & Welling, 2013) to image compression.
In subsequent work, Ball´e et al.
(2018) encoded images with a two-level VAE architecture involving a scale hyper-prior, which can be further improved by autoregressive structures (Minnen et al., 2018; Minnen & Singh, 2020) or by optimization at encoding time (Yang et al., 2020d).
Yang et al.
(2020e) and Flamich et al.
(2019) demonstrated competitive image compression performance without a pre-deﬁned quantization grid.
Neural Video Compression.
Compared to image compression, video compression is a signiﬁcantly more challenging problem, as statistical redundancies exist not only within each video frame (exploited by intra-frame compression) but also along the temporal dimension.
Early works by Wu et al.
; Djelouah et al.
and Han et al.
(2019) performed video compression by predicting future frames using a recurrent neural network, whereas Chen et al.
and
Chen et al.
(2017) used convolutional architectures within a traditional block-based motion estimation approach.
These early approaches did not outperform the traditional H.264 codec and barely surpassed the MPEGcodec.
Lu et al.
(2019) adopted a hybrid architecture that combined a pre-trained Flownet (Dosovitskiy et al., 2015) and residual compression, which leads to an elaborate training scheme.
Habibian et al.
and Liu et al.
combined 3D convolutions for dimensionality reduction with expressive autoregressive priors for better entropy modeling at the expense of parallelism and runtime efﬁciency.
Our method extends a low-latency model proposed by Agustsson et al.
(2020), which allows for end-to-end training, efﬁcient online encoding and decoding, and parallelism.
Sequential Deep Generative Models.
We drew inspiration from a body of work on sequential generative modeling.
Early deep learning architectures for dynamics forecasting involved RNNs (Chung et al., 2015).
Denton & Fergus (2018) and Babaeizadeh et al.
(2017) used VAE-based stochastic models in conjunction with LSTMs to model dynamics.
Li & Mandt (2018) introduced both local and global latent variables for learning disentangled representations in videos.
Other video generation models used generative adversarial networks (GANs) (Vondrick et al., 2016; Lee et al., 2018) or autoregressive models and normalizing ﬂows (Rezende & Mohamed, 2015;
Dinh et al., 2014; 2016; Kingma & Dhariwal, 2018; Kingma et al., 2016;
Papamakarios et al., 2017).
Recently, Marino et al.
(2020) proposed to combine latent variable models with autoregressive ﬂows for modeling dynamics at different levels of abstraction, which inspired our models and viewpoints.
V IDEO C OMPRESSION THROUGH D EEP A UTOREGRESSIVE M ODELING
We identify commonalities between hierarchical autoregressive ﬂow models (Marino et al., 2020) and state-of-the-art neural video compression architectures (Agustsson et al., 2020), and will use this viewpoint to propose improvements on existing models.
3. B ACKGROUND
We ﬁrst review VAE-based compression schemes (Ball ´ e et al., 2017; Theis et al., 2017) and formulate existing low-latency video codecs in this framework.
We also review ﬂow-based models.
Generative Modeling and Source Compression.
Let x 1: T ∈ R T × D be a sequence of video frames.
Lossy compression seeks to ﬁnd the shortest description R of x 1: T without exceeding a certain level of information loss D .
The distortion D measures how much reconstruction error accrues due to encoding x 1: T into a latent representation ¯ z 1: T and subsequently decoding it back to ˆx 1: T , while R measures the bit rate (ﬁle size).
In learned compression methods (Ball ´ e et al., 2017;
Theis et al., 2017), the above process is parameterized by ﬂexible functions f (“encoder”) and g (“decoder”) that map between the input video and its latent representation ¯z 1: T = f ( x 1: T ) .
Using a parameter β , both terms can be weighted against each other:
[Equation]
We adopt the end-to-end compression approach of Ball ´ e et al.
; Han et al.
(2019), which approximates the rounding operations ( (cid:98)•(cid:101) ) by uniform noise injection to enable gradient-based optimization during training.
With an appropriate choice of probability model p ( z 1: T ) , the approximated version of the above R-D objective then corresponds to the VAE objective:
[Equation]
In this model, the likelihood p ( x 1: T | z 1: T ) follows a Gaussian distribution with mean ˆx 1: T = g ( z 1: T ) and diagonal covariance
β 2log
I , the approximate posterior q is chosen to be a unit-width uniform (thus zero-differential-entropy) distribution whose mean ¯z 1: T is predicted by an amortized inference network f .
The prior density p ( z 1: T ) interpolates its discretized version, so that it measures the code length of discretized ¯z 1: T after entropy-coding.
Low-Latency Sequential Compression
We specialize Eq.
1 to make it suitable for low-latency video compression, widely used in both conventional and recent neural codecs (Rippel et al., 2019;
Agustsson et al., 2020).
To this end, we encode and decode individual frames x t in sequence.
Given the ground truth current frame x t and the previously reconstructed frames ˆx <t , the encoder is restricted to be of the form ¯z t = f ( x t , ˆx <t ) , and similarly the decoder computes reconstruction sequentially based on previous reconstructions and the current encoding, ˆx t =
g ( ˆx <t , (cid:98) ¯z t (cid:101) )) .
Existing codecs usually condition on a single reconstructed frame, substituting ˆx <t by ˆx t − 1 in favor of efﬁciency.
In the language of variational inference, the sequential encoder corresponds to a variational posterior of the form q ( z t | x t , z <t ) , i.e., ﬁltering, and the sequential decoder corresponds to the likelihood p ( x t | z ≤ t ) =
N ( ˆx t , β 2log2 I ) ; in both distributions, the probabilistic conditioning on z <t is based on the observation that ˆx t − 1 is a deterministic function of z <t .
As we show, all sequential compression approaches considered in this work follow this paradigm, and the form of the reconstruction transform ˆx determines the lowest hierarchy of the corresponding generative process of video x .
Masked Autoregressive Flow (MAF).
As a ﬁnal component in neural sequence modeling, we discuss MAF (Papamakarios et al., 2017), which models the joint distribution of a sequence p ( x 1: T ) in terms of a simpler distribution of its underlying noise variables y 1: T through the following autoregressive transform and its inverse:
[Equation]
The noise variable y t usually comes from a standard normal distribution.
While the forward MAF transforms a sequence of standard normal noises into a data sequence, the inverse ﬂow “whitens” the data sequence and removes temporal correlations.
Due to its invertible nature, MAF allows for exact likelihood computations, but as we will explain in Section 3.3, we will not exploit this aspect in compression but rather draw on its expressiveness in modeling conditional likelihoods.
3. A G ENERAL F RAMEWORK FOR G ENERATIVE V IDEO C ODING
We now describe a general framework that captures several existing low-latency neural compression methods as speciﬁc instances and gives rise to the exploration of new models.
To this end, we combine latent variable models with autoregressive ﬂows into a joint framework.
We consider a sequential decoding procedure of the following form:
[Equation]
Eq.
3 resembles the deﬁnition of the MAF in Eq. 2, but augments this transform with two sets of latent variables w t , v t ∼ p ( w t , v t ) .
Above, h µ and h σ are functions that transform the previous reconstructed data frame ˆx t − 1 along with w t into a shift and scale parameter, respectively.
The function g v ( v t , w t ) converts these latent variables into a noise variable that encodes residuals with respect to the mean next-frame prediction h µ ( ˆx t − 1 , w t ) .
This stochastic decoder model has several advantages over existing generative models for compression, such as simpler ﬂows or sequential VAEs.
First, the stochastic autoregressive transform h µ ( ˆx t − 1 , w t ) involves a latent variable w t and is therefore more expressive than a deterministic transform (Schmidt & Hofmann, 2018; Schmidt et al., 2019).
Second, compared to MAF, the additional nonlinear transform g v ( v t , w t ) enables more expressive residual noise, reducing the burden on the entropy model.
Finally, as visualized in Figure 2, the scale parameter h σ ( ˆx t − 1 , w t ) effectively acts as a gating mechanism, determining how much variance is explained in terms of the autoregressive transform and the residual noise distribution.
This provides an added degree of ﬂexibility, in a similar fashion to how RealNVP improves over NICE (Dinh et al., 2014; 2016).
Our approach is inspired by Marino et al.
(2020) who analyzed a restricted version of the model in Eq. 3, aiming to hybridize autoregressive ﬂows and sequential latent variable models for video prediction.
In contrast to Eq. 3, their model involved deterministic transforms as well as residual noise that came from a sequential VAE.
3. E XAMPLE M ODELS AND E XTENSIONS
Next, we will show that the general framework expressed by Eq.
3 captures a variety of state-of-theart neural video compression schemes and gives rise to extensions and new models.
Temporal Autoregressive Transform (TAT).
The ﬁrst special case among the class of models that are captured by Eq.
3 is the autoregressive neural video compression model by Yang et al.
(2020b), which we denote as temporal autoregressive transform (TAT).
Shown in Figure 1(a), the decoder g implements a deterministic scale-shift autoregressive transform on decoded noise y t ,
[Equation]
The encoder f inverts the transform to decorrelate the input frame x t into ¯y t and encodes the result as ¯z t = f ( x t , ˆx t − 1 ) =
f z ( ¯y t ) , where ¯y t = x t − h µ ( ˆx t − 1 ) h
σ ( ˆx t − 1 ) .
The shift h µ and scale h σ transforms are parameterized by neural networks, f z is a convolutional neural network (CNN), and g z is a deconvolutional neural network (DNN) that approximately inverts f z .
The TAT decoder is a simple version of the more general stochastic autoregressive transform in Eq 3, where h µ and h σ lack latent variables.
Indeed, interpreting the probabilistic generative process of
[Figure]
additional β from the inverse transform decorrelates raw to
DVC (Lu et al., 2019) and Scale-Space Flow (SSF, Agustsson et al.
The second class of models captured by Eq.
3 belong to the conventional video compression framework based on predictive coding (Cutler, 1952; Wiegand et al., 2003; Sullivan et al., 2012); both models make use of two sets of latent variables z 1: T = { w 1: T , v 1: T } to capture different aspects of information being compressed, where w captures estimated motion information used in warping prediction, and v helps capture residual error not predicted by warping.
Like most classical approaches to video compression by predictive coding, the reconstruction transform in the above models has the form of a prediction shifted by residual error (decoded noise), and lacks the scaling factor h σ compared to the autoregressive transform in Eq.
[Equation]
Above, g w and g v are DNNs, o t :=
g w ( w t ) has the interpretation of an estimated optical ﬂow (motion) ﬁeld, h warp is the computer vision technique of warping, and the residual r t :=
g v ( v t , w t ) =
ˆx t − h warp ( ˆx t − 1 , o t ) represents the prediction error unaccounted for by warping.
Lu et al.
(2019) only makes use of v t in the residual decoder g v , and performs simple 2D warping by bi-linear interpretation; SSF (Agustsson et al., 2020) augments the optical ﬂow (motion) ﬁeld o t with an additional scale ﬁeld, and applies scale-space-warping to the progressively blurred versions of ˆx t − 1 to allow for uncertainty in the warping prediction.
The encoding procedure in the above models compute the variational mean parameters as ¯w t = f w ( x t , ˆx t − 1 ) , ¯v t
= f v ( x t − h warp ( ˆx t − 1 , g w ( w t ))) , corresponding to a structured posterior q ( z t | x t , z <t ) =
q ( w t | x t , z <t ) q ( v t | w t , x t , z <t ) .
We illustrate the above generative and inference procedures in Figure 1(b).
Proposed: models based on Stochastic Temporal Autoregressive Transform.
Finally, we consider the most general models as described by the stochastic autoregressive transform in Eq. 3, shown in Figure 1(c).
We study two main variants, categorized by how they implement h µ and h σ :
STAT uses DNNs for h µ and h σ as in (Yang et al., 2020b), but complements it with the latent variable w t that characterizes the transform.
In principle, more ﬂexible transforms should give better compression performance, but we ﬁnd the following variant more parameter efﬁcient in practice:
STAT-SSF : a less data-driven variant of the above that still uses scale-space warping (Agustsson
[Figure]
in the three methods.
The proposed scale parameter ˆ σ t gives the model extra ﬂexibility when combining the noise ˆy t (decoded from ( (cid:98) ¯v t (cid:101) , (cid:98) ¯w t (cid:101) ) ) with the warping prediction ˆ µ t (decoded from (cid:98) ¯w t (cid:101) only) to form the reconstruction ˆx t
= ˆ µ t + ˆ σ t (cid:12) ˆy t : the scale ˆ σ t downweights contribution from the noise ˆy t in the foreground where it is very costly, and reduces the residual bit-rate R ( (cid:98) ¯v t (cid:101) ) (and thus the overall bit-rate) compared to STAT-SSF and
SSF (with similar reconstruction quality), as illustrated in the third and fourth ﬁgures in the top row.
et al., 2020) in the shift transform , i.e., h µ ( ˆx t − 1 , w t ) = h warp ( ˆx t − 1 , g w ( w t )) . This can also be seen as an extended version of the SSF model, whose shift transform h µ is preceded by a new learned scale transform h σ .
Structured Prior (SP).
Besides improving the autoregressive transform (affecting the likelihood model for x t ), one variant of our approach also improves the topmost generative hierarchy in the form of a more expressive latent prior p ( z 1: T ) , affecting the entropy model for compression.
We observe that motion information encoded in w t can often be informative of the residual error encoded in v t .
In other words, large residual errors v t incurred by the mean prediction h µ ( ˆx t − 1 , w t ) (e.g., the result of warping the previous frame h µ ( ˆx t − 1 )) are often spatially collocated with (unpredictable) motion as encoded by w t .
The original SSF model’s prior factorizes as p ( w t , v t ) = p ( w t ) p ( v t ) and does not capture such correlation.
We therefore propose a structured prior by introducing conditional dependence between w t and v t , so that p ( w t , v t ) = p ( w t ) p ( v t | w t ) .
At a high level, this can be implemented by introducing a new neural network that maps w t to parameters of a parametric distribution of p ( v t | w t )
(e.g., mean and variance of a diagonal Gaussian distribution).
This results in variants of the above models, STAT-SP and STAT-SSF-SP , where the structured prior is applied on top of the proposed STAT and STAT-SSF models.
E XPERIMENTS
In this section, we train our models both on existing dataset as well as our new YouTube-NT dataset, and improve over state-of-the-art neural as well as classical video compression methods when evaluating on several publically available benchmark datasets.
Lower-level modules and training scheme for our models largely follow Agustsson et al.
(2020); we provide detailed model diagrams and schematic implementation, including the proposed scaling transform and structured prior, in Appendix A.4.
We also implement a more computationally efﬁcient version of scale-space warping (Agustsson et al., 2020) based on Gaussian pyramid and interpolation (instead of naive Gaussian blurring); pseudocode is available at Appendix A.3.
4. T RAINING D ATASETS
Vimeo-90k (Xue et al., 2019) consists of 90,000 clips of 7 frames at 448x256 resolution collected from vimeo.com .
As this dataset has been used in previous works (Lu et al., 2019; Yang et al., 2020a; Liu et al., 2019), it provides a benchmark for comparing models.
While other publicly- available video datasets exist, e.g., Kinetics (Carreira & Zisserman, 2017), such datasets are lowerresolution and primarily intended for human action recognition.
Accordingly, previous works that have trained using Kinetics generally report sub-par PSNR-bitrate performance (Wu et al., 2018;
Habibian et al., 2019; Golinski et al., 2020).
Agustsson et al.
(2020) uses a signiﬁcantly larger and higher-resolution dataset collected from youtube.com , however, this is not publicly available.
YouTube-NT .
This is our new dataset.
We collected 8,000 nature videos and movie/video-game trailers from youtube.com and processed them into 300k high-resolution (720p) clips, which we refer to as YouTube-NT.
In contrast to existing datasets (Carreira & Zisserman, 2017; Xue et al., 2019), we provide YouTube-NT in the form of customizable scripts to enable future compression research.
Table 1 compares the current version of YouTube-NT with Vimeo-90k (Xue et al., 2019) and with Google’s proprietary training dataset (Agustsson et al., 2020).
In Figure 5b, we display the evaluation performance of the SSF model architecture after training on each dataset.
[Table]
4. T RAINING AND E VALUATION S CHEME
Training .
All models are trained with three consecutive frames and batchsize 8, which are randomly selected from each clip, then randomly cropped to 256x256.
We trained on MSE loss, following similar procedure to Agustsson et al.
(Refer to Appendix A.2 for details).
Evaluation .
We evaluate compression performance on the widely used UVG (Mercat et al., 2020) and MCL JCV (Wang et al., 2016) test datasets, which both consist of raw videos in YUV format.
UVG is widely used for testing the HEVC codec and contains seven 1080p videos at 120fps with smooth and mild motions or stable camera moving.
MCL JCV contains thirty 1080p videos at 30fps, which are generally more diverse, with a higher degree of motion and a more unstable camera.
We compute the bit rate (bits-per-pixel, BPP) and the reconstruction quality as measured in PSNR, averaged across all frames.
We note that PSNR is a more challenging metric than MS-SSIM (Wang et al., 2003) for learned codecs (Lu et al., 2019;
Agustsson et al., 2020; Habibian et al., 2019;
Yang et al., 2020a;c).
Since existing neural compress methods assume video input in RGB format (24bits/pixel), we follow this convention in our evaluations for meaningful comparisons.
We note that HEVC also has special support for YUV420 (12bits/pixel), allowing it to exploit this more compact ﬁle format and effectively halve the input bitrate on our test videos (which were coded in YUV420 by default), giving it an advantage over all neural methods.
Regardless, we report the performance of HEVC in YUV420 mode (in addition to the default RGB mode) for reference.
[Table]
4. B ASELINE A NALYSIS
We trained our models on Vimeo-90k data, in order to compare with published results of baseline models listed in Table 2.
Figure 3a compares our proposed models (STAT-SSF, STAT-SSF-SP) with
[Figure] [Figure]
on top.
Here, models with the proposed scale transform (STAT-SSF and STAT-SSF-SP) outperform the ones without, yielding visually more detailed reconstructions at lower rates; structured prior (STAT-SSF-SP) reduces the bit-rate further.
previous state-of-the-art classical codec HEVC and neural codecs on the UVG evaluation dataset.
Our STAT-SSF-SP model provides superior performance at bitrates ≥ 0 .
07 BPP, outperforming conventional HEVC even in its favored YUV 420 mode and state-of-the-art neural method SSF (Agustsson et al., 2020) , as well as the established DVC (Lu et al., 2019), which leverages a more complicated model and multi-stage training procedure.
We also note that, as expected, our proposed
[Figure] STAT model improves over TAT (Yang et al., 2020b), with the latter lacking stochasticity in the autoregressive transform compared to our proposed STAT and its variants.
Figure 3a shows that the performance ranking on MCL JCV is similar to on UVG, despite
MCL JCV having more diverse and challenging (e [Figure] g., animated) content (Agustsson et al., 2020).
We provide qualitative results in Figure 2 and 4, offering insight into the behavior of the proposed scaling transform and structured prior, as well as visual qualities of the top performing methods [Figure]
4. A BLATION A NALYSIS
Using the baseline SSF (Agustsson et al., 2020) model and YouTube-NT training dataset [Figure] we demonstrate the improvements of our proposed components, stochastic temporal autoregressive transform (STAT) and structured prior (SP), evaluated on UVG.
As shown in Figure 5a, STAT improves performance to a greater degree than SP, while SP alone cannot provide noticable improvement over
SSF. To quantify the effect of the training dataset on performance, we compare performance on UVG for the SSF model architecture after training on Vimeo-90k (Xue et al., 2019) and YouTube-NT.
We also compare with the reported results from Agustsson et al.
(2020), trained on a larger proprietary dataset.
This is shown in Figure 5b, where we see that training on YouTube-NT improves evaluation performance over Vimeo-90k, in some cases bridging the gap with the performance from the larger proprietary training dataset of Agustsson et al.
At higher bitrate, the model optimized with
Vimeo-90k(Xue et al., 2019) tends to have a similar performance with YouTube-NT.
This is likely because YouTube-NT currently only covers 8000 videos, limiting the diversity of the short clips.
D ISCUSSION
We provide a unifying perspective on sequential video compression and temporal autoregressive ﬂows (Marino et al., 2020), and elucidate the relationship between the two in terms of their underlying generative hierarchy.
From this perspective, we consider several video compression methods, particularly a state-of-the-art method Scale-Space-Flow (Agustsson et al., 2020), as instantiations of a more general stochastic temporal autoregressive transform, which allows us to naturally extend the
Scale-Space-Flow model
and obtain improved rate-distortion performance on two public benchmark datasets.
Further, we are providing scripts to generate a new high-resolution video dataset, YouTubeNT, which is substantially larger than current publicly-available datasets.
Together, we hope that this new perspective and dataset will drive further progress in the nascent yet highly impactful ﬁeld of learned video compression.
A A PPENDIX A. C OMMAND FOR HEVC CODEC
To avoid FFmpeg package taking the advantage of the input ﬁle color format (YUV420), we ﬁrst need to dump the video.yuv ﬁle to a sequence of lossless png ﬁles: ffmpeg -i video.yuv -vsync 0 video/%d.png
Then we use the default low-latency setting in ffmpeg to compress the dumped png sequences: ffmpeg -y -i video/%d.png -c:v libx265 -preset medium \ -x265-params bframes=
-crf {crf} video.mkv where crf is the parameter for quality control.
The compressed video is encoded by HEVC with
RGB color space.
To get the result of HEVC (YUV420), we directly execute:
[Equation]
video
[Equation]
[Equation]
video.mkv
A. T RAINING SCHEDULE
Training time is about four days on an NVIDIA Titan RTX.
Similar to Agustsson et al.
(2020), we use the Adam optimizer (Kingma & Ba, 2014), training the models for 1,050,000 steps.
The initial learning rate of 1e-4 is decayed to 1e-5 after 900,000 steps, and we increase the crop size to 384x for the last 50,000 steps.
All models are optimized using MSE loss.
A. E FFICIENT SCALE - SPACE - FLOW
Agustsson et al.
(2020) leverages a simple implementation of scale-space ﬂow by convolving the previous reconstructed frame ˆ x t − 1 with a sequence of Gaussian kernel σ 2 = { 0 , σ 20 , (2 σ 0 ) 2 , (4 σ 0 ) 2 , (8 σ 0 ) 2 , (16 σ 0 ) 2 } .
However, this may lead to a large kernel size when σ is increasing and signiﬁcantly reduce the efﬁciency.
For example, Gaussian kernel with σ 2 =usually requires kernel size 97x97 to avoid artifact (usually kernel size =
(6 ∗ σ + 1) 2 ).
To alleviate the problem, we leverage an efﬁcient version of Gaussian scale-space by using Gaussian pyramid with upsampling.
In our implementation, we use σ 2 = { 0 , σ 20 , σ 20 + (2 σ 0 ) 2 , σ 20 + (2 σ 0 )
+ (4 σ 0 ) 2 , σ 20 + (2 σ 0 )
+ (4 σ 0 )
+ (8 σ 0 ) 2 , σ 20 + (2 σ 0 )
+ (4 σ 0 )
+ (8 σ 0 )
+ (16 σ 0 ) 2 } , because by using Gaussian pyramid, we can always use Gaussian kernel with σ = σ 0 to consecutively blur and downsample the image to build a pyramid .
At the ﬁnal step, we only need to upsample all the downsampled images to the original size to approximate a scale-space 3D tensor.
Detailed algorithm is described in Algorithm 1.
A. A RCHITECTURE
Figure 6 illustrates the low-level encoder, decoder and hyper-en/decoder modules used in our proposed STAT-SSF and STAT-SSF-SP models, as well as in the baseline TAT and SSF models, based on Agustsson et al.
Figure 7 shows the encoder-decoder ﬂowchart for w t and v t separately, as well as their corresponding entropy models (priors), in the STAT-SSF-SP model.