I NSTANCE -A DAPTIVE V IDEO C OMPRESSION : I MPROVING N EURAL C ODECS BY T RAINING ON THE T EST S ET
Anonymous authors
Paper under double-blind review
A BSTRACT
We introduce a video compression algorithm based on instance-adaptive learning.
On each video sequence to be transmitted, we ﬁnetune a pretrained compression model.
The optimal parameters are transmitted to the receiver along with the latent code.
By entropy-coding the parameter updates under a suitable mixture model prior, we ensure that the network parameters can be encoded efﬁciently.
This instance-adaptive compression algorithm is agnostic about the choice of base model and has the potential to improve any neural video codec.
On UVG, HEVC, and Xiph datasets, our codec improves the performance of a low-latency scalespace
ﬂow model by between 21 % and 26 % BD-rate savings, and that of a stateof-the-art B-frame model by 17 to 20 % BD-rate savings.
We also demonstrate that instance-adaptive ﬁnetuning improves the robustness to domain shift.
Finally, our approach reduces the capacity requirements on compression models.
We show that it enables a competitive performance even after reducing the network size by 72 %.
I NTRODUCTION
Neural compression methods have enormous potential to improve the efﬁciency of video coding.
With video constituting the large majority of internet trafﬁc, this has signiﬁcant implications for the internet at large (Sandvine, 2019).
State-of-the-art algorithms model each frame by warping the previous one with a neural estimate of the optical ﬂow (Lu et al., 2019) or scale-space ﬂow (Agustsson et al., 2020) and adding residuals modeled by another network.
Both the optical ﬂow and the residuals are compressed with variational autoencoders.
Such neural codecs have recently achieved results on par with popular classical codecs (Agustsson et al., 2020;
Rippel et al., 2021; Pourreza &
Cohen, 2021) such as H.264 / AVC (Wiegand et al., 2003) and H.265 / HEVC (Sullivan et al., 2012).
However, relatively little research has focused on their computational complexity, and matching the rate-distortion performance of H.266 / VVC (Bross et al., 2018) is still an open problem.
Neural video codecs depend critically on generalization : they are developed assuming that a good performance on training and validation datasets translates to a good performance at test time.
However, this assumption does not always hold in practice, for instance because of limited training data or imperfect optimization.
Domain shift is a particularly challenging problem given the variety in video content and styles; for instance, neural video codecs trained on natural scene video data often perform poorly on animated sequences (Agustsson et al., 2020).
Here we address this problem by optimizing the autoencoder — ﬁnetuning its weights — for each single video to be compressed at test time.
After training on the test set in this way, the ﬁnetuned network parameters need to be made available on the receiver side: they are compressed, quantized, and entropy-coded under a suitable prior to the bitstream along with the latent code.
Optimizing the compression network for every video in this way relaxes the requirement of training data that closely matches the test distribution as well as the necessary model capacity.
It also allows trading off encoding compute for better compression efﬁciency.
The essential idea of instance-adaptive ﬁnetuning was recently proposed by van Rozendaal et al.
The authors demonstrated the idea on I-frame compression, i. e. compressing a set of similar images, but did not apply it to video sequences yet.
In this work we extend the idea to the compression of full videos, in which only periodic key frames are compressed as images and most frames are encoded relative to one or two reference frames.
Our method is general and can be applied to various settings and base models.
We ﬁrst demonstrate it in a low-latency setting with
I-frames and P-frames using a scale-space ﬂow architecture (Agustsson et al., 2020) as base model.
Next we show its performance in a B-frame setting, using the base model proposed by Pourreza &
Cohen (2021).
On the UVG-1k, HEVC class-B, and Xiph-5N datasets, our new instance-adaptive video compression codec yields BD-rate savings of 17 to 26 % over the respective base models and 5 to 44 % over the popular ffmpeg (x265) implementation of the H.265 codec (FFmpeg; VideoLAN, b).
In addition, instance-adaptive ﬁnetuning lends itself to a possible reduction in model size, because a smaller network may sufﬁce to model a single instance.
We show that in this framework smaller models can still outperform most neural codecs while reducing the computational complexity of the decoder by 72%.
Overall, we demonstrate that unlike other neural video codecs, our method can trade off encoding compute against compression performance, and that this trade-off is more effective than for classical codecs.
R ELATED WORK
Neural video compression The standard framework used by most neural compression codecs (either implicitly or explicitly) is that of the variational (Kingma & Welling, 2014) or compressive (Theis et al., 2017) autoencoder.
An encoder (or approximate posterior) q φ ( z | x ) maps a data point x to a latent representation z .
This latent is transmitted to the decoding party by means of entropy coding under a latent prior p θ ( z ) .
The receiver can then reconstruct the sample with a decoder (or likelihood) p θ ( x | z ) .
The encoder, prior, and decoder are neural networks, parameterized by weights φ , θ as indicated by the subscripts.
These models are trained by minimizing the rate-distortion loss
[Equation]
which combines a distortion metric D and a rate term R z that approximates the bitrate necessary to transmit the latent code z .
Up to a constant entropy term H [ q φ ] , this loss equals the β -weighted
VAE loss (Habibian et al., 2019; Higgins et al., 2017).
Much research has focused on the design of efﬁcient network architectures for neural video codecs, in particular with respect to the temporal structure.
While some earlier works were based on 3D convolutions (Habibian et al., 2019; Pessoa et al., 2020), recent models use optical ﬂow and residual modeling to exploit similarities between frames.
The latter class of approaches can be divided into predictive or P-frame temporal modeling, where the model for each frame x t is conditional on the previous frame(s)
(Agustsson et al., 2020; Chen et al., 2019; Liu et al., 2020; Lombardo et al., 2019; Lu et al., 2019; Rippel et al., 2019; Golinski et al., 2020; Rippel et al., 2021; Hu et al., 2020;
Yang et al., 2020b), and bidirectional or B-frame modeling, where each frame is conditioned on past as well as future reference frames (Cheng et al., 2019; Choi & Baji´c, 2019; Djelouah et al., 2019; Wu et al., 2018; Park & Kim, 2021; Pourreza & Cohen, 2021; Yang et al., 2020a).
Agustsson et al.
(2020) generalize optical ﬂow to scale-space ﬂow by adding dynamic blurring to the warping operation, improving the modeling of uncertainty and leading to state-of-the-art results.
Instance-adaptive compression Rate-distortion autoencoders are trained by minimizing the RD loss in Eq.
(1) over a training dataset D .
This approach relies on the assumptions that the model is not only able to ﬁt the training data well, but also generalizes to unseen datapoints at inference time.
In practice, however, ﬁnite training data, limited model capacity, optimization imperfections, and domain shift — differences between train and test distributions — can degrade the performance.
This problem can be solved by adapting video codecs to each sequence at test time. Several such methods have been proposed, see Appendix A for an in-depth overview that also covers image compression.
These works can be roughly categorized into encoder-only ﬁnetuning, limited decoder ﬁnetuning, and hybrid classical-neural codecs. Works in the ﬁrst category (e. g. Lu et al.
optimize the encoder parameters φ for each data point.
Such an update does not have to be communicated to the receiver side.
This approach alleviates the generalization problem for the encoder, but does not solve it for the decoder and prior.
We will demonstrate later that encoder-only ﬁnetuning leads to a limited improvement in the compression performance.
The second category of works also adapt parts of the prior and decoder for each video instance. Since these modules need to be known on the receiver side, such an update needs to be signalled in the bitstream.
Depending on the implementation, this may increase the bitrate substantially.
Aytekin et al. ; Zou et al. ; Wang et al. (2021) address this by only allowing limited changes to the decoder-side models, for instance choosing one out of a number of ﬁxed decoder networks, thus limiting the potential RD gains of these methods.
Finally, the third category of works propose a hybrid approach, where a video is ﬁrst compressed by a classical codec and a ﬁnetuned enhancement network is added to the bitstream (He et al., 2020; 2021; Klopp et al., 2020; 2021; Liu et al., 2021).
Although these methods have shown great performance at low bitrates, they tend to perform less well at higher bitrates, and do not beneﬁt from developments in end-to-end neural compression.
van Rozendaal et al.
(2021) suggest a method to adapt the full model to a single datapoint.
The key idea is that the parameter updates δ = θ − θ D , where θ D are the global model parameters resulting from training on D , can be be transmitted very efﬁciently.
After discretization, the quantized updates ¯ δ are entropy-coded under a parameter update prior p [ ¯ δ ] that assigns high probability mass (and thus low transmission cost) to zero-updates ¯ δ = 0 .
This procedure enables a neural codec in which the full model — encoder, decoder, and prior — is ﬁnetuned on each data sample at test time by minimizing the loss
[Equation]
φ, θ D + ¯ δ ) + β ( − log p ( δ ))
(cid:124) (cid:123)(cid:122) (cid:125)
R θ . The R θ term reﬂects the increased bitstream length from coding the parameter updates ¯ δ .
In van
Rozendaal et al.
(2021), this approach was successfully demonstrated on I-frame compression, essentially compressing a set of images with similar content.
In this work we adapt it to the compression of full video sequences, as we will describe in the following section.
I NSTANCE - ADAPTIVE VIDEO COMPRESSION
We now introduce our instance-adaptive compression codec for full video sequences, or InstA for short.
The key idea is to optimize the parameters of a rate-distortion VAE for each video sequence to be transmitted , and to send the relevant network parameters in a rate-efﬁcient way to the decoder.
This approach can be beneﬁcial whenever a standard rate-distortion autoencoder would not generalize well, for instance because of limited training data or domain shift.
In addition, instance adaptation allows us to use smaller models while maintaining most of the compression performance.
While encoding the network parameters to the bitstream increases the length of the code, we describe a compression scheme in which this cost is negligible, especially when amortized over many frames of a video sequence.
Our approach is agnostic about the base model and we demonstrate it on two different architectures: a scale-space ﬂow model in the low-latency setting and the B- EPIC model in
the B-frame setting In the following, we ﬁrst describe the base models, before describing how InstA compresses and decompresses video sequences.
Low-latency base model: Scale-space ﬂow (SSF)
We ﬁrst focus on the low-latency setting , which we deﬁne as having only access to current or previous frames when decoding a frame.
As a base model for this setting, we use the scale-space-ﬂow architecture introduced by Agustsson et al.
Any video sequence is ﬁrst split into groups of pictures (GoP).
The ﬁrst frame in each GoP is modeled as an image without any dependency on previous frames, i. e. as an I-frame.
All other frames are P-frames, modeled as
[Equation]
where x i − 1 is the reconstructed previous frame, g i is the estimated scale-space ﬂow ﬁeld, r i is the estimated residual, and the scale-space warping operation performs optical-ﬂow warping of the previous frame with a dynamic, position-dependent amount of Gaussian blur.
The I-frame images,
[Table] P-frame scale-space ﬂow g i , and P-frame residuals r i are compressed with separate hyperprior models (Ball´e et al., 2018), a hierarchical variational autoencoder architecture.
We largely follow the architecture choices of Agustsson et al.
and refer to this model as
SSF18 after its number of decoder-side parameters (in millions).
In Appendix C we describe the architecture in detail and highlight the differences to Agustsson et al.
Smaller low-latency base models As our instance-adaptive approach is based on ﬁnetuning the compression model on a low-entropy “dataset” (a single video sequence), we hypothesize that it does not require the full expressivity of the computationally complex SSF18 model.
We propose three alternative scale-space ﬂow architectures with a reduced computational complexity.
In Tbl.we list the parameter counts and the number of multiply-accumulate (MAC) operations required for decoding a sequence, which are reduced by 43–84% compared to the SSF18 baseline.
The architectures of the four SSF models are described in detail in Appendix C.
B-frame base model: B-EPIC
We also consider a less constrained setting, in which frames can be compressed as B-frames as well, i. e. using both a previous frame and a future frame as reference points.
This ﬂexibility allows for even more efﬁcient compression and is used for instance for ondemand video streaming.
As a base model we choose the B-EPIC architecture (Pourreza & Cohen, 2021).
Again, the video is split into GoPs.
The very ﬁrst frame of a video is modeled as an Iframe, while the last frame in each GoP is a P-frame, using the last frame of the preceeding GoP as reference; both I-frame and P-frames are compressed as in the SSF model.
Any other frame x i is modeled as a B-frame: it is assigned a past reference frame x j ( j < i ) and a future reference frame x k ( k > i ); an off-the-shelf Super-SloMo frame interpolator (Jiang et al., 2018) is used to interpolate between these two reference frames.
The interpolated frame is then used as a basis for scale-space ﬂow warping:
[Equation]
The interpolated frame provides a more useful starting point for optical-ﬂow warping than the previous frame, e. g. because the combination of past and future knowledge may avoid occlusion effects.
Again, I-frames, optical ﬂow, and residuals are modeled with three separate hyperprior models and the model is trained on the RD loss in Eq.
We only consider a single conﬁguration and use hyperparameters and checkpoints from Pourreza & Cohen (2021).
This model has 38.5 million parameters, 23.2 million of which are on the decoder side.
Encoding a video sequence Our procedure conceptually follows van Rozendaal et al.
For completeness we shall describe the full method here.
A video sequence x is compressed by:
Finetuning the model parameters ( θ, φ ) of the base model on the sequence x using Eq.
computing the latent codes z ∼ q φ ( z | x ) , 3. parameterizing the ﬁnetuned decoder and prior parameters as updates δ = θ − θ D , 4.
quantizing latent codes z and network updates δ , and 5.
compressing the quantized latents ¯ z and updates ¯ δ with entropy coding to the bitstream.
To ﬁnetune a pretrained base compression model, we start with the global model with parameters ( θ D , φ D ) and minimize the rate-distortion loss given in Eq.
(2) — but only over the single video sequence x .
This modiﬁed rate-distortion loss explicitly includes the bitrate required to send model updates δ under an update prior p ( δ ) .
We compute the regularizing R θ loss with the unquantized updates δ , but use the quantized parameter updates θ D + ¯ δ to calculate the D and R
z loss terms, using a straight-trough estimator (Bengio et al., 2013) in the backward pass.
Sending the updated network parameters of course adds to the length of the bitstream.
This is even the case when ﬁnetuning does not lead to changed parameters ( ¯ δ = 0 ).
Given the large size of the neural model we consider, it is therefore essential to choose an update prior that assigns a large probability mass to zero-updates ¯ δ = 0 .
This allows the network to transmit trivial updates at a negligible rate cost, while giving it the freedom to invest bit cost in non-trivial parameter updates that improve the performance substantially.
We use a spike-and-slab prior (Johnstone & Titterington, 2009; van Rozendaal et al., 2021), a mixture model of a narrow and a wide Gaussian distribution given by
[Equation]
where the “slab” component with variance σ 2 keeps the bitrate cost for sizable updates down, and the “spike” component with the narrow standard deviation s (cid:28) σ ensures cheap zero-updates.
The mixing weight α is a tunable hyperparameter.
At the beginning of the ﬁnetuning procedure, our neural model is equal to the global model.
Because of the spike-slab update prior, the rate cost is only marginally increased and the compression performance essentially equal to the global model.
During ﬁnetuning, the rate-distortion performance gradually improves, giving us an anytime algorithm that we can stop prematurely to get the best compression performance within a given encoder compute budget.
After the model has converged or a compute budget has been exhausted, we use the ﬁnetuned encoder q φ to ﬁnd the latent code z corresponding to the video sequence x .
Both z and the updates to prior and decoder δ , which will be necessary to decode the video sequence, are quantized.
To discretize the updates δ , we use a ﬁxed grid of n equal-sized bins of width t centered around δ = 0 .
We choose n such that p ( − n/t < δ < n/t )
≈ [Equation] − (cid:15) and clip values at the tails.
The quantization of z is analogous, except that we use a bin width of t = 1 and do not clip the values at the tails (in line with Ball´e et al.
Finally, we write the quantized updates ¯ δ and quantized codes ¯ z to the bitstream.
We use entropy coding under the quantized update prior p (¯ δ ) and ﬁnetuned prior p θ D
+¯ δ (¯ z ) , respectively.
For latent codes in the tail region we use Exp-Golomb coding (Wiegand et al., 2003).
Decoding a video sequence The receiver ﬁrst decodes the prior and decoder updates ¯ δ from the bitstream.
Once ¯ θ = θ D + ¯ δ is known on the decoder side, the latents z are decoded with the prior p ¯ θ ( z ) and the video sequence is reconstructed with the decoder p ¯ θ ( x | z ) following standard VAE protocol.
The only overhead on the decoder side is therefore the initial decoding of ¯ δ .
While this results in a small delay before the ﬁrst frame can be decoded, in practice this delay is very short (below 0.2 seconds in our experiments).
E XPERIMENTAL SETUP
Datasets In our experiments we use sequences from ﬁve different datasets.
The global models are trained on Vimeo90k (Xue et al., 2019).
We evaluate on the HEVC class-B test sequences (HEVC, 2013), on the UVG-1k (Mercat et al., 2020) dataset, and on Xiph-5N (van Rozendaal et al., 2021), which entails ﬁve Xiph.org test sequences.
The performance on out-of-distribution data is tested on two sequences from the animated short ﬁlm Big Buck Bunny, also part of the Xiph.org collection (Xiph.org).
We choose an “easy” (BBB-E) and a “hard” (BBB-H) clip of 10s each, see
Appendix B for details.
All of these datasets consist of videos in Full-HD resolution ( 1920 ×pixels).
In addition, we evaluate on HEVC class C with a lower resolution of 832 × 480 pixels.
Global models
The scale-space ﬂow models described in Sec.
3 are trained with the MSE training setup described in Agustsson et al.
(2020), except that we use the publicly available Vimeo-90k dataset.
For the B-EPIC model, we use the model trained by Pourreza & Cohen (2021) on Vimeo90k.
We describe the training schemes in detail in Appendix D.
Instance-adaptive ﬁnetuning On each instance, we ﬁnetune the models with the InstA objective in Eq. 3, using the same weight β as used to train the corresponding global model.
We ﬁnetune for up to two weeks, corresponding to an average of 300 000 steps.
Our training scheme is described in detail in Appendix D.
Baselines As our instance-adaptive video compression method can be applied to any compression architecture, we focus on comparing to the respective base models: we compare InstA-SSF to our reimplementation of the SSF model (Agustsson et al., 2020) and InstA-B-EPIC to the B-EPIC model (Pourreza & Cohen, 2021).
To better understand where the beneﬁts of instance-adaptive ﬁnetuning come from, we consider an additional SSF18 baseline in which only the encoder is ﬁnetuned on each instance.
Unlike our InstA method, encoder-only ﬁnetuning does not require sending any model updates in the bitstream.
In addition, we show the strongest or most closely related neural baselines, with further baselines in Appendix F.
We also run the popular classical codecs H.264 (AVC) (Wiegand et al., 2003) and H.
(HEVC)
(Sullivan et al., 2012) in the ffmpeg (x264 / x265) implementation (FFmpeg; VideoLAN, a;b) as well as the (slower but more effective) HM reference implementation (HEVC) of H.265.
To ensure a fair comparison, in the low-latency experiments we restrict all codecs to I-frames and
P-frames and ﬁx the GoP size to 12.
In the B-frame experiments, we use default settings, allowing the codecs to freely determine frame types and GoP size.
For ffmpeg, we compare several different encoder presets to study the trade-off between encoding time and RD performance.
In Appendix E we provide the exact commands used to generate these baseline results.
Metrics We evaluate the ﬁdelity of the reconstructions through the peak signal-to-noise ratio (PSNR) in RGB space.
To summarize the rate-distortion performance in a single number, we compute the Bjøntegaard Delta bitrate (BD-rate) (Bjøntegaard, 2001), the relative rate difference between two codecs averaged over a distortion range.
We evaluate the BD-rate savings of all methods relative to the HM reference implementation of HEVC, using PSNR as distortion metric, the intersection of the support of all RD curves as the distortion range, and cubic spline interpolation.
R ESULTS
Compression performance Figure 1 shows the rate-distortion curves of our instance-adaptive video codec (InstA) as well as neural and traditional baselines.
We compare to more baselines in Appendix F. Both for SSF in the low-latency and B-EPIC in the B-frame setting, the instanceadaptive models clearly outperform the corresponding base models.
Finetuning only the encoder — which does not require sending model updates in the bitstream — leads to a much more modest improvement over the base models.
In the low-latency setting, InstA-SSF outperforms all other neural models as well as the ffmpeg implementations of H.264 and H.265 at all studied bitrates, except for OTU and ELF-VC, which hold an edge in the low-bitrate region, and M-LVC, which is competitive at medium bitrates.
Within the restrictions of the low-latency setting and at medium to high bitrates, it is even competitive with the HM reference codec.
In the B-frame conﬁguration, InstA-B-EPIC outperforms all baselines from around 0.1 bits per pixel except HM.
Robustness to model-size reduction While the full-size InstA-SSF18 model leads to the best RD performance in the low-latency setting, the smaller InstA-SSF5 models hold their own.
Despite the 72% reduction in decoder complexity, Insta-SSF5 outperforms most neural baselines and the ffmpeg codecs, and achieves a performance close to that of InstA-SSF18.
In comparison, the gap between the SSF5 and SSF18 baselines is larger.
This provides evidence for the hypothesis that instance-adaptive ﬁnetuning reduces the capacity requirements on neural compression models.
To further investigate how the performance of our method depends on decoder complexity, we train two additional models, SSF3 and SSF8, as summarized in Tbl.
The results are in line with the trends already shown.
While the performance of the global models degrades quickly when reducing the model size, instance-adaptive ﬁnetuning can compensate for the reduced capacity and offers more “bang for the buck” in terms of (decoding-side) model complexity.
For detailed results we refer the readers to Appendix H.
Robustness to dataset variation Globally trained neural codecs rely on the similarity of the test instances to the training data.
In comparison, we expect our ﬁnetuned models to be more robust under domain shift and to perform well across different test sets.
To put this to the test, we evaluate the InstA-SSF models and low-latency baselines on different sequences, including two animated [Figure] scenes from the Big Buck Bunny video.
In Tbl.
2 we report the compression performance as BDrate savings relative to the HM reference codec, the corresponding RD curves can be found in
Appendix F. We ﬁnd that InstA performs strongly on all datasets.
On the animated sequences, the global SSF18 baseline performs poorly relative to ffmpeg H.265, showing its susceptibility to domain shift.
Our instance-adaptive models are much more robust to this shift and close the gap with classical codecs.
We illustrate this performance improvement on example frames from the Big Buck
Bunny video in Appendix G and provide reconstructed video data in the supplementary material.
Instance-adaptive compression is particularly powerful for high-resolution data, where the modelrate overhead is amortized over more pixels.
We test whether InstA still works at a lower resolution by evaluating on the HEVC class C test sequences, which have 81 % less pixels per frame than the
Full-HD sequences.
As shown in Appendix F, InstA indeed performs worse relative to the baselines than on the higher-resolution data.
Nevertheless, InstA-SSF18 still is among the best neural codecs.
To further quantify our robustness to test data characteristics, we investigate the effect of the video framerate on the model performance.
In Appendix H we study the performance of global and instance-adaptive SSF models on temporally subsampled videos and again ﬁnd that the instanceadaptive model is more robust to variations than the global SSF model.
While these ablations do not cover all realistic variations, they serve as examples of dataset characteristics that a global neural codec is likely sensitive to.
[Table]
Rate composition It is remarkable that InstA consistently improves the rate-distortion performance of neural models despite having to send updates for millions of neural network parameters.
We investigate this low rate overhead in Appendix H. We ﬁnd that the spike-and-slab prior is crucial: compared to Gaussian coding, it allows us to save around 7.5 bits per parameter.
InstA learns to distribute bits on model updates and latents.
The optimal trade-off generally depends on the bitrate, the size of the model and latent code, and on the test data: we expect that for strongly out-of-distribution data, more bits will be spent on model updates.
Indeed we ﬁnd that in almost all cases, less than 1% of the total rate is spent on model updates, while for the animated sequences from the Big Buck Bunny video between 1% and 3% of the total bitrate are invested in model updates.
We provide more details in Appendix H.
Computational complexity The improved compression performance of our instance-adaptive video codec comes at the price of increased encoding time.
In Fig.
2 we illustrate this trade-off for the low-latency setting.
We summarize the rate-distortion performance as BD-rate savings relative to HM and compare to the encoding time 1 .
For our instance-adaptive methods, this time includes the ﬁnetuning in addition to the forward pass of the encoder, the quantization, and the entropy coding (for both the latents and the parameters).
By choosing how long to run the ﬁnetuning, we can trade off RD performance against the encoding time.
Out of the studied codecs, ours can best exploit this trade-off: encoding time and compression performance are ﬁxed in other neural methods, while the existing handles in traditional codecs have a limited effect on performance.
We ﬁnd that instance-adaptive ﬁnetuning leads to state-of-the-art RD performance even when ﬁnetuning only for a few steps.
The performance gradually improves over the course of training and eventually ﬂattens out.
Care has to be taken in interpreting these results as the different codecs were designed for different hardware and our algorithms have not been optimized for performance.
Instance-adaptive video compression also adds some overhead on the decoder side, as the model parameters need to be decoded before any frame can be reconstructed.
For the full-size InstA-SSF model and without optimizing the code for performance, this initial delay is below 0.2 seconds.
The neural network forward pass alone is timed with a speed of around 0.01 seconds per frame on a
TeslaV100 GPU without any optimization.
Both the overhead and the network forward pass are thus negligible compared to the time it takes to entropy-decode the latents z : in our (not optimized) implementation, this takes around 1 second per frame, both for the global SSF models and InstASSF.
Compared to the neural baselines, InstA-SSF thus yields a better performance at comparable complexity.
Compared to traditional codecs, there is still a gap to bridge, and none of the neural codecs can decode at the framerate of the videos yet.
[Figure]
r e l a t i v e t
[Figure]
M [
[Figure] [Figure]
We summarize the RD performance as BD-rate difference relative to the HM implementation of
H.265 (lower is better).
InstA-SSF (red) is compared to SSF (orange) and traditional (grey) baselines.
Some ﬂuctuations in the measured walltime are due to varying load on our computational cluster.
Since we can stop the instance ﬁnetuning at any time, InstA-SSF allows us to continuously trade off encoder compute for compression performance.
At the same time, instance-adaptive ﬁnetuning allows us to use smaller architectures while maintaining a strong performance, as shown above.
This potential to reduce the decoder-side compute may outweigh the small overhead from the initial update decoding.
C ONCLUSIONS
We introduced InstA, a method for video compression that ﬁnetunes a pretrained neural compression model on each video sequence to be transmitted.
The ﬁnetuned network parameters are compressed and transmitted along with the latent code.
The method is general and can be applied to many different settings and architectures, we demonstrated it on scale-space ﬂow models in the low-latency setting as well as a state-of-the-art B-frame architecture.
Instance-adaptive video compression relaxes the requirement of a single global model that needs to be able to generalize to any unseen test sequence.
This can be beneﬁcial when the training data is limited, when there is domain shift between training and test distributions, or when the model class is not expressive enough.
While it comes at the cost of some overhead to the bitstream, we demonstrated that with a suitable compression scheme this overhead can be kept small, especially since it is amortized over the many pixels of a video sequence.
Our InstA models clearly outperform the base models in all considered settings, in several cases achieving a state-of-the-art rate-distortion performance.
They outperform their corresponding global base models with BD-rate savings of between 17 % and 26 % on the UVG-1k, HEVC class-B, and
Xiph-5N datasets, and the popular ffmpeg implementation of H.265 with BD-rate savings between 5 % and 44 %.
By evaluating models that were trained on natural scene data on animated sequences, we demonstrate that instance-adaptive ﬁnetuning improves the robustness of neural video codecs under domain shift.
Finally, instance-adaptive compression reduces the required model expressivity: even after reducing the model size by 72 %, InstA-SSF maintains a competitive performance.
The ability to trade off encoder-side compute for compression performance is a natural match for one-to-many transmission scenarios, in which a sequence is compressed once but transmitted and decoded often.
We demonstrated that even just ﬁnetuning for seconds per frame can lead to a substantial improvement in compression performance, so instance-adaptive compression may ultimately even be beneﬁcial in real-time applications. A S UMMARY OF I NSTANCE -A DAPTIVE COMPRESSION METHODS [Table] [Table]
In Tbl.
3 we categorize existing methods for neural instance-adaptive data compression. The categories are based on the extent of adaptation and we shall describe them one-by-one in the following paragraphs. Encoder-only instance-adaptive methods only optimize the encoder q φ (Yang et al., 2020c; Lu et al., 2020) or directly optimize the latents z (Campos et al., 2019; Guo et al., 2020) at inference time. These methods can decrease the amortization gap (the problem that the encoder is not a good ﬁt for the given decoder)
(Cremer et al., 2018).
However, as they cannot change any components required on the receiver side, their adaptation potential is fundamentally limited.
The next group of methods are the hybrid instance-adaptive compression methods.
These consist of a traditional codec which is ran at a low quality setting (Lam et al., 2019;
He et al., 2020; Klopp et al., 2020; He et al., 2021) or a low resolution (Klopp et al., 2021;
Liu et al., 2021), followed by a neural enhancement network which is overﬁtted to the test instance (and signalled in the bitstream).
The advantage of this approach is that it can build on established, high-performing traditional codecs such as the HM reference implementation of H.265.
As such, the rate-distortion performance of these methods can be very good, especially at lower bitrates, where neural codecs tend to perform more poorly (see Fig. 4).
The size and computational complexity of these enhancement networks is typically very low.
However, it should be noted that decoding the video still requires running the expensive base codec ﬁrst.
In contrast, we present a smaller neural codec InstA-SSF5 which does not require any compute besides running the neural decoder.
Another downside of the hybrid approaches is that end-to-end neural video codecs are continuously improving.
While our instanceadaptive method can be applied to any neural base model, it is not clear how hybrid approaches can beneﬁt from the developments in this ﬁeld.
The next group we consider consists of fully neural codecs that adapt the decoder and prior networks to each instance, albeit to a limited extent.
The works we consider (Aytekin et al., 2018; Zou et al., 2020; Wang et al., 2021)
all show results on image compression, with autoencoders that operate on patches or blocks of the image.
At training time, these methods train an ensemble of different decoders, which are then frozen as a part of the codec.
At test time, the optimal decoder is selected for every patch and communicated to the receiver.
This is often combined with encoder-side ﬁnetuning for increased rate-distortion performance.
Although this approach allows changing of all decoderside parameters, the actual degrees of freedom are very limited (up to 255 different possibilities in
Zou et al.
Furthermore, since the different possible decoders are determined during training these approaches is not able to solve the domain gap problem fully.
Finally we turn to the full-model instance-adaptive methods, in which all components of a neural codec (encoder, decoder, and prior) are overﬁtted at test time, and the changes to the decoder are signalled in the bitstream.
Due to the large size of neural codecs, one has to make sure that the overhead for signalling model updates will not dominate the bitrate.
This can be achieved by choosing a very small base model (Mikami et al., 2021) such that the rate overhead is minimized, but this does not scale to the architectures required for SOTA performance.
Instead van Rozendaal et al.
propose adapting the codec with a model-rate aware loss function, which includes a sparsity-inducing prior.
The authors only reported results on an I-frame compression task on a single dataset.
We are the ﬁrst to show that this type of full-model adaptation is highly efﬁcient for the compression of full video sequences with P-frames or B-frames.
B D ATASETS
UVG-1k
Following Refs.
(Wu et al., 2018; Lu et al., 2019; Golinski et al., 2020; Agustsson et al., 2020), we report results on the UVG-1k dataset, which consists of seven test video sequences totalling 3900 frames captured at 120 fps and natively available in 1080p resolution.
Note that these sequences are only a subset of the full UVG dataset (Mercat et al., 2020).
Xiph-5N We use the Xiph-5N dataset introduced in van Rozendaal et al.
It consists of the following ﬁve sequences from the Xiph.org collection: “in to tree”, “aspen”, “controlled burn”, “sunﬂower”, and “pedestrian area”.
Big Buck Bunny (BBB-E and BBB-H)
From the Big Buck Bunny video, which is also part of the Xiph.org media collection, we extract two subsequences of 10 s length.
They are chosen based on the frame-wise RD loss achieved by a global SSF18 model (with β = 0 . 0016 ).
We select the 10 s window with the lowest RD loss and the 10 s window with the highest RD loss, excluding the intro and outro of the video.
These signify the “easiest” and “hardest” parts of the video to compress and we refer to them as “BBB-E” and “BBB-H”, respectively.
In Fig.
3 we show the frame-wise RD loss and the selected sequences. [Figure] [Table]
C M ODEL ARCHITECTURES
Full-size SSF model We ﬁrst apply instance-adaptive video compression to the scale-space ﬂow (SSF) architecture proposed by Agustsson et al.
(Agustsson et al., 2020), using our own reimplementation of the model.
We mostly follow the setup described in Agustsson et al.
but there are some minor differences.
In the hyperencoder and hyperdecoder networks we use a slightly different architecture.
We list the architectures in Tbl. 4.
In addition, our implementation of the blur stack and of the scale-space-ﬂow warping operation differ from the one used by Ref. 4, but we have veriﬁed in experiments that this has a negligible effect.
We empirically validated our SSF re-implementation by comparing its performance to the original
SSF (using entropy-coding).
On the UVG-1k dataset, our SSF18 model performed slightly better than the results published by Agustsson et al.
Smaller SSF models In addition to the full-size SSF18 model, we consider three smaller scalespace ﬂow architectures.
They follow the same structure as SSF18 in Tbl.
4 but differ in the number of latent variables and convolutional channels, as shown in Tbl.
We refer to them as SSF8, SSF5, and SSF3 after the number of decoder-side network parameters (in millions).
B-EPIC model
As a second base model we use the B-frame codec developed by Pourreza &
Cohen (2021) (B-EPIC).
We follow the hierarchical IBP conﬁguration presented in that paper and use the model trained by the authors.
During training, a GoP consists of 4 frames arranged in an
IBBP pattern.
At test time, we use a GoP size of 12 frames and the following reference structure (where frame 0 refers to the last frame of the previous GoP):
• Frame 12: P-frame referring to frame• Frame 6: B-frame referring to frames 0 and• Frame 3: B-frame referring to frames
and• Frame 9: B-frame referring to frames 6 and• Frame 1: B-frame referring to frames 0 and• Frame 2: B-frame referring to frames 1 and [Table]
and so on for the remaining frames.
The very ﬁrst frame in a video is encoded as an I-frame.
Encoding and decoding in this order ensures that the reference frames will always be encoded or decoded before the frames for which they are needed.
Quantization settings Ablation studies showed insensitivity to network prior settings in the ranges of 10 ≤ α ≤ 1000 , 0 .
≤ t ≤ 0 .
005 , and 0 .
01 ≤ σ ≤ 0 .
For the experiments in this paper we use a bin width t = 0 .
001 , σ = 0 .
05 , s = t/ 6 , a spike-slab ratio α = 100 , and (cid:15) =
2 − 8 .
Entropy coding We implement an entropy coder for the latents and model updates in the SSF and
InstA-SSF models.
We veriﬁed that this achieves bitrates in agreement with the probability mass functions of p ( z ) and p ( δ ) .
For the B-EPIC and InstA-B-EPIC models, we report bitrates based on probability mass functions without actually implementing an entropy coder.
D T RAINING SCHEME
Global models
The scale-space ﬂow models described in Sec.
3 are trained with the MSE training setup described in Agustsson et al.
(2020), except that we use the publicly available Vimeo-90k dataset.
The D loss is computed with (integer) quantized latents ¯ z , while for the R z loss term we use noisy samples from U ( z − 12 , z + 12 ) .
Following Agustsson et al.
(2020), we ﬁrst train formillion steps on 256 × 256 crops with a learning rate of 10 − 4 .
We then conduct the MSE “ﬁnetune” stage of the training procedure from Agustsson et al.
(2020) (not to be confused with instanceadaptive ﬁnetuning) for the SSF18 model, where we train on crops of size h × w = 256 × 384 with a learning rate of 10 − 5 .
The models are trained with a GoP size of 3 frames, which means that we split the training video into chunks of 3 frames and randomly sample chunks during training.
We ﬁnally evaluate the models with a GoP size of 12.
This gives similar results to using inﬁnite unrolling at test time, as we demonstrate in Appendix C.
For the B-EPIC model, we use the model trained by Pourreza & Cohen (2021) on Vimeo-90k.
The setup is similar to that for the SSF models except that B-EPICs more complicated GoP structure requires training with a GoP size of 4 frames.
At test time we use a GoP size of 12, the frame conﬁgurations are described in Appendix C.
Instance-adaptive ﬁnetuning In the low-latency scenario we use a GoP size of 3 and ﬁnetune on full-resolution frames ( 1920 × 1080 pixels) with a batch size of 1 and a learning rate of 10 − 5 .
After ﬁnetuning, we transmit sequences with a GoP size of 12.
The prior and decoder updates δ = θ − θ D are quantized and coded under the spike-and-slab network prior described in Sec. 3.
While the SSF model follows a stationary distribution over frames, the B-EPIC model has a more complex frame structure: the distance between a frame and its reference points can vary betweento 6 frames.
For this reason, we found it beneﬁcial to use a GoP of 12 for InstA ﬁnetuning of the
B-EPIC model.
Due to memory limitations, we ﬁnetune on horizontal crops of size 256 × 1080 pixels instead of full-resolution frames.
For the low-bitrate working points use a learning rate of 10 − 5 , in line with Pourreza & Cohen (2021).
For the high-bitrate region ( β ≤ 0 . 0008 ), we found that we can get faster convergence by using a learning rate of 5 · 10 − 5 .
E C LASSICAL CODECS
We generate H.265 and H.264 results using version v3.4.8 of ffmpeg (FFmpeg).
When comparing to neural codecs in the low-latency setting, we also choose a low-latency setting that only allows Iframes and P-frames and ﬁx the GoP size to 12.
We use various encoder presets.
Example commands are ffmpeg -pix_fmt yuv420p -s 1920x1080 \ -r 120 -i raw_video.yuv \ -c:v libx264 -preset medium \ -crf 23 -tune zerolatency \ -x264-params \ "keyint=12:min-keyint=12:verbose=1" \ output.mkv for H.264 and ffmpeg -pix_fmt yuv420p
-s 1920x1080 \ -r 120 -i raw_video.yuv \ -c:v libx265 -preset medium \ -crf 23 -tune zerolatency \ -x265-params \ "keyint=12:min-keyint=12:verbose=1" \ output.mkv for H.265.
We also compress the test sequences with the HM reference implementation of HEVC, using the “lowlatencyP” setting with a ﬁxed GoP size of 12.
As baselines for the neural B-frame codecs, we also relax the GoP and frame-type restrictions for the classical codecs, using the default random-access settings with B-frames and variable GoP size.
F A DDITIONAL RATE - DISTORTION RESULTS
In Fig.
4, we show the rate-performance performance of our models relative to the HM baseline.
In addition to the classical codecs and our base models, we compare to several additional neural baselines: Lu et al.
(2019; 2020); Rippel et al.
(2021); Lin et al.
(2020); Klopp et al.
; Yang et al.
(2020a;c);
Hu et al.
(2021), and Hu et al.
Relative to classical codecs, almost all neural codecs perform better at higher bitrate.
The exception is Klopp et al.
(2020), which consists of a classical codec followed by neural enhancement.
With the exception of this method and in some bitrate regions ELF-VC
(Rippel et al., 2019) and M-LVC (Lin et al., 2020), our InstA method achieves a state-of-the-art neural compression performance.
Figures 5 and 6 show rate-distortion curves on the HEVC class C, Xiph-5N, and Big Buck Bunny datasets.
The test sequences of HEVC class C have a lower resolution of only 832 × 480 .
This is a challenging scenario for InstA, as the model rate overhead is amortized over less pixels than in higher-resolution videos.
Indeed, we ﬁnd that our method does not perform as strongly relative to classical and some neural baselines as on the higher-resolution datasets.
Nevertheless, it still leads to a sizable improvement over the SSF18 base model.
Finally, in Tbl.
6 we show the BD-rate savings of the InstA-SSF18, InstA-SSF5, and SSF18 models relative to two different H.265 implementations and relative to the SSF18 model. [Figure] [Figure] [Table]
[Table]
[Figure]
BD-rate savings are computed over a certain distortion range.
We choose the largest possible range for which we have results for our methods as well as the main baselines.
These are:
• UVG-1k: 33 . 5 dB ≤ PSNR ≤ 40 .
4 dB • HEVC class B: 30 . 3 dB ≤ PSNR ≤ 37 .
4 dB • HEVC class C: 27 . 2 dB ≤ PSNR ≤ 35 .
4 dB • Xiph-5N: 31 . 4 dB ≤ PSNR ≤ 39 .
5 dB • BBB-E: 29 . 3 dB ≤ PSNR ≤ 37 .
dB • BBB-H: 31 .
4 dB ≤ PSNR ≤ 39 .
5 dB
G P ERCEPTUAL QUALITY
We illustrate the performance on out-of-distribution data in Fig.
7, where we show original and compressed versions of two frames from the Big Buck Bunny video.
Both traditional codecs and the SSF18 baseline lose details and introduce artifacts at the chosen low-bitrate setting.
While the
InstA-SSF18 compressions still suffer from some blurriness, artifacts are less pronounced than in the global model.
In the supplementary materials, we attach the corresponding reconstructed video sequences.
To highlight the differences between the codecs, we again use a very low bitrate for all codecs (even though our InstA method is strongest at higher bitrates).
H A BLATION STUDIES
Model size ablation In the left panel of Fig.
8 we compare the compression performance of globally trained SSF18, SSF8, SSF5, and SSF3 models, as well as instance-adaptive versions InstASSF18, InstA-SSF8, InstA-SSF5, and InstA-SSF3.
Due to the training time requirement we limit this ablation to a single video, the “in to tree” sequence from the Xiph.org collection.
We ﬁnd that the larger models lead to a better compression performance, with more pronounced difference in the high-bitrate region.
SSF3 and InstA-SSF3 models perform poorly at high bitrates.
We attribute this partially to the small size of the latent space compared to all other models.
In general, we observe that instance-adaptive training improves the performance of models of any complexity. Instance
[Figure] adaption reduces the gap between smaller models and the full-size model, conﬁrming that less expressivity is necessary for good performance on a single instance compared to on the entire data distribution.
GoP size considerations Throughout this work, we use a GoP size of 12 at test time.
To test the robustness of this choice, in the right panel of Fig.
8 we compare to an inﬁnite GoP size (for the
SSF model).
In other words, we compress the ﬁrst frame as I-frame and all remaining frames as
P-frames.
The overall difference is slim, though the gap is larger for some videos than others.
Sensitivity to framerate variations To further quantify our robustness to test data characteristics, we investigate the effect of the video framerate on the model performance.
We study the performance of global and instance-adaptive SSF models on temporally subsampled videos, i. e. keeping one out of every four frames.
To save computational resources, this ablation is only performed on the “Bosphorus” sequence from the UVG-1k dataset.
The left panel of Fig.
9 shows our results.
While the SSF18 performance varies greatly between the original and the subsampled video, the difference in performance is much smaller for InstA-SSF18.
In addition, both models perform better in the 120 fps scenario due to the higher redundancy across frames.
Bitrate allocation Figure 10 disentangles the various contributions to the total bitrate.
We ﬁnd that the overhead from model updates is generally small (typically well below 1 % ), except on videos that differ substantially from the training data: on the animated Big Buck Bunny sequences, InstA learns to adapt more strongly and up to 3 % of the total rate are spent on model updates. We further break down the model rate in Fig. 11, where we show how the model rate is distributed over the various auto-encoders and submodules of our InstA-SSF18 and InstA-SSF5 model. It can
[Figure]
[Figure] be seen that the InstA-SSF5 model has a lot fewer parameters per module by design, but it is making up for its smaller capacity by spending a lot more bits/per parameter on model updates compared to InstA-SSF18 (often more than double). This explains why the rate-distortion performance of
InstA-SSF5 is close to that of InstA-SSF18. Even though the InstA-SSF5 model needs more bit per parameter to adapt to the data instances, the total bitrate spend on model updates is still smaller than that of InstA-SSF18. Focusing on the distribution of the model bitrate across auto-encoders, we see that more than half of this rate is allocated to the I-frame model. The ﬂow autoencoder is the next biggest consumer
[Figure]
a c t i o n
[Figure]
f [Figure] h [Figure]
a l r
[Figure]
t
[Figure]
of model rate
, while only few bits are spent to communicate updates to the residual autoencoder.
In terms of submodules, it turns out that even though the hyper-decoder has a lot more parameters than the decoder, the decider receives substantially larger update bits per parameter.
Together, these ﬁndings suggest that instance adaptation is most important for network components that are closer to the data space in the computational graph.
Update sparsity analysis The key design for achieving low model rate is in update sparsity.
We achieve this by using a spike-and-slab prior around global model parameter values.
Training with the spike-and-slab prior is sparsity enforcing (on average around 10 % of all quantized parameters receive a zero update).
To quantify savings of our chosen spike-and-slab prior, we compute the cross-entropy of our prior with the resulting model δ after training.
This is the number of bits we need to transmit the model update.
On the other hand, assume we had a Gaussian prior with same variance as our “slab” component, and assuming the resulting δ follows the same distribution, the entropy of this Gaussian prior gives the expected number of bits to transmit non-sparse model update.
We estimate the difference of the two scenarios to be around 7.5 bits per parameter.
Finetuning strategy for longer videos For longer videos with multiple scenes, one can consider several different strategies for instance-adaptive ﬁnetuning.
The simplest approach is ﬁnetuning a single model on the full video.
It may be beneﬁcial to instead ﬁnetune the model independently on each scene.
Even a hierarchical approach in which a model ﬁnetuned on the full video is further ﬁnetuned on each scene is an option.
The optimal strategy will in general depend on the particular video content and the similarity of the scenes as well as on the bitrate setting.
We perform a simple ﬁrst test by ﬁnetuning a single model on the entire Big Buck Bunny video, a 10 min video (14315 frames) with multiple scenes united by a common style.
The results in the right panel of Fig.
9 show that such a single ﬁnetuned model leads to clear RD improvements over the global SSF model.
However, in the large-bitrate region these improvements are smaller than the gains we achieve by ﬁnetuning on the BBB-E and BBB-H scenes individually.
H y p
e r P r i o r
I-frame 108 B
Flow 149 B Residual 193 B I-frame 93 B
Flow 123 B
Residual 152 B 0.
H y p
e r D e c o d e r 16.2 kB 12.0 kB 12.5 kB 8.7 kB 4.7 kB 5.4 kB    4 0. D e c o d e r ( b i t s / p a r a m e t e r ) 53.6 kB    4 #params (M) 8.2 kB     33.9 kB  47.6 kB 1 #params (M) 6.6 kB  31.9 kB InstA-SSF18 (136.9 kB) InstA-SSF5 (105.2 kB)
I E THICS STATEMENT
Video compression affects the life of millions of people and any algorithm and model should be carefully analyzed for potential ethical issues before deployment.
In particular, it is important to check for harmful biases, which may be due to imbalanced training data, but can also be caused or exacerbated by algorithmic choices.
To some extent, the instance-adaptive video compression method we are proposing may be helpful in combating such biases, as it can improve the compression performance on data that is different from typical samples in the training set.
However, it cannot solve the problem entirely, as more pronounced instance adaptation will increase the bitrate.
The environmental impact of our research is also relevant.
The energy cost of our experiments directly contributes to the climate catastrophe.
In the long term, our work may also have a beneﬁcial aspect, as it allows us to lower the bitrate and thus the energy cost of video streaming, which already now make up the majority of internet trafﬁc.
J R EPRODUCIBILITY STATEMENT
Our instance-adaptive video compression algorithm is described in detail in Sec.
The most complex components are the base compression models, which are described in more detail in the references we cite as well as in appendix C. With the details we provide, instance-adaptive video compression can be implemented and used for any base model with relatively little effort.
Finally, in the supplementary material we provide the performance of the various methods on each video sequence as a CSV ﬁle to aid comparisons.