VCT: A Video Compression Transformer
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We show how transformers can be used to vastly simplify neural video compression.
Previous methods have been relying on an increasing number of architectural biasesand priors, including motion prediction and warping operations, resulting in complex models.
Instead, we independently map input frames to representations anduse a transformer to model their dependencies, letting it predict the distribution offuture representations given the past.
The resulting video compression transformeroutperforms previous methods on standard video compression data sets.
Experiments on synthetic data show that our model learns to handle complex motionpatterns such as panning, blurring and fading purely from data.
Our approach iseasy to implement, and we release code to facilitate future research.
Introduction
Neural network based video compression techniques have recently emerged to rival their non-neuralcounter parts in rate-distortion performance [ e.g. , 1, 16, 29, 41].
These novel methods tend to incorporate various architectural biases and priors inspired by the classic, non-neural approaches.
Whilemany authors tend to draw a line between “hand-crafted” classical codecs and neural approaches,the neural approaches themselves are increasingly “hand-crafted”, with authors introducing complexconnections between the many sub-components.
The resulting methods are complicated, challengingto implement, and constrain themselves to work well only on data that matches the architecturalbiases.
In particular, many methods rely on some form of motion prediction followed by a warpingoperation [ e.g. , 1, 16, 18, 22, 41, 39].
These methods warp previous reconstructions with the predictedﬂow, and calculate a residual.
In this paper, we replace ﬂow prediction, warping, and residual compensation, with an elegantlysimple but powerful transformer-based temporal entropy model.
The resulting video compressiontransformer (VCT) outperforms previous methods on standard video compression data sets, whilebeing free from their architectural biases and priors.
Furthermore, we create synthetic data to explorethe effect of architectural biases, and show that we compare favourably to previous approaches onthe types videos that the architectural components were designed for (panning on static frames,or blurring), despite our transformer not relying on any of these components.
More crucially, weoutperform previous approaches on videos that have no obvious matching architectural component(sharpening, fading between scenes), showing the beneﬁt of removing hand-crafted elements andletting a transformer learn everything from data.
We use transformers to compress videos in two steps (see Fig. 1):
First, using lossy transformcoding [3], we map frames x i from image space to quantized representations y i , independently foreach frame .
From y i we can recover a reconstruction ˆ x i .
Second, we let a transformer leveragetemporal redundancies to model the distributions of the representations.
We use these predicteddistributions to losslessly compress the quantized y i using entropy coding [42, Sec 2.2.1].
The betterthe transformer predicts the distributions, the fewer bits are required to store the representations.
(cid:1) (cid:2)(cid:3) (cid:1) (cid:1) (cid:2)(cid:3) (cid:2) (cid:1) (cid:2) (cid:4)
[Table]
(cid:1) (cid:4)
[Table] (cid:2) (cid:1) (cid:1) [Table]
[Figure]
This setup avoids complex state transitions or warping operations by letting the transformer learn toleverage arbitrary relationships between frames.
As a bonus, we get rid of temporal error propagationby construction since the reconstruction ˆ x i does not depend on previous reconstructions.
Contrastwith warping-based approaches, where ˆ x i is a function of the warped ˆ x i  1 meaning that any visualerrors in ˆ x i will be propagated forward and require additional bits to correct with residuals.
VCT is based on the original language translation transformer [34]: We can view our problem as“translating” two previous representations y i  2 , y i  1 to y i .
However, there are various challenges 44 in the way of directly applying the NLP formulation.
Consider a 1080p video frame; using atypical neural image compression encoder [4] that downscales by a factor 16 and has 192 outputchannels, a (1080, 1920, 3)-dimensional input frame is mapped to a (68, 120, 192)-dimensionalfeature representation leading to approximately 1 . 6 million symbols.
Naively correlating all of thesesymbols to all symbols in a previous representation would yield a 1 . 6 M ⇥ 1 .
6 M-dimensional attentionmatrix.
To address this computationally impractical problem, we introduce independence assumptionsto shrink the attention matrix and enable parallel execution on subsets of the symbols.
Our model is easy to implement with contemporary machine learning frameworks, and we providean extensive code and model release to allow future work to build on this direction.
Related Work
Transformers were initially proposed for machine translation [34], where an encoder-decoder structurewas used for to obtain state-of-the-art results.
This led to a wide range of follow-up research, and stateof-the-art natural language processing (NLP) models are still based on transformers [ e.g. , 6, 8, 7, 11].
Motivated by these advancements, Dosovitski et al.
[10] replaced CNNs with a transformer-basedarchitecture to achieve state-of-the-art results in image classiﬁcation, which in turn spurred moreexploration of transformers in the computer vision community including both image tasks [ e.g. ,21, 36, 43] as well as video analysis [ e.g. , 2, 5, 12, 27, 31].
Recently, transformers were incorporated into neural image compression models.
Qian et al.
[28]replaced the autoregressive hyperprior [25] with a self-attention stack, and Zhu et al.
[44] replacedall convolutions in the standard approach [4, 26] with Swin Transformer [21] blocks.
Neural video compression remains CNN-based.
After initial work used frame interpolation [38,9], Lu et al.
[22] followed the more traditional approach of predicting optical ﬂow between theprevious reconstruction and the input, transmitting a compressed representation of the ﬂow, andalso transmitting a residual image to correct visual errors after warping.
Many papers extendedthis approach, for example Agustsson et al.
[1] introduced the notion of a ﬂow predictor thatalso supports blurring called “Scale Space Flow” (SSF), which became a building block for otherapproaches [41, 29].
Rippel et al.
[29] achieved state-of-the-art results by using SSF and morecontext to predict ﬂow.
RNNs and ConvLSTMs were used to build recurrent decoders [13] or entropymodels [39].
[Figure]
DCVC [18] estimated motion in pixel space but performed residual compensation in a feature space.
Liu et al.
[20] also losslessly encoded frame-level representations, but rely on CNNs for temporalmodelling.
Finally, recent work employed GAN losses to increase realism [23, 40].
Method
3.1 Overview and Background
Frame encoding and decoding A high-level overview of our approach is shown in Fig.
Wesplit video coding into two parts.
First, we independently encode each frame x i into a quantizedrepresentation y i =
b E ( x i ) e using a CNN-based image encoder E followed by quantization to aninteger grid.
The encoder downscales spatially and increases the channel dimension, resulting in y ibeing a ( H, W, d C ) -dimensional feature map, where H, W are 16 ⇥ smaller than the input imageresolution.
From y i , we can recover a reconstruction ˆ x i using the decoder D .
We train E, D usingstandard neural image compression techniques to be lossy transforms reaching nearly any desireddistortion d ( x i , ˆ x i ) by varying how large the range of each element in y i is.
For now, let us assumewe have a pair E, D reaching a ﬁxed distortion.
Naive approach After having lossily converted the sequence of input frames x i to a sequence ofrepresentations
y i
= b E ( x i ) e , one can naively store all y i to disk losslessly .
To see why this is suboptimal, let each element y i,j of y
i be a symbol in S =
{  L, . . .
, L } .
Assuming that all |S| symbols 93 appear with equal probability, i.e. , P ( y i,j ) = 1 / |S| , one can transmit y i using H · W · d C · log 2 |S| 94 bits.
Using a realistic L =32 , this implies that we would need 9 MB , or ⇡ 2Gbps at 30fps, to encode 95 a single HD frame ( where H · W · d C ⇡ 1 .
6 M, see Introduction).
While arguably inefﬁcient, this is 96 a valid compression scheme which will result in the desired distortion.
The aim of this work is toimprove this scheme by approximately two orders of magnitude.
An efﬁcient coding scheme Given a probability mass function (PMF) P estimating the truedistribution Q of symbols in y i , we can use entropy coding (EC) to transmit y i with H · W · d C ·
E y ⇠ Q ( y i )
[  log 2 P ( y )] bits.
By using EC, we can encode more frequently occurring values withfewer bits, and hence improve the efﬁciency.
Note that the expectation term representing the averagebit count corresponds to the cross-entropy of Q with respect to P .
Our main idea is to parameterize Pas a conditional distribution using very ﬂexible transformer models, and to minimize the cross-entropyand thus maximize coding efﬁciency.
We emphasize that we use P for lossless EC, we do not samplefrom the model to transmit data.
Even if the resulting model of P is sub-optimal, y i can still be storedlosslessly, albeit inefﬁciently.
[Table]
[Figure]
Why would one hope to do better than the uniform distribution over y i ?
In principle, the modelshould be able to exploit the temporal redundancy across frames, and the spatial consistency withinframes.
3.2 Transformer-based Temporal Entropy Model
To transmit a video of F frames, x 1 , . .
, x F , we ﬁrst map E over each frame obtaining quantizedrepresentations y 1 , . . .
, y F .
Let’s assume we already transmitted y 1 , . . .
, y i  1 .
To transmit y i , weuse the transformer to predict P ( y i |
y i  2 , y i  1 ) .
Using this distribution, we entropy code y i to create 114 a compressed, binary representation that can be transmitted.
To compress the full video, we simplyapply this procedure iteratively, letting the transformer predict P ( y j | y j  2 , y j  1 ) for j 2 { 1 , . . .
, F } , 116 padding with zeros when predicting distributions for y 1 , y 2 .
The receiver follows the same procedureto recover all y j , i.e. , it iteratively calculates P ( y j | y j  2 , y j  1 ) to entropy decode each y j .
Afterobtaining each representation, y 1 , y 2 , . .
, y F , the receiver generates reconstructions.
Tokens When processing the current representation y i , we split it spatially into non-overlappingblocks with size w c ⇥ w c as shown in Fig.
Previous representations y i  2 , y i  1 become correspond-
121 ing overlapping w p ⇥ w p blocks (where w p > w c ) to provide both temporal and spatial context forpredicting P ( y i |
y i  2 , y i  1 ) .
Intuitively, the larger spatial extent provides useful context to predict the 123 distribution of the current block.
Note that all blocks span a relatively large spatial region in imagespace due to the downscaling convolutional encoder E .
We ﬂatten each block spatially (see Fig.
2)to obtain tokens for the transformers.
The transformers then run independently on correspondingblocks/tokens, i.e. , tokens of the same color in Fig.
2 get processed together, trading reduced spatialcontext for parallel execution.
This independence assumption allows us to focus on a single set of blocks, e.g. , the pink blocks in
Fig.
In the following text and in Fig.
3, we thus show how we predict distributions for the w 2 c =16tokens t 1 , t 2 , . . .
, t 16 in block b i , given the 2 w 2 p =128 tokens from the previous blocks
b i  2 , b i  1 .
Step 1: Temporal Mixer We use two transformers to extract temporal information from b i  2 , b i  1 .
132 A ﬁrst transformer T sep operates separately on each previous block.
Then, we concatenate the outputsin the token dimension and run the second transformer, T joint , on the result to mix information acrosstime.
The output z joint is 2 w 2 p features, containing everything the model “knows” about the past.
Step
Block
Autoregression The second part of our method is the masked transformer
T cur , which predicts PMFs for each token using auto-regression within the block.
We obtain apowerful model by conditioning T cur on z joint as well as already transmitted tokens within the block.
For entropy coding, both the sender and the receiver must be able to obtain exactly the same PMFs,i.e. , T cur must be causal and start from a known initialization point.
For the latter, we learn a starttoken t S .
To send the tokens, we ﬁrst obtain z joint .
After that, we feed [ t S ] to T cur , obtain P ( t 1 | t S ; z joint ) , anduse entropy coding to store the d C symbols in token t 1 into a bitstream using P ( t 1 | t S ; z joint ) .
Then,we feed [ t S , t 1 ] , obtain P ( t 2 | t 1 , t S ; z joint ) , store t 2 in the bitstream, and so on.
The receiver gets theresulting bitstream and can obtain the same distributions, and thereby the tokens, by ﬁrst feeding[ t S ] to T cur , obtaining P ( t 1 | t S ; z joint ) , entropy decoding t 1 from the bitstream, then feeding [ t S , t 1 ] toobtain P ( t 2 | t 1 , t S ; z joint ) , and so on.
Fig.
3 visualizes this for P ( t 3 | . . . ) .
We run this procedure in parallel over all blocks, and thereby send/receive y i by running T cur w 2 c =16times.
Each run produces d H/w c e · d W/w c e · d C distributions.
To ensure causality of T cur during 149 training, we mask the self-attention blocks similar to [34].
Independence Apart from assuming blocks in y i are independent, we emphasize that each tokenis a vector and that we assume the symbols within each token are conditionally independent givenprevious tokens, i.e. , T cur predicts the d C distributions required for a token at once .
One could insteadpredict a joint distribution over all possible |S| d C realisations, use channel-autoregression [26], oruse vector quantization on tokens.
The latter two are interesting directions for future work.
Finally,we note that we do not rely on additional side information, in contrast to, e.g. , autoregressive imagecompression entropy models [25, 26].
3.3 Architectures
Transformers As visualized in Fig.
3, all of our transformers are based on standard architectures [34, 10].
We start by projecting the d C -dimensional tokens to a d T -dimensional space ( d T =768in our model) using a single fully connected layer, and adding a learned positional embedding.
While both T sep and T joint are stacks of multi-head self-attention (MHSA) layers, T cur uses masked“conditional” transformer layers, similar to Vaswani et al.
[34]: These alternate between masked
Components trained
Loss
B
[Equation]
LR
Steps steps/s
Stage I
[Equation] [Equation]
2M
Stage II
[Equation]
[Equation]
1M
Stage III
[Equation]
5 E
[Equation]
250k
[Figure]
MHSA layers and MHSA layers that use z joint as keys (K) and values (V), as shown in Fig.
Weuse 6 transformer layers for T sep , 4 for T joint , and 5 masked transformer layers for T cur .
We use 16attention heads everywhere.
We learn a separate temporal positional embedding to add to the input of
T joint .
Image encoder E , decoder D
The image encoder and decoder E, D are not the focus of thispaper, so we use architectures based on standard image compression approaches [25, 26].
For theencoder, we use 4 strided convolutional layers, downscaling by a factor 16 ⇥ in total.
For the decoder,we use transposed convolutions and additionally add residual blocks at the low resolutions.
We used ED =192 ﬁlters for all layers.
See App. A.1 for details and an exploration of architecture variants.
3.4 Loss and Training Process
The modeling choices introduced in the previous section allow for an efﬁcient training procedurewhere we decompose the training into three stages, which enables rapid experimentation (Tab. 1).
In Stage I we train the per-frame encoder E and decoder D by minimizing the rate-distortiontrade-off [42, Sec 3.1.1].
Let U denote a uniform distribution in [  0 . 5 , 0 . 5] .
We minimize
[Equation]
using ˜ y to refer to the unquantized representation, and x ⇠ p X are frames drawn from the training set.
Intuitively, we want to minimize the reconstruction error under the constraint that we can effectivelyquantize the encoder output, with  controlling the tradeoff.
For Stage I , we thus employ the meanscale hyperprior [25] approach to estimate p , the de facto standard in neural image compression,which we discard for later stages.
4 To enable end-to-end training, we also follow [25], adding i.i.d.uniform noise u to ˜ y when calculating r , and using straight-through estimation (STE) [32, 26] forgradients when rounding ˜ y to feed it to D .
For Stage II , we train the transformer to obtain p , and only minimize rate:
[Equation]
where ( x 1 , x 2 , x 3 ) ⇠ p X 1:3 are triplets of adjacent video frames.
We assume each of the d Cunquantized elements in each token follow a Gaussian distribution, p ⇠ N , and let the transformer
ˆ x i ˆ x i
Decode: 0 tokensblock 2 tokensblock 13 tokensblock 0kB 0.13kB
0.78kB predict d C means and d C scales per token.
Finally, we ﬁnetune everything jointly in Stage III , addingthe distortion loss to Eq. 2.
To obtain a discrete PMF P for the quantized symbols (for entropy coding), we again followstandard practice [4], convolving p with a unit-width box and evaluating it at discrete points, P ( y ) =
R u 2 U p ( y + u ) , y 2 Z [see, e.g. , 42, Sec. 3.3.3, for details].
To train, we use random spatio-temporalcrops of ( B, N F , 256 , 256 , 3) pixels, where B is the batch size, and N F the number of frames (valuesare given in Tab. 1).
We use the linearly decaying learning rate (LR) schedule with warmup, wherewe warmup for 10k steps and then linearly decay from the LR shown in the table to 1 E  5 .
Stage
I is trained using  =0 . 01 .
To navigate the rate-distortion trade-off and obtain results for multiplerates, we ﬁne-tune 9 models in Stage III , using  =0 .
01 · 2 i , i 2 {  3 , . . .
We train all models on
Google Cloud TPUv4 chips.
3.5 Latent Residual Predictor (LRP)
To further leverage the powerful representation that the transformer learns, we adapt the “latentresidual predictor” (LRP) from recent work in image compression [26]: The ﬁnal features z cur from
T cur have the same spatial dimensions as y i , and contain everything the transformer knows about thecurrent and previous representations.
Since we have to compute them to compute P , they constitute“free” extra features that are helpful to reconstruct ˆ x i .
We thus use z cur by feeding y {+
+} i =
y i + f LRP ( z cur )to D (we enable this in Stage III ), where f LRP consists of a 1 ⇥ 1 convolution mapping from d T tod ED followed by a residual block.
We note that this implies that ˆ x i
= D ( y 0 i ) indirectly depends ony i  2 , y i  1 , y i .
Since this is a bounded window into the past and y {+
+}
i does not depend on ˆ x j<i , weremain free from temporal error propagation.
Experiments
Data sets We train on one million Internet video clips, where each clip has nine frames.
Weobtained high-resolution videos which we downscale with a random factor (removing previouscompression artifacts), from which we get a central 256 crop.
Training batches are made up ofrandomly selected triplets of adjacent frames.
We evaluate on two common benchmark data sets: (1)
MCL-JCV [35, MIT Licence] made up of thirty 1080p videos captured at either 25 or 30FPS andaveraging 137 frames per video, and (2) UVG [24, CC-BY-NC Licence] containing twelve 1080p120FPS videos with either 300 or 600 frames each.
Synthetic videos We explore three parameterized synthetic data sets that we build by generatingvideos from still images from the CLIC2020 test set [33, Unsplash licence], (see Fig. 5).
Each data
[Table]
set has a parameter x that we vary, and we create 100 videos for each value of x .
Each video is 12frames of 512 ⇥ 512 px.
We explore: Shift , where we pan from the center of the image towards thelower right, shifting by x pixels in each step.
SharpenOrBlur , where if x  0 , we apply Gaussianblurring with sigma x · t at time step t .
If x< 0 , we create videos that get sharper over time by playinga video blurred with | x | in reverse.
Fade , where we linearly transition between two unrelated imagesusing alpha blending (as in a scene cut).
We release the code to synthesize these videos.
Models We refer to our video compression transformer as VCT .
We run the widely used, nonneural, standard codec HEVC [30] ( a.k.a. H.265) using ffmpeg in the medium and veryslow settings,as well as H.264 in the medium setting.
For a fair comparison to our method, we follow previouswork [1, 23, 29] in disabling B-Frames, but do not constrain the codecs in any other way.
We runthe public DVC [22] code, and additionally obtain numbers from the following papers: SSF [1],which introduced scale-space-ﬂow, an architectural component to support warping and blurring,commonly used in follow-up work, ELF-VC [29], to the best of our knowledge the state-of-the-artneural method in terms of PSNR on MCL-JCV, which extends the motion compensation of SSFwith more motion priors, FVC [16] and DCVC [18], both strong models based on warping plusresidual coding in a representation space, RLVC [39], using ConvLSTMs as a sequence model, and Liu et al.
[20], who study losslessly transmitting representations using CNNs for temporal entropymodelling.
To explore how architectural biases behave on synthetic data, we reproduce SSF.
Metrics We evaluate the common PSNR and MS-SSIM [37] in RGB.
We train all models using
MSE as a distortion and use 200 · (1  MS-SSIM ( x, ˆ x )) as the training objective in Stage III (
Tab. 1)to obtain MS-SSIM models.
Results
Comparison to State of the Art In Fig.
4, we depict rate distortion graphs for our method andthe neural video compression methods introduced in Sec. 4, on MCL-JCV and UVG.
Despite thesimplicity of our approach, and the fact that we use no motion or warping components, we outperformall methods in both PSNR and MS-SSIM.
Synthetic data In Fig.
5, we show how the transfomer learns to exploit various types of temporalpatterns by applying it to the synthetic data sets introduced in Sec. 4, and reporting the evaluation
R-D loss.
5 We compare to HEVC and SSF, which both have explicit support for shifting motion,while SSF also has explicit support for blurring.
We expect them to perform well on temporal patternsfor which they have corresponding architectural priors.
In contrast, VCT has no such priors.
For eachdata set, we explore different values for the parameter x
(see Sec. 4), a point in the plot represents theaverage evaluation loss over the 100 videos created with x .
We observe: a) On videos with shifting based motion, VCT obtains ⇡ 45% lower R-D loss comparedto SSF, which saturates at about x = 10 , presumably due to the shallow CNN used for ﬂow estimation.
Since HEVC supports motion compensating with arbitrary shifts of previous frames, it excels onthese kinds of videos.
For shifts that are a multiple of 16, the representations shifts by exactly 1symbol in each step, and VCT matches HEVC.
b) For blurring/sharpening, we outperform both HEVC and SSF, despite the latter having explicit support for blurring. Note that the curve for SSFis asymmetric: since it has built-in support for blurring, it gets a ⇡ 20% lower RD loss on blurring we
T
[Table]
compared to sharpening.
c) VCT learns to handle fading, exhibiting a near-constant RD loss as weincrease x , in contrast to the baselines, neither of which has a explicit support for fading.
SSF is⇡ 20% better than HEVC, possibly due to its blurrying capabilities.
For completely static videosx =0 , we observe that VCT is at a slight disadvantage compared to the previous approaches.
Overall,we believe that synthetic data can give better insight into the strengths and weaknesses of methods,and hope that future work can compare on these data sets.
Visualizing certainty during decoding After having seen k tokens in each block, the transformerpredicts a PMF P ( t k +1 | t  k , z joint ) .
This induces a joint distribution P ( t >k | . . . )
over all unseen (notyet decoded) tokens.
Intuitively, if the transformer is certain about the future, this distribution will beconcentrated on the actual future tokens we will decode.
In Fig.
6, we visualize the sample meanof this distribution by feeding it through D , i.e. , we sample N realisations of the unseen tokens ineach block, conditioned on the k already decoded ones, for k 2 { 0 , 2 , 13 } .
In the middle image in
Fig.
6, we show what the transformer expects at the current frame, before decoding any information( k = 0 , i.e. , 0 bits).
We observe that the model is able—to some degree—to learn second ordermotion implicitly.
The next two images shows that as we decode more tokens, the model gets morecertain, and the image sharpens.
Ablations In Tab. 2, we explore the importance of temporal context from previous frames andlatent residual prediction (LRP) on MCL-JCV.
We start from a baseline that does not use any previousframes, i.e. , an image model, used to independently code each frame.
Conditioning on one previousframe reduces bitrate by  58% .
Using two previous frames yields an additional improvement of 6% .
In the ﬁnal conﬁguration (our model, VCT), which adds LRP, we observe an increase in PSNRof 0.7dB at the same bitrate.
We did not observe further gains from more context.
Runtime To obtain runtimes of the transformers ( T sep , T joint , T cur ) and the decoder ( D ), we employa Google Cloud TPU v4 (single core) using Flax [15], which has an efﬁcient implementation forautoregressive transformers.
We use Tensorﬂow Compression to measure time spent entropy coding(EC), on an Intel Skylake CPU core.
In Tab. 3, we report numbers for 1280 ⇥ 720 px, 852 ⇥ 480 px,and 480 ⇥ 360 px.
Since this benchmark is not fully end-to-end, we only report an FPS estimate bycalculating 1000 / ( sum of individual runtimes in ms ) .
Note that running T cur at 720p once only takes⇡ 2 .
8 ms, but we run it w 2 c
=16 times to decode a frame.
To run T joint , we only have to run T sep onceper representation, since we can re-use the output of running T sep on the previous representation.
Conclusion and Future Work
We presented an elegantly simple transformer-based approach to neural video compression, outperforming previous methods without relying on architectural priors such as explicit motion prediction orwarping.
Notably, our results are achieved by conditioning the transformer only on a 2-frame windowinto the past.
For some types of videos, it would be interesting to scale this up, or to introduce anotion of more long-term memory, possibly leveraging arbitrary reference frames.
As mentioned towards the end of Sec. 3.2, various different ways to factorize the distributions couldbe explored, including vector quantization, channel-autoregression, or changing the independenceassumptions around how we split representations into blocks.
Societal Impact We hope our method can serve as the foundation for a new generation of videocodecs.
This could have a net-positive impact on society by reducing the bandwidth needed for videoconferencing and video streaming and to better utilize storage space, therefore increasing the capacityof knowledge preservation.