C ONDITIONAL C ODING FOR F LEXIBLE L EARNED V IDEO C OMPRESSION
A BSTRACT
This paper introduces a novel framework for end-to-end learned video coding.
Image compression is generalized through conditional coding to exploit information from reference frames, allowing to process intra and inter frames with the same coder.
The system is trained through the minimization of a rate-distortion cost, with no pre-training or proxy loss.
Its ﬂexibility is assessed under three coding conﬁgurations (All Intra, Low-delay P and Random Access), where it is shown to achieve performance competitive with the state-of-the-art video codec HEVC.
I NTRODUCTION
In the last few years ITU/MPEG video coding standards—HEVC (Sullivan et al., 2012) and VVC (J. Chen, 2020)—have been challenged by learning-based codecs.
The learned image coding framework introduced by Ball ´ e et al.
(2017; 2018) eases the design process and improves the performance by jointly optimizing all steps (encoder, decoder, entropy coding) given a rate-distortion objective.
The best learned coding system (Cheng et al., 2020) exhibits performance on par with the image coding conﬁguration of VVC.
In video coding, temporal redundancies are removed through motion compensation.
Motion information between frames are transmitted and used to interpolate reference frames to obtain a temporal prediction.
Then, only the residue (prediction error) is sent, reducing the rate.
Frames coded using references are called inter frames, while others are called intra frames.
Although most learning-based video coding systems follow the framework of Ball ´ e et al., the end-toend character of the training is often overlooked.
The coders introduced by Lu et al.
(2019) or (Liu et al., 2019) rely on a dedicated pre-training to achieve efﬁcient motion compensation.
Dedicated training requires proxy metrics not necessary in line with the real rate-distortion objective, leading to suboptimal systems.
Due to the presence of both intra and inter frames, learned video coding methods transmit two kinds of signal: image-domain signal for intra frames and residual-domain for inter frames.
Therefore, most works (Agustsson et al., 2020) adopt a two-coder approach, with separate coders for intra and inter frames, resulting in heavier and less factorizable systems.
This paper addresses these shortcomings by introducing a novel framework for end-to-end learned video coding, based on a single coder for both intra and inter frames.
Pursuing the work of Ladune et al.
(2020), the coding scheme is decomposed into two sub-networks: MOFNet and CodecNet.
MOFNet conveys motion information and a coding mode, which arbitrates between transmission with CodecNet or copy of the temporal prediction.
MOFNet and CodecNet use conditional coding to leverage information from the previously coded frames while being resilient to their absence.
This allows to process intra and inter frames with the same coder.
The system is trained as a whole with no pre-training or dedicated loss term for any of the components.
It is shown that the system is ﬂexible enough to be competitive with HEVC under three coding conﬁgurations.
P ROPOSED SYSTEM
Let { x i , i ∈ N } be a video sequence, each frame x i being a vector of C color channels 1 of height
H and width W .
Video codecs usually process Groups Of Pictures (GOP) of size N , with a regular frame organization.
Inside a GOP, all frames are inter-coded and rely on already sent frames called
[Figure]
references: B-frames use two references while P-frames use a single one.
The ﬁrst frame of the GOP relies either on a preceding GOP or on an intra-frame (I-frame) denoted as x 0 .
This work primarily targets the Random Access conﬁguration (Fig. 1), because it features I, P and B-frames.
Here, we consider the rate-distortion trade-off, weighted by λ , of a single GOP plus an initial I-frame x 0 :
L λ =
N (cid:88) t
[Equation]
D(ˆ x t , x t ) + λ R(ˆ x t ) , with D the MSE and R the rate.
2. B- FRAME C ODING
The proposed architecture processes the entire GOP (I, P and B-frames) using a unique neural-based coder.
B-frames coding is detailed here.
Thanks to conditional coding, I and P-frames are processed by simply bypassing some steps of the B-frame coding process as explained in Section 2.2.
Let x t be the current B-frame and (ˆ x p , ˆ x f ) two reference frames.
Figure 2 depicts the coding process of x t .
First, ( x t , ˆ x p , ˆ x f ) are fed to MOFNet which computes and conveys—at a rate R m —two optical ﬂows ( v p , v f ) , a pixel-wise prediction weighting β and a pixel-wise coding mode selection α .
The optical ﬂow v p (respectively v f ) represents a 2D pixel-wise motion from x t to ˆ x p (resp.
ˆ x f ).
It is used to interpolate the reference through a bilinear warping w .
The pixel-wise weighting β is applied to obtain the bi-directional weighted prediction ˜ x t : ˜ x t = β (cid:12) w (ˆ x p ; v p ) + (1 − β ) (cid:12) w (ˆ x f ; v f ) , (cid:26) (cid:12) is a pixel-wise multiplication, v p and v f ∈ R 2 × H × W , β ∈ [0 , 1] H × W (2)
The coding mode selection α ∈ [0 , 1] H × W arbitrates between transmission of x t using CodecNet versus Skip mode , a direct copy of ˜ x t .
CodecNet sends areas of x t selected by α , using information from ˜ x t to reduce its rate R c .
The total rate required for x t is R =
R m + R c and the decoded frame ˆ x t is the sum of both contributions:
ˆ x t = (1 − α ) (cid:12) ˜ x t (cid:124) (cid:123)(cid:122) (cid:125)
Skip + c ( α (cid:12) x t , α (cid:12) ˜ x t ) (cid:124) (cid:123)(cid:122) (cid:125) CodecNet .
2. C ONDITIONAL C ODING
Conditional coding (Ladune et al., 2020) allows to exploit decoder-side information more efﬁciently than residual coding.
Its architecture is similar to an auto-encoder (Ball´e et al., 2018), with one additional shortcut transform (Fig. 2).
It can be understood through the description of its 3 transforms.
Shortcut transform g (cid:48) a ( Decoder )—Its role is to extract information from the reference frames available at the decoder ( i.e. at no rate).
The information is computed as latents y (cid:48) .
Analysis transform g a ( Encoder )—It estimates and conveys the information not available at the decoder i.e. the unpredictable part.
The information is computed as latents ˆ y .
Synthesis transform g s ( Decoder )—Latents from the analysis and shortcut transforms are concatenated and synthesized to obtain the desired output.
Unlike residual coding, conditional coding leverages decoder-side information in the latent domain.
As noted by Djelouah et al.
(2019), this makes the system more resilient to the absence of information at the decoder ( i.e. for I-frames).
Thus, MOFNet and CodecNet implement conditional coding to be able to process I, P and B-frames as well as lowering their rate.
I and P-frames are compressed using the B-frames coding scheme, with the same parameters, and ignore the unavailable elements.
I-frame —Motion compensation is not available.
As such, MOFNet is ignored, α is set to 1 and
CodecNet conveys the whole frame, with its shortcut latents y (cid:48)
c set to 0 .
P-frame —Bi-directional motion compensation is not available.
β is set to 1 to only rely on the prediction from ˆ x p .
MOFNet shortcut latents y (cid:48) m are set to 0 .
[Figure]
T RAINING
The training aims at learning to code I, P and B-frames.
As such, it considers the smallest coding conﬁguration featuring all 3 types of frame: a GOP of size 2 plus the preceding I-frame.
Each training iteration consists in the coding of the 3 frames, followed by a single back-propagation to minimize the rate-distortion cost of equation 1.
Unlike previous works, the entire learning process is achieved through this rate-distortion loss.
No element of the system requires a pre-training or a dedicated loss term.
Moreover, coding the entire GOP in the forward pass enables the system to model the dependencies between coded frames, leading to better coding performance.
The training set is made of 400 000 videos crops of size 256 × 256 , with various resolutions (from 540p to 4K) and framerates (from 24 to 120 fps).
The original videos are from several datasets:
KonViD-1k (Hosu et al., 2017), CLIC20 P-frame and Youtube-NT
(Yang et al., 2020).
The batch size is 4 and the learning rate is set to 10 − 4 and decreased to 10 − 5 during the last epochs.
Ratedistortion curves are obtained by training systems for different λ .
V ISUAL I LLUSTRATIONS
This section shows the different quantities at stakes when coding a B-frame x t (Fig. 3a).
First,
MOFNet outputs two optical ﬂows ( v p , v f ) (Fig. 3d), the prediction weighting β (Fig. 3b) and the coding mode selection α .
The temporal prediction is then computed following equation 2.
Most of the time, β (cid:39) 0 .
5 , mitigating the noise from both bilinear warpings.
When the background is disoccluded by a moving object ( e.g. the woman), β equals 0 on one side of the object and 1 on the other side.
This allows to retrieve the background from where it is available.
The competition between Skip mode and CodecNet is weighted by α .
Here, most of ˆ x t comes from the Skip mode(Fig.
3c).
However, the less predictable parts, e.g. the woman, are sent by CodecNet.
To illustrate the conditional coding, v f is computed by the MOFNet synthesis transform using only the shortcut latents y (cid:48) m
(Fig. 3e), the transmitted ones ˆ y m (Fig. 3f) or both (Fig. 3d).
The shortcut transform captures the nature of the motion in y (cid:48) m , which allows to synthesize most of v f without any transmission involved.
In contrast, ˆ y m consists in a reﬁnement of the ﬂow magnitude.
The rate of y (cid:48) m is reduced by using a low spatial resolution, unlike y (cid:48) m which keeps all the spatial accuracy.
R ATE -D ISTORTION R ESULTS
The proposed system is assessed against x265 3 , an implementation of HEVC.
The quality is measured with the PSNR and the BD-rate (Bjontegaard, 2001) indicates the rate difference for the same distortion between two coders.
The test sequences are from the HEVC Common Test Conditions
[Figure]
(Bossen, 2013).
The system ﬂexibility is tested under three coding conﬁgurations: All Intra (AI) i.e. coding only the ﬁrst I-frame, Low-delay P (LDP) i.e. coding one I-frame plus 8 P-frames and
Random Access (RA) i.e. coding one I-frame plus a GOP of size 8.
BD-rates of the proposed coder against HEVC are presented in the Table 1. Coding
[Table]
All Intra (AI) [Table] Random
[Table]
RA)
[Table]
The proposed system outperforms HEVC in AI conﬁguration, proving that it properly handles Iframes.
It is on par with HEVC for RA coding and slightly worse than HEVC for LDP coding.
This shows that the same coder is also able to efﬁciently code P and B-frames, without affecting the
I-frames performance.
To the best of our knowledge, this is the ﬁrst system to achieve compelling performance under different coding conﬁgurations with a single end-to-end learned coder for the three types of frame.
C ONCLUSION
This paper proposes a new framework for end-to-end video coding.
It is based on MOFNet and
CodecNet, which use conditional coding to leverage the information present at the decoder.
Thanks to conditional coding, all types of frame (I, P & B) are processed using the same coder with the same parameters, offering a great ﬂexibility in the coding conﬁguration.
The entire training process is performed through the minimization of a unique rate-distortion cost.
Its ﬂexibility is illustrated under three coding conﬁgurations: All Intra, Low-delay P and Random Access, where the system achieves performance competitive with HEVC.
The main focus of this work is not in the internal design of the networks architecture (MOFNet and CodecNet).
Future work will investigate more advanced architectures, from the optical ﬂow estimation or the learned image coding literature, which should bring performance gains.
A S UPPLEMENTARY R ATE - DISTORTION R ESULTS A. S EQUENCE - BY - SEQUENCE BD- RATES
Table 2 details the sequence-by-sequence BD-rates that gives the averaged results presented in Table 1.
HEVC compression is achieved using ffmpeg with the following command:
[Table]
The BD-rate is computed using four quality factors QP
= { 27 , 32 , 37 , 42 } .
The Low-delay P conﬁguration is obtained by changing the tune option to zerolatency .
WxH denotes the video width and height.
[Table]
A. S UPPLEMENTARY A NCHORS
Previous work (Lu et al., 2019; Djelouah et al., 2019) uses AVC as an anchor.
Table 3 displays the
BD-rate of the proposed system against x264 through the following command:
[Table]
The proposed system consistently outperforms AVC in all classes under the three coding conﬁgurations.
[Table]
B S YSTEM B EHAVIOR ON THE FourPeople S EQUENCE
Additional illustrations of the system behavior are given here for the FourPeople sequence, extracted from the HEVC Common Test Conditions.
This sequence shows four people slightly moving in front of a still background.
The ﬁrst frame is coded as an I-frame and the next 8 frames are compressed using a (random access) GOP of size 8.
B. R ATE - DISTORTION CURVES
Rate-distortion results of the proposed coder against HEVC and AVC are presented in Figure 4.
On this sequence, the system signiﬁcantly outperforms HEVC across the entire rate range.
[Figure] [Figure]
distortion curves on FourPeople — BD-rate is -
[Figure]
3% w. [Figure] .t [Figure]
HEVC
[Figure]
B. R ATE D ISTRIBUTION INSIDE A GOP
The Figure 5 exhibits the distribution of the rate and of the PSNR across the frames of a GOP.
The PSNR is stable for all the coded frames, ensuring temporal consistency.
As their distortion is roughly the same, their rate is function of their predictability .
Indeed, the less predictable is a frame, the more information are transmitted.
As a result, frames B4 or P8 require more bits to be sent than other inter-frames because their references are temporally further, making them less predictable.
I
B B
B B
B B
B
P
Avg     8 .0 .0 .0 .0 .0 .0 .0 .1 .1 .
[Figure]
I
B B
B B
B B
B
P
Avg      37 .37 .37 .37 .37 .37 .37 .37 .37 .37 .
[Figure]
[Figure]
This section displays the quantities involved when coding a B-frame x t (Fig. 6a).
The two optical ﬂows v p , v f (Fig.
6c and 6d) and the pixel-wise bi-directional weighting β (Fig.
6f) are used to compute the temporal prediction ˜ x t as speciﬁed in equation 2.
Because both ﬂows represent the motion from x t to a reference, they are activated at the same spatial locations but with different directions, resulting in different visualization colors.
Some areas ( e.g. the left man’s hand) exhibits a motion which is not well captured by the system, causing checkerboard artifacts in the visualization.
Disocclusions occurring due to moving objects are handled using β = 0 on one side of the objects and β = 1 on the other side.
This behavior can be seen around the left man’s arm.
MOFNet also computes and transmits the coding mode selection α (Fig. 6e).
Areas in blue ( α = 0 ) rely on Skip mode to be reconstructed (Fig. 6h) while areas in red ( α = 1 ) are transmitted with
CodecNet (Fig. 6g).
Most of the decoded frame (Fig. 6b) comes from Skip mode whereas areas transmitted with CodecNet are only those not well predicted enough e.g. the left man’s hand.
Skip mode relevance is illustrated through the spatial distribution of CodecNet rate (Fig. 6i).
Thanks to Skip mode, only few areas of the frame use CodecNet, resulting in few areas for which bits are spent.
Lastly, the spatial distribution of MOFNet rate (Fig. 6j) shows that all of the coding scheme side-information ( α , β , v p , v f ) are transmitted for a low rate.
[Figure]
B. C ONDITIONAL C ODING
Conditional coding relevance is illustrated for CodecNet by synthesizing its output from the shortcut latents only 7b, the sent latents only 7c or both latents 7a.
Similarly to MOFNet, most of CodecNet output is retrieved from the shortcut latents and few information are transmitted, resulting in significant rate savings.
The shortcut latents are computed from the temporal prediction ˜ x t .
Therefore, a poor prediction results in shortcut latents lacking some information, requiring CodecNet to convey something in these areas.
Here they correspond to the quickly moving objects such as the people’s hands, whose prediction results from badly estimated ﬂows (Fig.
6c, 6d).
This example shows that even with an inaccurate ˜ x t , CodecNet exploits all information from ˜ x t and only transmits correction terms to obtain a proper reconstruction.
[Figure]
B. V ISUAL C OMPARISON
The Figure 8 offers a visual comparison of a B-frame, compressed by HEVC 4 and by the proposed system.
For a lower rate, the system achieves a higher PSNR than HEVC.
The zoom on the man’s hand shows that moving areas are well handled by the system.
The high frequencies in the background (the text) are properly recovered.
The system obtains a smoother reconstruction with fewer coding artifacts than HEVC, i.e. without blocking or rigging effects.
[Figure]
[Figure]
C S YSTEM B EHAVIOR ON THE BQMall S EQUENCE
The behavior of the proposed system is detailed on the BQMall sequence, extracted from the HEVC
Common Test Conditions.
This sequence features people walking in front of a static background.
In this example, the ﬁrst frame is coded as an I-frame and the next 8 frames are compressed using a (random access) GOP of size 8.
This appendix provides additional illustrations to the ones already shown in section 4.
C. R ATE - DISTORTION CURVES
The Figure 9 presents the rate-distortion results of the proposed coder against HEVC and AVC.
For this sequence the system outperforms HEVC at low rate.
However, the system quality starts saturating at high rate resulting in worse performance than HEVC.
We note that the quality saturation issue seems to be inherent to the auto-encoder architecture as noted by Helminger et al.
[Figure]
[Figure]
distortion curves on BQMall — BD-rate is -2.6% w.r.t. HEVC
[Figure]
C. R ATE D ISTRIBUTION INSIDE A GOP
The Figure 10 presents the distribution of the rate and of the PSNR across the frames of a GOP.
Similarly to the FourPeople sequence, the PSNR remains consistent for all the coded frames.
Because this sequence is less static than FourPeople , the inter-frame rates are higher.
I
B B
B B
B B
B
P
Avg    7 .0 .0 .0 .0 .0 .0 .0 .1 .1 .
[Figure]
I
B B
B B
B B
B
P
Avg      33 .
[Figure]
C. V ISUAL C OMPARISON
The Figure 11 offers a visual comparison of a B-frame, compressed by HEVC 5 and by the proposed system.
At a similar rate, HEVC achieves a better PSNR than the proposed coder and seems to retain more high frequency contents.
However, it comes as the cost of signiﬁcant blocking artifacts and pronounced ringing effects.
Due to its convolutional nature, the proposed system offers a smoother output, with fewer compression artifacts.
[Figure]
[Figure]
D D ESCRIPTION OF THE N ETWORK A RCHITECTURE
The architecture of MOFNet and CodecNet, presented in Figure 2, are described in this appendix.
D. B ASIC B UILDING B LOCKS
The system uses attention module to increase the capacity of its different transforms.
The attention modules are implemented as proposed by Cheng et al.
and are described in Figure 12.
[Figure]
D. H YPERPRIOR AND E NTROPY C ODING
The transmitted latents of MOFNet and CodecNet are conveyed using entropy coding, which requires an estimate of the latents probability density function (PDF).
Each element ˆ y i of the latents is described by a Laplace PDF, whose parameters µ i , σ i are conditioned on a hyperprior ˆ z (Ball ´ e et al., 2018).
The hyperprior is computed and transmitted from an auxiliary auto-encoder, described in
Figure 13a.
The hyperprior transmission uses entropy coding and a Laplace PDF, whose parameters are estimated with an auto-regressive model (see Fig. 13b) as proposed by Minnen et al.
Two hyperprior networks are implemented, one for MOFNet and one for CodecNet.
[Figure]
D. MOFN ET A RCHITECTURE
The detailed architecture of the three main transforms of MOFNet (analysis, shortcut and synthesis) is depicted in Figure 14.
[Figure]
D. C ODEC N ET A RCHITECTURE
The detailed architecture of the three main transforms of CodecNet (analysis, shortcut and synthesis) is depicted in Figure
[Figure]