L OSSY I MAGE C OMPRESSION WITH N ORMALIZING F LOWS
A BSTRACT
Deep learning based image compression has recently witnessed exciting progress and in some cases even managed to surpass transform coding based approaches.
However, state-of-the-art solutions for deep image compression typically employ autoencoders which map the input to a lower dimensional latent space and thus irreversibly discard information already before quantization.
In contrast, traditional approaches in image compression employ an invertible transformation before performing the quantization step.
In this work, we propose a deep image compression method that is similarly able to go from low bit-rates to near lossless quality, by leveraging normalizing ﬂows to learn a bijective mapping from the image space to a latent representation.
We demonstrate further advantages unique to our solution, such as the ability to maintain constant quality results through reencoding, even when performed multiple times.
To the best of our knowledge, this is the ﬁrst work leveraging normalizing ﬂows for lossy image compression.
I NTRODUCTION
In the recent years, several works (Toderici et al., 2016; Rippel & Bourdev, 2017; Mentzer et al., 2018; Minnen et al., 2018b) have explored the usage of deep learning for lossy image compression.
These methods quickly started to challenge decades of work in transform coding and they are now at the core of a number of learning based video compression techniques such as (Lu et al., 2019;
Rippel et al., 2018; Djelouah et al., 2019).
A common approach in deep lossy image compression is to map the images into a lower dimensional latent space in which a probability distribution is learned to allow for entropy coding.
For this mapping, Ball ´ e et al.
(2017) proposed to leverage specialized nonlinear transformations (GDN), while
Rippel & Bourdev (2017) use a pyramidal encoder to extract features of various scales.
However, the existing compression methods can typically be interpreted as variational autoencoders (VAEs), where the entropy model corresponds to the prior on the latent representation.
Since autoencoders map images to a lower dimensional latent space, they impose an implicit limit on the reconstruction quality.
In this work, we propose to leverage normalizing ﬂows as generative models in lossy image compression instead of autoencoders.
By learning bijective transforms from image space to latent space, we can cover a very wide range of quality levels and effectively enable to go from low bit-rates to near lossless quality.
Another interesting property of this bijective mapping is that a compressed image is always mapped to the same point.
Therefore consecutive compression steps can retain the same rate-distortion performance (see Figure 1).
Summing up, the contribution of this paper is three fold: (1) To the best of our knowledge, our work is the ﬁrst to explore normalizing ﬂows for lossy image compression.
(2) Our model achieves the widest range of quality levels among learned image compression methods.
(3) We demonstrate additional advantages such as the capacity to maintain rate-distortion performance during multiple reencodings.
L OSSY I MAGE C OMPRESSION WITH N ORMALIZING F LOWS
We propose using normalizing ﬂows for lossy image compression, and in particular we design an architecture based on additive coupling (Kingma & Dhariwal, 2018) and factor-out layers (Dinh et al., 2017).
The model is illustrated in Figure 2 and it consists of three different levels bijectively mapping any input image x ∈ X to its latent representation
z
=
( z 0 , z 1 , z 2 ) .
We express lossy image compression as the quantization and the entropy coding of this latent representation.
We can obtain an estimate of the data distribution by optimizing the negative log likelihood ( nll ) for normalizing ﬂows.
Adapted to our model this equation becomes
[Equation]
with all the log-determinant terms equal to 0 given that we only use additive layers.
Although the distributions p θ ( z 0 ) , p θ ( z 1 | z 0 ) and p θ ( z 2 | z 1 ) are computed, this model is however not adapted to image compression.
The reason for this is that the quantized latent values are not observed during the training as part of the dataset.
This leads to our proposed image compression objective.
Considering the decompressed image ˆ x obtained from the rounded latents ˆ z l = r ( z l ) [Equation]
[Equation]
the objective of image compression is to both minimize the entropy (Eq.1) and the deviation from the input data point.
This is expressed in the rate-distortion loss ( rdl )
[Equation]
with d ( x , ˆ x ) a term penalizing the deviation of ˆ x from the original image x .
In order to further reduce the bit-rate, we need to exploit the models’s hierarchical architecture and transmitting only a part of the latent space and sample the remaining latent values.
This is illustrated in Figure 2, where on the sampling path, only latent values z 0 are entropy coded while z 1 and zare respectively sampled as the most likely values in the distributions p ( z 1 | z 0 ) and p ( z 2 | z 1 ) .
This new sampling path is easily included in our model through an additional reconstruction loss on the decoded sample ˜x =
f θ (ˆ z 0 , ˜ z 1 , ˜ z 2 ) .
The ﬁnal objective is then deﬁned as follows:
[Equation]
Since the rounding operation is not differentiable, we use universal quantization (Choi et al., 2019), where every element of the latent space is shifted by the same random sample u ∼ U (cid:0) − ∆2 ∆2 (cid:1) .
Regarding probability models, we use the fully factorized distribution by Ball´e et al.
(2018) for the base distribution p θ ( z 0 ) while for the factor-out layers, we use a conditional distribution (see appendix for more details).
To get different quality levels on the rate-distortion curve, we train a single model using a ﬁxed λ value, and at test time, we reach different rate-distortion points by varying the quantization step size used in the latent representation.
To achieve best performance, we propose to ﬁne-tune the quantization step values by minimizing the rate-distortion loss for various values of λ .
E XPERIMENTS
The model we consider in our experiments uses 8 additive layers at each level.
Each of these coupling layers uses half of the channels.
The transformation network t ( · ) is a ResNet with two blocks.
In addition to the traditional image compression codecs, JPEG and BPG, we compare our model with recent learning based autoencoder models.
In order to properly compare the performance, we train on the same dataset and refer to the retrained versions of Ball ´ e et al.
, Ball ´ e et al.
(2018) and Minnen et al.
(2018a)
respectively as AE + Fully-Factorized , AE + Hyperprior and AE + CM
Hyperprior .
For completeness, we add the numbers published in the original paper when available.
To better understand the limits on autoencoder based models, we also train a version of the networks to achieve the highest reconstruction without any rate penalty, referred to as AE Limit .
Image Compression - Fixed Resolution When using normalizing ﬂows related work (Hoogeboom et al., 2019) only considered ﬁxed size images.
In the same spirit, our ﬁrst experiment is based on a ﬁxed size of 64 × 64 .
For this, we resized the images from ImageNet to a resolution of 64 × 64 as described in (Chrabaszcz et al., 2017).
Image compression results are presented in the supplementary material in Figure 6a.
The autoencoder based models have a hard limit on the image reconstruction quality contrary to our solution that consistently increases in quality as more information is transferred.
This limit is shown with a horizontal line and the value is obtained by training autoencoder networks to achieve the highest reconstruction without any rate penalty.
Image Compression - Arbitrary High Resolution The proposed architecture is fully convolutional and the probability distribution of the latents corresponding to the coarse level is modeled globally.
As a result, the model can be trained on a ﬁxed resolution and applied to arbitrary sized images.
We use the COCO dataset (Lin et al., 2014) for training and extract image patches of size 128 × 128 .
We evaluate the trained models on the full images of the Kodak 1 and the Tecnick (Asuni & Giachetti, 2013) data sets.
The results are visualized in Figure 3a and in the supplementary material in Figure 7.
In addition to this, the clear beneﬁt of our normalizing ﬂow based model is the capacity to target a large range of compression rates from low bit-rate to high quality results.
This allows to better
[Figure]
match the behavior of traditional codecs and thus largely closes an important gap between learning based and traditional approaches.
Another interesting property is the model’s possibility of progressively transmitting the image.
By leveraging the hierarchical architecture the latents can be sent level by level and hence progressively improving the quality of the image (Figure 4 in Appendix).
Quasi Lossless Reencoding Besides the capacity to handle a larger range of quality levels than existing learned image compression techniques, our proposed method offers additional advantages.
Since we use normalizing ﬂows, there is a one-to-one mapping between quantized latent values and the corresponding reconstructed image.
This means that compressing the decoded image again will result in the same latent values.
Once the image quality is ﬁxed, chaining multiple compression and decompression steps does not change the quality of the image and retains rate-distortion performance.
This is in contrast to existing autoencoder based solutions that can quickly deteriorate and produce visible artifacts as content is reencoded several times (see Figure 1).
This feature is important in video production pipelines where image and video content might be composited and edited by different parties requiring reencodings in the process.
An evaluation is shown in Figure 3b, where both architectures, ours and autoencoder based approaches, start from the same image quality.
All approaches achieve a constant bit-rate for all the steps, but the autoencoder reconstruction quality solution drops by 1 db after only 2 steps and up to 10 db after 15 steps.
Limitations In the low bit-rate part of the curve, autoencoder based solutions are the best performing.
This is due to the more powerful transformations that are in place, in particular the use of nonlinear transformations (GDN layers).
For the moment we only use additive coupling layers in our network and there is potential for substantial improvements as better bijective layers are developed for normalizing ﬂows.
C ONCLUSION
In this paper, we explored an approach for lossy image compression based on normalizing ﬂows.
To the best of our knowledge, we are the ﬁrst to evaluate the potential of normalizing ﬂows in the task of lossy compression.
Furthermore, we have showcased a number beneﬁcial properties inherent to our solution, such as the capacity to reencode images multiple times in a quasi lossless way closely retaining rate distortion performance.
This has been problematic with autoencoder based methods.
We believe the results obtained in this work are encouraging as there is still a lot of potential for improvement and would push new research works to explore new bijective layers, architecture designs and reﬁned training procedures.
A A PPENDIX
In this section we present an additional experiment on progressive transmission, describe the datasets and the details of the model’s architecture and provide additional RD-curves.
While some hyperparameters differ between datasets, the common architecture is the same for all experiments.
We further provide a pseudo code for the entropy coding used in our compression method.
A. D ETAILS ABOUT P ROBABILITY M ODELS
To model the base distribution p θ ( z 0 ) we use the fully factorized distribution by Ball´e et al.
For each channel dimension c , a neural network is used to learn the cumulative density function F c .
Considering probability independence between the latent elements ˆ z j,c 0 , we have
[Equation]
For the factor-out layers, we use a conditional distribution.
Let ˆ z j be an element in the latent encoding ˆ z l .
We model its probability with a discrete logistic distribution DLogistic (ˆ z j | µ j , σ j ) deﬁned as:
[Equation]
Parameters µ j and σ j at every position j are computed by a parameterized neural network φ ( h l ) .
Given this probability model, the sampling path amounts to using ˜ z
j = µ j .
A. A DDITIONAL E XPERIMENTS ON P ROGRESSIVE T RANSMISSION
Scalable video coding (SVC) (Schwarz et al., 2007) allows to transmit only parts of a bit stream with the goal of retaining rate-distortion performance w.r.t.
the partial stream that could represent a lower spatial resolution, a lower temporal resolution, or a lower ﬁdelity content.
As such SVC provides interesting opportunities for graceful degradation and ﬂexible bit-rate adaptation in the context of video streaming.
However, due to added complexity, it has not found widespread industry adoption.
Aside from SVC, image standards such as JPEG and JPEG2000 also allow progressive transmission.
This can be seen as a more ﬁne grained way of scaling image ﬁdelity.
[Figure]
A
[Figure]
H y p
e r P r i o r
( B a ll é e t a l .
[Figure]
Our solution uses a hierarchical model with factor out distributions and thus is designed to inherently allow us to store details in the top layers and coarse information in the bottom layers.
However, contrary to (Hoogeboom et al., 2019)
these intermediate layers are optimized to have best ratedistortion performance.
When only using the information stored in the bottom layer ˆ z 0 , we can automatically set the values on the remaining layers to their estimated means without requiring to transmit any further information.
In this way, we obtain a decoded image that has competitive ratedistortion performance w.r.t this partial stream of data.
I.e. the results are comparable with those produced by autoencoders in terms of rate and distortion (see Figure.
4, 1 level).
To incrementally scale to a higher quality image, it is sufﬁcient to only send the remaining latents ˆ z 1 (Figure. 4, 2 levels), or the full encodings ˆ z 2 (Figure.
4, 3 levels).
It is also possible to partially send data and achieve progressive scaling, following a predeﬁned pattern.
This is illustrated for the latent ˆ z 2 (in
Figure.
4, 2 . 5 levels) where only a fraction ( 50% ) of the values are sent.
A. T RAINING D ETAILS
We describe the model illustrated in Figure 3 in the main paper.
Independent of the dataset, the models have L = 3 levels with K steps each.
Each levels starts with a Squeeze -layer.
A single step consists of a random permutation and an additive transformation .
A ResNet (Figure 5) with B blocks and C hidden channels is used as transformer network t ( . )
(see also Figure 2 in the main paper).
Each factor-out layer divides the latents into two equal sized partitions, and only one of them is passed to the next level.
We use a fully factorized model as prior p ( z 0 ) and discrete logistic distributions as conditional distributions p ( z 1 | z 0 ) and p ( z 2 | z 1 ) .
The network φ ( . ) used to predict the parameters of the conditional distributions is also a ResNet B blocks and C channels.
During training, we use the AdaMax optimizer (Kingma & Ba, 2015) with a learning rate of 10 −(with (cid:15) = 10 − 7 ) and the quantization step is set to ∆ = 1 .
After training, to optimize the quantization steps ∆ for a ﬁxed λ ∈ (cid:2) 1 , 10 6 (cid:3) we use Adam optimizer (Kingma & Ba, 2015) with learning rate 10 − 1 . A. C OMPRESSION AT DIFFERENT QUALITY LEVELS . In order to avoid retraining image compression models for different rate-distortion levels Choi et al. (2019) use the Lagrange multiplier and the quantization bin size as conditioning variables to the network.
It is unclear how such a strategy would ﬁt in the normalizing ﬂows model. Instead, to get different quality levels on the rate-distortion curve, we train a single model using a ﬁxed λ value (see experiments for details), and at test time, we reach different rate-distortion points by varying the quantization step size used in the latent representation. In our case however, multiple levels are present and hence multiple step sizes must be set. To achieve best performance, we propose to ﬁne-tune the quantization step values by minimizing the rate-distortion loss for various values of λ :
[Equation]
where ∆ is the set of quantization steps to be estimated: a single scalar for each one of ˆ z 2 and ˆ z 1 , and a distinct value for each channel of ˆ z 0 .
A.4. I MAGE N ET
We resized the images from ImageNet to a resolution of 64 × 64 as described in (Chrabaszcz et al., 2017).
For this dataset, the model is trained for 25 epochs, where each epoch consists of 40 k batches of size 32 .
The results for the ImageNet64 dataset are visualized in Figure 6a. [Figure] Parameter Value
Levels L
[Figure] ResNet ( B = 2 , C = 64 ) [Figure] ResNet ( B =
[Figure]
C =
[Figure]
Baselines We used the same architecture proposed in (Ball ´ e et al., 2018) with M = 192 and N = 192 .
We optimized for each model ( AE + Fully-Factorized , AE + Hyperprior ) the rate-distortionloss for a ﬁxed λ = 0 .
3 with Adam optimizer and learning rate lr = 2 · 10 − 4 for ∼ 5 .
5 million iterations with batch size 32 .
A.4. COCO
The COCO dataset consists of ≈ 285 k images.
We used the original train/validation/test - split.
During training, we randomly cropped patches of size 128 × 128 due to the limitation of memory.
The model is trained for 50 epochs.
[Figure]
Baselines For all autoencoder based models ( AE-Limit , AE + Fully-Factorized , AE + Hyperprior and AE + CM Hyperprior )
we used the same architecture proposed in (Minnen et al., 2018a) with
M = 320 and N = 192 .
We optimized for each model ( AE + Fully-Factorized , AE + Hyperprior , AE + CM Hyperprior ) the rate-distortion-loss for a ﬁxed λ = 0 .
15 with Adam optimizer and learn-
[Figure]
A. E NTROPY C ODING : P SEUDO A LGORITHM
To further improve the bit-rate as the predictions reach higher probability values around the means (with respect to the quantization step), it becomes more beneﬁcial to simply sample this value.
In our compression scheme, we use a threshold on the probability value around the mean,
DLogistic ( µ j | µ j , σ j ) >
P thresh , for which no information is transferred and the mean µ j is used ( z j = µ j ).
In all our experiments P thresh = 0 .
The pseudo code in Algorithm 1 describes how our thresholding is considered in entropy encoding and decoding.
The thresholding is performed based on the probability of the predictions (see end of
Chapter 3 in the main paper).
As the threshold can be arbitrarily set in principle, it offers another degree of freedom when realizing progressive transmission. [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]