Hierarchical Image Compression Framework Conference Submissions
Abstract
In learning-based image compression approaches, compression models are based on variational autoencoder(VAE) framework and optimized by a ratedistortion objective function, which achieve better performance than hybrid codecs.
However, VAE maps the input to a lower dimensional latent space which becomes a bottleneck of reconstruction.
In this paper, we propose a deep Hierarchical Compression(HC) model, which can achieve good compression performance from low-bit to very high-bit.
HC model consists of two closely-related modules, including hierarchical latent compression module and Hierarchical Conditional Entropy(HCE) module.
Such a design transmits the details in the shallower layers and coarse information in the deeper layers and conditions the shallower entropy estimation on the deeper information.
Extensive experiments show that HC model could breakthrough the AE limit and achieve signiﬁcant improvements over stateof-the-art approaches in the high quality regime.
Introduction
In recent years, end-to-end optimized image compression methods based on variational autoencoder(VAE) have achieved better performance than hybrid codecs.
The methods utilize an end-to-end trainable model to jointly optimize the rate and distortion.
The stateof-the-art networks for image compression are (Agustsson et al., 2017; Theis et al., 2017;
Ball´e et al., 2016a;b; 2018; Mentzer et al., 2018; Lee et al., 2019; Minnen et al., 2018a;
Cheng et al., 2020; Qian et al., 2020; Hu et al., 2020).
A common approach in VAE is to map the images into a lower dimensional latent space in which a probability distribution is learned to allow for entropy coding, such as hyper-prior and context model.
However, the operation of the existing VAE models that mapping an image to a lower dimensional latent space impose an implicit limit on the reconstruction quality.
As such, they cannot address high quality levels well.
VAE models are limited on the reconstruction quality by the AE Limit line (Helminger et al., 2020).
According to the Nyquist–Shannon sampling theorem (Shannon, 1949), high-frequency signals and low-frequency signals need smaller and larger receptive ﬁelds respectively.
Shallower layer has a smaller receptive ﬁeld, which is good for capturing high-frequency information, and the deeper layer has a larger receptive ﬁeld, which is good for capturing low-frequency signals.
Besides, there is a correlation between deep and shallow features.
For these reasons, we explore the HC model.
The contributions of our work are as follows:
(1)We propose a hierarchical compression framework, the model allows us to transmit details in the shallower layers and coarse information in the deeper layers.
Entropy estimation of shallower features are conditioned on deeper features.
(2)We propose a residual compression module, which is diﬀerent from the widely used VAE framework and could achieve better performance.
(3)The experiment results show that HC model breaks the AE Limit, and outperforms the widely used VAE models with lower FLOPs.
The rest of this work is organized as follows.
In Section 2, we introduce the key approaches of hierarchical compression, and the experimental results are shown in section 3.
Finally, in
Section 4, we discuss the current work and future improvements.
Hierarchical Compression 2. Previous Works
Neural multi-scale image compression was ﬁrst proposed by (Nakanishi et al., 2018), which maps the image to multi-scale latents and estimates the probability by a parallel multi-scale
PixelCNN.
(Helminger et al., 2020) proposed normalizing ﬂows for lossy image compression and is superior to the existing VAE solutions in the high quality regime but at low bit-rates the performance is much lower.
2. Compression Based on Hierarchical Condition
The latent representations in deeper and shallower layers have a strong correlation.
We show three latent representations from the 2rd layer to the 4th layer in ﬁgure 1.
Visually, it is obvious that the features of diﬀerent layers are similar.
Shallower and deeper features store details and structural information respectively.
For this reason, we explore a HC model, as shown in the ﬁgure 2(left).
Our framework has three fundamental parametric transform functions: an analysis transform g a ( x ; ϕ g ), a synthesis transform g s (ˆ y ; θ g ), and a group of conditional transform h s (ˆ y ; θ h ).The
encoding process is described as follows: on the encoder side, g a encodes the input image into hierarchical features y i , i = 1 , 2 , ..., n which then quantized to form ˆ y i and lossless compressed using entropy coding techniques.
On the decoder side, an entropy decoder recovers ˆ y i from the compressed signal, and g s recovers the reconstructed image ˆ x from ˆ y .
The entropy coding needs the probability distribution of ˆ y , where ˆ y =
(ˆ y 1 , ˆ y 2 , ..., ˆ y n ).
We assumed a
Gaussian distribution for the Hierarchical Conditional Entropy(HCE), that is,
[Equation]
Since there is no prior for ˆ y n , so a factorized density model (Ball´e et al., 2018) is used to estimate the probability distribution.
Subsequently, the joint probability p (ˆ y ) is represented as the product of the hierarchical conditional distributions:
[Equation]
Rate is the expected code length (bit rate) of the compressed representation: assuming the entropy coding technique is operating eﬃciently, it can be written as a cross entropy:
[Equation]
The loss function is deﬁned as below:
[Equation]
where D is distortion, λ is a hyperparameter that controls the bpp.
2. Residual compression
As shown in ﬁgure 2(right), in latent compression, the quantization of the latents y can be expressed as follows: ˆ y = round ( y ).
This process will inevitably lead to a residual error r =
y − ˆ y in the latent space that manifests as extra distortion when ˆ y is transformed back into ˆ x .
In response to this problem, we use residual compression in our approach to reduce the residual error.
The quantization of residual compression can be expressed as follows:
ˆ y = round ( y − µ ) + µ , where µ is the mean value of Gaussian distribution output from the entropy estimation network.
We will show in our experiments that residual compression can reduce the residual error, and achieve better performance.
[Table]
Experiments 3. Experimental Setup
The detailed framework of our approach is shown in ﬁgure 3.
The backbone is similar to (Ball´e et al., 2018).
The channel of y 3 , y 2 , are set to 16, 4 respectively.
Entropy estimation of shallow features are conditioned on deeper ones.
For training, we use Imagenet (Deng et al., 2009) as training dataset.
The models are optimized using Adam (Kingma & Ba, 2015).
For each input min-batch, we randomly crop 8 patches with size of 256 × 256 from the training dataset.
It takes 6 epochs for training.
We set the initial learning rate to 10 − 4 and reduced by 0.5 times at the 4th epoch.
We optimized our models using mean square error ( MSE ), and λ belongs to the set { 0 .
3. Experimental Results
In this section, we evaluate the eﬀects of HC model.
In the HC model with N =
( HC N 128), only y 4 is transmitted when bpp ≤ 0 .
5, y 3 , y 4 are transmitted when 0 .
5 < bpp ≤ 1 . 0, and y 2 , y 3 , y 4 are transmitted when bpp > 1 . 0.
In the HC model with
N = 192( HC N 192), only y 4 is transmitted when bpp ≤ 1 . 0 , y 3 , y 4 are transmitted when 1 . 0 < bpp ≤ 2, and y 2 , y 3 , y 4 are transmiteed when bpp > 2 . 0.
Figure 4 compares the RD curves averaged over the Kodak image set.
We can see from the left plot that, HC N128 model is outperforms (Ball´e et al. , 2018)(N=128 and N=192).
From the middle plot, HC N192 model outperforms (Ball´e et al., 2018)(N=192 and N=256).
And from the right plot residual compression outperforms latent compression.
Take a 256 × 256 image as an example, The FLOPs of diﬀerent models are shown in the table 1.
It can be seen that our HC model is more lightweight.
Discussion
In this paper, we aimed to solve the problem of VAE limition by HC model.
We also proposed a latent residual compression model that eﬀectively improves the performance.
Experimental results show that our approach can break the limitation of VAE and achieve much better performance at high bit rates with smaller FLOPs.