Paper: Towards image compression with perfect realism at ultra-low bitrates (ICLR-2024)

Reviewer comment (Evaluation):
I am not really sure what is the point of evaluating LPIPS in Fig.3. Obviously MS-ILLM and HiFiC are trained with LPIPS, and this brings unfair advantage over the proposed approach. And as the authors are optimizing their approach towards divergence based perceptual quality [Blau 2018], using FID and KID should be enough. Reporting MS-SSIM and mIoU is also weird.

Author response:
We agree with the reviewer that our goal is to design a perceptual codec and that FID/KID should be sufficient to evaluate the perceptual quality of our model. As we wanted to compare in the setting of RDP theory [Blau 2019], we included an evaluation on image similarity metrics LPIPS and MS-SSIM because they are widely used in the compression litterature. As rightly pointed out, LPIPS and MS-SSIM are not very meaningful in the low bitrate regime (see Fig. 14 in Appendix), so we included mIoU and CLIP score as they offer alternative ways of assessing distribution matching (mIoU having spatial localization and CLIP using a different image recognition mechanism). In the revised manuscript we further clarify the selection of metrics for evaluation.

Reviewer comment (Evaluation):
The authors think achieveing a low MS-SSIM, PSNR and LPIPS is a weakness, which I can not agree. By RDP theory [Blau 2019], it is absolutely normal for perceptual codec being outperformed on image-wise distortion metrics.

Author response:
We agree with the reviewer. We include this discussion in 'Limitations' paragraph of the paper.

Reviewer comment (Presentation):
The authors are enouraged to present results in their original aspect ratio. Currently some results in Fig. 1 looks weird as they are squeezed horizontally.

Author response:
Thanks for the valuable remark. We update all figures in the paper with their original aspect ratio.

Reviewer comment (Experiments):
In MS-ILLM, the VQ-VAE is forzen once after pre-trained. While in this paper the VQ-VAE participate the end-to-end training. Then a natural question to ask is whether include the VQ-VAE of MS-ILLM in training improves the result?

Author response:
In our model, we use an autoencoder that maps RGB images to the latent space of the diffusion model. This autoencoder is pretrained and frozen when training our models and we train a small hyper-encoder in the latent space for vector quantization (see paragraph 'Implementation details' in Section 4.1). In MS-ILLM, the VQ-VAE is only used as a continuous-to-discrete mapping for the purpose of training an adversarial loss. Our model does not use an adversarial loss, so our assessment is the two are not comparable. If we misunderstood the reviewer’s intent, we could reconsider a rephrasing of this request.

Reviewer comment (Novelty):
A recent preprint [Conditional Perceptual Quality Preserving Image Compression] also justifies the adoptation of side information in image compression. The authors can probably connect their work to rate-distortion-perception trade-off with the approach similar the in preprint.

Author response:
Thank you for providing this relevant reference. This work introduces the notion of conditional perceptual quality, by extending perceptual quality to be conditioned on a specific side information. Given that we condition our generative codec on a losslessly encoded text description with the objective of achieving better preservation of text and local semantic information (measured by CLIP score/mIoU) and optimal image quality (FID), our work is very related to the idea of designing an optimal conditional perceptual quality preserving codec. We include this discussion in the related work.

Reviewer comment (Evaluation):
The training and inference complexity of such diffusion-based methods is usually large. The authors should provide some preliminary results like running time or memory consumption.

Author response:
As stated in the joint answer, we performed tests for encode and decode speed and added a table with the results to the Appendix.

Reviewer comment (Novelty):
Another existing diffusion-based image compression framework [1] combining text and image information can be discussed. [1] Pan, Zhihong, Xin Zhou, and Hao Tian. "Extreme Generative Image Compression by Learning Text Embedding from Diffusion Models." arXiv preprint arXiv:2211.07793 (2022).

Author response:
Thanks for pointing out this relevant reference. In [1], they optimize a textual embedding on top of a pretrained text-to-image diffusion model. They also design a compression guidance method used at each denoising step during inference to better reconstruct original images. Their lowest compression rate is 0.042 bits per pixel which is lower than most of the compression baselines, but still much higher than 10x our minimum bitrate of 0.0031 bits per pixel. We include this paper in the related work.

Reviewer comment (Writing):
On page 4, in the “Local spatial encoding” paragraph line 6, the statement “Third, we proceed to quantization of $H_s$ to obtain $z_g$ via vector quantization” may contain an error. From my understanding, $z_g$ should be $z_l$. Also, there is a typo: “quatization” should be “quantization”.

Author response:
Thanks a lot for pointing out the two typos, we correct them in the revised manuscript.

Reviewer comment (Theory):
The proposed method only optimizes the distortion/perceptual term but drops the rate term. It is not justified why the rate term can be dropped. What impact will have if include the rate term into optimization?

Author response:
In our approach, the rate is controlled through two hyper-parameter settings: the spatial resolution of the hyper-latents, and the vocabulary size. Therefore, the rate can be considered fixed when training our models, and therefore dropped from the objective function. To clarify this, we rephrase the first paragraph of section 3.1.

Reviewer comment (Experiments):
One limitation of diffusion model is the inference speed. This paper do not provide the running time of the proposed method, especifically the decoding time. A fair comparsion of running time (both encoding and decoding) with the competing methods (both learned and traditional) should be provided and discussed.

Author response:
As stated in the joint answer, we performed tests for encode and decode speed and added a table with the results to the Appendix.