Paper: Idempotence and Perceptual Image Compression (ICLR-2024)

Reviewer comment (Writing):
The relationship between idempotence and image compression is hard to understand in the current version.

Author response:
Although we have provided proof sketch in main text and complete proof in Appendix of __Theorem 1__ and __Theorem 2__, we do agree that they are quite abstract and not intuitive. However, theoretical results take $50$ percent of this paper. And it is a must to understand them correctly to understand this paper. To provide a more intuitive illustration, we will include an example with discrete finite alphabet (our theoretical results are appliable to any countable alphabet or Borel-measurable source $X$ and any deterministic measureable function $f(.)$):
    * __Example 1__ (1-dimension, discrete finite alphabet, optimal 1-bit codec): Consider discrete 1d random variable $X$, with alphabet $\mathcal{X} = \{0,1,2,3,4,5\}$, and $Y$ with alphabet $\mathcal{Y}=\{0,1\}$. We assume $P(X=i) = \frac{1}{6}$, i.e., $X$ follows uniform distribution. We consider a deterministic transform $f(X): \mathcal{X}\rightarrow \mathcal{Y} = \textrm{round}(X/3)$. Or to say, $f(.)$ maps $\{0,1,2\}$ to $\{0\}$, and $\{3,4,5\}$ to $\{1\}$. This is infact the MSE-optimal 1-bit codec for source $X$ (See Chapter 10 of [Elements of Information Theory]). And the joint distribution $P(X,Y)$, posterior $P(X|Y)$ is just a tabular
    *   | x | y | P(X=x,Y=y) | P(X=x\|Y=y) | f(X=x) |
        |---|---|------------|-------------|--------|
        | 0 | 0 | 1/6        | 1           | 0      |
        | 1 | 0 | 1/6        | 1           | 0      |
        | 2 | 0 | 1/6        | 1           | 0      |
        | 3 | 0 | 0          | 0           | 1      |
        | 4 | 0 | 0          | 0           | 1      |
        | 5 | 0 | 0          | 0           | 1      |
        | 0 | 1 | 0          | 0           | 0      |
        | 1 | 1 | 0          | 0           | 0      |
        | 2 | 1 | 0          | 0           | 0      |
        | 3 | 1 | 1/6        | 1           | 1      |
        | 4 | 1 | 1/6        | 1           | 1      |
        | 5 | 1 | 1/6        | 1           | 1      |
    * We first examine __Theorem 1__, which says that any sample from $X\sim P(X|Y=y)$ satisfies $f(X) = y$. Observing this table, this is indeed true. As for $f(X)\neq y$, the posterior $P(X=x|Y=y)$ is $0$.
    * We then examine __Theorem 2__, which says that sampling from $X\sim P(X)$ with $f(X)=y$ constraint is the same as sampling from $P(X|Y=y)$. We first identify the set such that $f(X)=y$. When $y=0$, this set is $\{0,1,2\}$. And when $y=1$, this set is $\{3,4,5\}$. And sampling from $p(X)$ with constrain $f(X)=y$ is equvalient to sampling from one of the two subset with probability $\propto p(X)$. And this probability is exactly $P(X|Y=y)$.

Reviewer comment (Experiments):
Authors should provide more evidence to support the points that Idempotence brings perceptual quality in Section 3.

Author response:
The "Idempotence brings perceptual quality" in Sec. 3 is __Theorem 2__. It is already theoretically supported by proof in Appendix. A. Furthermore, it is already empirically supported by the Sec. 5. In Tab. 2 and Fig. 5 of Sec. 5, we can clearly see that indeed optimizing idempotence loss brings perceptual image compression.

Reviewer comment (Evaluation):
I can't see the superiority of the proposed method in the visual comparison with HiFiC and ILLM, especially in Fig 1.

Author response:
It could be the problem of PDF compression. We have provided a version with larger resolution in https://ibb.co/NWww3RX. Probably zoom in can help (Reviewer VEis already finds qualitative results compelling with the PDF). We note that only our approach can reconstruct the beard of dog, while HiFiC and ILLM fail to do so. Further, our reconstruction has much more realistic texture of dog's hair.

Reviewer comment (Evaluation):
In BD-FID, the proposed method is better. But in BD-PSNR, others may be better. Could authors provide more evaluation metrics, such as MSSSIM and VMAF.

Author response:
For BD-PSNR, we emphasis that:
    * We can control our decoder to achieve an intermediate point between FID and MSE. In Appendix B.3 and Fig. 8, we can see that by trading FID for MSE, our approach achieves better FID with comparable MSE with ILLM. And it can also achieves comparable FID with better MSE compared with ILLM. We also note that it is impossible to achieve sota FID and sota PSNR at the same time by distortion-perception trade-off [Blau 2018]. Therefore, we can not achieve a PSNR as good as [Balle 2018] and [He 2022].
    * Our MSE is within 2x MSE of base codec, and our PSNR is within 3 dB of base codec, which is aligned to the theoretical result.
* For other metrics, we have evaluated our approach additionally on KID and LPIPS:

    | Method              | FFHQ      |          | ImageNet  |          | COCO      |          | CLIC      |          |
    |---------------------|-----------|----------|-----------|----------|-----------|----------|-----------|----------|
    |                     | BD-logKID↓| BD-LPIPS↓| BD-logKID↓| BD-LPIPS↓| BD-logKID↓| BD-LPIPS↓| BD-logKID↓| BD-LPIPS↓|
    | _MSE Baselines_     |           |          |           |          |           |          |           |          |
    | Hyper               | 0.0000    | 0.0000   | 0.0000    | 0.0000   | 0.0000    | 0.0000   | 0.0000    | 0.0000   |
    | ELIC                | -0.2320   | -0.0402  | -0.348    | -0.0580  | -0.236    | -0.0623  | -0.4067   | -0.0595  |
    | BPG                 | 0.1506    | -0.0102  | 0.0274    | -0.0102  | 0.1267    | -0.0116  | 0.03985   | -0.00795 |
    | VTM                 | -0.2320   | -0.0311  | -0.2975   | -0.0476  | -0.2157   | -0.0504  | -2.049    | -0.04812 |
    | _Conditional-based_ |           |          |           |          |           |          |           |          |
    | HiFiC               | -3.132    | -0.1087  | -2.274    | -0.1724  | -2.049    | -0.1723  | -1.925    | -0.1483  |
    | HiFiC*              | -4.261    | -0.1102  | -2.780    | -0.1734  | N/A       | N/A      | N/A       | N/A      |
    | Po-ELIC             | -3.504    | -0.1047  | -2.877    | -0.1674  | -2.671    | -0.1687  | -2.609    | -0.145   |
    | CDC                 | -2.072    | -0.0600  | -1.968    | -0.0985  | -1.978    | -0.1011  | -2.122    | -0.08437 |
    | ILLM                | -3.418    | -0.1092  | -2.681    | -0.1809  | -2.62     | -0.1805  | -2.882    | -0.1547  |
    | ILLM*               | -4.256    | -0.1062  | -2.673    | -0.1784  | N/A       | N/A      | N/A       | N/A      |
    |_Unconditional-based_|           |          |           |          |           |          |           |          |
    | Proposed (Hyper)    | -5.107    | -0.0859  | -4.271    | -0.0576  | -4.519    | -0.0826  | -3.787    | -0.0557  |
    | Proposed (ELIC)     | -5.471    | -0.0987  | -5.694    | -0.1062  | -5.36     | -0.113   | -4.046    | -0.0796  |
* The result of KID has same trend as FID, which means that our approach is sota. While many other approaches outperform our approach in LPIPS. This result is expected, as:
    * KID is a divergence based metric. And our approach is optimized for divergence.
    * LPIPS is a image-to-image distortion. By distortion-perception trade-off [Blau 2018], perception optimized approaches can not achieve SOTA LPIPS.
    * All other perceptual codec (HiFiC, ILLM, CDC, Po-ELIC) except for ours use LPIPS as loss function during training.
    * ILLM [Muckley 2023] also achieves sota FID, KID and visual quality at that time, but its LPIPS is outperformed by a autoencoder trained with LPIPS.

Sure, below we present the result of MS-SSIM:

|                  | FFHQ    | ImageNet | COCO    | CLIC    |
|------------------|---------|----------|---------|---------|
| Hyper            | 0.0000  | 0.0000   | 0.0000  | 0.0000  |
| ELIC             | 0.0098  | 0.0148   | 0.0146  | 0.0176  |
| HiFiC            | 0.0030  | 0.0064   | 0.0071  | 0.0058  |
| ILLM             | 0.0001  | 0.0041   | 0.0058  | 0.0042  |
| Proposed (Hyper) | -0.0065 | -0.0399  | -0.0262 | -0.0236 |
| Proposed (ELIC)  | -0.0029 | -0.0203  | -0.0135 | -0.0132 |

In terms of MS-SSIM, ELIC > HiFiC > ILLM > Hyper > Proposed (ELIC) > Proposed (Hyper). We are reluctant to use MS-SSIM as perceptual metric, as this result is obviously not aligned with visual quality, and should not be used when we considering divergence based perceptual quality as [Blau 2018], because:
* MS-SSIM is an image to image distortion. By [Blau 2018], it is in odd with divergence based metrics such as FID, KID. Theoretically, there does not exist a codec that achieve optimal FID and MS-SSIM at the same time.
* In terms of MS-SSIM, ELIC, a mse optimzied codec, is the sota. This indicates that MS-SSIM correlates poorly with human perceptual quality.
* Similar result is also reported in [Muckley 2023] Fig. 3, where the MS-SSIM of ILLM is not even as good as Hyper, which is a mse optimized codec. This indicates that MS-SSIM correlates poorly with human perceptual quality.
* In a perceptual codec competition CVPR CLIC 2021 (https://clic.compression.cc/2021/leaderboard/image-075/test/index.html), the human perceptual quality is tested as final result and MS-SSIM, FID are evaluated. The 1st place of human perceptual test result among 23 teams "MIATL_NG", also has lowest FID, which indicates FID correlates well with human perceptual quality. On the other hand, its MS-SSIM ranks 21st place among 23 teams, which indicates MS-SSIM correlates poorly with human perceptual quality.

We will include MS-SSIM result in our final version, but we emphasis that this result should be treated with care.

Reviewer comment (Experiments):
The primary weakness is a lack of comparison to other diffusion-based methods. The only diffusion method considered is CDC, which may be relatively underpowered vs. DIRAC (Ghouse, 2023). CDC is almost certainly underpowered vs. the work of (Hoogeboom, 2023). Also, there are images released from the work of (Agustsson, 2022), which are not included in the paper.

Author response:
The reason why we only compare to CDC among diffusion codec is that it is the only diffusion codec that provides test code. Furthermore, [Goose 2023] [Hoogeboom 2023] are not yet published. Not comparing to them should not be considered as a weakness. Our testing dataset has different size and resolution than [Agustsson 2022], and that is why we have not compared to the image released. However, we do note that [Muckley 2023] outperform the datapoint released by [Agustsson 2022] in terms of FID, and we outperform [Muckley 2023] in terms of FID.

Reviewer comment (Novelty):
The x-domain constraint seems identical to that of (Hoogeboom, 2023).

Author response:
We are not sure if we get this correctly, so please correct us if our understanding is wrong. It appears to us that [Hoogeboom 2023] is a conditional generative codec, while its condition is MSE reconstruction $g_0(f_0(x))$. The diffusion part takes $g_0(f_0(x))$ as condition, and rectified flow allows very fast sampling within a few steps. We search for the word "unconditional" in [Hoogeboom 2023] but we fail to find relevant information. Thus it appears to me that [Hoogeboom 2023] is a conditional codec using $g_0(f_0(x))$ as condition, while x-domain constraint is $||g_0(f_0(\hat{x}))-g_0(f_0(x))||$. They both contains $g_0(f_0(x))$, but we do not think they are identical.

Reviewer comment (Reproducibility):
The paper uses a third-party implementation of HiFiC with a different training set that may give divergent results from the original paper. The authors of the HiFiC paper have made their models available at https://github.com/tensorflow/compression.

Author response:
Sure, we re-evaluate the original tensorflow HiFiC, and the results are as follows:

    |                    | FFHQ   |         | ImageNet |         | COCO   |         | CLIC   |         |
    |--------------------|--------|---------|----------|---------|--------|---------|--------|---------|
    |                    | BD-FID↓| BD-PSNR↓| BD-FID↓  | BD-PSNR↓| BD-FID↓| BD-PSNR↓| BD-FID↓| BD-PSNR↓|
    | HiFiC (Pytorch)    | -48.35 | -2.036  | -44.52   | -1.418  | -44.88 | -1.276  | -36.16 | -1.621  |
    | HiFiC (Tensorflow) | -46.44 | -1.195  | -40.25   | -0.8287 | -46.45 | -0.9176 | -35.90 | -0.9202 |

* In terms of BD-FID, the torch version outperforms tf version in FFHQ, ImageNet and CLIC dataset, while it is outperformed by tf version in COCO. The overall conclusion is not impacted.

Reviewer comment (Evaluation):
Only two metrics are considered: FID and PSNR. Most other compression works include more metrics, such as LPIPS, to gauge model performance across a variety of axes.

Author response:
Sure, we have evaluated our approach additionally on KID and LPIPS:

    | Method              | FFHQ      |          | ImageNet  |          | COCO      |          | CLIC      |          |
    |---------------------|-----------|----------|-----------|----------|-----------|----------|-----------|----------|
    |                     | BD-logKID↓| BD-LPIPS↓| BD-logKID↓| BD-LPIPS↓| BD-logKID↓| BD-LPIPS↓| BD-logKID↓| BD-LPIPS↓|
    | _MSE Baselines_     |           |          |           |          |           |          |           |          |
    | Hyper               | 0.0000    | 0.0000   | 0.0000    | 0.0000   | 0.0000    | 0.0000   | 0.0000    | 0.0000   |
    | ELIC                | -0.2320   | -0.0402  | -0.348    | -0.0580  | -0.236    | -0.0623  | -0.4067   | -0.0595  |
    | BPG                 | 0.1506    | -0.0102  | 0.0274    | -0.0102  | 0.1267    | -0.0116  | 0.03985   | -0.00795 |
    | VTM                 | -0.2320   | -0.0311  | -0.2975   | -0.0476  | -0.2157   | -0.0504  | -2.049    | -0.04812 |
    | _Conditional-based_ |           |          |           |          |           |          |           |          |
    | HiFiC               | -3.132    | -0.1087  | -2.274    | -0.1724  | -2.049    | -0.1723  | -1.925    | -0.1483  |
    | HiFiC*              | -4.261    | -0.1102  | -2.780    | -0.1734  | N/A       | N/A      | N/A       | N/A      |
    | Po-ELIC             | -3.504    | -0.1047  | -2.877    | -0.1674  | -2.671    | -0.1687  | -2.609    | -0.145   |
    | CDC                 | -2.072    | -0.0600  | -1.968    | -0.0985  | -1.978    | -0.1011  | -2.122    | -0.08437 |
    | ILLM                | -3.418    | -0.1092  | -2.681    | -0.1809  | -2.62     | -0.1805  | -2.882    | -0.1547  |
    | ILLM*               | -4.256    | -0.1062  | -2.673    | -0.1784  | N/A       | N/A      | N/A       | N/A      |
    |_Unconditional-based_|           |          |           |          |           |          |           |          |
    | Proposed (Hyper)    | -5.107    | -0.0859  | -4.271    | -0.0576  | -4.519    | -0.0826  | -3.787    | -0.0557  |
    | Proposed (ELIC)     | -5.471    | -0.0987  | -5.694    | -0.1062  | -5.36     | -0.113   | -4.046    | -0.0796  |
* The result of KID has same trend as FID, which means that our approach is sota. While many other approaches outperform our approach in LPIPS. This result is expected, as:
    * KID is a divergence based metric. And our approach is optimized for divergence.
    * LPIPS is a image-to-image distortion. By distortion-perception trade-off [Blau 2018], perception optimized approaches can not achieve SOTA LPIPS.
    * All other perceptual codec (HiFiC, ILLM, CDC, Po-ELIC) except for ours use LPIPS as loss function during training.
    * ILLM [Muckley 2023] also achieves sota FID, KID and visual quality at that time, but its LPIPS is outperformed by a autoencoder trained with LPIPS.

Reviewer comment (Theory):
The failures of many of the generative models to give good results suggests a divergence between theory in practice, i.e., although the theory suggests any unconditional generator will do, the generators that we have available to us and their various samplers exhibit widely different properties. Did the authors consider discussing this in the Discussion?

Author response:
Thanks for the advice, we will add this discussion. Our theory relies on the assumption that the generative model perfectly learns the natural image distribution, while this is hard to achieve in practice. And we think this is where the gap lies. More specifically, GAN has better "precision" and worse "recall", while diffusion has a fair "precision" and fair "recall". The "precision" and "recall" is defined in [Improved precision and recall metric for assessing generative models]. Or to say, the distribution learned by GAN largely lies in the natural image distribution better, but many area of natural image distribution is not covered. The distribution learned by diffusion does not lies in natural image distribution as good as GAN, but it covers more area of natural image distribution. This phenomena is also observed and discussed by [Diffusion Models Beat GANs on Image Synthesis] and [Classifier-Free Diffusion Guidance]. The "precision" is more important for sampling, but "recall" is more important for inversion. That is why the GAN based approach fails. The MCG fails probably because it is not well suited for non-linear problems.

Reviewer comment (Evaluation):
Could you clarify why you used the OASIS FID implementation as opposed to the FID/256 method that is more standard in the compression community?

Author response:
[Agustsson 2022] [Muckley 2023] evaluate FID with pathsize 256 on 256x256 COCO dataset with 30000 images and around 2000x2000 CLIC dataset with 400 images. The former produces 30000 patches and the later produces around 15000 pathes. As our approach is more expensive to evaluate, we can not afford testing on that large dataset. Therefore, we test our approach on 256x256 COCO dataset with 1000 images and 256x256 CLIC dataset with 400 images. In that case, using FID/256 will produce only 1000 and 400 patches. As FID is biased [Effectively Unbiased FID and Inception Score and where to find them], 1000 datapoints is too small for FID evaluation. Therefore, we evaluate FID with 64x64 patches, which produce 16000 and 6400 datapoints, which is reasonablely large. And this patch number is close to [Agustsson 2022] [Muckley 2023]. Note FID is proposed to be used with 64x64 CelebA and 64x64 ImageNet [Two time-scale update rule for training GANs]. So we think it is fine to use FID with 64x64 patches.

Reviewer comment (Experiments):
The rate-distortion-perception tradeoff is portrayed in a confusing way in Figure 8 of the Appendix. Is the reason that you could not match bpps between methods as done in Figure 1 of (Muckley, 2023)?

Author response:
Sure, we can draw the perception-distortion trade-off as Fig. 1 of [Muckley 2023], and the result is shown in https://ibb.co/DrtyXTG. We choose this way because we can compare to other methods easily, while the way of [Muckley 2023] can only compare to ourselves as we can not match the bitrate of other methods without selecting the bitrate of our base model carefully.

Reviewer comment (Writing):
Would you consider to use language other than "perception" to describe distributional divergence? "Perception" is an overloaded term in the compression community and does not intuitively describe the phenomena of the paper. Although it is true that "perception" is the term of Blau/Michaeli to describe divergence phenomena, others in the community have adopted more specific terms such as "realism" or "statistical fidelity" that lead to less confusion with human perception.

Author response:
Thanks for this advice, we will revise it into divergence-based perception to distinguish it with human perception.

Reviewer comment (Experiments):
The issue regarding the image resolution should be paid with more attention. We can imagine that the gradient descent-based inversion would be hard to be implemented on high-resolution images. In the experimental section of this paper, it is stated like "central crop image by their short edge and rescale them to 256 × 256", which means almost all experiments are performed on this scale. On the one hand, some baseline models are trained for high-resolution generation or generative compression, and it may be unfair to directly compare with their pretrained models. On the other hand, it implies that the current method is hard to be applied to high-resolution images without cropping.

Author response:
As we have discussed in limitation, we agree that the resolution is a problem. We have done some very early attempt using diffusion model with larger resolution. For example, our approach works with 512x512 Stable Diffusion 1.5 and null text prompt. And a visual example with [Balle 2018] and ImageNet 512x512 Image is: https://ibb.co/xCd9gMB.  
We have not tried 1024x1024 Stable Diffusion 2.0. Note that as Stable Diffusion is a latent diffusion, the latent space size is much smaller than pixel size. And thus its complexity does not grow that much. Currently our higher-resolution codec relies on higher-resolution diffusion model. We are also aware that there are other pathch-wise solutions [DiffCollage: Parallel Generation of Large Content with Diffusion Models], but they are not the mainstream in diffusion community and we have not given them a try.  
For the baselines. we note that all the baselines are trained on 256x256 images. And to secure fair comparsion, we also provide the result of HiFiC and ILLM re-trained on 256x256 FFHQ and ImageNet in Tab.2.

Reviewer comment (Evaluation):
The approach proposed in this paper can utilize pre-trained unconditional generative model for compression at different bitrates. In Figure 9, it seems the advantage of the proposed method is more significant at 0.17bpp, compared with original ELIC. However, the result at 0.11 bpp seems to be weird, which is definitely what we do not want in practice. Is there is potential solution or idea to solve this issue?

Author response:
The weird reconstruction is due to an improper hyper-parameter. More specifically, the scale parameter $\zeta=0.3$ is too small, which lead to the reconstruction deviate from original image. We have shown additional results varying $\zeta=0.6$ in https://ibb.co/XXRQN2F. Clearly use $\zeta=0.6$ improve the result.

Reviewer comment (Novelty):
I admit that the findings of high correlation between the idempotence and perceptual image compression are interesting and meaningful. However, the subsequent idea of "perceptual image compression by inversion" does not bring new insights or knowledge.

Author response:
First, migrating from super-resolution to codec is not that straightforward. Unlike super-resolution and other restortation, codec is highly non-linear. And many previously proposed approaches that work well for super-resolution, such as [Menon 2020] [Daras 2021] [Chung 2022a], fail. As we have shown in Tab. 1, a thorough curation of different generative model and inversion approach is required to make it actually work. Furthermore, in Sec. 4.2, we additionally propose to use the straight through estimation to make codec differentiable, and the x-domain y-domain constraint. Those techniques are codec specific and have never been studied in super-resolution.
* Second, indeed we borrow the empirical approach from super-resolution. However, we also give back theoretical understanding. Despite inversion-based super-resolution has been studied for years empirically, their theoretical relationship with conditional model based super-resolution, and distortion-perception trade-off is in general unknown. PULSE [Menon 2020] is the poineer of this area, and they justify their approach by "natural image manifold" assumption. And later works in inversion-based super-resolution follow PULSE's story. On the other hand, our __Theorem 1__, __Theorem 2__ only limit the operator $f(.)$ to be deterministic, and it need not to be a codec. Then, our __Theorem 1__, __Theorem 2__ and __Corollary 1__ are also appliable to inversion based super-resolution, at least when the degeneration operator is deterministic. In other words, our theoretical results also implies that:
    * __Theorem 1'__ (super-resolution version) Conditional generative super-resolution also satisfies idempotence, that is, the up-sampled image can downsample into low-resolution image.
    * __Theorem 2'__ (super-resolution version) Inversion-based super-resolution is theoretically equvalient to conditional generative super-resolution.
    * __Corollary 1'__ (super-resolution version) Inversion-based super-resolution satisfies the theoretical results of distortion-perception trade-off, that is the MSE is at most double of best MSE.
* We believe that our result provides non-trivial theoretical insights to inversion-based super-resolution community. For example, [Menon 2020] claim that the advantage of inversion-based super-resolution over the conditional generative super-resolution is that inversion-based super-resolution "downscale correctly". However, with our theoretical result, we know that ideal conditional generative super-resolution also "downscale correctly". Currently they fail to achieve this due to implemention issue. Another example is that most inversion-based super-resolution [Menon 2020] [Daras 2021] [Chung 2022a] report no MSE comparsion with sota MSE super-resolution, as it is for sure that their MSE is worse and there seems to be no relationship between their MSE and sota MSE. However, with our theoretical result, we know that their MSE should be smaller than 2x sota MSE. And they should examine whether their MSE falls below 2x sota MSE.

Reviewer comment (Novelty):
Applying generative model inversion on the low-level tasks such as super-resolution has been well studied for several years. The proposed method seems to just change the task from super-resolution to image compression. Applying a proven methodology to a similar task, in my opinion, hardly meets the ICLR's bar.

Author response:
First, migrating from super-resolution to codec is not that straightforward. Unlike super-resolution and other restortation, codec is highly non-linear. And many previously proposed approaches that work well for super-resolution, such as [Menon 2020] [Daras 2021] [Chung 2022a], fail. As we have shown in Tab. 1, a thorough curation of different generative model and inversion approach is required to make it actually work. Furthermore, in Sec. 4.2, we additionally propose to use the straight through estimation to make codec differentiable, and the x-domain y-domain constraint. Those techniques are codec specific and have never been studied in super-resolution.
* Second, indeed we borrow the empirical approach from super-resolution. However, we also give back theoretical understanding. Despite inversion-based super-resolution has been studied for years empirically, their theoretical relationship with conditional model based super-resolution, and distortion-perception trade-off is in general unknown. PULSE [Menon 2020] is the poineer of this area, and they justify their approach by "natural image manifold" assumption. And later works in inversion-based super-resolution follow PULSE's story. On the other hand, our __Theorem 1__, __Theorem 2__ only limit the operator $f(.)$ to be deterministic, and it need not to be a codec. Then, our __Theorem 1__, __Theorem 2__ and __Corollary 1__ are also appliable to inversion based super-resolution, at least when the degeneration operator is deterministic. In other words, our theoretical results also implies that:
    * __Theorem 1'__ (super-resolution version) Conditional generative super-resolution also satisfies idempotence, that is, the up-sampled image can downsample into low-resolution image.
    * __Theorem 2'__ (super-resolution version) Inversion-based super-resolution is theoretically equvalient to conditional generative super-resolution.
    * __Corollary 1'__ (super-resolution version) Inversion-based super-resolution satisfies the theoretical results of distortion-perception trade-off, that is the MSE is at most double of best MSE.
* We believe that our result provides non-trivial theoretical insights to inversion-based super-resolution community. For example, [Menon 2020] claim that the advantage of inversion-based super-resolution over the conditional generative super-resolution is that inversion-based super-resolution "downscale correctly". However, with our theoretical result, we know that ideal conditional generative super-resolution also "downscale correctly". Currently they fail to achieve this due to implemention issue. Another example is that most inversion-based super-resolution [Menon 2020] [Daras 2021] [Chung 2022a] report no MSE comparsion with sota MSE super-resolution, as it is for sure that their MSE is worse and there seems to be no relationship between their MSE and sota MSE. However, with our theoretical result, we know that their MSE should be smaller than 2x sota MSE. And they should examine whether their MSE falls below 2x sota MSE.

Reviewer comment (Experiments):
In the experiments, a MSE optimized codec is used as the base model. Have the authors tried other codecs (perceptual optimized or mixed)?

Author response:
* __Q1__. By [Blau 2018] and __Corollary 1__, the MSE of inversion-based perceptual codec is bounded by 2x base codec. Using a perceptual codec as base codec:
    * does not help with perceptual quality, as theoretically any base codec inversion achieves perfect perceptual quality.
    * does loosen MSE upperbound, as the current upperbound is 2x MSE of a perceptual codec. And it is very likely that the MSE of our codec grows larger.
* Therefore, we choose to use a MSE optimized codec, to obtain the tightest MSE bound possible.

Reviewer comment (Evaluation):
Could the authors provide more results on other metrics, such as KID, LPIPS?

Author response:
* __Q2__. Sure, we have evaluated our approach additionally on KID and LPIPS:

    | Method              | FFHQ      |          | ImageNet  |          | COCO      |          | CLIC      |          |
    |---------------------|-----------|----------|-----------|----------|-----------|----------|-----------|----------|
    |                     | BD-logKID↓| BD-LPIPS↓| BD-logKID↓| BD-LPIPS↓| BD-logKID↓| BD-LPIPS↓| BD-logKID↓| BD-LPIPS↓|
    | _MSE Baselines_     |           |          |           |          |           |          |           |          |
    | Hyper               | 0.0000    | 0.0000   | 0.0000    | 0.0000   | 0.0000    | 0.0000   | 0.0000    | 0.0000   |
    | ELIC                | -0.2320   | -0.0402  | -0.348    | -0.0580  | -0.236    | -0.0623  | -0.4067   | -0.0595  |
    | BPG                 | 0.1506    | -0.0102  | 0.0274    | -0.0102  | 0.1267    | -0.0116  | 0.03985   | -0.00795 |
    | VTM                 | -0.2320   | -0.0311  | -0.2975   | -0.0476  | -0.2157   | -0.0504  | -2.049    | -0.04812 |
    | _Conditional-based_ |           |          |           |          |           |          |           |          |
    | HiFiC               | -3.132    | -0.1087  | -2.274    | -0.1724  | -2.049    | -0.1723  | -1.925    | -0.1483  |
    | HiFiC*              | -4.261    | -0.1102  | -2.780    | -0.1734  | N/A       | N/A      | N/A       | N/A      |
    | Po-ELIC             | -3.504    | -0.1047  | -2.877    | -0.1674  | -2.671    | -0.1687  | -2.609    | -0.145   |
    | CDC                 | -2.072    | -0.0600  | -1.968    | -0.0985  | -1.978    | -0.1011  | -2.122    | -0.08437 |
    | ILLM                | -3.418    | -0.1092  | -2.681    | -0.1809  | -2.62     | -0.1805  | -2.882    | -0.1547  |
    | ILLM*               | -4.256    | -0.1062  | -2.673    | -0.1784  | N/A       | N/A      | N/A       | N/A      |
    |_Unconditional-based_|           |          |           |          |           |          |           |          |
    | Proposed (Hyper)    | -5.107    | -0.0859  | -4.271    | -0.0576  | -4.519    | -0.0826  | -3.787    | -0.0557  |
    | Proposed (ELIC)     | -5.471    | -0.0987  | -5.694    | -0.1062  | -5.36     | -0.113   | -4.046    | -0.0796  |
* The result of KID has same trend as FID, which means that our approach is sota. While many other approaches outperform our approach in LPIPS. This result is expected, as:
    * KID is a divergence based metric. And our approach is optimized for divergence.
    * LPIPS is a image-to-image distortion. By distortion-perception trade-off [Blau 2018], perception optimized approaches can not achieve SOTA LPIPS.
    * All other perceptual codec (HiFiC, ILLM, CDC, Po-ELIC) except for ours use LPIPS as loss function during training.
    * ILLM [Muckley 2023] also achieves sota FID, KID and visual quality at that time, but its LPIPS is outperformed by a autoencoder trained with LPIPS.