Paper: The Uncertainty-Perception Tradeoff (ICLR-2024)

Reviewer comment (Experiments):
In my opinion, demonstrating how the framework can be used in practical settings is seriously lacking, somewhat undermining the potential impact and significance of the work. The authors claim that the introduced framework can be used to 1) assess estimator performance and 2) identify areas of improvement for image reconstruction techniques. Even though Figures 4 and 5 demonstrate that the framework gives sensible results that correspond to what we would expect from the theorems (higher perceptual quality results in higher uncertainty and higher distortion), it is not clear to me how much we can gain from this. In particular, I believe if we simply plotted LPIPS vs PSNR on various datasets for these techniques, we would see the same trend. That being said, the key contribution of the framework could be answering questions such as "How much can we improve the perceptual quality/distortion of this network based on our model of the trade-off?". Authors mention this direction repeatedly in the paper, but there are no experiments or further discussion supporting this. For instance, can we determine where a certain reconstruction technique lies in the uncertainty-perception plane and use that to derive useful clues what type of improvement we can still expect from our estimator? I believe that focusing on answering these questions on practical reconstruction techniques would very significantly increase the significance of the paper.

Author response:
We thank the reviewer for their detailed valuable feedback and constructive criticism, both of which we have carefully considered and addressed in our response below.

Major Issues
* Practical Implications:  We highly appreciate the reviewer's detailed feedback regarding the practical implications of our work. First, we completely agree with the reviewer that evaluating estimators’ performance using practical and established measures like LPIPS would yield similar trends to those presented. Our primary objective is to provide theoretical justification for such empirical observations by establishing a fundamental relationship between uncertainty and perception. This theoretical foundation serves as the core contribution of our research, with the numerical experiments primarily serving an illustrative purpose, aligning with previous works such as Blau & Michaeli (2018) and Freirich et al. (2021).

Despite the theoretical focus, our work offers several practical implications -

(1) Tradeoff Awareness: Developers should be aware of the tradeoff between high perceptual quality and low uncertainty when designing or optimizing restoration algorithms, prioritizing specific needs based on their application context.

(2) Uncertainty-Perception Plane: Plotting algorithms on this plane, using any established measures of uncertainty and perception, should reveal potential areas for improvement. In our case, enhancing DIP's uncertainty without compromising perceptual quality presents a potential optimization opportunity. Additionally, attempts to improve EDSR's perception will likely increase its uncertainty, while lowering ESRGAN's uncertainty may compromise perceptual quality.

Reviewer comment (Writing):
The paper has multiple typos and there is a repeated sentence in Section 2, please check.

Author response:
Minor Issue - In response to the reviewer's feedback, we have carefully proofread the manuscript to ensure accuracy and clarity.

Reviewer comment (Evaluation):
With respect to 1) in Weaknesses: Can the proposed framework reveal any insights about estimator performance that would not be possible by simply evaluating perceptual quality and distortion metrics?

Author response:
We thank the reviewer for their detailed valuable feedback and constructive criticism, both of which we have carefully considered and addressed in our response below.

Major Issues
* Practical Implications:  We highly appreciate the reviewer's detailed feedback regarding the practical implications of our work. First, we completely agree with the reviewer that evaluating estimators’ performance using practical and established measures like LPIPS would yield similar trends to those presented. Our primary objective is to provide theoretical justification for such empirical observations by establishing a fundamental relationship between uncertainty and perception. This theoretical foundation serves as the core contribution of our research, with the numerical experiments primarily serving an illustrative purpose, aligning with previous works such as Blau & Michaeli (2018) and Freirich et al. (2021).

Despite the theoretical focus, our work offers several practical implications -

(1) Tradeoff Awareness: Developers should be aware of the tradeoff between high perceptual quality and low uncertainty when designing or optimizing restoration algorithms, prioritizing specific needs based on their application context.

(2) Uncertainty-Perception Plane: Plotting algorithms on this plane, using any established measures of uncertainty and perception, should reveal potential areas for improvement. In our case, enhancing DIP's uncertainty without compromising perceptual quality presents a potential optimization opportunity. Additionally, attempts to improve EDSR's perception will likely increase its uncertainty, while lowering ESRGAN's uncertainty may compromise perceptual quality.

Reviewer comment (Evaluation):
With respect to 2) in Weaknesses: How can the proposed framework be used to identify areas of improvement concretely in practice?

Author response:
We thank the reviewer for their detailed valuable feedback and constructive criticism, both of which we have carefully considered and addressed in our response below.

Major Issues
* Practical Implications:  We highly appreciate the reviewer's detailed feedback regarding the practical implications of our work. First, we completely agree with the reviewer that evaluating estimators’ performance using practical and established measures like LPIPS would yield similar trends to those presented. Our primary objective is to provide theoretical justification for such empirical observations by establishing a fundamental relationship between uncertainty and perception. This theoretical foundation serves as the core contribution of our research, with the numerical experiments primarily serving an illustrative purpose, aligning with previous works such as Blau & Michaeli (2018) and Freirich et al. (2021).

Despite the theoretical focus, our work offers several practical implications -

(1) Tradeoff Awareness: Developers should be aware of the tradeoff between high perceptual quality and low uncertainty when designing or optimizing restoration algorithms, prioritizing specific needs based on their application context.

(2) Uncertainty-Perception Plane: Plotting algorithms on this plane, using any established measures of uncertainty and perception, should reveal potential areas for improvement. In our case, enhancing DIP's uncertainty without compromising perceptual quality presents a potential optimization opportunity. Additionally, attempts to improve EDSR's perception will likely increase its uncertainty, while lowering ESRGAN's uncertainty may compromise perceptual quality.

Reviewer comment (Theory):
Assumptions like unbiasedness and Markov chain may limit applicability in some cases.

Author response:
* Assumptions: 
(1) Unbiasedness - this assumption holds in our case without loss of generality, since our uncertainty measure, power entropy, is translation-invariant and thus insensitive to bias.
(2) Markov Chain - this assumption suggests that the estimation process effectively captures all relevant information about X from Y. As we do not have access to the true signal X, this is a reasonable assumption which aligns with  existing literature on restoration tasks, in particular with [Blau & Michaeli 18] and [Freirich et al. 21]). Intuitively, the details in the estimation that are independent of the true image can be seen as "hallucinations" that contribute to improved perceptual quality. However, these same details also lead to higher uncertainty, as they deviate from the ground truth and introduce additional variability into the estimation.
We assure the reviewer that the revised manuscript includes an extended discussion of these assumptions and clarifies the points mentioned above to ensure better understanding and transparency.

Reviewer comment (Experiments):
More analysis for other divergence measures besides Renyi could strengthen claims.

Author response:
* Additional Divergence Measures: We appreciate the reviewer's suggestion to explore other divergence measures. Following the approach of Blau & Michaeli (2018), we first define the uncertainty-perception function using a theoretic definition of perceptual quality as a divergence between the true and estimated distributions. This allows us to establish general properties of the function. Then, we follow the example of Freirich et al. (2021) and consider a specific perception measure, Renyi divergence. This choice aligns with widely used measures like KL divergence and Hellinger distance, enabling us to derive further insights about the uncertainty-perception function. We believe this approach ensures consistency with existing literature and applicability to many common cases. We emphasize these points in the revised manuscript.

Reviewer comment (Experiments):
Additional validation on diverse restoration tasks needed to fully support general claims.

Author response:
* Generalization to Other Models and Tasks:  We reiterate that the primary focus of this paper is the theoretical examination of the interplay between uncertainty and perception. Following Blau & Michaeli (2018) and Freirich et al. (2021), we utilize super-resolution algorithms as an illustrative example to demonstrate our theoretical results. Our analysis does not rely on any specific task or model and thus theoretically generalizes to other settings. We agree that further experiments involving various tasks and models would be valuable. However, as our current work emphasizes theoretical contributions, we leave such extensive empirical investigations for a future paper, similar to "The 2018 PIRM Challenge on Perceptual Image Super-resolution."

Reviewer comment (Experiments):
Does the proposed strategy work only for image super-resolution? Could it work for other models and tasks?

Author response:
* Generalization to Other Models and Tasks:  We reiterate that the primary focus of this paper is the theoretical examination of the interplay between uncertainty and perception. Following Blau & Michaeli (2018) and Freirich et al. (2021), we utilize super-resolution algorithms as an illustrative example to demonstrate our theoretical results. Our analysis does not rely on any specific task or model and thus theoretically generalizes to other settings. We agree that further experiments involving various tasks and models would be valuable. However, as our current work emphasizes theoretical contributions, we leave such extensive empirical investigations for a future paper, similar to "The 2018 PIRM Challenge on Perceptual Image Super-resolution."

Reviewer comment (Theory):
Can the theory guide development of new algorithms to achieve better uncertainty-perception tradeoffs?

Author response:
* Theory to Practice: First, we would like to emphasize that the primary focus of our work is theoretical analysis and its contribution lies in illuminating the relationship between uncertainty and perception in restoration tasks. Nevertheless, our theoretical results can offer practical insights for algorithms, both existing and new. The tradeoff we identified suggests that high perceptual quality usually leads to higher uncertainty, and vice versa. Developers should consider this tradeoff when setting requirements for their solutions. For example, examining Figure 4 suggests opportunities for improving DIP in terms of uncertainty without sacrificing perceptual quality. Additionally, attempts to improve EDSR's perception will likely increase its uncertainty, while lowering ESRGAN's uncertainty will likely come at the cost of decreased perception. 
We address these points in the revised manuscript, where we broaden the relevant discussions and emphasize the theoretical focus of our work. Furthermore, we will revise the title to " A Theory of Uncertainty-Perception Tradeoff" of the camera-ready version to better reflect this emphasis.

Reviewer comment (Writing):
Introduction is confusing for those not familiar with the "perceptual quality" concept (divergence between PDFs) introduced in [Blau & Michaeli 18].

Author response:
We appreciate the reviewer's concern about the potential ambiguity of "perceptual quality" and the suggestion to clarify its definition in our manuscript. We agree that quantifying this concept is a complex and evolving field with diverse approaches. To ensure consistency with previous works, we follow the theoretical definition of perceptual quality as the divergence between probability density functions (PDFs), as proposed by [Blau & Michaeli 18, 19]. This approach aligns with relevant prior work and establishes a clear framework for our analysis. 
Following the reviewer's feedback, we have clarified the above in the Introduction explicitly stating our chosen definition of perceptual quality and its connection to PDF divergence. We have added a new paragraph on "Perception Quantification" discussing the various ongoing efforts and diverse approaches in this field.

Reviewer comment (Theory):
Perceptual distances between samples (the regular perceptual quality concept) have been related to the signal PDFs as well.

Author response:
We appreciate the reviewer's concern about the potential ambiguity of "perceptual quality" and the suggestion to clarify its definition in our manuscript. We agree that quantifying this concept is a complex and evolving field with diverse approaches. To ensure consistency with previous works, we follow the theoretical definition of perceptual quality as the divergence between probability density functions (PDFs), as proposed by [Blau & Michaeli 18, 19]. This approach aligns with relevant prior work and establishes a clear framework for our analysis. 
Following the reviewer's feedback, we have clarified the above in the Introduction explicitly stating our chosen definition of perceptual quality and its connection to PDF divergence. We have added a new paragraph on "Perception Quantification" discussing the various ongoing efforts and diverse approaches in this field.

Reviewer comment (Evaluation):
Experimental illustration of super resolution is limited: no visual examples are given. Note that the "perceptual quality" concept defined from divergence is an abstraction. Actual assessment of the perceptual quality should be done through visual inspection, but no visual examples are given.

Author response:
We agree with the reviewer's comments and accept their suggestion to include a visual comparison of various estimators concerning uncertainty and perception in the main body of the paper.

Reviewer comment (Evaluation):
Numerical evaluation is prone to error because entropy and divergence values are based on estimations from samples in high-dimensional scenarios. This should be acknowledged.

Author response:
We agree with the reviewer's concern about the challenges of entropy and divergence estimation in high dimensions. Following previous works by [Blau & Michaeli 18] and [Freirich et al. 21], we perform our estimations at the patch level, where we have more than 16 million samples. However, we acknowledge that challenges and potential limitations remain associated with high-dimensional estimations, which we explicitly state in the revised manuscript.  We highly appreciate the reviewer's suggestions and pointers to improved estimators, which we hope to include in the camera-ready version.

Reviewer comment (Writing):
The introduction should include a citation to the "perceptual quality" concept defined from divergence between PDFs as done in [Blau & Michaeli 18, 19] because if not, the key question "Can one design an AI model of high perceptual quality which exhibits low uncertainty" is unclear or vague.

Author response:
We appreciate the reviewer's concern about the potential ambiguity of "perceptual quality" and the suggestion to clarify its definition in our manuscript. We agree that quantifying this concept is a complex and evolving field with diverse approaches. To ensure consistency with previous works, we follow the theoretical definition of perceptual quality as the divergence between probability density functions (PDFs), as proposed by [Blau & Michaeli 18, 19]. This approach aligns with relevant prior work and establishes a clear framework for our analysis. 
Following the reviewer's feedback, we have clarified the above in the Introduction explicitly stating our chosen definition of perceptual quality and its connection to PDF divergence. We have added a new paragraph on "Perception Quantification" discussing the various ongoing efforts and diverse approaches in this field.

Reviewer comment (Presentation):
The "related work" section should include a paragraph on "Perceptual quality quantification". This section should link the Blau & Michaeli definition with "no-reference distance" definitions based on the similarity between PDFs (e.g. citations to [32-34] and associated reasoning in [Blau & Michaeli 18]). However, note that this divergence concept is related to the usefulness of summary statistics to capture the nature of textures, as in [Portilla & Simoncelli IJCV 2000], in more recent style transfer algorithms based on difference of Gram matrices in VGG-like nets [Gatys et al. CVPR 2016]) or in state-of-the-art perceptual distortion metrics such as DISTS [Ding et al. IEEE PAMI 2020]. Also worth citing in that paragraph is the recent work relating perceptual distances and PDFs [Hepburn et al. ICLR 2022], which makes interesting points on the difference between individual distances between samples and averages over ensembles. In particular [Hepburn et al. ICLR 2022] shows that perceptual distances capture relevant information on the image PDF.

Author response:
We appreciate the reviewer's concern about the potential ambiguity of "perceptual quality" and the suggestion to clarify its definition in our manuscript. We agree that quantifying this concept is a complex and evolving field with diverse approaches. To ensure consistency with previous works, we follow the theoretical definition of perceptual quality as the divergence between probability density functions (PDFs), as proposed by [Blau & Michaeli 18, 19]. This approach aligns with relevant prior work and establishes a clear framework for our analysis. 
Following the reviewer's feedback, we have clarified the above in the Introduction explicitly stating our chosen definition of perceptual quality and its connection to PDF divergence. We have added a new paragraph on "Perception Quantification" discussing the various ongoing efforts and diverse approaches in this field.

Reviewer comment (Theory):
Please discuss if the proposed uncertainty-perception concept could be extended to rate-distortion? R(U,P) similarly to R(D,P) in [Blau & Michaeli 19].

Author response:
We strongly believe the uncertainty-perception tradeoff can be extended to incorporate rate, given the close connection between mutual information and entropy. While we do not have concrete theoretical results currently, we briefly mention this potential future direction in the revised manuscript, as suggested by the reviewer.

Reviewer comment (Evaluation):
The experiments depend on estimations of entropy and divergence between PDFs from samples of 243 dimensions. These estimations are risky and prone to high bias. How many samples did you use? Have you checked other estimators apart from Kozachenko-Leonenko? Ready-to-use alternatives include (1) an improved Kozachenko-Leonenko estimator [Marin-Franch & Foster IEEE PAMI 2013] available here https://github.com/imarinfr/klo, or (2) a Gaussianization-based algorithm which has proved to be better [Laparra et al. IEEE TNN 2011], see the comparisons in [Malo J. Math. Neurosci. 2020], or in [Laparra et al. 2023 https://arxiv.org/abs/2010.03807], available here https://isp.uv.es/RBIG4IT.htm https://github.com/IPL-UV/rbig. I suggest to repeat Fig. 4 and 5 with other estimators to have stronger evidences of the trend.

Author response:
We agree with the reviewer's concern about the challenges of entropy and divergence estimation in high dimensions. Following previous works by [Blau & Michaeli 18] and [Freirich et al. 21], we perform our estimations at the patch level, where we have more than 16 million samples. However, we acknowledge that challenges and potential limitations remain associated with high-dimensional estimations, which we explicitly state in the revised manuscript.  We highly appreciate the reviewer's suggestions and pointers to improved estimators, which we hope to include in the camera-ready version.

Reviewer comment (Presentation):
Please include a visual example for several points of Fig. 4. At least in the appendix if it doesn't fit in the main text. Remember that beyond conceptual definitions of "Perceptual quality", in this regard, nothing substitutes visual inspection of representative examples. In these examples please report the uncertainty (entropy of error) but also the MSE.

Author response:
We agree with the reviewer's comments and accept their suggestion to include a visual comparison of various estimators concerning uncertainty and perception in the main body of the paper.

Reviewer comment (Presentation):
Why in fig. 5 we have less points than in fig. 4?

Author response:
We have updated both figures to have the same number of data points.

Reviewer comment (Writing):
The third paragraph in page 2 has a repeated sentence "Conformal methods...".

Author response:
We have removed repeated sentences.

Reviewer comment (Writing):
Last sentence of the fourth paragraph of page 2 is confusing (particularly as "perceptual quality" as in Blau & Michaeli had not been defined). Currently it says "While the above studies address both uncertainty and perception, none of them explicitly quantify uncertainty as a function of perceptual quality"... Please clarify in which way the above studies talk about "perception" and how your work is different. Is it because this one uses the divergence definition? Probably it will be easier to clarify this if the suggested paragraph on "Perceptual Quality Quantification" is added before this comment.

Author response:
We have clarified the manuscript to explicitly state that the discussed studies do not analyze perceptual quality or its relation to uncertainty.

Reviewer comment (Presentation):
[Blau & Michaeli 18a] and [Blau & Michaeli 18b] are the same?

Author response:
We have fixed the duplicate citation error.