Paper: Wasserstein Distortion: Unifying fidelity and realism (ICLR-2024)

Reviewer comment (Novelty):
the Wasserstein distortion as introduced in section 2 is not really new and is in fact very close to the work of Freeman et al (2012) the main innovation being that the parameter sigma can freely be fixed at any position in the image instead of being constrained by the eccentricity of the visual receptive fields.

Author response:
Thank you for your review. We would respond to the weaknesses you mention as follows:
We assume the reference is to the paper:
J. Freeman and E. P. Simoncelli, “Metamers of the Ventral Stream,” Natural Neuroscience, vol. 14, pp. 1195-1201, 2011.


This paper is indeed the point of departure for our work, as noted in the introduction. Our paper extends Freeman and Simoncelli in several ways:

    Freeman and Simoncelli do not provide a distortion measure or loss function. They create perceptually indistinguishable images ("metamers") by constraining them to have the same local statistics across different patches. Thus the optimization problem they are solving has no objective; their goal is to find a feasible point, i.e., one that satisfies their constraints. Under their approach, whether two images are metamers is a binary notion; they do not quantify how far two images are from being metamers. Beyond being able to measure how close images are to being metamers (as in current Figure 12, Figure 10 in original submission), defining an explicit distortion measure enables the idea of measuring discrepancy between local statistics to be applied more broadly, such as in compression applications. It also allows one to precisely answer structural questions about the distortion measure, such as what properties of the pooling PMF we require, which brings us to the next point.

    Section 3 of the paper has been replaced with a discussion of what properties we require in order for Wasserstein distortion to be a proper metric (in particular, to satisfy that d(x,x’) = 0 iff x = x’). We find that both the raised-cosine-type PMF used by Freeman and Simoncelli and the uniform PMF used in most other works is problematic in this regard. We explain why and suggest PMFs that are superior for our purposes.

    Freeman and Simoncelli assume that the viewer focuses on the center of the image, and the pooling regions are the same for all images. We provide an automatic method for producing an image-specific sigma map from a given saliency map.

    We connect the pooling idea of Freeman and Simoncelli to the recent literature on the distortion-perception tradeoff. In that literature, distortion and perception (or realism) are viewed as disparate constraints that are fundamentally different in character. The main goal of this paper is to show that they can be lifted in a common framework, inspired by Freeman and Simoncelli. The paper thus has the potential to change the way both the machine learning and information theory communities view the distortion-perception tradeoff.

Reviewer comment (Writing):
the maths of section 2 are overly complicated for a naive reader : in the end the authors use discrete optimal transport between empirical distributions and assume Gaussiannity which in summary corresponds to adjust the mean and standard dev of the local features to a new image (initialized from a white noise image) to the mean and standard dev of the local features of an exemplar image.

Author response:
We believe that we made the math in Section 2 as simple as possible and that it is inline with other ICLR papers. The introduction provides a non-mathematical description of the ideas as a warm-up, and the paper is not especially notation-heavy by the standards of the community. We define Wasserstein distortion and prove our theorems in 1-D for simplicity’s sake. Note that we do not make a global Gaussian assumption. In particular, Theorems 3.1 and 3.2 in the original submission (which are moved to Appendix C now) make no reference to the Gaussian distribution. Our experiments resort to means and variances for computational reasons, but one cannot pose Wasserstein distortion in terms of means and variances without compromising the generality of the theorems.

Reviewer comment (Novelty):
the pooling distribution $q_\sigma$ corresponds to a local weighting of the statistics with width $\sigma$ (as in Freeman 2012).

Author response:
The sigma parameter does indeed correspond to the width of the pooling region in Freeman and Simoncelli, but we do not view this as a weakness of the paper.

Reviewer comment (Novelty):
when section 2 is well understood the theoretical results become trivial : (i) in the large $\sigma$ limit this is standard texture synthesis framework (Portilla-Simoncelli, Gatys). The specific setting of equation (12) has been empirically evaluated for texture synthesis against Gatys and Portilla-Simoncelli by Vacher et al (Neurips 2020). (ii) in the small $\sigma$ limit this is the exact reconstruction of an image from its feature.

Author response:
While we would not describe Theorems 3.1 and 3.2 in the original submission as trivial (they require proofs, and the proofs are more than a few lines), they are intended to confirm that Wasserstein distortion has the extremal properties that one intuitively expects. We have moved these theorems to the appendix to avoid giving the impression that these results are difficult or unexpected. We replaced them with a result and discussion on how the metric properties of Wasserstein distortion relates to the spectral properties of the pooling PMF, which we believe will be of greater interest.

Reviewer comment (Novelty):
the numerical experiments illustrate the nature of sigma but I do not see in which it contributes to vision study (even in terms of methods).

Author response:
We are not sure what “vision study” refers to. The goal of the experiments was to validate the proposed distortion measure using the method propounded by Ding et al. (2021), namely by generating random images that are close to a given image under the proposed measure. In particular, we sought to illustrate how it can be useful to interpolate between fidelity and realism constraints. The goal was not to contribute to the literature on the human visual system. We have replaced Experiment 1 with a different experiment that interpolates between fidelity and realism without any reference to peripheral vision, which hopefully helps clarify this point.

Reviewer comment (Writing):
the title is a bit misleading by talking about fidelity and realism, it is a bit disappointing that this corresponds to notion of pixel distance vs perceptual distance.

Author response:
We appreciate the prompt and detailed response. We understand the nature of the reviewer’s concerns now and are better able to address them.

**Regarding the objectives:** the title, abstract, and introduction place the work squarely in the recent literature on the distortion-perception tradeoff. This includes papers such as:
- Yochai Blau and Tomer Michaeli. The perception-distortion tradeoff. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, pp. 6228–6237, 2018. 					
- Yochai Blau and Tomer Michaeli. Rethinking lossy compression: The rate-distortion-perception tradeoff. In Proceedings of the 36th International Conference on Machine Learning (ICML), pp. 675–685, 2019. 
- Lucas Theis and Aaron B. Wagner. A coding theorem for the rate-distortion-perception function. In Neural Compression: From Information Theory to Applications – Workshop @ ICLR 2021, 2021. URL https://openreview.net/forum?id=BzUaLGtKecs
- Aaron B. Wagner. The rate-distortion-perception tradeoff: The role of common randomness. arXiv preprint arXiv:2202.04147, 2022. doi: 10.48550/arXiv.2202.04147.
- Jun Chen, Lei Yu, Jia Wang, Wuxian Shi, Yiqun Ge, and Wen Tong. On the rate-distortion-perception function. IEEE Journal on Selected Areas in Information Theory, 2022. doi: 10.1109/JSAIT.2022.3231820.
- George Zhang, Jingjing Qian, Jun Chen, and Ashish Khisti. Universal rate-distortion-perception representations for lossy compression. Advances in Neural Information Processing Systems, pp. 11517--11529, 2021.

The community interested in this topic lies in the intersection of image processing (especially image compression), information theory, and machine learning, and it is mathematically-oriented. Within the image processing and information theory communities in particular, the need for good distortion metrics has been recognized for so long as to be unquestioned, which is why we do not explicitly motivate it in the paper. The above papers trace their lineage to historical papers focused on distortion measures such as
- Zhou Wang, Alan Conrad Bovik, Hamid Rahim Sheikh, and Eero P. Simoncelli, Image quality assessment: From error visibility to structural similarity, IEEE Trans. Image Proc., Vol. 13, No. 4, pp. 600-612, 2004.

in image processing and
- James L. Mannos and David J. Sakrison, The effects of a visual fidelity criterion on the encoding of images, IEEE Trans. Inform. Theory, Vol. 4, pp. 525–536, 1974.

in information theory. The former is one of the most important papers in its field, with over 47K citations according to Google Scholar. The paper was informed by contemporary HVS studies, yet its contribution lies in proposing a distortion measure for engineering purposes. The measure is now used in a myriad of engineering applications and has won industry awards.

Likewise, the goal in our paper is to draw from, rather than contribute to, the HVS literature for purposes of developing a metric.  Contributions we make to the HVS literature would be serendipitous and incidental. We view our paper as belonging to the above literature on distortion-perception, and we hope it can be reviewed as such. We do not see it as a method/theory paper for vision studies as the reviewer suggests.

Reviewer comment (Experiments):
The paper is mostly a proof-of-concept at this stage. The author mentions using their method for image encoding, but this is not part of this work, and it is not clear that it would work well since, like recent super-resolution approaches based on single image patches approach, it could tend to fill in details with elements that would look realistic but are not real.

Author response:
Thank you for your careful and in-depth review. We especially appreciate your summary and description of the strengths of the paper. Regarding the weakness you mention, hallucination would appear to be intrinsic to any scheme that seeks a high level of realism without enforcing high pixel-level fidelity. Yet, image generation by minimization of Wasserstein distortion is less prone to hallucinating realistic elements that bear no resemblance to the original image because it relies entirely on local statistics of the image, without reference to the ensemble of all images. If the original image is, say, blurry, then we expect the reconstructed image to be blurry as well. In this sense the weakness you mention is less applicable to Wasserstein distortion than other methods of achieving realism, such as diffusion models. This was discussed in the last paragraph of the original submission. Also note that Wasserstein distortion, as a distortion measure derived from image statistics, is designed to quantify the extent to which a reproduced image contains realistic elements that are disconnected from the reference image. Rather than producing images that withstand close scrutiny in all regions, our image generation experiments are meant to demonstrate the properties of our metric, namely that it becomes increasingly permissive to error in the periphery, and that these errors are hard to spot when viewing the image at an appropriate distance.

Reviewer comment (Experiments):
Could the distortion measure be used to identify generated or edited content in an image, say to identify tampered images?

Author response:
Wasserstein distortion cannot detect generated or edited content because it is a full-reference distortion measure. Thus it can only measure the distance between two images. It cannot score the likelihood or realism of a single image in isolation.