Paper: Perceptual Scales Predicted by Fisher Information Metrics (ICLR-2024)

Reviewer comment (Reproducibility):
It is not clear how to generalize their approach to more complicated textures, where it is not obvious the features to describe them and how to compute Fisher information, J, from those.

Author response:
Parts of our results are generic in the sense our theoretical results hold as long as there is a generative model of your the image that is sufficient to compute the Fisher information (or estimate it numerically). We focus on the texture model that is capable of synthesizing a wide range of natural textures and that is often used in experimental vision studies (Gatys 2015, Wallis 2017, Vacher 2020, Ziemba 2021). The features we are testing are also the standard ones that are tested in vision studies.

Reviewer comment (Experiments):
In the more complicated scenario the authors only try a limited (non conclusive) set of features on some so-called naturalistic textures which are not very general either. All this reduces the strength of the experiments and the possible conclusions. Moreover, in some of the cases (e.g. pixel and wavelet features) they do not explain how to compute J.

Author response:
We acknowledge that we only run experiment with few textures. Though the goal of the current paper is to provide the technical results. We are already working in more general settings : generating image interpolations using deep generative models (eg VAEs) and evaluate the corresponding perceptual scale. So we expect to further extend the experimental work but this is beyond the scope of this paper. See the response to questions below for the calculation of Fisher information.

Reviewer comment (Theory):
Given the normalization of the response axes, the derived nonlinear functions cannot be used as a metric to compute differences between textures.

Author response:
The reviewer is correct, the measured scale is relative and cannot be used to measure distances between textures. However, this is not our goal here. Yet, we explain that we can measure the discrepancy between model predictions and empirical perceptual scales using the AMS score introduced in section 3.2. Such a score is a measure of how much a model is aligned to perception.

Reviewer comment (Experiments):
Please give examples interpolating between more general textures (e.g. a brick wall, and a flower field, or a pile of fruits -see examples of natural textures in Portilla & Simoncelli IJCV 2000-). Do you get similar results (theoretical predictions and human responses) in those cases?

Author response:
In the supplementary material of Vacher *et al.* 2020, you can see that indeed a wide range of textures can be synthesized. We can definitely run a few more experiments before the camera ready deadline if our work were to be accepted (before the review period ends is not possible because of administrative delays).

Reviewer comment (Writing):
The description of the method is confusing. For example, how do you get the nonlinear response in Fig. 1? I guess first you use the expressions in appendix A, then, you use \Psi in Eq. 6 and then you integrate, right? This is not clear in the text. Similarly, how do you get the predictions in Fig. 2? For the VGG response you assume they are Gaussian vectors, then you apply Proposition 4, and then you integrate Eq. 6?

Author response:
You are correct on everything. For Fig 1, it's Appendix A + Eq. 6 and for Fig 2, it's Proposition 4 + Eq. 6. We apologize if it's not clear we have clarified this in the Figures' caption and main text.

Reviewer comment (Theory):
The normalization of the response axes imply that all modifications of the stimuli lead to the same perceptual distortion. Simple visual inspection of the pairs in Fig. 2 shows that this is not correct. If the authors do not propose a way to give an absolute scaling in these different dimensions they should not claim that they are giving a metric of the image space. What they just provide is a nonlinear (up to a scalar) relation between displacements in certain directions and variations in the inner representation, but this is not a metric.

Author response:
Sorry for the confusion, we are not claiming that we are measuring a metric. In psychophysics, there is no absolute measure. Claiming to make an absolute measure of perception is highly controversial. That say, our point is that we are able to compare a theoretical "perceptual" scale as if a model was "perceiving" something to an empirical perceptual scale. We can compare both using the AMS score we introduce. So that we are comparing the displacement along a geodesic on the image manifold to how it is perceived. One could then claim to have matched the perception when the model prediction matches the empirical data.

Reviewer comment (Evaluation):
Following the above comment, the proposed method would be unable to give a measure to predict the Mean Opinion Score (MOS) on distortion in databases such as TID [Ponomarenko et al. 13] or KADID [Lin et al. 19]. If this is not the case, the authors should mention how to infer this MOS.

Author response:
Thank you for pointing out KADID and TID databases. We did not know these databases. Using our framework we could compute a score from the square root of the Fisher information (around a specific distortion). This would again require some assumption about the measurements, so we could obtain scores with pixels, images, wavelet or vgg19 and so on. The idea is that the scores correlate with the image quality judgment in that if we are more sensitive to a perturbation the image will appear as more degraded (lower score). This is very interesting, though it is a bit different from our approach. The derivative of the perceptual scale is a measure of sensitivity at every single point. This is what we are trying to predict here.

Reviewer comment (Presentation):
The title is too vague, it does not reflect the content of the paper, and actually overstresses "distances" and "metrics" not quite addressed in the work. What about changing the title by something like: "Nonlinear image representations in humans from Fisher information".

Author response:
Thanks for the alternative title suggestion. The previous title we had was more like your suggestion so we decided to follow you suggestion of changing the title. After discussion among the authors, we deciced to use : "Fisher Information Predicts the Perceptual Scale of Texture Interpolation Paths".

Reviewer comment (Writing):
In the abstract authors say "we demonstrate that it is related to the Fisher information of the generative model that underlies perception", while it should say "we demonstrate that it is related to the Fisher information of the generative model that underlies the stimuli".

Author response:
1. Both are correct, it depends on your perspective. We would like to find the right generative model of the stimuli, the one that matches the perception of an observer ie the one that have better chances to be the one used by an observer.

Reviewer comment (Writing):
In the first paragraph of page 6 the authors say "We will see that in both cases..." ... Where do the authors show this? (this is related to the confusing description of the method stated above).

Author response:
2. Right after Proposition 3.

Reviewer comment (Writing):
Typos:  
first paragraph of page 8 "VGG-19 to another (bottom-right of Fig. 2)" ---> "VGG-19 to another (bottom-left of Fig. 2)"  
second paragraph page 8 "frequency mode (Proposition 2 and Proposition 2)" ---> ?

Author response:
3. Corrected, thanks.

Reviewer comment (Theory):
The authors claim to have a convergence theorem, but what are the implications and practical relevance? Specifically, a Gaussian field is assumed, but high-dimensional natural images are highly non-Gaussian.

Author response:
The convergence theorem has been previously proven already. Here we explain how this result justifies the compatibility between different hypotheses made in Bayesian modeling of perception about an observer's measurement. See contribution (i) above. 
The result hold for stationary Gaussian images only. The question remains open for more complex images. Yet, we show that when this result does not hold it is necessary to make extra assumption about the observer's measurements and without the mathematical result, different assumptions could lead to different predictions  (Figure 6).

Reviewer comment (Experiments):
The function $\psi$ in Eq. (2) as the perceptual scale is tested in a very constrained scenario, without comparing to competing methods. For example, a recent computational model of the contrast sensitivity function considers spatio-temporal frequency, eccentricity, luminance, and area [C1].

Author response:
Thank you for pointing to stelaCSF model. We will cite it as an alternative line of work. The work is interesting, yet the goal is totally different from our. The stelaCSF model is a descriptive model (*i.e.* it does not provide an explanation for why the contrast functions are like they are) aiming at better detecting visual artifacts in computer vision applications such as Virtual and Augmented Reality. In contrast, our aim is to provide and test a normative model of perception (*i.e.* we assume that perception is driven by the inference of parameters from a generative model). To this date and up to our knowledge, we are not aware of any alternative (normative) models that provide predictions of the perceptual scales. Though, we are comparing different sub-assumption of our model about the nature of measurement $M$ (Figure 6b). In addition, the CSF is a measure of the limit of the perceptual system (between seen and not seen) while the perceptual scale is a measure of perceptual sensitive inside the window of visibility (Watson 1986)

Reviewer comment (Experiments):
The experimental results regarding texture interpolation are performed in a discriminative setting, not from a generative perspective (i.e., texture synthesis).

Author response:
To be precise, we are actually synthesizing interpolated textures using a generative model (Vacher *et al.* 2020) and the generated textures are used in the psychophysical experiment.

Reviewer comment (Writing):
The goals of the paper should be more precise.

Author response:
Hopefully, we have clarify our contributions above and we will update the section accordingly.

Reviewer comment (Experiments):
The comparison to previous methods should be performed, and performed in a comprehensive way.

Author response:
We are not aware of any models capable of predicting the perceptual scale that we could compare our model to. We compare different sub-assumptions of our model about the nature of measurement $M$.

Reviewer comment (Experiments):
The authors may want to test their models on natural photographic texture images, besides the synthetic and simplistic ones (as shown in Figs. 1 and 2).

Author response:
Ultimately the goal would be to test our model on natural photographic texture images. Yet, we have no generative model for true natural textures so we are using a state of the art model as an approximation. Note also that the images that you are judging as simplistic are not considered as such by the vision science community (e.g. Rust *et al.* 2005, Martinez-Garcia *et al.* 2019).

Reviewer comment (Theory):
The theoretical analysis is performed for the case of GRFs, which does not apply to naturalistic textures (note that Gaussian textures are a very limited class of images). Most of the Propositions are special cases of previous work, so what is the theoretical contribution of this work?

Author response:
> The theoretical analysis is performed for the case of GRFs, which does not apply to naturalistic textures (Note that Gaussian textures are a very limited class of images). 

We acknowledge that from the point of view of computer vision, Gaussian textures allow to synthesized a very limited class of images, mostly what is known as micro-textures (see Galerne 2011). Yet, in vision study most experiments are performed using simple and/or deterministic stimuli such as drifting grating, moving dots, moving bars, sparse Gabor functions... Moving toward naturalistic stimuli or even better natural images is still out of reach as we are not yet able to make sense of how to control the generation of those images. We have tool to do so thanks to deep generative models but they work as a black boxes which is not satisfactory for vision experiment purposes. 

> Most of the Propositions are special cases of previous work, so what is the theoretical contribution of this work?

We have made no new theoretical contributions to the fields of information theory (or more generally to mathematics) though we apply existing results to the theory of perception. In particular the consequence of Proposition 1 have never been explained previously. In the context of our paper, this shows that different hypothesis about the internal representation used by an observer leads to similar results though this is not always the case (*e.g.* see Vacher *et al.* 2018)

Reviewer comment (Experiments):
The different scaling experiments only involve a small set of pairs and 5 naive participants. There may be insufficient data for detailed analysis, and actually results were presented in Sec. 3 without any analysis.

Author response:
> The different scaling experiments only involve a small set of pairs and 5 naive participants. There may be insufficient data for detailed analysis, and actually results were presented in Sec. 3 without any analysis.

The number of participants is enough for this type of experiments (Devinck & Knoblauch 2012). We tested 15 pairs (3 in Fig.3, 10 in Fig. 4 and Fig. 5) which is a good sample size to start testing our model. The current work is not enough in term of experiment and we expect to test it further in a future work that will be more focused on experiments.

Reviewer comment (Writing):
The paper appears to be hastily written with many typos (e.g., page 5: e have -> we have; Fig. 2: the colors are wrong, and the caption's description conflicts with the main body, Page 8: Proposition 2 and Proposition 2; ...) and symbols are not always well-defined or explained.

Author response:
> The paper appears to be hastily written with many typos (e.g., page 5: e have -> we have; Fig. 2: the colors are wrong, and the caption's description conflicts with the main body, Page 8: Proposition 2 and Proposition 2; ...) and symbols are not always well-defined or explained.

We apologize for the typos and the wrong color in the captions. The second proposition 2 was refering to the one you can find in the appendix. This has been corrected, thank you ! Reviewer gWW2 has also highlighted that $s$ wasn't defined in Equation (1). This has also been corrected.

Reviewer comment (Writing):
Can the authors explain more on how measuring the perceptual scale helps estimate the perceptual geometry of images? This is claimed in the Abstract but rarely mentioned in the main body.

Author response:
2. Using texture interpolations (Vacher *et al.* 2020) we are controlling the change of the stimulus along a geodedic between two textures. Therefore, we are exploring a line on the manifold of natural textures (characterized by VGG feature statistics). The measure of perceptual scale reflects the sensitivity to this change. If we were to find a metric to interpolate between textures that leads to a linear perceptual scale, this would mean that we have found a representation of textures and a metric that is accounting for human perception (not the distance between textures but the sensitivity along the path joining those textures).