Paper: The Unreasonable Effectiveness of Linear Prediction as a Perceptual Metric (ICLR-2024)

Reviewer comment (Experiments):
The experimental dataset consists only BAPPS.

Author response:
We are aware of the existence of other full-reference (i.e. pairwise) IQA datasets, however BAPPS is the only one we know of that contains a rich set of distortions. Other established datasets such as TID2013 or LIVE do not contain CNN-based distortions such as warping, and as such are prone to overfitting to (which we believe has already happened with traditional metrics such as MS-SSIM). A topic for future work would be to collect our own data to validate on, but this is unfortunately outside of the scope of this paper.

Reviewer comment (Experiments):
An ablation study is missing for the causal neighborhood and non-causal neighborhood, as semantic understanding relies on contextual relationships around pixels or regions.

Author response:
We thank the reviewer for this suggestion of what is effectively a new version of our LASI metric.
This exact ablation has been run and is shown below.
The performance of the new suggestion is slightly worse, however, the implementation is simplified as computing the non-causal mask consumes less memory.

[Please click for ablation plot](https://postimg.cc/V5ZPm8jk)

Reviewer comment (Novelty):
The sentence in the section 4.1 ‘’Our method relies on self-supervision (at inference-time) to learn a representation for each pixel that captures global perceptual semantics of the image. The underlying assumption is that, given a representation vector for some pixel, it must successfully predict the value of other pixels in the image in order to capture useful semantics of the image’s structure.” But the relationship of the perceptual embedding in FR-IQA and the semantic extraction is confused in this work. Because many previous methods also use semantic features extracted by pre-trained models on high-level classification tasks to calculate perceptual distance. Is there any difference between this semantic feature and the embedding derived by Eq 1 in this work?

Author response:
We are not sure if we understand your question correctly, but we do not think that we are confusing two different concepts in this work. Computing a perceptual embedding of an image in many prior works is done via a function $f(x; \theta)$, where $x$ is the image and $\theta$ are, for example, neural network parameters that are determined by a separate training stage.
In LASI, computing the weights is done via a function $f(x)$ that does not have trainable parameters, hence no training stage is necessary. Computing the embedding is performed via solving a linear least squares problem, and hence there is no analytic formula for $f$.
However, that does not matter, since $f$ is still a deterministic function of the image: every time we solve for $w$ (the weights of the linear least squares problem), we get the same answer.

Reviewer comment (Novelty):
Since the LASI is designed on the semantic extraction in the images’ structure, so the correlation of the prediction task and perceptual task is good, which is a little obvious.

Author response:
There is no evidence in the psychophysical literature that points towards the human visual system performing low-level reconstruction tasks.
It is not immediately obvious to us that good perceptual embeddings can be learned from linear prediction of pixels.

However, we agree that it is intuitive to expect the correlation to be better for LASI (prediction) than for LPIPS (classification).
The reason being that a useful feature for classification is not necessarily useful for visual tasks regarding the image’s structure.

Reviewer comment (Experiments):
Perhaps LASI can only measure perceptual distance at patch level, and will it be useful for high-resolution images?

Author response:
Distortions are usually applied to the image as a whole and therefore different patches are distorted in a similar way.
It is common in the FR-IQA literature to scale up to larger images by computing the metric on smaller patches for this reason.
LASI can be run in the same way by parallelizing over patches and averaging the distance, a common technique in the literature.
We expect this to perform similarly, the same way other methods do, and will include an experiment on LIVE IQA [1] in the follow-up.

[1] Sheikh H. LIVE image quality assessment database release 2. http://live.ece.utexas.edu/research/quality. 2005.

Reviewer comment (Experiments):
An ablation study is missing for the causal neighborhood and non-causal neighborhood, as semantic understanding relies on contextual relationships around pixels or regions.

Author response:
We thank the reviewer for this suggestion of what is effectively a new version of our LASI metric.
This exact ablation has been run and is shown below.
The performance of the new suggestion is slightly worse, however, the implementation is simplified as computing the non-causal mask consumes less memory.

[Please click for ablation plot](https://postimg.cc/V5ZPm8jk)

Reviewer comment (Evaluation):
The paper claims that the advantage of the proposed method is that it requires no training data or DNN features. But, the method requires computations at inference time. Considering that once training is done, DNN-feature based methods do not require much of extra computations (hence the cost is amortized) while the proposed method requires extra computations, is requiring training data or DNN features really a bad thing?

Author response:
Even with amortization, computing DNN features requires more computation at inference time than LASI (ours).
DNN features require a forward-pass in the neural network to compute embeddings.
The plot below shows the wall-time for computing PIM (a DNN) and LASI (ours) distances, compiled with `tf.function` and `jax.jit`, respectively.
Note how LASI is faster for the neighborhood sizes considered in the experiments (i.e., $10$ to $12$).

[Please click for LASI and PIM comparison plot](https://postimg.cc/ygnnh7Vg)

Reviewer comment (Experiments):
All experiments are performed with 64x64 resolution, which seem rather small. Is the proposed method effective for larger images, compared to other works? Is the choice of $N$ the size of neighborhood robust against the image size? It seems $N$ needs to be tuned for each image resolution, which can be critical, since images can come in at various sizes.

Author response:
Distortions are usually applied to the image as a whole and therefore different patches are distorted in a similar way.
It is common in the FR-IQA literature to scale up to larger images by computing the metric on smaller patches for this reason.
LASI can be run in the same way by parallelizing over patches and averaging the distance, a common technique in the literature.
We expect this to perform similarly, the same way other methods do, and will include an experiment on LIVE IQA [1] in the follow-up.

[1] Sheikh H. LIVE image quality assessment database release 2. http://live.ece.utexas.edu/research/quality. 2005.

$N$ does not need to be tuned for each image resolution.
Our experiments show increasing $N$ gives better downstream performance (see Figure 3).
$N$ should be set to as large as possible while respecting the compute and memory budget of the application.

Reviewer comment (Experiments):
The paper claims that the method can be combined with LPIPS but I cannot find the experimental results on this. Does the combination actually bring improvements?

Author response:
The MAD experiment highlights that LPIPS and LASI fail in different ways.
We experimented with merging these models by taking geometric averages of their distances.
This resulted in a slightly better perceptual performance but not enough to justify merging, as the compute required is now that of LPIPS + LASI.

Merging these models is a non-trivial manner as the embedding dimensions differ in size making it difficult to perform simple aggregation techniques at the embedding level. Note this is true in general for any pair of models.

We will update the writing of the paper to make this clear in the main text.

Reviewer comment (Evaluation):
There is minimal discussion about the failure cases using the proposed method. Would be great to have some qualitative results and the probable reason we are seeing the results as we see it.

Author response:
The difference in performance between LASI (ours) and PIM is insignificant (less than $2.3\\%$ averaged across categories) leaving very little signal to perform a detailed analysis, which is the main point of the paper.

A qualitative analysis on the examples of the dataset yields no immediately obvious pattern that would explain this small difference in performance.

The comparison in Table 1 favors alternative metrics by design.
LASI (ours) is held fixed while the best performing models for other metrics are chosen for each category. 
For example, for LPIPS we picked the best performing embedding between VGG, SqueezeNet, and AlexNet, for each category, while for PIM we chose between PIM-1 and PIM-5 (whichever scored higher).
This cherry picking explains the gap: if we tune LASI (ours) by varying $\omega$, then the gap can be removed.

This is exactly the main message of the paper: we can achieve perceptual performance virtually indistinguishable from that of neural network methods while using no training data or deep features, even when neural network methods are allowed to be fine-tuned for each category.

Reviewer comment (Evaluation):
In the results presented in Table:1, why does MS-SSIM outdo the author's proposed method for the BAPPS-JND task.. whereas it outperforms it for the BAPPS-2AFC task?

Author response:
MS-SSIM does not outdo our method on the JND task.
This is only true in one category of the JND task and by less than $1\\%$, while in the other category our method is superior by $54.4\\%$.

In all other categories, for both 2AFC and JND, LASI (ours) outperforms MS-SSIM.

This is highlighted in Table 1, in the last row labeled "Improv. over MS-SSIM", where only the last column is negative ($-0.9\\%$) and all others are positive.

Reviewer comment (Evaluation):
As mentioned in the weakness section, it is imperative that authors present more details qualitative and quantitative about the failure cases seen using the LASI method proposed in the paper.

Author response:
The difference in performance between LASI (ours) and PIM is insignificant (less than $2.3\\%$ averaged across categories) leaving very little signal to perform a detailed analysis, which is the main point of the paper.

A qualitative analysis on the examples of the dataset yields no immediately obvious pattern that would explain this small difference in performance.

The comparison in Table 1 favors alternative metrics by design.
LASI (ours) is held fixed while the best performing models for other metrics are chosen for each category. 
For example, for LPIPS we picked the best performing embedding between VGG, SqueezeNet, and AlexNet, for each category, while for PIM we chose between PIM-1 and PIM-5 (whichever scored higher).
This cherry picking explains the gap: if we tune LASI (ours) by varying $\omega$, then the gap can be removed.

This is exactly the main message of the paper: we can achieve perceptual performance virtually indistinguishable from that of neural network methods while using no training data or deep features, even when neural network methods are allowed to be fine-tuned for each category.

Reviewer comment (Evaluation):
In Section 5.3: "....Results indicate LASI can find failure points for LPIPS and vice-versa......" Can authors elaborate on this point ?

Author response:
Images in the box labeled “LPIPS loss fixed” have the same LPIPS distance to the reference.
This means LPIPS says all images have equal perceptual quality relative to the reference.
However, clearly the bottom row has better perceptual quality than the top row.
This is known as a “failure mode” in the MAD literature.

Conceptually, this is analogous to finding adversarial examples for a classifier – all known perceptual metrics have failure modes. In conducting this experiment, we were hoping to find qualitative explanations for the unexpected performance of LASI, but the results have so far been inconclusive.