Paper: SEAL: A Framework for Systematic Evaluation of Real-World Super-Resolution (ICLR-2024)

Reviewer comment (Evaluation):
The definition of purity accuracy is fuzzy and will need better explanation.

Author response:
**1. The definition of purity accuracy is fuzzy and will need better explanation.**

Thank you for bringing this to our attention. We have revised the manuscript to incorporate detailed description of purity accuracy as highlighted in blue on page 17 of the Appendix. Purity [A] is an external criterion (similar to NMI, F measure) that assesses the alignment of the clustering outcome with the actual, known classes. It is defined as:
$$
\mbox{purity}(
\Omega,\mathbb{C}
) =
\frac{1}{N}
\sum_k \max_j
\vert\omega_k \cap
c_j\vert
$$
Where $N$ is the total number of data points, $\Omega = \{ \omega_1, \omega_2, \ldots, \omega_K \}$ is the set of clusters and $\mathbb{C} = \{ c_1,c_2,\ldots,c_J \}$ is the set of classes. To compute purity, each cluster is assigned to the class that appears most frequently within that cluster. The accuracy of this assignment is then evaluated by the number of correctly assigned points divided by the total number of points. 

[A] Schütze H, Manning C D, Raghavan P. Introduction to information retrieval[M]. Cambridge: Cambridge University Press, 2008.

Reviewer comment (Theory):
In Introduction, the authors argued that random selection may cause significant bias and randomness to the evaluation results. From my point of view, random selection will not cause bias because the posibility of choosing each degradation is identical. The authors are suggested to check this claim to avoid misunderstanding. Same issues exist in Section 5.3.

Author response:
Thank you for bringing this to our attention. We agree with the reviewer that there is no bias in the selection of degradation cases as they are chosen randomly. When we refer to the results as "biased", we mean to say that the evaluation results are skewed due to the unrepresentative nature of the randomly selected cases. These results only demonstrate the model's performance in these specific cases and do not provide a comprehensive view of its performance across the entire degradation space. To prevent any confusion or misinterpretation, we have modified the phrasing to "inconsistent and potentially misleading" and revised the manuscript accordingly.

Reviewer comment (Theory):
In Section 3.1, there may be a mistake in the calculation of the total combination of degradations. Since the order of degradation types can be switched, the total degradation combination for $s$ degradation type and $k$ degradation levels should be $A^{s}_{s} \times k^{s}$.

Author response:
Thank you for spotting the mistake! Assume there are only $s$ degradation types, and each type contains only $k$ discrete degradation levels. The correct calculation for the total degradation should be $A_s^s \times k^s$. With $s=10$ and $k=10$, the generated degradation space would have a magnitude of $(A_{10}^{10})*10^{10}$, which is $A_{10}^{10}$ times larger than the previous figure that did not consider the order. This number serves as a powerful illustration of the immense scale of the degradation space. We have revised the description on page 3 to reflect this correction.

Reviewer comment (Experiments):
In Section 3.2, the authors claimed that different combinations of degradation types may have similar visual quality and restoration defficulty. Consequently, the authors designed a clustering-based approach to handle this problem. However, this approach seems intuitive, and experiments should be conducted to validate whether the clustering works as expected, i.e., whether the examples within a cluster have similar visual quality and restoration defficulty whereas the examples in different clusters differ significantly. Only using Fig. 13 for demonstration is not enough.

Author response:
Thank you for the valuable suggestion. In Fig. 17 on page 20 of the Appendix, we’ve added the visual comparison of samples both between and within clusters. We randomly selected five clusters out of a total of 100 clusters and chose six samples from each cluster. We observed that samples within each cluster exhibit similar degradation patterns, indicating they share a comparable level of restoration difficulty. In contrast, samples belonging to different clusters display remarkably distinct degradation patterns. The results demonstrate that the clustering algorithm worked as expected.

Reviewer comment (Experiments):
The authors only used one image (i.e., lenna) to generate 100 degradation parameters. It can be observed in Fig. 9 that the image content can affect the final results (0.1-0.3 dB in PSNR). Consequently, have the authors considered using more images to generate more stable and representative clustering centers?

Author response:
We appreciate your careful reading and insightful comment. Based on your suggestion, we have incorporated a new experiment by utilizing the 5 reference images in Figure 9 for degradation clustering. To accomplish this, we computed the average of the similarity matrices induced by these images. We used the newly identified representative degradation cases to rank GAN-based methods and provided the results in Table 7 on page 13 of the Appendix. Notably, we have observed that this adjustment has led to a more conclusive ranking compared to the results obtained when using a single image (Table 2). In Table 2, we observed that SwinIR and MMRealSR had a very close difference of only 0.01 in the $AR$ metric. Consequently, their ranks needed to be determined using finer metrics such as $RPR_I$. However, in Table 7, we found that SwinIR (AR: 0.86) and MMRealSR (AR: 0.75) could be easily ranked based on the AR metric alone. This suggests that the degradation cases identified by combining the 5 images may indeed be more representative. It also emphasizes the potential benefits of applying enhanced clustering algorithms, which can further enhance the stability and representativeness of our SEAL framework.

Reviewer comment (Evaluation):
In Figure 1, the author compares the average performance of two randomly selected test sets for conventional evaluation. If evaluations were conducted on multiple randomized test sets, it could potentially provide a more comprehensive understanding of the model’s performance. It would allow for the observation of performance trends across different test sets and offer insights into the model’s consistency and reliability.

Author response:
We appreciate your valuable suggestion. We already provided the results on 20 randomized test sets in Figure 12 on page 16 of the Appendix. These results demonstrate a high level of randomness and significant variance. Across these 20 test sets, BSRNet and RealESRNet each outperform the other in approximately half of the cases, with their performance gap ranging from -0.22 to 0.18. It is difficult to definitively determine which method is superior. This observation further highlights the inadequacy of using randomized test sets for evaluation.

Reviewer comment (Evaluation):
Figure 3 presents a coarse-to-fine evaluation protocol. Initially, the authors utilize AR to validate the effectiveness of the real-SR model in comparison to the acceptance line. Subsequently, they employ coarse metrics to rank the real-SR model. However, actual evaluations may be contingent on user requirements. Could the sequence of fine-grained indicators possibly be adjusted to meet specific needs?

Author response:
We appreciate your insightful question. Similar to other components of our SEAL framework, the proposed coarse-to-fine evaluation metric can be adjusted according to the requirements and preferences of users. For instance, if enhancing the performance of acceptable degradation tasks is a priority, the user can choose to place $RPR_A$ before $RPR_I$.

Reviewer comment (Evaluation):
Why is it that the evaluation based on MSE-based real-SR employs multiple smaller models as reference lines, whereas the GAN-based real-SR utilizes a single model as reference lines?

Author response:
Thank you for the careful reading and thoughtful question. Our SEAL framework provides flexibility in aligning with specific requirements. Users can make different design choices when selecting the appropriate reference lines that best suit their needs. In the GAN-based setting, generating high-quality texture details with a small network can be challenging. To address this, we utilize RealESRGAN and RealHATGAN as reference models for acceptance and excellence, respectively. However, training these models individually on 100 tasks would be time-consuming due to their large size. As a result, we use their performance on the 100 tasks as reference points without further fine-tuning them on these tasks.

Reviewer comment (Reproducibility):
The authors use histograms as the basis for clustering degradations. However, the form of the histograms does not appear to be given in the paper (sections 3.2, 3.3). Are these histograms of r, g, and b separately? Is this a 2-D chromaticity histogram, a 3-D RGB histogram, or a set of histograms of filter bank outputs? It's not clear that all of these would be equally effective at capturing diverse degradations.

Author response:
Thank you for the question. The histograms of R, G, and B are generated separately and then concatenated into a single vector to represent the degraded image. We have included this information on page 16 of the new version and highlighted it in blue.

Reviewer comment (Evaluation):
The sole use of PSNR as a way to rank-order results may not be the best choice. PSNR rankings don't always reflect human rankings.

Author response:
Thank you for your insightful question. Our SEAL framework offers the flexibility to incorporate various IQA metrics, allowing users to customize the evaluation according to their specific requirements and objectives. In our experiments, we have explored and utilized multiple IQA metrics. Specifically, for MSE-based real-SR methods, we utilize the widely used PSNR metric. For GAN-based real-SR methods, we employ the perceptual metric LPIPS, as demonstrated in Table 2 on page 7. Additionally, we present evaluation results using the SSIM metric in Table 6 on page 13 of the Appendix.

Indeed, different metrics may yield different rankings. For instance, BSRNet ranks first in PSNR, while RealESRNet holds the first position in SSIM. This is because RealESRNet is trained with sharpened ground truth, which favors structural information. This variation in rankings highlights the utilization of different metrics in our evaluation framework can effectively capture and reflect the unique characteristics of real-SR methods.

Reviewer comment (Evaluation):
Why use PSNR as the basis for whether an image is Acceptable? Why not SSIM, perceptual distance, or some combination of the three? Do you get different rankings with different metrics under this evaluation system?

Author response:
Thank you for your insightful question. Our SEAL framework offers the flexibility to incorporate various IQA metrics, allowing users to customize the evaluation according to their specific requirements and objectives. In our experiments, we have explored and utilized multiple IQA metrics. Specifically, for MSE-based real-SR methods, we utilize the widely used PSNR metric. For GAN-based real-SR methods, we employ the perceptual metric LPIPS, as demonstrated in Table 2 on page 7. Additionally, we present evaluation results using the SSIM metric in Table 6 on page 13 of the Appendix.

Indeed, different metrics may yield different rankings. For instance, BSRNet ranks first in PSNR, while RealESRNet holds the first position in SSIM. This is because RealESRNet is trained with sharpened ground truth, which favors structural information. This variation in rankings highlights the utilization of different metrics in our evaluation framework can effectively capture and reflect the unique characteristics of real-SR methods.

Reviewer comment (Experiments):
The histograms used in the clustering are histograms of what?

Author response:
Thank you for the question. The histograms of R, G, and B are generated separately and then concatenated into a single vector to represent the degraded image. We have included this information on page 16 of the new version and highlighted it in blue.