Paper: PAC-FNO: Parallel-Structured All-Component Fourier Neural Operators for Recognizing Low-Quality Images (ICLR-2024)

Reviewer comment (Experiments):
Although the author compared super-resolution (SR) models for variable resolution inputs, the compared SR models are outdated and lack representation across various upscaling factors for super-resolution reconstruction. The field of super-resolution has seen significant advancements recently; thus, it is recommended to select more appropriate comparative algorithms.

Author response:
We provide an additional comparison to the method equipped with the latest super-resolution model. The results are in the following table. OSRT [1] is a state-of-the-art model in the super-resolution domain but it does not support $\times$8 upscale. Therefore, we only use the $\times$2 and $\times$4 upscale models of OSRT. In addition, we report OSRT (fine-tune) that combines super-resolution and fine-tuning methods. That is, we fine-tune the pre-trained classification model with low-resolution images upscaled by a super-resolution model.  As a result, PAC-FNO shows better performance than OSRT and OSRT (fine-tune) methods. We also report these results in Table 13 in Appendix F.3 of the revision. 

|  Imagenet-1k  |    Method   |        Metric       |    28    |    32    |    56    |    64    |    112   |    128   |  224 |
|:-------------:|:-----------:|:-------------------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:----:|
|              |     OSRT    |    Top-1 Acc (%)    |     -    |     -    |   61.4   |     -    |   75.4   |     -    | 82.5 |
|               |             | # of Parameters (M) |     -    |     -    |   11.93  |     -    |   11.93  |     -    |   -  |
|  ConvNeXt-Tiny  |     OSRT    |    Top-1 Acc (%)    |     -    |     -    |   71.2   |     -    |   78.4   |     -    | 82.1 |
|               | (Fine-tune) | # of Parameters (M) |     -    |     -    |   11.93  |     -    |   11.79  |     -    |   -  |
|               |   PAC-FNO   |    Top-1 Acc (%)    |  **58.9**   | **63.2** | **77.6** | **76.2** | **80.2** | **80.7** | 81.8 |
|              |             | # of Parameters (M) |          |          |          |   3.65   |          |          |      |

| Imagenet-C/P Fog |    Method   |        Metric       |    28    |    32    |    56    |    64    |    112   |    128   |    224   |
|:----------------:|:-----------:|:-------------------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|
|                 |     OSRT    |    Top-1 Acc (%)    |     -    |     -    |   19.4   |     -    |   37.9   |     -    |   58.4   |
|                  |             | # of Parameters (M) |     -    |     -    |   11.93  |     -    |   11.93  |     -    |     -    |
| ConvNeXt-Tiny  |     OSRT    |    Top-1 Acc (%)    |     -    |     -    |   42.3   |     -    |   56.4   |     -    |   59.4   |
|                  | (Fine-tune) | # of Parameters (M) |     -    |     -    |   11.93  |     -    |   11.79  |     -    |     -    |
|                  |   PAC-FNO   |    Top-1 Acc (%)    | **25.4** | **30.4** | **48.2** | **51.7** | **60.1** | **61.4** | **62.8** |
|         `        |             | # of Parameters (M) |          |          |          |   3.65   |          |          |          |

[1] Yu, Fanghua, et al. "OSRT: Omnidirectional image super-resolution with distortion-aware transformer." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Reviewer comment (Theory):
The primary advantage of Fourier Neural Operators (FNOs) lies in their use of frequency domain processing for resolution invariance. As a learnable enhancement operator, it's expected to exhibit some resilience to input natural variations. However, the author hasn't provided a detailed and explanatory analysis of the mechanisms where the operator shows robustness against natural variations.

Author response:
$\to$ As you mentioned, degradations change the high-frequency and low-frequency information in original images. Figure 8 in the Appendix F.8 shows that there are many changes in high-frequency when degradations are visualized in the frequency domain. Since existing FNO-based models that use a low-pass filter remove high-frequency, the degraded image loses not only the degradation factor but also the information of the original image. However, our proposed PAC-FNO uses high-frequency information, so it shows better performance in degradation.
Regarding the future work, we would like to address combined degradation types as the natural variations (e.g., mixed with motion blur and fog that may occur in the real-world), not try to address new types of variations.

Reviewer comment (Experiments):
The chosen input variations in the experiments, like fog, brightness, spatter, and saturate, represent basic degradation scenarios that can be addressed without deep learning methods. Therefore, regarding resilience to input natural variations, this might not be sufficiently emphasized as a highlight of the paper.

Author response:
$\to$ As you mentioned, degradations change the high-frequency and low-frequency information in original images. Figure 8 in the Appendix F.8 shows that there are many changes in high-frequency when degradations are visualized in the frequency domain. Since existing FNO-based models that use a low-pass filter remove high-frequency, the degraded image loses not only the degradation factor but also the information of the original image. However, our proposed PAC-FNO uses high-frequency information, so it shows better performance in degradation.
Regarding the future work, we would like to address combined degradation types as the natural variations (e.g., mixed with motion blur and fog that may occur in the real-world), not try to address new types of variations.

Reviewer comment (Experiments):
The paper suggests exploring degradation in real-world scenarios in future work, indicating that the authors are aware of the limitations in terms of experimental performance or the algorithm proposed. However, such scenarios represent fundamental problems studied in the field of Image Recognition (IR) and hold significant practical application implications.

Author response:
$\to$ As you mentioned, degradations change the high-frequency and low-frequency information in original images. Figure 8 in the Appendix F.8 shows that there are many changes in high-frequency when degradations are visualized in the frequency domain. Since existing FNO-based models that use a low-pass filter remove high-frequency, the degraded image loses not only the degradation factor but also the information of the original image. However, our proposed PAC-FNO uses high-frequency information, so it shows better performance in degradation.
Regarding the future work, we would like to address combined degradation types as the natural variations (e.g., mixed with motion blur and fog that may occur in the real-world), not try to address new types of variations.

Reviewer comment (Evaluation):
Certain degradation processes might affect high or low frequency details in the image's frequency domain. For instance, blur involves the loss of high-frequency details. This prompts the author to conduct a mechanistic analysis combining frequency domain and degradation processes to enhance this aspect's interpretability.

Author response:
$\to$ As you mentioned, degradations change the high-frequency and low-frequency information in original images. Figure 8 in the Appendix F.8 shows that there are many changes in high-frequency when degradations are visualized in the frequency domain. Since existing FNO-based models that use a low-pass filter remove high-frequency, the degraded image loses not only the degradation factor but also the information of the original image. However, our proposed PAC-FNO uses high-frequency information, so it shows better performance in degradation.
Regarding the future work, we would like to address combined degradation types as the natural variations (e.g., mixed with motion blur and fog that may occur in the real-world), not try to address new types of variations.

Reviewer comment (Evaluation):
The original intent behind the existence of ideal low-pass filters was to reduce the number of parameters and computational complexity. While the author's innovative design to remove the inherent low-pass filter is intuitively comprehensible, the associated trade-offs are not discussed in the manuscript. It would be beneficial to provide supplementary explanations to demonstrate the worthiness of such a modification.

Author response:
$\to$ We provide FLOPs and runtime on data at different resolutions. Existing FNO models used low-pass filters due to computational complexity, but the channel size was increased to fully utilize low-frequency components. However, PAC-FNO maintained the 3 channel size of the image to use both high-pass filter and low-pass filter. As a result, PAC-FNO showed similar levels of FLOPs and Runtimes as existing FNO models, but showed better performance because it used additional high-frequency components.

| ImageNet-1k |    Metrics   |    28   |   56   |   112   |  224  |
|:-----------:|:------------:|:-------:|:------:|:-------:|:-----:|
|    Resize   |    GFLOPs    |   8.96  |  8.96  |   8.96  |  8.96 |
|             | Runtimes (s) |  0.006  |  0.006 |  0.006  | 0.006 |
|  Fine-tune  |    GFLOPs    |   8.96  |  8.96  |   8.96  |  8.96 |
|             | Runtimes (s) |  0.006  |  0.006 |  0.006  | 0.006 |
|     DRLN    |    GFLOPs    |  180.66 | 412.05 | 1200.50 |  8.96 |
|             | Runtimes (s) |  0.378  |  0.498 |  0.913  | 0.007 |
|     DRPN    |    GFLOPs    | 1220.42 | 576.94 |  387.88 |  8.96 |
|             | Runtimes (s) |  0.1532 |  0.164 |  0.171  | 0.007 |
|     FNO     |    GFLOPs    |   9.78  |  9.78  |   9.78  |  9.78 |
|             | Runtimes (s) |  0.016  |  0.016 |  0.016  | 0.016 |
|     UNO     |    GFLOPs    |   9.10  |  9.10  |   9.10  |  9.10 |
|             | Runtimes (s) |  0.018  |  0.018 |  0.018  | 0.018 |
|     AFNO    |    GFLOPs    |   8.96  |  8.96  |   8.96  |  8.96 |
|             | Runtimes (s) |  0.010  |  0.010 |  0.010  | 0.010 |
|   PAC-FNO   |    GFLOPs    |   8.98  |  8.98  |   8.98  |  8.98 |
|             | Runtimes (s) |  0.013  |  0.013 |  0.013  | 0.013 |

Reviewer comment (Evaluation):
The experiments thoroughly prove the advantages of parallel architectures and claim that this approach encapsulates more frequency components. However, they lack further detailed explanations and justifications.

Author response:
$\to$ The parallel configuration of AC-FNO blocks captures more information than a serial structured model in the first layer, which is directly related to the data. We visualize what frequency the parallel configuration and serial configuration capture in Figure 7 in Appendix F.8 of revision. Figure 7 shows that the parallel configuration captures more high frequencies than the serial configuration. In other words, the parallel structured model captured both more low-frequency and high-frequency components than the serial structured model. As a result, parallel configuration of AC-FNO blocks show a better performance than serial configuration in Figure 4 in Section 4.3. We will add this explanation and justification in Appendix F.8 of the revision. Thank you!

Reviewer comment (Experiments):
Is there a more advanced choice for the SR baseline model used for comparison in your experimental setup? This will affect the fairness of the performance of your experiment?

Author response:
$\to$ For fairness of experiment in SR baseline, we will report additional baseline equipped with the latest super-resolution model in Table 13 in Appendix F.3 of the revision that will be updated shortly around November 20th AoE.

Reviewer comment (Evaluation):
It can be found that in ViT-B16, PAC-FNO shows not very good results at all low resolutions compared to other methods. What caused this phenomenon to occur? Is your method also unfriendly to other Transformer methods?

Author response:
$\to$ PAC-FNO generally performs well in low resolution but only worse than A-FNO. A-FNO is a model proposed as a token mixer for transformers, so it seems particularly friendly when combined with Transformer-based models.

Reviewer comment (Experiments):
The ideal low-pass filter in the FNO block removes detailed image signals that play an important role in classification in the fine-grained dataset. Is this conclusion applicable to Transformer based image classification methods? More quantitative results should be provided to confirm the universality of the proposed method.

Author response:
$\to$ Yes, high-frequency components play a more important role in image classification, especially fine-grained datasets. So, to see the effect of high frequency component removal for nuanced classification problem, we conduct  a fine-grained classification experiment with the ViT model. On the Oxford-IIIT Pets dataset, PAC-FNO, which captures the high-frequency components, performs better at all resolutions than FNO with an ideal low-pass filter.compared to FNO with an ideal low-pass filter. We add these results in Tables 16 and 17 in Appendix F.4 of the revision.

| Oxford-IIIT Pets |  28  |  32  |  56  |  64  |  112 |  128 |  224 |
|:----------------:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|
|        FNO       | 26.3 | 33.0 | 58.1 | 64.8 | 82.9 | 85.8 | 91.3 |
|      PAC-FNO     | 40.3 | 46.2 | 69.0 | 72.5 | 86.8 | 89.2 | 92.2 |

Reviewer comment (Experiments):
The abandonment of the low-pass filter is one of the main innovations in this work. Although the author provides an explanation for the motivation behind this operation, it is still recommended that the author conduct ablative experiments to analyze the impact of low-frequency/high-frequency information on accuracy/generalization.

Author response:
$\to$ Thank you for the suggestion! We provide an analysis of the impact of low and high-frequency information on accuracy/generalization through ablation experiments in Table 21 in Appendix F.7 of revision. Compared to PAC-FNO, accuracy and generalization decrease when using low-pass filter or high-pass filter. PAC-FNO with low pass filter show similar performance in ImageNet-1k compared with our PAC-FNO, but show a decrease in performance in terms of generalization in ImageNet-C/P Fog. On the other hand, when using a high-pass filter, it is expected to show good performance in ImageNet-C/P Fog, but it does not show good performance in terms of generalization because the performance is also poor in ImageNet-1k. Therefore, PAC-FNO, which uses both low-frequency and high-frequency components, only shows good performance in terms of accuracy/generalization. 

|           ImageNet-1k           |  32  |  64  |  128 |  224 |
|:-------------------------------:|:----:|:----:|:----:|:----:|
|  PAC-FNO (low pass filter) | 53.5 | 71.4 | 78.7 | 79.0 |
| PAC-FNO (high pass filter) | 21.6 | 49.4 | 68.2 | 74.8 |
|             PAC-FNO             | 58.9 | 74.5 | 80.2 | 81.5 |

|         ImageNet-C/P Fog        |  32  |  64  |  128 |  224 |
|:-------------------------------:|:----:|:----:|:----:|:----:|
|  PAC-FNO (low pass filter) | 18.0 | 41.7 | 52.4 | 54.4 |
| PAC-FNO (high pass filter) |  5.92  | 23.0 | 43.2 | 50.2 |
|             PAC-FNO             | 25.4 | 48.2 | 60.1 | 62.8 |

Reviewer comment (Theory):
As for parallel architecture, the relevant experimental results have indeed proven its effectiveness. However, the explanation of parallel architecture in the method section appears somewhat lacking. It is hoped that the author can provide further analysis of the mechanism that enables it to be effective.

Author response:
$\to$ The reason that it is effective is that the parallel configuration captures both low and high-frequency components while the serial architecture captures low-frequency components only (first paragraph in Sec. 3.2). Both low and high-frequency components must be captured to achieve good accuracy/generalization. This can be confirmed in Table 21 in Appendix F.7 in the revision. We visualize what frequency the parallel configuration and serial configuration capture in Figure 7 in Appendix F.8 of revision. Figure 7 shows that the parallel configuration more captures high frequencies than the serial configuration.

Reviewer comment (Experiments):
In terms of comparative experiments, the methods used by the author for comparison appear to be lacking in both quantity and novelty. The comprehensiveness of the complex scenarios considered by the author is commendable, but it is hoped that the author can still increase the comparison results with more advanced works to more effectively validate the superiority of the proposed method.

Author response:
$\to$ We believe that advanced work to effectively validate the superiority of our method is an advanced super-resolution method, as super-resolution models are relatively old models. Therefore, we provide additional comparison to the method equipped with the latest super-resolution model in Appendix F.4 of the revision that will be updated shortly around November 20th AoE. Furthermore, we report additional baselines that combine super-resolution and fine-tuning methods in rows 5-6 in Table 13 in Appendix F.4 of the revised paper. That is, we fine-tune the pre-trained classification model with low-resolution images upscaled by a super-resolution model. This method showed worse performance than our PAC-FNO. 
In addition, super-resolution methods are needed for each resolution. In other words, upscaling models of x8, x4, and x2 are needed to handle 28, 56, and 112 resolution, respectively. In contrast, our proposed PAC-FNO can handle images of all resolutions with an additional 3.65M network and shows good performance.
If the advanced works you intend are not super-resolution models, please let us know.

Reviewer comment (Evaluation):
The author mentions the advantages of this work in terms of efficiency, but it seems that no experimental analysis related to efficiency has been provided (such as FLOPs and runtime on data at different resolutions).

Author response:
$\to$ We did not mention the efficiency as the advantage of this work. Instead, we stated "efficacy of our neural operator-based mechanism" (not the “efficiency”) in the last paragraph of Section 4.4 is that neural operator-based mechanism is suitable for real-world applications because it can handle a variety of resolutions without the process of resizing to the target resolution. But for curiosity, we also compare the efficiency of our method to the other methods in the following table:

| ImageNet-1k |    Metrics   |    28   |   56   |   112   |  224  |
|:-----------:|:------------:|:-------:|:------:|:-------:|:-----:|
|    Resize   |    GFLOPs    |   8.96  |  8.96  |   8.96  |  8.96 |
|             | Runtimes (s) |  0.006  |  0.006 |  0.006  | 0.006 |
|  Fine-tune  |    GFLOPs    |   8.96  |  8.96  |   8.96  |  8.96 |
|             | Runtimes (s) |  0.006  |  0.006 |  0.006  | 0.006 |
|     DRLN    |    GFLOPs    |  180.66 | 412.05 | 1200.50 |  8.96 |
|             | Runtimes (s) |  0.378  |  0.498 |  0.913  | 0.007 |
|     DRPN    |    GFLOPs    | 1220.42 | 576.94 |  387.88 |  8.96 |
|             | Runtimes (s) |  0.1532 |  0.164 |  0.171  | 0.007 |
|     FNO     |    GFLOPs    |   9.78  |  9.78  |   9.78  |  9.78 |
|             | Runtimes (s) |  0.016  |  0.016 |  0.016  | 0.016 |
|     UNO     |    GFLOPs    |   9.10  |  9.10  |   9.10  |  9.10 |
|             | Runtimes (s) |  0.018  |  0.018 |  0.018  | 0.018 |
|     AFNO    |    GFLOPs    |   8.96  |  8.96  |   8.96  |  8.96 |
|             | Runtimes (s) |  0.010  |  0.010 |  0.010  | 0.010 |
|   PAC-FNO   |    GFLOPs    |   8.98  |  8.98  |   8.98  |  8.98 |
|             | Runtimes (s) |  0.013  |  0.013 |  0.013  | 0.013 |

As a result, PAC-FNO showed the most efficiency in terms of FLOPs and runtimes except AFNO.