Paper: FourierAugment: Frequency-Based Image Encoding for Resource-Constrained Vision Tasks (ICLR-2024)

Reviewer comment (Novelty):
Separating the frequency bands of the network input is not novel per se: for example, it is used from time to time in generative modeling, see e.g. [1], [2]. However, usually, it is done with the help of wavelets. I recommend adding this baseline to the comparison, namely, decomposing the input image with two or three iterations of discrete wavelet transform before feeding the result to the network.

Author response:
As per your comment, we have added wavelet to the baseline and conducted a comparative experiment; we added 2- and 3-level decomposition. Table r3 below shows that applying wavelet decomposition to mini-ImageNet results in performance degradation.

| Accuracy (%)  | mini-ImageNet 100 | mini-ImageNet 500 |
|----------|---|---|
| Baseline | 35.60 | 61.52 |
| 2-level wavelet | 34.00 | 59.73 |
| 3-level Wavelet | 34.45 | 59.59 |
| FA | 37.35 | 65.23 |

Table r3. Comparison with Wavelet.

Reviewer comment (Experiments):
The authors compare their method with a number of other Fourier-based methods (Appendix C.2.3) and conclude that those methods do not perform well in their settings because they were specifically designed for medical image processing. It would be great to evaluate the proposed FourierAugment method on the medical data to demonstrate if it is competitive in that field since medical image processing often suffers from a limited amount of data.

Author response:
We experimented on the chest X-ray dataset [1] to demonstrate that FourierAugment also works for medical images. The dataset contains 97 images and is divided into 17 disease classes. We separate 63 of 97 data into train-set and 34 into test-set, and compare FourierAugment with AugMix, RandAugment, and DeepAutoAugment using ResNet18 as in previous experiments. In Table r1, RandAugment and DeepAutoAugment were ineffective on medical datasets, AugMix achieved slightly higher performance than baseline, but FourierAugment improved performance to 29.41%.

|          | Accuracy (%) |
|----------|--------------|
| Baseline | 20.59        |
| AM       | 23.53        |
| RA       | 17.65        |
| DAA      | 20.59        |
| FA       | **29.41**        |

Table r1. Top-1 accuracies of AugMix (AM), RandAugment (RA), DeepAutoAugment (DAA), and FourierAugment (FA) on the chest X-ray dataset.

[1] TrainingData.pro. (2023). Chest X-ray - 17 Diseases [Dataset]. https://www.kaggle.com/datasets/trainingdatapro/chest-xray-17-diseases

Reviewer comment (Writing):
I find the name of the method, i.e. *FourierAugment*, misleading since it is just a way to present exactly the same information which the image contains, in a different way. In addition, it is deterministic, not random. This contradicts the way the word *augmentation* is typically used by the community nowadays.

Author response:
We described in the paper that our method is close to transformation, but we judged that names like Fourier-transformation could confuse the reader. The name FourierAugment explains the content and direction of the application of our research well because it can be used like any other data augmentation.

Reviewer comment (Presentation):
Fig. 2 contains too much empty space and does not tell a lot of useful information. For example, the horizontal axis is not very informative since there is nothing unexpected in the training dynamics, and the whole figure may be replaced with a table reporting the final performance.

Author response:
As per your comment, we have tabulated the contents of Fig 2 as follows.

| Accuracy (%) | 500 | 100 | 50 | 25 | 10 | 5 |
|---|---|---|---|---|---|---|
| Original | 61.52 | 35.60 | 25.50 | 16.17 | 10.03 | 7.12 |
| Low | 45.68 | 26.28 | 19.98 | 13.55 | 9.85 | 7.07 |
|High | 40.01 | 23.85 | 16.82 | 10.92 | 8.06 | 4.46 |

Table r4. The horizontal axis is the number of data per class, and the vertical axis is the frequency component used.

Reviewer comment (Theory):
The paper could benefit from a more comprehensive theoretical background and motivation section, providing a clear foundation for why FourierAugment was developed and the specific theoretical underpinnings.

Author response:
1. The theoretical underpinnings

We describe in detail the motivation of our study in section 4-1. While several studies have demonstrated the significance of frequency information in vision tasks ([1], [2]), a lack of research exists pertaining to the scrutiny of lightweight models from a frequency standpoint. We hypothesized that an analytical approach focusing on frequency could ameliorate the constraints associated with lightweight models. It appears necessary to train HFC in accordance with the findings obtained from the analysis, and we propose a suitable method.

Reviewer comment (Theory):
The theoretical underpinnings and motivations for proposing FourierAugment, including the specific theoretical underpinnings that guided its development, would add to the coherence of the paper if they were added in full.

Author response:
1. The theoretical underpinnings

We describe in detail the motivation of our study in section 4-1. While several studies have demonstrated the significance of frequency information in vision tasks ([1], [2]), a lack of research exists pertaining to the scrutiny of lightweight models from a frequency standpoint. We hypothesized that an analytical approach focusing on frequency could ameliorate the constraints associated with lightweight models. It appears necessary to train HFC in accordance with the findings obtained from the analysis, and we propose a suitable method.

Reviewer comment (Experiments):
The n value is chosen empirically? Is there a definite basis for setting 2 or 3?

Author response:
2. n value

Excessive separation may destroy and lose essential features that rely on the whole image. We anticipated that using small n would improve the performance of FourierAugemnt and conducted empirical experiments on n. Table 12 displays the effect of the number of filters on FourierAugment. The result demonstrates that subdivision of the frequency bands beyond moderation degrades the performance of the model. This phenomenon is expected due to the inability to explore meaningful information with the excessive growth and granularity of the data.

| Number of filters | Last session accuracy (%) |
|----------|-----|
| CEC | 52.63 |
| + FA (n=2) | **53.77** |
| + FA (n=3) | 53.41 |
| + FA (n=4) | 52.92 |
| + FA (n=5) | 51.50 |
| + FA (n=6) | 53.23 |
| + FA (n=7) | 51.58 |

Table 12. Ablation study results: the effect of the number of filters on the proposed FourierAugment method (mini-ImageNet).

Reviewer comment (Experiments):
How does the method proposed in this paper compare to the comparative baseline AugMix, RandAugment and Deep AutoAugment in terms of computational efficiency and generalizability to different visual tasks?

Author response:
4. Computational complexity

Table 2r below shows the time complexity and memory usage of FourierAugment, AugMix, RandAugment, and DeepAutoAugment. The experiment used mini-ImageNet to measure the time and memory usage required to augment the entire dataset. Computationally, FourierAugment is faster than AugMix and DeepAutoAugment.

|          | Baseline |AM|RA|DAA|FA|
|----------|--------------|-----|-----|-----|-----|
| Computational time (s)| 9.56  | 13.36 | 11.66 | 12.54 | 11.85 |

Table 2r. Computational time of data augmentation methods.

Reviewer comment (Evaluation):
Does adding the image enhancement proposed in this paper to the image classification and FSCIL tasks significantly extend the processing time of the tasks?

Author response:
4. Computational complexity

Table 2r below shows the time complexity and memory usage of FourierAugment, AugMix, RandAugment, and DeepAutoAugment. The experiment used mini-ImageNet to measure the time and memory usage required to augment the entire dataset. Computationally, FourierAugment is faster than AugMix and DeepAutoAugment.

|          | Baseline |AM|RA|DAA|FA|
|----------|--------------|-----|-----|-----|-----|
| Computational time (s)| 9.56  | 13.36 | 11.66 | 12.54 | 11.85 |

Table 2r. Computational time of data augmentation methods.

Reviewer comment (Novelty):
Learning both low- and high-frequency information in balance is not novel. There have been many works that promote uniformly learning various frequency domains by randomly masking frequency segments, but the authors did not analyze or compare their work to these related works.

Author response:
Unlike the masking-based frequency segments ([1], [2], [3], [4]), our method uses the entire frequency and is designed for improving the performance of lightweight models‒targeting a different research goal. Furthermore, we conducted a comparative study with other frequency-based data augmentation techniques (Table 9) and confirmed that techniques developed for other tasks performed poorly in general classification.

Reviewer comment (Experiments):
I am curious if other frequency domain augmentation methods can also enhance performance, such as perturbing amplitudes[5], or randomly masking frequencies in the frequency domain[3].

Author response:
Unlike the masking-based frequency segments ([1], [2], [3], [4]), our method uses the entire frequency and is designed for improving the performance of lightweight models‒targeting a different research goal. Furthermore, we conducted a comparative study with other frequency-based data augmentation techniques (Table 9) and confirmed that techniques developed for other tasks performed poorly in general classification.

Reviewer comment (Experiments):
Can the authors provide evidence that the performance improvement is not solely due to the increased parameters, considering that the number of input channels has been changed to nx3? Could you discuss the computational complexity of your proposed method compared to other methods?

Author response:
Table 6 displays the number of parameters when different models adopt FourierAugment. FourierAugment increases the number of channels, but FouireAugment minimally changes the number of parameters. In particular, we deleted the first convolution layer of ResNet18, thus the model can learn with fewer parameters. Therefore, the performance improvement is not solely due to an increase in parameters.

| Number of parameters | Original | +FA |
|---|---|---|
| ResNet18 | 11,688,512 | 11,646,092 |
| EfficientNet-lite0 | 3,499,108 | 3,499,972 |
| MobileNet-V2 |  3,504,872 | 3,505,736 |
| ShuffleNet-V2 | 2,278,604 | 2,279,252 |

Table 6. Comparison of the number of lightweight model parameters  

3. Computational complexity

Table r2 below shows the time complexity and memory usage of FourierAugment, AugMix, RandAugment, and DeepAutoAugment. The experiment used mini-ImageNet to measure the time and memory usage required to augment the entire dataset. Computationally, FourierAugment is faster than AugMix and DeepAutoAugment.

|          | Baseline | AM | RA | DAA | FA |
|----------|---|---|---|---|---|
| Computational time (s) | 9.56  | 13.36 | 11.66 | 12.54 | 11.85 |

Table r2. Computational time of data augmentation methods.

Reviewer comment (Experiments):
Can the authors switch the backbone to demonstrate that there is indeed an increase in learning high-frequency features in the early stages, rather than it being a proprietary feature of ResNet?

Author response:
We employ EfficientNet-lite0 for additional experiments. Table 7 shows the comparative study results of EfficientNet-lite0. FourierAugment outperforms other data augmentation methods in the case of EfficientNet-lite0, a model suitable for resource-constrained environments. Similar to the comparative study results with ResNet18, the proposed FourierAugment displays superior performance over other data augmentation baseline methods in all experiment conditions.

Reviewer comment (Evaluation):
It would be helpful to understand if there are any limitations to your method. Are there specific scenarios or datasets where your approach may not perform as well as existing methods? If so, what are the reasons for this?

Author response:
Table 8 shows the results of unconstrained environments. FourierAugment enhances the performance on mini-ImageNet (small dataset) and results in no significant performance improvement on ImageNet (large dataset). We analyze that this is because ResNet50 does not learn sufficiently various frequency components from small datasets requiring FourierAugment for enhancement, but it can learn sufficiently various frequency components from large datasets—corroborating the effectiveness of FourierAugment in resource-constrained environments.

| Method | mini-ImageNet | ImageNet |
|---|---|---|
| Baseline | 71.66 | 76.30 |
| +RA | 71.81 | 77.85 |
|+FA | 80.80 | 75.61 |

Table 8. Top-1 accuracies of RA and FA on ResNet50