Paper: Frequency-Aware Transformer for Learned  Image Compression (ICLR-2024)

Reviewer comment (Experiments):
It is interesting to find that FDWA achieves a significant improvement compared to 4x4 blocks and 16x16 blocks. However, it is not clear that how does the multiscale decomposition and the directional decomposition affect. The authors are suggested to provide an ablation study on FDWA.

Author response:
**W1: Ablation Study on FDWA:**

* Thanks for your valuable suggestion. We have performed an ablation study on our FDWA to demonstrate the effect of both multiscale and directional decomposition. Due to the limitations on time and computational resources, we only provide models trained for 500,000 training steps in this response. The Lagrangian multiplier $\lambda$ is set to 0.0483 for all the models.

  In the following table, we use **Multiscale** to denote anisotropic $4\times 4$ and $16\times 16$ windows, and we employ **Directional** to represent isotropic $4\times 16$ and $16\times 4$ windows. Besides, we provide a baseline result with only $4\times 4$ windows for reference. The PSNR, BPP, and R-D loss are averaged across 24 images of the Kodak dataset. Results show that both multiscale and directional decomposition enhance the R-D performance.


* | Models            | Multiscale | Directional | PSNR   | BPP    | R-D Loss |
|-------------------|:------------:|:-------------:|--------|--------|----------|
| Baseline          | x    | x    | 37.031 | 0.8949 | 1.552    |
| Multiscale-only   | ✔️         | x  | 37.025 | 0.8722 | 1.537    |
| Directional-only  | x   | ✔️         | 37.080 | 0.8889 | 1.540    |
| Proposed          | ✔️         | ✔️          | 37.248 | 0.8910 | 1.517    |

Reviewer comment (Experiments):
The frequency-aware transformer is realized by two modules, FDWA and FMFFN. Have you tried these two mechanisms in the entropy model? Entropy model plays a crucial role in learned image compression. In my opinion, it is more important to capture frequency component in entropy model. It would be a great improvement if the frequency-aware mechanism works.

Author response:
**W2: Capturing Frequency Components in Entropy Model:**

* Thanks for your suggestion. We agree that capturing frequency components can further benefit entropy models in LIC, and **we specifically propose a TC-A entropy model to exploit the correlations between the frequency components and reduce the coding budget**, as elaborated in Section~3.3.

  Since the latent representation extracted by the proposed FAT block consists of diverse frequency components across different channels, we employ channel attention in our TC-A entropy model to model the dependencies between these frequency components. Moreover, the channel-wise correlations (*i.e.*, frequencies correlations) captured by our TC-A can adaptively vary with input image, which cannot be achieved by the commonly used CNN-based channel-wise auto-regressive entropy model (*i.e.*, CHARM [R1]).

  **Apply FDWA and FMFFN in Entropy Model:**

* Thanks for your valuable comments. Theoretically, FDWA and FMFFN can be used in the entropy model. Here, we provide additional experimental results based on a recent work [R2]. In this experiment, we adopt the entropy model from [R2] as our baseline and conduct an ablation study to demonstrate the additional benefits of FDWA and FMFFN. The experimental setup follows the ablation study on the TC-A entropy model in Section 4.3 of our paper.

  Despite the Swin-Transformer being only a modest sub-module in the baseline entropy model [R2], our results demonstrate that FDWA and FMFFN can also improve its performance, showing the potential of FDWA and FMFFN in helping achieve more accurate probability distribution estimation.
  | FDWA         | FMFFN        | BD-rate (Anchor: BPG) |
  |:--------------:|:--------------:|:------------------------:|
  |      $\times$ |      $\times$ |         -17.4\%         |
  |✔️          |      $\times$ |         -18.1\%         |
  |      $\times$ | ✔️          |         -17.6\%         |
  |✔️         | ✔️         |         -18.2\%         |


  We do not directly apply FDWA and FMFFN in the proposed entropy model (*i.e.*, TC-A) since both these two modules are designed to enhance the Swin-Transformer based on **spatial self-attention**, while our **channel self-attention**-based TC-A is not constructed with the Swin-Transformer.

[R1] Minnen, David, and Saurabh Singh. "Channel-wise autoregressive entropy models for learned image compression." 2020 IEEE International Conference on Image Processing (ICIP). IEEE, 2020.

[R2] Liu, Jinming, Heming Sun, and Jiro Katto. "Learned image compression with mixed transformer-cnn architectures." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Reviewer comment (Experiments):
Have you ever tried other decomposition ways such as (1) smaller size and bigger size for square window (I am wondering how much improvement can be obtained in this mechanism), or (2) more diverse shapes.

Author response:
**Q1: Other decomposition ways**

We have tried the decomposition ways mentioned by the reviewer in our earlier experiments, but we did not adopt these configurations in our manuscript due to the out-of-memory (OOM) problem raised by using the bigger size for a square window ($32 \times 32$). Here, we also conduct experiments to comprehensively understand how different decomposition ways affect the performance.

**(1) OOM problem of 32 $\times$ 32 window**: We emphasize that employing a bigger square window ($32 \times 32$) results in significantly increased memory consumption and running time. For example, when we set the window size of **half of the attention heads** as $32\times 32$ and the remaining ones as $2 \times 2$, we meet the OOM problem on our Nvidia RTX 4090Ti GPU (with 24 GB memory) during the training process with each batch containing 8 images with a patch size of 256$\times$ 256.

**(2) Effect of different decomposition ways**: We explored various decomposition ways in this ablation study, including:

- **(a) With Only One Small Window** (*i.e.*, $4\times 4$).
- **(b) With Multiscale Decomposition**, including two types of windows （*i.e.*,$4\times4$ and $16\times16)$.
- **(c) With Smaller Size and Bigger Size for Square Window (Reviewer's Suggestion)**, including four types of windows (*i.e.*,$2\times2, 4\times4, 16\times16,$ and $ 32\times32)$.
- **(d) With Our FDWA**, including four types of window (*i.e.*,$4\times4, 16\times16, 4\times16, 16\times4$).
- **(e) With More Diverse Shapes (Reviewer's Suggestion)**, including 8 types of window (*i.e.*,$2\times2, 4\times4, 16\times16,  32\times32, 2\times32, 32\times2, 4\times16$, and $16\times4$).

In each of the above methods, the number of attention heads for each type of window attention is identical (*i.e.*, $K/N$ heads, where $K$ denotes the total number of attention heads, and $N$ denotes the number of window types). All the models trained for 500,000 training steps, and the Lagrangian multiplier $\lambda$ for all the models is 0.0483. We observe that the decomposition method employing more diverse shapes achieves slightly lower R-D loss, with a sacrifice in memory consumption and training speed.


| **Methods** | **PSNR** | **BPP** | **R-D Loss** | **GPU Memory for Training** | **Training Speed (steps/s)** |
|:------------:|:--------:|:-------:|:------------:|:---------------------------:|:-----------------------------:|
|     (a)      |  37.031  |  0.8949 |    1.552     |           10.36GB           |             3.33              |
|     (b)      |  37.025  |  0.8722 |    1.537     |           13.96GB           |             3.08              |
|     (c)      |  37.135  |  0.8802 |    1.525     |           20.22GB           |             2.16              |
|     (d)      |  37.248  |  0.8910 |    1.517     |           12.68GB           |             3.17              |
|     (e)      |  37.213  |  0.8817 |    1.513     |           15.02GB           |             2.57              |

Reviewer comment (Experiments):
What impact does the block size in FMFFN have? When you set the block size responding to the maximum window size in FDWA, does it mean that the FMFFN is used to refine the high-frequence of features?

Author response:
**Q2: Effect of the Block Size in FMFFN**

The block size in FMFFN influences the balance between modulating local and global features in FMFFN. A block size that is too small in FFT may result in the loss of some global features ( *i.e.,* low-frequency information), while a block size that is too large may impede high-frequency local structure and increase the parameters of the learnable filter matrix.

We want to clarify that FMFFN is not used to refine the high-frequency features only. Setting the block size corresponding to the maximum window size ( *i.e.,* 16$\times$ 16) in FDWA indicates that FMFFN operates within a similar frequency range as FDWA, so that we can comprehensively modulate both low and high-frequency components produced by FDWA. In addition, FFT with 16$\times$ 16 block size also resembles the 16$\times$ 16 block-wise DCT transforms in H.264. Considering these factors, we chose the 16$\times$ 16 block size for FMFFN in our manuscript.

We also conducted ablation studies on the block size of FMFFN, and the results are in the table below. All the models are trained for 500,000 training steps, and the Lagrangian multiplier $\lambda$ for all the models is 0.0483. It indicates that FMFFN with a block size of 8 achieves the best R-D performance in this training set up.

|   Block Size   |   PSNR   |   BPP   |   R-D Loss   |   # Params   |
|:--------------:|:--------:|:-------:|:-----------------------------:|:------------:|
|       4        |  37.117  | 0.8812  |            1.524              |   70.44M     |
|       8        |  37.211  | 0.8789  |            1.511              |   70.55M     |
|      16        |  37.248  | 0.8910  |            1.517              |   70.97M     |
|      32        |  37.092  | 0.8793  |            1.529              |   72.55M     |

Reviewer comment (Experiments):
If the decomposition of window size works, how about directly using different sizes of convolution?

Author response:
**Q3: Convolutions with Diverse Kernel Size.**
Several works have discussed the impact of convolution kernel sizes in learned image compression. Cheng *et al.* [R3] compare the learned image compression model with diverse convolutional kernel sizes (*i.e.*, $5\times 5$, $7\times 7$, and $9\times 9$) and observe that a larger convolutional kernel size would lead to better R-D performance. Moreover, a larger kernel size could be obtained by stacking small convolutional kernels. Cui *et al.* [R4] employ mask convolutions with diverse kernel sizes { *i.e.*, $3\times 3$, $5\times 5$, and $7\times 7$) to achieve a more precise autoregression model.

However, we have not found discussions on anisotropic kernel size or analyses on the effect of kernel size from the perspective of frequency decomposition for convolution-based LIC models. Some works [R5, R6, R7] show that convolutions with a local receptive size tend to present the characteristics of a high-frequency filter. While there is potential to achieve frequency decomposition with diverse kernel sizes of convolutions, achieving much larger kernel sizes may pose challenges in computation complexity or require stacking too many small kernel convolution layers.

We highlight that our FDWA can achieve multiscale and directional frequency decomposition **in each attention layer in a more effective and simple way.**

- [R3] Z. Cheng, H. Sun, M. Takeuchi, and J. Katto, "Deep Residual Learning for Image Compression," CVPR Workshops. 2019. 
 
- [R4] Z. Cui, J. Wang, S. Gao, T. Guo, Y. Feng, and B. Bai, "Asymmetric Gained Deep Image Compression With Continuous Rate Adaptation," 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 2021, pp. 10527-10536.

- [R5] Park, Namuk, and Songkuk Kim. "How Do Vision Transformers Work?." International Conference on Learning Representations. 2022.

- [R6] Yin, Dong, et al. "A fourier perspective on model robustness in computer vision." Advances in Neural Information Processing Systems 32 (2019).

- [R7] Wang, Haohan, et al. "High-frequency component helps explain the generalization of convolutional neural networks." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

We sincerely appreciate your dedicated review of our paper. The author response period for this submission has been extended until December 1st, and we are looking forward to your further response.

Reviewer comment (Novelty):
The idea of applying frequency processing to the learned compression framework is not new. For example, conv in the frequency domain [1] has been used in Balle’s early work [2]. Wavelet-based compression framework has also been proposed [3]. The authors should cite, compare, and identify their differences.

Author response:
We appreciate your insightful comments and valuable suggestions. We address your concerns as below.

**W1: Novelty of this paper.**

- First of all, we would like to clarify that **the novelty of this paper is the novel method to exploit multi-scale and directional frequency information in nonlinear transforms rather than employing frequency-based operations.** We HAVE NOT claimed that our work is the first to apply frequency-based operations in LIC. On the contrary, we break the limitation of existing transformer-based LIC models that cannot achieve multi-scale and directional analysis due to the isotropic window attention.

- Subsequently, we have discussed the three papers [R1, R2, R3] mentioned by the reviewer and clarified the difference between this paper and [R1, R2, R3]. In summary, [R1] and [R2] **aim to accelerate network optimization** and directly learn the parameters of CNN in the spectral domain. [R3] substitutes the filters in the traditional separable wavelet transforms with trained CNNs to improve the R-D performance of  JPEG2000. **These methods cannot achieve multi-scale directional analysis by leveraging isotropic DFT [R1] and DCT [R2] bases or introducing CNNs into 1-D Wavelet transforms [R3].** By contrast, in this paper, we propose to capture diverse frequency components with the frequency-aware transformer and achieve an end-to-end optimized multi-scale and directional analysis by anisotropic window attention. We elaborate our differences from [R1, R2, R3] as below.

  * **Difference from [R1] and [R2].** [R1] and [R2] parameterize CNNs in the frequency domain for fast convergence since spatial domain convolutions are equal to spectral domain multiplications. DCT [R1] and DFT [R2] are adopted as the transforms for the linear filters in CNNs. However, such parameterization only affects convergence speed in optimizing the parameters but does not change the network architectures.  It should be noted that this idea is also considered for speeding up various tasks such as  image classification [R1], inverting deep networks [R6], and mobile sensing [R7], rather than specifically designed for extracting frequency components in LIC.

    In contrast, our approach captures and processes diverse frequency components to achieve more  compact image representations for improved R-D performance.  **Different from [R1] and [R2], we design FMFFN specifically for LIC to to adaptively decide the needed frequency components** and further eliminate redundancy across different components, though FFT is used to transform the features into frequency domain.**

  * **Difference from [R3].** iWave [R3] is developed based on the JPEG2000 schemes by substituting the discrete wavelet transforms (*e.g.*, Daubechies 5/3 and 9/7 wavelets) with CNNs. iWave [R3] cannot be end-to-end optimized by fixing the entropy model (*i.e.*, the probability distribution of discretized symbols) for entropy coding and yields limited R-D performance gain over traditional JPEG2000.  iWave++ [R8] is later developed to enable end-to-end learning for LIC. However,  iWave [R3] and iWave++ [R8] realize  nonlinear transform by stacking1-D transforms on horizontal and vertical directions, inspired by JPEG2000 that leverages 2-D separable wavelet transforms realized independently on the horizontal and vertical directions. As a result, they cannot realize multi-dimensional directional transform and is restricted in achieving compact representation for image compression 
  * **Different from [R3] and [R8] that independently perform 1-D transforms, our method achieves directional transforms that process the horizontal and vertical directions simultaneously.**  We are the first to leverage the transformer to exploit the different frequency components, which has not been achieved in existing LIC model. Furthermore, we  propose the T-CA entropy model to model the correlations across different frequency components for end-to-end optimization.

We have cited these works as. Ball'e et al. (2016), Rippel et al. (2015), and Ma et al. (2019; 2020) and discussed them in the related work section in our revision.

Reviewer comment (Experiments):
The authors should compare with more existing works (e.g. [3,4]) to demonstrate the SOTA performance of the paper.

Author response:
**W2: SOTA performance.**
* We exactly achieve the SOTA performance and outperform [R4] and [R5] mentioned by the reviewer. [R4] is the current SOTA method for LIC and has been compared in the previously submitted manuscript. We show that, compared with [R4], we yield 2.6% BD-rate saving on the Kodak dataset. Moreover, [R5] is inferior to [R4] in R-D performance. Compared with VVC, [R5] yields 2.1% BD-rate reduction on the Kodak dataset, while **we can achieve a significantly higher 14.5% BD-rate reduction**. We have included the results for [R5] in Fig.~4 in the revised manuscript.

Reviewer comment (Reproducibility):
Table 1 and Table 5 in the article involve flop calculations, may I ask the authors what methods or tools they used to calculate the complexity? As far as I know, many existing tools that count complexity do not calculate correctly the complexity of the internal operators of the transformer. Besides, Table 5 is kind of confusing. Can the authors explain what the (1), (2), (3) in the table mean here.

Author response:
**Q1: We address the concerns on Tables 1 and 5 as below.**

**Table 5 for calculating the complexity:**
Thanks for your kind remainder. We noticed that *deepspeed* used in the previous transformer-based LIC work [R9] ignores the matrix multiplication and softmax operation of self-attention. Thus, in our evaluation, we used  the *FlopCountAnalys* function from *fvcore* library, which can appropriately process these internal operations of the transformer.

**Presentation of Table 1:**
We have revised Table 1 to improve its clarity. Here, (1) (2) (3) mean three different settings of architectures of FAT Block, and the number only represents the indices of different architectures.

Reviewer comment (Experiments):
Adding a transformer on the codec side would result in a longer training time and a larger memory requirement that is not hardware-friendly for the actual deployment of the compression model. Can the authors provide the training time as well as the corresponding average and peak memory consumption for both training and test? Besides, the three methods compared in Table 5 are all transformer-based methods, and there is no comparison of decoding time with, for example, Minnen[1] and GMM[2], where both the encoder and decoder use a CNN structure.

Author response:
**Q2: Comparison with CNN-based methods**

**The integration of transformers into LIC is a notable trend for enhancing compression performance.** However, existing transformer-based LIC methods cannot achieve multiscale and directional analysis, which is an important and indispensable topic spanning from conventional codecs to recent LIC methods. Therefore, in this paper, we propose a frequency-aware transformer for LIC to further enable multiscale and directional analysis. To this end, we focus on **achieving better R-D performance with equivalent complexity over existing transformer-based LIC.**

According to the reviewer's comments, we compare our method with existing CNN-based [R10, R11] and transformer-based methods [R4, R9, R12] in terms of training speed, memory requirement, and FLOPs in Tables 5 and 6 in the revision (also seen as below). Note that all the existing methods claim that they require 2M$\sim$3.5M training steps in total for convergence. Compared with CNN-based methods, transformer-based methods exhibit evidently high R-D performance at the cost of memory consumption, training speed, and coding time. Notably, compared with existing transformer-based methods, the proposed method achieves apparent R-D performance improvement with only a slightly higher computational complexity.  In addition, many existing works [R13,R14,R15] focus on achieving efficient transformer, these facts imply the potential of our method in further exploring the efficient transformer-based LIC. 



| **Model**      | **GFLOPs (Enc.)** | **GFLOPs (Dec.)** | **Inference Latency (Enc.) (ms)** | **Inference Latency (Dec.) (ms)** | **#Params** | **BD-rate** |
| ----------------------------- | :------: |:-:| :-: |:-:|:-:|-:|
| Cheng et al. (2020) [R10]    | 154               | 229               | $>$1000                          | $>$1000                          | 26.60M      | 3.6%        |
| Minnen & Singh (2020)  [R11]  | 101               | 100          | 56                               | 43                               | 55.13M      | 1.1%        |
| Zhu et al. (2022) [R9] | 116              | 116               | 110                              | 99                               | 32.71M      | -3.3%       |
|   Zou et al. (2022) [R12]       | 285               | 276               | 97                               | 101                              | 99.58M      | -4.3%       |
| Liu et al. (2023)  [R4]     | 317               | 453               | 255                              | 322                              | 76.57M      | -11.9%      |
| Ours                          | 141               | 349               | 125                              | 242                              | 70.97M      | -14.5%      |
| VTM-12.1                       | -                 | -                 | -                                | -                                | -           | 0%          |

| **Model**                       | **Peak GPU Memory (GB) for Training** | **Peak GPU Memory (GB) for Test** | **Training Speed (steps/s)** |
| :------------------------------- | :-----------------------------------:| :--------------------------------: | :-----------------------------: |
| Cheng et al. (2020) [R10]       | 2.32                                | 0.59                             | 14.23                         |
|Minnen & Singh (2020)  [R11]     | 3.48                                | 0.50                             | 8.26                          |
| Zhu et al. (2022) [R9] | 16.84                               | 2.10                             | 2.66                          |
|   Zou et al. (2022) [R12]           | 10.87                               | 0.68                             | 4.29                          |
| Liu et al. (2023)  [R4]          | 14.14                               | 1.71                             | 2.13                          |
| Ours                            | 12.68                               | 1.09                             | 3.17                          |


We sincerely thank you for your efforts in reviewing this paper, and we are looking forward to your response.

Reviewer comment (Experiments):
It is not entirely clear how much improvement is brought about by the specific structure of the FDWA and FMFFN blocks, rather than the fact that some learnable layers have been included which would increase the overall capacity of the network. The authors have provided a few ablations in Figures 5 and 6 which do somewhat support their claim, but this could be made stronger in future versions.

Author response:
**Q1: Improvement brought by FDWA and FMFFN**

We want to clarify that our FDWA and FMFFN only **introduce less than 1% increase in model parameters** compared with the baseline, and the performance improvement does come from their specific and well-designed structure. In Table 1 of the revised manuscript, we compare the computational complexity, number of parameters, and R-D performance of three transformer-based transforms to show the superiority of the proposed FDWA and FMFFN. The four included transformer-based transforms are as follows.

- SwinT ($4\times4$): Swin-Transformer with $4\times4$ windows.
- SwinT ($16\times16$): Swin-Transformer with $16\times16$ window.
- FAT w/o FMFFN: Swin-Transformer with FAT blocks of FDWA only.
- FAT: Swin-Transformer with full FAT blocks, including FDWA and FMFFN. 


| Methods             | FDWA | FMFFN | **GFLOPs** | **#Params** | **BD-rate** |
|---------------------|------|-------|------------|-------------|-------------|
| SwinT ($4\times4$)  |    x   |     x   | 77.3       | 70.29M      | -9.3%       |
| SwinT ($16\times16$)|  x  |    x    | 91.3       | 70.50M      | -11.2%      |
| FAT w/o FMFFN       | ✔️  |    x    | 82.2       | 70.36M      | -13.3%      |
| FAT                 | ✔️  | ✔️     | 83.1       | 70.97M      | -14.5%      |
| VTM-12.1            | -    | -     | -          | -           | 0%          |

The above table shows that the proposed FDWA and FAT achieve significant R-D performance improvement (at most -5.2\% BD-rate reduction) with negligible increase in the number of parameters (within 1\%). Therefore, the increased model capacity is not the main reason for the evident performance enhancement. Moreover, both FDWA and FMFFN are indispensable components of the proposed FAT, contributing to the R-D performance enhancement.

Reviewer comment (Experiments):
What if additional windows with different aspect ratios were used? Would these provide further improvement? This could be achieved by either increasing $K$ (the number of heads), or by using the same number of heads but partitioning into more groups to accommodate the different aspect ratios.

Author response:
**Q2: Additional Windows with Different Aspect Ratios**

Firstly, we want to clarify that we employed the same total number of attention heads and partitioned them into more groups to accommodate various window types in ablation studies.

Moreover, we have finished experiments on FAT with diverse window shapes. The four groups of attention windows used in the experiments are as follows, where the total number of attention heads is identical.

- (a) Window attention with $4\times 4$ windows.
- (b) Window attention with the proposed FDWA that comprises of four types of windows (*i.e.*, $4\times4, 16\times16, 4\times16, 16\times4$).
- (c) Window attention with more diverse types of windows (*i.e.*, $ 4\times4, 16\times16,  4\times16, 16\times4, 4\times16$, $8\times4$, $8\times16$, and $16\times8$).
- (d) Window attention with larger aspect ratios (*i.e.*, $2\times2, 4\times4, 16\times16,  32\times32, 2\times32, 32\times2, 4\times16$, and $16\times4$).

| **Methods** | **PSNR** | **BPP** | **R-D Loss ↓** | **GPU Memory for Training** | **Training Speed (steps/s)** |
|:-------------|:----------:|:---------:|:--------------:|:-----------------------------:|:-------------------------------:|
| (a)         | 37.031   | 0.8949  | 1.552        | 10.36GB                     | 3.33                          |
| (b)         | 37.248   | 0.8910  | 1.517        | 12.68GB                     | 3.17                          |
| (c)         | 37.176   | 0.8838  | 1.520        | 13.95GB                     | 2.84                          |
| (d)         | 37.213   | 0.8817  | 1.513        | 15.02GB                     | 2.57                          |

All the models were trained for 500,000 training steps due to the time limit, and the Lagrangian multiplier $\lambda$ for all the models is 0.0483. The PSNR, BPP, and R-D loss are averaged across 24 images of the Kodak dataset. The comparison results are presented in the above table.  We observe that employing more diverse window shapes (*i.e.*, method (c)) does not bring further improvement compared with our FDWA. Meanwhile, method 
(d) employing windows with larger aspect ratios achieves slightly lower R-D loss, with a sacrifice in memory consumption and training speed. Besides, the proposed FDWA can achieve **a better trade-off between R-D performance and overall complexity**.

Reviewer comment (Experiments):
The impact of the FMFFN block seems to be rather small. What if instead of a block FFT, a learnable conv layer (and deconv for the IFFT) was used?

Author response:
**Q3: Impact of FMFFN**

The FMFFN is a **simple yet effective module** of the proposed framework, which adaptively modulates the frequency components, further bringing a 1.2% BD-rate reduction with negligible complexity overhead.

The quantitative results in Fig. 6 of the manuscript also illustrate the effect of FMFFN. FMFFN can help the high-bit-rate model reserve more high-frequency information while guiding the low-bit-rate model to focus more on the low-frequency information. Thus, the redundancies in the latent representation can be further reduced.

**Replace FFT with Convolutional Layers**

Thanks for your suggestion. First, we clarify the necessity of the block-wise FFT/IFFT in FMFFN:

1. **FFT and IFFT** constitute a reversible transformation pair, guaranteeing that frequency modulation with matrix $W$ in the frequency domain is a comprehensive and meaningful operation.
  
2. **Block operation** further supports applying frequency modulation operation to input images with different resolutions, as the size of block-wise filter $W$ only depends on the block size.

To further demonstrate the effect of FMFFN, we have conducted experiments to replace the non-trainable block-wise FFT/IFFT with trainable convolution/deconvolution (Conv/Deconv) layers. We also drop our filter $W$ for trainable Conv/Deconv layers since the linear weight filter has been integrated into the Conv layer.

However, introducing Conv and Deconv layers **with stride** in the feedforward network (FFN) led to instability in the model convergence.
We used two $3\times 3$ Conv layers **without stride** as a substitute for FFT/IFFT. We have also conducted another experiment using two $3\times 3$ depth-wise Conv layers to prevent a significant parameter increase. The results are shown in the Table below.

| **Methods**                        | **PSNR** | **BPP** | **R-D Loss ↓** | **#Params** |
|-------------------------------------|----------|---------|--------------|-------------|
| block FFT/IFFT with $W$             | 37.248   | 0.8910  | 1.517        | 70.97M      |
| two Conv layers                     | 36.993   | 0.8624  | 1.532        | 84.79M      |
| two depth-wise Conv layers          | 37.003   | 0.8641  | 1.534        | 70.52M      |


All the models were trained for 500,000 training steps due to the time limit, and the Lagrangian multiplier $\lambda$ for all the models is 0.0483. From the above table, we find the following conclusions.

- Replacing the block-wise frequency modulation operation with the trainable Conv layers leads to a **degradation** in R-D performance, showing the crucial effect of our FMFFN.

- Increasing the overall capacity of the model may **not** bring much performance improvement, suggesting that the performance improvement of FMFFN and FDWA comes from their specific structure.