Paper: MULTISCALE ATTENTION VIA WAVELET NEURAL OPERATORS FOR VISION TRANSFORMER (ICLR-2024)

Reviewer comment (Evaluation):
Why MWA's Top-5% accuracy in Table 2 is not optimal?

Author response:
top-5% accuracy refers to the accuracy achieved by the model on the most difficult or challenging samples in the dataset.While the top-5% accuracy is indeed important and can provide insights into the model's performance on difficult cases, it may not reflect the overall performance of the model on the entire dataset. It is possible that the model performs better on the majority of the samples, but encounters some particularly challenging cases where its accuracy drops.Optimizing for top-5% accuracy alone not be the primary goal. Top 1 is more important.

Reviewer comment (Experiments):
Is the improvement in the performance of the proposed method due to this parallel structure of fusing features at different scales, or is it the discrete wavelet transform?

Author response:
both

Reviewer comment (Experiments):
Experiments are only performed on small datasets (CIFAR and Tiny-ImageNet). It might be difficult to generalize the conclusion to larger scale datasets (such as ImageNet).

Author response:
Due to the GPU limitation, we could not perform our experiments on large data  and large pach size  (all experiments were performed on 1 GPU), but we believe that our model performs better than other models on large data.

Reviewer comment (Experiments):
The motivation of this paper is to solve the quadratic complexity of self-attention for high resolution vision, but all experiments are performed on short sequences: 64 tokens (32 x 32 with patch size 4) or 256 tokens (64 x 64 with patch size 4).

Author response:
Due to the GPU limitation, we could not perform our experiments on large data  and large pach size  (all experiments were performed on 1 GPU), but we believe that our model performs better than other models on large data.

Reviewer comment (Writing):
In the preliminaries section, the phrase “... self-attention mechanism, an inductive bias that associates …” may be incorrect. The reviewer questions whether the authors mean "operation" instead of "inductive bias," as their understanding is that inductive bias is something different.

Author response:
Q1: ... self-attention mechanism, …” .  
In the context of self-attention mechanisms, "bias" is actually the correct term, not "operation". The phrase should be "an inductive bias that associates ...". The attention mechanism introduces a bias towards certain interactions or associations between elements.

Reviewer comment (Theory):
The claim “.. is a global operator, which means it may overlook or miss fine and moderate scale structures commonly present in natural images.” is disputed. The reviewer argues that this is only the case if the global kernel used is intrinsically low-frequent, and provides counterexamples from modern sequential models relying on global convolutions (Implicit Neural Representations, SSMs) which can describe high-frequencies and model long sequences like language and DNA in a fine-grained manner.

Author response:
Q2 :  “.. is a global operator, ....  
AFNO, as a global operator, may sometimes overlook or miss fine and moderate scale structures commonly found in natural images. It does not make a general claim about all global operators. The effectiveness of global operators in capturing fine and moderate scale structures depends on the characteristics and design of the specific operator or model used. Modern sequential models, like the Implicit Neural Representations (INRs) and Structured-Sparse Models (SSMs) you mentioned, have demonstrated the ability to model high frequencies and capture fine-grained details even with global convolutions.These models utilize parameterizations and structures that enable them to capture and represent both low and high-frequency components adequately. This allows them to describe long sequences, such as language and DNA sequences, in a fine-grained manner.Therefore, it is essential to consider the specific characteristics and design choices of the global operator or model being used before making broad statements about their ability to capture fine and moderate scale structures.

Reviewer comment (Theory):
The statement “FFT can extract periodic patterns due to the use of sine and cosine functions to analyze images, but it is not suitable for studying the spatial behavior of images with non-periodic patterns.” is considered incorrect. The reviewer notes that Fourier transform can describe arbitrary functions on a finite domain and that methods like JPEG prove its power. The periodicity assumption applies similarly to both Fourier and Wavelet transforms.

Author response:
Q3 : “FFT can extract periodic patterns.....  

The Fourier Transform, including its discrete version (FFT), is indeed a powerful tool for analyzing signals and images, not limited to periodic patterns. It can represent arbitrary functions in terms of sinusoidal components, and its application extends to a wide range of signal processing tasks, including image analysis.While it's true that Fourier analysis is particularly well-suited for periodic signals due to the use of sine and cosine functions, it doesn't mean it's limited to periodic patterns. In fact, as you pointed out, the Fourier Transform can be applied to describe arbitrary functions on a finite domain. But Wavelet transform works better than Fourier transform for the following reasons  
1. Time-Frequency Localization: Wavelets provide excellent time-frequency localization, which means they can capture both high-frequency and low-frequency details of a signal at different time points. This is particularly useful when analyzing signals with non-stationary characteristics, where frequency content changes over time.  
2. Multiresolution Analysis: Wavelets offer a multiresolution analysis, meaning that they can capture signal details at different scales. This allows for the identification of localized features within a signal. Fourier transforms, on the other hand, provide a global frequency representation, making it difficult to analyze localized features.  
3. Edge Detection: Wavelets are effective in detecting edges and discontinuities in signals due to their ability to capture

Reviewer comment (Writing):
In the method section, the phrase “we use all the coefficients of the last decomposition level” is unclear. The reviewer asks whether it means all coefficients up to that level or only those of that specific level, noting that in practice only one level of decomposition is used.

Author response:
Q4: “In this work, we use all the coefficients .....  
All the coefficients of the last level of analysis. We also did it with higher levels Decomposition.

Reviewer comment (Writing):
Relatedly, the statement “In DWT, the wavelet coefficients in the highest scale include the important features of the input…” is unclear to the reviewer.

Author response:
Q5:  “In DWT, the wavelet coefficients.....  
In DWT, the wavelet coefficients in the highest scale refer to the smallest details or features in the input. These details are considered important as they contribute to the overall understanding of the input.

Reviewer comment (Theory):
The author defines the Wavelet Neural Operator as a convolution in the Wavelet domain via multiplication, but Eq. 10 changes the operation to convolution on this space without motivation. The reviewer questions this choice, pointing out that convolution in Wavelet space equals multiplication in spatial domain, so the rationale is unclear.

Author response:
Q6 : The author defines the Wavelet Neural Operator as a convolution ....  

Convolution operates by sliding a small kernel over the input space, multiplying the kernel's values with the corresponding input values at each position, and summing them up. This process takes into account the neighboring values, allowing for the detection of patterns, edges, and other spatially dependent features.Multiplication, on the other hand, is a point-wise operation that does not consider the surrounding values. It is mainly used for scalar multiplication or element-wise operations, which may not capture the spatial relationships present in the data.By using convolution, we can effectively capture spatial information and exploit the locality of features in the input space, making it especially useful in tasks like image processing, where recognizing patterns and extracting relevant features require considering the context and relationships between neighboring pixels.

Reviewer comment (Reproducibility):
It is also unclear whether the Wavelet decomposition is used as a spatial input, or if each decomposition is considered independently from one another, and how frequency components are “mixed” across levels.

Author response:
Q8 :  is the Wavelet decomposition used as an spatial input, or is each decomposition considered independently from one another? How are frequency components “mixed” across levels of decomposition?  

In wavelet decomposition, the input signal is divided into different frequency bands or levels. Decomposition is done recursively, different frequency components are not mixed in each level. Instead, they are analyzed independently.

Reviewer comment (Experiments):
Can you design an experiment which will demonstrate that the improved results are indeed caused by a better analysis of tokens of different scales?

Author response:
Yes, the expriments can be shown on big data, but it requires gpu. We did all our expriments with 1 gpu.

Reviewer comment (Experiments):
What are the hyperparameters of the method? Can you perform ablation studies on them?

Author response:
Some common hyperparameters could include the number of decomposition levels, wavelet function,  kernel size...  
Yes, to perform ablation studies on these hyperparameters, one can systematically change one hyperparameter at a time while keeping the others constant.