Paper: When Semantic Segmentation Meets Frequency Aliasing (ICLR-2024)

Reviewer comment (Evaluation):
The authors must provide more visualizations on the three types of errors and the corresponding feature maps in frequence domain.

Author response:
Thank you for the valuable feedback and encouragement.

## W1, Q1, Q2: More visualizations

[Authors’ response]

We acknowledge the importance of visualization in elucidating how the proposed methods operate and rectify errors. In response to your valuable advice, we have included comprehensive visualizations and analyses in the revised appendix. As the figure cannot be added in the comment box, we recommend referring to Section J of the revised appendix for a detailed visual representation. Section J.1 discusses how the FAD improves the feature during downsampling (Figure 5), Section J.2 discusses how the FreqMix suppresses unnecessary high frequency in the background and object centers (Figure 6), and Section J.3 provides a more visualized analysis of how the FAD and FreqMix improve the feature and prediction while relieving three types of errors (Figure 7). Section K provides additional frequency analysis of features (Figures 8 and 9). For your convenience, we have also attached our description of the visualizations below.

In Figure 7 of the revised appendix, we randomly selected image patches with a high aliasing score from the Cityscapes dataset validation set. We observed that aliasing leads to a jagged phenomenon[1,2], disrupting object shapes and boundaries, resulting in false responses (see Figure 7(b) and (l) in the revised appendix) and displacement errors (see Figures 7(c), (d), and (e) in the revised appendix). DAF and FreqMix relieve this phenomenon and improve the feature representation thus resulting in lower false responses and displacement errors.

Besides, when high-frequency information is aliased, it transforms into false low-frequency information. For example, when two objects are close to each other, their high-frequency boundaries can be aliased to lower frequency during downsampling, causing the two objects to appear connected and their boundaries to be merged. This leads to merging errors (see Figures 7(a), (g), and (j) in the revised appendix). DAF/FreqMix solve this by removing/suppressing these high frequencies during downsampling/encoder block, leading to lower merging errors.

Moreover, high-frequency components in the object center or background can result in false responses (see Figures 7(i) and (k) in the revised appendix). FreqMix addresses this issue by suppressing the high frequency in the object center or background while preserving the high frequency at the boundaries (see Figures 6(a) and (b) in the revised appendix).

Thanks for your valuable comments that help us improve our manuscript and clarify the proposed method more clearly!

[1] Qian S, Shao H, Zhu Y, et al. Blending anti-aliasing into vision transformer. Advances in Neural Information Processing Systems, 2021, 34: 5416-5429.

[2] Zou, X., Xiao, F., Yu, Z., Li, Y., & Lee, Y. J. (2023). Delving deeper into anti-aliasing in convnets. *International Journal of Computer Vision*, *131*(1), 67-81.

Reviewer comment (Evaluation):
As to the DAF and FreqMix, how the filtered feature maps can help find the location of error map should be invastigated.

Author response:
Thank you for the valuable feedback and encouragement.

## W1, Q1, Q2: More visualizations

[Authors’ response]

We acknowledge the importance of visualization in elucidating how the proposed methods operate and rectify errors. In response to your valuable advice, we have included comprehensive visualizations and analyses in the revised appendix. As the figure cannot be added in the comment box, we recommend referring to Section J of the revised appendix for a detailed visual representation. Section J.1 discusses how the FAD improves the feature during downsampling (Figure 5), Section J.2 discusses how the FreqMix suppresses unnecessary high frequency in the background and object centers (Figure 6), and Section J.3 provides a more visualized analysis of how the FAD and FreqMix improve the feature and prediction while relieving three types of errors (Figure 7). Section K provides additional frequency analysis of features (Figures 8 and 9). For your convenience, we have also attached our description of the visualizations below.

In Figure 7 of the revised appendix, we randomly selected image patches with a high aliasing score from the Cityscapes dataset validation set. We observed that aliasing leads to a jagged phenomenon[1,2], disrupting object shapes and boundaries, resulting in false responses (see Figure 7(b) and (l) in the revised appendix) and displacement errors (see Figures 7(c), (d), and (e) in the revised appendix). DAF and FreqMix relieve this phenomenon and improve the feature representation thus resulting in lower false responses and displacement errors.

Besides, when high-frequency information is aliased, it transforms into false low-frequency information. For example, when two objects are close to each other, their high-frequency boundaries can be aliased to lower frequency during downsampling, causing the two objects to appear connected and their boundaries to be merged. This leads to merging errors (see Figures 7(a), (g), and (j) in the revised appendix). DAF/FreqMix solve this by removing/suppressing these high frequencies during downsampling/encoder block, leading to lower merging errors.

Moreover, high-frequency components in the object center or background can result in false responses (see Figures 7(i) and (k) in the revised appendix). FreqMix addresses this issue by suppressing the high frequency in the object center or background while preserving the high frequency at the boundaries (see Figures 6(a) and (b) in the revised appendix).

Thanks for your valuable comments that help us improve our manuscript and clarify the proposed method more clearly!

[1] Qian S, Shao H, Zhu Y, et al. Blending anti-aliasing into vision transformer. Advances in Neural Information Processing Systems, 2021, 34: 5416-5429.

[2] Zou, X., Xiao, F., Yu, Z., Li, Y., & Lee, Y. J. (2023). Delving deeper into anti-aliasing in convnets. *International Journal of Computer Vision*, *131*(1), 67-81.

Reviewer comment (Theory):
I need more clarification about the robustness of ESR (equivalent sampling rate). Clearly, the filters are orthogonal for the shown example (Fig 3). So there is no loss of information. But what if the filters are not orthogonal to each other? Assume a worst-case scenario where all filters are equal. In that case, the proposed ESR will give us a wrong sampling rate. So, is the proposed ESR intended to replace the regular “sampling rate,” or is it intended as a heuristic to select cutoff frequency? If it is the second, then the paper should clarify it.

Author response:
We appreciate your inspiring comments. The proposed Equivalent Sampling Rate (ESR) is designed as a heuristic for selecting the cutoff frequency, not as a replacement for the regular sampling rate. We have clarified this distinction in the conclusion of the revised main paper.

We agree that the generalization of the proposed ESR calculation is crucial, and it is computed under the assumption that downsampling filters are orthogonal. We have provided this missing part in the revised appendix (Section H, Table 3, Figure 4), the main analysis results are attached below. In our analysis, we quantitatively assessed the orthogonality of filters utilized in prominent models, including ResNet, Swin, ConvNeXt, HorNet, and DiNAT. Their weights are obtained by training on ImageNet. The average absolute cosine similarity, serving as a measure of orthogonality, ranges from 0.046 to 0.072, indicating a high degree of orthogonality (<< 1.0, which indicates all filters are equal). This finding supports the introduced ESR.

| Model        | ResNet-18 | ConvNeXt-T | Swin-T | HorNet-T | DiNAT-L |
| ------------ | --------- | ---------- | ------ | -------- | ------- |
| Abs. CosSim. | 0.067     | 0.072      | 0.06   | 0.069    | 0.046   |

Besides, your comments are insightful and inspiring, we think how to adjust the equivalent sampling rate based on the orthogonality of the filter is a very interesting and important problem which worth further investigation.

Reviewer comment (Writing):
Section 3.2, “Metrics for three errors.” — I think the middle one should be the definition of MErr.

Author response:
We appreciate your careful comments. We have corrected it and thoroughly reviewed the section to ensure the accuracy of the content and to prevent similar situations.

Reviewer comment (Evaluation):
“aligning with the observations in Figure 1 that false responses and merging mistakes predominantly exist in areas with relatively low aliasing scores.” — could you clarify how we are associating aliasing scores with a region of an image?

Author response:
Thank you for your careful comments. The reference to the figure was indeed incorrect. We have rectified this to *"Align with the observations in Figure **2**, where false responses and merging mistakes predominantly exist in areas with relatively low aliasing scores."* 

In Table 1 of the main paper, we provide evidence indicating that increasing the blur kernel leads to a reduction in the overall aliasing score. Consequently, there is a decrease in displacement error, while the occurrences of false responses and merging mistakes exhibit an upward trend. This aligns with the statistical findings presented in Figure 2 of the main paper, illustrating the distribution of three errors. In this figure, we observed that the aliasing score of false responses and merging mistakes is lower than that of displacement error, providing consistent support for our observations.

We also have included comprehensive visualizations and analyses in the revised appendix (Figure 7 in the appendix). As the figure cannot be added in the comment box, we recommend referring to Section J of the revised appendix for a detailed visual representation.

Reviewer comment (Evaluation):
“However, for deeper features at the second and third stages, the aliasing ratio decreases. This decrease does not signify a reduction in aliasing; rather, it occurs because the earlier high level of aliasing results in severe degradation. Consequently, the features become ‘hard pixels,’ meaning that the following stage struggles to extract useful information and loses response to objects.” — How can we confirm that it is caused by the side effect of aliasing? Applying significant noise can generate an out-of-distribution (OOD) sample of the internal layers, which can also lead to poor feature extraction.

Author response:
Thank you for your inspiring comments. We provide visualizations in Figure 5 of the revised appendix. As the figure cannot be added in the comment box, for your convenience, we have also attached our description of the visualization below. Please refer to Section J and Figure 5 of the revised appendix for a detailed visual representation.

In Figure 5 of the revised appendix, we have marked areas with high aliasing scores in the features with red boxes. Owing to the aliasing, the downsampled features exhibit a severe jagged phenomenon[2, 3], resulting in the degraded representation of object shapes and boundaries (see Figures 5(c) and (e) in the revised appendix). When downsampling increases to 4×, the response of some objects fades or is lost (see Figure 5(e) in the revised appendix). It is noteworthy that widely used state-of-the-art models, such as Swin Transformer and ConvNeXt, adopt a total downsampling stride of 32$\times$, potentially leading to an even more severe loss of response. After applying the proposed DAF to address the aliasing artifact, the jagged phenomenon is relieved (see Figures 5(d) and (f) in the revised appendix) and the response of objects still remains strong in the 4× downsampled features (see Figure 5(e) in the revised appendix). This indicates a strong correlation between the aliasing effect and a loss of object response.

Recent work [2] also observes a similar phenomenon, where an increase in image noise raises the high frequency in features (leading to aliasing), resulting in deeper features losing the response of interested objects. Applying a low-pass filter (which can be regarded as an anti-aliasing filter) can alleviate this problem and recover the response of interested objects.

[2] Qian S, Shao H, Zhu Y, et al. Blending anti-aliasing into vision transformer. Advances in Neural Information Processing Systems, 2021, 34: 5416-5429.

[3] Zou, X., Xiao, F., Yu, Z., Li, Y., & Lee, Y. J. (2023). Delving deeper into anti-aliasing in convnets. *International Journal of Computer Vision*, *131*(1), 67-81.

[4] Chen, L., Fu, Y., Wei, K., Zheng, D., & Heide, F. (2023). Instance Segmentation in the Dark. *International Journal of Computer Vision*, 1-21.