Paper: Multi-Scale Representations by Varying Window Attention for Semantic Segmentation (ICLR-2024)

Reviewer comment (Presentation):
The manuscript requires structural refinement. Although the paper proposes a new methodology, the empirical validation predominantly resides in the appendix. The core content appears to be overly enmeshed in granular theoretical derivations, leading to a somewhat convoluted narrative structure.

Author response:
Thanks for the advice and careful review. We will refine the structure, delineation, and core content. We will check all captions, e.g. explaining PE in Fig. 2(b), and adjust the font size in all Figures to make readers clearly understand each of them. But with openness, I think it should be noted that two reviewers of all 4 think the presentation is good, one reviewer of all 4 thinks the presentation is fair, and two reviewers of all 4 think it is easy to follow.

Reviewer comment (Presentation):
The delineation between the figures, tables, and the main text is ambiguous. It is advisable to render the captions in a font slightly smaller than the primary text to enhance clarity. The layout of the illustrations also warrants improvement. For instance, in Figure 2, it would be beneficial to incorporate descriptions of specific letters either within the caption or designated sections of the image. The term "PE" is presented here, yet a precise elucidation is deferred until Section 3.3.

Author response:
Thanks for the advice and careful review. We will refine the structure, delineation, and core content. We will check all captions, e.g. explaining PE in Fig. 2(b), and adjust the font size in all Figures to make readers clearly understand each of them. But with openness, I think it should be noted that two reviewers of all 4 think the presentation is good, one reviewer of all 4 thinks the presentation is fair, and two reviewers of all 4 think it is easy to follow.

Reviewer comment (Evaluation):
The visualization results do not substantiate the issues you've raised. The quantified outcomes displayed in Figure 5 merely illustrate the discriminative results of the proposed network in comparison to Segformer for a specific image region. If the method you presented exhibits a substantial improvement over Segformer, I believe it would be acceptable. However, given that the data only indicates approximately a 1% enhancement relative to Segformer, there is a lack of targeted visualization results addressing the issues of scale inadequacy and field inactivation.

Author response:
Your concern involves that Figure 5 can not substantiate the issues we have raised, i.e. scale inadequacy and field inactivation, and the improvement of about 1% is not substantial over segformer to show the effectiveness of addressing these issues. We will answer this concern from TWO perspectives.

**(1)** I think visualization results showing the issues are not in Figure 5 but in Figure 1. In the beginning, the paper used EFR visualization to show the issue. It can be seen from Fig. 1(e) that the multi-scale feature of SegFormer only has the local and global scales(i.e. scale inadequacy) and some areas within the receptive field have almost zero values(i.e. field inactivation), as analyzed in Appendix A. The visualization of Figure 1 is averaged on many ADE20k val images, so I guess it might be too abstract to substantiate the proposed issue. THEREFORE, we sincerely request you to take a look at this [ANONYMOUS LINK](https://drive.google.com/file/d/1fKaeJBH8fFKDA4VM9X-ds4GTttfUdJ_E/view?usp=sharing) in which the specific ADE20K val image with EFRs of segformer and VWFormer are analyzed contrastively. We hope this new visualization will help to understand the receptive issue of SegFormer and show the strengths of VWFormer's multi-scale learning.

**(2)** For comparison of VWFormer to segformer, meanIoU is not the only metric to be considered. From Table 1, you can see that besides meanIoU Table 1 reported params, flops, and memory usage. Why do we think the overall metrics comparison is meaningful? First, An improvement with no more computational budgets is a PARETO improvement, and especially considering that our motivation is to improve multi-scale representations of classical methods, the PARETO optimum is more favorable. Second, VWFormer is architecturally similar to adding VWA to the MLP-decoder so you naturally propose the **Question 4**. The answer can be very simple. Towards PARETO, We weakened the powerful multi-layer aggregation (MLA) module of VWFormer by reducing its original dimensionality (this is one step. Also along with LLE introduced in Sec.4.3 and a more efficient setting of VWFormer introduced in Appendix F) so our VW. can even be much more efficient than Seg. according to Table 1. At the beginning of this research, we ran VWFormer by keeping the original channel setting of MLA (768 or 2048), the improvements of VW. over Seg. become around 1.5% to 2.3%. 
|mIoU on ADE20K | B0 | B1 | B2 | B3 | B4 | B5 |
| :- | :-: | :-: | :-: | :-: | :-: | :-: | 
|Seg. (original setting, shown in **Table 1**) | 38.0 | 43.1 | 47.5 | 50.0 | 51.1 | 51.8|
|VW. (keeping original setting, **NOT** shown in Table 1) | 40.3 | 44.8 | 49.5 | 51.6 | 52.6 | 53.5|
|VW. (with channel reduction and LLE, shown in **Table 1**) | 39.6 | 44.0 | 49.2 | 50.9 |  51.6 | 52.7|

Reviewer comment (Experiments):
The author introduced "varying window attention," but neglected a comparative analysis with other similar methods, such as the 'Shifted Window' in the Swin-Transformer.

Author response:
> neglected comparative analysis to similar methods, e.g. Swin.

This paper just begins by comparing similar methods including the Swin. For Swin, the analysis reveals that the 'Shifted-Window' mechanism directly incurs the field inactivation issue (please see **Introduction, Figure 1(d)** showing the shift-pattern of Swin's receptive field, and its captions cited **Appendix A**). 

> an operation akin to that of Swin

After reading your questions 2&3, I believe a clearer distinction between VWA and Swin attention may be helpful : ) **a**. Swin is of Self-attention. VWA is of Cross-attention. **b**. Swin shifts every window's location and does nothing about its size. VWA fixes every window's location and varies (enlarges) its context window's size (so VWA needs no mask). **c**. If Swin increases the receptive field for multi-scale learning, it can only stack itself upon itself and finally will meet the field inactivation issue. But if VWA increases the receptive field, it simply varies the context window's size, which should be computationally expensive but its incorporated DOPE and pre-scaling will handle this.

Reviewer comment (Experiments):
Some state-of-the-art segmentation methods like HRViT, SegViT-v2, ViT-adapter, Lawin Transformer, etc. could be compared in the experiments to better verify the superiority of your proposed architecture.

Author response:
Experiments in the paper were conducted mostly to show that VWFormer is the best MSD for semantic segmentation.


>HRViT-b1,b2, and b3

HRViT takes the MLP-decoder as the default MSD. If HRViT is paired with VWFormer, the improvements are reported as follows:
| mIoU on ADE20K | HRViT-b1 | HRViT-b2 | HRViT-b3 |
| :- | :-: | :-: | :-: | 
|HRViT-MLP | 45.6 | 48.8 | 50.2|
| HRViT-VW. | 46.9 | 50.4 | 51.6|

>SegViT-V2

SegViT-V2 is a decoder designed for ViT. We evaluate VW. paired with BEiTv2.

| mIoU on ADE20K | BEiT-V2-L  |
| :- | :-: | 
| SegViT-V2 |  58.2 |
| VW. | 59.0 |

>ViT-Adapter 

Adapter is equipped to ViT paired with UperNet as MSD. The UperNet is replaced with VWFormer and we get the following improvements:
| mIoU on ADE20K | ViT-B | ViT-L | 
| :- | :-: | :-: | 
|Adapter-Uper. | 52.5 | 54.4|
|Adapter-VW. | 54.1 | 55.8 |


> Lawin Transformer

Indeed, VW. is Version 2 (or called thoroughly improved version) of Lawin.

Reviewer comment (Experiments):
Why not experiment with the same Encoder paired with different decoders (more than three)? This can help better show the consistent effectiveness of your approach.

Author response:
Good idea to show our consistent superiority. The backbone choice depends on what the decoder's official paper used or what most papers paired it with. And the order in our paper is based on the performance of compared decoders.  Below is the result obtained by the same Encoder--Swin Transformer--paired with FPN, MLP-decoder, UPerNet, and our VW, respectively on ADE20K, and the order of other decoders is module-size ascending.

| mIoU | Swin-Ti | Swin-S | Swin-B | Swin-L | 
| :----: | :----: | :----: | :----: | :----: | 
| FPN | 44.7 |  48.0 | 51.7 | 53.3 | 
| MLP | 46.2 | 48.4 | 52.5 | 53.9 | 
| Uper | 45.8 | 49.2 | 52.4 | 54.1 | 
| VW. | 46.9 | 50.4 | 53.6 | 55.4 | 

Likewise, we will rearrange the order of the compared modules in the paper according to their sizes.

Reviewer comment (Experiments):
The computational load shown in Equation (2) appears highly reminiscent of the Swin Transformer. Could you possibly incorporate a comparison with the Swin Transformer?

Author response:
The only similarity between Swin and VWA is they are both as computationally efficient as LWA. Question 2 also focuses on the computational load. To avoid misunderstandings because in Summary you said "the authors introduced LWA", it needs to be clarified that LWA computed by Eq. 2 is a preliminary idea, neither our contribution nor Swin's contribution. Whether it's the Swin or our proposed VWA, both are extensions of LWA. As explained in point **b** above, Swin only changes the location of the Local Window but does not change its size. Hence, Swin's computational load is the same as that of LWA, given by $4(HW)(C^2) + 2(HW)(P^2)C$ in Eq. 2. 

Tip: In the paper, we presented the computation of LWA straightforwardly. Since you asked this question, if you want more details about how $4(HW)(C^2) + 2(HW)(P^2)C$ is obtained we will show you here. If you already know it, just skip to <For Weakness 5>. Assuming the feature map is of shape ($H, W, C$) and it can be split uniformly into ($H/P \times W/P = \frac{HW}{P^2}$) patches with shapes of ($P \times P \times C$), for the LWA implements self-attention on one patch, its computation consists of two parts, first is mapping query, key, value, and output, so the first part of computation is ($4 \times P \times P \times C_{in} \times C_{out} = 4{P^2}{C^2}$). And second is attention computation including calculating attention maps and doing weighted summation, both of which are ($P^2 \times P^2 \times C = {P^4}C$) indicating that every pixel's feature of the patch will multiply itself elementwise and every pixel does weighted summation over all pixels according to the attention maps, so the second part of the computation is ($2 {P^4}C$). Now that for one patch the computation cost is $4{P^2}{C^2}+2{P^4}C$, for the whole feature map the computation cost is $[\frac{HW}{P^2} \times (4{P^2}{C^2}+2{P^4}{C})] = 4{(HW)}{(C^2)}+2{(HW)}{(P^2)}C$

Reviewer comment (Theory):
SegFormer utilizes an MLP Decoder, which doesn't encompass an attention mechanism. Yet, in your VWA mechanism, you've incorporated an attention mechanism. How does this lead to a reduction in parameter count? This can be better analyzed.

Author response:
Second, VWFormer is architecturally similar to adding VWA to the MLP-decoder so you naturally propose the **Question 4**. The answer can be very simple. Towards PARETO, We weakened the powerful multi-layer aggregation (MLA) module of VWFormer by reducing its original dimensionality (this is one step. Also along with LLE introduced in Sec.4.3 and a more efficient setting of VWFormer introduced in Appendix F) so our VW. can even be much more efficient than Seg. according to Table 1. At the beginning of this research, we ran VWFormer by keeping the original channel setting of MLA (768 or 2048), the improvements of VW. over Seg. become around 1.5% to 2.3%. 
|mIoU on ADE20K | B0 | B1 | B2 | B3 | B4 | B5 |
| :- | :-: | :-: | :-: | :-: | :-: | :-: | 
|Seg. (original setting, shown in **Table 1**) | 38.0 | 43.1 | 47.5 | 50.0 | 51.1 | 51.8|
|VW. (keeping original setting, **NOT** shown in Table 1) | 40.3 | 44.8 | 49.5 | 51.6 | 52.6 | 53.5|
|VW. (with channel reduction and LLE, shown in **Table 1**) | 39.6 | 44.0 | 49.2 | 50.9 |  51.6 | 52.7|

Reviewer comment (Presentation):
Too much space is spent to explain the memory usage problem, resulting in some experimental results, such as Table 9 and some ablation experiments, which can only be placed in the appendix.

Author response:
Thanks for the structure advice. We will move key ablation studies to the main text and some equations explaining memory usage to the appendix, re-balancing the math and experiments.

Reviewer comment (Experiments):
Lack of ablation experiments on the copy-shift padding mode. It is mentioned in the paper that filling with zero will cause attention collapse, but there are no experimental results to explain it.

Author response:
> Lack of ablation experiments on the copy-shift padding mode

The ablation experiment on the copy-shift padding mode is **Table 8 Left** in the paper. It will be moved to the main text.

> Why not take the adjacent part to padding

I think you refer to adjacent part padding as the symmetric padding mode in Pytorch (If I mistake it please tell me). In this research, we have tried many techniques to tackle attention collapse, the symmetric padding mode has also been a candidate but it did not provide high improvements like CSP. This is because the symmetric mode only copies the already existing feature within the context window but CSP offers the window at the edge and corner a wider contextual receptivity.

What does the "wider contextual receptivity" indicate? For example, given a feature map of $64\times64$, the local window is $8\times8$, and the context window is $32\times32$. If the VWA is at the upper left corner, the context window's coverage on the feature map is the feature map's [-12:20, -12:20] (here minus means outside feature map), so $12\times20+20\times12 +12\times12$ paddings are needed for the context window. That means, with symmetric padding mode, the padding elements are shifted from [0:12, 0:20], [0:20, 0:12], and [0:12, 0:12] of the feature map, all of which are already covered by the context window. But if using CSP, from Eq. 17-18, the padding elements are shifted from [20:32, 0:20], [0:20, 20:32], and [20:32, 20:32], and now the context window's coverage is changing and not initial [-12:20, -12:20] of the feature map but becomes [0:32, 0:32] of the feature map.

Below is our previous trial result on the adjacent padding. It can be seen the adjacent padding's improvement is not that high compared to copy&shift.
| padding mode | mIoU(/MS) |
| :----: | :----: | 
| zero padd. | 51.6 / 52.6 | 
| adjacent |  51.9 / 52.9 | 
|copy&shift |  52.3 / 53.6|

Reviewer comment (Experiments):
There is no comparison between VWA and LWA.

Author response:
Sec 3.2 and Fig. 2 show how LWA becomes VWA and conclude that LWA is a special VWA with $R=1$. Table 6 analyzes the impact of LWA on VWFormer. If you want some particular comparison, please let me know and I will show them soon : )

Reviewer comment (Experiments):
The pre-scaling principle is to reduce memory usage, and the impact of pre-scaling on performance is not reflected.

Author response:
>  The impact of pre-scaling on performance is not reflected & What does pre. or post. in Table 7 refer to?

The impact of pre-scaling is reflected in Table 7 because pre. or post. refers to Pre-scaling and Post-scaling.

Sec. 3.2 shows that VWA with **no scaling** leads to unaffordable computation cost and memory usage. Sec. 3.3 shows that  **post-scaling** is a naive solution able to remove the extra computation but unable to handle the memory usage, and **pre-scaling** is proposed for handling both of computation cost and memory usage. Table 7 compares these three strategies. Then you question about:

> PE refers to padding with zero or CSP?

PE refers to patch embedding, just the same as 'patch embedding' in ViT[1]. Regardless of the post-scaling or pre-scaling, there must be specific scaling operators. PE is the operator doing post-scaling. For pre-scaling, we propose DOPE paired with PE, specifically DOPE--> PE, to implement it, as introduced in Sec 3.3.

Reviewer comment (Novelty):
The method part is not abundant, such as the short path branch. Overall, it seems that the original is not high, mainly VWA.

Author response:
> What is a short path branch

A short path (means with no heavy components along this path) branch refers to a residual connection[2] or an identity mapping i.e. 1x1 conv[3]. 

> The original is not high, mainly VMA

The basic idea of VMA is original. The architecture of VWFormer, even if it has no VWA, is original. The DOPE with pre-scaling strategy is original because it is a new way to embed patches for a Transformer-like vision framework. The copy&shift padding is original because it is a new padding mode different from off-the-shelf padding modes existing in Numpy or Pytorch. 



Ref.

-- [1] An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2022

-- [2] Deep residual learning for image recognition. In CVPR, 2016.

-- [3] Identity mappings in deep residual networks. In ECCV, 2016

Reviewer comment (Evaluation):
Inference time is not reflected in the experiments.

Author response:
> Inference time

To compare the inference time, we supplement Tables 1, 2, and 3 in the paper by assessing the frame per second (FPS), respectively. 

Comparison of Ours to SegFormer
| FPS | MiTB0 | MiTB1| MiTB2| MiTB3 | MiTB4| MiTB5| 
| :----: | :----: | :----: | :----: | :----: | :----: | :----: | 
| Seg. | 50.5 | 46.2 | 30.9 | 22.1 | 15.5 | 11.9 | 
| VW. | 53.4 | 49.8 | 35.3 | 26.8 | 19.2 | 14.0 |

Comparison of Ours to Swin-Uper.
| FPS | Swin-Ti | Swin-S | Swin-B | Swin-L | 
| :----: | :----: | :----: | :----: | :----: | 
| Uper. | 18.5 | 15.2 | 8.7 | 6.2 | 
| VW. | 24.4 | 22.0 | 15.9 | 8.8 | 

Comparison of Ours to Swin-MaskFormer
| FPS | Swin-Ti | Swin-S | Swin-B | Swin-L | 
| :----: | :----: | :----: | :----: | :----: | 
| Mask.-FPN | 22.1 | 19.6 | 12.6 | 7.9 | 
| Mask.-VW. | 23.1 | 20.7 | 12.9 | 7.9 | 

In summary, our inference time is faster than SegFormer and MaskFormer, much faster than UperNet, and slightly faster than FPN.

Reviewer comment (Experiments):
For copy-shift padding mode (CSP), why not take the adjacent part to padding?

Author response:
I think you refer to adjacent part padding as the symmetric padding mode in Pytorch (If I mistake it please tell me). In this research, we have tried many techniques to tackle attention collapse, the symmetric padding mode has also been a candidate but it did not provide high improvements like CSP. This is because the symmetric mode only copies the already existing feature within the context window but CSP offers the window at the edge and corner a wider contextual receptivity.

What does the "wider contextual receptivity" indicate? For example, given a feature map of $64\times64$, the local window is $8\times8$, and the context window is $32\times32$. If the VWA is at the upper left corner, the context window's coverage on the feature map is the feature map's [-12:20, -12:20] (here minus means outside feature map), so $12\times20+20\times12 +12\times12$ paddings are needed for the context window. That means, with symmetric padding mode, the padding elements are shifted from [0:12, 0:20], [0:20, 0:12], and [0:12, 0:12] of the feature map, all of which are already covered by the context window. But if using CSP, from Eq. 17-18, the padding elements are shifted from [20:32, 0:20], [0:20, 20:32], and [20:32, 20:32], and now the context window's coverage is changing and not initial [-12:20, -12:20] of the feature map but becomes [0:32, 0:32] of the feature map.

Below is our previous trial result on the adjacent padding. It can be seen the adjacent padding's improvement is not that high compared to copy&shift.
| padding mode | mIoU(/MS) |
| :----: | :----: | 
| zero padd. | 51.6 / 52.6 | 
| adjacent |  51.9 / 52.9 | 
|copy&shift |  52.3 / 53.6|

Reviewer comment (Writing):
What is the short path branch?

Author response:
A short path (means with no heavy components along this path) branch refers to a residual connection[2] or an identity mapping i.e. 1x1 conv[3].

Reviewer comment (Presentation):
What does pre. or post. in Table 7 refer to? PE refers to padding with zero or CSP?

Author response:
PE refers to patch embedding, just the same as 'patch embedding' in ViT[1]. Regardless of the post-scaling or pre-scaling, there must be specific scaling operators. PE is the operator doing post-scaling. For pre-scaling, we propose DOPE paired with PE, specifically DOPE--> PE, to implement it, as introduced in Sec 3.3.

Reviewer comment (Experiments):
The proposed copy-shift padding is a bit weird. It mimics the rolling operation in numpy and torch. I wonder why adopting this instead of symmetric or constant padding.

Author response:
### **For Weakness 1 and Question 1**: 
---
> Copy&shift mimics the rolling operation

Very insightful opinions. CSP seems like the roll function but very very vdifferent from it. If only seeing Figure 3, CSP is like "copy&shift" and the roll function is like "cut&shift". HOWEVER, more significantly, the roll function can only operate the elements of the feature map's edge parts, cutting an edge and shifting it to another edge. But from Eq. 17 and 18, CSP is proposed for copying the elements that are nearby outside the context window's inside-feature edges and then shifting these elements to padding so that the context window can cover them instead of zeros when the context window is at the edge or corner, which is the roll function in Numpy or Pytorch can not achieve. 

> Any ablation studies indicating the effectiveness

At the bottom of **page 15**, **Table 8 Left**, shows the effectiveness of CSP and that the zero padding mode impairs performance largely.

> Symmetric or constant padding.

In this research, we have tried many techniques to tackle attention collapse, and symmetric padding mode has also been a candidate but it did not provide high improvements like CSP. We think this is because the symmetric mode only copies the already existing feature within the context window but CSP offers the window at the edge or corner a wider contextual receptivity [1]. For constant padding, zero padding is a kind of constant padding so we are not very clear on what constants may be useful for this problem and all candidates having been tried by us are dynamic methods.
| padding mode | mIoU(/MS) |
| :----: | :----: | 
| zero padd. | 51.6 / 52.6 | 
| symmetric |  51.9 / 53.0 | 
|copy&shift |  52.3 / 53.6|


Tip:

[1] What does the "wider contextual receptivity" indicate? For example, given a feature map of $64\times64$, the local window is $8\times8$, and the context window is $32\times32$. If the VWA is at the upper left corner, the context window's coverage on the feature map is the feature map's [-12:20, -12:20] (here minus means outside feature map), so $12\times20+20\times12 +12\times12$ paddings are needed for the context window. That means, with symmetric padding mode, the padding elements are copied from [0:12, 0:20], [0:20, 0:12], and [0:12, 0:12] of the feature map, all of which are already covered by the context window. But if using CSP, from Eq. 17-18, the padding elements are copied from [20:32, 0:20], [0:20, 20:32], and [20:32, 20:32], and now the context window's coverage is not initial [-12:20, -12:20] of the feature map but becomes [0:32, 0:32] of the feature map.

Reviewer comment (Presentation):
Fig 2b and Sec 3.2 are not well-aligned. In Fig 2b there's a PE operation, which doesn't occur in Sec 3.2, which can confuse the readers.

Author response:
### **For Weakness 2**: 
---
We will revise the text-figure alignment issue in the paper to make readers clear. And we will polish every caption to let readers know what the used term in every Figure means : )

Reviewer comment (Experiments):
Are there any ablation studies indicating the effectiveness of copy-shift padding?

Author response:
### **For Weakness 1 and Question 1**: 
---
> Copy&shift mimics the rolling operation

Very insightful opinions. CSP seems like the roll function but very very vdifferent from it. If only seeing Figure 3, CSP is like "copy&shift" and the roll function is like "cut&shift". HOWEVER, more significantly, the roll function can only operate the elements of the feature map's edge parts, cutting an edge and shifting it to another edge. But from Eq. 17 and 18, CSP is proposed for copying the elements that are nearby outside the context window's inside-feature edges and then shifting these elements to padding so that the context window can cover them instead of zeros when the context window is at the edge or corner, which is the roll function in Numpy or Pytorch can not achieve. 

> Any ablation studies indicating the effectiveness

At the bottom of **page 15**, **Table 8 Left**, shows the effectiveness of CSP and that the zero padding mode impairs performance largely.

> Symmetric or constant padding.

In this research, we have tried many techniques to tackle attention collapse, and symmetric padding mode has also been a candidate but it did not provide high improvements like CSP. We think this is because the symmetric mode only copies the already existing feature within the context window but CSP offers the window at the edge or corner a wider contextual receptivity [1]. For constant padding, zero padding is a kind of constant padding so we are not very clear on what constants may be useful for this problem and all candidates having been tried by us are dynamic methods.
| padding mode | mIoU(/MS) |
| :----: | :----: | 
| zero padd. | 51.6 / 52.6 | 
| symmetric |  51.9 / 53.0 | 
|copy&shift |  52.3 / 53.6|


Tip:

[1] What does the "wider contextual receptivity" indicate? For example, given a feature map of $64\times64$, the local window is $8\times8$, and the context window is $32\times32$. If the VWA is at the upper left corner, the context window's coverage on the feature map is the feature map's [-12:20, -12:20] (here minus means outside feature map), so $12\times20+20\times12 +12\times12$ paddings are needed for the context window. That means, with symmetric padding mode, the padding elements are copied from [0:12, 0:20], [0:20, 0:12], and [0:12, 0:12] of the feature map, all of which are already covered by the context window. But if using CSP, from Eq. 17-18, the padding elements are copied from [20:32, 0:20], [0:20, 20:32], and [20:32, 20:32], and now the context window's coverage is not initial [-12:20, -12:20] of the feature map but becomes [0:32, 0:32] of the feature map.

Reviewer comment (Experiments):
From Table 7 in Supp, it seems avg_pool can be a substitution of PE, then I wonder what the performance will be like if the rescaling method "DOPE --> avg_pool --> 1x1 conv" is taken.

Author response:
### **For Question 2**: 
---
Very good question, you must have carefully read the paper. Although avg_pool can be a substitution for PE, the 2nd and 3rd rows in Table 7 show that avg_pool leads to an unignorable performance drop (53.7%-->52.8%). So if using avg_pool-->conv1x1 to substitute PE in the "DOPE-->PE" operation, there will be an unavoidable decrease in performance. From another view of efficiency, the 2nd and 3rd rows in Table 7 show that avg_pool--> conv1x1 is much more efficient than PE. But that works with the post-scaling strategy. With the pre-scaling strategy, according to the analysis in Sec. 3.3 and Eq. 13-15, it can be seen that PE is the same as a linear mapping. So PE in the "DOPE-->PE" operation with a pre-scaling strategy is efficient enough and it is redundant to replace it with avg_pool --> conv1x1 for more efficiency. Below is the comparison of "DOPE-->PE" with "DOPE-->avg_pool-->conv1x1", showing the performance drop caused by the substitution.
| rescaling method | mIoU(/MS) |
| :---- | :----: | 
| DOPE --> PE | 52.3 / 53.6 | 
| DOPE --> avg_pool-->conv1x1 |  51.8 / 52.8 |

Reviewer comment (Novelty):
The paper may not sufficiently differentiate VWA from existing local window attention mechanisms.

Author response:
### **For Weakness 1 and Question 1**: 
---
> How does VWA DIFFER from existing window attention mechanisms in handling multi-scale representation learning

In the paper we called them 'multi-scale attention' introduced in the last line of Sec. 2.1. Please see the second paragraph of **Sec. 2.1, Fig. 6, Appendix B, and Table 9** (both ISANet and ANN in Table 9 are 'multi-scale attention').

I am glad to present a summary of these related methods here (If you have some particular methods for comparing with our VWA, please let me know and I will compare them. To my knowledge, the listed works in the paper are all existing ones). ISANet[1] and GG-Transformer[2] have the same core idea but apply it to different tasks, semantic segmentation and image recognition, respectively. This idea first splits the image into windows to learn local representations and then interlaces the pixels among these windows so (pixels in) every window can learn global representations. The biggest problem with the ISANet-like method is they did not provide a possibility to learn representations of other scales, as illustrated in Fig. 6 (a). Table 9 compares ISANet with ours, showing that Our VWA outperforms them by ISANet margins. 

ANN[3], Focal Transformer[4], and Shunted Transformer[5] share the same core idea but apply it to different tasks. ANN is for semantic segmentation, and both Focal and Shunted are for image recognition. This idea depends on pooling filters to learn multi-scale representations (Appendix A analyzes the pooling's issue on multi-scale learning). One may question the proposed VWA also applies DOPE to downsample the image like pooling. The key point lies in that VWA changes the context window size so pixels in windows have varying receptive fields. However, the ANN-like method does not deal with the context window size so its receptive field is indeed fixed, as illustrated in Fig. 6(b). Table 9 also shows that VWA outperforms ANN by large margins.

In short word, existing multi-scale attention, whether based on ISANet or ANN, confines the representation learning within the window. Only VWA liberates the representation learning beyond the window, covering any region.

Ref.

-- [1] Interlaced sparse self-attention for semantic segmentation. In IJCV, 2021

-- [2] Glance-and-gaze
vision transformer. In NIPS, 2022

-- [3] Asymmetric non-local neural
networks for semantic segmentation. In CVPR, 2019

-- [4] Focal self-attention for local-global interactions in vision transformers. In NIPS, 2022

-- [5] Shunted self-attention
via multi-scale token aggregation. In CVPR, 2022

**To be continued.**

Reviewer comment (Novelty):
How does VWA fundamentally differ from existing attention mechanisms in handling multi-scale representations?

Author response:
### **For Weakness 1 and Question 1**: 
---
> How does VWA DIFFER from existing window attention mechanisms in handling multi-scale representation learning

In the paper we called them 'multi-scale attention' introduced in the last line of Sec. 2.1. Please see the second paragraph of **Sec. 2.1, Fig. 6, Appendix B, and Table 9** (both ISANet and ANN in Table 9 are 'multi-scale attention').

I am glad to present a summary of these related methods here (If you have some particular methods for comparing with our VWA, please let me know and I will compare them. To my knowledge, the listed works in the paper are all existing ones). ISANet[1] and GG-Transformer[2] have the same core idea but apply it to different tasks, semantic segmentation and image recognition, respectively. This idea first splits the image into windows to learn local representations and then interlaces the pixels among these windows so (pixels in) every window can learn global representations. The biggest problem with the ISANet-like method is they did not provide a possibility to learn representations of other scales, as illustrated in Fig. 6 (a). Table 9 compares ISANet with ours, showing that Our VWA outperforms them by ISANet margins. 

ANN[3], Focal Transformer[4], and Shunted Transformer[5] share the same core idea but apply it to different tasks. ANN is for semantic segmentation, and both Focal and Shunted are for image recognition. This idea depends on pooling filters to learn multi-scale representations (Appendix A analyzes the pooling's issue on multi-scale learning). One may question the proposed VWA also applies DOPE to downsample the image like pooling. The key point lies in that VWA changes the context window size so pixels in windows have varying receptive fields. However, the ANN-like method does not deal with the context window size so its receptive field is indeed fixed, as illustrated in Fig. 6(b). Table 9 also shows that VWA outperforms ANN by large margins.

In short word, existing multi-scale attention, whether based on ISANet or ANN, confines the representation learning within the window. Only VWA liberates the representation learning beyond the window, covering any region.

Ref.

-- [1] Interlaced sparse self-attention for semantic segmentation. In IJCV, 2021

-- [2] Glance-and-gaze
vision transformer. In NIPS, 2022

-- [3] Asymmetric non-local neural
networks for semantic segmentation. In CVPR, 2019

-- [4] Focal self-attention for local-global interactions in vision transformers. In NIPS, 2022

-- [5] Shunted self-attention
via multi-scale token aggregation. In CVPR, 2022

**To be continued.**

Reviewer comment (Evaluation):
Include a breakdown of performance gains across different classes or segments within the datasets used.

Author response:
### **For Weakness 2 and Question 2**: 
---
>Scalability or Generalizability

Good points, we will conduct in-depth research on them.

BUT now, some good indicators can be presented. Please see Appendix F. It introduces that our method has two settings of different model sizes to match backbones with different model capacities, including the efficient setting specially designed for lightweight backbones like SegFormer-B0 and B1 compared with ours in Table 1, and a normal setting for regular backbones. We have also experimented with scalability on the MLA module of VWFormer, for example, scaling the channel from 512 to 2048. With Mask2Former and Swin-L, our method can improve mIoU from 58.3% to 58.8%. To show generalizability, the proposed method has been evaluated with MiT (B0-->B5), Swin Transformer (Ti-->L), MaskFormer, Mask2Former, Deformable Attention, and ResNet(50-->101) in the paper. The benchmarks contain three challenging datasets as introduced in Appendix G. 

> A breakdown of performance gains

I will show a breakdown of performance gains within Cityscapes which has 19-class segments. The upper results are obtained by MiT-B5 paired with SegFormer head (mIoU 82.26%) and the lower results are obtained by MiT-B5 paired with our VWFormer (mIoU 82.87%). 

|method | road | side. | build. | wall | fence | pole | light | sign | vege. | terrain | sky | man | rider | car | truck | bus | train | motor | bike|
| :-| :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | 
|Seg. | 98.5 | 87.3 | 93.7 | **68.6** | 65.7 | 69.5 | 75.6 | 81.8 | 93.2 | 66.2 | **95.7** | 85.3 | 69.6 | 95.6 | **85.7** | 91.7 | **84.7** | 73.8 | 80.7|
|VW. |98.5 | 87.6 | 94.0 | 68.4 | 68.7 | 73.0 | 77.3 | 84.5 | 93.5 | 66.3 | 95.4 | 86.8 | 71.2 | 96.2 | 77.4 | 93.1 | 84.6 | 75.2 | 82.2|
 
The bold number is the class that the counterpart performs better than ours. Except for the "truck" class where SegFormer outperforms ours largely, which seems like a biased result, on the 'wall', 'sky', and 'train' SegFormer only slightly outperforms ours (by avg. 0.2%). And on the other 15 classes, Ours shows consistent superiority to SegFormer (by avg. 1.4%).

Besides, we did a breakdown of performance gains within ADE20K which has 150-class segments. Since it has too many classes, I will list some important statistics. The comparable results are obtained from Swin-L-Mask2Former and Swin-L-Mask2Former with VWFormer in place of Deform-FPN. (1) The pure Mask2Former achieves 56.96% mIoU and ours achieves 58.27% mIoU. (2) In all 150 classes, their methods outperform ours in 58 classes, while ours outperform theirs in 92 classes. (3) In the 58 categories where their methods outperform ours, the average improvement is -4.7%. In the 92 categories where ours outperforms theirs, the average improvement is 5.2%.