Original: Q UADTREE A TTENTION FOR V ISION T RANSFORMERS
Revised: Q UADTREE A TTENTION FOR V ISION T RANSFORMERS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Transformers have been successful in many vision tasks, thanks to their capability of capturing long-range dependency.
Revised: Transformers have been successful in many vision tasks, thanks to their capability of capturing long-range dependency.

Original: However, their quadratic computational complexity poses a major obstacle for applying them to vision tasks requiring dense predictions, such as object detection, feature matching, stereo, etc.
Revised: However, their quadratic computational complexity poses a major obstacle for applying them to vision tasks requiring dense predictions, such as object detection, feature matching, stereo, etc.

Original: We introduce QuadTree Attention, which reduces the computational complexity from quadratic to linear.
Revised: We introduce QuadTree Attention, which reduces the computational complexity from quadratic to linear.

Original: Our quadtree transformer builds token pyramids and computes attention in a coarse-to-ﬁne manner.
Revised: Our quadtree transformer builds token pyramids and computes attention in a coarse-to-ﬁne manner.

Original: At each level, the top K patches with the highest attention scores are selected, such that at the next level, attention is only evaluated within the relevant regions corresponding to these top K patches.
Revised: At each level, the top K patches with the highest attention scores are selected, such that at the next level, attention is only evaluated within the relevant regions corresponding to these top K patches.

Original: We demonstrate that quadtree attention achieves state-of-the-art performance in various vision tasks, e.g. with 2.7% improvement in feature matching on ScanNet, about 50% ﬂops reduction in stereo, 0.9% improvement in top-1 accuracy on ImageNet classiﬁcation, and 0.3% improvement and 30% ﬂops reduction on COCO object detection.
Revised: We demonstrate that quadtree attention achieves state-of-the-art performance in various vision tasks, e.g. with 2.7% improvement in feature matching on ScanNet, about 50% ﬂops reduction in stereo matching, 0.6-1.4% improvement in topaccuracy on ImageNet classiﬁcation, and 1.3-1.6% improvement with 30% ﬂops reduction on COCO object detection.

Original:  I NTROCUTION
Revised:  I NTROCUTION

Original: Transformers can capture long-range dependencies by the attention module and have demonstrated tremendous success in natural language processing tasks.
Revised: Transformers can capture long-range dependencies by the attention module and have demonstrated tremendous success in natural language processing tasks.

Original: In recent years, transformers have also been adapted to computer vision tasks for image classiﬁcation (Dosovitskiy et al., 2020), object detection (Wang et al., 2021c), semantic segmentation (Liu et al., 2021), feature matching (Sarlin et al., 2020), and stereo (Li et al., 2021), etc.
Revised: In recent years, transformers have also been adapted to computer vision tasks for image classiﬁcation (Dosovitskiy et al., 2020), object detection (Wang et al., 2021c), semantic segmentation (Liu et al., 2021), feature matching (Sarlin et al., 2020), and stereo (Li et al., 2021), etc.

Original: Typically, images are divided into patches and these patches are ﬂattened and fed to a transformer as word tokens to evaluate attention scores.
Revised: Typically, images are divided into patches and these patches are ﬂattened and fed to a transformer as word tokens to evaluate attention scores.

Original: However, transformers have quadratic computational complexity in terms of the number of tokens, i.e. number of image patches.
Revised: However, transformers have quadratic computational complexity in terms of the number of tokens, i.e. number of image patches.

Original: Thus, applying transformers to computer vision applications requires careful simpliﬁcation of the involved computation.
Revised: Thus, applying transformers to computer vision applications requires careful simpliﬁcation of the involved computation.

Original: To utilize the standard transformer in vision tasks, many works opt to apply it on low resolution or sparse tokens.
Revised: To utilize the standard transformer in vision tasks, many works opt to apply it on low resolution or sparse tokens.

Original: ViT (Dosovitskiy et al., 2020) uses coarse image patches of 16 × 16 pixels to limit the number of tokens.
Revised: ViT (Dosovitskiy et al., 2020) uses coarse image patches of 16 × 16 pixels to limit the number of tokens.

Original: DPT (Ranftl et al., 2021) up-samples low-resolution results from ViT to high resolution maps to achieve dense predictions.
Revised: DPT (Ranftl et al., 2021) up-samples low-resolution results from ViT to high resolution maps to achieve dense predictions.

Original: SuperGlue
Revised: SuperGlue

Original: (Sarlin et al., 2020) applies transformer on sparse image keypoints.
Revised: (Sarlin et al., 2020) applies transformer on sparse image keypoints.

Original: Focusing on correspondence and stereo matching applications, Germain et al.
Revised: Focusing on correspondence and stereo matching applications, Germain et al.

Original: (2021) and Li et al.
Revised: (2021) and Li et al.

Original: (2021) also apply transformers at a low resolution feature map.
Revised: (2021) also apply transformers at a low resolution feature map.

Original: However, as demonstrated in several works (Wang et al.
Revised: However, as demonstrated in several works (Wang et al.

Original: (2021c)
Revised: (2021c)

Original: ; Liu et al.
Revised: ; Liu et al.

Original: ; Sun et al.
Revised: ; Sun et al.

Original: Li et al.
Revised: Li et al.

Original: (2021)), applying transformers on high resolution is beneﬁcial for a variety of tasks.
Revised: (2021)), applying transformers on high resolution is beneﬁcial for a variety of tasks.

Original: Thus, many efforts have been made to design efﬁcient transformers to reduce computational complexity.
Revised: Thus, many efforts have been made to design efﬁcient transformers to reduce computational complexity.

Original: Linear approximate transformers (Katharopoulos et al., 2020; Wang et al., 2020) approximate standard attention computation with linear methods.
Revised: Linear approximate transformers (Katharopoulos et al., 2020; Wang et al., 2020) approximate standard attention computation with linear methods.

Original: However, empirical studies (Germain et al., 2021;
Revised: However, empirical studies (Germain et al., 2021;

Original: Chen et al., 2021) show those linear transformers are inferior in vision tasks.
Revised: Chen et al., 2021) show those linear transformers are inferior in vision tasks.

Original: To reduce the computational cost, the PVT (Wang et al., 2021c) uses downsampled keys and values, which is harmful to capture pixel-level details.
Revised: To reduce the computational cost, the PVT (Wang et al., 2021c) uses downsampled keys and values, which is harmful to capture pixel-level details.

Original: In comparison, the Swin-Transformer (Liu et al., 2021) restricts the attention in local windows, which might hurt long-range dependencies, the most important merit of transformers.
Revised: In comparison, the Swin Transformer (Liu et al., 2021) restricts the attention in local windows in a single attention block, which might hurt long-range dependencies, the most important merit of transformers.

Original: Unlike all these previous works, we design an efﬁcient vision transformer that captures both ﬁne image details and long-range dependencies.
Revised: Unlike all these previous works, we design an efﬁcient vision transformer that captures both ﬁne image details and long-range dependencies.

Original: Inspired by the observation that most image regions are irrelevant, we build token pyramids and compute attention in a coarse to ﬁne manner.
Revised: Inspired by the observation that most image regions are irrelevant, we build token pyramids and compute attention in a coarse to ﬁne manner.

Original: In this way, we can quickly skip irrelevant regions in the ﬁne level if their corresponding coarse level regions are
Revised: In this way, we can quickly skip irrelevant regions in the ﬁne level if their corresponding coarse level regions are

Original: [Figure]
Revised: [Figure]

Original: not promising.
Revised: not promising.

Original: For example, as in Figure 1, at the 1st level, we compute the attention of the blue image patch in image A with all the patches in image B and choose the top K (here, K = 2 ) patches which are also highlighted in blue.
Revised: For example, as in Figure 1, at the 1st level, we compute the attention of the blue image patch in image A with all the patches in image B and choose the top K (here, K = 2 ) patches which are also highlighted in blue.

Original: In the 2nd level, for the four framed sub-patches in image A (which are children patches of the blue patch at the 1st level), we only compute their attentions with the sub-patches corresponding to the top K patches in image B at the 1st level.
Revised: In the 2nd level, for the four framed sub-patches in image A (which are children patches of the blue patch at the 1st level), we only compute their attentions with the sub-patches corresponding to the top K patches in image B at the 1st level.

Original: All the other shaded sub-patches are skipped to reduce computation.
Revised: All the other shaded sub-patches are skipped to reduce computation.

Original: We highlight two sub-patches inimage Ain yellowand green.
Revised: We highlight two sub-patches in A yellow green.

Original: Their corresponding topK patches in image B are also highlightedin the same color.
Revised: Their corresponding top highlighted

Original: This process is iterated in the 3rd level, where we only show the sub-sub-patches corresponding to the green sub-patch at the 2nd level.
Revised: This process is iterated in the 3rd level, where we only show the sub-sub-patches corresponding to the green sub-patch at the 2nd level.

Original: In this manner, our method can both obtain ﬁne scale attention and retain long-range connections.
Revised: In this manner, our method can both obtain ﬁne scale attention and retain long-range connections.

Original: Most importantly, only sparse attention is evaluated in the whole process.
Revised: Most importantly, only sparse attention is evaluated in the whole process.

Original: Thus, our method has low memory and computational costs.
Revised: Thus, our method has low memory and computational costs.

Original: Since a quadtree structure is formed in this process, we refer to our method as QuadTree Attention, or QuadTree Transformer.
Revised: Since a quadtree structure is formed in this process, we refer to our method as QuadTree Attention, or QuadTree Transformer.

Original: In experiments, we demonstrate the effectiveness of our quadtree transformer in both tasks requiring cross attention, e.g. feature matching and stereo, and tasks only utilizing self-attention, e.g. image classiﬁcation and object detection.
Revised: In experiments, we demonstrate the effectiveness of our quadtree transformer in both tasks requiring cross attention, e.g. feature matching and stereo, and tasks only utilizing self-attention, e.g. image classiﬁcation and object detection.

Original: Our method achieves state-of-the-art performance with signiﬁcantly reduced computation comparing to relevant efﬁcient transformers (Katharopoulos et al.
Revised: Our method achieves state-of-the-art performance with signiﬁcantly reduced computation, comparing to relevant efﬁcient transformers (Katharopoulos et al.

Original: (2020); Wang et al.
Revised: (2020); Wang et al.

Original: (2021c)
Revised: (2021c)

Original: ; Liu et al.
Revised: ; Liu et al.

Original: In feature matching, we achieve 60.3 AUC@20 ◦ in ScanNet (Dai et al., 2017), 2.7 higher than the linear transformer (Katharopoulos et al., 2020) but with similar ﬂops.
Revised: In feature matching, we achieve 60.3 AUC@20 ◦ in ScanNet (Dai et al., 2017), 2.7 higher than the linear transformer (Katharopoulos et al., 2020) but with similar ﬂops.

Original: In stereo matching, we achieve a similar end-point-error as standard transformer (Li et al., 2021) but with about 50% ﬂops reduction and 40% memory reduction.
Revised: In stereo matching, we achieve a similar end-point-error as standard transformer, (Li et al., 2021) but with about 50% ﬂops reduction and 40% memory reduction.

Original: In image classiﬁcation, we achieve 82.6% top-1 accuracy in ImageNet (Deng et al., 2009), 6.2% higher than
Revised: In image classiﬁcation, we achieve 82.6% top-1 accuracy in ImageNet (Deng et al., 2009), 6.2% higher than

Original: ResNet (He et al., 2016) and 1.3% higher than the Swin-Transformer-T (Liu et al., 2021), with less parameters and ﬂops.
Revised: ResNet (He et al., 2016) and 1.3% higher than the Swin Transformer-T (Liu et al., 2021), with fewer parameters and ﬂops.

Original: In object detection, our QuadTree Attention + RetinaNet achieves 44.9 AP in
Revised: In object detection, our QuadTree Attention + RetinaNet achieves 46.2 AP in

Original: COCO (Lin et al., 2014), 6.4 higher than the backbone ResNet101 (He et al., 2016) but around 35% reduction in ﬂops.
Revised: COCO (Lin et al., 2014), 1.6 higher than the backbone PVTv2 (Wang et al., 2021b) but around 35% reduction in ﬂops.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Efﬁcient Transformers.
Revised: Efﬁcient Transformers.

Original: Transformers have shown great success in both natural language processing and computer vision.
Revised: Transformers have shown great success in both natural language processing and computer vision.

Original: Due to the quadratic computational complexity, the computation of full attention is unaffordable when dealing with long sequence tokens.
Revised: Due to the quadratic computational complexity, the computation of full attention is unaffordable when dealing with long sequence tokens.

Original: Therefore, many works design efﬁcient transformers, aiming to reduce computational complexity (Katharopoulos et al., 2020;
Revised: Therefore, many works design efﬁcient transformers, aiming to reduce computational complexity (Katharopoulos et al., 2020;

Original: Choromanski et al., 2020;
Revised: Choromanski et al., 2020;

Original: Wang et al., 2020; Lee et al., 2019;
Revised: Wang et al., 2020; Lee et al., 2019;

Original: Ying et al., 2018).
Revised: Ying et al., 2018).

Original: Current efﬁcient transformers can be categorized into three classes.
Revised: Current efﬁcient transformers can be categorized into three classes.

Original: 1) Linear approximate attention (Katharopoulos et al., 2020; Choromanski et al., 2020; Wang et al., 2020; Beltagy et al., 2020; Zaheer et al., 2020) approximates the full attention matrix by linearizing the softmax attention and thus can accelerate the computation by ﬁrst computing the product of keys and values.
Revised: 1) Linear approximate attention (Katharopoulos et al., 2020; Choromanski et al., 2020; Wang et al., 2020; Beltagy et al., 2020; Zaheer et al., 2020) approximates the full attention matrix by linearizing the softmax attention and thus can accelerate the computation by ﬁrst computing the product of keys and values.

Original: 2) Inducing point-based linear transformers (Lee et al., 2019;
Revised: 2) Inducing point-based linear transformers (Lee et al., 2019;

Original: Ying et al., 2018) use learned inducing points with ﬁxed size to compute attention with input tokens, thus can reduce the computation to linear complexity.
Revised: Ying et al., 2018) use learned inducing points with ﬁxed size to compute attention with input tokens, thus can reduce the computation to linear complexity.

Original: However, these linear transformers are shown to have inferior results than standard transformers in different works (Germain et al., 2021; Chen et al., 2021).
Revised: However, these linear transformers are shown to have inferior results than standard transformers in different works (Germain et al., 2021; Chen et al., 2021).

Original: 3) Sparse attention, including Longformer (Beltagy et al., 2020), Big Bird (Zaheer et al., 2020), etc, attends each query token to part of key and value tokens instead of the entire sequence.
Revised: 3) Sparse attention, including Longformer (Beltagy et al., 2020), Big Bird (Zaheer et al., 2020), etc, attends each query token to part of key and value tokens instead of the entire sequence.

Original: Unlike these works, our quadtree attention can quickly skip the irrelevant tokens according to the attention scores at coarse levels.
Revised: Unlike these works, our quadtree attention can quickly skip the irrelevant tokens according to the attention scores at coarse levels.

Original: Thus, it achieves less information loss while keeps high efﬁciency.
Revised: Thus, it achieves less information loss while keeps high efﬁciency.

Original: Vision Transformers.
Revised: Vision Transformers.

Original: Transformers have shown extraordinary performance in many vision tasks.
Revised: Transformers have shown extraordinary performance in many vision tasks.

Original: ViT (Dosovitskiy et al., 2020) applies transformers to image recognition, demonstrating the superiority of transformers for image classiﬁcation at a large scale.
Revised: ViT (Dosovitskiy et al., 2020) applies transformers to image recognition, demonstrating the superiority of transformers for image classiﬁcation at a large scale.

Original: However, due to the computational complexity of full attention, it is hard to apply transformers in dense prediction tasks, e.g. object detection, semantic segmentation, etc.
Revised: However, due to the computational complexity of full attention, it is hard to apply transformers in dense prediction tasks, e.g. object detection, semantic segmentation, etc.

Original: To address this problem, Swin-Transformer (Liu et al., 2021) restricts attention computation in a local window.
Revised: To address this problem, Swin Transformer (Liu et al., 2021) restricts attention computation in a local window.

Revised: Focal transformer (Yang et al., 2021) uses twolevel windows to increase the ability to capture long-range connection for local attention methods.

Original: Pyramid vision transformer (PVT) (Wang et al., 2021c) downsamples key and value tokens to reduce computation.
Revised: Pyramid vision transformer (PVT) (Wang et al., 2021c) reduce the computation of global attention methods by downsampling key and value tokens.

Original: Although these methods have shown improvements in various tasks, they have drawbacks either in capturing long-range dependencies (Liu et al., 2021) or ﬁne level attention (Wang et al., 2021c).
Revised: Although these methods have shown improvements in various tasks, they have drawbacks either in capturing long-range dependencies (Liu et al., 2021) or ﬁne level attention (Wang et al., 2021c).

Revised: Different from these methods, our method simultaneously capture both local and global attention by computing attention from full image levels to the ﬁnest token levels with token pyramids in one single block.

Original: The K-NN transformers (Wang et al., 2021a; Zhao et al., 2019) aggregate messages from top K most similar tokens as ours, but they compute the attention scores among all pairs of query and key tokens, and thus still has quadratic complexity.
Revised: Besides, the K-NN transformers (Wang et al., 2021a; Zhao et al., 2019) aggregate messages from top K most similar tokens as ours, but they compute the attention scores among all pairs of query and key tokens, and thus still has quadratic complexity.

Original: Beyond self-attention, many tasks can largely beneﬁt from cross attention.
Revised: Beyond self-attention, many tasks can largely beneﬁt from cross attention.

Original: Superglue (Sarlin et al., 2020) processes detected local descriptors with self- and cross attention and shows signiﬁcant improvement in feature matching.
Revised: Superglue (Sarlin et al., 2020) processes detected local descriptors with self- and cross attention and shows signiﬁcant improvement in feature matching.

Original: Standard transformers can be applied in SuperGlue because only sparse keypoints are considered.
Revised: Standard transformers can be applied in SuperGlue because only sparse keypoints are considered.

Original: SGMNet (Chen et al., 2021) further reduces the computation by attending to seeded matches.
Revised: SGMNet (Chen et al., 2021) further reduces the computation by attending to seeded matches.

Original: LoFTR (Sun et al., 2021) utilizes linear transformer (Katharopoulos et al., 2020) on low-resolution feature maps to generate dense matches.
Revised: LoFTR (Sun et al., 2021) utilizes linear transformer (Katharopoulos et al., 2020) on low-resolution feature maps to generate dense matches.

Original: For stereo matching, STTR (Li et al., 2021) applies self- and cross attention along epipolar lines and reduces the memory by gradient checkpointing engineering techniques.
Revised: For stereo matching, STTR (Li et al., 2021) applies self- and cross attention along epipolar lines and reduces the memory by gradient checkpointing engineering techniques.

Original: However, due to the requirement of processing a large number of points, these works either use linear transformers, which compromise performance, or a standard transformer, which compromises efﬁciency.
Revised: However, due to the requirement of processing a large number of points, these works either use linear transformers, which compromise performance, or a standard transformer, which compromises efﬁciency.

Original: In contrast, our transformer with quadtree attention achieves a signiﬁcant performance boost compared with linear transformer or efﬁciency improvement compared with standard transformer.
Revised: In contrast, our transformer with quadtree attention achieves a signiﬁcant performance boost compared with linear transformer or efﬁciency improvement compared with standard transformer.

Original: Besides, it can be applied to both self-attention and cross attention.
Revised: Besides, it can be applied to both self-attention and cross attention.

Original: Tree structure.

Original: Many works adopt quadtree or octree structure (Riegler et al., 2017; Wang et al., 2017; Jayaraman et al., 2018; Sullivan & Baker, 1994; Spann & Wilson, 1985; Samet, 1984) to accelerate data processing time by focusing on the interesting subsets of data.

Original: For example, in 3D object recognition tasks, Riegler et al.

Original: (2017) hierarchically partitions the 3D space using a set of unbalanced octrees to focus computation to the relevant dense regions.

Original: Jayaraman et al.

Original: (2018) decomposes and represents the image as a linear quadtree that is only reﬁned in the non-empty portions of the image.

Original: Following the spirit of these works, our quadtree attention quickly skips irrelevant regions in the ﬁne level according to the results in the coarse level.

Original:  M ETHOD
Revised:  M ETHOD

Original: We ﬁrst brieﬂy review the attention mechanism in transformers in Section 3.1 and then formulate our quadtree attention in Section 3.2.
Revised: We ﬁrst brieﬂy review the attention mechanism in transformers in Section 3.1 and then formulate our quadtree attention in Section 3.2.

Original: 3. A TTENTION IN T RANSFORMER
Revised: 3. A TTENTION IN T RANSFORMER

Original: Vision transformers have shown great success in many tasks.
Revised: Vision transformers have shown great success in many tasks.

Original: At the heart of a transformer is the attention module, which can capture long-range information between feature embeddings.
Revised: At the heart of a transformer is the attention module, which can capture long-range information between feature embeddings.

Original: Given two image embeddings X 1 and X 2 , the attention module passes information between them.
Revised: Given two image embeddings X 1 and X 2 , the attention module passes information between them.

Original: Selfattention is the case when X 1 and X 2 are the same, while cross attention covers a more general situation when X 1 and X 2 are different.
Revised: Selfattention is the case when X 1 and X 2 are the same, while cross attention covers a more general situation when X 1 and X 2 are different.

Original: It ﬁrst generates the query Q , key K , and value V by the levelq
Revised: It ﬁrst generates the query Q , key K , and value V by the [Figure]

Original:  i
Revised: q i

Original: Q UADTREE A TTENTION FOR V ISION T RANSFORMER
Revised: Q UADTREE A TTENTION FOR V ISION T RANSFORMER

Original: Anonymous authors Paper under double-blind review
Revised: Anonymous authors Paper under double-blind review

Original: Q
Revised: Q

Original: K ⌦ li ⌦ l +1 i ⌦ l +2 i m li m l +

Revised: K ⌦ li ⌦ l +1 i ⌦ l +2 i m li m l +

Original: i m l +

Revised: i m l +

Original: i  li
Revised: i  li

Original:  l +1 i
Revised:  l +1 i

Original:  l +

Revised:  l +

Original: i
Revised: i

Original: R EFERENCES
Revised: R EFERENCES

Original: Under review as a conference paper at ICLR
Revised: Under review as a conference paper at ICLR

Original: Q UADTREE A TTENTION FOR V ISION T RANSFORMER
Revised: Q UADTREE A TTENTION FOR V ISION T RANSFORMER

Original: Anonymous authors Paper under double-blind review
Revised: Anonymous authors Paper under double-blind review

Original: Q
Revised: Q

Original: K \ V ⌦ li ⌦ l +1 i ⌦ l +2 i m li m l +

Revised: K \ V ⌦ li ⌦ l +1 i ⌦ l +2 i m li m l +

Original: i m l +

Revised: i m l +

Original: i  li
Revised: i  li

Original:  l +1 i
Revised:  l +1 i

Original:  l +

Revised:  l +

Original: i ⇤ li
Revised: i ⇤ li

Original: ⇤ l
Revised: ⇤ l

Original: +

Revised: +

Original: i
Revised: i

Original: ⇤ l +

Revised: ⇤ l +

Original: i
Revised: i

Original: R EFERENCES level 2 level
Revised: R EFERENCES [Figure]

Original: QuadTree-A
Revised: QuadTree-A

Original: Under review as  conference 
Revised: Under review as a conference paper at ICLR

Original: Q UADTREE A TTENTION FOR V ISION T RANSFORMER
Revised: Q UADTREE A TTENTION FOR V ISION T RANSFORMER

Original: Anonymous authors Paper under double-blind review
Revised: Anonymous authors Paper under double-blind review

Original: Q
Revised: Q

Original: K \ V ⌦ 1 i ⌦ 2 i ⌦ 

Revised: K \ V ⌦ 1 i ⌦ 2 i ⌦

Original: i m 1 i m 2 i m 3 i ⇤ l i ⇤ l
Revised: i m 1 i m 2 i m 3 i ⇤ li ⇤ l

Original: +

Revised: +

Original: i
Revised: i

Original: ⇤ l +

Revised: ⇤ l +

Original: i
Revised: i

Original: R EFERENCES
Revised: R EFERENCES

Original: Under review as a conference paper at ICLR
Revised: QuadTree-B

Original: Q UADTREE A TTENTION FOR V ISION T RANSFORMER
Revised: Q UADTREE A TTENTION FOR V ISION T RANSFORMER

Original: Anonymous authors Paper under double-blind review
Revised: Anonymous authors Paper under double-blind review

Original: Q
Revised: Q

Original: K \ V ⌦ 1 i ⌦ 2 i ⌦ 

Revised: K \ V ⌦ 1 i ⌦ 2 i ⌦

Original: i m 1 i m 2 i m [-3 i ⇤ li ⇤ l +
-]{+
+}
Revised: i m 1 i m 2 i m [-3 i ⇤ li ⇤ l +
-]{+
+}

Original: i
Revised: i

Original: [-⇤ l +
-]
Revised: [-⇤ l +
-]

Original:  i R EFERENCES{+ 2 i  
+}
Revised:  1 i {+ 2 i  
+}

Original: Under review as a conference paper at ICLR Q UADTREE A TTENTION FOR V ISION T RANSFORMER Anonymous authors Paper under double-blind review
Revised: i ⇤ li

Original: [-Q K \ V ⌦ 1 i ⌦ 2 i ⌦ 
 i m 1 i m 2 i m 3 i ⇤ li-] ⇤ l
Revised: [-Q K \ V ⌦ 1 i ⌦ 2 i ⌦ 
 i m 1 i m 2 i m 3 i ⇤ li-] ⇤ l

Original: +

Revised: +

Original: i
Revised: i

Original: ⇤ l +

Revised: ⇤ l +

Original: i
Revised: i

Original: R EFERENCES
Revised: R EFERENCES

Revised: [Figure]

Original: Under review as a conference paper at ICLR  Q UADTREE A TTENTION FOR V ISION T RANSFORMER Anonymous authors
Revised: Under review as a conference paper at ICLR 2022 Q UADTREE A TTENTION FOR V ISION T RANSFORMER Anonymous authors

Original: Paper under double-blind review

Original: Q
Revised: Q

Original: K \ V[-⌦ 1 i ⌦ 2 i ⌦ 
 i m 1 i m 2 i m 3 i ⇤ li ⇤ l +
 i-]
Revised: K \ V[-⌦ 1 i ⌦ 2 i ⌦ 
 i m 1 i m 2 i m 3 i ⇤ li ⇤ l +
 i-]

Original: [-⇤ l +
-]
Revised: [-⇤ l +
-]

Original: [-i R EFERENCES Under review as a conference paper at ICLR 2022 Q UADTREE A TTENTION FOR V ISION T RANSFORMER Anonymous authors Paper under double-blind review Q ⌦ 1 i ⌦ 2 i ⌦ 
-]
Revised: [-i R EFERENCES Under review as a conference paper at ICLR 2022 Q UADTREE A TTENTION FOR V ISION T RANSFORMER Anonymous authors Paper under double-blind review Q ⌦ 1 i ⌦ 2 i ⌦ 
-][Figure]

Original: i m 1 i m 2 i m 3 i ⇤ li ⇤ l

Original: [-+
 i ⇤ l +
 i-]
Revised: [-+
 i ⇤ l +
 i-]⌦

Original: R EFERENCES

Original: [Figure]
Revised: [Figure]

Original: Q UADTREE A TTENTION FOR V ISION T RANSFORMER Anonymous authors Paper under double-blind review Q K \ V ⌦
Revised: m

Original: [Figure]
Revised: [Figure]

Revised: m

Original: i
Revised: [Figure] m [Figure]

Original: [Figure]
Revised: [Figure]

Original: i ⌦
Revised:  [Figure]

Original: [Figure]
Revised: [Figure]

Revised: ⇤ li

Revised: ⇤ l

Original: {++
+} i[-m 
-]
Revised: {++
+} i[-m 
-]

Original: {+⇤ l +2 i level 1 level 2 levelm 1 i m 
+}
Revised: {+⇤ l +2 i level 1 level 2 levelm 1 i m 
+}

Original: i m 2 i[-m 
-]
Revised: i m 3 i[-m 
-]

Original: i  1 i  2 i 
Revised: R EFERENCES Under review as a conference paper at ICLR 2022 Q UADTREE A TTENTION FOR V ISION T RANSFORMER [Figure] Under review as a conference paper at ICLR 2022 Q UADTREE A TTENTION FOR V ISION T RANSFORMER

Original: [Figure]
Revised: [Figure]

Original: {+Under review as a conference paper at ICLR 2022 Q UADTREE A TTENTION FOR V ISION T RANSFORMER Anonymous authors Paper under double-blind review Q K \ V ⌦ 1 i ⌦ 2 i ⌦ 
+}
Revised: {+Under review as a conference paper at ICLR 2022 Q UADTREE A TTENTION FOR V ISION T RANSFORMER Anonymous authors Paper under double-blind review Q K \ V ⌦ 1 i ⌦ 2 i ⌦ 
+}

Original: {+i m 1 i m 2 i m 
+}
Revised: {+i m 1 i m 2 i m 
+}

Original: i
Revised: i

Original: [Figure]{+ 1 i  2 i  
+}
Revised: {+ 1 i  2 i  
+}

Original: l + [Figure]
Revised: i ⇤ li

Original: ⇤ l
Revised: ⇤ l

Original: [Figure]
Revised: +1 i

Revised: ⇤

Original: {++
+} iR
Revised: {++
+} i

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: following equation,
Revised: following equation,

Original: [Equation]
Revised: [Equation]

Original: where W q , W k and W v are learnable parameters.
Revised: where W q , W k and W v are learnable parameters.

Original: Then, it performs message aggregation by computing the attention scores between query and key as following,
Revised: Then, it performs message aggregation by computing the attention scores between query and key as following,

Original: [Equation]
Revised: [Equation]

Original: where C is the embedding channel dimension.
Revised: where C is the embedding channel dimension.

Original: The above process has O ( N 2 ) computational complexity, where N is the number of image patches in a vision transformer.
Revised: The above process has O ( N 2 ) computational complexity, where N is the number of image patches in a vision transformer.

Original: This quadratic complexity hinders transformers from being applied to tasks requiring high resolution output.
Revised: This quadratic complexity hinders transformers from being applied to tasks requiring high resolution output.

Original: To address this problem, PVT (Wang et al.
Revised: To address this problem, PVT (Wang et al.

Original: (2021c))
Revised: (2021c))

Original: downsamples K and V , while Swin-Transformer (Liu et al.
Revised: downsamples K and V , while Swin Transformer (Liu et al.

Original: (2021)) limits the attention computation within local windows.
Revised: (2021)) limits the attention computation within local windows.

Original: 3. Q UAD T REE A TTENTION
Revised: 3. Q UAD T REE A TTENTION

Original: In order to reduce the computational cost of vision transformers, we present QuadTree Attention.
Revised: In order to reduce the computational cost of vision transformers, we present QuadTree Attention.

Original: As the name implies, we borrow the idea from quadtrees, which are often used to partition a twodimensional space by recursively subdividing it into four quadrants or regions.
Revised: As the name implies, we borrow the idea from quadtrees, which are often used to partition a twodimensional space by recursively subdividing it into four quadrants or regions.

Original: Quadtree attention computes attention in a coarse to ﬁne manner.
Revised: Quadtree attention computes attention in a coarse to ﬁne manner.

Original: According to the results at the coarse level, irrelevant image regions are skipped quickly at the ﬁne level.
Revised: According to the results at the coarse level, irrelevant image regions are skipped quickly at the ﬁne level.

Original: This design achieves less information loss while keeping high efﬁciency.
Revised: This design achieves less information loss while keeping high efﬁciency.

Original: The same as the regular transformers, we ﬁrst linearly project X 1 and X 2 to the query, key, and value tokens.
Revised: The same as the regular transformers, we ﬁrst linearly project X 1 and X 2 to the query, key, and value tokens.

Original: To facilitate fast attention computation, we construct L -level pyramids for query Q , key K , and value V tokensrespectively by 2 × 2 average pooling.
Revised: To facilitate fast attention computation, we construct L -level pyramids for query Q , key K , and value V tokens by downsampling feature maps. For query and key tokens, we use average pooling layers.

Revised: For value tokens, average pooling is used for cross attention tasks and convolutionalnormalization-activation layers with stride 2 are used for self attention tasks if no special statement.

Original: As shown in Figure 1, after computing attention scores in the coarse level, for each query token, we select the top K key tokens with the highest attention scores.
Revised: As shown in Figure 1, after computing attention scores in the coarse level, for each query token, we select the top K key tokens with the highest attention scores.

Original: At the ﬁne level, query sub-tokens only need to be evaluated with those key sub-tokens that correspond to one of the selected K key tokens at the coarse level.
Revised: At the ﬁne level, query sub-tokens only need to be evaluated with those key sub-tokens that correspond to one of the selected K key tokens at the coarse level.

Original: This process is repeated until the ﬁnest level.
Revised: This process is repeated until the ﬁnest level.

Original: After computing the attention scores, we aggregate messages at all levels, where we design two architectures named as QuadTree-A and QuadTree-B .
Revised: After computing the attention scores, we aggregate messages at all levels, where we design two architectures named as QuadTree-A and QuadTree-B .

Original: QuadTree-A .
Revised: QuadTree-A .

Original: Considering the i -th query token q i at the ﬁnest level, we need to compute its received message m i from all key tokens.
Revised: Considering the i -th query token q i at the ﬁnest level, we need to compute its received message m i from all key tokens.

Original: This design assembles the full message by collecting partial messages from different pyramid levels.
Revised: This design assembles the full message by collecting partial messages from different pyramid levels.

Original: Speciﬁcally,
Revised: Speciﬁcally,

Original: [Equation]
Revised: [Equation]

Original: where m li indicates the partial message evaluated at level l .
Revised: where m li indicates the partial message evaluated at level l .

Original: This partial message m li assemble messages at the l -th level from tokens within the region Ω li , which will be deﬁned later.
Revised: This partial message m li assemble messages at the l -th level from tokens within the region Ω li , which will be deﬁned later.

Original: In this way, messages from less related regions are computed from coarse levels, while messages from highly related regions are computed in ﬁne levels.
Revised: In this way, messages from less related regions are computed from coarse levels, while messages from highly related regions are computed in ﬁne levels.

Original: This scheme is illustrated in Figure 2 (b), message m i is generated by assembling three partial messages that are computed from different image regions with different colors, which collectively cover the entire image space.
Revised: This scheme is illustrated in Figure 2 (b), message m i is generated by assembling three partial messages that are computed from different image regions with different colors, which collectively cover the entire image space.

Original: The green region indicates the most relevant region and is evaluated at the ﬁnest level, while the red region is the most irrelevant region and is evaluated at the coarsest level.
Revised: The green region indicates the most relevant region and is evaluated at the ﬁnest level, while the red region is the most irrelevant region and is evaluated at the coarsest level.

Original: The region Ω li can be deﬁned as Γ li − Γ l +1 i , where the image region Γ li corresponds to the top K tokens at the level l − 1 .
Revised: The region Ω li can be deﬁned as Γ li − Γ l +1 i , where the image region Γ li corresponds to the top K tokens at the level l − 1 .

Original: The regions Γ li are illustrated in
Revised: The regions Γ li are illustrated in

Original: Figure 2 (c).
Revised: Figure 2 (c).

Original: The region Γ 1 i covers the entire image.
Revised: The region Γ 1 i covers the entire image.

Original: The partial messages are computed as,
Revised: The partial messages are computed as,

Original: [Equation]
Revised: [Equation]

Original: where s lij is the attention score between the query and key tokens at level l .
Revised: where s lij is the attention score between the query and key tokens at level l .

Original: Figure 2 (a) highlights query and key tokens involved in computing m li with the same color as Ω li .
Revised: Figure 2 (a) highlights query and key tokens involved in computing m li with the same color as Ω li .

Original: Attention scores are computed recursively,
Revised: Attention scores are computed recursively,

Original: [Equation]
Revised: [Equation]

Original: Here, s l − 1 ij is the score of corresponding parent query and key tokens and s 1 ij = 1 .
Revised: Here, s l − 1 ij is the score of corresponding parent query and key tokens and s 1 ij = 1 .

Original: The tentative attention score t lij is evaluated according to Equation 1 among the 2 × 2 tokens of the same parent query token.
Revised: The tentative attention score t lij is evaluated according to Equation 1 among the 2 × 2 tokens of the same parent query token.

Revised: For QuadTree-A, we use average pooling layers to downsample all query, key and value tokens.

Original: QuadTree-B .
Revised: QuadTree-B .

Original: The attention scores s lij in QuadTree-A are recursively computed from all levels, which makes scores smaller at ﬁner levels and reduces the contributions of ﬁne image features.
Revised: The attention scores s l ij in QuadTree-A are recursively computed from all levels, which makes scores smaller at ﬁner levels and reduces the contributions of ﬁne image features.

Original: Besides, ﬁne level scores are also largely affected by the inaccuracy at coarse levels.
Revised: Besides, ﬁne level scores are also largely affected by the inaccuracy at coarse levels.

Original: So we design a different scheme, referred as QuadTree-B in this paper, to address this problem.
Revised: So we design a different scheme, referred as QuadTree-B in this paper, to address this problem.

Original: Speciﬁcally, we compute m i as a weighted average of the partial messages from different levels,
Revised: Speciﬁcally, we compute m i as a weighted average of the partial messages from different levels,

Original: [Equation]
Revised: [Equation]

Original: where w li is a learned weight.
Revised: where w li is a learned weight.

Original: As shown in Figure 2 (c), the partial messages here overlap with each other, which are computed as,
Revised: As shown in Figure 2 (c), the partial messages here overlap with each other, which are computed as,

Original: [Equation]
Revised: [Equation]

Original: where Attention is the attention message computation as Equation 1.
Revised: where Attention is the attention message computation as Equation 1.

Original: Here, K l Γ li and V l Γ li are matrices formed by stacking all keys and values within the region Γ li .
Revised: Here, K l Γ li and V l Γ li are matrices formed by stacking all keys and values within the region Γ li .

Original: Both QuadTree-A and QuadTree-B involve only sparse attention evaluation.
Revised: Both QuadTree-A and QuadTree-B involve only sparse attention evaluation.

Original: Thus, our method largely reduces computational complexity.
Revised: Thus, our method largely reduces computational complexity.

Original: As analyzed in Appendix A.1, the computational complexity of our quadtree attention is linear to the number of tokens.
Revised: As analyzed in Appendix A.1, the computational complexity of our quadtree attention is linear to the number of tokens.

Original: [Table]
Revised: [Table]

Revised: Multiscale position encoding .

Revised: The computation of attention is permutation invariant to tokens, and thus positional information is missed.

Revised: To address this problem, we adopt the locally-enhanced positional encoding (LePE)

Revised: (Dong et al., 2021) at each level to design a multiscale position encoding.

Revised: Speciﬁcally, for level l , we apply unshared depth-wise convolution layers to value tokens V l to encode the positional information.

Original:  E XPERIMENT
Revised:  E XPERIMENT

Original: We experiment our quadtree transformer with four representative tasks, including feature matching, stereo, image classiﬁcation [Table] and object detection.
Revised: We experiment our quadtree transformer with four representative tasks, including feature matching, stereo, image classiﬁcation, and object detection.

Original: The ﬁrst two tasks require cross attention to fuse information across different images, while the latter two involve only self-attention.
Revised: The ﬁrst two tasks require cross attention to fuse information across different images, while the latter two involve only self-attention.

Original: We implement our quadtree transformer using PyTorch and CUDA kernels.
Revised: We implement our quadtree transformer using PyTorch and CUDA kernels.

Original: More implementation details are provided in Appendix B.
Revised: More implementation details are provided in Appendix B.

Original: 4. C ROSS ATTENTION TASKS 4.1. F EATURE MATCHING
Revised: 4. C ROSS ATTENTION TASKS 4.1. F EATURE MATCHING

Original: Finding feature correspondence
Revised: Finding feature correspondence

Original: (Luo et al., 2019; DeTone et al., 2018) across different images is a precedent problem for many 3D computer vision tasks.
Revised: (Luo et al., 2019; DeTone et al., 2018) across different images is a precedent problem for many 3D computer vision tasks.

Original: It is typically evaluated by the accuracy of the camera pose estimated from the corresponding points.
Revised: It is typically evaluated by the accuracy of the camera pose estimated from the corresponding points.

Original: We follow the framework proposed in a recent state-of-the-art work LoFTR
Revised: We follow the framework proposed in a recent state-of-the-art work LoFTR

Original: (Sun et al., 2021), which consists of a CNN-based feature extractor and a transformer-based matcher.
Revised: (Sun et al., 2021), which consists of a CNN-based feature extractor and a transformer-based matcher.

Original: We replace the linear transformer (Katharopoulos et al., 2020) in LoFTR with our quadtree transformer.
Revised: We replace the linear transformer (Katharopoulos et al., 2020) in LoFTR with our quadtree transformer.

Original: Besides, we also implement a new version of LoFTR with the spatial reduction (SR) attention (Wang et al., 2021c) for additional comparison.
Revised: Besides, we also implement a new version of LoFTR with the spatial reduction (SR) attention (Wang et al., 2021c) for additional comparison.

Original: Setting.
Revised: Setting.

Original: We experiment on ScanNet (Dai et al., 2017) with 1,513 scans.
Revised: We experiment on ScanNet (Dai et al., 2017) with 1,513 scans.

Original: In order to accelerate training, we design the LoFTR-lite setting, which uses half of the feature channels of LoFTR andtraining scans.
Revised: In order to accelerate training, we design the LoFTR-lite setting, which uses half of the feature channels of LoFTR andtraining scans.

Original: Ablation studies in section 4.3 are conducted in this setting.
Revised: Ablation studies in section 4.3 are conducted in this setting.

Original: We train both LoFTRlite and LoFTR for 30 epochs with batch size 8.
Revised: We train both LoFTRlite and LoFTR for 30 epochs with batch size 8.

Original: For quadtree transformer, we build pyramids of three levels with the coarsest resolution at 15 × 20 pixels.
Revised: For quadtree transformer, we build pyramids of three levels with the coarsest resolution at 15 × 20 pixels.

Original: We set the parameter K to 8 at the ﬁnest level, and double it at coarser levels.
Revised: We set the parameter K to 8 at the ﬁnest level, and double it at coarser levels.

Original: For the SR attention, we average pool the value and key tokens to the size 8 × 8 to keep similar memory usage and ﬂops as our quadtree attention.
Revised: For the SR attention, we average pool the value and key tokens to the size 8 × 8 to keep similar memory usage and ﬂops as our quadtree attention.

Original: More details are included in Appendix B.1.
Revised: More details are included in Appendix B.1.

Original: Results.
Revised: Results.

Original: Table 1 shows the AUC of camera pose errors 1 under (5 ◦ , 10 ◦ , 20 ◦ ).
Revised: Table 1 shows the AUC of camera pose errors 1 under (5 ◦ , 10 ◦ , 20 ◦ ).

Original: We can see that the SR attention achieves similar results with linear transformer.
Revised: We can see that the SR attention achieves similar results with linear transformer.

Original: In comparison, both QuadTree-A  and QuadTree-B outperform linear transformer and SR attention by a large margin.
Revised: In comparison, both QuadTree-A [Table] and QuadTree-B outperform linear transformer and SR attention by a large margin.

Original: Quadtree-B generally performs better than Quadtree-A. Quadtree-B has 2.6 and 1.9 improvements in terms of
Revised: Quadtree-B generally performs better than Quadtree-A. Quadtree-B has 2.6 and 1.9 improvements in terms of

Original: AUC@20 ◦ over linear transformer on LoFTR-lite and LoFTR respectively.
Revised: AUC@20 ◦ over linear transformer on LoFTR-lite and LoFTR respectively.

Original: To further enhance the results, we train a model with K = 16 and initialize the backbone ResNet-18 with the weights published in (Sun et al., 2021).
Revised: To further enhance the results, we train a model with K = 16 and initialize the backbone ResNet-18 with the weights published in (Sun et al., 2021).

Original: This model achieves 2.7 improvements on AUC@20 ◦ over (Sun et al., 2021), where the LoFTR model is trained with a batch size of 64 with 64 GPUs, a more preferable setting leading to slightly better results than our linear transformer implementation shown in Table 1.[Table]
Revised: This model achieves 2.7 improvements on AUC@20 ◦ over (Sun et al., 2021), where the LoFTR model is trained with a batch size of 64 with 64 GPUs, a more preferable setting leading to slightly better results than our linear transformer implementation shown in Table 1.

Original: 4.1. S TEREO M ATCHING
Revised: 4.1. S TEREO M ATCHING

Original: Stereo matching aims to ﬁnd corresponding pixels on epipolar lines between two rectiﬁed images.
Revised: Stereo matching aims to ﬁnd corresponding pixels on epipolar lines between two rectiﬁed images.

Original: The recent work STTR (Li et al., 2021) applies transformers to feature points between epipolar lines and achieves state-of-the-art performance.
Revised: The recent work STTR (Li et al., 2021) applies transformers to feature points between epipolar lines and achieves state-of-the-art performance.

Original: Note here, both self- and cross attention are applied along epipolar lines, pixels across different lines are not considered in the attention computation.
Revised: Note here, both self- and cross attention are applied along epipolar lines, pixels across different lines are not considered in the attention computation.

Original: We replace the standard transformer in STTR (Li et al., 2021) with our quadtree transformer.
Revised: We replace the standard transformer in STTR (Li et al., 2021) with our quadtree transformer.

Original: Setting.
Revised: Setting.

Original: We experiment on the Scene Flow FlyingThings3D (Mayer et al., 2016) synthetic dataset, which contains 25,466 images with a resolution of 960 × 540 .
Revised: We experiment on the Scene Flow FlyingThings3D (Mayer et al., 2016) synthetic dataset, which contains 25,466 images with a resolution of 960 × 540 .

Original: We build pyramids of four levels to evaluate quadtree attention.
Revised: We build pyramids of four levels to evaluate quadtree attention.

Original: While the STTR is applied to features of 1/3 of image resolution, we use feature maps of 1/2 of image resolution.
Revised: While the STTR is applied to features of 1/3 of image resolution, we use feature maps of 1/2 of image resolution.

Original: More details about the network are included in
Revised: More details about the network are included in

Original: Appendix B.2.
Revised: Appendix B.2.

Original: Results.
Revised: Results.

Original: We report EPE (End-Point-Error) in non-occluded regions and IOU (Intersection-overUnion) for occlusion estimation in Table 2 as (Li et al., 2021).
Revised: We report EPE (End-Point-Error) in non-occluded regions and IOU (Intersection-overUnion) for occlusion estimation in Table 2 as (Li et al., 2021).

Original: Computational complexity and memory usage are also reported.
Revised: Computational complexity and memory usage are also reported.

Original: Compared with STTR based on the standard transformer, our quadtree transformer achieves similar EPE (0.45 px vs 0.46 px) and higher IOU for occlusion estimation, but with much lower computational and memory costs, with only 52% FLOPs and 63% memory consumption.
Revised: Compared with STTR based on the standard transformer, our quadtree transformer achieves similar EPE (0.45 px vs 0.46 px) and higher IOU for occlusion estimation, but with much lower computational and memory costs, with only 52% FLOPs and 63% memory consumption.

Original: 4. S ELF - ATTENTION TASK
Revised: 4. S ELF - ATTENTION TASK

Original: This section presents results on image classiﬁcation and object detection.
Revised: This section presents results on image classiﬁcation and object detection.

Original: In the past, convolutional neural networks (CNNs) have dominated these tasks for a long time.
Revised: In the past, convolutional neural networks (CNNs) have dominated these tasks for a long time.

Original: Recently, vision transformers (Dosovitskiy et al., 2020; Liu et al., 2021; Wang et al., 2021c) show excellent potential on these problems, thanks to their capability in capturing long-range interactions.
Revised: Recently, vision transformers (Dosovitskiy et al., 2020; Liu et al., 2021; Wang et al., 2021c) show excellent potential on these problems, thanks to their capability in capturing long-range interactions.

Original: To compare our method with these vision transformers on image classiﬁcation, we use the public codes of PVTv2 (Wang et al., 2021c) and replace all the spatial reduction attention with our quadtree attention.
Revised: To compare our method with these vision transformers on image classiﬁcation, we use the public codes of PVTv2 (Wang et al., 2021c) and replace all the spatial reduction attention with our quadtree attention.

Original: For object detection, we further apply a representative object detection framework, RetinaNet (Lin et al., 2017), which is a widely used single-stage object detector.
Revised: For object detection, we further apply a representative object detection framework, RetinaNet (Lin et al., 2017), which is a widely used single-stage object detector.

Original: 4.2. I MAGE CLASSIFICATION
Revised: 4.2. I MAGE CLASSIFICATION

Original: Settings.
Revised: Settings.

Original: We evaluate image classiﬁcation on the ImageNet-1K dataset (Deng et al., 2009), which consists of 1.28M training images and 50K validation images from 1,000 categories.
Revised: We evaluate image classiﬁcation on the ImageNet-1K dataset (Deng et al., 2009), which consists of 1.28M training images and 50K validation images from 1,000 categories.

Original: We build token pyramids with the coarsest level at a resolution of 7 × 7 and set K = 8 .
Revised: We build token pyramids with the coarsest level at a resolution of 7 × 7 and set K = 8 .

Original: We crop and resize the input images to 224 × 224 pixels and train the model with a mini-batch of 128.
Revised: We crop and resize the input images to 224 × 224 pixels and train the model with a mini-batch of 128.

Original: All models are trained for 300 epochs from scratch on 8 GPUs.
Revised: All models are trained for 300 epochs from scratch on 8 GPUs.

Original: All the other training settings are the same as in (Wang et al., 2021c).
Revised: All the other training settings are the same as in (Wang et al., 2021c).

Original: We build three different quadtree transformers at different complexity, named as b0, b1, b2.
Revised: We build three different quadtree transformers at different complexity, named as b0, [Table] DeiT [Table]

Revised: CoaT

Original: These models are gradually deeper and wider.
Revised: [Table] Lite Small ( [Table] PVTv

Revised: [Table]

Revised: [Table]

Revised: b1, b2.

Revised: These models are gradually deeper and wider.

Original: There are 8 transformer blocks in b0 and b1, while there are 16 transformer blocks in b2.
Revised: There are 8 transformer blocks in b0 and b1, while there are 16 transformer blocks in b2.

Original: Furthermore, the number of feature channels in b1 is two times of that in b0.
Revised: Furthermore, the number of feature channels in b1 is two times of that in b0.

Original: More conﬁguration details can be found in Appendix.
Revised: More conﬁguration details can be found in Appendix.

Original: B.3.
Revised: B.3.

Original: Results.
Revised: Results.

Original: We provide the top-1 accuracy of various methods and network settings in Table 3.
Revised: We provide the top-1 accuracy of various methods and network settings in Table 3.

Original: These results are grouped into three sections, each with several methods of similar network complexity as indicated by the number of parameters.
Revised: These results are grouped into three sections, each with several methods of similar network complexity as indicated by the number of parameters.

Original: As shown in Table 3, QuadTree-B outperforms PVTv by 0.5%-0.9% in top-1 accuracy with fewer parameters.
Revised: As shown in Table 3, QuadTree-B outperforms PVTv by 0.5%-0.9% in top-1 accuracy with fewer parameters.

Original: Swin-Transformer-T adopts local attention and is surpassed by our QuadTree-B-b2 by 1.3% in top-1 accuracy, although it has 5.8M more parameters and 0.2 more ﬂops.
Revised: Swin Transformer-T adopts local attention and is surpassed by our QuadTree-B-b2 by 1.3% in top-1 accuracy, although it has 5.8M more parameters and 0.2 more ﬂops.

Original: This result proves that global information is important.
Revised: This result proves that global information is important.

Original: In general,

Original: [Table]

Original: DeiT

Original: [Table]

Original: CoaT

Original: [Table]

Original: Lite Small (

Original: [Table]

Original: PVTv [Table] [Table] our quadtree transformer leverages both global information at the coarse level and local information at ﬁne levels, and outperforms both PVTv2 and Swin-transformer.
Revised: In general, our quadtree transformer leverages both global information at the coarse level and local information at ﬁne levels, and outperforms both PVTv2 and Swin Transformer

Original: 4.2. O BJECT DETECTION
Revised: 4.2. O BJECT DETECTION

Original: Settings.
Revised: Settings.

Original: We experiment on the COCO dataset.
Revised: We experiment on the COCO dataset.

Original: All models are trained on COCO train2017 (118k images) and evaluated on val2017 (5k images).
Revised: All models are trained on COCO train2017 (118k images) and evaluated on val2017 (5k images).

Original: We initialize the quadtree backbone with the weights pre-trained on ImageNet.
Revised: We initialize the quadtree backbone with the weights pre-trained on ImageNet.

Original: We adopt the same setting as PVTv2, training the model with a batch size of 16 and AdamW optimizer with an initial learning rate of 1 × 10 − 4 for 12 epochs.
Revised: We adopt the same setting as PVTv2, training the model with a batch size [Table] Liu [Table]

Revised: [Table]

Original: We use the standard metric average precision to evaluate our method.
Revised: Focal Attention (Yang et al., 2021) [Table] [Table] [Table]

Original: Results.
Revised: Results.

Original: We mainly compare our method with PVTv2, ResNet (He et al., 2016), and ResNeXt (Xie et al., 2017), which are state-of-the-art backbones for dense prediction.
Revised: We mainly compare our method with PVTv2, ResNet (He et al., 2016), and ResNeXt (Xie et al., 2017), which are state-of-the-art backbones for dense prediction.

Original: Table 4 lists the average precision of different methods and their backbone ﬂops for images of resolution of 800 × 1 , 333 .
Revised: Table 4 lists the average precision of different methods and their backbone ﬂops for images of resolution of 800 × 1 , 333 .

Original: Beneﬁting from the coarse to ﬁne mechanism, a small K is enough for our method.
Revised: Beneﬁting from the coarse to ﬁne mechanism, a small K is enough for our method.

Original: Thus, the computation can be reduced when using high resolution images.
Revised: Thus, the computation can be reduced when using high resolution images.

Original: We can see that QuadTree-B achieves slightly higher performance, but with much fewer ﬂops than PVTv2.
Revised: We can see that QuadTree-B achieves slightly higher performance, but with much fewer ﬂops than PVTv2.

Original: Our quadtree transformer also outperforms ResNet and ResNeXt.
Revised: Our quadtree transformer also outperforms ResNet and ResNeXt.

Original: For example, QuadTree-B-b2 outperform ResNet101 and
Revised: For example, QuadTree-B-b2 outperform ResNet101 and

Original: ResNeXt101-32x4d by 6.4 AP and 5.0 AP respectively with about 40% backbone ﬂops reduction.[Table] [Table] [Figure]
Revised: ResNeXt101-32x4d by 6.4 AP and 5.0 AP respectively with about 40% backbone ﬂops reduction.

Original: 4. A NALYSIS
Revised: 4. A NALYSIS

Original: Top K numbers.
Revised: Top K numbers.

Original: Table 5 and Table 6 shows the performance of QuadTree-B architecture with different value of K for object detection and feature matching respectively.
Revised: Table 6 and Table 7 shows the performance of QuadTree-B architecture with different value of K for object detection and feature matching respectively.

Original: The performance is improved when K becomes larger and saturates quickly.
Revised: The performance is improved when K becomes larger and saturates quickly.

Original: This indicates only a few tokens with high attention scores should be subdivided in the next level for computing attentions.
Revised: This indicates only a few tokens with high attention scores should be subdivided in the next level for computing attentions.

Original: QuadTree-A vs QuadTree-B .
Revised: Comparison with other attention mechanisms.

Original: QuadTree-B architecture consistently outperforms QuadTree-A in feature matching, image classiﬁcation, and detection experiments. We analyze its reason as shown in Figure 3, where (d) and (e) show the attention score maps of QuadTree-A and QuadTree-B at different levels for the same point in the query image shown in (a).
Revised: For a fair comparison with other attention mechanisms, we test these attention mechanisms under the same backbone and training settings.

Original: It is clear that the QuadTree-B has more accurate score maps, and is less affected by the inaccuracy in coarse level score estimation.
Revised: Speciﬁcally, we replace the original attention module in PVTv2-B0 with the attention method used in Swin

Original:  We further visualize the attention scores of spatial reduction (SR) attention (Wang et al., 2021c) and linear transformer (Katharopoulos et al., 2020) in (b) and (c).
Revised: Transformer and Focal Transformer. We also augment PVTv2 with the same positional encoding (Dong et al., 2021) for a fair comparison. As shown in Table.

Original: We can see that SR attention  and linear transformer attend the query token on large unrelated regions due to the loss of ﬁne-grained information.
Revised: 5, QuadTree attention obtain consistently better performance than Swin and PVTv2 in both classiﬁcation task and detection task. Compared with focal attention, our method gets 1.0 higher results in object detection, which might be because that QuadTree attention can always cover the whole images, while Focal attention only covers 1 / 6 of the image in the ﬁrst stage. For cross attention tasks, we provide visualization of attention score as shown in Fig.5 in Appendix E.

Original: In contrast, our quadtree transformer focus on the most relevant area.
Revised: Our method can attend to much more related regions than PVT (Wang et al., 2021b) and Linear attention (Katharopoulos et al., 2020).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We introduce QuadTree Attention to reduce the computational complexity of vision transformers from quadratic to linear.
Revised: We introduce QuadTree Attention to reduce the computational complexity of vision transformers from quadratic to linear.

Original: Quadtree transformers build token pyramids and compute attention in a coarse-to-ﬁne manner.
Revised: Quadtree transformers build token pyramids and compute attention in a coarse-to-ﬁne manner.

Original: At each level, top K regions with the highest attention scores are selected, such that in the next level, computation in irrelevant regions can be quickly skipped.
Revised: At each level, top K regions with the highest attention scores are selected, such that in ﬁner level, computation in irrelevant regions can be quickly skipped.

Original: Quadtree attention can be applied to cross attention as well as self-attention.
Revised: Quadtree attention can be applied to cross attention as well as self-attention.

Original: It achieves state-of-the-art performance in various tasks including feature matching, stereo, image classiﬁcation, and object detection.
Revised: It achieves state-of-the-art performance in various tasks including feature matching, stereo, image classiﬁcation, and object detection.

Original: A A PPENDIX A. C OMPLEXITY ANALYSIS
Revised: A A PPENDIX A. C OMPLEXITY ANALYSIS

Original: In this section we analyze the computational complexity of quadtree attention.
Revised: In this section, we analyze the computational complexity of quadtree attention.

Original: Suppose the lengths of the query tokens, key tokens and value tokens are all H × W .
Revised: Suppose the lengths of the query tokens, key tokens, and value tokens are all H × W .

Original: We build token pyramids of L levels, the l th level has a token length of HW 4 l − 1 .
Revised: We build token pyramids of L levels, the l th level has a token length of HW 4 l − 1 .

Original: The ﬂops of computing quadtree attention is,
Revised: The ﬂops of computing quadtree attention is,

Original: [Equation]
Revised: [Equation]

Original: Here, H 0 and W 0 are the height and width of the coarsest level of token pyramids.
Revised: Here, H 0 and W 0 are the height and width of the coarsest level of token pyramids.

Original: Therefore, H 20 Wis a constant and the computational complexity is O ( KHW ) .
Revised: Therefore, H 20 Wis a constant and the computational complexity is O ( KHW ) .

Original: Since K is a constant number, the complexity of quadtree attention is linear to the number of tokens.
Revised: Since K is a constant number, the complexity of quadtree attention is linear to the number of tokens.

Original: B A DDITIONAL TASK INTRODUCTION AND IMPLEMENTATION DETAILS B. F EATURE M ATCHING
Revised: B A DDITIONAL TASK INTRODUCTION AND IMPLEMENTATION DETAILS B. F EATURE M ATCHING

Original: SuperGlue (Sarlin et al., 2020) is the ﬁrst feature matching method that shows transformers can signiﬁcantly improve performance.
Revised: SuperGlue (Sarlin et al., 2020) is the ﬁrst feature matching method that shows transformers can signiﬁcantly improve performance.

Original: It ﬁrst detects interest points and extracts local descriptors using convolutional neural networks.
Revised: It ﬁrst detects interest points and extracts local descriptors using convolutional neural networks.

Original: Then, cross attention and self-attention are chained alternatively to generate new descriptors from the extracted descriptors.
Revised: Then, cross attention and self-attention are chained alternatively to generate new descriptors from the extracted descriptors.

Original: SuperGlue uses full attention, since the number of interest points is small, typically less than 1,000.
Revised: SuperGlue uses full attention, since the number of interest points is small, typically less than 1,000.

Original: Recently, another work, LoFTR (Sun et al., 2021) abandons interest point detection and directly matches dense pixels on a 60 × 80 feature map.
Revised: Recently, another work, LoFTR (Sun et al., 2021) abandons interest point detection and directly matches dense pixels on a 60 × 80 feature map.

Original: After obtaining the coarse match results, the positions are reﬁned by ﬁne matches in local regions through a linear transformer.
Revised: After obtaining the coarse match results, the positions are reﬁned by ﬁne matches in local regions through a linear transformer.

Original: However, due to the computational costs of full attention,
Revised: However, due to the computational costs of full attention,

Original: LoFTR compromises to apply linear attention to process high resolution images.
Revised: LoFTR compromises to apply linear attention to process high resolution images.

Original: It ﬁrst uses ResNet18 (He et al., 2016) to extract image features from two images and leverages four self- and cross linear attentions to process those features.
Revised: It ﬁrst uses ResNet18 (He et al., 2016) to extract image features from two images and leverages four self- and cross linear attentions to process those features.

Original: The costs are computed for features after transformer.
Revised: The costs are computed for features after transformer.

Original: It shows signiﬁcant improvement over SuperGlue.
Revised: It shows signiﬁcant improvement over SuperGlue.

Original: Implementation details.
Revised: Implementation details.

Original: We train and evaluate the model in ScanNet (Dai et al., 2017), where 230M image pairs is sampled for training, with overlapping scores between 0.4 and 0.8.
Revised: We train and evaluate the model in ScanNet (Dai et al., 2017), where 230M image pairs is sampled for training, with overlapping scores between 0.4 and 0.8.

Original: ScanNet provides
Revised: ScanNet provides

Original: RGB images, depth maps, and ground truth camera poses on a well-deﬁned training and testing split.
Revised: RGB images, depth maps, and ground truth camera poses on a well-deﬁned training and testing split.

Original: Following the same evaluation settings as Sarlin et al.
Revised: Following the same evaluation settings as Sarlin et al.

Original: and Sun et al.
Revised: and Sun et al.

Original: (2021), we evaluate our method on the 1,500 testing pairs from (Sarlin et al., 2020).
Revised: (2021), we evaluate our method on the 1,500 testing pairs from (Sarlin et al., 2020).

Original: For both trainig and testing, all images and depth maps are resized to 640 × 480.
Revised: For both trainig and testing, all images and depth maps are resized to 640 × 480.

Original: Following (Sun et al., 2021), we compute the camera pose by solving the essential matrix from predicted matches with RANSAC.
Revised: Following (Sun et al., 2021), we compute the camera pose by solving the essential matrix from predicted matches with RANSAC.

Original: We report the AUC of the pose error at thresholds (5 ◦ , 10 ◦ , 20 ◦ ), where the pose error is deﬁned as the maximum of angular error in rotation and translation.
Revised: We report the AUC of the pose error at thresholds (5 ◦ , 10 ◦ , 20 ◦ ), where the pose error is deﬁned as the maximum of angular error in rotation and translation.

Original: We only replace the coarse level transformer with quadtree attention.
Revised: We only replace the coarse level transformer with quadtree attention.

Original: B. S TEREO MATCHING
Revised: B. S TEREO MATCHING

Original: Our network is based on the STTR (Li et al., 2021), where we replace the standard transformer with our quadtree transformer.
Revised: Our network is based on the STTR (Li et al., 2021), where we replace the standard transformer with our quadtree transformer.

Original: The network consists of a CNN backbone which outputs feature maps of 1/2 image resolution, a quadtree transformer with both self- and cross attention, a regression head with optimal transport layers (Cuturi, 2013), and a context adjust layer to reﬁne the disparity.
Revised: The network consists of a CNN backbone which outputs feature maps of 1/2 image resolution, a quadtree transformer with both self- and cross attention, a regression head with optimal transport layers (Cuturi, 2013), and a context adjust layer to reﬁne the disparity.

Original: Six self- and cross attention layers are used with 128 channels.
Revised: Six self- and cross attention layers are used with 128 channels.

Original: We build pyramids with four levels for quadtree attention, and apply the Sinkhorn algorithm (Cuturi, 2013) for 10 iteration for optimal transport.
Revised: We build pyramids with four levels for quadtree attention, and apply the Sinkhorn algorithm (Cuturi, 2013) for 10 iteration for optimal transport.

Original: We follow STTR to train the network, with 15 epochs of AdamW optimizer.
Revised: We follow STTR to train the network, with 15 epochs of AdamW optimizer.

Original: OneCycle learning rate scheduler is used with a leaning rate of 6e-4 and a batch size of 8.
Revised: OneCycle learning rate scheduler is used with a leaning rate of 6e-4 and a batch size of 8.

Original: B. I MAGE CLASSIFICATION
Revised: B. I MAGE CLASSIFICATION

Original: This paragraph introduces the details of PVTv2-b0, b1, b2.
Revised: This paragraph introduces the details of PVTv2-b0, b1, b2.

Original: All these three networks have 4 stages.
Revised: All these three networks have 4 stages.

Original: Each stage is down-sampled from the previous stage by a stride of 2.
Revised: Each stage is down-sampled from the previous stage by a stride of 2.

Original: The feature resolutions for each stage is H 4 × W 4 , H 8 × W 8 , H 16 × W 16 and H 32 ×
Revised: The feature resolutions for each stage is H 4 × W 4 , H 8 × W 8 , H 16 × W 16 and H 32 ×

Original: W 32 respectively, where H and W is the image height and width.
Revised: W 32 respectively, where H and W is the image height and width.

Original: For each stage, M quadtree transformers are used with a channel number of I and head number of J .
Revised: For each stage, M quadtree transformers are used with a channel number of I and head number of J .

Original: For the network PVTv2-b0, the parameters M , I , J are set to [2 , 2 , 2 , 2] , [32 , 64 , 160 , 256] , [1 , 2 , 5 , 8] at each stage respectively.
Revised: For the network PVTv2-b0, the parameters M , I , J are set to [2 , 2 , 2 , 2] , [32 , 64 , 160 , 256] , [1 , 2 , 5 , 8] at each stage respectively.

Original: For the network PVTv2-b1, the parameters
Revised: For the network PVTv2-b1, the parameters

Original: [Figure] M , I , J are set to [2 , 2 , 2 , 2] , [64 , 128 , 320 , 512] , [1 , 2 , 5 , 8] respectively.
Revised: M , I , J are set to [2 , 2 , 2 , 2] , [64 , 128 , 320 , 512] , [1 , 2 , 5 , 8] respectively.

Original: Finally, for PVTv2-b2, the parameters M , I , J are set to [3 , 4 , 6 , 3] , [64 , 128 , 320 , 512] , [1 , 2 , 5 , 8] respectively.
Revised: Finally, for PVTv2-b2, the parameters M , I , J are set to [3 , 4 , 6 , 3] , [64 , 128 , 320 , 512] , [1 , 2 , 5 , 8] respectively.

Original: C PLOT OF L OSS AND AUC FOR IMAGE MATCHING
Revised: C T RAINING LOSS

Revised: Feature matching.

Original: We plot the training loss and validation performance for LoFTR-lite in Figure 4 for different efﬁcient transformers, including spatial reduction (SR) transformer (Wang et al., 2021c), linear transformer (Katharopoulos et al., 2020), our Quadtree-A, and Quadtree-B transformers.
Revised: We plot the training loss and validation performance for LoFTR-lite in Figure 3 for different efﬁcient transformers, including spatial reduction (SR) transformer (Wang et al., 2021c), linear transformer (Katharopoulos et al., 2020), our Quadtree-A, and Quadtree-B transformers.

Original: We can see quadtree-B transformer obtains consistently lower training loss and higher performance over other three transformers.
Revised: We can see quadtree-B transformer obtains consistently lower training loss and higher performance over other three transformers.

Original: In addition, it is also noted that the spatial reduction (SR)  transformer has lower training but worse AUC@20 ◦ than QuadTree-A attention, which indicates that it cannot generalize well.
Revised: In addition, it is also noted that the spatial reduction (SR) [Figure] [Figure] transformer has lower training but worse AUC@20 ◦ than QuadTree-A attention, which indicates that it cannot generalize well.

Revised: Image classiﬁcation .

Revised: We also show traning and validation curve for image classiﬁcation task with respective to different attentions in Fig. 4.

Revised: Compared with Swin Transformer (Liu et al., 2021) and

Revised: PVT (Wang et al., 2021c), the loss of Quadtree attention is consistently lower and the top 1 accuracy is higher.

Revised: D R UNNING TIME

Revised: Currently, we only implement a naive CUDA kernel without mamy optimizations and it is not as efﬁcient as the well-optimized dense GPU matrix operation.

Revised: We test the running time of Retinanet under PVTv2-b0 architecture.

Revised: For PVTv2-b0, The running time is 0.026s to forward one image and for Quadtree-b0, the running time is 0.046s for forwarding once.

Revised: However, Quadtree-b0 has much lower memory usage than PVTv2-b0.

Revised: Quadtree-b0 consumes about 339MB while PVTv2-b consumes about 574MB for one 800 × 1333 image.

Revised: [Table]

Revised: [Figure]

Revised: [Figure]

Revised: [Table]

Revised: E A BLATIONS

Revised: QuadTree-A vs QuadTree-B .

Revised: QuadTree-B architecture consistently outperforms QuadTree-A in feature matching, image classiﬁcation, and detection experiments.

Revised: We analyze its reason as shown in Figure 5, where (d) and (e) show the attention score maps of QuadTree-A and QuadTree-B at different levels for the same point in the query image shown in (a).

Revised: It is clear that the QuadTree-B has more accurate score maps, and is less affected by the inaccuracy in coarse level score estimation.

Revised: We further visualize the attention scores of spatial reduction (SR) attention (Wang et al., 2021c) and linear transformer (Katharopoulos et al., 2020) in (b) and (c).

Revised: We can see that SR attention and linear transformer attend the query token on large unrelated regions due to the loss of ﬁne-grained information.

Revised: In contrast, our quadtree transformer focus on the most relevant area.

Revised: Comparison with Swin Transformer.

Revised: We compare with Swin Transformer in Table.

Revised: 8 using the released codes.

Revised: We replace the corresponding attention in Swin Transformer with Quadtree-B attention.

Revised: Our method obtains 0.9% higher top 1 accuracy than Swin Transformer.

Revised: Multiscale position encoding.

Revised: We compare our method with or without multiscale position encoding (MPE).

Revised: For Quadtree-B-b2 model, MPE can bring an improvement of 1.3 on object detection.