Original: V I DT: A N E FFICIENT AND E FFECTIVE F ULLY T RANSFORMER - BASED O BJECT D ETECTOR
Revised: V I DT: A N E FFICIENT AND E FFECTIVE F ULLY T RANSFORMER - BASED O BJECT D ETECTOR

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Transformers are transforming the landscape of computer vision, especially for recognition tasks.
Revised: Transformers are transforming the landscape of computer vision, especially for recognition tasks.

Original: Detection transformers are the ﬁrst fully end-to-end learning systems for object detection, while vision transformers are the ﬁrst fully transformer-based architecture for image classiﬁcation.
Revised: Detection transformers are the ﬁrst fully end-to-end learning systems for object detection, while vision transformers are the ﬁrst fully transformer-based architecture for image classiﬁcation.

Original: In this paper, we integrate
Revised: In this paper, we integrate

Original: Vision and Detection Transformers (ViDT) to build an effective and efﬁcient object detector.
Revised: Vision and Detection Transformers (ViDT) to build an effective and efﬁcient object detector.

Original: ViDT introduces a reconﬁgured attention module to extend the recent
Revised: ViDT introduces a reconﬁgured attention module to extend the recent

Original: Swin Transformer to be a standalone object detector, followed by a computationally efﬁcient transformer decoder that exploits multi-scale features and auxiliary techniques essential to boost the detection performance without much increase in computational load.
Revised: Swin Transformer to be a standalone object detector, followed by a computationally efﬁcient transformer decoder that exploits multi-scale features and auxiliary techniques essential to boost the detection performance without much increase in computational load.

Original: Extensive evaluation results on the Microsoft COCO benchmark dataset demonstrate that ViDT obtains the best AP and latency trade-off among existing fully transformer-based object detectors, and achieves 49.2 AP owing to its high scalability for large models.
Revised: Extensive evaluation results on the Microsoft COCO benchmark dataset demonstrate that ViDT obtains the best AP and latency trade-off among existing fully transformer-based object detectors, and achieves 49.2 AP owing to its high scalability for large models.

Original: Wewill release the code and trained models upon acceptance.
Revised: We release the code and trained models at https://github.com/naver-ai/vidt.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Object detection is the task of predicting both bounding boxes and object classes for each object of interest in an image.
Revised: Object detection is the task of predicting both bounding boxes and object classes for each object of interest in an image.

Original: Modern deep object detectors heavily rely on meticulously designed components, such as anchor generation and non-maximum suppression (Papageorgiou & Poggio, 2000; Liu et al., 2020).
Revised: Modern deep object detectors heavily rely on meticulously designed components, such as anchor generation and non-maximum suppression (Papageorgiou & Poggio, 2000; Liu et al., 2020).

Original: As a result, the performance of these object detectors depend on speciﬁc postprocessing steps, which involve complex pipelines and make fully end-to-end training difﬁcult.
Revised: As a result, the performance of these object detectors depend on speciﬁc postprocessing steps, which involve complex pipelines and make fully end-to-end training difﬁcult.

Original: Motivated by the recent success of Transformers (Vaswani et al., 2017) in NLP, numerous studies introduce Transformers into computer vision tasks.
Revised: Motivated by the recent success of Transformers (Vaswani et al., 2017) in NLP, numerous studies introduce Transformers into computer vision tasks.

Original: Carion et al.
Revised: Carion et al.

Original: (2020) proposed Detection
Revised: (2020) proposed Detection

Original: Transformers (DETR) to eliminate the meticulously designed components by employing a simple transformer encoder and decoder architecture, which serves as a neck component to bridge a CNN body for feature extraction and a detector head for prediction.
Revised: Transformers (DETR) to eliminate the meticulously designed components by employing a simple transformer encoder and decoder architecture, which serves as a neck component to bridge a CNN body for feature extraction and a detector head for prediction.

Original: Thus, DETR enables end-to-end training of deep object detectors.
Revised: Thus, DETR enables end-to-end training of deep object detectors.

Original: By contrast, Dosovitskiy et al.
Revised: By contrast, Dosovitskiy et al.

Original: (2021) showed that a fully-transformer
Revised: (2021) showed that a fully-transformer

Original: [Figure]
Revised: [Figure]

Original: backbone without any convolutional layers, Vision Transformer (ViT) , achieves the state-of-theart results in image classiﬁcation benchmarks.
Revised: backbone without any convolutional layers, Vision Transformer (ViT) , achieves the state-of-theart results in image classiﬁcation benchmarks.

Original: Approaches like ViT have been shown to learn effective representation models without strong human inductive biases, e.g., meticulously designed components in object detection (DETR), locality-aware designs such as convolutional layers and pooling mechanisms.
Revised: Approaches like ViT have been shown to learn effective representation models without strong human inductive biases, e.g., meticulously designed components in object detection (DETR), locality-aware designs such as convolutional layers and pooling mechanisms.

Original: However, there is a lack of effort to synergize
Revised: However, there is a lack of effort to synergize

Original: DETR and ViT for a better object detection architecture.
Revised: DETR and ViT for a better object detection architecture.

Original: In this paper, we integrate both approaches to build a fully transformer-based, end-to-end object detector that achieves state-of-the-art performance without increasing computational load.
Revised: In this paper, we integrate both approaches to build a fully transformer-based, end-to-end object detector that achieves state-of-the-art performance without increasing computational load.

Original: A straightforward integration of DETR and ViT can be achieved by replacing the ResNet backbone
Revised: A straightforward integration of DETR and ViT can be achieved by replacing the ResNet backbone

Original: [Figure]
Revised: [Figure]

Original: (body) of DETR with ViT – Figure 2(a).
Revised: (body) of DETR with ViT – Figure 2(a).

Original: This naive integration, DETR (ViT) 1 , has two limitations.
Revised: This naive integration, DETR (ViT) 1 , has two limitations.

Original: First, the canonical ViT suffers from the quadratic increase in complexity w.r.t.
Revised: First, the canonical ViT suffers from the quadratic increase in complexity w.r.t.

Original: image size, resulting in the lack of scalability.
Revised: image size, resulting in the lack of scalability.

Original: Furthermore, the attention operation at the transformer encoder and decoder (i.e., the “neck” component) adds signiﬁcant computational overhead to the detector.
Revised: Furthermore, the attention operation at the transformer encoder and decoder (i.e., the “neck” component) adds signiﬁcant computational overhead to the detector.

Original: Therefore, the naive integration of DETR and ViT show very high latency – the blue lines of Figure 1.
Revised: Therefore, the naive integration of DETR and ViT show very high latency – the blue lines of Figure 1.

Original: Recently, Fang et al.
Revised: Recently, Fang et al.

Original: (2021) propose an extension of ViT to object detection, named YOLOS , by appending the detection tokens [ DET ] to the patch tokens [ PATCH ] (Figure 2(b)), where [ DET ] tokens are learnable embeddings to specify different objects to detect.
Revised: (2021) propose an extension of ViT to object detection, named YOLOS , by appending the detection tokens [ DET ] to the patch tokens [ PATCH ] (Figure 2(b)), where [ DET ] tokens are learnable embeddings to specify different objects to detect.

Original: YOLOS is a neck-free architecture and removes the additional computational costs from the neck encoder.
Revised: YOLOS is a neck-free architecture and removes the additional computational costs from the neck encoder.

Original: However, YOLOS shows limited performance because it cannot use additional optimization techniques on the neck architecture, e.g., multi-scale features and auxiliary loss.
Revised: However, YOLOS shows limited performance because it cannot use additional optimization techniques on the neck architecture, e.g., multi-scale features and auxiliary loss.

Original: In addition, YOLOS can only accommodate the canonical transformer due to its architectural limitation, resulting in a quadratic complexity w.r.t.
Revised: In addition, YOLOS can only accommodate the canonical transformer due to its architectural limitation, resulting in a quadratic complexity w.r.t.

Original: the input size.
Revised: the input size.

Original: In this paper, we propose a novel integration of Vi sion and D etection T ransformers ( ViDT ) (Figure 2(c)).
Revised: In this paper, we propose a novel integration of Vi sion and D etection T ransformers ( ViDT ) (Figure 2(c)).

Original: Our contributions are three-folds.
Revised: Our contributions are three-folds.

Original: First, ViDT introduces a modiﬁed attention mechanism, named Reconﬁgured Attention Module (RAM), that facilitates any ViT variant to handle the appended [ DET ] and [ PATCH ] tokens for object detection.
Revised: First, ViDT introduces a modiﬁed attention mechanism, named Reconﬁgured Attention Module (RAM), that facilitates any ViT variant to handle the appended [ DET ] and [ PATCH ] tokens for object detection.

Original: Thus, we can modify the latest Swin Transformer (Liu et al., 2021) backbone with RAM to be an object detector and obtain high scalability using its local attention mechanism with linear complexity.
Revised: Thus, we can modify the latest Swin Transformer (Liu et al., 2021) backbone with RAM to be an object detector and obtain high scalability using its local attention mechanism with linear complexity.

Original: Second, ViDT adopts a lightweight encoder-free neck architecture to reduce the computational overhead while still enabling the additional optimization techniques on the neck module.
Revised: Second, ViDT adopts a lightweight encoder-free neck architecture to reduce the computational overhead while still enabling the additional optimization techniques on the neck module.

Original: Note that the neck encoder is unnecessary because RAM directly extracts ﬁne-grained representation for object detection, i.e., [ DET ] tokens.
Revised: Note that the neck encoder is unnecessary because RAM directly extracts ﬁne-grained representation for object detection, i.e., [ DET ] tokens.

Original: As a result, ViDT obtains better performance than neck-free counterparts.
Revised: As a result, ViDT obtains better performance than neck-free counterparts.

Original: Finally, we introduce a new concept of token matching for knowledge distillation, which brings additional performance gains from a large model to a small model without compromising detection efﬁciency.
Revised: Finally, we introduce a new concept of token matching for knowledge distillation, which brings additional performance gains from a large model to a small model without compromising detection efﬁciency.

Original: ViDT has two architectural advantages over existing approaches.
Revised: ViDT has two architectural advantages over existing approaches.

Original: First, similar to YOLOS, ViDT takes [ DET ] tokens as the additional input, maintaining a ﬁxed scale for object detection, but constructs hierarchical representations starting with small-sized image patches for [ PATCH ] tokens.
Revised: First, similar to YOLOS, ViDT takes [ DET ] tokens as the additional input, maintaining a ﬁxed scale for object detection, but constructs hierarchical representations starting with small-sized image patches for [ PATCH ] tokens.

Original: Second, ViDT can use the hierarchical (multi-scale) features and additional techniques without a signiﬁcant computation overhead.
Revised: Second, ViDT can use the hierarchical (multi-scale) features and additional techniques without a signiﬁcant computation overhead.

Original: Therefore, as a fully transformer-based object detector, ViDT facilitates better integration of vision and detection transformers.
Revised: Therefore, as a fully transformer-based object detector, ViDT facilitates better integration of vision and detection transformers.

Original: Extensive experiments on Microsoft COCO benchmark (Lin et al., 2014) show that ViDT is highly scalable even for large ViT models, such as
Revised: Extensive experiments on Microsoft COCO benchmark (Lin et al., 2014) show that ViDT is highly scalable even for large ViT models, such as

Original: Swin-base with 0.1 billion parameters, and achieves the best AP and latency trade-off.
Revised: Swin-base with 0.1 billion parameters, and achieves the best AP and latency trade-off.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Vision transformers process an image as a sequence of small-sized image patches, thereby allowing all the positions in the image to interact in attention operations (i.e., global attention).
Revised: Vision transformers process an image as a sequence of small-sized image patches, thereby allowing all the positions in the image to interact in attention operations (i.e., global attention).

Original: However, the canonical ViT (Dosovitskiy et al., 2021) is not compatible with a broad range of vision tasks due to its high computational complexity, which increases quadratically with respect to image size.
Revised: However, the canonical ViT (Dosovitskiy et al., 2021) is not compatible with a broad range of vision tasks due to its high computational complexity, which increases quadratically with respect to image size.

Original: The Swin Transformer (Liu et al., 2021) resolves the complexity issue by introducing the notion of shifted windows that support local attention and patch reduction operations, thereby improving compatibility for dense prediction task such as object detection.
Revised: The Swin Transformer (Liu et al., 2021) resolves the complexity issue by introducing the notion of shifted windows that support local attention and patch reduction operations, thereby improving compatibility for dense prediction task such as object detection.

Original: A few approaches use vision transformers as detector backbones but achieve limited success (Heo et al., 2021; Fang et al., 2021).
Revised: A few approaches use vision transformers as detector backbones but achieve limited success (Heo et al., 2021; Fang et al., 2021).

Original: Detection transformers eliminate the meticulously designed components (e.g., anchor generation and non-maximum suppression) by combining convolutional network backbones and Transformer encoder-decoders.
Revised: Detection transformers eliminate the meticulously designed components (e.g., anchor generation and non-maximum suppression) by combining convolutional network backbones and Transformer encoder-decoders.

Original: While the canonical DETR (Carion et al., 2020) achieves high detection performance, it suffers from very slow convergence compared to previous detectors.
Revised: While the canonical DETR (Carion et al., 2020) achieves high detection performance, it suffers from very slow convergence compared to previous detectors.

Original: For example, DETR requires 500 epochs while the conventional Faster R-CNN (Ren et al., 2015) training needs onlyepochs (Wu et al., 2019).
Revised: For example, DETR requires 500 epochs while the conventional Faster R-CNN (Ren et al., 2015) training needs onlyepochs (Wu et al., 2019).

Original: To mitigate the issue, Zhu et al.
Revised: To mitigate the issue, Zhu et al.

Original: (2021) propose Deformable DETR which introduces deformable attention for utilizing multi-scale features as well as expediting the slow training convergence of DETR.
Revised: (2021) propose Deformable DETR which introduces deformable attention for utilizing multi-scale features as well as expediting the slow training convergence of DETR.

Original: In this paper, we use the Deformable DETR as our base detection transformer framework and integrate it with the recent vision transformers.
Revised: In this paper, we use the Deformable DETR as our base detection transformer framework and integrate it with the recent vision transformers.

Original: DETR (ViT) is a straightforward integration of DETR and ViT, which uses ViT as a feature extractor, followed by the transformer encoder-decoder in DETR.
Revised: DETR (ViT) is a straightforward integration of DETR and ViT, which uses ViT as a feature extractor, followed by the transformer encoder-decoder in DETR.

Original: As illustrated in Figure 2(a), it is a body–neck–head structure; the representation of input [ PATCH ] tokens are extracted by the ViT backbone and then directly fed to the transformer-based encoding and decoding pipeline.
Revised: As illustrated in Figure 2(a), it is a body–neck–head structure; the representation of input [ PATCH ] tokens are extracted by the ViT backbone and then directly fed to the transformer-based encoding and decoding pipeline.

Original: To predict multiple objects, a ﬁxed number of learnable [ DET ] tokens are provided as additional input to the decoder.
Revised: To predict multiple objects, a ﬁxed number of learnable [ DET ] tokens are provided as additional input to the decoder.

Original: Subsequently, output embeddings by the decoder produce ﬁnal predictions through the detection heads for classiﬁcation and box regression.
Revised: Subsequently, output embeddings by the decoder produce ﬁnal predictions through the detection heads for classiﬁcation and box regression.

Original: Since DETR (ViT) does not modify the backbone at all, it can be ﬂexibly changed to any latest ViT model, e.g., Swin Transformer.
Revised: Since DETR (ViT) does not modify the backbone at all, it can be ﬂexibly changed to any latest ViT model, e.g., Swin Transformer.

Original: Additionally, its neck decoder facilitates the aggregation of multi-scale features and the use of additional techniques, which help detect objects of different sizes and speed up training (Zhu et al., 2021).
Revised: Additionally, its neck decoder facilitates the aggregation of multi-scale features and the use of additional techniques, which help detect objects of different sizes and speed up training (Zhu et al., 2021).

Original: However, the attention operation at the neck encoder adds signiﬁcant computational overhead to the detector.
Revised: However, the attention operation at the neck encoder adds signiﬁcant computational overhead to the detector.

Original: In contrast, ViDT resolves this issue by directly extracting ﬁne-grained [ DET ] features from Swin
Revised: In contrast, ViDT resolves this issue by directly extracting ﬁne-grained [ DET ] features from Swin

Original: Transformer with RAM without maintaining the transformer encoder in the neck architecture.
Revised: Transformer with RAM without maintaining the transformer encoder in the neck architecture.

Original: YOLOS (Fang et al., 2021) is a canonical ViT architecture for object detection with minimal modiﬁcations.
Revised: YOLOS (Fang et al., 2021) is a canonical ViT architecture for object detection with minimal modiﬁcations.

Original: As illustrated in Figure 2(b), YOLOS achieves a neck-free structure by appending randomly initialized learnable [ DET ] tokens to the sequence of input [ PATCH ] tokens.
Revised: As illustrated in Figure 2(b), YOLOS achieves a neck-free structure by appending randomly initialized learnable [ DET ] tokens to the sequence of input [ PATCH ] tokens.

Original: Since all the embeddings for [ PATCH ] and [ DET ] tokens interact via global attention, the ﬁnal [ DET ] tokens are generated by the ﬁne-tuned ViT backbone and then directly generate predictions through the detection heads without requiring any neck layer.
Revised: Since all the embeddings for [ PATCH ] and [ DET ] tokens interact via global attention, the ﬁnal [ DET ] tokens are generated by the ﬁne-tuned ViT backbone and then directly generate predictions through the detection heads without requiring any neck layer.

Original: While the naive DETR (ViT) suffers from the computational overhead from the neck layer, YOLOS enjoys efﬁcient computations by treating the [ DET ] tokens as additional input for ViT. YOLOS shows that 2D object detection can be accomplished in a pure sequence-to-sequence manner, but this solution entails two inherent limitations:
Revised: While the naive DETR (ViT) suffers from the computational overhead from the neck layer, YOLOS enjoys efﬁcient computations by treating the [ DET ] tokens as additional input for ViT. YOLOS shows that 2D object detection can be accomplished in a pure sequence-to-sequence manner, but this solution entails two inherent limitations:

Original: 1) YOLOS inherits the drawback of the canonical ViT; the high computational complexity attributed to the global attention operation.
Revised: 1) YOLOS inherits the drawback of the canonical ViT; the high computational complexity attributed to the global attention operation.

Original: As illustrated in Figure
Revised: As illustrated in Figure

Original: YOLOS shows very poor latency compared with other fully transformer-based detectors, especially when its model size becomes larger, i.e., small → base.
Revised: YOLOS shows very poor latency compared with other fully transformer-based detectors, especially when its model size becomes larger, i.e., small → base.

Original: Thus, YOLOS is not scalable for the large model.
Revised: Thus, YOLOS is not scalable for the large model.

Original: 2) YOLOS cannot beneﬁt from using additional techniques essential for better performance, e g., multi-scale features, due to the absence of the neck layer.
Revised: 2) YOLOS cannot beneﬁt from using additional techniques essential for better performance, e.g., multi-scale features, due to the absence of the neck layer.

Original: Although YOLOS used the same DeiT backbone with Deformable DETR (DeiT), its AP was lower than the straightforward integration.
Revised: Although YOLOS used the same DeiT backbone with Deformable DETR (DeiT), its AP was lower than the straightforward integration.

Original: In contrast, the encoder-free neck architecture of ViDT enjoys the additional optimization techniques from Zhu et al.
Revised: In contrast, the encoder-free neck architecture of ViDT enjoys the additional optimization techniques from Zhu et al.

Original: (2021), resulting in the faster convergence and the better performance.
Revised: (2021), resulting in the faster convergence and the better performance.

Original: Further, our
Revised: Further, our

Original: RAM enables to combine Swin Transformer and the sequence-to-sequence paradigm for detection.
Revised: RAM enables to combine Swin Transformer and the sequence-to-sequence paradigm for detection.

Original:  V I DT: V ISION AND D ETECTION T RANSFORMERS
Revised:  V I DT: V ISION AND D ETECTION T RANSFORMERS

Original: ViDT ﬁrst reconﬁgures the attention model of Swin Transformer to support standalone object detection while fully reusing the parameters of Swin Transformer.
Revised: ViDT ﬁrst reconﬁgures the attention model of Swin Transformer to support standalone object detection while fully reusing the parameters of Swin Transformer.

Original: Next, it incorporates an encoder-free neck layer to exploit multi-scale features and two essential techniques: auxiliary decoding loss and iterative box reﬁnement.
Revised: Next, it incorporates an encoder-free neck layer to exploit multi-scale features and two essential techniques: auxiliary decoding loss and iterative box reﬁnement.

Original: We further introduce knowledge distillation with token matching to beneﬁt from large ViDT models.
Revised: We further introduce knowledge distillation with token matching to beneﬁt from large ViDT models.

Original: 3. R ECONFIGURED A TTENTION M ODULE
Revised: 3. R ECONFIGURED A TTENTION M ODULE

Original: Applying patch reduction and local attention scheme of Swin Transformer to the sequence-tosequence paradigm is challenging because (1) the number of [ DET ] tokens must be maintained at a ﬁxed-scale and (2) the lack of locality between [ DET ] tokens.
Revised: Applying patch reduction and local attention scheme of Swin Transformer to the sequence-tosequence paradigm is challenging because (1) the number of [ DET ] tokens must be maintained at a ﬁxed-scale and (2) the lack of locality between [ DET ] tokens.

Original: To address this challenge, we introduce a reconﬁgured attention module (RAM) 2 that decomposes a single global attention associated with [ PATCH ] and [ DET ] tokens into the three different attention, namely [ PATCH ] × [ PATCH ] , [ DET ] × [ DET ] ,
Revised: To address this challenge, we introduce a reconﬁgured attention module (RAM) 2 that decomposes a single global attention associated with [ PATCH ] and [ DET ] tokens into the three different attention, namely [ PATCH ] × [ PATCH ] , [ DET ] × [ DET ] ,

Original: V [PATCH]
Revised: V [PATCH]

Original: Tokens
Revised: Tokens

Original: FeatureMap FeatureMap
Revised: FeatureMap FeatureMap

Original: Replicate
Revised: Replicate

Original: Spatial Positional Encoding
Revised: Spatial Positional Encoding

Original: Detection Token Encoding
Revised: Detection Token Encoding

Original: Q V [PATCH] × [PATCH] with Window Partitions
Revised: Q V [PATCH] × [PATCH] with Window Partitions

Original: Q
Revised: Q

Original: V K + + × [PATCH]
Revised: V K + + × [PATCH]

Original: Tokens [DET] × [DET & PATCH] K
Revised: Tokens [DET] × [DET & PATCH] K

Original: Q K V
Revised: Q K V

Original: Q
Revised: Q

Original: K
Revised: K

Original: K ×
Revised: K ×

Original: V
Revised: V

Original: V ×
Revised: V ×

Original: [DET] Tokens × Q
Revised: [DET] Tokens × Q

Original: K Relative Pos.
Revised: K Relative Pos.

Original: Reassemble [DET] Tokens and [ DET ] × [ PATCH ] attention.
Revised: Reassemble [DET] Tokens and [ DET ] × [ PATCH ] attention.

Original: Based on the decomposition, the efﬁcient schemes of Swin Transformer are applied only to [ PATCH ] ×
Revised: Based on the decomposition, the efﬁcient schemes of Swin Transformer are applied only to [ PATCH ] ×

Original: [ PATCH ] attention, which is the heaviest part in computational complexity, without breaking the two constraints on [ DET ] tokens.
Revised: [ PATCH ] attention, which is the heaviest part in computational complexity, without breaking the two constraints on [ DET ] tokens.

Original: As illustrated in Figure 3, these modiﬁcations fully reuse all the parameters of Swin Transformer by sharing projection layers for [ DET ] and [ PATCH ] tokens, and perform the three different attention operations:
Revised: As illustrated in Figure 3, these modiﬁcations fully reuse all the parameters of Swin Transformer by sharing projection layers for [ DET ] and [ PATCH ] tokens, and perform the three different attention operations:

Original: • [ PATCH ] ×
Revised: • [ PATCH ] ×

Original: [ PATCH ] Attention: The initial [ PATCH ] tokens are progressively calibrated across the attention layers such that they aggregate the key contents in the global feature map (i
Revised: [ PATCH ] Attention: The initial [ PATCH ] tokens are progressively calibrated across the attention layers such that they aggregate the key contents in the global feature map (i

Original: e., spatial form of [ PATCH ] tokens) according to the attention weights, which are computed by (cid:104) query, key (cid:105) pairs.
Revised: e., spatial form of [ PATCH ] tokens) according to the attention weights, which are computed by (cid:104) query, key (cid:105) pairs.

Original: For [ PATCH ] × [ PATCH ] attention, Swin Transformer performs local attention on each window partition, but its shifted window partitioning in successive blocks bridges the windows of the preceding layer, providing connections among partitions to capture global information.
Revised: For [ PATCH ] × [ PATCH ] attention, Swin Transformer performs local attention on each window partition, but its shifted window partitioning in successive blocks bridges the windows of the preceding layer, providing connections among partitions to capture global information.

Original: Without modifying this concept, we use the same policy to generate hierarchical [ PATCH ] tokens.
Revised: Without modifying this concept, we use the same policy to generate hierarchical [ PATCH ] tokens.

Original: Thus, the number of [ PATCH ] tokens is reduced by a factor of 4 at each stage; the resolution of feature maps decreases from H/ 4 × W/ 4 to H/ 32 × W/ 32 over a total of four stages, where H and W denote the width and height of the input image, respectively.
Revised: Thus, the number of [ PATCH ] tokens is reduced by a factor of 4 at each stage; the resolution of feature maps decreases from H/ 4 × W/ 4 to H/ 32 × W/ 32 over a total of four stages, where H and W denote the width and height of the input image, respectively.

Original: • [ DET ] ×
Revised: • [ DET ] ×

Original: [ DET ] Attention: Like YOLOS, we append one hundred learnable [ DET ] tokens as the additional input to the [ PATCH ] tokens.
Revised: [ DET ] Attention: Like YOLOS, we append one hundred learnable [ DET ] tokens as the additional input to the [ PATCH ] tokens.

Original: As the number of [ DET ] tokens speciﬁes the number of objects to detect, their number must be maintained with a ﬁxed-scale over the transformer layers.
Revised: As the number of [ DET ] tokens speciﬁes the number of objects to detect, their number must be maintained with a ﬁxed-scale over the transformer layers.

Original: In addition, [ DET ] tokens do not have any locality unlike the [ PATCH ] tokens.
Revised: In addition, [ DET ] tokens do not have any locality unlike the [ PATCH ] tokens.

Original: Hence, for [ DET ] × [ DET ] attention, we perform global self-attention while maintaining the number of them; this attention helps each [ DET ] token to localize a different object by capturing the relationship between them.
Revised: Hence, for [ DET ] × [ DET ] attention, we perform global self-attention while maintaining the number of them; this attention helps each [ DET ] token to localize a different object by capturing the relationship between them.

Original: • [ DET ] ×
Revised: • [ DET ] ×

Original: [ PATCH ] Attention: This is cross-attention between [ DET ] and [ PATCH ] tokens, which produces an object embedding per [ DET ] token.
Revised: [ PATCH ] Attention: This is cross-attention between [ DET ] and [ PATCH ] tokens, which produces an object embedding per [ DET ] token.

Original: For each [ DET ] token, the key contents in [ PATCH ] tokens are aggregated to represent the target object.
Revised: For each [ DET ] token, the key contents in [ PATCH ] tokens are aggregated to represent the target object.

Original: Since the [ DET ] tokens specify different objects, it produces different object embeddings for diverse objects in the image.
Revised: Since the [ DET ] tokens specify different objects, it produces different object embeddings for diverse objects in the image.

Original: Without the crossattention, it is infeasible to realize the standalone object detector.
Revised: Without the crossattention, it is infeasible to realize the standalone object detector.

Original: As shown in Figure 3, ViDT binds [ DET ] × [ DET ] and [ DET ] × [ PATCH ] attention to process them at once to increase efﬁciency.
Revised: As shown in Figure 3, ViDT binds [ DET ] × [ DET ] and [ DET ] × [ PATCH ] attention to process them at once to increase efﬁciency.

Original: We replace all the attention modules in Swin Transformer with the proposed RAM, which receives [ PATCH ] and [ DET ] tokens 
Revised: We replace all the attention modules in Swin Transformer with the proposed RAM, which receives [ PATCH ] and [ DET ] tokens (as shown in “Body”

Original: as
Revised: of

Original: shown in

Original: “Body”

Original: of Figure
Revised: Figure

Original: c)) and then outputs their calibrated new tokens by performing the three different attention operations in parallel.
Revised: c)) and then outputs their calibrated new tokens by performing the three different attention operations in parallel.

Original: Positional Encoding.
Revised: Positional Encoding.

Original: ViDT adopts different positional encodings for different types of attention.
Revised: ViDT adopts different positional encodings for different types of attention.

Original: For [ PATCH ] × [ PATCH ] attention, we use the relative position bias (Hu et al., 2019) originally used in Swin
Revised: For [ PATCH ] × [ PATCH ] attention, we use the relative position bias (Hu et al., 2019) originally used in Swin

Original: Transformer.
Revised: Transformer.

Original: In contrast, the learnable positional encoding is added for [ DET ] tokens for [ DET ] × [ DET ] attention because there is no particular order between [ DET ] tokens.
Revised: In contrast, the learnable positional encoding is added for [ DET ] tokens for [ DET ] × [ DET ] attention because there is no particular order between [ DET ] tokens.

Original: However, for [ DET ] × [ PATCH ] attention, it is crucial to inject spatial bias to the [ PATCH ] tokens due to the permutation-equivariant in transformers, ignoring spatial information of the feature map.
Revised: However, for [ DET ] × [ PATCH ] attention, it is crucial to inject spatial bias to the [ PATCH ] tokens due to the permutation-equivariant in transformers, ignoring spatial information of the feature map.

Original: Thus, ViDT adds the sinusoidalbased spatial positional encoding to the feature map, which is reconstructed from the [ PATCH ] tokens for [ DET ] ×
Revised: Thus, ViDT adds the sinusoidalbased spatial positional encoding to the feature map, which is reconstructed from the [ PATCH ] tokens for [ DET ] ×

Original: [ PATCH ] attention, as can be seen from the left side of Figure 3.
Revised: [ PATCH ] attention, as can be seen from the left side of Figure 3.

Original: We present a thorough analysis of various spatial positional encodings in Section 4.2.1.
Revised: We present a thorough analysis of various spatial positional encodings in Section 4.2.1.

Original: Use of [ DET ] × [ PATCH ] Attention.
Revised: Use of [ DET ] × [ PATCH ] Attention.

Original: Applying cross-attention between [ DET ] and [ PATCH ] tokens adds additional computational overhead to Swin Transformer, especially when it is activated at the bottom layer due to the large number of [ PATCH ] tokens.
Revised: Applying cross-attention between [ DET ] and [ PATCH ] tokens adds additional computational overhead to Swin Transformer, especially when it is activated at the bottom layer due to the large number of [ PATCH ] tokens.

Original: To minimize such computational overhead,
Revised: To minimize such computational overhead,

Original: ViDT only activates the cross-attention at the last stage (the top level of the pyramid) of Swin Transformer, which consists of two transformer layers that receives [ PATCH ] tokens of size H/ 32 × W/ 32 .
Revised: ViDT only activates the cross-attention at the last stage (the top level of the pyramid) of Swin Transformer, which consists of two transformer layers that receives [ PATCH ] tokens of size H/ 32 × W/ 32 .

Original: Thus, only self-attention for [ DET ] and [ PATCH ] tokens are performed for the remaining stages except the last one.
Revised: Thus, only self-attention for [ DET ] and [ PATCH ] tokens are performed for the remaining stages except the last one.

Original: In Section 4.2.2 we show that this design choice helps achieve the highest FPS, while achieving similar detection performance as when cross-attention is enabled at every stage.
Revised: In Section 4.2.2 we show that this design choice helps achieve the highest FPS, while achieving similar detection performance as when cross-attention is enabled at every stage.

Original: We provide more details on RAM including its complexity analysis and algorithmic design in Appendix A.
Revised: We provide more details on RAM including its complexity analysis and algorithmic design in Appendix A.

Original: 3. E NCODER - FREE N ECK S TRUCTURE
Revised: 3. E NCODER - FREE N ECK S TRUCTURE

Original: To exploit multi-scale feature maps, ViDT incorporates a decoder of multi-layer deformable transformers (Zhu et al., 2021).
Revised: To exploit multi-scale feature maps, ViDT incorporates a decoder of multi-layer deformable transformers (Zhu et al., 2021).

Original: In the DETR family (Figure 2(a)), a transformer encoder is required at the neck to transform features extracted from the backbone for image classiﬁcation into the ones suitable for object detection; the encoder is generally computationally expensive since it involves [ PATCH ] × [ PATCH ] attention.
Revised: In the DETR family (Figure 2(a)), a transformer encoder is required at the neck to transform features extracted from the backbone for image classiﬁcation into the ones suitable for object detection; the encoder is generally computationally expensive since it involves [ PATCH ] × [ PATCH ] attention.

Original: However, ViDT maintains only a transformer decoder as its neck, in that Swin Transformer with RAM directly extracts ﬁne-grained features suitable for object detection as a standalone object detector.
Revised: However, ViDT maintains only a transformer decoder as its neck, in that Swin Transformer with RAM directly extracts ﬁne-grained features suitable for object detection as a standalone object detector.

Original: Thus, the neck structure of ViDT is computationally efﬁcient.
Revised: Thus, the neck structure of ViDT is computationally efﬁcient.

Original: The decoder receives two inputs from Swin Transformer with RAM: (1) [ PATCH ] tokens generated from each stage (i.e., four multi-scale feature maps, { x l } Ll =1 where L = 4 ) and (2) [ DET ] tokens generated from the last stage.
Revised: The decoder receives two inputs from Swin Transformer with RAM: (1) [ PATCH ] tokens generated from each stage (i.e., four multi-scale feature maps, { x l } Ll =1 where L = 4 ) and (2) [ DET ] tokens generated from the last stage.

Original: The overview is illustrated in “Neck” of Figure 2(c).
Revised: The overview is illustrated in “Neck” of Figure 2(c).

Original: In each deformable transformer layer, [ DET ] × [ DET ] attention is performed ﬁrst.
Revised: In each deformable transformer layer, [ DET ] × [ DET ] attention is performed ﬁrst.

Original: For each [ DET ] token, multi-scale deformable attention is applied to produce a new [ DET ] token, aggregating a small set of key contents sampled from the multi-scale feature maps { x l } Ll =1 ,
Revised: For each [ DET ] token, multi-scale deformable attention is applied to produce a new [ DET ] token, aggregating a small set of key contents sampled from the multi-scale feature maps { x l } Ll =1 ,

Original: MSDeformAttn([ DET ] , { x l } Ll =1 ) =
Revised: MSDeformAttn([ DET ] , { x l } Ll =1 ) =

Original: M (cid:88) m
Revised: M (cid:88) m

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:104)
Revised: (cid:104)

Original: [Equation]
Revised: [Equation]

Original: (cid:88) l=
Revised: (cid:88) l

Revised: [Equation]

Original: K (cid:88) k
Revised: K (cid:88) k

Original: =
Revised: =

Original: Amlk
Revised: A

Original: [Equation]
Revised: [Equation]

Original: W
Revised: W

Original: [Equation]
Revised: [Equation]

Original: where m indices the attention head and K is the total number of sampled keys for content aggregation.
Revised: where m indices the attention head and K is the total number of sampled keys for content aggregation.

Original: In addition, φ l ( p ) is the reference point of the [ DET ] token re-scaled for the l -th level feature map, while ∆ p mlk is the sampling offset for deformable attention; and A mlk is the attention weights of the K sampled contents.
Revised: In addition, φ l ( p ) is the reference point of the [ DET ] token re-scaled for the l -th level feature map, while ∆ p mlk is the sampling offset for deformable attention; and A mlk is the attention weights of the K sampled contents.

Original: W m and W (cid:48)
Revised: W m and W (cid:48)

Original: m are the projection matrices for multi-head attention.
Revised: m are the projection matrices for multi-head attention.

Original: Auxiliary Techniques for Additional Improvements.
Revised: Auxiliary Techniques for Additional Improvements.

Original: The decoder of ViDT follows the standard structure of multi-layer transformers, generating reﬁned [ DET ] tokens at each layer.
Revised: The decoder of ViDT follows the standard structure of multi-layer transformers, generating reﬁned [ DET ] tokens at each layer.

Original: Hence, ViDT leverages the two auxiliary techniques used in (Deformable) DETR for additional improvements:
Revised: Hence, ViDT leverages the two auxiliary techniques used in (Deformable) DETR for additional improvements:

Original: • Auxiliary Decoding Loss: Detection heads consisting of two feedforward networks (FNNs) for box regression and classiﬁcation are attached to every decoding layer.
Revised: • Auxiliary Decoding Loss: Detection heads consisting of two feedforward networks (FNNs) for box regression and classiﬁcation are attached to every decoding layer.

Original: All the training losses from detection heads at different scales are added to train the model.
Revised: All the training losses from detection heads at different scales are added to train the model.

Original: This helps the model output the correct number of objects without non-maximum suppression (Carion et al., 2020).
Revised: This helps the model output the correct number of objects without non-maximum suppression (Carion et al., 2020).

Original: • Iterative Box Reﬁnement: Each decoding layer reﬁnes the bounding boxes based on predictions from the detection head in the previous layer.
Revised: • Iterative Box Reﬁnement: Each decoding layer reﬁnes the bounding boxes based on predictions from the detection head in the previous layer.

Original: Therefore, the box regression process progressively improves through the decoding layers (Zhu et al., 2021).
Revised: Therefore, the box regression process progressively improves through the decoding layers (Zhu et al., 2021).

Original: These two techniques are essential for transformer-based object detectors because they signiﬁcantly enhance detection performance without compromising detection efﬁciency.
Revised: These two techniques are essential for transformer-based object detectors because they signiﬁcantly enhance detection performance without compromising detection efﬁciency.

Original: We provide an ablation study of their effectiveness for object detection in Section 4.3.1.
Revised: We provide an ablation study of their effectiveness for object detection in Section 4.3.1.

Original: 3. K NOWLEDGE D ISTILLATION WITH T OKEN M ATCHING FOR O BJECT D ETECTION
Revised: 3. K NOWLEDGE D ISTILLATION WITH T OKEN M ATCHING FOR O BJECT D ETECTION

Original: While a large model has a high capacity to achieve high performance, it can be computationally expensive for practical use.
Revised: While a large model has a high capacity to achieve high performance, it can be computationally expensive for practical use.

Original: As such, we additionally present a simple knowledge distillation approach that can transfer knowledge from the large ViDT model by token matching.
Revised: As such, we additionally present a simple knowledge distillation approach that can transfer knowledge from the large ViDT model by token matching.

Original: Based on the fact that all ViDT models has exactly the same number of [ PATCH ] and [ DET ] tokens regardless of their scale, a small ViDT model (a student model) can easily beneﬁt from a pre-trained large ViDT (a teacher model) by matching its tokens with those of the large one, thereby bringing out higher detection performance at a lower computational cost.
Revised: Based on the fact that all ViDT models has exactly the same number of [ PATCH ] and [ DET ] tokens regardless of their scale, a small ViDT model (a student model) can easily beneﬁt from a pre-trained large ViDT (a teacher model) by matching its tokens with those of the large one, thereby bringing out higher detection performance at a lower computational cost.

Original: Matching all the tokens at every layer is very inefﬁcient in training.
Revised: Matching all the tokens at every layer is very inefﬁcient in training.

Original: Thus, we only match the tokens contributing the most to prediction.
Revised: Thus, we only match the tokens contributing the most to prediction.

Original: The two sets of tokens are directly related: (1) P : the set of [ PATCH ] tokens used as multi-scale feature maps, which are generated from each stage in the body, and (2) D : the set of [ DET ] tokens, which are generated from each decoding layer in the neck.
Revised: The two sets of tokens are directly related: (1) P : the set of [ PATCH ] tokens used as multi-scale feature maps, which are generated from each stage in the body, and (2) D : the set of [ DET ] tokens, which are generated from each decoding layer in the neck.

Original: Accordingly, the distillation loss based on token matching is formulated by
Revised: Accordingly, the distillation loss based on token matching is formulated by

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: where the subscripts s and t refer to the student and teacher model.
Revised: where the subscripts s and t refer to the student and teacher model.

Original: P [ i ] and D
Revised: P [ i ] and D

Original: [ i ] return the i -th [ PATCH ] and [ DET ] tokens, n -dimensional vectors, belonging to P and D , respectively.
Revised: [ i ] return the i -th [ PATCH ] and [ DET ] tokens, n -dimensional vectors, belonging to P and D , respectively.

Original: λ dis is the coefﬁcient to determine the strength of (cid:96) dis , which is added to the detection loss if activated.
Revised: λ dis is the coefﬁcient to determine the strength of (cid:96) dis , which is added to the detection loss if activated.

Original:  E VALUATION
Revised:  E VALUATION

Original: In this section
Revised: In this section

Original: [Table]
Revised: [Table]

Original: we show that ViDT achieves the best trade-off between accuracy and speed (Section 4.1).
Revised: we show that ViDT achieves the best trade-off between accuracy and speed (Section 4.1).

Original: Then, we conduct detailed ablation study of the reconﬁgured attention module (Section 4.2) and additional techniques to boost detection performance (Section 4.3).
Revised: Then [Table] we conduct detailed ablation study of the reconﬁgured attention module (Section 4.2) and additional techniques to boost detection performance (Section 4.3).

Original: Finally, we provide a complete analysis of all components available for ViDT (Section 4.4).
Revised: Finally, we provide a complete analysis of all components available for ViDT (Section 4.4).

Original: Dataset.
Revised: Dataset.

Original: We carry out object detection experiments on the Microsoft COCO 2017 benchmark dataset (Lin et al., 2014).
Revised: We carry out object detection experiments on the Microsoft COCO 2017 benchmark dataset (Lin et al., 2014).

Original: All the fully transformer-based object detectors are trained on 118K training images and tested on 5K validation images following the literature (Carion et al., 2020).
Revised: All the fully transformer-based object detectors are trained on 118K training images and tested on 5K validation images following the literature (Carion et al., 2020).

Original: Algorithms.
Revised: Algorithms.

Original: We compare ViDT with two existing fully transformer-based object detection pipelines, namely DETR (ViT) and YOLOS.
Revised: We compare ViDT with two existing fully transformer-based object detection pipelines, namely DETR (ViT) and YOLOS.

Original: Since DETR (ViT) follows the general pipeline of (Deformable)
Revised: Since DETR (ViT) follows the general pipeline of (Deformable)

Original: DETR by replacing its ResNet backbone with other ViT variants; hence, we use one canonical ViT and one latest ViT variant, DeiT and Swin Transformer, as its backbone without any modiﬁcation.
Revised: DETR by replacing its ResNet backbone with other ViT variants; hence, we use one canonical ViT and one latest ViT variant, DeiT and Swin Transformer, as its backbone without any modiﬁcation.

Original: In contrast, YOLOS is the canonical ViT architecture, thus only DeiT is available.
Revised: In contrast, YOLOS is the canonical ViT architecture, thus only DeiT is available.

Original: Table 1 summarizes all the ViT models pre-trained on ImageNet used for evaluation.
Revised: Table 1 summarizes all the ViT models pre-trained on ImageNet used for evaluation.

Original: Note that publicly available pretrained models are used except for Swin-nano.
Revised: Note that publicly available pretrained models are used except for Swin-nano.

Original: We newly conﬁgure Swin-nano 3 comparable to DeiTtiny, which is trained on ImageNet with the identical setting.
Revised: We newly conﬁgure Swin-nano 3 comparable to DeiTtiny, which is trained on ImageNet with the identical setting.

Original: Overall, with respect to the number of parameters, Deit-tiny, -small, and -base are comparable to Swin-nano, -tiny, and -base, respectively.
Revised: Overall, with respect to the number of parameters, Deit-tiny, -small, and -base are comparable to Swin-nano, -tiny, and -base, respectively.

Original: Please see Appendix B.2 for the detailed pipeline of compared detectors.
Revised: Please see Appendix B.2 for the detailed pipeline of compared detectors.

Original: Implementation Details.
Revised: Implementation Details.

Original: All the algorithms are implemented using PyTorch and executed using eight NVIDIA Tesla V100 GPUs.
Revised: All the algorithms are implemented using PyTorch and executed using eight NVIDIA Tesla V100 GPUs.

Original: We train ViDT using AdamW (Loshchilov & Hutter, 2019) with the same initial learning rate of 10 − 4 for its body, neck and head.
Revised: We train ViDT using AdamW (Loshchilov & Hutter, 2019) with the same initial learning rate of 10 − 4 for its body, neck and head.

Original: In contrast, following the (Deformable) DETR setting, DETR (ViT) is trained with the initial learning rate of 10 − 5 for its pretrained body (ViT backbone) and 10 − 4 for its neck and head.
Revised: In contrast, following the (Deformable) DETR setting, DETR (ViT) is trained with the initial learning rate of 10 − 5 for its pretrained body (ViT backbone) and 10 − 4 for its neck and head.

Original: YOLOS and ViDT (w.o. Neck) are trained with the same initial learning rate of 5 × 10 − 5 , which is the original setting of YOLOS for the neck-free detector.
Revised: YOLOS and ViDT (w.o. Neck) are trained with the same initial learning rate of 5 × 10 − 5 , which is the original setting of YOLOS for the neck-free detector.

Original: We do not change any hyperparameters used in transformer encoder and decoder for (Deformable) DETR; thus, the neck decoder of ViDT also consists of six deformable transformer layers using exactly the same hyperparameters.
Revised: We do not change any hyperparameters used in transformer encoder and decoder for (Deformable) DETR; thus, the neck decoder of ViDT also consists of six deformable transformer layers using exactly the same hyperparameters.

Original: The only new hyperparameter introduced, the distillation coefﬁcient λ dis in Eq.
Revised: The only new hyperparameter introduced, the distillation coefﬁcient λ dis in Eq.

Original: (2), is set to be 4.
Revised: (2), is set to be 4.

Original: For fair comparison, knowledge distillation is not applied for ViDT in the main experiment in Section 4.1.
Revised: For fair comparison, knowledge distillation is not applied for ViDT in the main experiment in Section 4.1.

Original: The efﬁcacy of knowledge distillation with token matching is veriﬁed independently in Section 4.3.2.
Revised: The efﬁcacy of knowledge distillation with token matching is veriﬁed independently in Section 4.3.2.

Original: Auxiliary decoding loss and iterative box reﬁnement are applied to the compared methods if applicable.
Revised: Auxiliary decoding loss and iterative box reﬁnement are applied to the compared methods if applicable.

Original: Regarding the resolution of input images, we use scale augmentation that resizes them such that the shortest side is at least 480 and at most 800 pixels while the longest at most 1333 (Wu et al., 2019).
Revised: Regarding the resolution of input images, we use scale augmentation that resizes them such that the shortest side is at least 480 and at most 800 pixels while the longest at most 1333 (Wu et al., 2019).

Original: More details of the experiment conﬁguration can be found in Appendix B.3–B.5.
Revised: More details of the experiment conﬁguration can be found in Appendix B.3–B.5.

Original: All the source code and trained models will be made available to the public.
Revised: All the source code and trained models will be made available to the public at https://github.com/naver-ai/vidt.

Original: 4. M AIN E XPERIMENTS WITH M ICROSOFT COCO B ENCHMARK
Revised: 4. M AIN E XPERIMENTS WITH M ICROSOFT COCO B ENCHMARK

Original: Table 2 compares ViDT with DETR (ViT) and YOLOS w.r.t their AP, FPS, # parameters, where the two variants of DETR (ViT) are simply named DETR and Deformable DETR.
Revised: Table 2 compares ViDT with DETR (ViT) and YOLOS w.r.t their AP, FPS, # parameters, where the two variants of DETR (ViT) are simply named DETR and Deformable DETR.

Original: We report the result of ViDT without using knowledge distillation for fair comparison.
Revised: We report the result of ViDT without using knowledge distillation for fair comparison.

Original: A summary plot is provided in
Revised: A summary plot is provided in

Original: Figure 1.
Revised: Figure 1.

Original: The experimental comparisons with CNN backbones are provided in Appendix C.1.
Revised: The experimental comparisons with CNN backbones are provided in Appendix C.1.

Original: [Table]
Revised: [Table]

Original: Highlights.
Revised: Highlights.

Original: ViDT achieves the best trade-off between AP and FPS.
Revised: ViDT achieves the best trade-off between AP and FPS.

Original: With its high scalability, it performs well even for Swin-base of 0 .
Revised: With its high scalability, it performs well even for Swin-base of 0 .

Original: 1 billion parameters, which is 2x faster than Deformable DETR with similar AP.
Revised: 1 billion parameters, which is 2x faster than Deformable DETR with similar AP.

Original: Besides, ViDT shows 40 .
Revised: Besides, ViDT shows 40 .

Original: 4 AP only with 16M parameters; it is 6 . 3 – 12 .
Revised: 4 AP only with 16M parameters; it is 6 . 3 – 12 .

Original: 6 AP higher than those of DETR (swin-nano) and DETR (swin-tiny), which exhibit similar FPS of 19 . 3 – 24 . 7 .
Revised: 6 AP higher than those of DETR (swin-nano) and DETR (swin-tiny), which exhibit similar FPS of 19 . 3 – 24 . 7 .

Original: ViDT vs. Deformable DETR.
Revised: ViDT vs. Deformable DETR.

Original: Thanks to the use of multi-scale features, Deformable DETR exhibits high detection performance in general.
Revised: Thanks to the use of multi-scale features, Deformable DETR exhibits high detection performance in general.

Original: Nevertheless, its encoder and decoder structure in the neck becomes a critical bottleneck in computation.
Revised: Nevertheless, its encoder and decoder structure in the neck becomes a critical bottleneck in computation.

Original: In particular, the encoder with multi-layer deformable transformers adds considerable overhead to transform multi-scale features by attention.
Revised: In particular, the encoder with multi-layer deformable transformers adds considerable overhead to transform multi-scale features by attention.

Original: Thus, it shows very low FPS although it achieves higher AP with a relatively small number of parameters.
Revised: Thus, it shows very low FPS although it achieves higher AP with a relatively small number of parameters.

Original: In contrast, ViDT removes the need for a transformer encoder in the neck by using Swin Transformer with RAM as its body, directly extracting multi-scale features suitable for object detection.
Revised: In contrast, ViDT removes the need for a transformer encoder in the neck by using Swin Transformer with RAM as its body, directly extracting multi-scale features suitable for object detection.

Original: ViDT (w.o. Neck) vs. YOLOS.
Revised: ViDT (w.o. Neck) vs. YOLOS.

Original: For the comparison with YOLOS, we train ViDT without using its neck component.
Revised: For the comparison with YOLOS, we train ViDT without using its neck component.

Original: These two neck-free detectors show relatively low AP compared with other detectors in general.
Revised: These two neck-free detectors show relatively low AP compared with other detectors in general.

Original: In terms of speed, YOLOS exhibits much lower FPS than ViDT (w.o. Neck) because of its quadratic computational complexity for attention.
Revised: In terms of speed, YOLOS exhibits much lower FPS than ViDT (w.o. Neck) because of its quadratic computational complexity for attention.

Original: However, ViDT (w.o. Neck) extends
Revised: However, ViDT (w.o. Neck) extends

Original: Swin Transformers with RAM, thus requiring linear complexity for attention.
Revised: Swin Transformers with RAM, thus requiring linear complexity for attention.

Original: Hence, it shows AP comparable to YOLOS for various backbone size, but its FPS is much higher.
Revised: Hence, it shows AP comparable to YOLOS for various backbone size, but its FPS is much higher.

Original: One might argue that better integration could be also achieved by (1) Deformable DETR without its neck encoder because its neck decoder also has [ DET ] × [ PATCH ] cross-attention, or (2) YOLOS with VIDT’s neck decoder because of the use of multiple auxiliary techniques.
Revised: One might argue that better integration could be also achieved by (1) Deformable DETR without its neck encoder because its neck decoder also has [ DET ] × [ PATCH ] cross-attention, or (2) YOLOS with VIDT’s neck decoder because of the use of multiple auxiliary techniques.

Original: Such integration is actually not effective; the former signiﬁcantly drops AP, while the latter has a much greater drop in
Revised: Such integration is actually not effective; the former signiﬁcantly drops AP, while the latter has a much greater drop in

Original: FPS than an increase in AP.
Revised: FPS than an increase in AP.

Original: The detailed analysis can be found in Appendix C.2.
Revised: The detailed analysis can be found in Appendix C.2.

Original: 4. A BLATION S TUDY ON R ECONFIGURED A TTENTION M ODULE (RAM)
Revised: 4. A BLATION S TUDY ON R ECONFIGURED A TTENTION M ODULE (RAM)

Original: We extend Swin Transformer with RAM to extract ﬁne-grained features for object detection without maintaining an additional transformer encoder in the neck.
Revised: We extend Swin Transformer with RAM to extract ﬁne-grained features for object detection without maintaining an additional transformer encoder in the neck.

Original: We provide an ablation study on the two main considerations for RAM, which leads to high accuracy and speed.
Revised: We provide an ablation study on the two main considerations for RAM, which leads to high accuracy and speed.

Original: To reduce the inﬂuence of secondary factors, we mainly use our neck-free version, ViDT (w.o. Neck), for the ablation study.
Revised: To reduce the inﬂuence of secondary factors, we mainly use our neck-free version, ViDT (w.o. Neck), for the ablation study.

Original: 4.2. S PATIAL P OSITIONAL E NCODING
Revised: 4.2. S PATIAL P OSITIONAL E NCODING

Original: [Table]
Revised: [Table]

Original: Spatial positional encoding is essential for [ DET ] × [ PATCH ] attention in RAM.
Revised: Spatial positional encoding is essential for [ DET ] × [ PATCH ] attention in RAM.

Original: Typically, the spatial encoding can be added to the [ PATCH ] tokens before or after the projection layer in Figure 3.
Revised: Typically, the spatial encoding can be added to the [ PATCH ] tokens before or after the projection layer in Figure 3.

Original: We call the former “pre-addition” and the latter “postaddition”.
Revised: We call the former “pre-addition” and the latter “postaddition”.

Original: For each one, we can design the encoding in a sinusoidal or learnable manner (Carion et al., 2020).
Revised: For each one, we can design the encoding in a sinusoidal or learnable manner (Carion et al., 2020).

Original: Table 3 contrasts the results with different spatial positional encodings with ViDT (w.o.
Revised: Table 3 contrasts the results with different spatial positional encodings with ViDT (w.o.

Original: Neck).
Revised: Neck).

Original: Overall, pre-addition results in performance improvement higher than post-addition, and speciﬁcally, the sinusoidal encoding is better than the learnable one; thus, the 2D inductive bias of the sinusoidal spatial encoding is more helpful in object detection.
Revised: Overall, pre-addition results in performance improvement higher than post-addition, and speciﬁcally, the sinusoidal encoding is better than the learnable one; thus, the 2D inductive bias of the sinusoidal spatial encoding is more helpful in object detection.

Original: In particular, pre-addition with the sinusoidal encoding increases AP by 5 . 

Revised: In particular, pre-addition with the sinusoidal encoding increases AP by 5 .

Original: compared to not using any encoding.
Revised: compared to not using any encoding.

Original: 4.2. S ELECTIVE [ DET ] × [ PATCH ] C ROSS -A TTENTION
Revised: 4.2. S ELECTIVE [ DET ] × [ PATCH ] C ROSS -A TTENTION

Original: The addition of cross-attention to Swin Transformer inevitably entails computational overhead, particularly when the number of [ PATCH ] is large.
Revised: The addition of cross-attention to Swin Transformer inevitably entails computational overhead, particularly when the number of [ PATCH ] is large.

Original: To alleviate such overhead, we selectively enable cross-attention in RAM at the last stage of Swin Transformer; this is shown to greatly improve FPS, but barely drop AP.
Revised: To alleviate such overhead, we selectively enable cross-attention in RAM at the last stage of Swin Transformer; this is shown to greatly improve FPS, but barely drop AP.

Original: Table 4 summarizes AP and FPS when used different selective strategies for the cross-attention, where Swin Transformer consists of four stages in total.
Revised: Table 4 summarizes AP and FPS when used different selective strategies for the cross-attention, where Swin Transformer consists of four stages in total.

Original: It is interesting that all the strategies exhibit similar AP as long as cross-attention is activated at the last stage.
Revised: It is interesting that all the strategies exhibit similar AP as long as cross-attention is activated at the last stage.

Original: Since features are extracted in a bottom-up manner as they go through the stages, it seems difﬁcult to directly obtain useful information about the target object at the low level of stages.
Revised: Since features are extracted in a bottom-up manner as they go through the stages, it seems difﬁcult to directly obtain useful information about the target object at the low level of stages.

Original: Thus, only using the last stage is the best design choice in terms of high AP and FPS due to the smallest number of [ PATCH ] tokens.
Revised: Thus, only using the last stage is the best design choice in terms of high AP and FPS due to the smallest number of [ PATCH ] tokens.

Original: Meanwhile, the detection fails completely or the performance signiﬁcantly drops if all the stages are not involved due to the lack of interaction between [ DET ] and [ PATCH ] tokens that spatial positional encoding is associated with.
Revised: Meanwhile, the detection fails completely or the performance signiﬁcantly drops if all the stages are not involved due to the lack of interaction between [ DET ] and [ PATCH ] tokens that spatial positional encoding is associated with.

Original: A more detailed analysis of the [ DET ] × [ PATCH ] cross-attention
Revised: A more detailed analysis of the [ DET ] × [ PATCH ] cross-attention

Original: and [ DET ] × [ DET ] self-attention is provided in appendices C.3 and C.4.
Revised: and [ DET ] × [ DET ] self-attention is provided in appendices C.3 and C.4.

Original: [Table]
Revised: [Table]

Original: 4. A BLATION S TUDY ON A DDITIONAL T ECHNIQUES
Revised: 4. A BLATION S TUDY ON A DDITIONAL T ECHNIQUES

Original: We analyze the performance improvement of two additional techniques, namely auxiliary decoding loss and iterative box reﬁnement [Table] and the proposed distillation approach in Section 3.3.
Revised: We analyze the performance improvement of two additional techniques, namely auxiliary decoding loss and iterative box reﬁnement, and the proposed distillation approach in Section 3.3.

Original: Furthermore, we introduce a simple technique that can expedite the inference speed of ViDT by dropping unnecessary decoding layers at inference time.
Revised: Furthermore, we introduce a simple technique that can expedite the inference speed of ViDT by dropping unnecessary decoding layers at inference time.

Original: 4.3. A UXILIARY D ECODING L OSS AND I TERATIVE B OX R EFINEMENT
Revised: 4.3. A UXILIARY D ECODING L OSS AND I TERATIVE B OX R EFINEMENT

Original: [Table]
Revised: [Table]

Original: T
Revised: T

Original: [Table]
Revised: [Table]

Original: To thoroughly verify the efﬁcacy of auxiliary decoding loss and iterative box reﬁnement, we extend them even for the neck-free detector like YOLOS; the principle of them is applied to the encoding layers in the body, as opposed to the conventional way of using the decoding layers in the neck.
Revised: To thoroughly verify the efﬁcacy of auxiliary decoding loss and iterative box reﬁnement, we extend them even for the neck-free detector like YOLOS; the principle of them is applied to the encoding layers in the body, as opposed to the conventional way of using the decoding layers in the neck.

Original: Table 5 shows the performance of the two neck-free detectors, YOLOS and ViDT (w.o. Neck), decreases considerably with the two techniques.
Revised: Table 5 shows the performance of the two neck-free detectors, YOLOS and ViDT (w.o. Neck), decreases considerably with the two techniques.

Original: The use of them in the encoding layers is likely to negatively affect feature extraction of the transformer encoder.
Revised: The use of them in the encoding layers is likely to negatively affect feature extraction of the transformer encoder.

Original: In contrast, an opposite trend is observed with the neck component.
Revised: In contrast, an opposite trend is observed with the neck component.

Original: Since the neck decoder is decoupled with the feature extraction in the body, the two techniques make a synergistic effect and thus show signiﬁcant improvement in AP.
Revised: Since the neck decoder is decoupled with the feature extraction in the body, the two techniques make a synergistic effect and thus show signiﬁcant improvement in AP.

Original: These results justify the use of the neck decoder in ViDT to boost object detection performance.
Revised: These results justify the use of the neck decoder in ViDT to boost object detection performance.

Original: 4.3. K NOWLEDGE D ISTILLATION WITH T OKEN M ATCHING
Revised: 4.3. K NOWLEDGE D ISTILLATION WITH T OKEN M ATCHING

Original: [Table]
Revised: [Table]

Original: We show that a small ViDT model can beneﬁt from a large ViDT model via knowledge distillation.
Revised: We show that a small ViDT model can beneﬁt from a large ViDT model via knowledge distillation.

Original: The proposed token matching is a new concept of knowledge distillation for object detection, especially for a fully transformer-based object detector.
Revised: The proposed token matching is a new concept of knowledge distillation for object detection, especially for a fully transformer-based object detector.

Original: Compared to very complex distillation methods that rely on heuristic rules with multiple hyperparameters (Chen et al., 2017; Dai et al., 2021), it simply matches some tokens with a single hyperparameter, the distillation coefﬁcient λ dis .
Revised: Compared to very complex distillation methods that rely on heuristic rules with multiple hyperparameters (Chen et al., 2017; Dai et al., 2021), it simply matches some tokens with a single hyperparameter, the distillation coefﬁcient λ dis .

Original: Table 6 summarizes the AP improvement via knowledge distillation with token matching with varying distillation coefﬁcients.
Revised: Table 6 summarizes the AP improvement via knowledge distillation with token matching with varying distillation coefﬁcients.

Original: Overall, the larger the size of the teacher model, the greater gain to the student model.
Revised: Overall, the larger the size of the teacher model, the greater gain to the student model.

Original: Regarding coefﬁcients, in general, larger values achieve better performance.
Revised: Regarding coefﬁcients, in general, larger values achieve better performance.

Original: Distillation increases AP by 1 . 0 – 1 . 7 without affecting the inference speed of the student model
Revised: Distillation increases AP by 1 . 0 – 1 . 7 without affecting the inference speed of the student model

Original: [Table]
Revised: [Table]

Original: 4.3. D ECODING L AYER D 
Revised: 4.3. D ECODING L AYER D ROP

Original: [Table]
Revised: [Table]

Original: ViDT has six layers of transformers as its neck decoder.
Revised: ViDT has six layers of transformers as its neck decoder.

Original: We emphasize that not all layers of the decoder are required at inference time for high performance.
Revised: We emphasize that not all layers of the decoder are required at inference time for high performance.

Original: Table 7 show the performance of
Revised: Table 7 show the performance of

Original: ViDT when dropping its decoding layer one by one from the top in the inference step.
Revised: ViDT when dropping its decoding layer one by one from the top in the inference step.

Original: Although there is a trade-off relationship between accuracy and speed as the layers are detached from the model, there is no signiﬁcant AP drop even when the two layers are removed.
Revised: Although there is a trade-off relationship between accuracy and speed as the layers are detached from the model, there is no signiﬁcant AP drop even when the two layers are removed.

Original: This technique is not designed for performance evaluation in Tablewith other methods, but we can accelerate the inference speed of a trained ViDT model to over 10% by dropping its two decoding layers without a much decrease in AP.
Revised: This technique is not designed for performance evaluation in Tablewith other methods, but we can accelerate the inference speed of a trained ViDT model to over 10% by dropping its two decoding layers without a much decrease in AP.

Original: 4. C OMPLETE C OMPONENT A NALYSIS
Revised: 4. C OMPLETE C OMPONENT A NALYSIS

Original: In this section, we combine all the proposed components (even with distillation and decoding layer drop) to achieve high accuracy and speed for object detection.
Revised: In this section, we combine all the proposed components (even with distillation and decoding layer drop) to achieve high accuracy and speed for object detection.

Original: As summarized in Table 8, there are four components: (1) RAM to extend Swin Transformer as a standalone object detector, (2) the neck decoder to exploit multi-scale features with two auxiliary techniques, (3) knowledge distillation to beneﬁt from a large model, and (4) decoding layer drop to further accelerate inference speed.
Revised: As summarized in Table 8, there are four components: (1) RAM to extend Swin Transformer as a standalone object detector, (2) the neck decoder to exploit multi-scale features with two auxiliary techniques, (3) knowledge distillation to beneﬁt from a large model, and (4) decoding layer drop to further accelerate inference speed.

Original: The performance of the ﬁnal version is very outstanding; it achieves 41.7AP with reasonable FPS by only using 13 M parameters when used Swin-nano as its backbone.
Revised: The performance of the ﬁnal version is very outstanding; it achieves 41.7AP with reasonable FPS by only using 13 M parameters when used Swin-nano as its backbone.

Original: Further, it only loses 2.7 FPS while exhibiting 46.4AP when used Swin-tiny.
Revised: Further, it only loses 2.7 FPS while exhibiting 46.4AP when used Swin-tiny.

Original: This indicates that a fully transformer-based object detector has the potential to be used as a generic object detector when further developed in the future.
Revised: This indicates that a fully transformer-based object detector has the potential to be used as a generic object detector when further developed in the future.

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have explored the integration of vision and detection transformers to build an effective and efﬁcient object detector.
Revised: We have explored the integration of vision and detection transformers to build an effective and efﬁcient object detector.

Original: The proposed ViDT signiﬁcantly improves the scalability and ﬂexibility of transformer models to achieve high accuracy and inference speed.
Revised: The proposed ViDT signiﬁcantly improves the scalability and ﬂexibility of transformer models to achieve high accuracy and inference speed.

Original: The computational complexity of its attention modules is linear w.r
Revised: The computational complexity of its attention modules is linear w.r.t.

Original: [Table]

Original: t. image size
Revised: image size

Original: [Table]
Revised: [Table]

Original: and ViDT synergizes several essential techniques to boost the detection performance.
Revised: and ViDT synergizes several essential techniques to boost the detection performance.

Original: On the Microsoft COCO benchmark, ViDT achieves 49 .
Revised: On the Microsoft COCO benchmark, ViDT achieves 49 .

Original: 2 AP with a large Swin-base backbone
Revised: 2 AP with a large Swin-base backbone

Original: [Table]
Revised: [Table]

Original: and 41 .
Revised: and 41 .

Original: 7 AP with the smallest Swin-nano backbone and only 13M parameters, suggesting the beneﬁts of using transformers for complex computer vision tasks.
Revised: 7 AP with the smallest Swin-nano backbone and only 13M parameters, suggesting the beneﬁts of using transformers for complex computer vision tasks.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: This paper deals with the topic of general object detection in computer vision.
Revised: This paper deals with the topic of general object detection in computer vision.

Original: We propose a novel integration of vision and detection transformers for a fully transformer-based object detector.
Revised: We propose a novel integration of vision and detection transformers for a fully transformer-based object detector.

Original: Therefore, we do not expect any potential negative social impact of our work.
Revised: Therefore, we do not expect any potential negative social impact of our work.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: For reproducibility we provide a detailed description of our experiment and hyperparameter settings in Appendix B.
Revised: For reproducibility we provide a detailed description of our experiment and hyperparameter settings in Appendix B.

Original: It includes the Swin-nano architecture (Appendix B.1), the pipelines of all compared object detectors (Appendix B.2), hyperparameters of neck transformers (Appendix B.3), detailed implementation (Appendix B.4) and training conﬁguration (Appendix B
Revised: It includes the Swin-nano architecture (Appendix B.1), the pipelines of all compared object detectors (Appendix B.2), hyperparameters of neck transformers (Appendix B.3), detailed implementation (Appendix B.4) and training conﬁguration (Appendix B

Original: We will release the code and trained models upon acceptance
Revised: We will release the code and trained models upon acceptance.

Revised: ACKNOWLEDGMENTS

Revised: We thank NAVER AI Lab members for valuable discussion and advice.

Revised: NAVER Smart Machine

Revised: Learning (NSML) (Kim et al., 2018) has been used for experiment.

Revised: M.-H. Yang is supported in part by the NSF CAREER grant

Original: A R ECONFIGURED A TTENTION M ODULE
Revised: A R ECONFIGURED A TTENTION M ODULE

Original: The proposed RAM in Figure 3 performs three attention operations, namely [ PATCH ] × [ PATCH ] , [ DET ] × [ PATCH ] , and [ DET ] × [ DET ] attention.
Revised: The proposed RAM in Figure 3 performs three attention operations, namely [ PATCH ] × [ PATCH ] , [ DET ] × [ PATCH ] , and [ DET ] × [ DET ] attention.

Original: This section provides (1) computational complexity analysis and (2) further algorithmic design for [ DET ] tokens.
Revised: This section provides (1) computational complexity analysis and (2) further algorithmic design for [ DET ] tokens.

Original: A. C OMPUTATIONAL C OMPLEXITY A NALYSIS
Revised: A. C OMPUTATIONAL C OMPLEXITY A NALYSIS

Original: We analyze the computational complexity of the proposed RAM compared with the attention used in YOLOS.
Revised: We analyze the computational complexity of the proposed RAM compared with the attention used in YOLOS.

Original: The analysis is based on the computational complexity of basic building blocks for
Revised: The analysis is based on the computational complexity of basic building blocks for

Original: Canonical and Swin Transformer, which is summarized in Table 9 4 , where T 1 and T 2 is the number of tokens for self- and cross-attention, and d is the embedding dimension.
Revised: Canonical and Swin Transformer, which is summarized in Table 9 4 , where T 1 and T 2 is the number of tokens for self- and cross-attention, and d is the embedding dimension.

Original: [Table]
Revised: [Table]

Original: Complexity
Revised: [Table] [Equation] [Table] [Equation]

Original: O (
Revised: [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: O
Revised: [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: Let P and D be the number of [ PATCH ] and [ DET ] tokens ( D << P in practice, e.g., P = 66 , 650 and
Revised: Let P and D be the number of [ PATCH ] and [ DET ] tokens ( D << P in practice, e.g., P = 66 , 650 and

Original: D = 100 at the ﬁrst stage of ViDT).
Revised: D = 100 at the ﬁrst stage of ViDT).

Original: Then, the computational complexity of the attention module for
Revised: Then, the computational complexity of the attention module for

Original: YOLOS and ViDT (RAM) is derived as below, also summarized in Table 10:
Revised: YOLOS and ViDT (RAM) is derived as below, also summarized in Table 10:

Original: • YOLOS Attention : [ DET ] tokens are simply appended to [ PATCH ] tokens to perform global selfattention on [ PATCH , DET ] tokens (i.e., T 1 = P + D ).
Revised: • YOLOS Attention : [ DET ] tokens are simply appended to [ PATCH ] tokens to perform global selfattention on [ PATCH , DET ] tokens (i.e., T 1 = P + D ).

Original: Thus, the computational complexity is O
Revised: Thus, the computational complexity is O

Original: ( d 2 ( P + D ) + d ( P + D ) 2 ) , which is quadratic to the number of [ PATCH ] tokens.
Revised: ( d 2 ( P + D ) + d ( P + D ) 2 ) , which is quadratic to the number of [ PATCH ] tokens.

Original: If breaking down the total complexity, we obtain O (cid:0) ( d 2 P + d P 2 ) + ( d 2 D + d D 2 ) + d PD (cid:1) , where the ﬁrst and second terms are for the global self-attention for [ PATCH ] and [ DET ] tokens, respectively
Revised: If breaking down the total complexity, we obtain O (cid:0) ( d 2 P + d P 2 ) + ( d 2 D + d D 2 ) + d PD (cid:1) , where the ﬁrst and second terms are for the global self-attention for [ PATCH ] and [ DET ] tokens, respectively

Original: and the last term is for the global cross-attention between them.
Revised: and the last term is for the global cross-attention between them.

Original: • ViDT (RAM) Attention : RAM performs the three different attention operations: (1) [ PATCH ] × [ PATCH ] local self-attention with window partition, O ( d 2 P + dk 2 P ) ; (2) [ DET ] × [ DET ] global selfattention, O ( d 2 D + d D 2 ) ; (3) [ DET ] × [ PATCH ] global cross-attention, O ( d 2 ( D + P ) + d DP ) .
Revised: • ViDT (RAM) Attention : RAM performs the three different attention operations: (1) [ PATCH ] × [ PATCH ] local self-attention with window partition, O ( d 2 P + dk 2 P ) ; (2) [ DET ] × [ DET ] global selfattention, O ( d 2 D + d D 2 ) ; (3) [ DET ] × [ PATCH ] global cross-attention, O ( d 2 ( D + P ) + d DP ) .

Original: In total, the computational complexity of RAM is O ( d 2 ( D + P ) + dk 2 P + d D 2 + d DP ) , which is linear to the number of [ PATCH ] tokens.
Revised: In total, the computational complexity of RAM is O ( d 2 ( D + P ) + dk 2 P + d D 2 + d DP ) , which is linear to the number of [ PATCH ] tokens.

Original: Consequently, the complexity of RAM is much lower than the attention module used in YOLOS since D << P .
Revised: Consequently, the complexity of RAM is much lower than the attention module used in YOLOS since D << P .

Original: Note that only RAM achieves the linear complexity to the patch tokens.
Revised: Note that only RAM achieves the linear complexity to the patch tokens.

Original: In addition, one might argue that YOLOS can be efﬁcient if the cross-attention is selectively removed similar to
Revised: In addition, one might argue that YOLOS can be efﬁcient if the cross-attention is selectively removed similar to

Original: RAM.
Revised: RAM.

Original: Even if we remove the complexity O ( d PD ) for the global cross-attention, the computational complexity is O
Revised: Even if we remove the complexity O ( d PD ) for the global cross-attention, the computational complexity is O

Original: ( d 2 ( P + D )+ d P 2 + d D 2 ) , which is still quadratic to the number of [ PATCH ] tokens.
Revised: ( d 2 ( P + D )+ d P 2 + d D 2 ) , which is still quadratic to the number of [ PATCH ] tokens.

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: A. A LGORITHMIC D ESIGN FOR [ DET ] T OKENS A.2. B INDING [ DET ] × [ DET ] AND 
Revised: A. A LGORITHMIC D ESIGN FOR [ DET ] T OKENS A.2. B INDING [ DET ] × [ DET ] AND [ DET ] × [ PATCH ] ATTENTION

Original: DET ] × [ PATCH ] ATTENTION

Original: Binding the two attention modules is very simple in implementation.
Revised: Binding the two attention modules is very simple in implementation.

Original: [ DET ] × [ DET ] and [ DET ] × [ PATCH ] attention is generating a new [ DET ] token, which aggregates relevant contents in [ DET ] and [ PATCH ] tokens, respectively.
Revised: [ DET ] × [ DET ] and [ DET ] × [ PATCH ] attention is generating a new [ DET ] token, which aggregates relevant contents in [ DET ] and [ PATCH ] tokens, respectively.

Original: Since the two attention share exactly the same [ DET ] query embedding obtained after the projection as shown in Figure 3, they can be processed at once by performing the scaled-dot product between [ DET ] Q and (cid:2)
Revised: Since the two attention share exactly the same [ DET ] query embedding obtained after the projection as shown in Figure 3, they can be processed at once by performing the scaled-dot product between [ DET ] Q and (cid:2)

Original: [ DET ] K , [ PATCH ] K (cid:3) embeddings, where Q , K are the key and query, and [ · ] is the concatenation.
Revised: [ DET ] K , [ PATCH ] K (cid:3) embeddings, where Q , K are the key and query, and [ · ] is the concatenation.

Original: Then, the obtained attention map is applied to the (cid:2)
Revised: Then, the obtained attention map is applied to the (cid:2)

Original: [ DET ] V , [ PATCH ] V (cid:3) embeddings, where V is the value and d is the embedding dimension,[Equation]
Revised: [ DET ] V , [ PATCH ] V (cid:3) embeddings, where V is the value and d is the embedding dimension,

Original: [Equation]
Revised: [Equation]

Original: This approach is commonly used in the recent Transformer-based architectures, such as YOLOS.
Revised: This approach is commonly used in the recent Transformer-based architectures, such as YOLOS.

Original: A.2. E MBEDDING D IMENSION OF [ DET ] TOKENS
Revised: A.2. E MBEDDING D IMENSION OF [ DET ] TOKENS

Original: [ DET ] × [ DET ] attention is performed across all the stages, and the embedding dimension of [ DET ] tokens increases gradually like [ PATCH ] tokens.
Revised: [ DET ] × [ DET ] attention is performed across all the stages, and the embedding dimension of [ DET ] tokens increases gradually like [ PATCH ] tokens.

Original: For the [ PATCH ] token, its embedding dimension is increased by concatenating nearby [ PATCH ] tokens in a grid.
Revised: For the [ PATCH ] token, its embedding dimension is increased by concatenating nearby [ PATCH ] tokens in a grid.

Original: However, this mechanism is not applicable for [ DET ] tokens since we maintain the same number of [ DET ] tokens for detecting a ﬁxed number of objects in a scene.
Revised: However, this mechanism is not applicable for [ DET ] tokens since we maintain the same number of [ DET ] tokens for detecting a ﬁxed number of objects in a scene.

Original: Hence, we simply repeat a [ DET ] token multiple times along the embedding dimension to increase its size.
Revised: Hence, we simply repeat a [ DET ] token multiple times along the embedding dimension to increase its size.

Original: This allows [ DET ] tokens to reuse all the projection and normalization layers in Swin Transformer without any modiﬁcation.
Revised: This allows [ DET ] tokens to reuse all the projection and normalization layers in Swin Transformer without any modiﬁcation.

Original: B E XPERIMENTAL D ETAILS B. 
Revised: B E XPERIMENTAL D ETAILS B.

Original: SWIN -
Revised: S

Original: [Table]
Revised: [Table]

Original: A RCHITECTURE
Revised: A RCHITECTURE

Original: [Table]
Revised: [Table]

Original: Due to the absence of Swin models comparable to
Revised: Due to the absence of Swin models comparable to

Original: Deit-tiny, we conﬁgure Swin-nano, which is a 0.25 × model of Swin-tiny such that it has 6 M training parameters comparable to Deit-tiny.
Revised: Deit-tiny, we conﬁgure Swin-nano, which is a 0.25 × model of Swin-tiny such that it has 6 M training parameters comparable to Deit-tiny.

Original: Table 11 summarizes the conﬁguration of Swin Transformer models available, including the newly introduced Swinnano; S1–S4 indicates the four stages in Swin Transformer.
Revised: Table 11 summarizes the conﬁguration of Swin Transformer models available, including the newly introduced Swinnano; S1–S4 indicates the four stages in Swin Transformer.

Original: The performance of all the pre-trained Swin
Revised: The performance of all the pre-trained Swin

Original: Transformer models are summarized in Table 1 in the manuscript.
Revised: Transformer models are summarized in Table 1 in the manuscript.

Original: B. D ETECTION P IPELINES OF A LL C OMPARED D ETECTORS
Revised: B. D ETECTION P IPELINES OF A LL C OMPARED D ETECTORS

Original: All the compared fully transformer-based detectors are composed of either (1) body–neck–head or (2) body–head structure, as summarized in Table 12.
Revised: All the compared fully transformer-based detectors are composed of either (1) body–neck–head or (2) body–head structure, as summarized in Table 12.

Original: The main difference of ViDT is the use of reconﬁgured attention modules (RAM) for Swin Transformer, allowing the extraction of ﬁne-grained detection features directly from the input image.
Revised: The main difference of ViDT is the use of reconﬁgured attention modules (RAM) for Swin Transformer, allowing the extraction of ﬁne-grained detection features directly from the input image.

Original: Thus, Swin Transformer is extended to a standalone object detector called ViDT (w.o. Neck).
Revised: Thus, Swin Transformer is extended to a standalone object detector called ViDT (w.o. Neck).

Original: Further, its extension to ViDT allows to use multi-scale features and multiple essential techniques for better detection, such as auxiliary decoding loss and iterative box reﬁnement, by only maintaining a transformer decoder at the neck.
Revised: Further, its extension to ViDT allows to use multi-scale features and multiple essential techniques for better detection, such as auxiliary decoding loss and iterative box reﬁnement, by only maintaining a transformer decoder at the neck.

Original: Except for the two neck-free detector, YOLOS and ViDT (w.o. Neck), all the pipelines maintain multiple FFNs; that is, a single FFNs for each decoding layer at the neck for box regression and classiﬁcation.
Revised: Except for the two neck-free detector, YOLOS and ViDT (w.o. Neck), all the pipelines maintain multiple FFNs; that is, a single FFNs for each decoding layer at the neck for box regression and classiﬁcation.

Original: We believe that our proposed RAM can be combined with even other latest efﬁcient vision transformer architectures, such as PiT (Heo et al., 2021), PVT (Wang et al., 2021) and Cross-ViT
Revised: We believe that our proposed RAM can be combined with even other latest efﬁcient vision transformer architectures, such as PiT (Heo et al., 2021), PVT (Wang et al., 2021) and Cross-ViT

Original: (Chen et al., 2021).
Revised: (Chen et al., 2021).

Original: We leave this as future work.
Revised: We leave this as future work.

Original: B. H YPERPARAMETERS OF N ECK T RANSFORMERS
Revised: B. H YPERPARAMETERS OF N ECK T RANSFORMERS

Original: The transformer decoder at the neck in ViDT introduces multiple hyperparameters.
Revised: The transformer decoder at the neck in ViDT introduces multiple hyperparameters.

Original: We follow exactly the same setting used in Deformable DETR.
Revised: We follow exactly the same setting used in Deformable DETR.

Original: Speciﬁcally, we use six layers of deformable
Revised: Speciﬁcally, we use six layers of deformable

Original: [Table]
Revised: [Table]

Original: transformers with width 256; thus, the channel dimension of the [ PATCH ] and [ DET ] tokens extracted from Swin Transformer are reduced to 256 to be utilized as compact inputs to the decoder transformer.
Revised: transformers with width 256; thus, the channel dimension of the [ PATCH ] and [ DET ] tokens extracted from Swin Transformer are reduced to 256 to be utilized as compact inputs to the decoder transformer.

Original: For each transformer layer, multi-head attention with eight heads is applied, followed by the point-wise FFNs of 1024 hidden units.
Revised: For each transformer layer, multi-head attention with eight heads is applied, followed by the point-wise FFNs of 1024 hidden units.

Original: Furthermore, an additive dropout of 0.1 is applied before the layer normalization.
Revised: Furthermore, an additive dropout of 0.1 is applied before the layer normalization.

Original: All the weights in the decoder are initialized with Xavier initialization.
Revised: All the weights in the decoder are initialized with Xavier initialization.

Original: For (Deformable) DETR
Revised: For (Deformable) DETR

Original: [Table]
Revised: [Table]

Original: the tranformer decoder receives a ﬁxed number of learnable detection tokens.
Revised: the tranformer decoder receives a ﬁxed number of learnable detection tokens

Revised: [Table]

Original: We set the number of detection tokens to
Revised: We set the number of detection tokens to

Original: [Table]
Revised: [Table]

Original: which is the same number used for YOLOS and ViDT
Revised: which is the same number used for YOLOS and ViDT

Original: [Table]
Revised: [Table]

Original: B. I MPLEMENTATION B.4. D ETECTION H EAD FOR P REDICTION
Revised: B. I MPLEMENTATION B.4. D ETECTION H EAD FOR P REDICTION

Original: The last [ DET ] tokens produced by the body or neck are fed to a 3 -layer FFNs for bounding box regression and linear projection for classiﬁcation,
Revised: The last [ DET ] tokens produced by the body or neck are fed to a 3 -layer FFNs for bounding box regression and linear projection for classiﬁcation,

Original: ˆ B = FFN 3-layer (cid:0) [ DET ] (cid:1) and ˆ P =
Revised: ˆ B = FFN 3-layer (cid:0) [ DET ] (cid:1) and ˆ P =

Original: Linear (cid:0) [ DET ] (cid:1) .
Revised: Linear (cid:0) [ DET ] (cid:1) .

Original: For box regression, the FFNs produce the bounding box coordinates for d objects, ˆ B ∈ [0 , 1] d × 4 , that encodes the normalized box center coordinates along with its width and height.
Revised: For box regression, the FFNs produce the bounding box coordinates for d objects, ˆ B ∈ [0 , 1] d × 4 , that encodes the normalized box center coordinates along with its width and height.

Original: For classiﬁcation, the linear projection uses a softmax function to produce the classiﬁcation probabilities for all possible classes including the background class, ˆ P ∈ [0 , 1] d × ( c +1) , where c is the number of object classes.
Revised: For classiﬁcation, the linear projection uses a softmax function to produce the classiﬁcation probabilities for all possible classes including the background class, ˆ P ∈ [0 , 1] d × ( c +1) , where c is the number of object classes.

Original: When deformable attention is used on the neck in Table 12, only c classes are considered without the background class for classiﬁcation.
Revised: When deformable attention is used on the neck in Table 12, only c classes are considered without the background class for classiﬁcation.

Original: This is the original setting used in DETR, YOLOS (Carion et al., 2020; Fang et al., 2021) and Deformable DETR (Zhu et al., 2021).
Revised: This is the original setting used in DETR, YOLOS (Carion et al., 2020; Fang et al., 2021) and Deformable DETR (Zhu et al., 2021).

Original: B.4. L OSS F UNCTION FOR T RAINING
Revised: B.4. L OSS F UNCTION FOR T RAINING

Original: All the methods adopts the loss function of (Deformable) DETR.
Revised: All the methods adopts the loss function of (Deformable) DETR.

Original: Since the detection head return a ﬁxed-size set of d bounding boxes, where d is usually larger than the number of actual objects in an image, Hungarian matching is used to ﬁnd a bipartite matching between the predicted box ˆ B and the ground-truth box B .
Revised: Since the detection head return a ﬁxed-size set of d bounding boxes, where d is usually larger than the number of actual objects in an image, Hungarian matching is used to ﬁnd a bipartite matching between the predicted box ˆ B and the ground-truth box B .

Original: In total, there are three types of training loss: a classiﬁcation loss (cid:96) cl 5 , a box distance (cid:96) l 1 , and a GIoU loss (cid:96) iou (Rezatoﬁghi et al., 2019), (cid:96) cl
Revised: In total, there are three types of training loss: a classiﬁcation loss (cid:96) cl 5 , a box distance (cid:96) l 1 , and a GIoU loss (cid:96) iou (Rezatoﬁghi et al., 2019), (cid:96) cl

Original: ( i ) =
Revised: ( i ) =

Original: − log ˆ P σ ( i ) ,c i , (cid:96) (cid:96) 1 ( i ) =
Revised: − log ˆ P σ ( i ) ,c i , (cid:96) (cid:96) 1 ( i ) =

Original: ||
Revised: ||

Original: B i − ˆ B σ ( i )
Revised: B i − ˆ B σ ( i )

Original: || 1 , and (cid:96) iou ( i )
Revised: || 1 , and (cid:96) iou ( i )

Original: =
Revised: =

Original: 1 − (cid:0) | B i ∩ ˆ B σ ( i ) | | B i ∪ ˆ B σ ( i ) | − | B
Revised: 1 − (cid:0) | B i ∩ ˆ B σ ( i ) | | B i ∪ ˆ B σ ( i ) | − | B

Original: ( B i , ˆ B σ ( i ) )
Revised: ( B i , ˆ B σ ( i ) )

Original: \ B i ∪ ˆ B σ ( i ) | | B ( B i 
Revised: \ B i ∪ ˆ B σ ( i ) | | B ( B i ,

Original: [Equation]
Revised: [Equation]

Original: where c i and σ ( i ) are the target class label and bipartite assignment of the i -th ground-truth box, and B returns the largest box containing two given boxes.
Revised: where c i and σ ( i ) are the target class label and bipartite assignment of the i -th ground-truth box, and B returns the largest box containing two given boxes.

Original: Thus, the ﬁnal loss of object detection is a linear combination of the three types of training loss,
Revised: Thus, the ﬁnal loss of object detection is a linear combination of the three types of training loss,

Original: [Equation]
Revised: [Equation]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: The coefﬁcient for each training loss is set to be λ cl = 1 , λ (cid:96) 1 = 5 , and λ iou = 2 .
Revised: The coefﬁcient for each training loss is set to be λ cl = 1 , λ (cid:96) 1 = 5 , and λ iou = 2 .

Original: If we leverage auxiliary decoding loss, the ﬁnal loss is computed for every detection head separately and merged with equal importance.
Revised: If we leverage auxiliary decoding loss, the ﬁnal loss is computed for every detection head separately and merged with equal importance.

Original: Additionally, ViDT adds the distillation loss in Eq.
Revised: Additionally, ViDT adds the distillation loss in Eq.

Original: (2) to the ﬁnal loss if the distillation approach in Section 3.3 is enabled for training.
Revised: (2) to the ﬁnal loss if the distillation approach in Section 3.3 is enabled for training.

Original: B. T RAINING C ONFIGURATION
Revised: B. T RAINING C ONFIGURATION

Original: We train ViDT for 50 epochs using AdamW (Loshchilov & Hutter, 2019) with the same initial learning rate of 10 − 4 for its body, neck and head.
Revised: We train ViDT for 50 epochs using AdamW (Loshchilov & Hutter, 2019) with the same initial learning rate of 10 − 4 for its body, neck and head.

Original: The learning rate is decayed by cosine annealing with batch size of 16 , weight decay of 1 × 10 − 4 , and gradient clipping of 0 .
Revised: The learning rate is decayed by cosine annealing with batch size of 16 , weight decay of 1 × 10 − 4 , and gradient clipping of 0 .

Original: In contrast, ViDT (w.o.
Revised: In contrast, ViDT (w.o.

Original: Neck) is trained for 150 epochs using AdamW with the initial learning rate of 5 × 10 − 5 by cosine annealing.
Revised: Neck) is trained for 150 epochs using AdamW with the initial learning rate of 5 × 10 − 5 by cosine annealing.

Original: The remaining conﬁguration is the same as for ViDT.
Revised: The remaining conﬁguration is the same as for ViDT.

Original: Regarding DETR (ViT), we follow the setting of Deformable DETR.
Revised: Regarding DETR (ViT), we follow the setting of Deformable DETR.

Original: Thus, all the variants of this pipeline are trained for 50 epochs with the initial learning rate of 10 − 5 for its pre-trained body (ViT backbone) and 10 − 4 for its neck and head.
Revised: Thus, all the variants of this pipeline are trained for 50 epochs with the initial learning rate of 10 − 5 for its pre-trained body (ViT backbone) and 10 − 4 for its neck and head.

Original: Their learning rates are decayed at the 40-th epoch by a factor of 0.1.
Revised: Their learning rates are decayed at the 40-th epoch by a factor of 0.1.

Original: Meanwhile, the results of YOLOS are borrowed from the original paper (Fang et al., 2021) except YOLOS (DeiT-tiny); since the result of YOLOS (DeiT-tiny) for 800 × 1333 is not reported in the paper, we train it by following the training conﬁguration suggested by authors.
Revised: Meanwhile, the results of YOLOS are borrowed from the original paper (Fang et al., 2021) except YOLOS (DeiT-tiny); since the result of YOLOS (DeiT-tiny) for 800 × 1333 is not reported in the paper, we train it by following the training conﬁguration suggested by authors.

Original: C S UPPLEMENTARY E VALUATION C. C OMPARISON WITH O BJECT D ETECTOR USING CNN B ACKBONE
Revised: C S UPPLEMENTARY E VALUATION C. C OMPARISON WITH O BJECT D ETECTOR USING CNN B ACKBONE

Original: We compare ViDT with (Deformable) DETR using the ResNet-50 backbone, as summarized in
Revised: We compare ViDT with (Deformable) DETR using the ResNet-50 backbone, as summarized in

Original: Table 13, where all the results except ViDT are borrowed from (Carion et al., 2020; Zhu et al., 2021), and DETR-DC5 is a modiﬁcation of DETR to use a dilated convolution at the last stage in ResNet.
Revised: Table 13, where all the results except ViDT are borrowed from (Carion et al., 2020; Zhu et al., 2021), and DETR-DC5 is a modiﬁcation of DETR to use a dilated convolution at the last stage in ResNet.

Original: For a fair comparison, we compare ViDT (Swin-tiny) with similar parameter numbers.
Revised: For a fair comparison, we compare ViDT (Swin-tiny) with similar parameter numbers.

Original: In general, ViDT shows a better trade-off between AP and FPS even compared with (Deformable)
Revised: In general, ViDT shows a better trade-off between AP and FPS even compared with (Deformable)

Original: DETR with the ResNet-50.
Revised: DETR with the ResNet-50.

Original: Speciﬁcally, ViDT achieves FPS much higher than DETR-DC5 and
Revised: Speciﬁcally, ViDT achieves FPS much higher than DETR-DC5 and

Original: Deformable DETR with competitive AP.
Revised: Deformable DETR with competitive AP.

Original: Particularly when training ViDT for 150 epochs, ViDT outperforms other compared methods using the ResNet-50 backbone in terms of both AP and FPS.
Revised: Particularly when training ViDT for 150 epochs, ViDT outperforms other compared methods using the ResNet-50 backbone in terms of both AP and FPS.

Original: C. V ARIATIONS OF E XISTING P IPELINES
Revised: C. V ARIATIONS OF E XISTING P IPELINES

Original: We study more variations of existing detection methods by modifying their original pipelines in
Revised: We study more variations of existing detection methods by modifying their original pipelines in

Original: Table 12.
Revised: Table 12.

Original: Thus, we remove the neck encoder of Deformable DETR to increase its efﬁciency, while adding a neck decoder to YOLOS to leverage multi-scale features along with auxiliary decoding
Revised: Thus, we remove the neck encoder of Deformable DETR to increase its efﬁciency, while adding a neck decoder to YOLOS to leverage multi-scale features along with auxiliary decoding

Original: [Figure]
Revised: [Figure]

Original: loss and iterative box reﬁnement.
Revised: loss and iterative box reﬁnement.

Original: Note that these modiﬁed versions follow exactly the same detection pipeline with ViDT, maintaining a encoder-free neck between their body and head.
Revised: Note that these modiﬁed versions follow exactly the same detection pipeline with ViDT, maintaining a encoder-free neck between their body and head.

Original: Tablesummarizes the performance of all the variations in terms of AP, FPS, and the number of parameters.
Revised: Tablesummarizes the performance of all the variations in terms of AP, FPS, and the number of parameters.

Original: Deformable DETR shows signiﬁcant improvement in FPS ( +14 . 4 ) but its AP drops sharply ( − 9 . 1 ) when its neck encoder is removed.
Revised: Deformable DETR shows signiﬁcant improvement in FPS ( +14 . 4 ) but its AP drops sharply ( − 9 . 1 ) when its neck encoder is removed.

Original: Thus, it is difﬁcult to obtain ﬁne-grained object detection representation directly from the raw ViT backbone without using an additional neck encoder.
Revised: Thus, it is difﬁcult to obtain ﬁne-grained object detection representation directly from the raw ViT backbone without using an additional neck encoder.

Original: However, ViDT compensates for the effect of the neck encoder by adding [ DET ] tokens into the body (backbone), thus successfully removing the computational bottleneck without compromising AP; it maintains 6.4 higher AP compared with the neck encoder-free Deformable DETR (the second row) while achieving similar FPS.
Revised: However, ViDT compensates for the effect of the neck encoder by adding [ DET ] tokens into the body (backbone), thus successfully removing the computational bottleneck without compromising AP; it maintains 6.4 higher AP compared with the neck encoder-free Deformable DETR (the second row) while achieving similar FPS.

Original: This can be attributed to that RAM has a great contribution to the performance w.r.t AP and FPS, especially for the trade-off between them.
Revised: This can be attributed to that RAM has a great contribution to the performance w.r.t AP and FPS, especially for the trade-off between them.

Original: YOLOS shows a signiﬁcant gain in AP ( +7 . 7) while losing FPS ( − 11 . 0 ) when the neck decoder is added.
Revised: YOLOS shows a signiﬁcant gain in AP ( +7 . 7) while losing FPS ( − 11 . 0 ) when the neck decoder is added.

Original: Unlike Deformable DETR, its AP signiﬁcantly increases even without the neck encoder due to the use of a standalone object detector as its backbone (i.e., the modiﬁed DeiT in Figure 2(b)).
Revised: Unlike Deformable DETR, its AP signiﬁcantly increases even without the neck encoder due to the use of a standalone object detector as its backbone (i.e., the modiﬁed DeiT in Figure 2(b)).

Original: However, its AP is lower than ViDT by 2.3AP.
Revised: However, its AP is lower than ViDT by 2.3AP.

Original: Even worse, it is not scalable for large models because of its quadratic computational cost for attention.
Revised: Even worse, it is not scalable for large models because of its quadratic computational cost for attention.

Original: Therefore, in the aspects of accuracy and speed, ViDT maintains its dominance compared with the two carefully tuned baselines.
Revised: Therefore, in the aspects of accuracy and speed, ViDT maintains its dominance compared with the two carefully tuned baselines.

Original: For a complete analysis, we additionally add a neck encoder to ViDT.
Revised: For a complete analysis, we additionally add a neck encoder to ViDT.

Original: The inference speed of ViDT degrades drastically by 13 . 7 because of the self-attention for multi-scale features at the neck encoder.
Revised: The inference speed of ViDT degrades drastically by 13 . 7 because of the self-attention for multi-scale features at the neck encoder.

Original: However, it is interesting to see the improvement of AP by 5 . 7 while adding only 3M parameters; it is 3 . 

Revised: However, it is interesting to see the improvement of AP by 5 . 7 while adding only 3M parameters; it is 3 .

Original: higher even than Deformable DETR.
Revised: higher even than Deformable DETR.

Original: This indicates that lowering the computational complexity of the encoder and thus increasing its utilization could be another possible direction for a fully transformer-based object detector.
Revised: This indicates that lowering the computational complexity of the encoder and thus increasing its utilization could be another possible direction for a fully transformer-based object detector.

Original: C. [ DET ] × [ PATCH ] A TTENTION IN RAM
Revised: C. [ DET ] × [ PATCH ] A TTENTION IN RAM

Original: In Section 4.2.2, it turns out that the cross-attention in RAM is only necessary at the last stage of
Revised: In Section 4.2.2, it turns out that the cross-attention in RAM is only necessary at the last stage of

Original: Swin Transformer; all the different selective strategies show similar AP as long as cross-attention is activated at the last stage.
Revised: Swin Transformer; all the different selective strategies show similar AP as long as cross-attention is activated at the last stage.

Original: Hence, we analyze the attention map obtained by the cross-attention in
Revised: Hence, we analyze the attention map obtained by the cross-attention in

Original: RAM.
Revised: RAM.

Original: Figure 4 shows attention maps for the stages of Swin Transformer where cross-attention is utilized; it contrasts (a) ViDT with cross-attention at all stages and (b) ViDT with cross-attention at the last stage.
Revised: Figure 4 shows attention maps for the stages of Swin Transformer where cross-attention is utilized; it contrasts (a) ViDT with cross-attention at all stages and (b) ViDT with cross-attention at the last stage.

Original: Regardless of the use of cross-attention at the lower stage, it is noteworthy that the ﬁnally obtained attention map at the last stage is almost the same.
Revised: Regardless of the use of cross-attention at the lower stage, it is noteworthy that the ﬁnally obtained attention map at the last stage is almost the same.

Original: In particular, the attention map at
Revised: In particular, the attention map at

Original: Stage 1–3 does not properly focus the features on the target object, which is framed by the bounding box.
Revised: Stage 1–3 does not properly focus the features on the target object, which is framed by the bounding box.

Original: In addition, the attention weights (color intensity) at Stage 1–3 are much lower than those at
Revised: In addition, the attention weights (color intensity) at Stage 1–3 are much lower than those at

Original: Stage 4.
Revised: Stage 4.

Original: Since features are extracted from a low level to a high level in a bottom-up manner as they go through the stages, it seems difﬁcult to directly get information about the target object with such low-level features at the lower level of stages.
Revised: Since features are extracted from a low level to a high level in a bottom-up manner as they go through the stages, it seems difﬁcult to directly get information about the target object with such low-level features at the lower level of stages.

Original: Therefore, this analysis provides strong empirical evidence for the use of selective [ DET ] × [ PATH ] cross-attention.
Revised: Therefore, this analysis provides strong empirical evidence for the use of selective [ DET ] × [ PATH ] cross-attention.

Original: C. 
Revised: C.

Original: [Table]
Revised: [Table]

Original: DET

Original: [Table]

Original: A TTENTION

Original: [Table] RAM [Table] Another possible consideration for ViDT is the use of [ DET ] × [ DET ] self-attention in RAM.
Revised: Another possible consideration for ViDT is the use of [ DET ] × [ DET ] self-attention in RAM.

Original: We conduct an ablation study by removing the [ DET ] × [ DET ] attention one by one from the bottom stage, and summarize the results in Table 15.
Revised: We conduct an ablation study by removing the [ DET ] × [ DET ] attention one by one from the bottom stage, and summarize the results in Table 15.

Original: When all the [ DET ] × [ DET ] self-attention are removed, (5) the AP drops by 0.7, which is a meaningful performance degradation.
Revised: When all the [ DET ] × [ DET ] self-attention are removed, (5) the AP drops by 0.7, which is a meaningful performance degradation.

Original: On the other hand, as long as the selfattention is activated at the last two stages, (1) – (3) all the strategies exhibit similar AP.
Revised: On the other hand, as long as the selfattention is activated at the last two stages, (1) – (3) all the strategies exhibit similar AP.

Original: Therefore, only keeping [ DET ] × [ DET ] self-attention at the last two stages can further increase FPS ( +0 . 2 ) without degradation in AP.
Revised: Therefore, only keeping [ DET ] × [ DET ] self-attention at the last two stages can further increase FPS ( +0 . 2 ) without degradation in AP.

Original: This observation could be used as another design choice for the AP and FPS trade-off.
Revised: This observation could be used as another design choice for the AP and FPS trade-off.

Original: Therefore, we believe that [ DET ] × [ DET ] self-attention is meaningful to use in RAM.
Revised: Therefore, we believe that [ DET ] × [ DET ] self-attention is meaningful to use in RAM.

Original: D P RELIMINARIES : T RANSFORMERS
Revised: D P RELIMINARIES : T RANSFORMERS

Original: A transformer is a deep model that entirely relies on the self-attention mechanism for machine translation (Vaswani et al., 2017).
Revised: A transformer is a deep model that entirely relies on the self-attention mechanism for machine translation (Vaswani et al., 2017).

Original: In this section, we brieﬂy revisit the standard form of the transformer.
Revised: In this section, we brieﬂy revisit the standard form of the transformer.

Original: Single-head Attention.
Revised: Single-head Attention.

Original: The basic building block of the transformer is a self-attention module, which generates a weighted sum of the values (contents), where the weight assigned to each value is the attention score computed by the scaled dot-product between its query and key.
Revised: The basic building block of the transformer is a self-attention module, which generates a weighted sum of the values (contents), where the weight assigned to each value is the attention score computed by the scaled dot-product between its query and key.

Original: Let W Q , W K , and
Revised: Let W Q , W K , and

Original: W V be the learned projection matrices of the attention module, and then the output is generated by
Revised: W V be the learned projection matrices of the attention module, and then the output is generated by

Original: [Equation]
Revised: [Equation]

Original: where W Q , W K , W V ∈ Rd × d .
Revised: where W Q , W K , W V [Equation] R

Revised: [Equation]

Revised: d .

Original: Multi-head Attention.
Revised: Multi-head Attention.

Original: It is beneﬁcial to maintain multiple heads such that they repeat the linear projection process k times with different learned projection matrices.
Revised: It is beneﬁcial to maintain multiple heads such that they repeat the linear projection process k times with different learned projection matrices.

Original: Let W Q
Revised: Let W Q

Original: i , W K i , and W V i be the learned projection matrices of the i -th attention head.
Revised: i , W K i , and W V i be the learned projection matrices of the i -th attention head.

Original: Then, the output is generated by the concatenation of the results from all heads,
Revised: Then, the output is generated by the concatenation of the results from all heads,

Original: [Equation]
Revised: [Equation]

Revised: where ∀

Revised: i

Revised: [Equation]

Revised: Q i

Revised: [Equation]

Revised: K

Revised: [Equation]

Original: Typically, the dimension of each head is divided by the total number of heads.
Revised: Typically, the dimension of each head is divided by the total number of heads.

Original: Feed-Forward Networks (FFNs).
Revised: Feed-Forward Networks (FFNs).

Original: The output of the multi-head attention is fed to the point-wise
Revised: The output of the multi-head attention is fed to the point-wise

Original: FFNs, which performs the linear transformation for each position separately and identically to allow the model focusing on the contents of different representation subspaces.
Revised: FFNs, which performs the linear transformation for each position separately and identically to allow the model focusing on the contents of different representation subspaces.

Original: Here, the residual connec- tion and layer normalization are applied before and after the FFNs.
Revised: Here, the residual connec- tion and layer normalization are applied before and after the FFNs.

Original: The ﬁnal output is generated by
Revised: The ﬁnal output is generated by

Original: [Equation]
Revised: [Equation]

Original: where H (cid:48) =
Revised: where H (cid:48) =

Original: FFN ( H (cid:48)(cid:48) ) and H (cid:48)(cid:48) =
Revised: FFN ( H (cid:48)(cid:48) ) and H (cid:48)(cid:48) =

Original: LayerNorm ( Dropout ( Multi-Head ( Z ))
Revised: LayerNorm ( Dropout ( Multi-Head ( Z ))

Original: + Z ) .
Revised: + Z ) .

Original: [Equation]
Revised: [Equation]

Original: Multi-Layer Transformers.
Revised: Multi-Layer Transformers.

Original: The output of a previous layer is fed directly to the input of the next layer.
Revised: The output of a previous layer is fed directly to the input of the next layer.

Original: Regarding the positional encoding, the same value is added to the input of each attention module for all layers.
Revised: Regarding the positional encoding, the same value is added to the input of each attention module for all layers.