Original: S PARSE DETR: E FFICIENT E ND - TO -E ND O BJECT D ETECTION WITH L EARNABLE S PARSITY
Revised: S PARSE DETR: E FFICIENT E ND - TO -E ND O BJECT D ETECTION WITH L EARNABLE S PARSITY

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: DETR is the ﬁrst end-to-end object detector using a transformer encoder-decoder architecture and demonstrates competitive performance but low computational efﬁciency.
Revised: DETR is the ﬁrst end-to-end object detector using a transformer encoder-decoder architecture and demonstrates competitive performance but low computational efﬁciency on high resolution feature maps.

Original: The subsequent work, DeformableDETR, enhances the efﬁciency of
Revised: The subsequent work, Deformable

Original:  DETR by replacing dense attention with deformable attention, which achieves 10 × faster convergence and improved performance.
Revised: DETR, enhances the efﬁciency of DETR by replacing dense attention with deformable attention, which achieves 10 × faster convergence and improved performance.

Original: Using the multiscale feature to ameliorate performance, however, the number of encoder queries increases by 20 × compared to DETR, and the computation cost of the encoder attention remains a bottleneck.
Revised: Deformable DETR uses the multiscale feature to ameliorate performance, however, the number of encoder tokens increases by 20 × compared to DETR, and the computation cost of the encoder attention remains a bottleneck.

Original: We observe that theencoder queries referenced by the decoder account for only 45% of the total, and ﬁnd out the detection accuracy does not deteriorate signiﬁcantly even if only the referenced queries are polished in the encoder block.
Revised: In our preliminary experiment, we observe that the detection performance hardly deteriorates even if only a part of the encoder token is updated.

Original: Inspired by this observation, we propose Sparse DETR that selectively updates only the queries expected to be referenced by the decoder, thus help the model effectively detect objects.
Revised: Inspired by this observation, we propose Sparse DETR that selectively updates only the tokens expected to be referenced by the decoder, thus help the model effectively detect objects.

Original: In addition, we show that applying an auxiliary detection loss on the selected queries in the encoder improves the performance while minimizing computational overhead.
Revised: In addition, we show that applying an auxiliary detection loss on the selected tokens in the encoder improves the performance while minimizing computational overhead.

Original: We validate that Sparse DETR achieves better performance than Deformable DETR even with only 10% encoder queries on the COCO dataset.
Revised: We validate that Sparse DETR achieves better performance than Deformable DETR even with only 10% encoder tokens on the COCO dataset.

Original: Albeit only the encoder queries are sparsiﬁed, the total computation cost decreases by 38% and the frames per second (FPS) increases by 42% compared to Deformable DETR.
Revised: Albeit only the encoder tokens are sparsiﬁed, the total computation cost decreases by 38% and the frames per second (FPS) increases by 42% compared to Deformable DETR.

Original: Code will be released.
Revised: Code will be released.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In recent years, we have witnessed the dramatic advancement and the success of object detection in deep learning.
Revised: In recent years, we have witnessed the dramatic advancement and the success of object detection in deep learning.

Original: Diverse object detection methods have been proposed, but the existing algorithms that perform positive matching with the ground truth as a heuristic way require non-maximum suppression (NMS) post-processing of near-duplicate predictions.
Revised: Diverse object detection methods have been proposed, but the existing algorithms that perform positive matching with the ground truth as a heuristic way require non-maximum suppression (NMS) post-processing of near-duplicate predictions.

Original: Recently, Carion et al.
Revised: Recently, Carion et al.

Original: (2020) has introduced a fully end-to-end detector DETR by eliminating the need for NMS post-processing through a set-based objective.
Revised: (2020) has introduced a fully end-to-end detector DETR by eliminating the need for NMS post-processing through a set-based objective.

Original: The training objective is designed by employing the Hungarian algorithm that considers both classiﬁcation and regression costs, and achieves highly competitive performance.
Revised: The training objective is designed by employing the Hungarian algorithm that considers both classiﬁcation and regression costs, and achieves highly competitive performance.

Original: However, DETR is unable to use multi-scale features such as feature pyramid networks (Lin et al., 2017), which are commonly used in object detection to improve the detection of small objects.
Revised: However, DETR is unable to use multi-scale features such as feature pyramid networks (Lin et al., 2017), which are commonly used in object detection to improve the detection of small objects.

Original: The main reason is increased memory usage and computation by adding Transformer (Vaswani et al., 2017) architecture.
Revised: The main reason is increased memory usage and computation by adding Transformer (Vaswani et al., 2017) architecture.

Original: As a result, its ability to detect small objects is relatively poor.
Revised: As a result, its ability to detect small objects is relatively poor.

Original: To address this problem, Zhu et al.
Revised: To address this problem, Zhu et al.

Original: (2021) has proposed a deformable-attention inspired by the deformable convolution (Dai et al., 2017) and reduced the quadratic complexity to linear complexity through key sparsiﬁcation in the attention module.
Revised: (2021) has proposed a deformable-attention inspired by the deformable convolution (Dai et al., 2017) and reduced the quadratic complexity to linear complexity through key sparsiﬁcation in the attention module.

Original: By using deformable attention, deformable
Revised: By using deformable attention, deformable

Original: DETR addresses the slow convergence and high complexity issue of DETR, which enables the encoder to use multi-scale features as an input and signiﬁcantly improves performance on detecting small objects.
Revised: DETR addresses the slow convergence and high complexity issue of DETR, which enables the encoder to use multi-scale features as an input and signiﬁcantly improves performance on detecting small objects.

Original: However, using the multi-scale features as an encoder input increases the number of tokens to be processed by about 20 times.
Revised: However, using the multi-scale features as an encoder input increases the number of tokens to be processed by about 20 times.

Original: Eventually, despite efﬁcient computation for the same token length, the overall complexity increases back again, making the model inference slower even than vanilla DETR.
Revised: Eventually, despite efﬁcient computation for the same token length, the overall complexity increases back again, making the model inference slower even than vanilla DETR.

Original: In general, natural images often contain large background regions irrelevant to the objects of interest, and accordingly, in end-to-end detectors, the tokens corresponding to the background also occupy a signiﬁcant portion.
Revised: In general, natural images often contain large background regions irrelevant to the objects of interest, and accordingly, in end-to-end detectors, the tokens corresponding to the background also occupy a signiﬁcant portion.

Original: In addition, the importance of each regional feature is not identical, which has been proven by the two-stage detectors successfully do their job by focusing only on the foreground.
Revised: In addition, the importance of each regional feature is not identical, which has been proven by the two-stage detectors successfully do their job by focusing only on the foreground.

Original: It suggests that there exists considerable regional redundancy that can be reduced in the detection tasks and seeking to devise an efﬁcient detector focusing on the salient regions is a necessary and natural direction.
Revised: It suggests that there exists considerable regional redundancy that can be reduced in the detection tasks and seeking to devise an efﬁcient detector focusing on the salient regions is a necessary and natural direction.

Original: In this work, we observe that the encoder queries referenced by the decoder, using the trained Deformable DETR model, accounts for only about 45% of the total, and ﬁnd out polishing only thepart of the encoder queries referenced by the decoder’s object queries does not signiﬁcantly degrade the detection performance (0.1 AP degradation on COCO dataset).
Revised: In our preliminary experiments, we observe the following: (a) during inference of a fully-converged Deformable DETR model on the COCO validation dataset, the encoder tokens referenced by the decoder account for only about 45% of the total, and (b) retraining a new detector while updating only the encoder tokens preferred by the decoder from another fully-trained detector, barely suffers performance loss(0.1 AP degradation). See Appendix A.9 for the details.

Original: Inspired by this observation, we propose a learnable decoder cross-attention map predictor to sparsify encoder queries.
Revised: Inspired by this observation, we propose a learnable decoder cross-attention map predictor to sparsify encoder tokens.

Original: In the existing methods (Carion et al., 2020; Zhu et al., 2021), the encoder takes all the queries, i.e. the backbone features combined with corresponding positional embeddings, as input without discrimination.
Revised: In the existing methods (Carion et al., 2020; Zhu et al., 2021), the encoder takes all the tokens, i.e. the backbone features combined with corresponding positional embeddings, as input without discrimination.

Original: Meanwhile, our approach distinguishes encoder queries to be referenced later in the decoder and considers only those queries in self-attention.
Revised: Meanwhile, our approach distinguishes encoder tokens to be referenced later in the decoder and considers only those tokens in self-attention.

Original: Therefore, this can signiﬁcantly reduce the number of encoder queries involved in the computation and reduce the total computational cost for detection.
Revised: Therefore, this can signiﬁcantly reduce the number of encoder tokens involved in the computation and reduce the total computational cost for detection.

Original: We further propose the encoder auxiliary loss only for selected encoder queries to improve detection performance while minimizing computational overhead.
Revised: We further propose the encoder auxiliary loss only for selected encoder tokens to improve detection performance while minimizing computational overhead.

Original: The proposed auxiliary loss not only improves performance, but also allows training a larger number of encoder layers.
Revised: The proposed auxiliary loss not only improves performance, but also allows training a larger number of encoder layers.

Original: Extensive experiments on the COCO 2017 benchmark (Lin et al., 2014) demonstrate that
Revised: Extensive experiments on the COCO 2017 benchmark (Lin et al., 2014) demonstrate that

Original: Sparse
Revised: Sparse

Original: DETR effectively reduces computational cost while achieving better detection performance.
Revised: DETR effectively reduces computational cost while achieving better detection performance.

Original: Without bells and whistles, Sparse DETR using Swin-T (Liu et al., 2021) backbone achieves 48.2 AP with 38% reduction of the entire computational cost compared to the 48.0 AP baseline and 49.2 AP with 23% reduction.
Revised: Without bells and whistles, Sparse DETR using Swin-T (Liu et al., 2021) backbone achieves 48.2 AP with 38% reduction of the entire computational cost compared to the 48.0 AP baseline and 49.2 AP with 23% reduction.

Original: In the case of the experiment that achieves 48.2 AP using only 10% of encoder queries, the computational cost of the transformer encoder block is reduced by approximately 82%.
Revised: In the case of the experiment that achieves 48.2 AP using only 10% of encoder tokens, the computational cost of the transformer encoder block is reduced by approximately 82%.

Original: We summarize our contributions as follows:
Revised: We summarize our contributions as follows:

Original: • We propose encoder query sparsiﬁcation method for an efﬁcient end-to-end object detector, by which we lighten the attention complexity in the encoder.
Revised: • We propose encoder token sparsiﬁcation method for an efﬁcient end-to-end object detector, by which we lighten the attention complexity in the encoder.

Original: This efﬁciency enables stacking more encoder layers than Deformable DETR, leading to performance improvement within the same computational budget.
Revised: This efﬁciency enables stacking more encoder layers than Deformable DETR, leading to performance improvement within the same computational budget.

Original: • We propose two novel sparsiﬁcation criteria to sample the informative subset from the entire query set: Objectness Score (OS) and Decoder cross-Attention Map (DAM) .
Revised: • We propose two novel sparsiﬁcation criteria to sample the informative subset from the entire token set: Objectness Score (OS) and Decoder cross-Attention Map (DAM) .

Original: Based on the decoder cross-attention map criterion, the sparsiﬁed model preserves detection performance even when using only 10% of the whole queries.
Revised: Based on the decoder cross-attention map criterion, the sparsiﬁed model preserves detection performance even when using only 10% of the whole tokens.

Original: • We adopt an encoder auxiliary loss only for the selected queries.
Revised: • We adopt an encoder auxiliary loss only for the selected tokens.

Original: This additional loss not only stabilizes the learning process, but also greatly improves performance, with only marginally increased training time.
Revised: This additional loss not only stabilizes the learning process, but also greatly improves performance, with only marginally increased training time.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Efﬁcient computation in vision transformers.
Revised: Efﬁcient computation in vision transformers.

Original: It is a well-known problem that the attention computation in Transformers incurs the high time and memory complexity.
Revised: It is a well-known problem that the attention computation in Transformers incurs the high time and memory complexity.

Original: The vision transformers need to digest even bigger token sets as input so that a large body of works (Parmar et al., 2018; Child et al., 2019; Ho et al., 2019; Wang et al., 2020; Katharopoulos et al., 2020; Choromanski et al., 2021; Kitaev et al., 2020) has been proposed lightweight attention mechanisms for them.
Revised: The vision transformers need to digest even bigger token sets as input so that a large body of works (Parmar et al., 2018; Child et al., 2019a; Ho et al., 2019; Wang et al., 2020; Katharopoulos et al., 2020; Choromanski et al., 2021; Kitaev et al., 2020) has been proposed lightweight attention mechanisms for them.

Original: Most of those works shed light on the complexity that resides only in a single-scale attention module, which hinders direct extension to the multi-scale features generally required in object detection.
Revised: Most of those works shed light on the complexity that resides only in a single-scale attention module, which hinders direct extension to the multi-scale features generally required in object detection.

Original: One of the promising approaches for the lighter transformer attention is input-dependent token sparsiﬁcation.
Revised: One of the promising approaches for the lighter transformer attention is input-dependent token sparsiﬁcation.

Original: DynamicViT (Rao et al., 2021) and IA-RED 2 (Pan et al., 2021), similar to our work, both propose jointly-learned token selectors generating the sparsity patterns to be overlaid on the input tokens.
Revised: DynamicViT (Rao et al., 2021) and IA-RED 2 (Pan et al., 2021), similar to our work, both propose jointly-learned token selectors generating the sparsity patterns to be overlaid on the input tokens.

Original: Those approaches mainly focus on sparsifying a backbone network evaluated on the classiﬁcation tasks, while our interest lies in a sparse encoder of the end-to-end object detectors.
Revised: Those approaches mainly focus on sparsifying a backbone network evaluated on the classiﬁcation tasks, while our interest lies in a sparse encoder of the end-to-end object detectors.

Original: On the other hand, there has been a line of works sharing the spirit with ours in that they aim at sparse transformers in the DETR-based framework.
Revised: On the other hand, there has been a line of works sharing the spirit with ours in that they aim at sparse transformers in the DETR-based framework.

Original: Deformable DETR (Zhu et al., 2021) conducts sparse attention computation by sampling only a fraction of the entire key set with learnable 2-d offsets, which enables to use multi-scale feature maps with a reasonable computational cost.
Revised: Deformable DETR (Zhu et al., 2021) conducts sparse attention computation by sampling only a fraction of the entire key set with learnable 2-d off- sets, which enables to use multi-scale feature maps with a reasonable computational cost.

Original: It can be viewed as a key sparsiﬁcation method but with dense queries, while our approach further reduces the query set pursuing even more sparsity.
Revised: It can be viewed as a key sparsiﬁcation method but with dense queries, while our approach further reduces the query set pursuing even more sparsity.

Original: PnP-DETR (Wang et al., 2021) shortens the token length of the transformer encoder by introducing the Polling and Pull (PnP) module to sample the foreground tokens and condense the background tokens into a smaller set.
Revised: PnP-DETR (Wang et al., 2021) shortens the token length of the transformer encoder by introducing the Polling and Pull (PnP) module to sample the foreground tokens and condense the background tokens into a smaller set.

Original: However, their method cannot naively be integrated with Deformable DETR, since their sparsiﬁcation breaks the 2d spatial structure of the key set assumed in the deformable attention.
Revised: However, their method cannot naively be integrated with Deformable DETR, since their sparsiﬁcation breaks the 2d spatial structure of the token set assumed in the deformable attention.

Original: On the contrary, Sparse DETR preserves the 2d sample space of the set and can be seamlessly combined with the deformable attention, which facilitates handling the multi-scale features.
Revised: On the contrary, Sparse DETR preserves the 2d sample space of the set and can be seamlessly combined with the deformable attention, which facilitates handling the multi-scale features.

Original: Thus, our approach gets beneﬁts from both the deformable key sampling and the proposed query sparsiﬁcation.
Revised: Thus, our approach gets beneﬁts from both the deformable key sampling and the proposed query sparsiﬁcation.

Original: Most of all, we proposethe explicit objectives for the token selection network, whereas the aforementioned works have no explicit objective implying their beliefs in a good selection strategy, merely relying on the ﬁnal detection objective.
Revised: Most of all, we propose explicit objectives for the token selection network, whereas the aforementioned works have no explicit objective implying their beliefs in a good selection strategy, merely relying on the ﬁnal detection objective.

Original: Auxiliary Loss.
Revised: Auxiliary Loss.

Original: Auxiliary loss (Szegedy et al., 2015) is widely adopted to deliver gradients to the early layers of deep networks.
Revised: Auxiliary loss (Lee et al., 2015; Szegedy et al., 2015) is widely adopted to deliver gradients to the early layers of deep networks.

Original: DETR variants employ auxiliary Hungarian matching objectives at the end of every decoder layer with extra FFN heads so that each decoder layer directly learns to detect the correct number of objects out of the decoder’s outputs.
Revised: DETR variants employ auxiliary Hungarian matching objectives at the end of every decoder layer with extra FFN heads so that each decoder layer directly learns to detect the correct number of objects out of the decoder’s outputs.

Original: Unlike the decoder’s  queries whose number is relatively small(e.g.
Revised: Unlike the decoder’s object queries whose number is relatively small(e.g.

Original: 300), the number of encoder’s
Revised: 300), the number of encoder’s

Original: queries has much larger scales when using multi-scale features.
Revised: tokens has much larger scales when using multi-scale features.

Original: Thus, extending the layerwise auxiliary loss to the multi-scale encoder increases the training time cost by feeding too many tokens to the attached FFN heads.
Revised: Thus, extending the layerwise auxiliary loss to the multi-scale encoder increases the training time cost by feeding too many tokens to the attached FFN heads.

Original: In Sparse DETR, thanks to the sparsity already induced in the encoder, we can instantly economize that cost while enjoying the auxiliary gradients in a wider range of intermediate layers.
Revised: In Sparse DETR, thanks to the sparsity already induced in the encoder, we can instantly economize that cost while enjoying the auxiliary gradients in a wider range of intermediate layers.

Original:  A PPROACH
Revised:  A PPROACH

Original: In this section, we present our main contributions: (a) formulating a generalized saliency-based query sparsiﬁcation scheme for the encoder, (b) proposing the effective saliency criteria with which that scheme can practically work, and (c) employing the  top- k decoder query selectionand encoder auxiliary losses to improve the performance.
Revised: In this section, we present our main contributions: (a) formulating a generalized saliency-based token sparsiﬁcation scheme for the encoder, (b) proposing the effective saliency criteria with which that scheme can practically work, and (c) employing the encoder auxiliary losses and the top- k decoder query selection to improve the performance.

Original: Before describing the details, we revisit the key components of DETR (Carion et al., 2020) and Deformable DETR (Zhu et al., 2021).
Revised: Before describing the details, we revisit the key components of DETR (Carion et al., 2020) and Deformable DETR (Zhu et al., 2021).

Original: 3. P RELIMINARY
Revised: 3. P RELIMINARY

Original: DETR.
Revised: DETR.

Original: DETR takes the ﬂattened spatial feature map x bb ∈ R N × D from a backbone network into the transformer encoder, where N denotes the number of queries (i.e. features) and D denotes query dimension.
Revised: DETR takes the ﬂattened spatial feature map x feat ∈ R N × D from a backbone network into the transformer encoder, where N denotes the number of tokens (i.e. features) and D denotes token dimension.

Original: The encoder iteratively polishes x bb by several vanilla self-attention modules.
Revised: The encoder iteratively updates x feat by several vanilla self-attention modules.

Original: Then, the transformer decoder takes both the reﬁned queries and M learnable object queries { q i } i =1 ··· M as inputs and predicts a tuple of a class score c ∈ [0 , 1] C and a bounding box b ∈ [0 , 1] 4 for each object query
Revised: Then, the transformer decoder takes both the reﬁned encoder tokens (i.e. encoder output) and M learnable object queries { q i } i =1 ··· M as inputs and predicts a tuple of a class score c ∈ [0 , 1] C and a bounding box b ∈ [0 , 1] 4 for each object query

Original: q i , denoted as { ˆy i } = { ( c i , b i ) } , where C denotes the number of classes.
Revised: q i , denoted as { ˆy i } = { ( c i , b i ) } , where C denotes the number of classes.

Original: All components including the backbone network are jointly trained by performing the bipartite matching between the ground truth { y i } and predictions { ˆy i } .
Revised: All components including the backbone network are jointly trained by performing the bipartite matching between the ground truth { y i } and predictions { ˆy i } .

Original: Deformable DETR.
Revised: Deformable DETR.

Original: Deformable DETR replaces the vanilla dense attention, which is the main computational bottleneck in DETR, with a deformable attention module.
Revised: Deformable DETR replaces the vanilla dense attention, which is the main computational bottleneck in DETR, with a deformable attention module.

Original: This signiﬁcantly reduces the computational cost and improves the convergence.
Revised: This signiﬁcantly reduces the computational cost and improves the convergence.

Original: Suppose that we have the same size of a set of queries (denoted as Ω q ) and a set of keys (denoted as Ω k ), which means | Ω q | = | Ω k | =
Revised: Suppose that we have the same size of a set of queries (denoted as Ω q ) and a set of keys (denoted as Ω k ), which means | Ω q | = | Ω k | =

Original: N .
Revised: N .

Original: The conventional dense attention computes the attention weight
Revised: The conventional dense attention computes the attention weight

Original: A qk for every pair { ( q, k ) :
Revised: A qk for every pair { ( q, k ) :

Original: q ∈ Ω q , k ∈ Ω k } , resulting in quadratic complexity with respect to N .
Revised: q ∈ Ω q , k ∈ Ω k } , resulting in quadratic complexity with respect to N .

Original: Deformable attention reduces this quadratic complexity into the linear one by only considering relevant keys for each query.
Revised: Deformable attention reduces this quadratic complexity into the linear one by only considering relevant keys for each query.

Original: Speciﬁcally, deformable attention computes attention weight
Revised: Speciﬁcally, deformable attention computes attention weight

Original: A qk for all queries and a small set of keys: { ( q, k ) :
Revised: A qk for all queries and a small set of keys: { ( q, k ) :

Original: q ∈ Ω q , k ∈ Ω qk } , where Ω qk ⊂ Ω k and | Ω qk | = K (cid:28) N .
Revised: q ∈ Ω q , k ∈ Ω qk } , where Ω qk ⊂ Ω k and | Ω qk | = K (cid:28) N .

Original: Due to this key sparsiﬁcation, Deformable DETR is able to use the multi-scale features of the backbone network, improving the detection performance of small objects signiﬁcantly.
Revised: Due to this key sparsiﬁcation, Deformable DETR is able to use the multi-scale features of the backbone network, improving the detection performance of small objects signiﬁcantly.

Original: Paradoxically, using the multi-scale feature increases the number of queries in the transformer encoder by about 20 × compared to DETR, making that the encoder becomes the computational bottleneck of deformable
Revised: Paradoxically, using the multi-scale feature increases the number of tokens in the transformer encoder by about 20 × compared to DETR, making that the encoder becomes the computational bottleneck of deformable

Original: DETR.
Revised: DETR.

Original: This motivates us to develop a sparsiﬁcation method to reduce the number of queries in the encoder aggressively, which is described in the next sections.
Revised: This motivates us to develop a sparsiﬁcation method to reduce the number of tokens in the encoder aggressively, which is described in the next sections.

Original: [Figure]
Revised: [Figure]

Original: 3. E NCODER Q UERY S PARSIFICATION
Revised: 3. E NCODER T OKEN S PARSIFICATION

Original: In this section, we introduce our query sparsiﬁcation scheme that the encoder module selectively reﬁnes a small number of encoder queries.
Revised: In this section, we introduce our token sparsiﬁcation scheme that the encoder module selectively reﬁnes a small number of encoder tokens.

Original: This encoder query subset is obtained from the backbone feature map x bb with a certain criterion, which is described in the subsequent section.
Revised: This encoder token subset is obtained from the backbone feature map x feat with a certain criterion, which is described in the subsequent section.

Original: For features that are not updated in this process, the values of x bb are passed through the encoder layers without being changed.
Revised: For features that are not updated in this process, the values of x feat are passed through the encoder layers without being changed.

Original: Formally, suppose that we have a scoring network g : R d → R that measures saliency of each query in x bb .
Revised: Formally, suppose that we have a scoring network g : R d → R that measures saliency of each token in x feat .

Original: We then deﬁne ρ -salient regions Ω ρs as the top- ρ % queries with the highest scores, for a given keeping ratio ρ , i.e. S = | Ω ρs | = ρ · | Ω q | (cid:28) | Ω q | = N .
Revised: We then deﬁne ρ -salient regions Ω ρs as the top- ρ % tokens with the highest scores, for a given keeping ratio ρ , i.e. S = | Ω ρs | = ρ · | Ω q | (cid:28) | Ω q | = N .

Original: Then, the i -th encoder layer polishes the features x i − 1 by:
Revised: Then, the i -th encoder layer updates the features x i − 1 by:

Original: [Equation]
Revised: [Equation]

Original: where DFAttn refers to deformable attention, LN to layer normalization (Ba et al., 2016), and FFN to a feed-forward network.
Revised: where DefAttn refers to deformable attention, LN to layer normalization (Ba et al., 2016), and FFN to a feed-forward network.

Original: Even in the case of unselected queries, the values are still passed through the encoder layer, so they can be referenced  when updating the selected queries.
Revised: Even in the case of unselected tokens, the values are still passed through the encoder layer, so they can be referenced as keys when updating the selected tokens.

Original: This means that the unselected queries can hand over information to the selected queries without losing the value of themselves while minimizing the computational cost.
Revised: This means that the unselected tokens can hand over information to the selected tokens without losing the value of themselves while minimizing the computational cost.

Original: Here, we use deformable attention for reﬁning queries in Ω ρs , but the proposed encoder query sparsiﬁcation is applicable regardless of which attention method the encoder uses.
Revised: Here, we use deformable attention for reﬁning tokens in Ω ρs , but the proposed encoder token sparsiﬁcation is applicable regardless of which attention method the encoder uses.

Original: Complexity of Attention Modules in Encoder.
Revised: Complexity of Attention Modules in Encoder.

Original: Deformable DETR reduces the attention complexity through key sparsiﬁcation, and we further reduce the attention complexity through query sparsiﬁcation, as shown in Fig. 1.
Revised: Deformable DETR reduces the attention complexity through key sparsiﬁcation, and we further reduce the attention complexity through query sparsiﬁcation, as shown in Fig. 1.

Original: Conventional dense attention in DETR requires quadratic complexity O ( N 2 ) , where N is the query length.
Revised: Conventional dense attention in DETR requires quadratic complexity O ( N 2 ) , where N is the query length.

Original: Deformable attention requires linear complexity
Revised: Deformable attention requires linear complexity

Original: O ( NK ) , where K (cid:28) N is the number of keys for each query.
Revised: O ( NK ) , where K (cid:28) N is the number of keys for each query.

Original: Sparse attention requires only
Revised: Sparse attention requires only

Original: O ( SK ) , where S (cid:28) N is the number of salient encoder queries.
Revised: O ( SK ) , where S (cid:28) N is the number of salient encoder queries.

Original: 3. F INDING S ALIENT E NCODER Q UERIES
Revised: 3. F INDING S ALIENT E NCODER T OKENS

Original: In this section, we introduce how to ﬁnd a salient query set Ω ρs from a backbone feature x bb .
Revised: In this section, we introduce how to ﬁnd a salient token set Ω ρs from a backbone feature x feat .

Original: We propose a method for determining saliency using a cross attention map from the transformer decoder.
Revised: We propose a method for determining saliency using a cross attention map from the transformer decoder.

Original: Before presenting our approach, we ﬁrst discuss a simple yet effective method based on the objectness scores obtained from a separate detection head.
Revised: Before presenting our approach, we ﬁrst discuss a simple yet effective method based on the objectness scores obtained from a separate detection head.

Original: The limitation of this simple approach motivates us to develop an advanced one, which is described in the following paragraph.
Revised: The limitation of this simple approach motivates us to develop an advanced one, which is described in the following paragraph.

Original: Objectness Score.
Revised: Objectness Score.

Original: Measuring objectness per each query is very natural to determine which ones from a backbone feature should be further updated in the transformer encoder.
Revised: Measuring objectness per each input token (i.e. feature x feat ) of encoder is very natural to determine which ones from a backbone feature should be further updated in the transformer encoder.

Original: It is widely known 
Revised: It is widely known that feature map from a pretrained backbone network is

Original: [Figure]
Revised: [Figure]

Original: that feature map from a pretrained backbone network is able to ﬁnd the saliency of objects, which is why the region proposal network (RPN) has been successfully adopted in many object detectors (Ren et al., 2015; Dai et al., 2016;{+
+}
Revised: I [Figure] able to ﬁnd the saliency of objects, which is why the region proposal network (RPN) has been successfully adopted in many object detectors (Ren et al., {+
+}

Original:  He et al., 2017).
Revised: ; Dai et al., 2016; He et al., 2017).

Original: Inspired by this observation, we introduce an additional detection head and Hungarian loss to the backbone feature map, where the structure of the newly added head is the same as the one of the ﬁnal detection head.
Revised: Inspired by this observation, we introduce an additional detection head and Hungarian loss to the backbone feature map, where the structure of the newly added head is the same as the one of the ﬁnal detection head in the decoder.

Original: Then, we can select the top- ρ % encoder queries with the highest class scores as a salient query set Ω ρs .
Revised: Then, we can select the top- ρ % encoder tokens with the highest class scores as a salient token set Ω ρs .

Original: This approach is effective to sparsify encoder queries, but we believe that it is sub-optimal to the transformer decoder, because the selected queries from the separate detection head are not explicitly considered for the decoder.
Revised: This approach is effective to sparsify encoder tokens, but we believe that it is sub-optimal to the transformer decoder, because the selected encoder tokens from the separate detection head are not explicitly considered for the decoder.

Original: Decoder Cross-Attention Map.
Revised: Decoder Cross-Attention Map.

Original: We consider another approach to select a subset of encoder queries that are highly relevant to the decoder in a more explicit manner.
Revised: We consider another approach to select a subset of encoder tokens that are highly relevant to the decoder in a more explicit manner.

Original: We observe that the crossattention maps from the transformer decoder could be used for measuring the saliency, because the decoder gradually attends to a subset of encoder queries that are favorable to detect objects as training continues.
Revised: We observe that the crossattention maps from the transformer decoder could be used for measuring the saliency, because the decoder gradually attends to a subset of encoder output tokens that are favorable to detect objects as training continues.

Original: Motivated by this, we introduce a scoring network that predicts a pseudo groundtruth of the saliency deﬁned by decoder cross-attention maps, and use it to determine which queries should be further reﬁned on the ﬂy.
Revised: Motivated by this, we introduce a scoring network that predicts a pseudo groundtruth of the saliency deﬁned by decoder cross-attention maps, and use it to determine which encoder tokens should be further reﬁned on the ﬂy.

Original: Fig.
Revised: Fig.

Original: 2 summarizes how to train a scoring network, and details are presented below.
Revised: 2 summarizes how to train a scoring network, and details are presented below.

Original: To determine the saliency of each query of  x bb , we have to aggregate the decoder cross-attentions between all object queries and the encoder query.
Revised: To determine the saliency of each input token of encoder x feat , we have to aggregate the decoder cross-attentions between all object queries and the encoder output.

Original: This procedure produces a single map of the same size as the feature map from the backbone, which is deﬁned as Decoder cross-Attention Map (DAM).
Revised: This procedure produces a single map of the same size as the feature map from the backbone, which is deﬁned as Decoder crossAttention Map (DAM).

Original: In the case of the dense attention, DAM can be easily obtained by summing up attention maps from every decoder layer.
Revised: In the case of the dense attention, DAM can be easily obtained by summing up attention maps from every decoder layer.

Original: In case of deformable attention, for each encoder query, the corresponding value of DAM can be obtained by accumulating the attention weights of decoder  queries whose attention offsets are directed toward the encoder query.
Revised: In case of deformable attention, for each encoder token, the corresponding value of DAM can be obtained by accumulating the attention weights of decoder object queries whose attention offsets are directed toward the encoder output tokens.

Original: Refer to the Appendix A.2 for the details in the DAM creation.
Revised: Refer to the Appendix A.2 for the details in the DAM creation.

Original: To train the scoring network, we binarize DAM so that the top- ρ % (by attention weights) of queries is only retained.
Revised: To train the scoring network, we binarize DAM so that the top- ρ % (by attention weights) of encoder tokens is only retained.

Original: This is because our goal is to ﬁnd a small subset of queries that the decoder references the most, rather than precisely predicting how much each encoder query will be referenced by the decoder.
Revised: This is because our goal is to ﬁnd a small subset of encoder tokens that the decoder references the most, rather than precisely predicting how much each encoder token will be referenced by the decoder.

Original: This binarized DAM implies the one-hot target that indicates whether each query is included in the top- ρ % most referenced queries.
Revised: This binarized DAM implies the one-hot target that indicates whether each encoder token is included in the top- ρ % most referenced encoder tokens.

Original: Then, we consider a 4 -layer scoring network g to predict how likely a given query is included in the top- ρ % most referenced queries, and the network is trained by minimizing the binary cross entropy (BCE) loss between the binarized DAM and prediction:
Revised: Then, we consider a 4 -layer scoring network g to predict how likely a given encoder token is included in the top- ρ % most referenced tokens, and the network is trained by minimizing the binary cross entropy (BCE) loss between the binarized DAM and prediction:

Original: [Equation]
Revised: [Equation]

Original: where DAM bin i means the binarized DAM value of the i th encoder query.
Revised: where DAM bin i means the binarized DAM value of the i th encoder token.

Original: [Figure]
Revised: [Figure]

Original: One may say that since DAM in the early phase of training is not accurate, pruning out the encoder queries based on the result in the decoder degrades the ﬁnal performance or hurts the convergence.
Revised: One may say that since DAM in the early phase of training is not accurate, pruning out the encoder tokens based on the result in the decoder degrades the ﬁnal performance or hurts the convergence.

Original: However, we empirically observe that the optimization is very stable even in the early phase of training, and achieves better performance compared to the method based on objectness score.
Revised: However, we empirically observe that the optimization is very stable even in the early phase of training, and achieves better performance compared to the method based on objectness score.

Original: We describe detailed comparisons in the experiments section.
Revised: We describe detailed comparisons in the experiments section.

Original: 3. A DDITIONAL C OMPONENTS
Revised: 3. A DDITIONAL C OMPONENTS

Original: In this section, we introduce two additional components: (a)top- k decoder queries selection and (b) applying auxiliary losses on the encoder queries.
Revised: In this section, we introduce two additional components: (a) auxiliary losses on the encoder tokens and (b) top- k decoder queries selection.

Original: We empirically observe that these greatly help improve the ﬁnal performance and stabilize the optimization.
Revised: We empirically observe that these greatly help improve the ﬁnal performance and stabilize the optimization.

Original: The overall architecture of Sparse DETR including these components is depicted in Fig. 
Revised: The overall architecture of Sparse DETR including these components is depicted in Fig. Encoder Auxiliary Loss.

Revised: In DETR variants, auxiliary detection heads are attached to decoder layers, but not to encoder layers.

Revised: Due to a signiﬁcantly larger number of encoder tokens (about 18k tokens) compared to decoder tokens (about 300), encoder auxiliary heads will heavily increase the computational cost.

Revised: In Sparse DETR, however, only part of encoder tokens are reﬁned by the encoder, and adding auxiliary heads only for sparsiﬁed encoder tokens is not a big burden.

Revised: We empirically observe that applying an auxiliary detection head along with Hungarian loss on the selected tokens stabilizes the convergence of deeper encoders by alleviating the vanishing gradient issue and even improves the detection performance.

Original:  Top-
Revised: We conjecture that, following the analysis in Sun et al. (2021), applying Hungarian loss at the intermediate layers helps distinguish the confusing features in the encoder, which contributes to the detection performance in the ﬁnal head. Top-

Original: k Decoder Queries.
Revised: k Decoder Queries.

Original: In DETR and Deformable DETR, decoder queries are given by only learnable  queries or with predicted reference points via another head after the encoder.
Revised: In DETR and Deformable DETR, decoder queries are given by only learnable object queries or with predicted reference points via another head after the encoder.

Original: InEfﬁcient
Revised: In

Original:  DETR (Yao et al., 2021) , decoder takes a part of encoder output as input, similar to RoI Pooling (Ren et al., 2015).
Revised: Efﬁcient DETR (Yao et al., 2021), decoder takes a part of encoder output as input, similar to RoI Pooling (Ren et al., 2015).

Original: Here, an auxiliary detection head is attached to the encoder output x enc and the head calculates the objectness (class) score of each encoder output.
Revised: Here, an auxiliary detection head is attached to the encoder output x enc and the head calculates the objectness (class) score of each encoder output.

Original: Based on the score, the top- k encoder outputs are passed as decoder queries, similar to objectness score-based encoder query sparsiﬁcation.
Revised: Based on the score, the top- k encoder outputs are passed as decoder queries, similar to objectness score-based encoder token sparsiﬁcation.

Original: Since this outperforms the methods based on learnable  queries or the two-stage scheme, we include this top- k decoder query selection in our ﬁnal architecture.
Revised: Since this outperforms the methods based on learnable object queries or the two-stage scheme, we include this top- k decoder query selection in our ﬁnal architecture.

Original: Encoder Auxiliary Loss.

Original: In DETR, auxiliary detection heads are attached to decoder layers, but not to encoder layers. Due to a signiﬁcantly larger number of encoder queries (about 18k queries) compared to decoder queries (about 300), encoder auxiliary heads will heavily increase the computational cost. In Sparse DETR, however, only part of encoder queries are reﬁned by the encoder, and adding auxiliary heads only for sparsiﬁed encoder queries is not a big burden. We empirically observe that applying an auxiliary detection head along with Hungarian loss on the selected queries improves the ﬁnal detection performance.
Revised: [Table]

Original: Following the analysis of Sun et al.

Original: (2021), we conjecture that this performance gain is obtained by the Hungarian loss, which helps distinguish the confusing features in the encoder.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We compare Sparse DETR with the conventional object detectors, including the recently proposed ones in the DETR family.
Revised: We compare Sparse DETR with the conventional object detectors, including the recently proposed ones in the DETR family.

Original: In addition, we conduct an ablation study to support our claims in section 3, presenting the performance comparison between query selection criteria (OS vs. DAM), the effectiveness of the encoder auxiliary loss, and the dynamic sparsiﬁcation during inference.[Table]
Revised: In addition, we conduct an ablation study to support our claims in Section 3, presenting the performance comparison between token selection criteria (OS vs. DAM), the effectiveness of the encoder auxiliary loss, and the dynamic sparsiﬁcation during inference.

Original: Implementation Details.
Revised: Implementation Details.

Original: We use ResNet50 (He et al., 2016) and Swin Transformer (Liu et al., 2021) as pre-trained backbone networks, where Swin Transformer is one of the state-of-the-art architecture in the ViT family.
Revised: We use ResNet-50 (He et al., 2016) and Swin Transformer (Liu et al., 2021) as pre-trained backbone networks, where Swin Transformer is one of the state-of-the-art architecture in the ViT family.

Original: We stack 6 encoder and 6 decoder layers, each with an auxiliary head at the end.
Revised: We stack 6 encoder and 6 decoder layers, each with an auxiliary head at the end.

Original: We train the model on a 4 × V100 GPU machine with a total batch size of [Table] for 50 epochs, where the initial learning rate is 0 . 0002 and decayed by 1/10 at the 40 epoch.
Revised: We train the model on a 4 × V100 GPU machine with a total batch size of 16, for 50 epochs, where the initial learning rate is 0 . 0002 and decayed by 1/10 at the 40 epoch.

Original: Unless otherwise speciﬁed, we use the same hyperparameters as in Deformable DETR.
Revised: Unless otherwise speciﬁed, we use the same hyperparameters as in Deformable DETR.

Original: 4. C OMPARISON W ITH O BJECT D ETECTION B ASELINES
Revised: 4. C OMPARISON W ITH O BJECT D ETECTION B ASELINES

Original: Baselines.
Revised: Baselines.

Original: We compare Sparse DETR with Faster-RCNN with FPN (Lin et al., 2017), DETR (Carion et al., 2020), Deformable DETR (Zhu et al., 2021), and PnP DETR (Wang et al., 2021).
Revised: We compare Sparse DETR with Faster-RCNN with FPN (Lin et al., 2017), DETR (Carion et al., 2020), Deformable DETR (Zhu et al., 2021), and PnP DETR (Wang et al., 2021).

Original: We also compare with DETR and Deformable DETR that uses Swin-Tiny (Liu et al., 2021) as a backbone.[Figure] [Figure]
Revised: We also compare with DETR and Deformable DETR that uses Swin-Tiny (Liu et al., 2021) as a backbone.

Original: Here, for brevity, we denote Deformable DETR with the top- k object query selection and bounding box reﬁnement, as Deformable DETR+.
Revised: Here, for brevity, we denote Deformable DETR with the top- k object query selection and bounding box reﬁnement, as Deformable DETR+.

Original: In Sparse DETR, encoder queries are sparsiﬁed with keeping ratios of 10%, 20%, 30%, 40%, and 50%, using DAM criterion.
Revised: In Sparse DETR, encoder tokens are sparsiﬁed with keeping ratios of 10%, 20%, 30%, 40%, and 50%, using DAM criterion.

Original: We demonstrate detection performance and inference costs on COCO val2017 dataset.
Revised: We demonstrate detection performance and inference costs on COCO val2017 dataset.

Original: Result.
Revised: Result.

Original: Table 1 shows the results of Sparse DETR and the other baselines on COCO val set.
Revised: Table 1 shows the results of Sparse DETR and the other baselines on COCO val set.

Original: Remarkably, on the ResNet50 backbone, Sparse DETR with a keeping ratio over 30% outperforms all the baselines while minimizing the computational cost.
Revised: Remarkably, on the ResNet-50 backbone, Sparse DETR with a keeping ratio over 30% outper- [Figure] [Figure] forms all the baselines while minimizing the computational cost.

Original: Even with the keeping ratio reduced down to 10%, Sparse DETR still performs better than most baselines except for Deformable
Revised: Even with the keeping ratio reduced down to 10%, Sparse DETR still performs better than most baselines except for Deformable

Original: DETR+.
Revised: DETR+.

Original: More surprisingly, on the Swin-T backbone, Sparse DETR only with the keeping ratio 10% outperforms all the baselines with no exception, while improving FPS by 38% compared to
Revised: More surprisingly, on the Swin-T backbone, Sparse DETR only with the keeping ratio 10% outperforms all the baselines with no exception, while improving FPS by 38% compared to

Original: Deformable DETR+.
Revised: Deformable DETR+.

Original: Remark that, compared to the most competitive baseline, Deformable DETR+, the improvement in
Revised: Remark that, compared to the most competitive baseline, Deformable DETR+, the improvement in

Original: AP L is relatively noticeable on the Swin-T backbone even under the extreme sparsity of 10%, while the performance gap on the ResNet50 backbone comes evenly from different sizes of objects.
Revised: AP L is relatively noticeable on the Swin-T backbone even under the extreme sparsity of 10%, while the performance gap on the ResNet-50 backbone comes evenly from different sizes of objects.

Original: We conjecture  it is because a single token in Swin-T can hold a wider region of information than the one in ResNet50, so even if we aggressively sparsify the encoder query, the network seems to have enough information to detect objects.
Revised: We conjecture that it is because a single token in Swin-T can hold a wider region of information than the one in ResNet-50, so even if we aggressively sparsify the encoder token, the network seems to have enough information to detect objects.

Original: 4. C OMPARISON B ETWEEN Q UERY S ELECTION C RITERIA
Revised: 4. C OMPARISON B ETWEEN TOKEN S ELECTION C RITERIA

Original: Baselines.
Revised: Baselines.

Original: To verify the beneﬁts of the proposed saliency criteria, we compare three query sparsiﬁcation criteria: random, Objectness Score (OS), and Decoder cross-Attention Map (DAM).
Revised: To verify the beneﬁts of the proposed saliency criteria, we compare three token sparsiﬁcation criteria: random, Objectness Score (OS), and Decoder cross-Attention Map (DAM).

Original: The random baseline samples a ﬁxed ratio of arbitrary queries.
Revised: The random baseline samples a ﬁxed ratio of arbitrary tokens.

Original: Note that the proposed encoder auxiliary loss is applied for all the methods.
Revised: Note that the proposed encoder auxiliary loss is applied for all the methods.

Original: Result.
Revised: Result.

Original: As illustrated in Fig.
Revised: As illustrated in Fig.

Original: 4, the random strategy suffers noticeable performance degradation.
Revised: 4, the random strategy suffers noticeable performance degradation.

Original: On the other hand, the DAM-based model outperforms the OS-based model at every ratio and almost catches up with the non-sparse baseline when using 50% of queries.
Revised: On the other hand, the DAM-based model outperforms the OS-based model at every ratio and almost catches up with the non-sparse baseline when using 50% of encoder tokens.

Original: See the Appendix A.4 for detailed results of this experiment.
Revised: See the Appendix A. for detailed results of this experiment.

Original: To analyze the reason that DAM-based model outperforms its counterpart, we measure the overlap between the queries referred by the decoder and the queries reﬁned by the encoder.
Revised: To analyze the reason that DAM-based model outperforms its counterpart, we measure the overlap between the encoder tokens referred by the decoder and the tokens reﬁned by the encoder.

Original: As a metric, we compute a scalar correlation Corr as:
Revised: As a metric, we compute a scalar correlation Corr as:

Original: [Equation]
Revised: [Equation]

Original: where Ω D is the query set referred by the decoder and DAM x is the DAM-value corresponding to query x .
Revised: where Ω D is the encoder token set referred by the decoder and DAM x is the DAM-value corresponding to token x .

Original: This Corr metric indicates the ratio of queries polished by the encoder among the queries referred by the decoder.
Revised: This Corr metric indicates the ratio of tokens polished by the encoder among the tokens referred by the decoder.

Original: Fig.
Revised: Fig.

Original: 5 demonstrates that Corr of DAM-based model rises higher than that of OS-based model.
Revised: 5 demonstrates that Corr of DAM-based model rises higher than that of OS-based model.

Original: This result implies that DAM-based model is a more suitable sparsiﬁcation method for the decoder,[Figure] [Figure] because the queries referenced by the decoder are explicitly reﬁned in the encoder, which achieves better detection performance.
Revised: This result implies that DAM-based model is a more suitable sparsiﬁcation method for the decoder, because the tokens referenced by the decoder are explicitly reﬁned in the encoder, which achieves better detection performance.

Original: See the Appendix A.4 for detailed results of this experiment. 
Revised: See the Appendix A.4 for detailed results of this experiment. [Figure] [Figure]

Original: 4. E FFECTIVENESS OF THE ENCODER AUXILIARY LOSS.
Revised: 4. E FFECTIVENESS OF THE ENCODER AUXILIARY LOSS

Original: Owing to the sparsiﬁed query set in our model, we can apply the auxiliary loss to the encoder layers without sacriﬁcing too much computational cost.
Revised: Owing to the sparsiﬁed token set in our model, we can apply the auxiliary loss to the encoder layers without sacriﬁcing too much computational cost.

Original: Apart from improving efﬁciency and performance, we ﬁnd another beneﬁt of the encoder auxiliary loss that allows us to safely stack more encoder layers without failing to converge.
Revised: Apart from improved efﬁciency and performance, we ﬁnd another beneﬁt of the encoder auxiliary loss that allows us to safely stack more encoder layers without failing to converge.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 6, the encoder auxiliary loss not only enhances detection performance, but also consistently increases detection performance as the encoder layers doubled to 12.
Revised: 6, the encoder auxiliary loss not only enhances detection performance, but also consistently increases detection performance as the encoder layers doubled to 12.

Original: However, we observe that the training without its assistance utterly fails when using 12 encoder layers.
Revised: However, we observe that the training without its assistance utterly fails when using 12 encoder layers.

Original: We conjecture that gradient propagated through decoder cross-attention may vanish as we stack more encoder layers, thus intermediate gradients from the auxiliary loss are required. 
Revised: We argue that gradient propagated through decoder cross-attention vanishes as we stack more encoder layers, thus intermediate gradients from the auxiliary loss are required. The observations reported in

Original: See the Appendix A.5 for detailed results of this experiment.
Revised: Appendix A.5 supports this assertion and Appendix A.6 details the results of Fig. 6.

Original: 4. D YNAMIC S PARSIFICATION FOR I NFERENCE S TAGE
Revised: 4. D YNAMIC S PARSIFICATION FOR I NFERENCE S TAGE

Original: To deploy the models in various hardware conditions of real-world applications, one often should retrain them at different scales according to the performance-computation trade-off required.
Revised: To deploy the models in various hardware conditions of real-world applications, one often should retrain them at different scales according to the performance-computation trade-off required.

Original: We evaluate if our model trained with a ﬁxed sparsity can adapt well to dynamic sparsity at inference time to check out Sparse DETR can avoid that hassle.
Revised: We evaluate if our model trained with a ﬁxed sparsity can adapt well to dynamic sparsity at inference time to check out Sparse DETR can avoid that hassle.

Original: Figure 7 shows the performance under the varied keeping ratio ( ρ ) during inference when the model trained using the Swin-T backbone and 30% of encoder queries with the DAM-based method.
Revised: Figure 7 shows the performance under the varied keeping ratio ( ρ ) during inference when the model trained using the Swin-T backbone and 30% of encoder tokens with the DAM-based method.

Original: When the inference keeping ratio is small, the performance of dynamic sparsiﬁcation is slightly degraded, but the overall performance is satisfactory at various keeping ratios given that only a single model is used.
Revised: When the inference keeping ratio is small, the performance of dynamic sparsiﬁcation is slightly degraded, but the overall performance is satisfactory at various keeping ratios given that only a single model is used.

Original: PnP DETR introduces dynamic ratio training to achieve similar performance to the ﬁxed keeping ratio counterpart.
Revised: PnP DETR introduces dynamic ratio training to achieve similar performance to the ﬁxed keeping ratio counterpart.

Original: However, without the additional trick, it suffers signiﬁcant performance degradation, for instance, 5.0 AP drop when training/inference keeping ratio is 0.33/0.5, despite the increased number of queries.
Revised: However, without the additional trick, it suffers signiﬁcant performance degradation, for instance, 5.0 AP drop when training/inference keeping ratio is 0.33/0.5, despite the increased number of encoder tokens.

Original: On the contrary, Sparse DETR achieves 0.2 AP improvement in a similar condition where the training/inference keeping ratio is 0.3/0.5.
Revised: On the contrary, Sparse DETR achieves 0.2 AP improvement in a similar condition where the training/inference keeping ratio is 0.3/0.5.

Original: To conclude, our method shows better robustness compared to PnP DETR without further treatment, showing a greater potential of dynamic adaptation to different hardware environments.
Revised: To conclude, our method shows better robustness compared to PnP DETR without further treatment, showing a greater potential of dynamic adaptation to different hardware environments.

Original: Note that any technique such as dynamic ratio training is orthogonal to our method and introducing it may bring even more robustness.
Revised: Note that any technique such as dynamic ratio training is orthogonal to our method and introducing it may bring even more robustness.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we have presented encoder query sparsiﬁcation algorithm that lowers the computational cost of the encoder, which is a computational bottleneck in the DETR and Deformable 
Revised: In this paper, we have presented encoder token sparsiﬁcation algorithm that lowers the computational cost of the encoder, which is a computational bottleneck in the DETR and Deformable DETR.

Original: DETR. By doing so, the proposed Sparse DETR architecture outperforms the Deformable DETR even when using only 10 % of the encoder query, and decreases the overall computation by 38%, and increases the FPS by 42% compared to the Deformable DETR.
Revised: By doing so, the proposed Sparse DETR architecture outperforms the Deformable DETR even when using only 10 % of the encoder token, and decreases the overall computation by 38%, and increases the FPS by 42% compared to the Deformable DETR.

Original: We hope that our proposed method will provide insights to effectively detect objects in the transformer structure in the future.
Revised: We hope that our proposed method will provide insights to effectively detect objects in the transformer structure in the future.

Original: A A PPENDIX A. I MPLEMENTATION D ETAILS OF THE S CORING N ETWORK.
Revised: A A PPENDIX A. I MPLEMENTATION D ETAILS OF THE S CORING N ETWORK

Original: The scoring network is consists of 4 linear layers where Layer Normalization (Ba et al., 2016) comes before the ﬁrst layer and every layer except for the last one is followed by GELU (Hendrycks & Gimpel, 2016) activation.
Revised: The scoring network is consists of 4 linear layers where Layer Normalization (Ba et al., 2016) comes before the ﬁrst layer and every layer except for the last one is followed by GELU (Hendrycks & Gimpel, 2016) activation.

Original: The output dimension of the 1st layer is 256 and halved to 128 andat the 2nd and 3rd layers.
Revised: The output dimension of the 1st layer is 256 and halved to 128 andat the 2nd and 3rd layers.

Original: The last layer outputs 1-d logit for the BCE loss.
Revised: The last layer outputs 1-d logit for the BCE loss.

Original: Since the network locally processes the input tokens in a token-wise manner, the ﬁnal decisions may overlook global statistics without additional treatment.
Revised: Since the network locally processes the input tokens in a token-wise manner, the ﬁnal decisions may overlook global statistics without additional treatment.

Original: To remedy this issue, we set aside half of the output dimension of the ﬁrst layer as a global feature, and average them across the whole token set, then concatenate it with each of the remained local features to maintain the original dimension.
Revised: To remedy this issue, we set aside half of the output dimension of the ﬁrst layer as a global feature, and average them across the whole token set, then concatenate it with each of the remained local features to maintain the original dimension.

Original: We also exclude the tokens that correspond to the zero-padded area when selecting top- ρ % scores, thereby we can prevent those tokens from participating in Hungarian matching process and getting meaningless gradients from the detection objective.
Revised: We also exclude the tokens that correspond to the zero-padded area when selecting top- ρ % scores, thereby we can prevent those tokens from participating in Hungarian matching process and getting meaningless gradients from the detection objective.

Original: A. DAM C REATION IN D EFORMABLE A TTENTION
Revised: A. DAM C REATION IN D EFORMABLE A TTENTION

Original: As attention offset calculated in deformable attention is a fractional position, deformable attention uses bilinear interpolation to get values.
Revised: As attention offset calculated in deformable attention is a fractional position, deformable attention uses bilinear interpolation to get values.

Original: Thus, we also use bilinear interpolation to obtain DAM.
Revised: Thus, we also use bilinear interpolation to obtain DAM.

Original: Assume that, one of the attention offsets, weights and the reference point of  query q is calculated as p , A and r , respectively.
Revised: Assume that, one of the attention offsets, weights and the reference point of decoder object query q is calculated as p , A and r , respectively.

Original: Then, deformable attention takes values as:
Revised: Then, deformable attention takes values as:

Original: [Equation]
Revised: [Equation]

Original: , where x enumerates all integral spatial locations in the feature map, G ( · , · ) is the bilinear interpolation kernel deﬁned as G ( a, b ) = max(0 , 1 − | a x − b x | ) · max(0 , 1 − | a y − b y | ) and v is the values.
Revised: , where x enumerates all integral spatial locations in the feature map, G ( · , · ) is the bilinear interpolation kernel deﬁned as G ( a, b ) = max(0 , 1 − | a x − b x | ) · max(0 , 1 − | a y − b y | ) and v is the values.

Original: Similarly, we accumulate DAM-value for location x as:
Revised: Similarly, we accumulate DAM-value for location x as:

Original: [Equation]
Revised: [Equation]

Revised: + p ) .

Original: Then, we accumulate DAM over every  object query.
Revised: Then, we accumulate DAM over every decoder object query.

Original: A. A LTERNATIVE OBJECTIVES FOR DAM- BASED MODEL.
Revised: A. A LTERNATIVE OBJECTIVES FOR DAM- BASED MODEL

Original: [Figure]
Revised: [Figure]

Original: As a training objective of the scoring network using DAM, we can consider other alternatives as long as they can encourage the predicted scores to represent the relative saliency of the queries.
Revised: As a training objective of the scoring network using DAM, we can consider other alternatives as long as they can encourage the predicted scores to represent the relative saliency of the encoder tokens.

Original: One of the naive alternatives is the regression loss by which the scoring network directly predicts the values in DAM.
Revised: One of the naive alternatives is the regression loss by which the scoring network directly predicts the values in DAM.

Original: The ranking loss can be another choice with which the network focuses more on learning the relativeness rather than estimating the set of salient queries.
Revised: The ranking loss can be another choice with which the network focuses more on learning the relativeness rather than estimating the set of salient tokens.

Original: Figure 8 shows the default BCE loss outperforms the alternatives.
Revised: Figure 8 shows the default BCE loss outperforms the alternatives.

Original: First, it is well-known that the regression problem is much harder than classiﬁcation.
Revised: First, it is well-known that the regression problem is much harder than classiﬁcation.

Original: Furthermore, since the value of DAM changes during training, the regression loss to predict the accurate value is more difﬁcult.
Revised: Furthermore, since the value of DAM changes during training, the regression loss to predict the accurate value is more difﬁcult.

Original: In case of the pairwise ranking loss, ranking the DAM elements may also be unstable as DAM gradually evolves.
Revised: In case of the pairwise ranking loss, ranking the DAM elements may also be unstable as DAM gradually evolves.

Original: Meanwhile, the BCE loss may reduce those element-level noises down to the set-level in that its binary (keep or drop) targets retain more consistency compared to the exact values or ranks.
Revised: Meanwhile, the BCE loss may reduce those element-level noises down to the set-level in that its binary (keep or drop) targets retain more consistency compared to the exact values or ranks.

Original: Refer to Table 2 to see the exact values of the points represented in Figure 8.
Revised: Refer to Table 2 to see the exact values of the points represented in Figure 8.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: random 10% 43.
Revised: random 10% 43.

Original: 41M 102G 27.
Revised: 41M 102G 27.

Original: 41M 110G 25.
Revised: 41M 110G 25.

Original: 41M 117G 24.
Revised: 41M 117G 24.

Original: 41M 125G 22.
Revised: 41M 125G 22.

Original: 41M 133G 21.
Revised: 41M 133G 21.

Original: OS 10% 44.
Revised: OS 10% 44.

Original: 41M 106G 26.
Revised: 41M 106G 26.

Original: 41M 114G 24.
Revised: 41M 114G 24.

Original: 41M 121G 23.
Revised: 41M 121G 23.

Original: 41M 129G 21.
Revised: 41M 129G 21.

Original: 41M 136G 20.
Revised: 41M 136G 20.

Original: DAM 10% 45.
Revised: DAM 10% 45.

Original: 41M 105G 26.
Revised: 41M 105G 26.

Original: 41M 113G 24. 
Revised: 41M 113G 24. 41M 121G 23. 41M 128G 21. 41M 136G 20.

Revised: Swin-T backbone:

Revised: N/A 100% 49.

Original: 41M 121G 23.
Revised: 41M 185G 15.

Original: [Table]
Revised: 37M 96G 26. random 10% 45. 41M 110G 22. 41M 118G 20.

Revised: 41M 125G 19.

Revised: 41M 133G 18.

Revised: 41M 141G 17.

Revised: OS 10% 48.

Revised: 42M 114G 21.

Revised: 42M 122G 20.

Revised: 42M 129G 18.

Revised: 42M 137G 18.

Revised: 42M 145G 17.

Revised: DAM 10% 48.

Revised: 41M 113G 21.

Revised: 41M 121G 20.

Revised: 41M 129G 18.

Revised: 41M 136G 18.

Revised: 41M 144G 17.

Revised: [Figure]

Revised: layers are stacked, while the Pre-LN technique preserves gradient magnitude even in deeper early layers.

Revised: (c) :

Revised: Layerwise gradient norm of Deformable-DETR with Swin-T backbone.

Revised: In this case, the Pre-LN fails to prevent vanishing gradient while the encoder auxiliary loss(denoted as Post-LN + EncAux) proposed in this paper effectively resolves this issue.

Original: A. E XPERIMENTAL DETAILS FOR DIFFERENT QUERY SELECTION CRITERIA
Revised: A. E XPERIMENTAL DETAILS FOR DIFFERENT TOKEN SELECTION CRITERIA

Original: Table 3and 4 contains  speciﬁc values used to plot (a) ResNet50 and (b) Swin-T backbone in Figure 4, respectively.
Revised: Table 3 contains the speciﬁc values used to plot (a) ResNet-50 and (b) Swin-T backbone in Figure 4.

Original: Additionally, they also include a lower-bound baseline that has no scoring method with keeping ratio 0%, meaning that the entire encoder block is removed and the backbone features are directly passed to the decoder.
Revised: Additionally, they also include a lower-bound baseline that has no scoring method with keeping ratio 0%, meaning that the entire encoder block is removed and the backbone features are directly passed to the decoder.

Original: Even with the lowest keeping ratio 10%, all the scoring methods including random criterion outperform this lower-bound baseline.
Revised: Even with the lowest keeping ratio 10%, all the scoring methods including random criterion outperform this lower-bound baseline.

Original: Note that all experiments reported in Table 3and 4 except for the keeping ratio 0% use the encoder auxiliary loss for training.
Revised: Note that all experiments reported in Table 3 except for the keeping ratio 0% use the encoder auxiliary loss for training.

Original: A. E XPERIMENTAL DETAILS FOR EFFECTIVENESS OF THE ENCODER AUXILIARY LOSS .
Revised: A. V ANISHING GRADIENT PROBLEM IN THE DEEP END - TO - END DETECTORS

Revised: As shown in Section 4.2 in Carion et al.

Revised: (2020), they observe that the performance of DETR gradually improves with more encoder layers.

Revised: To reproduce this result, we used the default settings of the ofﬁcial code, but only changed the number of encoder layers.

Revised: However, we fail to train the DETR model when using more than 9 encoder layers, which is probably due to different hyperparameters from the ones used in their experiments.

Revised: Interestingly, we also found that the DETR model converges stably with the Pre-LN architecture(Baevski & Auli, 2019; Child et al., 2019b; Wang et al., 2019) that is known to be a better choice than the canonical Post-LN when the number of layers of

Revised: [Table]

Revised: transformer increases.

Revised: We used the pre norm option the authors already have implemented in their code.

Revised: Figure 9(a) and 9(b) illustrate that gradient norm of each layer from bottom to top in 6 and 12 encoder layers when using Post-LN and Pre-LN, respectively.

Revised: We compute the (cid:96) 2 -norm of the gradients for all parameters in a particular layer, as if they are concatenated into a single vector.

Revised: To see training dynamics in the early stage of training, we track the gradients computed on a ﬁxed set of training data and average them over the ﬁrst 150 steps with a batch size of 2.

Original: Table 5 presents the detailed values of Figure 5.
Revised: Note that we applied Pre-LN only to the encoder module for a fair comparison between only the early modules although it could be used in any other transformer modules, e.g. backbone or decoder. We found that the vanishing gradient issue is generally observed regardless of the encoder size. When we double the size of the encoder, as one can expect, the gradient in the early layers ends up with an even smaller scale, which may have caused the convergence failure. Meanwhile, the PreLN technique seems to signiﬁcantly alleviate this issue even for the deeper encoder by maintaining the gradient scale evenly through the encoder layers and conveying a strong training signal to the backbone layers.

Original: As shown in  the table, training of a deeper encoder of more than 9 layers fails without the auxiliary loss, but if it is adopted, the convergence becomes feasible as the intermediate gradients provided to the earlyencoder layers may augment the vanishing gradients propagated from the decoder layers.
Revised: On the other hand, as shown in Figure 9(c), Deformable-DETR suffers from the same problem of vanishing gradient and even the Pre-LN technique does not help in this case. Meanwhile, the encoder auxiliary loss proposed in our paper drastically ampliﬁes the gradient magnitude in the early layers by providing aggressive intermediate objectives for each encoder layer.

Original: [Table] [Table] 41M 136G 18. 10% (cid:88) 48.
Revised: Note that it also creates good synergy with our main sparsiﬁcation strategy owing to reduced training cost.

Original: 41M 113G 21.
Revised: We claim that this observation supports our motivation of introducing the encoder auxiliary loss.

Original: 20% (cid:88) 48. 41M 121G 20. 30% (cid:88) 49.
Revised: A. E XPERIMENTAL DETAILS FOR EFFECTIVENESS OF THE ENCODER AUXILIARY LOSS

Original: 41M 129G 18. 40% (cid:88) 49. 41M 136G 18.
Revised: Table 4 presents the detailed values of Figure 4. As shown in the Table 4 and discussed in Section

Original: 100% (cid:88) 49. 44M 220G 12. 100% (cid:88) 50. 46M 261G 11.
Revised: A.5, training of a deeper encoder of more than 9 layers fails without the auxiliary loss, but if it is adopted, the convergence becomes feasible as the intermediate gradients provided to the early encoder layers augment the vanishing gradient back-propagated from the decoder module.

Original: 10% (cid:88) 49.
Revised: [Table]

Original: 46M 128G 19. 20% (cid:88) 49. 46M 143G 17.
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: % (cid:88) 49.

Original: 46M 158G 15.
Revised: 101M 358G 8.

Original: 40% (cid:88) 49.
Revised: Recently, many self-supervised learning methods through contrastive learning have been studied, and in particular, methods for obtaining dense representations with better performance for localization downstream tasks such as object detection are in the spotlight. In order to check whether our proposed method is effective even when such dense representation is used, the backbone network is initialized with the SCRL (Roh et al., 2021) model that aims to learn dense representations in a selfsupervised way instead of initializing with the ImageNet (Deng et al., 2009) pre-trained one.

Original: 46M 173G 14.
Revised: Just as the SCRL model outperformed the ImageNet pre-trained model in various localization downstream tasks, our proposed method, Sparse DETR, also shows better performance in all keeping ratios( ρ ) without the inﬂuence of encoder token sparsiﬁcation.

Original: A. U SING A LARGER TRANSFORMER - BASED BACKBONE (S WIN -B)
Revised: A. U SING A LARGER TRANSFORMER - BASED BACKBONE (S WIN -B)

Original: We perform experiments on Sparse DETR with Swin-Base(Liu et al., 2021) backbone to see if our method shows similar efﬁciency and performance gain even when using a heavier transformer-based backbone.
Revised: We perform experiments on Sparse DETR with Swin-Base(Liu et al., 2021) backbone to see if our method shows similar efﬁciency and performance gain even when using a heavier transformer-based backbone.

Original: Table 6 illustrates a comparison of COCO detection performance between Swin-T and
Revised: Table 6 illustrates a comparison of COCO detection performance between Swin-T and

Original: Swin-B backbone under the varied sparsity.
Revised: Swin-B backbone under the varied sparsity.

Original: Due to the increased capacity, using Swin-B backbone signiﬁcantly boosts up the baseline AP up to 52.5 (+4.5) but with 2.4 × parameters and 2.1 × computational cost.
Revised: Due to the increased capacity, using Swin-B backbone signiﬁcantly boosts up the baseline AP up to 52.5 (+4.5) but with 2.4 × parameters and 2.1 × com- Ratio of [Figure] I m age

Revised: [Figure]

Original:  With the keeping ratio of 40% and the encoder auxiliary loss, the performance gap remains at a similar level(+4.1).
Revised: =0. [Table] putational cost. With the keeping ratio of 40% and the encoder auxiliary loss

Original:  We can also observe consistent performance gains as the keeping ratio gets higher while the increasing gap converges more quickly than Swin-T.
Revised: [Table] the performance gap remains at a similar level(+4.1). We can also observe consistent performance gains as the keeping ratio gets higher while the increasing gap converges more quickly than Swin-T.

Original: It may be because a single visual token of Swin-B can incorporate a wider range of information due to the deeper at- tention hierarchies and a smaller number of tokens is required to fully represent all the objects in an image.
Revised: It may be because a single visual token of Swin-B can incorporate a wider range of information due to the deeper attention hierarchies and a smaller number of tokens is required to fully represent all the objects in an image.

Original: Note that the efﬁciency of a backbone network is behind the scope of this paper.
Revised: Note that the efﬁciency of a backbone network is behind the scope of this paper.

Original: Our work is orthogonal to the backbone sparsiﬁcation approaches, e.g. DynamicViT
Revised: Our work is orthogonal to the backbone sparsiﬁcation approaches, e.g. DynamicViT

Original: (Rao et al., 2021), and we leave the integration with those works as future work.
Revised: (Rao et al., 2021), and we leave the integration with those works as future work.

Revised: A. T HE PRELIMINARY EXPERIMENTS : W HY PURSUE A SPARSE ENCODER ?

Revised: Using a model trained with Deformable-DETR, we have analyzed the number of encoder output tokens referenced by the decoder’s object query.

Revised: Unlike using the bilinear interpolation described in the appendix A.2 to generate DAMs for training with pseudo-labels, in this analysis, we do not use bilinear interpolation to calculate how many encoder tokens are directly referenced by the decoder object query.

Revised: To analyze the non-zero values of DAM, we use a Deforamble DETR model trained with Top- k sampling strategy (Yao et al., 2021) and bounding box reﬁnement (Zhu et al., 2021) using ResNet-50 backbone.

Original: [Table]
Revised: Fig. 10 illustrates the distribution of the ratio of non-zero values of DAM on COCO val2017 dataset. As shown in Fig.

Revised: 10, on average, only 45% of encoder tokens were referenced by object queries.

Revised: This observation naturally raises a question: Can we preserve the detection performance even if we focus, in the ﬁrst place, only on the encoder tokens that the decoder might have preferred?

Revised: As a preliminary experiment to answer this question, we trained the detector restricting token updates to the subset to which the decoder could have referred if there had been no such restriction.

Revised: To this end, we performed the two-stage learning as follows: (i)

Revised: We ﬁrst obtained the DAMs of the entire training data by feeding them to a fully-trained Deformable-DETR model.

Revised: (ii)

Revised: Then, we retrained another model from the scratch by updating only a subset of tokens determined by the binarized

Revised: DAM preserving top- ρ % of the elements(refer to Section3.3 for more details).

Revised: Table 7 shows the performance on COCO detection for different keeping ratio ρ .

Revised: We found that the two-stage model almost catches up with the baseline( ρ = 100% ) as the keeping ratio is raised close to 45%, namely the percentage of non-zero values in DAM computed on the validation dataset earlier.

Revised: These observations have strongly motivated us to develop the encoder token sparsiﬁcation method presented in the main text.

Revised: Note that our main algorithm differs from this preliminary experiment in some aspects: (a) A DAM is obtained from the jointly learning decoder, not from the separately trained decoder, and (b) a binarized DAM is utilized as a prediction target of the scoring network rather than used directly as a sparsiﬁcation mask.

Original: A. V ISUALIZATIONS OF S ELECTED E NCODER Q UERIES
Revised: A. V ISUALIZATIONS OF S ELECTED E NCODER TOKENS

Original: We visualize selected encoder queries and top- k decoder queries for each criterion, OS, and DAM.
Revised: We visualize selected encoder tokens and top- k decoder queries for each criterion, OS, and DAM.

Original: In the ﬁrst row, selected encoder queries from the backbone feature map are visualized as yellow regions, whereas unselected queries are visualized as purple regions.
Revised: In the ﬁrst row, selected encoder tokens from the backbone feature map are visualized as yellow regions, whereas unselected tokens are visualized as purple regions.

Original: In the second row, selected top- k decoder queries from encoder output are visualized in the same manner.
Revised: In the second row, selected topk decoder queries from encoder output are visualized in the same manner.

Original: In the ﬁnal row, DAM values and Corr metrics are visualized.
Revised: In the ﬁnal row, DAM values and Corr metrics are visualized.

Original: Corr is measured as in section 4.2.
Revised: Corr is measured as in Section 4.2.

Original: Interestingly, the DAM-based selection seems to better capture the objects than OS-based selection.
Revised: Interestingly, the DAM-based selection seems to better capture the objects than OS-based selection.

Original: The OS-based selection also captures objects well, but it typically focuses on the high-frequency edges that are not only in the foreground but also in the background.
Revised: The OS-based selection also captures objects well, but it typically focuses on the high-frequency edges that are not only in the foreground but also in the background.

Original: On the other hand, the DAMbased selection captures the boundary of the objects and also their inner areas and is less distracted from the background edges.
Revised: On the other hand, the DAMbased selection captures the boundary of the objects and also their inner areas and is less distracted from the background edges.

Original: We analyze that DAM focuses on the boundary of objects to lower the regression loss, and attends to the inside of objects to lower the classiﬁcation loss.
Revised: We analyze that DAM focuses on the boundary of objects to lower the regression loss, and attends to the inside of objects to lower the classiﬁcation loss.

Original: Finally, the scoring network predicts such a DAM well, and reﬁning the encoder queries according to it ﬁnally helps achieve better detection performance.
Revised: Finally, the scoring network predicts such a DAM well, and reﬁning the encoder tokens according to it ﬁnally helps achieve better detection performance.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]