Original: L EARNING D YNAMIC Q UERY C OMBINATIONS FOR T RANSFORMER - BASED O BJECT D ETECTION AND
Revised: L EARNING D YNAMIC Q UERY C OMBINATIONS FOR T RANSFORMER - BASED O BJECT D ETECTION AND

Original: S
Revised: S

Original: EGMENTATION
Revised: EGMENTATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Transformer-based detection and segmentation methods use a list of learned detection queries to retrieve information from the transformer network and learn to predict the location and category of one specific object from each query.
Revised: Transformer-based detection and segmentation methods use a list of learned detection queries to retrieve information from the transformer network and learn to predict the location and category of one specific object from each query.

Original: We empirically find that random convex combinations of the learned queries are still good queries for the corresponding models.
Revised: We empirically find that random convex combinations of the learned queries are still good queries for the corresponding models.

Original: We then propose to learn a convex combination with dynamic coefficients based on the high-level semantics of the image.
Revised: We then propose to learn a convex combination with dynamic coefficients based on the high-level semantics of the image.

Original: The generated dynamic queries better capture the prior of object locations and categories in the different images.
Revised: The generated dynamic queries better capture the prior of object locations and categories in the different images.

Original: Equipped with our dynamic queries, a wide range of DETR-based models achieve consistent and superior performance across multiple tasks (object detection, instance segmentation, panoptic segmentation) and on different benchmarks (MS COCO, CityScapes, YoutubeVIS).
Revised: Equipped with our dynamic queries, a wide range of DETR-based models achieve consistent and superior performance across multiple tasks (object detection, instance segmentation, panoptic segmentation) and on different benchmarks (MS COCO, CityScapes, YoutubeVIS).

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Object detection is a fundamental yet challenging task in computer vision, which aims to localize and categorize objects of interest in the images simultaneously.
Revised: Object detection is a fundamental yet challenging task in computer vision, which aims to localize and categorize objects of interest in the images simultaneously.

Original: Traditional detection models (Ren et al., 2015; Cai & Vasconcelos, 2019; Duan et al., 2019; Lin et al., 2017b;a) use complicated anchor designs and heavy post-processing steps such as Non-Maximum-Suppression (NMS) to remove duplicated detections.
Revised: Traditional detection models (Ren et al., 2015; Cai & Vasconcelos, 2019; Duan et al., 2019; Lin et al., 2017b;a) use complicated anchor designs and heavy post-processing steps such as Non-Maximum-Suppression (NMS) to remove duplicated detections.

Original: Recently, Transformer-based object detectors such as DETR (Carion et al., 2020) have been introduced to simplify the process.
Revised: Recently, Transformer-based object detectors such as DETR (Carion et al., 2020) have been introduced to simplify the process.

Original: In detail, DETR combines convolutional neural networks (CNNs) with Transformer (Vaswani et al., 2017) by introducing an encoder-decoder framework to generate a series of predictions from a list of object queries.
Revised: In detail, DETR combines convolutional neural networks (CNNs) with Transformer (Vaswani et al., 2017) by introducing an encoder-decoder framework to generate a series of predictions from a list of object queries.

Original: Following works improve the efficiency and convergence speed of DETR with modifications to the attention module (Zhu et al., 2021; Roh et al., 2021), and divide queries into positional and content queries (Liu et al., 2022; Meng et al., 2021).
Revised: Following works improve the efficiency and convergence speed of DETR with modifications to the attention module (Zhu et al., 2021; Roh et al., 2021), and divide queries into positional and content queries (Liu et al., 2022; Meng et al., 2021).

Original: This paradigm is also adopted for instance/panoptic segmentation, where each query is associated with one specific object mask in the decoding stage of the segmentation model (Cheng et al., 2021a).
Revised: This paradigm is also adopted for instance/panoptic segmentation, where each query is associated with one specific object mask in the decoding stage of the segmentation model (Cheng et al., 2021a).

Original: The existing DETR-based detection models always use a list of fixed queries, regardless of the input image.
Revised: The existing DETR-based detection models always use a list of fixed queries, regardless of the input image.

Original: The queries will attend to different objects in the image through a multi-stage attention process.
Revised: The queries will attend to different objects in the image through a multi-stage attention process.

Original: Here, the queries are served as global priors for the location and semantics of target objects in the image.
Revised: Here, the queries are served as global priors for the location and semantics of target objects in the image.

Original: In this paper, we would like to associate the detection queries with the content of the image, i.e., adjusting detection queries based on the high-level semantics of the image in order to capture the distribution of object locations and categories in this specific scene.
Revised: In this paper, we would like to associate the detection queries with the content of the image, i.e., adjusting detection queries based on the high-level semantics of the image in order to capture the distribution of object locations and categories in this specific scene.

Original: For example, when the highlevel semantics show the image is a group photo, we know that there will be a group of people (category) inside the image and they are more likely to be close to the center of the image (location).
Revised: For example, when the highlevel semantics show the image is a group photo, we know that there will be a group of people (category) inside the image and they are more likely to be close to the center of the image (location).

Original: Since the detection queries are implicit features that do not directly relate to specific locations and object categories in the DETR framework, it is hard to design a mechanism to change the queries while keeping them within a meaningful “query” subspace to the model.
Revised: Since the detection queries are implicit features that do not directly relate to specific locations and object categories in the DETR framework, it is hard to design a mechanism to change the queries while keeping them within a meaningful “query” subspace to the model.

Original: Through an empirical study, we notice that convex combinations of learned queries are still good queries to different
Revised: Through an empirical study, we notice that convex combinations of learned queries are still good queries to different

Original: DETR-based models, achieving similar performance as the originally learned queries (See Section 3.2).
Revised: DETR-based models, achieving similar performance as the originally learned queries (See Section 3.2).

Original: Motivated by this, we propose a method to generate dynamic detection queries based on the high-level semantics of the image in DETR-based methods while constraining the generated queries in a sequence of convex hulls spanned by the static queries.
Revised: Motivated by this, we propose a method to generate dynamic detection queries based on the high-level semantics of the image in DETR-based methods while constraining the generated queries in a sequence of convex hulls spanned by the static queries.

Original: Therefore, the generated detection queries are more related to the target objects in the image and stay in a meaningful subspace.
Revised: Therefore, the generated detection queries are more related to the target objects in the image and stay in a meaningful subspace.

Original: Extensive experiments on MS COCO (Lin et al., 2014), CityScapes (Cordts et al., 2016) and YouTube-VIS
Revised: Extensive experiments on MS COCO (Lin et al., 2014), CityScapes (Cordts et al., 2016) and YouTube-VIS

Original: (Yang et al., 2019) benchmarks with multiple tasks, including object detection, instance segmentation, and panoptic segmentation show superior performance of our approach combined with a wide range of DETR-based models.
Revised: (Yang et al., 2019) benchmarks with multiple tasks, including object detection, instance segmentation, and panoptic segmentation show superior performance of our approach combined with a wide range of DETR-based models.

Original: In Figure 1, we show the performance of our method on object detection combined with two baseline models.
Revised: In Figure 1, we show the performance of our method on object detection combined with two baseline models.

Original: When integrated with our proposed method, the mAP of recent detection models DAB-Deformable-DETR (Liu et al., 2022) can be increased by 1 . 6% .
Revised: When integrated with our proposed method, the mAP of recent detection models DAB-Deformable-DETR (Liu et al., 2022) can be increased by 1 . 6% .

Original: With fewer dynamic detection queries and less computation in the transformer decoder, our method can still achieve better performance than baseline models on both Deformable-DETR and
Revised: With fewer dynamic detection queries and less computation in the transformer decoder, our method can still achieve better performance than baseline models on both Deformable-DETR and

Original: DAB-Deformable-DETR.
Revised: DAB-Deformable-DETR.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Transformers for object detection.
Revised: Transformers for object detection.

Original: Traditional CNN-based object detectors require manually designed components such as anchors (Ren et al., 2015; Tian et al., 2019) or post-processing steps such as NMS(Neubeck & Van Gool, 2006; Hosang et al., 2017).
Revised: Traditional CNN-based object detectors require manually designed components such as anchors (Ren et al., 2015; Tian et al., 2019) or post-processing steps such as NMS(Neubeck & Van Gool, 2006; Hosang et al., 2017).

Original: Transformer-based detectors directly generate predictions for a list of target objects with a series of learnable queries.
Revised: Transformer-based detectors directly generate predictions for a list of target objects with a series of learnable queries.

Original: Among them, DETR (Carion et al., 2020) first combines the sequence-to-sequence framework with learnable queries and CNN features for object detection.
Revised: Among them, DETR (Carion et al., 2020) first combines the sequence-to-sequence framework with learnable queries and CNN features for object detection.

Original: Following DETR, multiple works were proposed to improve its convergence speed and accuracy.
Revised: Following DETR, multiple works were proposed to improve its convergence speed and accuracy.

Original: Deformable-DETR (Zhu et al., 2021) and SparseDETR (Roh et al., 2021) replace the self-attention modules with more efficient attention operations where only a small set of key-value pairs are used for calculation.
Revised: Deformable-DETR (Zhu et al., 2021) and SparseDETR (Roh et al., 2021) replace the self-attention modules with more efficient attention operations where only a small set of key-value pairs are used for calculation.

Original: Conditional-DETR (Tian et al., 2020) changes the queries in DETR to be conditional spatial queries, which speeds up the convergence process.
Revised: Conditional-DETR (Tian et al., 2020) changes the queries in DETR to be conditional spatial queries, which speeds up the convergence process.

Original: Anchor-DETR (Wang et al., 2021a) generates the object queries using anchor points rather than a set of learnable embeddings.
Revised: Anchor-DETR (Wang et al., 2021b) generates the object queries using anchor points rather than a set of learnable embeddings.

Original: DAB-DETR (Liu et al., 2022) directly uses learnable box coordinates as queries which can be refined in the Transformer decoder layers.
Revised: DAB-DETR (Liu et al., 2022) directly uses learnable box coordinates as queries which can be refined in the Transformer decoder layers.

Original: DINO (Zhang et al., 2022) introduces a strategy to train models with noisy ground truthsand select a subset of queries in inference to improve the performance.
Revised: DINO (Zhang et al., 2022) and DN-DETR Li et al. (2022) introduce a strategy to train models with noisy ground truths to help the model learn the representation of the positive samples more efficiently. Recently, Group-DETR Chen et al.

Revised: and

Original:  All these Transformer-based detection methods use fixed initial detection queries learned on the whole dataset.
Revised: HDETR Jia et al. both added auxiliary queries and a one-to-many matching loss to improve the convergence of the DETR-based models. They still use static queries which does not change the general architecture of DETR. All these Transformerbased detection methods use fixed initial detection queries learned on the whole dataset.

Original: In contrast, we propose to modulate the queries based on the image’s content, which generates more effective queries for the current image.
Revised: In contrast, we propose to modulate the queries based on the image’s content, which generates more effective queries for the current image.

Original: Transformers for object segmentation.
Revised: Transformers for object segmentation.

Original: Besides object detection, Transformer-based models are also proposed for object segmentation tasks including image instance segmentation, panoptic segmentation (Kirillov et al., 2019) and video instance segmentation (VIS) (Yang et al., 2019).
Revised: Besides object detection, Transformer-based models are also proposed for object segmentation tasks including image instance segmentation, panoptic segmentation (Kirillov et al., 2019; Wang et al., 2021a; Zhang et al., 2021) and video instance segmentation (VIS) (Yang et al., 2019).

Original: In DETR (Carion et al., 2020), a mask head is introduced on top of the decoder outputs to generate the predictions for panoptic segmentation.
Revised: In DETR (Carion et al., 2020), a mask head is introduced on top of the decoder outputs to generate the predictions for panoptic segmentation.

Original: Following DETR, ISTR (Hu et al., 2021) generates lowdimensional mask embeddings, which are matched with the ground truth mask embeddings using Hungarian Algorithm for instance segmentation.
Revised: Following DETR, ISTR (Hu et al., 2021) generates low-dimensional mask embeddings, which are matched with the ground truth mask embeddings using Hungarian Algorithm for instance segmentation.

Original: SOLQ (Dong et al., 2021) uses a unified query representation for class, location, and object mask.
Revised: SOLQ (Dong et al., 2021) uses a unified query representation for class, location, and object mask.

Original: Mask2Former(Cheng et al., 2021a) introduces
Revised: Mask2Former

Original: [Table]
Revised: [Table]

Original:  masked attention to extract localized features, and predictoutput for panoptic, instance and semantic segmentation in a unified framework. SeqFormer (Wu et al., utilizes video level instance queries where each query attends to a specific object across frames in the video. These Transformerbased models follow the general paradigm of DETR and use fixed queries regardless of the input
Revised: introduces masked attention to extract localized features predict

Original: Dynamic deep neural networks.
Revised: Dynamic deep neural networks.

Original: Dynamic deep neural networks (Han et al., 2021) aim at adjusting the computation procedure of a neural network adaptively in order to reduce the overall computation cost or enhance the model capacity.
Revised: Dynamic deep neural networks (Han et al., 2021) aim at adjusting the computation procedure of a neural network adaptively in order to reduce the overall computation cost or enhance the model capacity.

Original: Slimmable networks (Yu et al., 2018; Yu & Huang, 2019; Li et al., 2021) introduce a strategy to adapt to multiple devices by simply changing channel numbers without the need for retraining.
Revised: Slimmable networks (Yu et al., 2018; Yu & Huang, 2019; Li et al., 2021) introduce a strategy to adapt to multiple devices by simply changing channel numbers without the need for retraining.

Original: Dynamic Convolution (Chen et al., 2020) proposes a dynamic perceptron that uses dynamic attention weights to aggregate multiple convolution kernels based on the input features.
Revised: Dynamic Convolution (Chen et al., 2020) proposes a dynamic perceptron that uses dynamic attention weights to aggregate multiple convolution kernels based on the input features.

Original: On object detection, Dynamic R-CNN (Zhang et al., 2020) proposes a new training strategy to dynamically adjust the label assignment for two-stage object detectors based on the statics of proposals.
Revised: On object detection, Dynamic R-CNN (Zhang et al., 2020) proposes a new training strategy to dynamically adjust the label assignment for two-stage object detectors based on the statics of proposals.

Original: Cui et al.
Revised: Cui et al.

Original: (2022) proposes to train a single detection model which can adjust the number of proposals based on the complexity of the input image.
Revised: (2022) proposes to train a single detection model which can adjust the number of proposals based on the complexity of the input image.

Original: Wang et al.
Revised: Wang et al.

Original: (2021b) introduces a Dynamic Transformer to determine the number of tokens according to the input image for efficient image recognition, by stacking multiple Transformer layers with increasing numbers of tokens.
Revised: (2021c) introduces a Dynamic Transformer to determine the number of tokens according to the input image for efficient image recognition, by stacking multiple Transformer layers with increasing numbers of tokens.

Original: In contrast to the existing work, we explore generating dynamic queries for a wide range of
Revised: In contrast to the existing work, we explore generating dynamic queries for a wide range of

Original: DETR-based models using the same framework.
Revised: DETR-based models using the same framework.

Original: Our focus is not to reduce the computation cost of DETR-based models, but to improve the model performances with queries more related to the content of each individual image.
Revised: Our focus is not to reduce the computation cost of DETR-based models, but to improve the model performances with queries more related to the content of each individual image.

Original:  M ETHODOLOGY 3. P RELIMINARY
Revised:  M ETHODOLOGY 3. P RELIMINARY

Original: We first summarize the inference process of the existing Transformer-based models for a series of tasks, including object detection, instance segmentation, and panoptic segmentation, as the following
Revised: We first summarize the inference process of the existing Transformer-based models for a series of tasks, including object detection, instance segmentation, and panoptic segmentation, as the following

Original: Equation:
Revised: Equation:

Original: [Equation]
Revised: [Equation]

Original: For the object detection task, given the input image I , multi-scale features F are extracted from the backbone network and then fed into a Transformer encoder N enc .
Revised: For the object detection task, given the input image I , multi-scale features F are extracted from the backbone network and then fed into a Transformer encoder N enc .

Original: After processing the features with multiple encoder layers, the output features are fed into a Transformer decoder N dec together with n randomly initialized query vectors Q ∈ R n × f , where n and f denote the number of queries and length of each query respectively.
Revised: After processing the features with multiple encoder layers, the output features are fed into a Transformer decoder N dec together with n randomly initialized query vectors Q ∈ R n × f , where n and f denote the number of queries and length of each query respectively.

Original: Each query can be a feature vector (Carion et al., 2020; Zhu et al., 2021), or a learned anchor box (Liu et al., 2022).
Revised: Each query can be a feature vector (Carion et al., 2020; Zhu et al., 2021), or a learned anchor box (Liu et al., 2022).

Original: The outputs of N dec is then fed into a task head N t to generate the final predictions Y = { ( b i , c i ) , i = 1 , 2 , . . . , n } , where b i , c
Revised: The outputs of N dec is then fed into a task head N t to generate the final predictions Y = { ( b i , c i ) , i = 1 , 2 , . . . , n } , where b i , c

Original: i represent the bounding boxes and their corresponding categories of the detected objects.
Revised: i represent the bounding boxes and their corresponding categories of the detected objects.

Original: Then, the predictions are matched with the ground truths Y ⋆ using the Hungarian Algorithm (Carion et al., 2020) to generate a bipartite matching.
Revised: Then, the predictions are matched with the ground truths Y ⋆ using the Hungarian Algorithm (Carion et al., 2020) to generate a bipartite matching.

Original: Then, the final loss is computed based on this bipartite matching:
Revised: Then, the final loss is computed based on this bipartite matching:

Original: [Equation]
Revised: [Equation]

Original: For the segmentation tasks, the final predictions are updated to Y = { ( b i , c i , m i ) , i = 1 , 2 , . . . , n } , where m i denotes the predicted masks for different object instances.
Revised: For the segmentation tasks, the final predictions are updated to Y = { ( b i , c i , m i ) , i = 1 , 2 , . . . , n } , where m i denotes the predicted masks for different object instances.

Original: Since there is no direct correspondence of the predictions with the ground truth annotations, a bipartite matching is also com- puted to find the correspondence of the predictions and the ground truths Y ⋆ .
Revised: Since there is no direct cor- respondence of the predictions with the ground truth annotations, a bipartite matching is also computed to find the correspondence of the predictions and the ground truths Y ⋆ .

Original: The final loss is then computed based on the matching.
Revised: The final loss is then computed based on the matching.

Original: In some models such as Mask2Former (Cheng et al., 2021a), there will be no Transformer encoder N enc to enhance the feature representations, while the other computational components follow the same paradigm.
Revised: In some models such as Mask2Former (Cheng et al., 2021a), there will be no Transformer encoder N enc to enhance the feature representations, while the other computational components follow the same paradigm.

Original: 3. F IXED Q UERY C OMBINATIONS
Revised: 3. F IXED Q UERY C OMBINATIONS

Original: Though some existing works analyze the contents of the queries for the decoder, such as ConditionalDETR (Tian et al., 2020) and Anchor-DETR (Wang et al., 2021a), they always exam each query individually.
Revised: Though some existing works analyze the contents of the queries for the decoder, such as ConditionalDETR (Tian et al., 2020) and Anchor-DETR (Wang et al., 2021b), they always exam each query individually.

Original: To the best of our knowledge, there is no work studying the interaction between the queries in Q .
Revised: To the best of our knowledge, there is no work studying the interaction between the queries in Q .

Original: Here, we would like to explore what kind of transformations conducted between the learned queries still generate “good” queries.
Revised: Here, we would like to explore what kind of transformations conducted between the learned queries still generate “good” queries.

Original: If we compute the average of a few queries, is it still an effective query?
Revised: If we compute the average of a few queries, is it still an effective query?

Original: If we use different types of linear transformations, which would be better to produce good queries?
Revised: If we use different types of linear transformations, which would be better to produce good queries?

Original: We conduct experiments to analyze the results of queries generated by different perturbations from the original queries.
Revised: We conduct experiments to analyze the results of queries generated by different perturbations from the original queries.

Original: The procedure of the experiments is as follows: given a welltrained Transformer-based model, the initial queries for the decoder are denoted as Q P = (cid:8) q P 1 , q P 2 , . .
Revised: The procedure of the experiments is as follows: given a welltrained Transformer-based model, the initial queries for the decoder are denoted as Q P = (cid:8) q P 1 , q P 2 , . .

Original: , q P n (cid:9) ∈ R n × f .
Revised: , q Pn (cid:9) ∈ R n × f .

Original: The first type of perturbation uses linear combinations of the original queries.
Revised: The first type of perturbation uses linear combinations of the original queries.

Original: We first separate the n queries into m groups, where each group has r = nm queries and generates one new query.
Revised: We first separate the n queries into m groups, where each group has r = nm queries and generates one new query.

Original: Then, we initialize the combination coefficients W ∈ R m × r , where w ij ∈ W is the coefficient used for the i -th group, j -th queries, denoted as q Pij , to generate a group of new queries Q C = { q [Equation] 1 , q [Equation]
Revised: Then, we initialize the combination coefficients W ∈ R m × r , where w ij ∈ W is the coefficient used for the i -th group, j -th queries, denoted as q Pij , to generate a group of new queries Q C = { q C 1 , q C 2 , . . .

Original:  q [Equation] } ∈ R m × f .
Revised: , q Cm } ∈ R m × f .

Original: The process can be summarized as:
Revised: The process can be summarized as:

Original: [Equation]
Revised: [Equation]

Original: We use three settings to evaluate the impact of different coefficients in Equation 3, namely Convex
Revised: We use three settings to evaluate the impact of different coefficients in Equation 3, namely Convex

Original: Combination, Non-convex Combination, and Averaged Combination.
Revised: Combination, Non-convex Combination, and Averaged Combination.

Original: In Convex Combination, q Ci is within the convex hull of q Pij , j = 1 , 2 , . . .
Revised: In Convex Combination, q Ci is within the convex hull of q Pij , j = 1 , 2 , . . .

Original: , r .
Revised: , r .

Original: The combination coefficients w ij are randomly initialized using uniform distribution in [ − 1 , 1] and then passed through a softmax function to satisfy the criteria: w ij ≥ 0 , (cid:80) rj
Revised: The combination coefficients w ij are randomly initialized using uniform distribution in [ − 1 , 1] and then passed through a softmax function to satisfy the criteria: w ij ≥ 0 , (cid:80) rj

Original: =1 w ij = 1 .
Revised: =1 w ij = 1 .

Original: For Non-convex Combination, w ij are initialized in the same way as those in the convex combination, and the sum of w ij is forced to be 1 .
Revised: For Non-convex Combination, w ij are initialized in the same way as those in the convex combination, and the sum of w ij is forced to be 1 .

Original: However, there is no guarantee on its range and w ij can be negative values.
Revised: However, there is no guarantee on its range and w ij can be negative values.

Original: For Averaged Combination, we generate q Ci by averaging q Pij , j = 1 , 2 , . . .
Revised: For Averaged Combination, we generate q Ci by averaging q Pij , j = 1 , 2 , . . .

Original: , r .
Revised: , r .

Original: As a baseline, we evaluate the model on m queries randomly sampled from Q P .
Revised: As a baseline, we evaluate the model on m queries randomly sampled from Q P .

Original: The experiments are conduct on MS COCO benchmark (Lin et al., 2014) for object detection, and instance segmentation, using DAB-DETR (Liu et al., 2022), DeformableDETR (Zhu et al., 2021) and Mask2Former (Cheng et al., 2021a), with ResNet-50 as the backbone.
Revised: The experiments are conduct on MS COCO benchmark (Lin et al., 2014) for object detection, and instance segmentation, using DAB-DETR (Liu et al., 2022), DeformableDETR (Zhu et al., 2021) and Mask2Former (Cheng et al., 2021a), with ResNet-50 as the backbone.

Original: The results are summarized in Table 1.
Revised: The results are summarized in Table 1.

Original: From Table 1, we notice that Convex Combination achieves the best results among all the compared settings except the baseline.
Revised: From Table 1, we notice that Convex Combination achieves the best results among all the compared settings except the baseline.

Original: Convex Combination only degenerate slightly compared with learned queries on DAB-DETR and Mask2Former.
Revised: Convex Combination only degenerate slightly compared with learned queries on DAB-DETR and Mask2Former.

Original: In addition, the performance of Convex Combination only has very small variances across different models, proving that convex combinations of the group-wise learned queries are naturally high-quality object queries for different Transformer-based models on both detection and segmentation tasks.
Revised: In addition, the performance of Convex Combination only has very small variances across different models, proving that convex combinations of the group-wise learned queries are naturally high-quality object queries for different Transformer-based models on both detection and segmentation tasks.

Original: n is set to 300 for detection models and 100 for Mask2Former.
Revised: n is set to 300 for detection models and 100 for Mask2Former.

Original: We run each setting 6 times to compute the variance.
Revised: We run each setting 6 times to compute the variance.

Original: 3. D YNAMIC Q UERY C OMBINATIONS
Revised: 3. D YNAMIC Q UERY C OMBINATIONS

Original: From the previous section, we learn that fixed convex combinations of learned queries are still able to produce a reasonable accuracy compared to the learned queries.
Revised: From the previous section, we learn that fixed convex combinations of learned queries are still able to produce a reasonable accuracy compared to the learned queries.

Original: In this section, we propose a strategy to learn dynamic query combinations for the Transformer-based models instead of randomly generating the coefficients w ij for query combinations.
Revised: In this section, we propose a strategy to learn dynamic query combinations for the Transformer-based models instead of randomly generating the coefficients w ij for query combinations.

Original: Our model predicts their values according to the high-level content of the input images.
Revised: Our model predicts their values according to the high-level content of the input images.

Original: Therefore, each input image will have a distinct set of object queries fed into the Transformer decoder.
Revised: Therefore, each input image will have a distinct set of object queries fed into the Transformer decoder.

Original: To generate dynamic queries, a naive idea is to generate the modulated queries directly from the input features F .
Revised: To generate dynamic queries, a naive idea is to generate the modulated queries directly from the input features F .

Original: This method will increase the number of parameters dramatically, causing it difficult  to optimize and inevitably computationally inefficient.
Revised: This method will increase the number of parameters dramatically, causing it difficult [Figure] to optimize and inevitably computationally inefficient.

Original: To verify this, we conduct an experiment on
Revised: To verify this, we conduct an experiment on

Original: [Figure]

Original: Deformable-DETR (Zhu et al., 2021) with ResNet-50 as the backbone.
Revised: Deformable-DETR (Zhu et al., 2021) with ResNet-50 as the backbone.

Original: We replace the original randomly initialized queries with those generated by a multi-layer perceptron (MLP), which transforms the image feature F to Q .
Revised: We replace the original randomly initialized queries with those generated by a multi-layer perceptron (MLP), which transforms the image feature F to Q .

Original: With 50 epochs, the model only achieves 45 .
Revised: With 50 epochs, the model only achieves 45 .

Original: 1% mAP, which is lower than the original model with 46 . 2% .
Revised: 1% mAP, which is lower than the original model with 46 . 2% .

Original: Inspired by the dynamic convolution (Chen et al., 2020), which aggregates the features with multiple kernels in each convolutional layer, we propose a query modulation method.
Revised: Inspired by the dynamic convolution (Chen et al., 2020), which aggregates the features with multiple kernels in each convolutional layer, we propose a query modulation method.

Original: We first introduce two types of queries: basic queries Q B ∈ R n × f and modulated queries Q M ∈ R m × f , where n, m are the number of queries and n = rm .
Revised: We first introduce two types of queries: basic queries Q B ∈ R n × f and modulated queries Q M ∈ R m × f , where n, m are the number of queries and n = rm .

Original: Equation 3 is updated as:
Revised: Equation 3 is updated as:

Original: [Equation]
Revised: [Equation]

Original: where W D ∈ R m × r is the combination coefficient matrix and w Dij ∈ W D is the coefficient for the i -th group, j -th query in Q B , denoted as q Bij .
Revised: where W D ∈ R m × r is the combination coefficient matrix and w Dij ∈ W D is the coefficient for the i -th group, j -th query in Q B , denoted as q Bij .

Original: To guarantee our query combinations to be convex, we add extra constraints to the coefficients as w Dij ≥ 0 ,
Revised: To guarantee our query combinations to be convex, we add extra constraints to the coefficients as w Dij ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: In our dynamic query combination module, the coefficient matrix W D is learned based on the input feature F through a mini-network, as:
Revised: In our dynamic query combination module, the coefficient matrix W D is learned based on the input feature F through a mini-network, as:

Original: [Equation]
Revised: [Equation]

Original: where A is a global average pooling to generate a global feature from the feature map F , θ is an MLP, σ is a softmax function to guarantee the elements of W D satisfy the convex constraints.
Revised: where A is a global average pooling to generate a global feature from the feature map F , θ is an MLP, σ is a softmax function to guarantee the elements of W D satisfy the convex constraints.

Revised: Here we try to make the mini-network as simple as possible to show the potential of using dynamic queries.

Revised: This attention-style structure happens to be a simple and effective design choice.

Original: During the training process, we feed both Q M and Q B to the same decoder to generate the corresponding predictions Y M and Y B as follows,
Revised: During the training process, we feed both Q M and Q B to the same decoder to generate the corresponding predictions Y M and Y B as follows,

Original: [Equation]
Revised: [Equation]

Original: The final training loss is then updated to
Revised: The final training loss is then updated to

Original: [Equation]
Revised: [Equation]

Original: where β is a hyperparameter.
Revised: where β is a hyperparameter.

Original: During the inference, only Q M is used to generate the final predictions
Revised: During the inference, only Q M is used to generate the final predictions

Original: Y M while the basic queries Q B are not used.
Revised: Y M while the basic queries Q B are not used.

Original: Therefore, the computational complexity increasesfor our models are negligible compared to the original DETR-based models. The only difference in the computation is that we have an additional MLP and a convex combination to generate the
Revised: Therefore, the computational complexity increases

Original: [Table]
Revised: [Table]

Original: modulated queries.
Revised: based models.

Revised: The only difference in the computation is that we have an additional MLP and a convex combination to generate the modulated queries.

Original: Since the role of modulated queries in our model is exactly the same as the fixed object queries in the original models, we use modulated queries and queries interchangeably to refer to the modulated queries.
Revised: Since the role of modulated queries in our model is exactly the same as the fixed object queries in the original models, we use modulated queries and queries interchangeably to refer to the modulated queries.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: To evaluate the effectiveness of our proposed methods [Table] we first conduct experiments on a series of tasks, including object detection, instance segmentation, panoptic segmentation, and video instance segmentation with different DETR-based models.
Revised: To evaluate the effectiveness of our proposed methods, we first conduct experiments on a series of tasks, including object detection, instance segmentation, panoptic segmentation, and video instance segmentation with different DETR-based models.

Original: Then we conduct several ablation studies to investigate the impact of different hyperparameters in our model.
Revised: Then we conduct several ablation studies to investigate the impact of different hyperparameters in our model.

Original: Detailed experiment setups and visualization examples are proviede in the supplementary materials
Revised: Detailed experiment setups and visualization examples are provided in the supplementary materials.

Original: . 4. E XPERIMENT S ETUP
Revised: 4. E XPERIMENT S ETUP

Original: Datasets.
Revised: Datasets.

Original: For the object detection task, we use MS COCO benchmark (Lin et al., 2014) for evaluation, which contains 118 , 287 images for training and 5 , 000 for validation.
Revised: For the object detection task, we use MS COCO benchmark (Lin et al., 2014) for evaluation, which contains 118 , 287 images for training and 5 , 000 for validation.

Original: For instance and panoptic segmentation, besides the MS COCO benchmark ( 80 “things” and 53 “stuff” categories), we also conduct experiments on the CityScapes (Cordts et al., 2016) benchmark ( 8 “things” and“stuff” categories) to validate the effectiveness of our proposed method.
Revised: For instance and panoptic segmentation, besides the MS COCO benchmark ( 80 “things” and 53 “stuff” categories), we also conduct experiments on the CityScapes (Cordts et al., 2016) benchmark ( 8 “things” and“stuff” categories) to validate the effectiveness of our proposed method.

Original: For the VIS task, YouTubeVIS-2019 (Yang et al., 2019) is used for evaluation.
Revised: For the VIS task, YouTubeVIS-2019 (Yang et al., 2019) is used for evaluation.

Original: For experiments on VIS, we pretrain our models on MS COCO and finetune them on the training set of YouTube-VIS-2019.
Revised: For experiments on VIS, we pretrain our models on MS COCO and finetune them on the training set of YouTube-VIS-2019.

Original: Evaluation metrics.
Revised: Evaluation metrics.

Original: For panoptic segmentation, the standard PQ (panoptic quality) metric (Kirillov et al., 2019) is used for evaluation.
Revised: For panoptic segmentation, the standard PQ (panoptic quality) metric (Kirillov et al., 2019) is used for evaluation.

Original: For instance segmentation (image or video) and object detection, we use the standard mAP (mean average precision) metric for evaluation.
Revised: For instance segmentation (image or video) and object detection, we use the standard mAP (mean average precision) metric for evaluation.

Original: For VIS, mAP and AR (average recall) on video instances are the evaluation metrics.
Revised: For VIS, mAP and AR (average recall) on video instances are the evaluation metrics.

Original: Implementation details.
Revised: Implementation details.

Original: The query ratio r used to generate the combination coefficients is set to 4 by default.
Revised: The query ratio r used to generate the combination coefficients is set to 4 by default.

Original: β is set to be 1 .
Revised: β is set to be 1 .

Original: θ is implemented as a two-layer MLP with ReLU as nonlinear activations.
Revised: θ is implemented as a two-layer MLP with ReLU as nonlinear activations.

Original: The output size of its first layer is 512, and that of the second layer is the length of
Revised: The output size of its first layer is 512, and that of the second layer is the length of

Original: W D in corresponding models.
Revised: W D in corresponding models.

Original: For detection models, we use 300 queries if not specified otherwise.
Revised: For detection models, we use 300 queries if not specified otherwise.

Original: For segmentation models, we use 50 queries for Mask2Former on image segmentation tasks,queries for Mask2Former on VIS, and 300 queries for SeqFormer on VIS.
Revised: For segmentation models, we use 50 queries for Mask2Former on image segmentation tasks,queries for Mask2Former on VIS, and 300 queries for SeqFormer on VIS.

Original: ResNet50 is used as the backbone for different models. 
Revised: ResNet50 is used as the backbone for different models. [Table] [Table] [Table]

Original: 4. M AIN R ESULTS
Revised: 4. M AIN R ESULTS

Original: Object detection.
Revised: Object detection.

Original: We evaluate our proposed methods with the DETR-based models DeformableDETR (Zhu et al., 2021), SMCA-DETR (Gao et al., 2021), Conditional-DETR (Tian et al., 2020),
Revised: We evaluate our proposed methods with the DETR-based models DeformableDETR (Zhu et al., 2021), SMCA-DETR (Gao et al., 2021), Conditional-DETR (Tian et al., 2020),

Original: DAB-DETR and DAB-Deformable-DETR (Liu et al., 2022) for object detection on the MS COCO benchmark.
Revised: DAB-DETR and DAB-Deformable-DETR (Liu et al., 2022) for object detection on the MS COCO benchmark.

Original: For a fair comparison, we run the original model integrated with and without our proposed dynamic queries using the same experimental settings, including the number of queries [Table]
Revised: For a fair comparison, we run the original model integrated with and without our proposed dynamic queries using the same experimental settings, including the number of queries and epochs.

Original: [Table] [Table] and epochs. The  equipped with our  are denoted as DQDeformable-DETR, DQ-SMCA-DETR, DQ-Conditional-DETR, DQ-DAB-DETR, and DQ-DABDeformable-DETR, respectively.
Revised: The models equipped with our dynamic query combinations are denoted as DQDeformable-DETR, DQ-SMCA-DETR, DQ-Conditional-DETR, DQ-DAB-DETR, and DQ-DABDeformable-DETR, respectively.

Original: The results are shown as in Table 2.
Revised: The results are shown as in Table 2.

Original: From Table 2, when integrated with our proposed method, mAP can be improved consistently by at least 0 .
Revised: From Table 2, when integrated with our proposed method, mAP can be improved consistently by at least 0 .

Original: 8% for all the models listed in the table.
Revised: 8% for all the models listed in the table.

Original: For DAB-Deformable-DETR, the performance can be improved byup to 1 . 
Revised: For DAB-Deformable-DETR, the mAP can be improved by 1 . 6% with ResNet backbone and 1 . 1% with Swin-B backbone. For Deformable-DETR, the mAP can be improved significantly by 2 . 3% with Swin-B backbone.

Revised: This proves the benefit of our method with different backbones.

Original: Note that models with dynamic queries only have negligible increased computation cost compared to the original models.
Revised: Note that models with dynamic queries only have negligible increased computation cost compared to the original models.

Original: Instance/panoptic segmentation.
Revised: Instance/panoptic segmentation.

Original: Mask2Former(Cheng et al., 2021a) is a recent state-of-the-art model that can be used for different segmentation tasks with a unified model architecture.
Revised: Mask2Former(Cheng et al., 2021a) is a recent state-of-the-art model that can be used for different segmentation tasks with a unified model architecture.

Original: We compare Mask2Former with/without our dynamic queries for image instance and panoptic segmentation tasks on the MS COCO (Lin et al., 2014) and CityScapes (Cordts et al., 2016) benchmarks.
Revised: We compare Mask2Former with/without our dynamic queries for image instance and panoptic segmentation tasks on the MS COCO (Lin et al., 2014) and CityScapes (Cordts et al., 2016) benchmarks.

Original: The model plugged with dynamic queries is named DQ-Mask2Former.
Revised: The model plugged with dynamic queries is named DQ-Mask2Former.

Original: The results are shown as in Table 3, 4 and 5.
Revised: The results are shown as in Table 3, 4 and 5.

Original: We use 50 queries for all the listed settings.
Revised: We use 50 queries for all the listed settings.

Original: For instance segmentation (Table 3 and 5), our model DQ-Mask2Former achieves consistent improvement across different metrics compared to the original Mask2Former.
Revised: For instance segmentation (Table 3 and 5), our model DQ-Mask2Former achieves consistent improvement across different metrics compared to the original Mask2Former.

Original: For example, the performance on mAP is improved by around 0 .
Revised: For example, the performance on mAP is improved by around 0 .

Original: 8% on both MS COCO and CityScapes.
Revised: 8% on both MS COCO and CityScapes.

Original: For panoptic segmentation, as shown in Table 4 and 5, DQMask2Former again significantly outperforms Mask2Former (Cheng et al., 2021a) across all the evaluation metrics on both MS COCO and CityScapes. 
Revised: For panoptic segmentation, as shown in Table 4 and 5, DQMask2Former again significantly outperforms Mask2Former (Cheng et al., 2021a) across all the evaluation metrics on both MS COCO and CityScapes. [Table] [Table]

Original: Video instance segmentation.
Revised: Video instance segmentation.

Original: Besides image tasks, we also evaluate our method on the video instance segmentation task.
Revised: Besides image tasks, we also evaluate our method on the video instance segmentation task.

Original: We evaluated our method on two state-of-the-art VIS methods
Revised: We evaluated our method on two state-of-the-art VIS methods

Original: Mask2Former (Cheng et al., 2021b) and SeqFormer (Wu et al., 2021).
Revised: Mask2Former (Cheng et al., 2021b) and SeqFormer (Wu et al., 2021).

Original: Results are shown in Table 6.
Revised: Results are shown in Table 6.

Original: It can be seen from Table 

Revised: It can be seen from Table

Original: that when integrated with our dynamic queries, mAP and AR of
Revised: that when integrated with our dynamic queries, mAP and AR of

Original: Mask2Former are improved by at around 1 . 0% .
Revised: Mask2Former are improved by at around 1 . 0% .

Original: The mAP of SeqFormer is significantly boosted by 1 . 5% .
Revised: The mAP of SeqFormer is significantly boosted by 1 . 5% .

Original: Note the additional computation cost are negligible for these two models.[Table] [Table]
Revised: Note the additional computation cost are negligible for these two models.

Original: 4. M ODEL A NALYSIS
Revised: 4. M ODEL A NALYSIS

Original: Analysis of number of queries.
Revised: Analysis of number of queries.

Original: We use Deformable-DETR and DAB-Deformable-DETR as baseline models to study the effects of number of queries on the performance of object detection.
Revised: We use Deformable-DETR and DAB-Deformable-DETR as baseline models to study the effects of number of queries on the performance of object detection.

Original: We compare the baseline models with DQ-Deformable-DETR and DQ-DAB-Deformable-DETR integrated with different numbers of queries as in Figure 1.
Revised: We compare the baseline models with DQ-Deformable-DETR and DQ-DAB-Deformable-DETR integrated with different numbers of queries as in Figure 1.

Original: Note that we include the additional components of our models into the FLOPs computation of the decoder.
Revised: Note that we include the additional components of our models into the FLOPs computation of the decoder.

Original: When integrated with our method, even by reducing the number of queries from 300 to 100 , the mAP of DQ-Deformable-DETR and
Revised: When integrated with our method, even by reducing the number of queries from 300 to 100 , the mAP of DQ-Deformable-DETR and

Original: DQ-DAB-Deformable-DETR are still better than the baseline models with 300 queries.
Revised: DQ-DAB-Deformable-DETR are still better than the baseline models with 300 queries.

Original: We are also able to reduce the computation cost of the decoders of Deformable-DETR and DAB-DeformableDETR by about 14% and 24% by using our method with 100 queries, respectively.
Revised: We are also able to reduce the computation cost of the decoders of Deformable-DETR and DAB-DeformableDETR by about 14% and 24% by using our method with 100 queries, respectively.

Original: However, we do not observe significant speedup using our method with fewer queries mainly because the main computation costs are from the backbones and the transformer encoders.
Revised: However, we do not observe significant speedup using our method with fewer queries mainly because the main computation costs are from the backbones and the transformer encoders.

Original: Analysis of number of training epochs.
Revised: Analysis of number of training epochs.

Original: In Figure 3 (a), we show the impact of the number of training epochs on a sample model DQ-Deformable-DETR together with the original DeformableDETR.
Revised: In Figure 3 (a), we show the impact of the number of training epochs on a sample model DQ-Deformable-DETR together with the original DeformableDETR.

Original: From the figure, the mAP of DQ-Deformable-DETR is always better than that of the original Deformable-DETR at different epochs on the MS COCO benchmark.
Revised: From the figure, the mAP of DQ-Deformable-DETR is always better than that of the original Deformable-DETR at different epochs on the MS COCO benchmark.

Original: At early epochs around 30, DQ-Deformable-DETR achieves an even more significant performance gain compared then
Revised: At early epochs around 30, DQ-Deformable-DETR achieves an even more significant performance gain compared then

Original: Deformable-DETR compared with later epochs.
Revised: Deformable-DETR compared with later epochs.

Original: Analysis of β .
Revised: Analysis of β .

Original: We analyze the impact of the scale of β on models equipped with our dynamic queries.
Revised: We analyze the impact of the scale of β on models equipped with our dynamic queries.

Original: We conduct experiments using Deformable-DETR with ResNet-50 as the backbone on the
Revised: We conduct experiments using Deformable-DETR with ResNet-50 as the backbone on the

Original: MS COCO benchmark with different values of β .
Revised: MS COCO benchmark with different values of β .

Original: Results are shown in Table 7.
Revised: Results are shown in Table 7.

Original: As shown in the table, when β is set to be 0 , where no loss is directly computed with the prediction from the basic queries, the performance drops by 2 . 4% compared to the original setting.
Revised: As shown in the table, when β is set to be 0 , where no loss is directly computed with the prediction from the basic queries, the performance drops by 2 . 4% compared to the original setting.

Original: In this case, the basic queries are not necessarily proper queries for the detection model, which will affect the quality of the modulated queries produced by them.
Revised: In this case, the basic queries are not necessarily proper queries for the detection model, which will affect the quality of the modulated queries produced by them.

Original: The performance can be improved by increasing the value of β to 0.5.
Revised: The performance can be improved by increasing the value of β to 0.5.

Original: Empirically we find β = 1 is a good choice to balance the scale of losses between the basic and modulated queries.
Revised: Empirically we find β = 1 is a good choice to balance the scale of losses between the basic and modulated queries.

Original: Analysis of query ratio.
Revised: Analysis of query ratio.

Original: We use DQ-Deformable-DETR (Zhu et al., 2021) to analyze the performance of our proposed methods with different query ratios 2, 4, and 8, as in Figure 3 (b). 
Revised: We use DQ-Deformable-DETR (Zhu et al., 2021) to analyze the performance of our proposed methods with different query ratios 2, 4, and 8, as in Figure 3 (b). From

Original: From the figure, using 4 as the query ratio achieves the best performance for DQ-Deformable-DETR with 300 queries.
Revised: [Figure] the figure, using 4 as the query ratio achieves the best performance for DQ-Deformable-DETR with 300 queries.

Original: However, other query ratio choices still generate better accuracies than the original
Revised: However, other query ratio choices still generate better accuracies than the original

Original: Deformable-DETR, which validates the effectiveness and robustness of our method.
Revised: Deformable-DETR, which validates the effectiveness and robustness of our method.

Original: Visualization of W D .
Revised: Visualization of W D .

Original: Since W D is conditioned on the high-level content of the image, we conjecture that images with similar scenes or object categories may have similar W D parameters.We
Revised: Since W D is conditioned on the high-level content of the image, we conjecture that images with similar scenes or object categories may have similar W D parameters.

Original: [Figure] choose 200 images from the validation set of MS COCO and compute their W D from DQ-DABDETR with 300 queries.
Revised: We choose 200 images from the validation set of MS COCO and compute their W D from DQ-DABDETR with 300 queries.

Original: The resulting W D are first flattened into vectors, and then projected onto a two-dimensional space using t-SNE(Van der Maaten & Hinton, 2008).
Revised: The resulting W D are first flattened into vectors, and then projected onto a two-dimensional space using t-SNE(Van der Maaten & Hinton, 2008).

Original: We visualize the projected
Revised: We visualize the projected

Original: W D parameters along with their corresponding input images as Figure 4.
Revised: W D parameters along with their corresponding input images as Figure 4.

Original: We can see that some object categories tend to be clustered.
Revised: We can see that some object categories tend to be clustered.

Original: For example, we can see a lot of transportation vehicles in the top right corner of the figure, and wild animals tend to be in the lower part of the figure, which indicates that the model uses some high-level semantics of the image to produce the combination coefficients.
Revised: For example, we can see a lot of transportation vehicles in the top right corner of the figure, and wild animals tend to be in the lower part of the figure, which indicates that the model uses some high-level semantics of the image to produce the combination coefficients.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we propose to use dynamic queries depending on the input image to enhance DETRbased detection and segmentation models.
Revised: In this paper, we propose to use dynamic queries depending on the input image to enhance DETRbased detection and segmentation models.

Original: We find that convex combinations of learned queries are naturally high-quality object queries for the corresponding models.
Revised: We find that convex combinations of learned queries are naturally high-quality object queries for the corresponding models.

Original: Based on this observation, we design a pipeline to learn dynamic convex combinations of the basic queries, adapting object queries according to the high level semantics of the input images.
Revised: Based on this observation, we design a pipeline to learn dynamic convex combinations of the basic queries, adapting object queries according to the high level semantics of the input images.

Original: This approach consistently improves the performance of a wide range of DETR-based models on object detection and segmentation tasks.
Revised: This approach consistently improves the performance of a wide range of DETR-based models on object detection and segmentation tasks.

Original: The gain of our model is agnostic to the different designs of the Transformer decoders and different types of object queries.
Revised: The gain of our model is agnostic to the different designs of the Transformer decoders and different types of object queries.

Original: We believe this approach opens the door to designing dynamic queries and creates a new perspective for Transformer-based models.
Revised: We believe this approach opens the door to designing dynamic queries and creates a new perspective for Transformer-based models.