Original: FP-DETR: D ETECTION T RANSFORMER A DVANCED BY F ULLY P RE - TRAINING
Revised: FP-DETR: D ETECTION T RANSFORMER A DVANCED BY F ULLY P RE - TRAINING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Large-scale pre-training has proven to be effective for visual representation learning on downstream tasks, especially for improving robustness and generalization.
Revised: Large-scale pre-training has proven to be effective for visual representation learning on downstream tasks, especially for improving robustness and generalization.

Original: However, the recently developed detection transformers only employ pre-training on its backbone while leaving the key component, i.e., a 12-layer transformer, being trained from scratch, which prevents the model from above beneﬁts.
Revised: However, the recently developed detection transformers only employ pre-training on its backbone while leaving the key component, i.e., a 12-layer transformer, being trained from scratch, which prevents the model from above beneﬁts.

Original: This separated training paradigm is mainly caused by the discrepancy between the upstream and downstream tasks.
Revised: This separated training paradigm is mainly caused by the discrepancy between the upstream and downstream tasks.

Original: To mitigate the issue, we propose FP-DETR, a new method that F ully P re-Trains an encoder-only transformer and smoothly ﬁnetunes it for object detection via a task adapter.
Revised: To mitigate the issue, we propose FP-DETR, a new method that F ully P re-Trains an encoder-only transformer and smoothly ﬁnetunes it for object detection via a task adapter.

Original: Inspired by the success of textual prompts in NLP, we treat query positional embeddings as visual prompts to help the model attend to the target area (prompting) and recognize the object.
Revised: Inspired by the success of textual prompts in NLP, we treat query positional embeddings as visual prompts to help the model attend to the target area (prompting) and recognize the object.

Original: To this end, we propose the task adapter which leverages self-attention to model the contextual relation between object query embedding.
Revised: To this end, we propose the task adapter which leverages self-attention to model the contextual relation between object query embedding.

Original: Experiments on the challenging
Revised: Experiments on the challenging

Original: COCO dataset demonstrate that our PT-DETR achieves competitive performance.
Revised: COCO dataset demonstrate that our FP-DETR achieves competitive performance.

Original: Moreover, it enjoys better robustness to common corruptions and generalization to small-size datasets than state-of-the-art detection transformers.
Revised: Moreover, it enjoys better robustness to common corruptions and generalization to small-size datasets than state-of-the-art detection transformers.

Original: The source code will be made publicly available.
Revised: Code will be made publicly available at https://github.com/encounter1997/FP-DETR .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Since the surge of deep learning-based object detector (Girshick et al., 2014), pre-training the backbone on large-scale datasets like ImageNet (Deng et al., 2009) and ﬁne-tuning the model on downstream tasks has become a well-established paradigm.
Revised: Since the surge of deep learning-based object detector (Girshick et al., 2014), pre-training the backbone on large-scale datasets like ImageNet (Deng et al., 2009) and ﬁne-tuning the model on downstream tasks has become a well-established paradigm.

Original: Pre-training signiﬁcantly improves robustness (Hendrycks et al., 2019) and may enhance the model performance, especially on small datasets.
Revised: Pre-training signiﬁcantly improves robustness (Hendrycks et al., 2019) and may enhance the model performance, especially on small datasets.

Original: Following this paradigm, modern object detectors, like Faster RCNN (Ren et al., 2016) and
Revised: Following this paradigm, modern object detectors, like Faster RCNN (Ren et al., 2016) and

Original: YOLO (Redmon et al., 2016), generally add a lightweight task-speciﬁc layers on top of ImageNet pre-trained backbones, so that the model can enjoy the aforementioned beneﬁts of pre-training.
Revised: YOLO (Redmon et al., 2016), generally add a lightweight task-speciﬁc layers on top of ImageNet pre-trained backbones, so that the model can enjoy the aforementioned beneﬁts of pre-training.

Original: However, the recently developed detection transformers 1 , e.g., DETR (Carion et al., 2020) and Deformable DETR (Zhu et al., 2020), only pre-trains its CNN backbone, while leaving the core module, a 12-layer transformer (including both encoder layers and decoder layers), trained from scratch in downstream tasks.
Revised: However, the recently developed detection transformers 1 , e.g., DETR (Carion et al., 2020) and Deformable DETR (Zhu et al., 2020), only pre-trains its CNN backbone, while leaving the core module, a 12-layer transformer (including both encoder layers and decoder layers), trained from scratch in downstream tasks.

Original: As a result, the model suffers from limited robustness against common corruptions and the reliance on a large amount of training data for ﬁne-tuning.
Revised: As a result, the model suffers from limited robustness against common corruptions and the reliance on a large amount of training data for ﬁne-tuning.

Original: Though UP-DETR
Revised: Though UP-DETR

Original: (Dai et al., 2020) attempts to mitigate this problem by unsupervised pre-training, it requires an off-theshelf CNN backbone that has already be pre-trained.
Revised: (Dai et al., 2020) attempts to mitigate this problem by unsupervised pre-training, it requires an off-theshelf CNN backbone that has already be pre-trained.

Original: Moreover, the two-stage separate pre-training methods may impair the model performance, since the two parts work together for object detection during ﬁne-tuning.
Revised: Moreover, the two-stage separate pre-training methods may impair the model performance, since the two parts work together for object detection during ﬁne-tuning.

Original: The key reason that hinders existing detection transformers from beneﬁting from large-scale pretraining can be attributed to the discrepancy between the upstream task and the downstream task.
Revised: The key reason that hinders existing detection transformers from beneﬁting from large-scale pretraining can be attributed to the discrepancy between the upstream task and the downstream task.

Original: Firstly, the object detection-oriented model structures are difﬁcult to adapt to the ImageNet classiﬁcation task.
Revised: Firstly, the object detection-oriented model structures are difﬁcult to adapt to the ImageNet classiﬁcation task.

Original: For example, the transformer decoder requires multiple query embeddings for detecting objects, while for ImageNet classiﬁcation, there is only a single query embedding (class token) used.
Revised: For example, the transformer decoder requires multiple query embeddings for detecting objects, while for ImageNet classiﬁcation, there is only a single query embedding (class token) used.

Original: If the decoder is included during pre-training, both the self-attention layers and the projections on query embeddings in cross-attention layers may easily overﬁt to the single class token, making it is hard to pre-train the decoder.
Revised: If the decoder is included during pre-training, both the self-attention layers and the projections on query embeddings in cross-attention layers may easily overﬁt to the single class token, making it is hard to pre-train the decoder.

Original: Secondly, the upstream ImageNet classiﬁcation task misses some crucial components for the downstream object detection task.
Revised: Secondly, the upstream ImageNet classiﬁcation task misses some crucial components for the downstream object detection task.

Original: For example, while the downstream object detection task requires both localization and classiﬁcation for the objects of interest, the upstream classiﬁcation task only focuses on the latter.
Revised: For example, while the downstream object detection task requires both localization and classiﬁcation for the objects of interest, the upstream classiﬁcation task only focuses on the latter.

Original: Moreover, the object relation modeling in object detection, which is important for removing heuristic post-processing like non-maximum suppression (NMS) (Carion et al., 2020), is absent in image classiﬁcation.
Revised: Moreover, the object relation modeling in object detection, which is important for removing heuristic post-processing like non-maximum suppression (NMS) (Carion et al., 2020), is absent in image classiﬁcation.

Original: To mitigate these problems, we propose FP-DETR, a novel method that reformulates the pre-training and ﬁne-tuning phases for detection transformers.
Revised: To mitigate these problems, we propose FP-DETR, a novel method that reformulates the pre-training and ﬁne-tuning phases for detection transformers.

Original: It fully pre-trains a detection transformer on ImageNet classiﬁcation task and smoothly ﬁne-tune it for object detection task though a task adaptor.
Revised: It fully pre-trains a detection transformer on ImageNet classiﬁcation task and smoothly ﬁne-tune it for object detection task though a task adaptor.

Original: Concretely, during pre-training, we introduce an encoder-only transformer structure by removing the decoder that can hardly be well pre-trained on the ImageNet classiﬁcation task.
Revised: Concretely, during pre-training, we introduce an encoder-only transformer structure by removing the decoder that can hardly be well pre-trained on the ImageNet classiﬁcation task.

Original: Moreover, since both the CNN backbone and the transformer encoder in detection transformers can be seen as feature extractor, we replace the complex CNN backbone with a simple multi-scale tokenizer and only uses the transformer encoder for feature extraction.
Revised: Moreover, since both the CNN backbone and the transformer encoder in detection transformers can be seen as feature extractor, we replace the complex CNN backbone with a simple multi-scale tokenizer and only uses the transformer encoder for feature extraction.

Original: The resultant architecture is an efﬁcient transformer encoder-only model.
Revised: The resultant architecture is an efﬁcient transformer encoder-only model.

Original: During ﬁne-tuning on the object detection task, we take inspiration from the success of textual prompt (Liu et al., 2021a) in NLP and treat the query positional embeddings as visual prompts to help the model attend to the target areas (prompting) and recognize the object.
Revised: During ﬁne-tuning on the object detection task, we take inspiration from the success of textual prompt (Liu et al., 2021a) in NLP and treat the query positional embeddings as visual prompts to help the model attend to the target areas (prompting) and recognize the object.

Original: Our key intuition is that if the model knows where to look at, it simply needs to identify the category of the object in hand, as did during pre-training.
Revised: Our key intuition is that if the model knows where to look at, it simply needs to identify the category of the object in hand, as did during pre-training.

Original: Based on this motivation, we devise a lightweight task adaptor to enhance the prompt ability of query positional embeddings and hence bridge the gap between upstream classiﬁcation task and downstream object detection task.
Revised: Based on this motivation, we devise a lightweight task adaptor to enhance the prompt ability of query positional embeddings and hence bridge the gap between upstream classiﬁcation task and downstream object detection task.

Original: It leverages self-attention to emphasize the relation between query embeddings, such that the inter-object relationships, which have been neglected during pre-training, can be captured during ﬁne-tuning.
Revised: It leverages self-attention to emphasize the relation between query embeddings, such that the inter-object relationships, which have been neglected during pre-training, can be captured during ﬁne-tuning.

Original: Experiments show that the proposed FP-DETR achieves competitive performance on the challenging COCO dataset (Lin et al., 2014), and a better trade-off between the number of parameters and detection accuracy.
Revised: Experiments show that the proposed FP-DETR achieves competitive performance on the challenging COCO dataset (Lin et al., 2014), and a better trade-off between the number of parameters and detection accuracy.

Original: Moreover, FP-DETR is more robust against common corruptions and can generalize well to small-size datasets like Cityscapes (Cordts et al., 2016), attributing to the effective fully pre-training.
Revised: Moreover, FP-DETR is more robust against common corruptions and can generalize well to small-size datasets like Cityscapes (Cordts et al., 2016), attributing to the effective fully pre-training.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Object Detection.
Revised: Object Detection.

Original: Object detection is one of the fundamental tasks in computer vision (CV).
Revised: Object detection is one of the fundamental tasks in computer vision (CV).

Original: Representative object detection methods can be roughly categorized as the one-stage object detector, e.g., YOLO (Redmon et al., 2016), SSD (Liu et al., 2016), and two-stage object detectors like
Revised: Representative object detection methods can be roughly categorized as the one-stage object detector, e.g., YOLO (Redmon et al., 2016), SSD (Liu et al., 2016), and two-stage object detectors like

Original: Faster-RCNN (Ren et al., 2016).
Revised: Faster-RCNN (Ren et al., 2016).

Original: While signiﬁcant progress has been made, these methods are not end-to-end and rely on heuristic components like non-max suppression (NMS) and rule-based label assignments.
Revised: While signiﬁcant progress has been made, these methods are not end-to-end and rely on heuristic components like non-max suppression (NMS) and rule-based label assignments.

Original: Recently, DETR (Carion et al., 2020) formulates the object detection task as a set prediction problem and proposes an end-to-end pipeline to exploit transformer and bipartite matching for detection.
Revised: Recently, DETR (Carion et al., 2020) formulates the object detection task as a set prediction problem and proposes an end-to-end pipeline to exploit transformer and bipartite matching for detection.

Original: The success of DETR brought the recent surge of detection transformers.
Revised: The success of DETR brought the recent surge of detection transformers.

Original: For example, Deformable DETR (Zhu et al., 2020) proposes deformable attention to signiﬁcantly accelerate the model convergence and reduce the computational cost, while allowing the model to beneﬁt from multi-scale features.
Revised: For example, Deformable DETR (Zhu et al., 2020) proposes deformable attention to signiﬁcantly accelerate the model convergence and reduce the computational cost, while allowing the model to beneﬁt from multi-scale features.

Original: Conditional DETR (Meng et al., 2021) narrows down the spatial range for localizing the object regions via learning the decoder embedding conditioned on a spatial query.
Revised: Conditional DETR (Meng et al., 2021) narrows down the spatial range for localizing the object regions via learning the decoder embedding conditioned on a spatial query.

Revised: REGO-DETR (Chen et al., 2021) mitigate the training difﬁculty through Region-of-Interest (RoI) based detection reﬁnement.

Original: While effective, the crucial component in these models, i.e., a 12-layer transformer, is trained from scratch, which limits the robustness and generalization ability of the detection models.
Revised: While effective, the crucial component in these models, i.e., a 12-layer transformer, is trained from scratch, which limits the robustness and generalization ability of the detection models.

Original: Pre-training for Object Detection.
Revised: Pre-training for Object Detection.

Original: The seminal work RCNN (Girshick et al., 2014) demonstrates the many beneﬁts of ImageNet pre-training for object detection, e.g., better detection performance and faster convergence.
Revised: The seminal work RCNN (Girshick et al., 2014) demonstrates the many beneﬁts of ImageNet pre-training for object detection, e.g., better detection performance and faster convergence.

Original: Since then, pre-training and ﬁne-tuning has become a well-established paradigm for object detection.
Revised: Since then, pre-training and ﬁne-tuning has become a well-established paradigm for object detection.

Original: Recently, He et.al challenge the common belief by showing that  training from scratch on datasets like COCO with a longer schedule can also produce a competitive detection performance (He et al., 2019).
Revised: Recently, He et.al challenge the common belief by showing that [Figure] training from scratch on datasets like COCO with a longer schedule can also produce a competitive detection performance (He et al., 2019).

Original: However, Hendrycks et.al show that though comparable in-domain performance can be achieved, training from scratch results in much worse out-of-domain generalization (Hendrycks et al., 2019).
Revised: However, Hendrycks et.al show that though comparable in-domain performance can be achieved, training from scratch results in much worse out-of-domain generalization (Hendrycks et al., 2019).

Original: Inspired by the success of pre-training for CNN-based object detectors, UP-DETR
Revised: Inspired by the success of pre-training for CNN-based object detectors, UP-DETR

Original: (Dai et al., 2020) proposes a novel random query patch detection pretraining task to improve DETR’s performance and convergence.
Revised: (Dai et al., 2020) proposes a novel random query patch detection pretraining task to improve DETR’s performance and convergence.

Original: Though progress has been made,

Original: [Figure] CLS [Figure] their method relies on a pre-trained off-the-shelf CNN backbone.
Revised: Though progress has been made, their method relies on a pre-trained off-the-shelf CNN backbone.

Original: Moreover, the improvement is limited, since the CNN backbone and the transformer are pre-trained separately while they need to work jointly in the downstream task.
Revised: Moreover, the improvement is limited, since the CNN backbone and the transformer are pre-trained separately while they need to work jointly in the downstream task.

Original: Probably the most similar work to ours is YOLOS (Fang et al., 2021), which also pre-trains and ﬁne-tunes an encoder-only transformer for object detection.
Revised: Probably the most similar work to ours is YOLOS (Fang et al., 2021), which also pre-trains and ﬁne-tunes an encoder-only transformer for object detection.

Original: However, it aims at demonstrating the feasibility of detecting objects using BERT (Devlin et al., 2018), with minimum inductive bias.
Revised: However, it aims at demonstrating the feasibility of detecting objects using BERT (Devlin et al., 2018), with minimum inductive bias.

Original: As a result, the discrepancy between the upstream and downstream tasks, which signiﬁcantly degenerates the model performance, is not considered in YOLOS.
Revised: As a result, the discrepancy between the upstream and downstream tasks, which signiﬁcantly degenerates the model performance, is not considered in YOLOS.

Original: By contrast, we aim at maximizing the beneﬁts of pre-training for the downstream object detection task.
Revised: By contrast, we aim at maximizing the beneﬁts of pre-training for the downstream object detection task.

Original: To this end, we treat query positional embeddings as visual prompts and devise a novel task adaptor to smoothly ﬁne-tune the pre-trained model on downstream tasks.
Revised: To this end, we treat query positional embeddings as visual prompts and devise a novel task adaptor to smoothly ﬁne-tune the pre-trained model on downstream tasks.

Original: Moreover, since object detection task often requires high-resolution images as input, inductive bias like sparsity is introduced to reduce the computational costs.
Revised: Moreover, since object detection task often requires high-resolution images as input, inductive bias like sparsity is introduced to reduce the computational costs.

Original: Prompt-based Learning in NLP.
Revised: Prompt-based Learning in NLP.

Original: Large-scale pre-trained models like GPT-3 (Brown et al., 2020) advance the ﬁeld of NLP.
Revised: Large-scale pre-trained models like GPT-3 (Brown et al., 2020) advance the ﬁeld of NLP.

Original: However, such large-scale models are not designed for ﬁne-tuning.
Revised: However, such large-scale models are not designed for ﬁne-tuning.

Original: To better utilize these models for downstream tasks, the “pre-train, prompt, and predict” paradigm emerges in the NLP community, and has drawn increasing attention (Liu et al., 2021a).
Revised: To better utilize these models for downstream tasks, the “pre-train, prompt, and predict” paradigm emerges in the NLP community, and has drawn increasing attention (Liu et al., 2021a).

Original: The new paradigm reformulates the downstream task to mimic the task during pre-training with the help of a textual prompt, to bridge the gap between upstream and downstream tasks.
Revised: The new paradigm reformulates the downstream task to mimic the task during pre-training with the help of a textual prompt, to bridge the gap between upstream and downstream tasks.

Original: Speciﬁcally, pioneer work introduces manually designed prompts to improve the ﬁne-tuning (Schick & Sch ¨ utze, 2020), or searches the textual prompt for downstream tasks (Gao et al., 2020; Jiang et al., 2020).
Revised: Speciﬁcally, pioneer work introduces manually designed prompts to improve the ﬁne-tuning (Schick & Sch¨utze, 2020), or searches the textual prompt for downstream tasks (Gao et al., 2020; Jiang et al., 2020).

Original: Though improvements have been made, these prompts correspond to natural language phrases in discrete space can be suboptimal, since the neural networks are inherently continuous.
Revised: Though improvements have been made, these prompts correspond to natural language phrases in discrete space can be suboptimal, since the neural networks are inherently continuous.

Original: To tackle this problem, P-tuning (Liu et al., 2021b) proposes to optimize contiguous prompt embedding to better close the gap between pre-training and ﬁne-tuning.
Revised: To tackle this problem, P-tuning (Liu et al., 2021b) proposes to optimize contiguous prompt embedding to better close the gap between pre-training and ﬁne-tuning.

Original: In this paper, instead of reformulating the downstream task, we propose a new perspective and treat the query positional embeddings in detection transformers as visual prompts.
Revised: In this paper, instead of reformulating the downstream task, we propose a new perspective and treat the query positional embeddings in detection transformers as visual prompts.

Original: To this end, a task adaptor is devised to facilitate ﬁne-tuning on the downstream task.
Revised: To this end, a task adaptor is devised to facilitate ﬁne-tuning on the downstream task.

Original:  M ETHOD
Revised:  M ETHOD

Original: In this section, we ﬁrst revisit the preliminaries on detection transformers, then introduce the pretraining and ﬁne-tuning phases of our method as shown in Figure 1.
Revised: In this section, we ﬁrst revisit the preliminaries on detection transformers, then introduce the pretraining and ﬁne-tuning phases of our method as shown in Figure 1.

Original: Through exploiting the largescale pre-training on ImageNet, our detection transformer can enjoy better detection performance, improved robustness, and enhanced generalization.
Revised: Through exploiting the largescale pre-training on ImageNet, our detection transformer can enjoy better detection performance, improved robustness, and enhanced generalization.

Original: 3. P RELIMINARIES ON D ETECTION T RANSFORMERS
Revised: 3. P RELIMINARIES ON D ETECTION T RANSFORMERS

Original: Detection transformers generally consist of the following parts: a backbone, a transformer encoder, a transformer decoder, and a task-speciﬁc feed-forward network (FFN).
Revised: Detection transformers generally consist of the following parts: a backbone, a transformer encoder, a transformer decoder, and a task-speciﬁc feed-forward network (FFN).

Original: The backbone F can be decomposed as multiple stages, i.e., F = f l ◦ · · · ◦ f 2 ◦ f 1 , where each stage f i takes the output from the previous stage as input and outputs the down-sampled feature map at stage i .
Revised: The backbone F can be decomposed as multiple stages, i.e., F = f l ◦ · · · ◦ f 2 ◦ f 1 , where each stage f i takes the output from the previous stage as input and outputs the down-sampled feature map at stage i .

Original: For an input image I ∈
Revised: For an input image I ∈

Original: R H
Revised: R H

Original: × W × 3 , the backbone extracts multi-scale features, which are ﬂattened and projected to 1-D sequence representation.
Revised: × W × 3 , the backbone extracts multi-scale features, which are ﬂattened and projected to 1-D sequence representation.

Original: Afterwards, the sequence representation is added with position and level embedding to obtained x ∈ R N × D , where N = (cid:80) Ll =1 HW/S 2 l is the sequence length, S l is the down-sampling rate at l -th feature level, and D is the dimension of the feature embedding.
Revised: Afterwards, the sequence representation is added with position and level embedding to obtained x ∈ R N × D , where N = (cid:80) Ll =1 HW/S 2 l is the sequence length, S l is the down-sampling rate at l -th feature level, and D is the dimension of the feature embedding.

Original: The transformer general consists of both encoder and decoder, and takes the sequence representation extracted from the image and the query embeddings as input for context modeling.
Revised: The transformer general consists of both encoder and decoder, and takes the sequence representation extracted from the image and the query embeddings as input for context modeling.

Original: The ﬁnal prediction is made by applying FFN to the object representations.
Revised: The ﬁnal prediction is made by applying FFN to the object representations.

Original: While Deformable DETR (Zhu et al., 2020) contains four different feature levels, and the downsampling rate ranges from 8, 16, 32, to 64, respectively, most other detection transformers, including 
Revised: While Deformable DETR (Zhu et al., 2020) contains four different feature levels with different down-sampling rates, most other detection transformers, including DETR (Carion et al., 2020),

Original: DETR (Carion et al., 2020), Conditional DETR (Meng et al., 2021), contains only one feature level, whose down-sampling rate is 32.
Revised: Conditional DETR (Meng et al., 2021), contains only a single feature level.

Original: 3. P RE - TRAINING
Revised: 3. P RE - TRAINING

Original: Encoder-only Transformer.
Revised: Encoder-only Transformer.

Original: As described in Section 1, for image classiﬁcation, only one class token is required for context aggregation.
Revised: As described in Section 1, for image classiﬁcation, only a single class token is required for context aggregation.

Original: The six decoder layers trained on only one class token is prone to overﬁtting.
Revised: The six decoder layers trained on only one class token is prone to overﬁtting.

Original: To tackle this problem, we remove the decoder during pre-training on the image classiﬁcation task, which leads to an encoder-only transformer structure.
Revised: To tackle this problem, we remove the decoder during pre-training on the image classiﬁcation task, which leads to an encoder-only transformer structure.

Original: We follow Deformable
Revised: We follow Deformable

Original: DETR (Zhu et al., 2020) to adopt multi-scale deformable attention with four different feature levels to build our model.
Revised: DETR (Zhu et al., 2020) to adopt multi-scale deformable attention with four different feature levels to build our model.

Original: It largely alleviates the slow convergence of DETR (Carion et al., 2020), with the help of inductive bias like sparsity.
Revised: It largely alleviates the slow convergence of DETR (Carion et al., 2020), with the help of inductive bias like sparsity.

Original: To perform image classiﬁcation, an additional class token added with corresponding positional embedding x cls ∈ R D is concatenated with the sequence feature, for aggregating global context from the image tokens.
Revised: To perform image classiﬁcation, an additional class token added with corresponding positional embedding x cls ∈ R D is concatenated with the sequence feature, for aggregating global context from the image tokens.

Original: Thus the input to the encoder-only transformer z 0 can be written as,
Revised: Thus the input to the encoder-only transformer z 0 can be written as,

Original: [Equation]
Revised: [Equation]

Original: Afterward, the input sequence feature is processed by T different transformer encoder layers, each consists of a multi-scale deformable self-attention (MSDSA) layer and an MLP layer.
Revised: Afterward, the input sequence feature is processed by T different transformer encoder layers, each consists of a multi-scale deformable self-attention (MSDSA) layer and an MLP layer.

Original: Residual connections and layer norm (LN) are applied after each sub-layer, i.e.,
Revised: Residual connections and layer norm (LN) are applied after each sub-layer, i.e.,

Original: [Equation]
Revised: [Equation]

Original: Final predictions are made on class token at the T -th layer, i.e., z 0 T .
Revised: Final predictions are made on class token at the T -th layer, i.e., z 0 T .

Original: During implementation, the class token is treated as an additional feature level, with only one resolution.
Revised: During implementation, the class token is treated as an additional feature level, with only one resolution.

Original: Since multi-scale deformable attention assumes different levels of feature maps are spatially aligned, which does not hold for the class token in x cls , attention mask is utilized to remove the information ﬂow from x cls .
Revised: Since multi-scale deformable attention assumes different levels of feature maps are spatially aligned, which does not hold for the class token in x cls , attention mask is utilized to remove the information ﬂow from x cls .

Original: Lightweight Multi-scale Tokenizer.
Revised: Lightweight Multi-scale Tokenizer.

Original: The transformer encoder’s ability to serve as a general feature extractor has been proved by recent advances in vision transformers (Dosovitskiy et al., 2020).
Revised: The transformer encoder’s ability to serve as a general feature extractor has been proved by recent advances in vision transformers (Dosovitskiy et al., 2020).

Original: While existing detection transformers contain both sophisticated backbone and transformer encoder for feature extraction, we argue that such design brings unnecessary complexity to the detection model.
Revised: While existing detection transformers contain both sophisticated backbone and transformer encoder for feature extraction, we argue that such design brings unnecessary complexity to the detection model.

Original: Instead, we replace the complex backbone with simple convolutional layers that work as a multi-scale tokenizer.
Revised: Instead, we replace the complex backbone with simple convolutional layers that work as a multi-scale tokenizer.

Original: Speciﬁcally, the feature extractor at each stage f
Revised: Speciﬁcally, the feature extractor at each stage f

Original: i is a simple single-layer convolutional layer that is designed only for down-sampling the feature map to expected resolution.
Revised: i is a simple single-layer convolutional layer that is designed only for down-sampling the feature map to expected resolution.

Original: Such design signiﬁcantly simpliﬁes the model and makes it easy to pre-train the detection model on ImageNet classiﬁcation task.
Revised: Such design signiﬁcantly simpliﬁes the model and makes it easy to pre-train the detection model on ImageNet classiﬁcation task.

Original: The detailed structure of our multi-scale tokenizer is shown in
Revised: The detailed structure of our multi-scale tokenizer is shown in

Original: Appendix A.1.
Revised: Appendix A.1.

Original: Discussion.
Revised: Discussion.

Original: Most of the existing vision transformers adopt an encoder-only structure (Dosovitskiy et al., 2020; Liu et al., 2021c).
Revised: Most of the existing vision transformers adopt an encoder-only structure (Dosovitskiy et al., 2020; Liu et al., 2021c; Xu et al., 2021).

Original: By reformulating the detection transformer to an encoder-only structure, we solve the problem that the decoder is difﬁcult to pre-train and simplify the model structure.
Revised: By reformulating the detection transformer to an encoder-only structure, we solve the problem that the decoder is difﬁcult to pre-train and simplify the model structure.

Original: Moreover, it allows us to take advantage of the existing experience in training vision transformers.In
Revised: Moreover, it allows us to take advantage of the existing experience in training vision transformers.

Original:  Appendix A.2, we explore pre-training on ImageNet using the encoder-decoder structure, however, the performance is signiﬁcantly worse than that of the encoder-only structure. 
Revised: In Appendix A.2, we explore pre-training on ImageNet using the encoderdecoder structure, however, the performance is much worse than that of the encoder-only structure. [Figure] love this [Figure]

Revised: [mask]

Revised: [Figure]

Original: 3. F INE - TUNING
Revised: F INE - TUNING

Original: During ﬁne-tuning, the single class token is replaced by N q query content embeddings (Carion et al., 2020) x obj ∈ R N q × D for object detection.
Revised: During ﬁne-tuning, the single class token is replaced by N q query content embeddings (Carion et al., 2020) x obj ∈ R N q × D for object detection.

Original: The content embeddings are also added with query[Figure] positional embeddings (Carion et al., 2020) p ∈ R N q × D before feed into the transformer encoder.
Revised: The content embeddings are also added with query positional embeddings (Carion et al., 2020) p ∈ R N q × D before feed into the transformer encoder.

Original: Thus the input to the transformer encoder is:
Revised: Thus the input to the transformer encoder is:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The class token for image classiﬁcation and the query content embeddings for object detection are both designed for context aggregation.
Revised: The class token for image classiﬁcation and the query content embeddings for object detection are both designed for context aggregation.

Original: While the former focuses on global context aggregation, the latter focus on aggregating the local context for one speciﬁc object instance.
Revised: While the former focuses on global context aggregation, the latter focus on aggregating the local context for one speciﬁc object instance.

Original: Query Positional Embeddings as Visual Prompts.
Revised: Query Positional Embeddings as Visual Prompts.

Original: The textual prompt in NLP is shown in Figure 2.
Revised: The textual prompt in NLP is shown in Figure 2.

Original: It reformulates the downstream task to mimic the pre-training one, so that the pre-trained model can better handle the downstream task.
Revised: It reformulates the downstream task to mimic the pre-training one, so that the pre-trained model can better handle the downstream task.

Original: However, ﬁnding the optimal textual prompt is not easy, recent work such as P-tuning (Liu et al., 2021b) searches prompts in a contiguous space to bridge the gap between the upstream and downstream tasks.
Revised: However, ﬁnding the optimal textual prompt is not easy, recent work such as P-tuning (Liu et al., 2021b) searches prompts in a contiguous space to bridge the gap between the upstream and downstream tasks.

Original: Object detection can be decomposed into two subtasks, i.e., localization and classiﬁcation.
Revised: Object detection can be decomposed into two subtasks, i.e., localization and classiﬁcation.

Original: Our intuition is that if the classiﬁer pre-trained on ImageNet knows where to look at, it can easily recognize the object within the speciﬁed region, as did during ImageNet classiﬁcation pre-training.
Revised: Our intuition is that if the classiﬁer pre-trained on ImageNet knows where to look at, it can easily recognize the object within the speciﬁed region, as did during ImageNet classiﬁcation pre-training.

Original: In our detection transformer, the query position embeddings are mapped to the reference points, which guide different query content embeddings to sample the corresponding context from the image content.
Revised: In our detection transformer, the query position embeddings are mapped to the reference points, which guide different query content embeddings to sample the corresponding context from the image content.

Original: In other words, the query position embedding works as a visual cue to point out the image region that the model should focus on, as shown in ﬁgure 2 (b).
Revised: In other words, the query position embedding works as a visual cue to point out the image region that the model should focus on, as shown in ﬁgure 2 (b).

Original: From this perspective, the query position embedding serves as a visual prompt, which is analogous to the textual prompt in NLP.
Revised: From this perspective, the query position embedding serves as a visual prompt, which is analogous to the textual prompt in NLP.

Original: Further, the process of training the model to localize objects corresponds to the process of searching textual prompts in contiguous space.
Revised: Further, the process of training the model to localize objects corresponds to the process of searching textual prompts in contiguous space.

Original: And the process of ﬁnal classiﬁcation corresponds to the process of ﬁlling in the blanks and answers mapping in NLP (Liu et al., 2021a).
Revised: And the process of ﬁnal classiﬁcation corresponds to the process of ﬁlling in the blanks and answers mapping in NLP (Liu et al., 2021a).

Original: Improving Fine-tuning with Task Adaptor.
Revised: Improving Fine-tuning with Task Adaptor.

Original: Based on the above motivation, we propose a task adaptor to improve the query positional embeddings’ ability to prompt.
Revised: Based on the above motivation, we propose a task adaptor to improve the query positional embeddings’ ability to prompt.

Original: Speciﬁcally, the inter-object relationship can help the model better identify object regions.
Revised: Speciﬁcally, the inter-object relationship can help the model better identify object regions.

Original: However, this crucial part is missing during ImageNet classiﬁcation pre-training, since only one class token is used.
Revised: However, this crucial part is missing during ImageNet classiﬁcation pre-training, since only one class token is used.

Original: To this end, we devise the task adapter to improve the modeling of the inter-object relationships.
Revised: To this end, we devise the task adapter to improve the modeling of the inter-object relationships.

Original: It pre-processes the visual prompts before they are sent to each pre-trained encoder layer, i.e.,
Revised: It pre-processes the visual prompts before they are sent to each pre-trained encoder layer, i.e.,

Original: [Equation]
Revised: [Equation]

Original: By default, the task adaptor is instantiated as a self-attention layer.
Revised: By default, the task adaptor is instantiated as a self-attention layer.

Original: Afterward, the sequence is processed by the transformer encoder layers, as did in Equation 2 and 3.
Revised: Afterward, the sequence is processed by the transformer encoder layers, as did in Equation 2 and 3.

Original: The object pattern embedding in the last encoder layer, i.e., z 0: N q t is used for ﬁnal prediction.
Revised: The object pattern embedding in the last encoder layer, i.e., z 0: N q t is used for ﬁnal prediction.

Original: Our task adaptor helps the visual  prompts for individual objects to make associations with other instances, making them more suitable for the downstream object detectiontask.
Revised: Our task adaptor helps the visual [Table] prompts for individual objects to make associations with other instances, making them more suitable for the downstream object detection

Original: Discussion.
Revised: Discussion.

Revised: Recently, Shen et al.

Revised: ﬁnd the connection between named entity recognition (NER) in NLP and object detection in CV, and propose a two-stage method for NER.

Original:  We propose a new perspective to view query positional embeddings as visual prompts.
Revised: Similarly, We propose a new perspective to view query positional embeddings as visual prompts.

Original: However, the existing pre-trained visual models are not as powerful as pre-trained language models (Brown et al., 2020).
Revised: However, the existing pre-trained visual models are not as powerful as pre-trained language models (Brown et al., 2020).

Original: As a result, we are unable to ﬁx the pre-trained models and only tune[Table] the visual prompts, to perform few-shot or even zero-shot learning on downstream tasks, as did in NLP.
Revised: As a result, we are unable to ﬁx the pre-trained models and only tune the visual prompts, to perform few-shot or even zero-shot learning on downstream tasks, as did in NLP.

Original: But still, this new perspective helps us to better understand the detection transformers and the discrepancy between the pre-training and downstream tasks, which further helps us design the task adaptor to smoothly ﬁne-tuning the pre-trained model on downstream tasks.
Revised: But still, this new perspective helps us to better understand the detection transformers and the discrepancy between the pre-training and downstream tasks, which further helps us design the task adaptor to smoothly ﬁne-tune the pre-trained model on downstream tasks.

Original:  E XPERIMENTS 4. I MPLEMENTATION D ETAILS
Revised:  E XPERIMENTS 4. I MPLEMENTATION D ETAILS

Original: Datasets.
Revised: Datasets.

Original: Following the common practice, our detector is pre-trained on ImageNet (Deng et al., 2009) and ﬁne-tuned on COCO 2017 (Lin et al., 2014) train set.
Revised: Following the common practice, our detector is pre-trained on ImageNet (Deng et al., 2009) and ﬁne-tuned on COCO 2017 (Lin et al., 2014) train set.

Original: Evaluation results on the val set of COCO 2017 are reported.
Revised: Evaluation results on the val set of COCO 2017 are reported.

Original: To evaluate the models’ robustness against common corruptions, we report the model performances on COCO-C (Michaelis et al., 2019), which is obtained by applying corruption synthesis algorithm to COCO.
Revised: To evaluate the models’ robustness against common corruptions, we report the model performances on COCO-C (Michaelis et al., 2019), which is obtained by applying corruption synthesis algorithm to COCO.

Original: Besides, we evaluated the model’s generalization ability by ﬁne-tuning on the small-size dataset, i.e., Cityscapes dataset (Cordts et al., 2016).
Revised: Besides, we evaluated the model’s generalization ability by ﬁne-tuning on the small-size dataset, i.e., Cityscapes dataset (Cordts et al., 2016).

Original: Model Variants
Revised: Model Variants

Original: The lite version of our model follows the hyper-parameters of the Deformable
Revised: The lite version of our model follows the hyper-parameters of the Deformable

Original: Transformer in Deformable DETR (Zhu et al., 2020).
Revised: Transformer in Deformable DETR (Zhu et al., 2020).

Original: Speciﬁcally, the encoder-only transformer contains 12 self-attention layers, each layer with 8 heads, the dimension of each head is 32, and the dimension of the MLP layer in FFN is 1024.
Revised: Speciﬁcally, the encoder-only transformer contains 12 self-attention layers, each layer with 8 heads, the dimension of each head is 32, and the dimension of the MLP layer in FFN is 1024.

Original: Furthermore, we introduce two other model variants by changing the number of heads, or dimensions per head, while keeping the model depth unchanged.
Revised: Furthermore, we introduce two other model variants by changing the number of heads, or dimensions per head, while keeping the model depth unchanged.

Original: Table 1 summarizes the different variants of our model.
Revised: Table 1 summarizes the different variants of our model.

Original: Training & Fine-tuning.
Revised: Training & Fine-tuning.

Original: Our FP-DETR is pre-trained on ImageNet-1k (Deng et al., 2009) for 300 epochs with AdamW (Loshchilov & Hutter, 2018) optimizer and cosine learning rate scheduler.
Revised: By default, our FP-DETR is pre-trained on ImageNet-1k (Deng et al., 2009) for 300 epochs with AdamW (Loshchilov & Hutter, 2018) optimizer and cosine learning rate scheduler.

Original: Training strategies in DeiT (Touvron et al., 2021) are adopted, and the image size is set as 224 × 224 .
Revised: Training strategies in DeiT (Touvron et al., 2021a) are adopted, and the image size is set as 224 × 224 .

Original: We use a batch size of 1,024 for training, and the initial learning rate is set as 5 × 10 − 4 .
Revised: We use a batch size of 1,024 for training, and the initial learning rate is set as 5 × 10 − 4 .

Original: After pretraining, models are ﬁne-tuned for 50 epochs with AdamW optimizer on the downstream tasks.
Revised: After pre-training, models are ﬁne-tuned for 50 epochs with AdamW optimizer on the downstream tasks.

Original: The learning rate is initialized as 1 × 10 − 4 and decreased by a factor of 0.1 at the 40th epoch.
Revised: The learning rate is initialized as 1 × 10 − 4 and decreased by a factor of 0.1 at the 40th epoch.

Original: We follow the implementation of Deformable DETR.
Revised: We follow the implementation of Deformable DETR to apply deep supervision (Lee et al., 2015) on the last six encoder layers.

Revised: The class token and the query content embedding are concatenated with the image tokens at an intermediate (the 7th) encoder layer, following CaiT (Touvron et al., 2021b).

Original: Speciﬁcally, both the number of sampling points and the feature levels in multi-scale deformable attentionare set as 4, and the number of object query embeddingsis set as 300.
Revised: Besides, we set both the number of sampling points and the feature levels in multi-scale deformable attention as 4, and the number of object query embeddings as 300.

Original: Models are ﬁne-tuned with a batch size of 32.
Revised: Models are ﬁne-tuned with a batch size of 32.

Original: All experiments are implemented on the NVIDIA A100 GPU.
Revised: All experiments are implemented on the NVIDIA A100 GPU.

Original: 4. C OMPARISON WITH S TATE - OF - THE - ARTS
Revised: 4. C OMPARISON WITH S TATE - OF - THE - ARTS

Original: As can be seen in Table 2, FP-DETR achieves competitive performance on COCO 2017 val set.
Revised: As can be seen in Table 2, FP-DETR achieves competitive performance on COCO 2017 val set.

Original: Speciﬁcally, FP-DETR-Base is comparable with state-of-the-art Deformable DETR, while its small variant with 24M parameters can match the performance of UP-DETR and outperforms Conditional
Revised: Speciﬁcally, FP-DETR-Base is comparable with state-of-the-art Deformable DETR, while its small variant with 24M parameters can match the performance of UP-DETR and outperforms Conditional

Original: DETR and DETR with about 40M parameters.
Revised: DETR and DETR with about 40M parameters.

Original: It performs better on detecting small objects and has better localization ability.
Revised: It performs better on detecting small objects and has better localization ability.

Revised: [Table]

Original:  Moreover, FP-DETR shows a better trade-off between model parameters and detection accuracy compared to Deformable DETR.
Revised: FP-DETR [Table] [Table] Moreover, FP-DETR shows a better trade-off between model parameters and detection accuracy compared to Deformable DETR.

Original: For example, FP-DETR-Small outperforms Deformable DETR with ResNet-18 (He et al., 2016) backbone, which has a similar amount of parameters.
Revised: For example, FP-DETR-Small outperforms Deformable DETR with ResNet-18 (He et al., 2016) backbone, which has a similar amount of parameters.

Original: It also slightly outperforms Deformable DETR with ResNet-34 backbone, which has about 10M more parameters.
Revised: It also slightly outperforms Deformable DETR with ResNet-34 backbone, which has about 10M more parameters.

Original: The smallest version of our model, FP-DETR-lite, achieves 37.2 mAP on COCOwith only 11M parameters.
Revised: The smallest version of our model, FP-DETR-lite, achieves 37.2 mAP on COCOwith only 11M parameters.

Original: Besides, FP-DETR signiﬁcantly outperforms YOLOS-Base which also adopts a transformer encoder-only structure, in terms of both model efﬁciency and detection accuracy.This is mainly [Table] FP
Revised: Besides, FP-DETR signiﬁcantly outperforms YOLOS-Base which also adopts a transformer encoder-only structure, in terms of both model efﬁciency and detection accuracy.

Original: [Table] DETR [Table] due to the inductive bias introduced in our model as Deformable DETR and the prompt-inspired task adaptor for more effective ﬁne-tuning, which better bridges the gap between pre-training and ﬁne-tuning.
Revised: This is mainly due to the inductive bias introduced in our model as Deformable DETR and the prompt-inspired task adaptor for more effective ﬁne-tuning, which better bridges the gap between pre-training and ﬁne-tuning.

Original: The last row shows the result of our FP-DETR-Base ﬁne-tuned from ImageNet-21 pre-training, denoted as FP-DETR-Base † .
Revised: The last row shows the result of our FP-DETR-Base ﬁne-tuned from ImageNet-21 pre-training, denoted as FP-DETR-Base † .

Original: As can be seen, FP-DETR can beneﬁt from pre-training at a larger scale, and FP-DETR-Base † with 36M parameters matches the performance of Deformable DETR with 40M parameters.
Revised: As can be seen, FP-DETR can beneﬁt from pre-training at a larger scale, and FP-DETR-Base † with 36M parameters matches the performance of Deformable DETR with 40M parameters.

Original: In Appendix A.3, we also show the result of Deformable DETR with ResNetbackbone pre-trained on ImageNet-21k.
Revised: In Appendix A.3, we also show the result of Deformable DETR with ResNetbackbone pre-trained on ImageNet-21k.

Original: Interestingly, the result is slightly worse than Deformable
Revised: Interestingly, the result is slightly worse than Deformable

Original: DETR with ImageNet-1k pre-trained backbone.
Revised: DETR with ImageNet-1k pre-trained backbone.

Original: We conjuncture that only pre-training part of the model makes it hard for the randomly initialized transformer adapts to the pre-trained CNN backbone, especially when the backbone has already been well-trained.
Revised: We conjuncture that only pre-training part of the model makes it hard for the randomly initialized transformer adapts to the pre-trained CNN backbone, especially when the backbone has already been well-trained.

Original: [Table]
Revised: [Table]

Original: 11M

Original: 4. A BLATIONS ON THE T ASK A DAPTOR
Revised: 4. A BLATIONS ON THE T ASK A DAPTOR

Original: To gain a better understanding of the effectiveness of the proposed task adaptor, we perform ablation studies on the lite version of FP-DETR, as shown in Table 3.
Revised: To gain a better understanding of the effectiveness of the proposed task adaptor, we perform ablation studies on the lite version of FP-DETR, as shown in Table 3.

Original: We have the following observations.
Revised: We have the following observations.

Original: First, removing the task adaptor results in signiﬁcant drops in model performance.
Revised: First, removing the task adaptor results in signiﬁcant drops in model performance.

Original: Without the task adaptor, the model can not well capture the inter-object relationships, which is crucial for removing duplicates and improving object recognition.
Revised: Without the task adaptor, the model can not well capture the inter-object relationships, which is crucial for removing duplicates and improving object recognition.

Original: Second, the task adaptor instantiated with a bidirectional-LSTM layer also helps to adapt the pre-trained model to the downstream task, but it works slightly worse than a self-attention layer.
Revised: Second, the task adaptor instantiated with a bidirectional-LSTM layer also helps to adapt the pre-trained model to the downstream task, but it works slightly worse than a self-attention layer.

Original: We conjuncture that the bidirectional-LSTM is slightly worse on modeling long-range dependency compared to the self-attention layer, especially when the number of query embeddings reaches 300.
Revised: We conjuncture that the bidirectional-LSTM is slightly worse on modeling long-range dependency compared to the self-attention layer, especially when the number of query embeddings reaches 300.

Original: By default, we adopt the self-attention layer for the task adaptor.
Revised: By default, we adopt the self-attention layer for the task adaptor.

Original: In this way, our FP-DETR is a pure transformer encoder structure.
Revised: In this way, our FP-DETR is a pure transformer encoder structure.

Original: Third, interestingly, a single task adaptor shared by visual prompts from different transformer encoder layers performs slightly better than specialized task adaptors for each layer.
Revised: Third, interestingly, a single task adaptor shared by visual prompts from different transformer encoder layers performs slightly better than specialized task adaptors for each layer.

Original: Since the task adaptor is trained from scratch on the downstream task, we conjuncture that the shared task adaptor sees diverse data from different levels compared to the non-shared ones, and is thus more sufﬁciently trained.
Revised: Since the task adaptor is trained from scratch on the downstream task, we conjuncture that the shared task adaptor sees diverse data from different levels compared to the non-shared ones [Table] and is thus more sufﬁciently trained.

Original: These experiments validate the effectiveness of our task adaptor on modeling object relationships, which is crucial for object detection.
Revised: These experiments validate the effectiveness of our task adaptor on modeling object relationships, which is crucial for object detection.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 36M

Original: 4. R OBUSTNESS TO C OMMON C ORRUPTIONS
Revised: 4. R OBUSTNESS TO C OMMON C ORRUPTIONS

Original: Model robustness  is critical to trustworthy AI applications like autonomous driving.
Revised: Model robustness (He & Tao, 2020) is critical to trustworthy AI applications like autonomous driving.

Original: To this end, we evaluate the object detectors’ robustness against common corruptions on COCO-C (Michaelis et al., 2019).
Revised: To this end, we evaluate the object detectors’ robustness against common corruptions on COCOC (Michaelis et al., 2019).

Original: As shown in Table 4, all detectors suffer signiﬁcant performance drops under this rigorous condition.
Revised: As shown in Table 4, all detectors suffer signiﬁcant performance drops under this rigorous condition.

Original: However, FP-DETR suffers the least performance drop compared to existing detection transformers.
Revised: However, FP-DETR suffers the least performance drop compared to existing detection transformers.

Original: Notably, FP-DETR-Base performs best on 14 out of 15 types of corruptions, though it has comparable performance to Deformable DETR on the clean COCO 2017 val set.
Revised: Notably, FP-DETR-Base performs best on 14 out of 15 types of corruptions, though it has comparable performance to Deformable DETR on the clean COCOval set.

Original: This is because thelarge-scale pre-training on ImageNet helps FP-DETR learn more generalizable representation.
Revised: This is because the pre-training on ImageNet helps FP-DETR learn more generalizable representation.

Original: This is coherent with the observation in Hendrycks et al.
Revised: This is coherent with the observation in Hendrycks et al.

Original: The ﬁrst three rows in Figure 3 provide some qualitative results of our FP-DETR-Base on the COCOC dataset.
Revised: The ﬁrst two rows in Figure 3 provide some qualitative results of our FP-DETR-Base on the COCOC dataset.

Original: As can be seen, the images are signiﬁcantly degenerated due to the existence of various corruptions, like shot noise, motion blur, fog, etc.
Revised: As can be seen, the images are signiﬁcantly degenerated due to the existence of various corruptions, like elastic transform, zoom blur, fog, etc.

Original: However, FP-DETR-Base still manages to produce plausible results under low visibility and large distortion, which manifests its robustness.
Revised: However, FP-DETR-Base still manages to produce plausible results under low visibility and large distortion, which manifests its robustness.

Original: 4. G ENERALIZATION TO S MALL - SIZE D ATASET
Revised: 4. G ENERALIZATION TO S MALL - SIZE D ATASET

Original: In real-world applications, collecting a large amount of data is often infeasible.
Revised: In real-world applications, collecting a large amount of data is often infeasible.

Original: As a result, the models are required to perform well by training on only limited data.
Revised: As a result, the models are required to perform well by training on only limited data.

Original: Table 5 shows the result of models ﬁne-tuned on the Cityscapes dataset with only 2,975 training images.
Revised: Table 5 shows the result of models ﬁne-tuned on the Cityscapes dataset with only 2,975 training images.

Original: Though Conditional DETR can achieve competitive performance after training on COCO 2017 for 50 epochs, it fails to converge on the much smaller Cityscapes dataset with the same training schedule.
Revised: All models are trained with a batch size of 8 to guarantee enough training iterations.

Original: Thus, we instead train it for 150 epochs.
Revised: The results are averaged over ﬁve repeated runs with different random seeds.

Original: As can be seen, most detection transformers, including DETR, UP-DETR, Condi- tional DETR, and YOLOS-B, perform poorly under this condition.
Revised: As can be seen, most detection transformers, including DETR, UP-DETR, Conditional DETR, and YOLOS-B, perform poorly under this condition.

Original: This is partly due to the lack of inductive bias in those models.
Revised: This is partly due to the lack of inductive bias in those models.

Original: As a result, the model requires a large amount of data to learn the sparsity that is beneﬁcial for object detection on images.
Revised: As a result, the model requires a large amount of data to learn the sparsity that is beneﬁcial for object detection on images.

Original: Deformable DETR performs better, thanks to the sparsity inductive bias introduced in the deformable transformer.
Revised: Deformable DETR performs better, thanks to the sparsity inductive bias introduced in the deformable transformer.

Original: However, it still performs worse than our FP-DETR, since the transformer in Deformable DETR is trained from scratch.
Revised: However, it still performs worse than our FP-DETR, since the transformer in Deformable DETR is trained from scratch.

Original: Speciﬁcally, both FP-DETR-Base and FP-DETR-Small outperform Deformable DETR with more parameters.
Revised: Speciﬁcally, both FP-DETR-Base and FP-DETR-Small outperform Deformable DETR with more parameters.

Original: Notably, our FP-DETR-Lite with only 11M parameters matches the performance of 40M Deformable DETR, attributing to the end-to-end fully pre-training of the transformer.
Revised: Notably, our FP-DETR-Lite with only 11M parameters matches the performance of 40M Deformable DETR, attributing to the end-to-end fully pre-training of the transformer.

Original: The last row in Figure 3 shows some qualitative results of our FP-DETR-Base on the Cityscapes dataset.
Revised: The last row in Figure 3 shows some qualitative results of our FP-DETR-Base on the Cityscapes dataset.

Original: As can be seen, FP-DETR quickly adapts to the small-size dataset, and produces accurate object detection results, even for the small cars in the distant area.
Revised: As can be seen, FP-DETR quickly adapts to the small-size dataset, and produces accurate object detection results, even for the small cars in the distant area.

Original: These results demonstrate the generalization ability of our method.
Revised: These results demonstrate the generalization ability of our method.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we propose a novel detection transformer named FP-DETR that can take advantage of pre-training onlarge-scale upstream datasets to enable better robustness and generalization.
Revised: In this paper, we propose a novel detection transformer named FP-DETR that can take advantage of pre-training on upstream datasets to enable better robustness and generalization.

Original: FPDETR contains a simple encoder-only structure for fully pre-training and can be smoothly ﬁne-tuned for object detection via a task adapter.
Revised: FP-DETR contains a simple encoder-only structure for fully pre-training and can be smoothly ﬁne-tuned for object detection via a task adapter.

Original: It leverages query positional embeddings as visual prompts to help the model attend to the target area (prompting) and recognize the object, effectively mitigating the gap between the upstream ImageNet classiﬁcation task and the downstream object detection task.
Revised: It leverages query positional embeddings as visual prompts to help the model attend to the target area (prompting) and recognize the object, effectively mitigating the gap between the upstream ImageNet classiﬁcation task and the downstream object detection task.

Original: Experiments show that FP-DETR not only achieves competitive performance on the challenging
Revised: Experiments show that FP-DETR not only achieves competitive performance on the challenging

Original: COCO dataset, but also gains better robustness against common corruptions and generalization to small-size datasets. 
Revised: COCO dataset, but also gains better robustness against common corruptions and generalization to small-size datasets. A CKNOWLEDGEMENT

Revised: This work is supported by National Key R&D Program of China under Grant 2020AAA0105701,

Revised: National Natural Science Foundation of China (NSFC) under Grants 61872327, and Major Special

Original:  E THICS S TATEMENT
Revised: Science and Technology Project of Anhui (No. 012223665049). Dr Jing Zhang is supported by ARC FL-170100117. E THICS S TATEMENT

Original: We acknowledge that all co-authors of this work have read and commit to adhering to the ICLR
Revised: We acknowledge that all co-authors of this work have read and commit to adhering to the ICLR

Original: Code of Ethics.
Revised: Code of Ethics.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: The authors strive to ensure the reproducibility of the experimental results.
Revised: The authors strive to ensure the reproducibility of the experimental results.

Original: Speciﬁcally, implementation details are provided in Section 4.1, Section A. and Section A.3.
Revised: Speciﬁcally, implementation details are provided in Section 4.1, Section A. and Section A.3.

Original: The related studies with the same experimental setup are carefully cited and referred to.
Revised: The related studies with the same experimental setup are carefully cited and referred to.

Original: Moreover, the code will be made publicly available to ensure reproducibility
Revised: Moreover, the code will be made publicly available to ensure reproducibility

Original: A A PPENDIX A. S TRUCTURE OF THE M ULTI - SCALE T OKENIZER
Revised: A A PPENDIX A. S TRUCTURE OF THE M ULTI - SCALE T OKENIZER

Original: Our lightweight multi-scale tokenizer extracts feature maps of 4 different levels to construct the input sequences to the encoder-only transformer.
Revised: Our lightweight multi-scale tokenizer extracts feature maps of 4 different levels to construct the input sequences to the encoder-only transformer.

Original: Different from existing detection transformers, which use a sophisticated ResNet-50 backbone for feature extraction, our multi-scale tokenizer simply downsamples the input image to desired feature resolution.
Revised: Different from existing detection transformers, which use a sophisticated ResNet-50 backbone for feature extraction, our multi-scale tokenizer simply downsamples the input image to desired feature resolution.

Original: Speciﬁcally, we perform nonoverlapping patch embedding, similar to the seminal ViT (Dosovitskiy et al., 2020).
Revised: Speciﬁcally, we perform nonoverlapping patch embedding, similar to the seminal ViT (Dosovitskiy et al., 2020).

Original: The only difference is that we perform multi-scale tokenization and adopt a hierarchical structure to progressively embed the tokens of large resolution.
Revised: The only difference is that we perform multi-scale tokenization and adopt a hierarchical structure to progressively embed the tokens of large resolution.

Original: The detailed structure of our multi-scale tokenizer is shown in
Revised: The detailed structure of our multi-scale tokenizer is shown in

Original: Table 6.
Revised: Table 6.

Original: The dimension D (cid:48) is 192, 384, and 384, respectively for our Lite, Small, and Base model.
Revised: The dimension D (cid:48) is 192, 384, and 384, respectively for our Lite, Small, and Base model.

Original: After patch embedding, the 2D sequence is ﬂattened to 1D sequence and further projected to tokens with dimension D .
Revised: After patch embedding, the 2D sequence is ﬂattened to 1D sequence and further projected to tokens with dimension D .

Original: [Table]
Revised: [Table]

Original: A. E XPLORING E NCODER -D ECODER S TRUCTURE
Revised: A. E XPLORING E NCODER -D ECODER S TRUCTURE

Original: While most detection transformers contain an encoder-decoder transformer, existing study on vision transformers extensively explores the encoder-only transformer for image classiﬁcation.
Revised: While most detection transformers contain an encoder-decoder transformer, existing study on vision transformers extensively explores the encoder-only transformer for image classiﬁcation.

Original: This results in a discrepancy between the upstream ImageNet classiﬁcation task and the downstream object detection task.
Revised: This results in a discrepancy between the upstream ImageNet classiﬁcation task and the downstream object detection task.

Original: To mitigate this problem, we ﬁrst explore pre-training the encoder-decoder transformer on the ImageNet classiﬁcation task, using existing pre-training technologies (Touvron et al., 2021).
Revised: To mitigate this problem, we ﬁrst explore pre-training the encoder-decoder transformer on the ImageNet classiﬁcation task, using existing pre-training technologies (Touvron et al., 2021a).

Original: Speciﬁcally, we pre-train and ﬁne-tune an encoder-decoder transformer structure, denoted as FPEncDec, on the ImageNet-1k dataset, follow the same implementation in 4.1.
Revised: Speciﬁcally, we pre-train and ﬁne-tune an encoder-decoder transformer structure, denoted as FP-EncDec, on the ImageNet-1k dataset, follow the same implementation in 4.1.

Original: The model follows the design of our FP-DETR-Lite, except the last 6 layers of self-attention layers are replaced by 6 decoder layers and the task adaptor is removed.
Revised: The model follows the design of our FP-DETR-Lite, except the last 6 layers of self-attention layers are replaced by 6 decoder layers and the task adaptor is removed.

Original: In each decoder layer, a standard self-attention layer (Vaswani et al., 2017) and a multi-scale deformable attention layer (Zhu et al., 2020) are applied sequentially, following Deformable DETR.
Revised: In each decoder layer, a standard self-attention layer (Vaswani et al., 2017) and a multi-scale deformable attention layer (Zhu et al., 2020) are applied sequentially, following Deformable DETR.

Original: Two variants of FP-EncDec are considered: (1) pre-training with a single class token, follow the common practice in training vision transformers (Dosovitskiy et al., 2020); and (2) pre-training with 300 class tokens, which equals the number of object query embeddings for ﬁne-tuning.
Revised: Two variants of FP-EncDec are considered: (1) pre-training with a single class token, follow the common practice in training vision transformers (Dosovitskiy et al., 2020); and (2) pre-training with 300 class tokens, which equals the number of object query embeddings for ﬁne-tuning.

Original: The 300 class tokens are pooled into a single class token before ﬁnal classiﬁcation.
Revised: The 300 class tokens are pooled into a single class token before ﬁnal classiﬁcation.

Original: [Table]
Revised: [Table]

Original: The results for both pre-training on ImageNet-1k and ﬁne-tuning on COCO 2017 dataset are shown in Table 7.
Revised: The results for both pre-training on ImageNet-1k and ﬁne-tuning on COCO 2017 dataset are shown in Table 7.

Original: As can be seen, pre-training the transformer decoder with a single class token performs poorly on the ImageNet classiﬁcation task, since both the self-attention layers and the projections on the class token in the cross-attention layer are trained on a single class token, which could easily lead to overﬁtting.
Revised: As can be seen, pre-training the transformer decoder with a single class token performs poorly on the ImageNet classiﬁcation task, since both the self-attention layers and the projections on the class token in the cross-attention layer are trained on a single class token, which could easily lead to overﬁtting.

Original: The low pre-training accuracy also limits the model’s detection performance on the downstream task.
Revised: The low pre-training accuracy also limits the model’s detection performance on the downstream task.

Original: By contrast, FP-EncDec pre-trained with 300 class tokens performs better on ImageNet-1k classiﬁcation, the top-1 accuracy is 4.4 higher than the single class token counterpart, since the decoder is more sufﬁciently trained.
Revised: By contrast, FP-EncDec pre-trained with 300 class tokens performs better on ImageNet-1k classiﬁcation, the top-1 accuracy is 4.4 higher than the single class token counterpart, since the decoder is more sufﬁciently trained.

Original: However, it is still inferior to our encoder-only
Revised: However, it is still inferior to our encoder-only

Original: FP-DETR-Lite.
Revised: FP-DETR-Lite.

Original: We conjuncture that this is caused by (1) existing training techniques for vision transformers have been heavily tuned towards the encoder-only structure, which can be sub-optimal for the encoder-decoder transformer; and (2) the discrepancy between upstream and downstream tasks also degenerates the model’s performance on object detection, even if the decoder has been pre-trained.
Revised: We conjuncture that this is caused by (1) existing training techniques for vision transformers have been heavily tuned towards the encoder-only structure, which can be sub-optimal for the encoder-decoder transformer; and (2) the discrepancy between upstream and downstream tasks also degenerates the model’s performance on object detection, even if the decoder has been pre-trained.

Original: We expect these ﬁndings may provide useful insights to the community to rethink the current paradigm of pre-training vision transformers, and pay more attention to the pre-training of encoder-decoder transformers.
Revised: We expect these ﬁndings may provide useful insights to the community to rethink the current paradigm of pre-training vision transformers, and pay more attention to the pre-training of encoder-decoder transformers.

Original: A. P RE - TRAINING ON I MAGE N ET -21 K
Revised: A. P RE - TRAINING ON I MAGE N ET -21 K

Original: For ImageNet-21k pre-training, we follow the pipeline in Ridnik et al.
Revised: For ImageNet-21k pre-training, we follow the pipeline in Ridnik et al.

Original: The ImageNet-21k is ﬁrst pre-processed by three steps: (1) cleaning invalid classes; (2) validation split; and (3) image resizing.
Revised: The ImageNet-21k is ﬁrst pre-processed by three steps: (1) cleaning invalid classes; (2) validation split; and (3) image resizing.

Original: Afterward, the model is trained on the processed dataset (ImageNet-21k-P) using the semantic softmax training (Ridnik et al., 2021).
Revised: Afterward, the model is trained on the processed dataset (ImageNet-21k-P) using the semantic softmax training (Ridnik et al., 2021).

Original: Speciﬁcally, our model is pre-trained on ImageNet-21k-P with a batch size of 4096 for 80 epochs.
Revised: Speciﬁcally, our model is pre-trained on ImageNet-21k-P with a batch size of 4096 for 80 epochs.

Original: The model is initialized from ImageNet-1K pre-trained weights.
Revised: The model is initialized from ImageNet-1K pre-trained weights.

Original: The learning rate is initialized as 3e-4 and scheduled using one-cycle policy (Smith, 2018).
Revised: The learning rate is initialized as 3e-4 and scheduled using one-cycle policy (Smith, 2018).

Original: RandAugment (Cubuk et al., 2020), Cutout (DeVries & Taylor, 2017), Label-smoothing (Szegedy et al., 2016), and True-weight-decay (Loshchilov & Hutter, 2018) are adopted for regularization.
Revised: RandAugment (Cubuk et al., 2020), Cutout (DeVries & Taylor, 2017), Label-smoothing (Szegedy et al., 2016), and True-weight-decay (Loshchilov & Hutter, 2018) are adopted for regularization.

Original: For more details on pre-training, please refer to Ridnik et al.
Revised: For more details on pre-training, please refer to Ridnik et al.

Original: For ﬁne-tuning FP-DETR further, we adopt the same implementation details in 4.1.
Revised: For ﬁne-tuning FP-DETR further, we adopt the same implementation details in 4.1.

Original: For a fair comparison, we also ﬁnetuned Deformable DETR (Zhu et al., 2020) with an ImageNet-21k-P pretrained ResNet-50 backbone.
Revised: For a fair comparison, we also ﬁnetuned Deformable DETR (Zhu et al., 2020) with an ImageNet-21k-P pretrained ResNet-50 backbone.

Original: The ResNet-50 pre-trained weight is taken from the ofﬁcial releaseof Ridnik et al.
Revised: The ResNet-50 pre-trained weight is taken from the ofﬁcial releaseof Ridnik et al.

Original: [Table]
Revised: [Table]

Original: The result is shown in Table 8.
Revised: The result is shown in Table 8.

Original: As can be seen, our FP-DETR can beneﬁt from fully pre-training on the larger-scale ImageNet-21k dataset as well as ﬁne-tuning with the task adaptor.
Revised: As can be seen, our FP-DETR can beneﬁt from fully pre-training on the larger-scale ImageNet-21k dataset as well as ﬁne-tuning with the task adaptor.

Original: Counterintuitively, the performance of Deformable DETR degenerates when using the weights pre-trained on ImageNet-21k.
Revised: Counterintuitively, the performance of Deformable DETR degenerates when using the weights pre-trained on ImageNet-21k.

Original: We conjuncture that only pre-training the CNN backbone of the model makes it difﬁcult for the randomly initialized transformer to adapt to the pre-trained backbone, especially when the backbone has already been well-trained.
Revised: We conjuncture that only pre-training the CNN backbone of the model makes it difﬁcult for the randomly initialized transformer to adapt to the pre-trained backbone, especially when the backbone has already been well-trained.

Original: More research efforts should be made to further dig into this problem.
Revised: More research efforts should be made to further dig into this problem.