Original: Deformable Vision Transformer Based Single-Stage Pedestrian Detector Anonymous Author(s)
Revised: Effectiveness of Vision Transformer for Fast and Accurate Single-Stage Pedestrian Detection

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Pedestrian detection is a challenging field in computer vision, which requires bothfast inference and high accuracy.
Revised: Vision transformers have demonstrated remarkable performance on a variety of computer vision tasks.

Original: Single-stage detectors are faster than region ofinterest based two-stage detectors at the expense ofaccuracy mainly due to the lackof spatially adaptive features. To this end, we propose asingle-stage anchor-freepedestrian detector with enhanced spatial and multi-scale features based on thedeformable vision transformer aiming to achieve the  balance between speed andaccuracy.
Revised: In this paper, we illustrate the effectiveness of the deformable vision transformer for single-stage pedestrian detection and propose a spatial and multi-scale feature enhancement module, which aims to achieve the optimal balance between speed and accuracy. Performance improvement with the use of vision transformers on various commonly used single-stage structures is demonstrated.

Original: The design of the  architecture is investigated in depth.
Revised: The design of the proposed architecture is investigated in depth.

Original: Comprehensivecomparisons with state-of-the-art single- and two- stage detectors on various pedestrian datasets are performed.
Revised: Comprehensive comparisons with state-of-the-art single- and two-stage detectors on various pedestrian datasets are performed.

Original: The proposed detector achieves leading performanceon both Caltech and Citypersons datasets among single- and two- stage methodsusing less parameters compared to the baseline.
Revised: The proposed detector achieves leading performance on Caltech and Citypersons datasets among single- and two-stage methods using fewer parameters than the baseline.

Original: The log-average miss rates for Reasonable(3.8%) and Heavy(36.5%) are decreased to 2.6% and 28.0% on Caltech, and 10.6% and 36.7% on  Citypersons validation datasets respectively.
Revised: The log-average miss rates for Reasonable and Heavy are decreased to 2.6% and 28.0% on the Caltech test set, and 10.9% and 38.6% on the Citypersons validation set respectively.

Original: It evenoutperforms SOTA two-stage detectors in  Heavy subsetby 3% on Citypersonsvalidation set.
Revised: The proposed method outperforms SOTA two-stage detectors in the Heavy subset on the Citypersons validation set with considerably faster inference speed.

Original:  Introduction
Revised:  Introduction

Original: Pedestrian detection is a popular task subordinate to object detection in computer vision.
Revised: Pedestrian detection is a popular task subordinate to object detection in computer vision.

Original: This taskaims at accurately locating and classifying pedestrians in images or videos obtained via visible-lightcameras, infrared cameras etc.
Revised: This task aims towards accurately locating and classifying pedestrians in images or videos obtained via visiblelight cameras, infrared cameras etc.

Original: In this paper, we focus on the detection from RGB images.
Revised: In this paper, we focus on the detection based on RGB images.

Original: Pedestriandetection is very important as it serves as the prerequisite of various vision tasks [1], such as humancentric tasks (person re-identification [2, 3], person search [4], human pose estimation [5] etc.) andmore generic multi-object tracking [6].
Revised: Pedestrian detection is very important as it serves as the prerequisite of various vision tasks [1], such as human-centric tasks (person re-identification [2, 3], person search [4], human pose estimation [5] etc.) and more generic multi-object tracking [6].

Original: It has been applied to autonomous driving [7, 8], videosurveillance [9] and action tracking.
Revised: It has been applied to autonomous driving [7, 8], video surveillance [9] and action tracking.

Original: Pedestrian detection is very challenging mainly due to the varying appearances resulted from thesignificant occlusion and varying scales.
Revised: Pedestrian detection suffers from significant occlusion and varying scales.

Original: On the one hand, significant intra- and inter- class occlusionoccur when a pedestrian is occluded by other pedestrians or objects like cars, bicycles etc., in busyareas.
Revised: Intra- and inter-class occlusion occur when a pedestrian is occluded by other pedestrians or objects like cars, bicycles etc.

Original: Due to the heavy occlusion, only a small part of the human body is visible which significantlyreduces the discriminative features.
Revised: Both significantly reduce the discriminative features and destroy the regular shape of pedestrians.

Original: On the other hand, when the target is far away from the camera,its height and width reflected in the image decrease, and vice versa.

Original: Large pedestrians tend to havemore informative features but are difficult to be fully extracted from a vast region while featuresof small targets are compact but relatively ambiguous with less preserved details.
Revised: For varying scales, large pedestrians tend to have more informative features but are difficult to be fully extracted from a vast region while features of small targets are compact but relatively ambiguous with less preserved details.

Original: In summary, thefluctuating amount and varying shape of effective features are the root problems.
Revised: In summary, the fluctuating amount and varying shape of effective features are the core problems.

Original: These intrinsicproblems challenges the capabilities of feature extraction modules, which act as a long-standingbottleneck in pedestrian detection and almost all other object detection tasks.
Revised: They challenge the capabilities of feature extraction modules, which act as a long-standing bottleneck in pedestrian detection.

Original: To deal with these problems,deep Convolutional Neural Networks (CNNs) based approaches,especially two-stage methods [10–15] based on Fast  and Faster R-CNN [16, 17] have pervaded pedestrian detection tasks owing to the high detection accuracy.
Revised: To deal with these problems, two-stage methods [10–15] based on Fast [16] and Faster R-CNN [16] have pervaded pedestrian detection tasks owing to the high detection accuracy.

Original: These methods first make coarsepredictions of targets in the Region Proposal Network (RPN), then refine the bounding boxes andpredict the final scores based on the features inside the proposal regions.
Revised: These methods first make coarse predictions of targets via the Region Proposal Network (RPN), then refine the bounding boxes and predict the final scores based on the features inside the proposal regions.

Original: In addition to methodsfor general object detection, pedestrian detectors take the advantage of unique characteristics oftargets, such as the mask of visible parts [49, 10] and key-points of human bodies [25].
Revised: In addition to methods for general object detection, pedestrian detectors take the advantage of unique characteristics of targets, such as the mask of visible parts [17, 10] and key points of human bodies [18].

Original: However,the inference speed of these methods is limited by the repeated prediction. This requires longertraining time per epoch and is hard to be applied tosome real-world scenes.
Revised: However, the inference speed of these methods is limited by the repeated predictions which makes them hard to be applied to real-world scenes.

Original: Secondly, the design ofthe anchors in RPN is seriously parameter depended.

Original: Lastly, the procedures of jointly training twonetworks are complicated.

Original: To achieve faster inference and easier training procedures, single-stage approaches, which only makea single round prediction, are developed [18] and applied to pedestrian detection [19–23].
Revised: To achieve faster inference, single-stage approaches, which only make one round prediction, are developed [19] and applied to pedestrian detection [20–24].

Original: However,they suffer from decreased detection accuracy.
Revised: However, they suffer from decreased detection accuracy.

Original: The miss rates of two-stage methods in Reasonablesubset on Caltech are smaller than 4% [14][10, 24, 25], while those for single-stage methods arelarger than 4.5%[21].
Revised: The miss rates of two-stage methods in the Reasonable subset on Caltech are reduced to less than 4% [14, 10, 25, 18], while those for single-stage methods are larger than 4.5%

Original: For  Citypersons validation set, the miss rates for the former is less than 40%in Heavy subset, which is much lower than the latter (42% [23]).
Revised: For the Citypersons validation set, the miss rates for the former are less than 40% in the Heavy subset, which is much lower than the latter (42% [24]).

Original: Our research purpose is to improvethe detection performance of single-stage pedestrian detectors, especially the detection accuracy of Reasonable and Heavy subsets, and to narrow the gap between single- and two- stage methods.
Revised: In this paper, our aim is to improve the detection accuracy, especially in Reasonable and Heavy subsets, and to narrow the gap between single- and two-stage methods with fast inference. Typical single-stage detectors use anchors (SSD [26]) or are anchor-free (CSP [22]).

Revised: The former generates rectangular bounding boxes with different aspect ratios and scales centered at each pixel of the feature maps at certain levels.

Original: Massive studies have been carried out to enhance the single-stage pedestrian detectors based on
Revised: Anchor scales are designed smaller at lower levels to facilitate the detection of small objects. These methods predict the offsets w.r.t.

Original: Center and Scale Prediction (CSP) [21], a typical single-stage and anchor-free pedestrian detectorwhich significantly accelerates the inference [1].
Revised: the upper left position, height and width of the anchor. The latter method, taking CSP as an example, only predicts the logarithm height and offsets w.r.t.

Original:  In the past year, most research focuses on the fusionof representative features [26–31].
Revised: centers of each pixel. For anchor-based single-stage detectors, ALFNet [21] refines anchors progressively with stacked prediction blocks to remedy the lack of proposal regions. In the past year, most research focuses on the fusion of representative features [27–32] to improve single-stage methods.

Original: For example, [29] enhancesthe features via increasing semanticinformation at low-level and enriches the localization information at high-level.
Revised: For example, [30] enhances features via increasing semantic information at a low level and enriches the localization information at a high level.

Original: Similarly, [31] fusesthe feature maps with different scales in adequate proportions.
Revised: Similarly, [32] fuses the feature maps with different scales in adequate proportions.

Original: [28, 30] also explore new strategies toaggregate multi-level features.
Revised: [29, 31] also explore new strategies to aggregate multi-level features.

Original: These feature enhancements are mainly performed along the featuremap level dimension, which makes sense because of  the intrinsically unbalanced feature informationbetween shallower feature maps and deeper ones.
Revised: These feature enhancements are mainly performed along the dimension of feature level due to the intrinsically unbalanced feature information between shallow and deep feature levels.

Original: However, this unbalanceproblem also exists intwo-stage detectors.
Revised: However, this unbalance also exists in two-stage detectors.

Original: This means that it is not the root cause of the poorer accuracy of single-stagemethods.
Revised: As such, this is not the particular reason for the poorer accuracy of single-stage methods.

Original: To address the root cause, the general architecture of single- and two- stage detectors are compared in Figure 1.
Revised: The general architecture of single- and two-stage detectors are compared in Figure 1.

Original: Assume that the training strategies and the detection head are the same for both methods,the difference in the architechture lies in the information fed into the detection head.
Revised: Assuming that the training strategies and the detection head are the same for both methods, the difference in the architecture lies in the information fed into the detection head.

Original: For two-stagemethods, both positions and features of the proposal regions are fed into the detection head.
Revised: For two-stage methods, both positions and features of the proposal regions are fed into the detection head.

Original: Theseproposals contain potential pedestrians.
Revised: These proposals contain potential pedestrians.

Original: Thereby, the detection head classifiesvia spatially targetfocused features with less background interruptions and refines the bonding boxes by predictingsmall offsets from the proposal positions.
Revised: Thereby, the detection head classifies spatially target-focused features with fewer background interruptions and refines the bounding boxes by predicting small offsets from the proposal positions.

Original: For single-stage methods, each pixel in the feature mapserves as the ‘proposal region’ with no pre-estimated positions.
Revised: For single-stage methods, each pixel in the feature map serves as the ‘proposal region’ with no pre-estimated positions.

Original: These  pixels share the same receptivefield size which may be too small to include sufficient information for large targets or so largethat the background information overwhelms the useful features for detection.
Revised: These receptive fields of these pixels share the same size, which may be too small to include sufficient information for large targets or so large that the background information overwhelms the useful features.

Original: Obviously, thisis more challenging for the classifier compared to two-stage methods.
Revised: This is more challenging for the classifier compared to two-stage methods.

Original: Additionally, single-stagemethods have to regress from scratch, which is more difficult than simple refinement in two-stagemethods.
Revised: Additionally, single-stage methods have to regress from scratch, which is more difficult than simple refinement.

Original: According to the analysis, the lack of spatially target-focused feature representation and theprediction of bounding boxes from scratch serve as two key bottlenecks hindering the improvementof single-stage detectors.
Revised: Thus, the lack of spatially target-focused feature representation and the prediction of bounding boxes from scratch are the two key bottlenecks hindering the improvement of single-stage detectors.

Original: To make the features fed into the detection head concentrating on the targets or other useful information automatically without the assistance of proposal regions,  we take the advantage of visiontransformers.
Revised: To make the features fed into the detection head concentrate on the targets or other useful information automatically without the assistance of proposal regions, in this paper we take the advantage of vision transformers.

Original: Vision transformers describe the pairwise dependency of each entity in the featuremap with attention weights.
Revised: Vision transformers describe the pairwise dependency of each entity in the feature map with attention weights.

Original: The output weighted averaged feature is the adaptive aggregation ofimportant entities (with higher attention weights) while the disturbing information is suppressed(with lower attention weights).
Revised: The output weighted averaged feature is the adaptive aggregation of important entities (with higher attention weights) while the disturbing information (with lower attention weights) is suppressed.

Original: Using such attention mechanism on top of the backbone enables thesingle-stage detector to supplement spatially filtered features easily for subsequent classification andregression tasks.
Revised: Using such attention mechanism on top of the backbone enables the single-stage detector to supplement spatially filtered features easily for subsequent classification and regression.

Original: Thus, the modified detector makes the best use of the fast inference originated fromsingle-stage methods and more effectiveand cleaner features.
Revised: In this case, the modified detector makes the best use of the fast inference originating from single-stage methods and more effective features.

Original: Our main contributions are as follows:
Revised: Our main contributions are as follows

Original:  • Proof the effectiveness ofusing the deformable vision transformer to improve the accuracy of  single-stagepedestrian detectors while maintaining fast inference.
Revised: : • Demonstrate the effectiveness of the deformable vision transformer in improving the accuracy of commonly used single-stage detectors on pedestrian datasets.

Original: • Propose a novel feature extractor which enhance features along both scale and spatial dimensions.
Revised: • Extend the application of vision transformers on top of the backbone in pedestrian detection tasks.

Original: Comprehensive experiments show that the  performance gap between thepresented method and two-stage methods is narrowed onvarious pedestrian datasets. • Extend the application of vision transformers on top of  the backbone in pedestrian detection tasks.
Revised: • Achieve the best performance among single-stage detectors on the Caltech test set and Citypersons validation set while maintaining fast inference and reducing the number of parameters. • Narrow the gap of detection accuracy between the single- and two-stage methods in pedestrian detection.

Original:  Related works
Revised:  Related works

Original: Vision transformers in Computer Vision (CV) originates from the transformers widely utilizedin Natural Language Processing (NLP).

Original: Thanks to the attention mechanism, vision transformersare applied to image feature extraction as (like ViT [32] and Swin Transformer [33]) or  on topof the backbone like DETR [34].
Revised: Currently, vision transformers are used to establish general-purpose backbones (ViT [33] and Swin Transformer [34]) or stack on top of the backbone (DETR [35]).

Original: Our detector belongs to the latter category.
Revised: Considering that we focus on pedestrian detection, this paper explores the latter case.

Original: DETR consists ofthe convolutional backbone, six encoder-decoder pairs and the prediction head.
Revised: DETR consists of the convolution backbone, six encoders and decoders and the prediction head.

Original: It is an inspiringend-to-end detector which has aroused plenty studies based on it.
Revised: It is an inspiring end-to-end detector but is memory-consuming.

Original: One typical variant of DETR is thedeformable DETR [35].

Original: It is known that DETR requires massive memory to store the self-attentionweights within each Multi-Head Self-Attention (MHSA) layer.
Revised: DETR requires massive memory to store the self-attention weights within each Multi-Head Self-Attention (MHSA) layer.

Original: The memory cost is linear to thenumber of attention heads and is square to the number of pixels in the down sampled feature map.
Revised: The memory cost is linear to the number of attention heads and is square to the number of pixels in the down-sampled feature map.

Original: Additionally, first and second order momentum in optimizers introducein further memory cost intraining.
Revised: Additionally, first and second order momentums in optimization introduce further memory cost in the training procedure.

Original: To achieve accurate detection, more attention heads and relatively large down sampledfeature map are preferred, which results in memory explosion.
Revised: In pedestrian detection, more attention heads and relatively large down-sampled feature maps are preferred to enhance detection accuracy, especially for small targets.

Original:  Toovercome this problem, thefast-converging and memory-saving deformable DETR  is proposed.
Revised: This results in memory explosion using DETR. To this end, deformable DETR [36] is proposed.

Original: It only calculates the attentionweights at several sampling locations rather than each pixel in the feature map, so that the memoryis linear to the number of pixels, which makes training with high resolutionsand limited memoryresources possible.
Revised: It only needs the attention weights at several sampling locations rather than each pixel in the feature map. The memory cost of the attention weights is linear to the number of pixels, which makes training with high resolutions possible.

Original: Experiments show that  deformable DETR outperforms  Faster R-CNN [17] and
Revised: Experiments show that the deformable DETR outperforms the Faster R-CNN [37] and

Original: DETR on COCO 2017 validation set [36].
Revised: DETR on COCO 2017 validation set [38].

Original: So far, vision transformers seem inspiring and potential, but it is rarely  applied in pedestrian detectionin the form of DETR or its variants.
Revised: So far, vision transformers show great potential, but they have rarely been applied to pedestrian detection in the form of DETR or its variants.

Original: [37] found that deformable DETR not only behaves worsethan Faster R-CNN on CrowdHuman dataset [38], but also costs tenfold training time.
Revised: This is because it has been observed that they perform worse than commonly used Faster R-CNN on CrowdHuman dataset [39] and require tenfold training time [40].

Original: Althoughit narrowed the performance gap using dense object queries and the rectified attention field, themodified deformable DETR still show limited advantage over the traditional Faster R-CNN.
Revised: Although [40] proposed the use of dense object queries and the rectified attention field to enhance scale-adaptive feature extraction in the decoding phase, the modified deformable DETR still shows a limited advantage over the traditional Faster R-CNN.

Original: Thisimplies that rigidly putting the whole six encoder-decoder pairs into the pedestrian detector maynot be cost-effective.
Revised: This implies that rigidly putting the whole six encoder-decoder pairs into the pedestrian detector may not be cost-effective.

Original: As an example, BoTNet [39] only substitutes the convolutional layers inresidual bottleneck blocks in the last stage of ResNet [40] for MHSA layers, but it produces strongperformance on ImageNet validation set [41].
Revised: As an example, BoTNet [41] only substitutes the convolution layers in residual bottleneck blocks in the last stage of ResNet [42] for MHSA layers, but it produces strong performance on ImageNet validation set [43].

Original: Inspired by this and the aforementioned limitation,our work only utilizes a single encoder of the deformable DETR as an adaptive feature extractor andapplies it to the single-stage anchor-free CSP.
Revised: Inspired by this and the aforementioned limitation, our work only utilizes a single encoder of the deformable vision transformer as an adaptive feature extractor and applies it to commonly used single-stage detectors for better detection accuracy and fast inference.

Original:  Method
Revised:  Method

Original: Deformable  Transformer Encoder : Figure 2 shows the deformableencoder in the multi-scaledeformable vision transformer. This encoder  takes the multi-scale feature maps (cid:8) z l (cid:9)
Revised: Deformable Vision Transformer Encoder : The deformable vision transformer encoder (Figure 2) takes the L feature maps (cid:8) z l (cid:9)

Original: L l =1( L = 3 isthe number of feature maps) with height H l and width W l at scale l and reference points, which are the positions of grid centers of the feature maps, as inputs.
Revised: L l =1 with height H l and width W l at scale l and reference points, which are the positions of grid centers of the feature maps, as inputs.

Original: It  the enhanced feature maps with the same resolution as the input.
Revised: It outputs the enhanced feature maps with the same resolution as the input.

Original: The input feature maps are first added by fixed encoded positional[34] and learnable level information [35] to disambiguate spatial and scale positions, then projected tothe query feature map z q via  linear projection.
Revised: The input feature maps are first added by fixed encoded positional [35] and learnable level information [36] to disambiguate spatial and scale positions, then projected to the query feature map z q via a linear layer.

Original: The feature maps also generate the value feature mapsz v with a linear layer but no encodings.
Revised: The feature maps also generate the value feature maps z v with a linear layer but without encoding.

Original: The query feature map z q , value feature map z v togetherwith the pre-generated reference points are sent to the Multi-Scale Deformable Attention (MSDA)layer to enhance spatially adaptive features, followed by a Fee-Forward Network (FFN).
Revised: The query feature map z q , value feature map z v together with the pre-generated reference points are sent to the Multi-Scale Deformable Attention (MSDA) layer to enhance spatially adaptive features, followed by a Feed-Forward Network (FFN).

Original: In summary, the encoder supplements the semantic information of input feature maps via the embedded attention layer and FFN.
Revised: In summary, the encoder supplements the semantic information of input feature maps via the embedded attention layer and FFN.

Original: Multi-Scale Deformable Attention : The MSDA layer sums the selected entities  in the value featuremap z vat sampling locations with predicted attention weights for each corresponding query entity.
Revised: Multi-Scale Deformable Attention : The MSDA layer sums the selected entities at sampling locations in the value feature map z v with predicted attention weights by each corresponding query entity.

Original: These attention weights W are the linear projection of query features z q followed by a softmaxoperator along scale and sampling point dimensions.
Revised: These attention weights W are the linear projection of query features z q followed by a softmax operator along scale and sampling point dimensions.

Original: For a single query entity, it only needs N h N l N pattention weights, representing the significance of selected value features at different attention heads,scales, and points.
Revised: For a single query entity, it only needs N h N l N p attention weights, representing the significance of selected value features at different attention heads, scales, and points.

Original: Selections are decided by the sampling locations which are the summation ofreference points p and sampling offsets ∆ p which is the embedding of the query features.
Revised: Selections are decided by the sampling locations which are the summation of reference points p and sampling offsets ∆ p which is the embedding of the query features.

Original: At eachfloat sampling location,  selected value feature is bilinear interpolated for accuracy and training offsetpredictor.
Revised: At each float sampling location, the selected value feature is bilinear interpolated for accuracy and training offset predictor.

Original: With weight, sampling locations and selected value features prepared, the q -th element ofthe separate output feature z so ,h ∈ R N q × c v ( N q = 
Revised: With weight, sampling locations and selected value features prepared, the q -th element of the separate output feature z so ,h ∈ R N q × c v ( N q = (cid:80) Ll =1 H l W l ) at attention head h

Original: [Equation]
Revised: (total

Original: Ll =1 H l W l ) at attention head
Revised: N h heads) is z so ,h q =

Original: h (total N h heads)is
Revised: N p (cid:88) p =

Original: [Equation]
Revised: [Equation]

Original: where p , q , l and h index the sampling offsets, elements of the deformable attention feature z o ,the scale of value v and attention head.
Revised: where p , q , l and h index the sampling offsets, elements of the deformable attention feature z o , the scale of value v and attention head.

Original: W plhq is a value from the weight W ∈ R N q × N h
Revised: W plhq is a value from the weight W ∈ R N q × N h

Original: × L × N p .p ∈
Revised: × L × N p .

Original:  R N q × L × 2 and ∆ p ∈ R N q × N h
Revised: p ∈ R N q × L × 2 and ∆ p ∈ R N q × N h

Original: × L × N p × 2 are the reference points and sampling offsets.
Revised: × L × N p × 2 are the reference points and sampling offsets.

Original: p ql and∆ p plhq denote the position of a single reference point and one of its corresponding sampling offsetsrespectively.
Revised: p ql and ∆ p plhq denote the position of a single reference point and one of its corresponding sampling offsets respectively.

Original: The separate output features from N h attention heads are projected to the q -th elementof the final output deformable attention feature z o by a linear layer expressed as
Revised: The separate output features from N h attention heads are projected to the q -th element of the final output deformable attention feature z o by a linear layer expressed as

Original: [Equation]
Revised: [Equation]

Original: where W ′ h ∈ R c
Revised: where W ′ h ∈ R c

Original: ×
Revised: ×

Original: c
Revised: c

Original: v denotes the learnable weight for the h -th attention head.
Revised: v denotes the learnable weight for the h -th attention head.

Original: Center and Scale Prediction :The architecture of CSP is simple. It concatenates multi-scale featuremaps and fed them to the detection head. Two main innovations of CSP are the anchor-free structureand predicting the center heatmap, height ("scale" is used in the original work, however, we use"height" in case of confusion with feature map scale) map and the optional offsets w.r.t. the centersinstead of the traditional bounding boxes.
Revised: Proposed Feature Enhancement Module :

Original: With no anchors applied, less hyper-parameters are required which simplifies training and design. The sample center and scale ground truths are shown in Figure3. Centers of all pedestrians are assigned as positives in the center heatmap, while negatives for otherpositions. For the height map, these points are assigned with the logarithm of the object height.
Revised: The features are enhanced owing to the self-attention mechanism to supplement the spatially adaptive features across multiple scales.

Original: Proposed Single-stage Detector with Enhanced Features : The proposed detector consists of  the ResNet 50 backbone, the detection neck, and head as shown in Figure 3.
Revised: The proposed module simply consists of convolution/deconvolution and normalization layer pairs ahead of and after the deformable encoder, and a final feature fusion step 3.

Original: The detection head predictsthe center and scale (height) maps and offsets to slightly adjust the positions of pedestrians.

Original: Thefeatures are enhanced in the detection neck where we take the advantage of the deformable transformerencoder, particularly the embedded MSDA mechanism to supplement the spatially adaptive featuresacross multiple scales.

Original: In the neck, input feature maps  are first upsampled with deconvolutional layersor encoded with the convolutional layer to generated multi-scale feature maps (cid:8) z l (cid:9) 3 l =1 .
Revised: In this module, input feature maps from the backbone are first upsampled with deconvolution layers or encoded with the convolution layer to generate multi-scale feature maps (cid:8) z l (cid:9) 3 l =1 .

Original: They arefollowed by group normalization to prevent the Internal Covariate Shift (ICS) that might be inductedby subsequent linear operations in the deformable encoder.
Revised: They are followed by group normalization to prevent the Internal Covariate Shift (ICS) that might be inducted by subsequent linear operations in the deformable encoder.

Original: The encoder yields enhanced multi-scalefeature maps.
Revised: The encoder yields enhanced multi-scale feature maps.

Original: Enhanced features are upsampled to keep the resolution as ( H/ 4 , W/ 4) for accuratedetection.
Revised: Enhanced features are upsampled to keep the resolution as ( H/ 4 , W/ 4) for accurate detection.

Original: They are normalized via L2Norm [21] before concatenation along the channel dimension.
Revised: They are normalized via L2Norm [22] before concatenation along the channel dimension.

Original: This makes the features at different scales contribute equally to the final feature representation fedinto the detection head.
Revised: This makes the features at different scales contribute equally to the final feature representation fed into the detection head.

Original: Concatenated features are compressed along the channel dimension to reducethe network parameters.
Revised: Concatenated features are compressed along the channel dimension to reduce the network parameters.

Original: Compared to the architecture of CSP, the proposed detector adopts the novelmulti-scale deformable attention mechanism in the detection neck. Other parts maintain the sameexcept for an additional convolutional layer is applied in  CSP detection head before the branches.
Revised: The output feature maps can be fed into the detection head used in SSD, CSP etc. Except for this standard structure, the use of convolution/deconvolution layers can be adjusted according to the resolution of input, and the concatenation step can be removed if predictions are made at separate levels.

Original: Training : Same ground truth and loss functions as [21] are utilized.
Revised: Training : For anchor-free cases, namely CSP, the loss function follows [22].

Original: The overall loss consists of threeparts as: [Equation] ) where  the center heatmap loss L center , height map lossL height and offset loss L offset are given inthe supplementary.
Revised: The overall loss consists of three parts as Equation(3) where L c , L h and L o stand for the center heatmap loss, height map loss and offset loss.

Original: Weights for each loss λ c , λ s and λ o are set as 0.01, 1 and 0.1 [21].
Revised: Weights for each loss λ c , λ s and λ o are set as 0.01, 1 and 0.1 [22].

Revised: For anchor-based cases, namely SSD or ALFNet, the multi-task loss function is formulated with two objectives as

Original: Inference : Predicted scores above 0.01 are kept.
Revised: Equation(4) [21] where λ cls is experimentally set as 0.01 in the following experiments.

Original: The predicted pedestrian centers are the summationof the offsets and positions of those qualified scores.
Revised: [Equation]

Original: The predicted width is the height multiplied bythe uniform aspect ratio 0.41 [21].
Revised: Inference : For anchor-free single-stage methods, the predicted width is the height multiplied by the uniform aspect ratio 0.41 [22].

Original: The predicted sizes are remapped to the original image size andfed to NMS.
Revised: If not specified, bounding boxes with scores above 0.01 are kept and merged by Non-Maximum Suppression (NMS) with the IoU threshold of 0.5.

Original:  Experiments 
Revised:  Experiments 4.1 Settings

Original: 4.1 Settings

Original: Datasets : The proposed detector is evaluated on two commonly used public pedestrian datasets:
Revised: Datasets : The proposed detector is evaluated on two commonly used public pedestrian datasets:

Original: Caltech [42] and Citypersons [43].
Revised: Caltech [44] and Citypersons [45].

Original: The Caltech dataset is an approximately 10 hours of 480x640video taken in a single urban city.
Revised: The Caltech dataset is an approximately 10 hours of 480x video taken in a single urban city.

Original: The standard training set contains about 250k frames with 350kbounding boxes.
Revised: The standard training set contains about 250k frames with 350k bounding boxes.

Original: In our experiments, the training data augmented by 10 folds containing 42782 images with 13674 persons and the standard test set containing 4024 images with corresponding newannotations [44] and fixed aspect-ratio for bounding boxes [45] are used.
Revised: In our experiments, the training data augmented by 10 folds containingimages with 13674 persons and the standard test set containing 4024 images with corresponding new annotations [46] and fixed aspect-ratio for bounding boxes [47] are used.

Original: Citypersons training setrecorded across 18 different cities, 3 seasons and various weather conditions with 19654 persons in2975 high resolution (1024x2048) images.
Revised: Citypersons training set recorded across 18 different cities, 3 seasons and various weather conditions with 19654 persons in 2975 high resolution (1024x2048) images.

Original: The validation set contains 500 images across 3 cities.
Revised: The validation set contains 500 images across 3 cities.

Original: Training Details : Our network is based on the CSP reproduction provided by Pedestron [45]. ResNet50 pretrained on ImageNet, Adam with moving average weights [46] and step learning rate scheduleare applied.
Revised: Training Details : If not specified, the ResNet50/VGG16 pre-trained on ImageNet, Adam with moving average weights [48] and step learning rate schedule are applied.

Original: Data augmentation techniques including random horizontal flip  and scaling are applied.
Revised: Data augmentation techniques including random horizontal flip with a probability of 0.5 and scaling are applied.

Original: For Caltech, additional random color distortion and cropping are implemented.
Revised: For Caltech, additional random color distortion and cropping are implemented.

Original: The input imagesare rescaled to 336x448for Caltech and 640x1280 for  Citypersons dataset.
Revised: The input images are rescaled to 336x448 and 640x1280 for Caltech and Citypersons datasets respectively.

Original: The proposed network istrained with a single NVIDIA GeForce RTX 3090 GPU for 10 and 75 epochs with batch size 16 and4 on Caltech and Citypersons respectively.
Revised: The detectors are trained with a single NVIDIA GeForce RTX 3090 GPU for 10 and 75 epochs with batch size 16 and 4 on Caltech and Citypersons respectively using the anchor-free CSP detection head.

Original: The  learning rate  and related hyperparameters will bepresented afterwards according to different designs of architectures.
Revised: The base learning rate is 0.5e-4 and decreased by a factor of 0.5 after 6 and 60 epochs.

Original: All the training is performedwith a randomly chosen and fixed seed.
Revised: Initialization is performed with a randomly chosen and fixed seed.

Revised: Tools provided by [47] are used in the experiments.

Original: Metrics : Log-average Miss Rate (denoted as MR − 2 or miss rate in this paper) over False Positive
Revised: Metrics : Log-average Miss Rate (denoted as MR − 2 or miss rate in this paper) over False Positive

Original: Per Image (FPPI) in the range [10 − 2 , 10 0 ] is calculated over Reasonable, Small, Heavy, and Allsubsets defined in Table 1.
Revised: Per Image (FPPI) in the range [10 − 2 , 10 0 ] is calculated over Reasonable, Small, Heavy, and All subsets defined in Table 1.

Original: The lower the MR − 2 the better.
Revised: The lower the MR − 2 the better.

Original: [Table]
Revised: [Table]

Original: 4.2 Ablation studies
Revised: 4.2 Ablation Experiments

Revised: Effectiveness of Enhanced Feature Maps : For generality, the proposed module is applied to three commonly used single-stage structures in pedestrian detection: anchor-based, progressive refinement, and anchor free.

Revised: For anchor-based single-stage methods, like SSD300 [26], Table 2 shows that the proposed module shows stable and significant improvement in all three subsets with an IoU threshold of 0.5 in NMS.

Revised: For refined anchors, we append two Convolutional Predictor Blocks [21] after the feature maps at the last three stages of the backbone and an extra stage.

Revised: The second block refines the coarse anchors predicted by the first block.

Original: Feature Maps Scales : Different combinations of multi-scale feature maps (cid:8) z l (cid:9) 3 l =1 with threedownsampling ratios ( 1 / 4 , 1 / 8 and 1 / 16 ) are compared in Table 2.
Revised: Table 3 demonstrates that even the anchors are refined progressively to remedy the lack of proposal regions, the separate enhancement at each level can bring improvement in certain subsets (level 0 improves Reasonable and All subsets, level 1, 2 improves Heavy subset). The combination of multi-level inputs brings in stable improvement in all the subsets. For anchor-free methods, We evaluated the influence of the proposed module on the baseline CSP [22] detector on Caltech (Table 4) and Citypersons (Table 5). Results from both datasets indicate that with the enhanced feature maps, the miss rates are decreased significantly in Reasonable, Heavy and

Original: In this comparison, only z o 3which has the same resolution as z 3 upsampled to ( H/ 4 , W/ 4) ,and is fed into the detection headwithout intermediate concatenation and L2Norm.
Revised: All subsets, in particular, the Heavy subset witnesses a decrease of up to 7.9%, and the Reasonable subset up to 3.1%. The additional enhancement module increases the inference time, however, they contain fewer learnable parameters as shown in Table 6. As the combination of CSP achieves the best results, subsequent experiments follow this implementation. According to the above, the proposed module is effective for general single-stage detectors on pedestrian datasets, owing to the multi-scale deformable self-attention mechanism to enhance spatially adaptive features across levels.

Original: Results show that feature maps  with  ratios  1 / 4 ,1 / 8, and 1 / 6 for each scale produce the best performance, which is yield by upsampling c 1 and c 2by two times while maintaining the scale of c 3 .
Revised: [Table] Feature Maps Scales : Different combinations of multi-scale feature maps (cid:8) z l (cid:9) 3 l =1 with three downsampling ratios ( 1 / 4 , 1 / 8 and 1 / 16 ) are compared in Table 7. In this comparison, only z o [Table]

Original: [Table]
Revised: [Table]

Revised: which has the same resolution as z 3 upsampled to ( H/ 4 , W/ 4) , and is fed into the detection head without intermediate concatenation and L2Norm.

Revised: Results show that feature maps with ratios 1 / 4 , 1 / 8 , and 1 / 6 for each scale produce the best performance, which is yielded by upsampling c 1 and cby two times while maintaining the scale of c 3 .

Original: Enhanced Feature Map Scales : Fix the downsampling ratios of (cid:8) z l (cid:9) 3 l =1 as 1 / 4 , 1 / 8 , and 1 / 16 ,feed different collections of multi-scale enhanced feature maps { z o l } 3 l =1 to the detection head.
Revised: Enhanced Feature Map Scales : Fix the downsampling ratios of (cid:8) z l (cid:9) 3 l =1 as 1 / 4 , 1 / 8 , and 1 / 16 , feed different collections of multi-scale enhanced feature maps { z o l } 3 l =1 to the detection head.

Original: Notethat the deformable encoder keeps the resolutions of the input feature maps, the downsampling ratiosfor { z o l } 3 l =1 are 1 / 4 , 1 / 8 and 1 / 6 respectively.
Revised: Note that the deformable encoder keeps the resolutions of the input feature maps, the downsampling ratios for { z o l } 3 l =1 are 1 / 4 , 1 / 8 and 1 / 6 respectively.

Original: All the enhanced feature maps are first upsampled to( H/ 4 , W/ 4), if needed, followed by normalization when multiple feature maps are utilized.
Revised: All the enhanced feature maps are first upsampled to ( H/ 4 , W/ 4) if needed, followed by normalization when multiple feature maps are utilized.

Original: They arethen processed in three ways: 1.
Revised: They are then processed in three ways: 1.

Original: Cat: Concatenate them along the channel dimension followed by acompression layer to reduce the number of channels to 256. 2.
Revised: Cat: Concatenate them along the channel dimension followed by a compression layer to reduce the number of channels to 256. 2.

Original: Add: Implement elementwise sum ofthe enhanced feature maps.
Revised: Add: Implement element-wise sum of the enhanced feature maps.

Original: Sep: No fusion operations across multi-scales; send them separatelyto the detection head which doubles the number of predictions.
Revised: Sep: No fusion operations across multi-scales; send them separately to the detection head which doubles the number of predictions.

Original: Table 3 compares various strategiesto fuse the multi-scale enhanced feature maps.
Revised: Table 8 compares various strategies to fuse the multi-scale enhanced feature maps.

Original: It shows that concatenation followed by L2Normproduces the overall best results in both Reasonable and Heavy subsets on  Caltech test set.
Revised: It shows that concatenation followed by L2Norm produces the overall best results in both Reasonable and Heavy subsets on the Caltech test set.

Original: Choice of the Normalization Method Applied to (cid:8) c l (cid:9) 3 l =1 : As Table 3 shows, GN performs best in  Reasonable subset while L2N in  Heavy subset.
Revised: Choice of the Normalization Method Applied to (cid:8) c l (cid:9) 3 l =1 : As Table 8 shows, GN performs best in the Reasonable subset while L2N in the Heavy subset.

Original: Considering that the miss rate of GN is 11.6% lowerin Reasonable subset and only 1.2% higher in  Heavy subset compared to L2N, GN is utilized in ourexperiments as the blue blocks above enhanced feature maps in Figure 3 indicate. [Table]
Revised: Considering that the miss rate of GN is 11.6% lower in the Reasonable subset and only 1.2% higher in the Heavy subset compared to L2N, GN is utilized in our experiments before the encoder as presented in Figure 3.

Original: Number of Encoders :
Revised: Number of Encoders :

Original: Based on the settings in the last part, different numbers of encoders are tested.
Revised: Based on the settings in the last part, different numbers of encoders are tested.

Original: These encoders are connected in series.
Revised: These encoders are connected in series.

Original: Table 4 presents that although more encoders can providehigher level semantic information, the best result is observed when only a single encoder is applied.
Revised: Table 9 presents that although more encoders can provide higher-level semantic information, the best result is observed when only a single encoder is applied.

Original: This phenomenon supports the design of BotNet [39] to some extent, which indicates that followingthe whole set of six encoder-decoder pairs in (deformable) DETR may not be suitable for specificobject detection tasks.
Revised: This phenomenon supports the design of BotNet [41] to some extent, which indicates that following the whole set of six encoders and decoders in (deformable) DETR may not be suitable for specific [Table]

Original: A single encoder can also work effectively with least number of learnableparameters, which prevents overfitting.
Revised: + Enhancement module

Original: [Table]
Revised: [Table]

Original: Importance of Enhanced Feature Maps : We evaluated the baseline CSP detector and the proposeddetector which improves CSP by adopting the deformable encoder-based feature enhancementmodule in the detection neck. Both the single and concatenated enhanced feature map(s) are assessed. Evaluation is conducted on Caltech (Table 5) and Citypersons (Table 6) respectively. Results fromboth datasets indicate that with the enhanced feature map(s), the miss rates are decreased significantlyin Reasonable, Heavy and All subsets, in particular, the Heavy subset witnesses a decrease up to 7.9%,and Reasonable subset up to 3.1%.
Revised: [Table]

Original: This means that the idea of extracting spatially adaptive featuresworks effectively for the single-stage detector. With the concatenation of multi-scale enhancedfeature maps (third row), the performance is further improved comprehensively on Caltech datasetcompared to the single feature map (second row).
Revised: [Table]

Original: This owes to the deformable sampling locationsand attention weight maps allowing to adaptively extract useful features, such as visible features,for the task. Furthermore, the feature map at scale 1 focuses on the visible part of the pedestrianwhen the reference point locates on the visible part of the target, complementing the lack of occludedtarget’s information with the sole use of the feature map at scale 3 as shown in the supplementary.
Revised: object detection tasks.

Original: Due to the additional interpolation operation in the deformable self-attention layer, the proposeddetectors require longer inference time compared to the baseline, however, they contain less learnableparameters as shown in Table 7. [Table] [Table] [Table]
Revised: A single encoder can also work effectively with the least number of learnable parameters, which prevents overfitting.

Original: 4.3 Comparison with the state-of-the-arts
Revised: 4.3 Comparison with the state-of-the-arts

Original: We compared two proposed architectures, namely the single enhanced feature map z

Original: o 3 and theconcatenation of z o 3 and z o 1 , with the state-of-the-art single- and two- stage detectors on Caltech and

Original: Citypersons datasets.

Original: Proposed detectors are trained with the hyperparameters presented in Table 8.

Original: Table 9 and Table 10 show that the proposed detectors achieve lowest miss rates in Reasonable and Heavy subsets and leading performance in  All subset on both datasets among presented single-stagedetectors.
Revised: Table 10 and Table 11 show that the proposed module achieves the lowest miss rates in Reasonable and Heavy subsets and leading performance in the All subset on both datasets among presented single-stage detectors.

Original: [Table] For  Caltech dataset, the lowest miss rate (3.7%) of the proposed detector in  Reasonable subset is 0.2%smaller than that of the two-stage KGSNet presented in the upper part of Table 9.
Revised: For the Caltech dataset, the lowest miss rate (3.7%) of the proposed detector in the Reasonable subset is 0.2% smaller than that of the two-stage KGSNet presented in the upper part of Table 10.

Original: With pretrainingon Citypersons dataset, the miss rates on all three subsets of  Caltech dataset are reduced significantlyand reach the lowest compared to other pretrained detectors as shown in the bottom part of Table9.
Revised: With pretraining on the Citypersons dataset, the miss rates on all three subsets of the Caltech dataset are reduced significantly and reach the lowest compared to other pretrained detectors as shown in the bottom part of Table 10.

Original: For  Citypersons dataset, the proposed two detectors even outperform the competitive two-stagedetectors in  Reasonable and especially Heavy subset with  miss rate 36.7% which is 3% lower thanthat of the best of two-stage detectors.
Revised: For the Citypersons dataset, the proposed two detectors even outperform the competitive two-stage detectors in the Reasonable and especially Heavy subset with a miss rate of 38.6% which is 1.1% lower than that of the best of two-stage detectors.

Original: Overall,it can be observed that with the enhanced spatially adaptive and multi-scale features, thegap between single- and two- stage detectors in Reasonable and Heavy subsets on different detectorshas been narrowed.
Revised: Overall, with the enhanced spatially adaptive and multi-scale features, the gap between single- and two-stage detectors in Reasonable and Heavy subsets on different detectors has been narrowed.

Original: Surprisingly, the proposed single-stage method can produce better results thanaccurate two-stage methods on certain pedestrian dataset, such as  Citypersons dataset.
Revised: Surprisingly, the proposed single-stage method outperforms the accurate two-stage methods on certain pedestrian datasets, such as the Citypersons dataset.

Original: Therefore,our enhanced single-stage pedestrian detector not only predicts quickly, but also achieves accuracycomparable to complicated two-stage methods.
Revised: It should be noted that two-stage methods usually produce overall better accuracy than single-stage methods with the advantage of the region proposal network and refined bounding boxes and at the expense of inference time. Apart from accuracy, fast inference also matters for pedestrian detection in practical scenes. The combination of CSP and the proposed module has a simple structure, which is easy to perform and effective.

Original: Conclusion
Revised: On the contrary, the leading two-stage KGSNet, for example, takes the advantage of the additional proposal [Table] [Table] [Table]

Original: The paper proposes a single-stage anchor-free pedestrian detectorwith enhanced spatial and multiscale features.
Revised: generation network, the refined bounding boxes, the key-point detector and the super-resolution network. With these components, the inference speed of KGSNet is 5.9 FPS and 3.2 FPS (Titan X GPU, not including the time of using ALFNet to generate the candidate proposals) on Caltech and

Original: This is fulfilled by the combination of deformable self-attention mechanism and Center and Scale Prediction (CSP).
Revised: Citypersons datasets [18] while ours achieves 29.5 FPS and 6.8 FPS (RTX 3090 GPU) respectively.

Original: Comprehensive experiments over the architecture design in thedetection neck and comparison with the state-of-the-arts are performed. More than 40% parametersare reduced in the detection neck compared to CSP; however, this detector still achieves the best
Revised: Therefore, the enhanced single-stage pedestrian detector is cost-effective with fast inference and competitive accuracy compared to complicated two-stage methods

Original: [Table] [Table]
Revised: [Table] [Table]

Original: results in Reasonable and Heavy subsets among presented single-stage  detectors on both Caltech and
Revised: The paper proposes a module to enhance spatial and multi-scale features based on a single encoder of the deformable vision transformer to improve the detection accuracy of single-stage pedestrian detectors with fast inference.

Original:  Citypersons datasets.
Revised: This module is effective on commonly used single-stage structures, including (progressively refined) anchor-based and anchor-free cases. With the CSP detection head, more than 40% parameters are reduced in the detection neck compared to CSP; however, this combination still achieves the best results in Reasonable and Heavy subsets among presented single-stage detectors on both Caltech and Citypersons datasets.

Original: With pretraining, the miss rates for these subsets can be decreased to 2.6% and28.0% which are far better than other single-stage methods and comparable to two-stage methods.
Revised: Utilising pre-training, the miss rates for these subsets can be decreased to 2.6% and 28.0% respectively, which are far better than other single-stage methods and comparable to two-stage methods.

Original: Iteven outperforms SOTA two-stage detectors in  Heavy subset by 3% on Citypersons. The proposeddetector significantly improves the accuracy of the single-stage method with slightly decreasedinference speed.
Revised: The proposed method outperforms SOTA two-stage detectors in the Heavy subset by 1.1% on Citypersons with slightly decreased inference speed.

Original: These demonstrate that single-stage detectors can do better if spatially adaptive andmulti-scale features are jointly adopted,thus making them cost-effective and promising.
Revised: This demonstrates that single-stage detectors can be improved if spatially adaptive and multi-scale features are jointly adopted, making them cost-effective and promising.

Original: It shouldbe mentioned that, false positives are introduced if the attention points of a negative reference pointextend to the target areas.
Revised: It should be mentioned that false positives appear if the attention points of a negative reference point extend to the target areas.

Original: Although the performance has been improved with  current deformabletransformer, the generation of attention weights and sampling locations can be carefully designed tosuppress the false positives for further improvements.
Revised: Although the performance has been improved with the current module, the generation of attention weights and sampling locations can be carefully designed to suppress the false positives for further improvements.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] (c) Did you discuss any potential negative societal impacts of your work?
Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original: [No](d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [No] The code will be published online in the future.
Revised: [No] The code will be published online in the future.

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters, how they were chosen)?
Revised: g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e
Revised: [Yes] (c) Did you report error bars (e

Original: g., with respect to the random seed after running experiments multiple times)?
Revised: g., with respect to the random seed after running experiments multiple times)?

Original: [No] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: [No] (d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] 
Revised: [Yes] 4.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [TODO] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [TODO] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [Yes] The data are available online.
Revised: [Yes] The data are available online.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [No] 
Revised: [No] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and  the total amount
Revised: Memory Cost of Self-attention Weights in DETR : Figure 4 presents the structure of the encoder, decoder, and embedded Multi-Head Self-Attention (MHSA) layer. DETR has six encoder-decoder pairs. Each MHSA layer has a self-attention weight tensor produced by the multiplication of Query and Key as shown in Figure 4.

Original: spent on participant compensation?
Revised: The memory to store this tensor during training under different hyperparameter settings and optimization strategies are plotted in Figure 5. It shows that more attention heads, and especially large downsampling ratios, increase the memory cost significantly. Additionally, Adam and AdamW optimizers, which are commonly used to train vision transformers, take more memory compared to simple SGD. For pedestrian detection tasks, we normally choose head=8, downsampling ratio=0.25 and Adam optimizer, it takes more than 10 3 GB memory to store a single attention weight tensor even when the batch size is 1.

Revised: Thus, deformable DETR is used in our work to save memory resources.

Revised: Architecture : The overall architecture of the proposed enhancement module combined with the CSP detection head is illustrated in Figure 6.

Revised: the backbone neck

Revised: Training :

Revised: For anchor-free methods, the same ground truth and loss functions as CSP are utilized.

Revised: The overall loss consists of three parts as:

Original: [N/A]
Revised: [Equation]

Revised: 5) where the center heatmap loss L c , height map loss L h and offset loss L o are formulated as

Revised: [Equation]

Revised: [Equation]

Revised: (10) where p ij ∈ (0 , 1) denotes the predicted score in the center heatmap; s k and o k are the predicted height and offset at k -th pedestrian with the ground truth t k

Revised: and ot k ; M is a Gaussian mask map with variances proportional to the height and width of each one of the K pedestrians.

Revised: The two hyper-parameters β and γ are set as 4 and 2 empirically in accord with CSP settings.

Revised: Weights λ c , λ s and λ o for losses in equation Eq.

Revised: (5) are set as 0.01, 1 and 0.1 according to CSP.

Revised: For anchor-based methods the overall loss consists of two parts:

Revised: [Equation]

Revised: where focusing parameters α and γ are set as 0.25 and 2 [51].

Revised: S + and S − stand for positive and negative sets.

Revised: Smooth L1 loss is used for localization.

Revised: Visualization : Figure 7 presents some detected pedestrian samples under various circumstances including relatively large, small, and significantly inter- and intra- class occluded ones.

Revised: To better understand how the Multi-Scale Deformable Attention (MSDA) layer works, the sampling locations and their attention weights are presented in Figure 8.

Revised: The center of a single pedestrian corresponds to two reference points in feature maps at the first and third scale with size ( H/ 4 , W/ 4) and ( H/ 16 , W/ 16) .

Revised: Each reference point corresponds to learned sampling locations across 8 heads and 3 scales with learned attention weights which assign large values to important features.

Revised: The

Revised: MSDA layer accumulates the features at these locations with the weights to fuse the multi-scale and spatial adaptive information together.

Revised: According to Figure 8, the distribution patterns of sampling locations are similar in presented samples, however, the differences lie in the attention weights.

Revised: It is observed that high scales tend to focus on a sparser area with mild attention weights while low scales concentrate on the area close to the reference point with relatively sharp weights.

Revised: As the heat maps at a higher scale (third) show, important sampling locations of the relatively large pedestrian seem to be more reasonable as the head, shoulder and leg are highlighted.