Original: MsSVT: Mixed-scale Sparse Voxel Transformer for 3D Object Detection on Point Clouds
Revised: MsSVT: Mixed-scale Sparse Voxel Transformer for 3D Object Detection on Point Clouds

Original: (s)
Revised: Author(s)

Original: AffiliationAddress
Revised: AffiliationAddress

Original: Abstract 3D object detection from the LiDAR point cloud is fundamental to autonomousdriving.
Revised: Abstract 3D object detection from the LiDAR point cloud is fundamental to autonomousdriving.

Original: Large-scale outdoor scenes usually feature significant variance in instancescales, thus requiring features rich in long-range and fine-grained information tosupport accurate detection.
Revised: Large-scale outdoor scenes usually feature significant variance in instancescales, thus requiring features rich in long-range and fine-grained information tosupport accurate detection.

Original: Recent detectors leverage the power of window-basedtransformers to model long-range dependencies but tend to blur out fine-graineddetails.
Revised: Recent detectors leverage the power of window-basedtransformers to model long-range dependencies but, unfortunately, tend to blurout fine-grained details.

Original: To mitigate this gap, we present a novel Mixed-scale Sparse Voxel Transformer, named MsSVT, which can well capture both types of information simultaneously by the divide-and-conquer philosophy.
Revised: To mitigate this gap, we present a novel Mixed-scale Sparse Voxel Transformer, named MsSVT, which can well capture both types ofinformation simultaneously by the divide-and-conquer philosophy.

Original: Specifically, MsSVT explicitlydivides attention heads into multiple groups, each in charge of attending to information within a particular range.
Revised: Specifically, MsSVT explicitly divides attention heads into multiple groups, each in charge ofattending to information within a particular range.

Original: All groups’ output is merged to obtain the finalmixed-scale features.
Revised: The output from all the groupsis merged to obtain the final mixed-scale features.

Original: Moreover, we provide a novel chessboard sampling strategyto reduce the computational complexity of applying a window-based transformer in3D voxel space. To improve efficiency, we also implement the voxel sampling andgathering operations sparsely with a hash map.
Revised: Moreover, we provide a novelchessboard sampling strategy to reduce the computational complexity of applyinga window-based transformer in 3D voxel space and implement the voxel samplingand gathering operations sparsely with a hash map to improve efficiency.

Original: Endowed by the powerful capabilityand high efficiency of modeling mixed-scale information, our single-stage detectorbuilt on top of MsSVT surprisingly outperforms state-of-the-art two-stage detectorson Waymo.
Revised: Endowedby the powerful capability and high efficiency of modeling mixed-scale information,our single-stage detector built on top of MsSVT surprisingly outperforms state-ofthe-art two-stage detectors on Waymo.

Original: Code will soon be available.
Revised: Code will soon be available.

Original:  Introduction
Revised:  Introduction

Original: 3D object detection has received increasing attention due to its successful autonomous drivingapplications.
Revised: 3D object detection has received increasing attention due to its successful autonomous drivingapplications.

Original: Unlike 2D images with a regular structure of pixels, LiDAR point clouds are naturallyirregular and unordered.
Revised: Unlike 2D images with a regular structure of pixels, LiDAR point clouds are naturallyirregular and unordered.

Original: Hence directly applying CNN-like operations [11, 12] to them can bedifficult.
Revised: Hence directly applying CNN-like operations [10, 11] to them can bedifficult.

Original: To solve this, many researchers have rasterized point clouds into regular voxel grids [24]and employed 3D CNNs to extract high-dimensional voxel features.
Revised: To solve this, many researchers have rasterized point clouds into regular voxel grids [21]and employed 3D CNNs to extract high-dimensional voxel features.

Original: With the recent rise of visiontransformer (ViT) [42] on 2D images, some attempts have been made to generalize global or moreefficient window-based transformers to 3D voxels [23] or pillars [7].
Revised: With the recent rise of visiontransformer (ViT) [35] on 2D images, some attempts have been made to generalize global or moreefficient window-based transformers to 3D voxels [20] or pillars [6].

Original: These methods successfully seeklong-range context by utilizing transformers’ powerful abilities in modeling long-range information.
Revised: These methods successfully seeklong-range context by utilizing transformers’ powerful abilities in modeling long-range information.

Original: However, they ignore that blindly increasing receptive fields would easily blur fine-grained details,especially in sparse 3D space, crucial to accurate object recognition and localization.
Revised: However, they ignore that blindly increasing receptive fields would easily blur fine-grained details,especially in sparse 3D space, crucial to accurate object recognition and localization.

Original: Standard window-based transformers update the features of queries in a local window by attending tokeys from the same window.
Revised: Standard window-based transformers update the features of queries in a local window by attending tokeys from the same window.

Original: Hence simultaneously aggregating long-range context and fine-graineddetails require enlarging the window size to embrace local and distant voxels.
Revised: Hence simultaneously aggregating long-range context and fine-graineddetails require enlarging the window size to embrace local and distant voxels.

Original: Nevertheless, directlygathering all the voxels within the window as keys suffers a cubical growth of the computationalload with the window size.
Revised: Nevertheless, directlygathering all the voxels within the window as keys suffer a cubical growth in computational cost w.r.t.the window size.

Original: Some attempts  alleviate this by sampling only a certain number of keyvoxels [23].
Revised: Some attempts [20] alleviate this by sampling only a certain number of key voxels.

Original: While a trivial sampling strategy quickly leads tothe sparse sampling of local voxels(Fig. 1 b), thus bias mainly on long-range context.
Revised: While a trivial sampling strategy quickly leads to sparse sampling of local voxels (Fig. 1 b), thusbias mainly on long-range context.

Original: To mitigate the above, we try to set up multiple key windows of varying sizes centered on a query window and sample the same number of local and distant key voxelsseparately from the smaller and larger windows.
Revised: To mitigate the above, we try to set up multiple key windowsof varying sizes centered on a query window and sample the same number of local and distant key voxels from the smaller and larger windows separately.

Original: As a result, we can keep finergranularity in the local region to retain fine-grained details while collecting distant voxels roughly toenlarge the receptive field (Fig.
Revised: As a result, we can keep finer granularity inthe local region to retain fine-grained details while collecting distant voxels roughly to enlarge thereceptive field (Fig.

Original: 1 (c)).
Revised: 1 (c)).

Original: With the sampled voxels ready, the next question is how toeffectively attend to voxels from differentwindows andsimultaneously capture long-range context and fine-grained details.
Revised: With the sampled voxels ready, the next question is how to attend to voxels from different windowseffectively and capture long-range context and fine-grained details simultaneously.

Original: We argue that thedivide-and-conquer philosophy can satisfactorily resolve this issue.
Revised: We argue thatthe divide-and-conquer philosophy can satisfactorily resolve this issue.

Original: Specifically, inspired by therecent findings [42, 56, 26] that transformers learn different levels of self-attention by different heads,we propose a novel Mixed-scale Sparse Voxel Transformer (MsSVT), which explicitly divides thetransformer heads into multiple groups.
Revised: Specifically, inspired by therecent findings [35, 48, 23] that transformers learn different levels of self-attention by different heads,we propose a novel Mixed-scale Sparse Voxel Transformer (MsSVT), which explicitly divides thetransformer heads into multiple groups.

Original: Different head groups accept voxels sampled from windowsof different sizes, so they are each in charge of capturing information of a particular scale.
Revised: Different head groups accept voxels sampled from windowsof different sizes, such that they are each in charge of capturing information of a particular scale.

Original: Combiningthe outputs from all the head groups,we can capture mixed-scale information, i.e. , long-range context,and fine-grained details.
Revised: Bycombining the outputs from all the head groups, mixed-scale information, i.e. , long-range context,and fine-grained details can be well captured simultaneously.

Original: We also design a novel scale-aware relative position encoding strategy toadaptively adjust the position encoding used in each head group according to the range ofthe keys.
Revised: We also design a novel scale-awarerelative position encoding strategy to adaptively adjust the position encoding used in each headgroup according to the range of keys.

Original: We provide some resulting attention maps by two different head groups (Fig. 1 (d), (e)).
Revised: We provide some resulting attention maps by two differenthead groups (Fig. 1 (d), (e)).

Original: It is alsoworth mentioning that the mixed-scale attention enables information exchange across local windows,making MsSVT more compact by saving additional shift window operation  commonly required bywindow-based transformers [21, 7].
Revised: It is also worth mentioning that the mixed-scale attention meanwhileenables information exchange across local windows, which makes MsSVT more compact by savingadditional shift window operation that is commonly required by window-based transformers [18, 6].

Original: Moreover, to improve the efficiency of applying transformers in 3D voxel space, we strive to reducecomputational costs in two ways.
Revised: Moreover, to improve the efficiency of applying transformers in 3D voxel space, we strive to reducecomputational costs in two ways.

Original: First, we propose a novel chessboard sampling (CBS) strategyto reduce the number of query voxels that need to be sampled within the query window, to reducecomputational costs without losing information.
Revised: First, we propose a novel chessboard sampling (CBS) strategyto reduce the number of query voxels that need to be sampled within the query window, to reducecomputational costs without losing information.

Original: Specifically, we partition the query window intochess-like spaced, and termed as " × ", " ⃝ ", " △ ", " □ " positions separately.
Revised: Specifically, we partition the query window intochess-like spaced, and termed as " × ", " ⃝ ", " △ ", " □ " positions separately.

Original: During each attentionlayer, only one specific position of voxels is sampled and updated by serving as queries, and theupdates of the other voxels can be obtained by interpolation.
Revised: During each attentionlayer, only one specific position of voxels is sampled and updated by serving as queries, and theupdates of the other voxels can be obtained by interpolation.

Original: Four positions are selected in the circular.
Revised: Four positions are selected in the circular.

Original: Thus, we can update all the voxels without introducing deviation.
Revised: Thus, we can update all the voxels without introducing deviation.

Original: Second, we take advantage ofnon-empty voxels’ sparsity by performing mixed-scale window-based attention solely on non-emptysites in 3D space. Furthermore, we parallelize the search and feature gathering for non-empty voxelsusing hash mapping for further acceleration.
Revised: Second, we take advantage ofnon-empty voxels’ sparsity by performing mixed-scale window-based attention solely on non-emptysites in 3D space and parallelizing the search and feature gathering for non-empty voxels using hashmapping for further acceleration.

Original: We build a 3D detector by replacing the original sparse 3D CNN backbone in SECOND [47] with our
Revised: We build a 3D detector by replacing the original sparse 3D CNN backbone in SECOND [39] with our

Original: MsSVT and conduct extensive experiments on the large-scale Waymo open dataset [38].
Revised: MsSVT and conduct extensive experiments on the large-scale Waymo open dataset [32].

Original: Benefitingfrom the powerful capability of abstractingmixed-scale voxel features, our single-stage detectorbased on MsSVT surprisingly outperforms state-of-the-art two-stage detectors.
Revised: Benefitingfrom the powerful capability of abstracting voxel features of mixed-scales, our single-stage detectorbased on MsSVT surprisingly outperforms state-of-the-art two-stage detectors.

Original: We summarize ourcontributions as follows:
Revised: We summarize ourcontributions as follows:

Original: • We present a novel Mixed-scale Sparse Voxel Transformer (MsSVT), which simultaneouslyabstracts voxel features with  long-range context and fine-grained details.•
Revised: • We present a novel Mixed-scale Sparse Voxel Transformer (MsSVT), which abstracts voxelfeatures with both long-range context and fine-grained details simultaneously.•

Original: We design an efficient chessboard sampling strategy to vastly reduce the computational costof applying a voxel-based transformer in 3D space and sparsely implement all operations toimprove efficiency.• OurMsSVT-based single-stage detector  outperforms state-of-the-art two-stage detectors onWaymo.
Revised: We design an efficient chessboard sampling strategy to vastly reduce the computational costof applying a voxel-based transformer in 3D space and sparsely implement all operations toimprove efficiency.• Our single-stage detector based on MsSVT even outperforms state-of-the-art two-stagedetectors on Waymo.

Original:  Related work
Revised:  Related work

Original: Outdoor 3D detection on point clouds.
Revised: Outdoor 3D detection on point cloud.

Original: The mainstream outdoor 3D object detectors are basedon voxels [3, 48, 49, 37, 17, 14, 57, 47] or pillars [16, 32].
Revised: The mainstream outdoor 3D detection models are based onvoxel [2, 40, 41, 31, 15, 13, 49, 39] or pillar [14].

Original: VoxelNet [57] utilizes PointNet [27,32] to aggregate features within each voxel and then apply sparse 3D convolution to generatedetection results.
Revised: VoxelNet [49] utilizes a point net to aggregate thefeatures within each voxel and is further processed by sparse 3D convolution and 2D convolutionto generate detection results.

Original: SECOND [47] investigates  improved sparse convolution to further improve speed.
Revised: SECOND [39] investigates an improved sparse convolution method,which dramatically improves the model’s speed.

Original: Pointpillar [16] converts the point cloud into pillars such that 2D CNNs can be applied to trade offbetween accuracy and efficiency. Rapoport-Lavie et al . [28] introduce the Cylindrical Coordinatesto leverage the natural scanning pattern from LiDAR sensors.
Revised: Pointpillar [14] utilizes a pillar (voxel with infinitelength in the vertical direction) to quantify the point cloud, so it can fully apply the mature 2D

Original: Chen et al .
Revised: CNN, considering both efficiency and performance.

Original: [2] further operate onboth Bird Eye View (BEV) and Range View (RV) in the unified Hybrid-Cylindrical-Spherical (HCS)voxel representation. Reconfigurable Voxels [43] improves the local neighbors searching of eachvoxel using a random walk scheme.
Revised: Two-stage models [17, 29, 30, 26] aggregatethe original point cloud or voxel features on the proposal of the single-stage model, which is used toadjust the bounding box.

Original: Voxel-FPN [15] and Pillar-in-Pillar [41] adopt a multi-scalevoxelization strategy to rasterize the input point cloud into multi-sized voxels and incorporate FPN [20] to aggregate multi-resolution feature maps. Two-stage detectors [19, 35, 36, 31] refine thebounding boxes output by a single-stage detector by aggregating raw point clouds or voxel features,yielding state-of-the-art performance.
Revised: At present, they have state-of-the-art performance.

Original: Vision transformer.
Revised: Vision transformer.

Original: Transformer [42, 5] has recently achieved great success  in  computer vision [6,1 , 21, 18, 58, 50, 44].
Revised: Inspired by the great success of Transformer [35, 4] in NLP, some worksemploy it in the field of computer vision [5, 1, 18, 16, 50, 42, 36].

Original: Swin-transformer [21] restricts self-attention  to non-overlapping local windowswhile allowing cross-window connection to improve efficiency.
Revised: Swin-transformer [18] increasesefficiency by limiting self-attention computation to non-overlapping local windows while allowing forcross-window connection.

Original: SSA [29] divides  attention headsinto multiple groups to aggregate image features with different granularities.
Revised: SSA [24] divides the attention head into multiple groups and aggregatesimage features with different granularity, respectively, which has achieved excellent performance.

Original:  Guo et al.
Revised: In the field of point cloud, Guo et al.

Original: [9] and Zhao et al.
Revised: [8] and Zhao et al.

Original: [55] make the first step towards introducing the transformer for point cloud analysis.
Revised: [47] introduce transformer paradigm forpoint cloud classification and segmentation.

Original: Recently, many approaches [45, 54, 25, 10, 22] apply local self-attention  on voxels to learn richerfeature representation.
Revised: Recently, many methods [37, 46, 22, 9, 19] applylocal self-attention mechanism on voxels to learn richer feature representations.

Original: Our work extends the window-based attention on 3D voxelsby introducingscale-aware attention learning equipped with novel sampling strategies for the queries and the keys toimprove both accuracy and efficiency.
Revised: Our model extendswindow attention to 3D voxels and flexibly combines different window sizes to capture multi-scaleand multi-granularity features while shunting them to different attention heads in SSA.

Original: Voxel transformer for 3D detection.
Revised: Voxel transformer backbone in 3D detection.

Original: VoTr [23] introduces a voxel-based transformer backbonethat performs self-attention on sparse voxels with the local  and  the dilated attention mechanisms.
Revised: VoTr [20] introduces two kinds of sparse voxelattention, including local attention and extended attention. Each voxel serves as a query and attendswith the neighbor voxels.

Original: Our work improves VoTr by introducing the window-based attention and optimizing the sparseoperation.
Revised: We optimize the sparse operation and introduce window attention, whichsignificantly improves efficiency and performance.

Original: The recent SST [7] follows a single-stride design and the swin-transformer architecture,which performs well on small objects.
Revised: The recent SST [6] adopts a single-stride designand swin-transformer structure, which performs well on small objects.

Original: Nevertheless, SST is implemented based on pillars.
Revised: However, SST is implementedbased on a pillar.

Original: Thesingle window size is not conducive to capturing multi-scale features, resulting in unsatisfactoryperformance on Vehicle whensimultaneously detecting multiple categories.
Revised: The single window size is not conducive to capturing multi-scale features, resultingin the low performance of vehicles when multiple categories are detected simultaneously.

Original: In comparison, our MsSVT can capture mixed-scale information to boost detection of objects of various scales.
Revised: In contrast,our MsSVT can capture fine and global features and better performance on large and small objects.

Original:  Method
Revised:  Method

Original: This section first gives details of the MsSVT block, then its efficient sparse implementation, andfinally the 3D detector based on MsSVT.
Revised: [Figure] In this section, We first detail the pipeline of the Mixed-scale Sparse Voxel Transformer (MsSVT),then move to the novel chessboard sampling strategy and efficient sparse implementation, and build acomplete 3D detector based on the above components.

Original: 3.1 Mixed-scale Sparse Voxel Transformer
Revised: 3.1 Mixed-scale Sparse Voxel Transformer

Revised: [Figure]

Original: Fig.
Revised: Fig.

Original: 2 illustrates the overall architecture of the MsSVT block. We first gather the query and the keyvoxels via chessboard sampling  and balanced multi-window sampling, respectively.
Revised: 2 illustrates the overall framework of our proposed MsSVT, consisting of two core modules: abalanced multi-window sampling module and a scale-aware head attention module.

Original: The obtainedqueries and keys are then fed into multiple  head groups to  capture mixed-scale information throughscale-aware attention learning.
Revised: We first get thequery, key, and value voxels balanced sampled from multiple windows through the former and thenfeed them into our scale-aware head attention to simultaneously capture mixed-scale information.

Original: We further incorporate scale-aware relative position encoding to makebetter use of position information  in different head groups. [Figure]
Revised: We also add the scale-aware relative position encoding to the attention mechanism so that even thesame relative position information can play various roles in different scales.

Original: We gather thenon-empty voxels within the query window {+size (e.g. s 
 = (2 , 2 , 2) )+} andapply Chessboard Sampling (CBS) to sample the queries.{+key window sizes (e.g. s 
 =+}
Revised: In MsSVT block, we gather the voxels with the query window {+size (e.g. s 
 = (2 , 2 , 2) )+} and the {+key window sizes (e.g. s 
 =+}

Original: For the keys, we gather the non-empty voxels from the key windows of different sizes separately, and {+s 
 = (4 , 4 , 4) ), respectively. Then, we apply CBS to obtain sampled queries, while employ FPS to+} get multiple sets of keys through Balanced Multi-window Sampling, with each set representing information of a specific scale.
Revised: (2 , 2 , 2) and {+s 
 = (4 , 4 , 4) ), respectively. Then, we apply CBS to obtain sampled queries, while employ FPS to+} get sampled keys and values of same number (e.g. N K = 3 ) among all key windows.

Original: Keys from windows of different sizes are assigned to differenthead groups to perform scale-aware attention learning, thus simultaneously capturing both long-range context and fine-grained details.
Revised: Finally, we feed the same query and the different keys into different attention head groups, respectively.

Original: 3.1.1 Balanced Multi-window Sampling
Revised: 3.1.1 Balanced Multi-window Sampling

Original: Let { s k | s k ∈ Z 3 } Mk =0 denote a series of window sizes, where s 0 is the size of the query windowand s 1 ,...,M are the sizes of M successively larger key windows.
Revised: Given a series of window sizes { s k | s k ∈ Z 3 } Mk =0 , where s 0 denotes the query window size ands 1 ,...,M denote M different key window sizes from small to large.

Original: Let V
Revised: Let V

Original: = { v i | v i =
Revised: = { v i | v i =

Original: ( x i , f i ) } |V| i =1 bethe input voxel set, with xyz coordinates x i ∈ Z 3 and feature vector f i ∈ R C for voxel
Revised: ( x i , f i ) } |V| i =

Original: i .
Revised: 1be

Original:  We firstpartition the voxel set into non-overlapping 3D windowseach of size s 0, and find the non-emptyones as query windows with their centers denoted by { c i
Revised: the input voxel set, where x i ∈ Z 3 denotes voxel coordinates and f i ∈ R C denotes voxel features. We first partition the voxel set into non-overlap 3D windows s 0 by finding non-empty window centers{ c i

Original: | c i ∈
Revised: | c i ∈

Original: Z 3 } Li =0 , where L is the total numberof query windows.
Revised: Z 3 } Li =0 , where L is the number of non-empty windows.

Original: To get query  voxels V c i , s 0for the query window centered on c i, one can simplygather all the non-empty voxels within the window  as the queries.
Revised: To get query voxels, we can simplygather all the non-empty voxels V c i , s 0 centered on c i within query window s 0 to guarantee that everyvoxel can serve as a query and be updated after one attention layer.

Original: While keeping efficiency in mind,we present a novel chessboard samplingstrategy, which will be detailed in Section 3.1.2.
Revised: We also prsent another novelchessboard sampling strategy for the query voxel sampling which will be discussed in Section 3.2.

Original:  As for the key voxels, instead of sampling within a single large window at once as previous methods [23], which inevitably biases oneither local or distant voxels, we simultaneously search for theneighbors for each center ci within multiple key windows of different sizes.
Revised: For easy understanding, readers can regrard all the non-empty voxels in each query window as thequeries. As for the key voxels, instead of sampling in a single large window at once like previous methods [20],which inevitably bias on local or distant voxels, we simultaneously research the neighboors for eachcenter c

Original: For the key window ofsize s k , we gather no more than N P non-empty voxelsV c
Revised: i with different window size s k and gather no more than m non-empty voxels

Original:  i , s k
Revised: V c i , s k

Original: = { v j | − s k < x j − c
Revised: ={ v j |− s k < x j − c

Original: i < s k } N P j =1 ,where N P is a pre-set number.
Revised: i < s k } mj =1 , where m is a pre-defined number for all key windows.

Original: Furthermore, to reduce computational cost and keep balanced sampling, we adoptthe farthest point sampling (FPS) algorithm to uniformly sample N K voxels from V c i , s k to obtain the final key voxels
Revised: Furthermore, toreduce the computational cost and keep balanced sampling, we adopt farthest point sampling (FPS) touniformly sample N K voxels for each window s k to obatain the final key voxels

Original: V fps c i , s k , k = 1 , ..., M at different scales, where N K is a pre-setmaximum number of sampled voxels.
Revised: V fps c i , s k , k = 1 , ..., Mat different scales, where N K is a predefined maximum number of sampled voxels.

Original: Benefitting from the multi-window strategy equipped withuniform sampling by FPS, we can achieve balanced sampling of key voxels at various scales, whichis crucial to capturing mixed-scale information.
Revised: Benefit from themulti-window sampling combined with the uniform FPS operation, we can achieve more balancedkey voxels sampling from various scales which is crucial to capturing mixed-scale information.

Original: 3.1.2 Chessboard Sampling

Original: The prerequisite for us to sample the keys is that we do not have to preserve all the key voxels andonly need to select some representative ones to reduce computational redundancy.

Original: While thingsare different for the queries.

Original: On the one hand, we should retain and update every query voxel afteran attention layer; otherwise, the information could be irreversibly lost.

Original: But on the other hand,the computational cost and memory footprint grow dramatically with the window size, makingimplementation impractical if the queries are not reduced.

Original: Since the positions of non-empty voxels remain unchanged after a series of attention blocks, an effective solution is to sample a subset of query voxelseach time for feature updating, and then use thesampled voxels to update those unsampled.

Original: Motivated by the above, we present Chessboard Sampling (CBS) as illustrated in Fig.

Original: Each voxel inthe query window is marked with " × ", " ⃝ ", " △ ",or " □ ", in an evenly spaced pattern.

Original: In each MsSVTblock, we sample queries from the non-empty voxelswith one specific type of the marks, and update thequeries’ features through attention learning.

Original: Thenfor each unsampled non-empty voxel, we update itsfeature by finding the K-nearest query voxels (bydefault K = 3 ) and by linearly interpolating theirupdated features.

Original: The four masks are used in the circular to sample query voxels in stacked blocks.

Original: This way, we retain the original structure and cover all voxels as comprehensively as possible.

Original: Notethat we can apply the interval sampling on any of the x , y , z axes to achieve a sampling rate of 1/2,1/4, or 1/8.

Original: Generally, we apply it in the horizontal x - y plane (as illustrated in Fig. 3).

Original: 3.1.3 Scale-aware Head Attention
Revised: 3.1.2 Scale-aware Head Attention

Original: Given the query voxels V c
Revised: Once receiving the query voxels V c

Original: i , s 

Revised: i , s

Original: = ( X 0 , F 0 )with voxel coordinates X 0 ∈ Z N Q × 3 and featurevectors F 0 ∈ R N Q × C , and the multi-scale key voxels
Revised: = ( X 0 , F 0 ) and sampled key voxels

Original: V fps c i , s k =
Revised: V fps c i , s k =(

Original: ( X k , F k ) , k = 1 , ..., M with voxelcoordinates X k ∈ 
Revised: X k , F k ) , k = 1 , ..., M at different scales, where X 0 ∈ Z N Q × 3 , F 0 ∈ R N Q × C denote queryvoxel coordinates and features, X k ∈ Z N K × 3 ,

Original: Z N K × 3 and feature vectors F k ∈ R N K × C , we first get queries Q , keys { K k } Mk =1and values { V k } Mk =1 as
Revised: F k ∈ R N K × C denote key voxel coordinates andfeatures, we first get queries Q , keys { K k } Mk =1 and values { V k } Mk =1 as

Original: [Equation]
Revised: [Equation]

Original: where W Q , W K , W V ∈ R C
Revised: where W Q , W K , W V ∈ R C

Original: ×
Revised: ×

Original: C are linear projections.
Revised: C are linear projection.

Original: To achieve scare-aware attention learning, wedivide multiple attention heads into M  groups and assignthe keys from windows of differentsizes todifferent head groups.
Revised: To make our attention module aware todifferent scales, we divide the attention heads into M multiple groups and assign keys from differentwindows to different head groups so that the queries Q should only attend K k in the k -th head group.

Original: Accordingly, we also split the feature channels of the queries Q into M groups. The k -th channel group of Q , denoted by Q k =
Revised: We also split the feature channel into M parts and explicitly make each part represent the informationof a specific scale. We denote Q k =

Original: Q
Revised: Q

Original: [: , ( k − 1) × C/M : k × C/M ] , k = 1 , ..., M ,is fed into the k -th head group. As a result, each head group is responsible for learning to attendinformation at a specific scale.
Revised: [: , ( k − 1) × C/M : k × C/M ] , k = 1 , ..., M as the k -thscale features for queries and the same is true for K and V .

Original: The attended feature for the k -th head group is
Revised: So for the k -th head group, we can getthe attended feature as:

Original: [Equation]
Revised: [Equation]

Original: where MHA ( · ) denotes multi-head-group attention and RPE ( · ) represents the newly proposedrelative position encoding which will be detailed in Section 3.1.4.
Revised: where MHA ( · ) denotes multi-head-group attention and RPE ( · ) stands for our relative positionencoding which will be discussed in Section 3.1.3.

Original: Each window size corresponds toa head group with one or more attention heads.
Revised: Each window size corresponds to a "headgroup" that includes one or more attention heads.

Original: We concatenate the output from all heads groups{ ˜ Y k } Mk =1 to ˜ Y ∈ R L × N Q × C , which is further fed into a feed-forward network (FFN) implementedby multi-layer perceptron (MLP) to obtain thefinal mixed-scale feature Y :
Revised: Finally, we concatenate the output of all headsgroup { ˜ Y k } Mk =1 to ˜ Y ∈ R L × N Q × C , and feed into FFN, where L denotes the total number ofnon-empty query windows. After that, we obtain the mixed-scale features Y :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where LN ( · ) represents layer normalization.

Original: 3.1.4 Scale-aware Relative Position Encoding
Revised: 3.1.3 Scale-aware Relative Position Encoding

Original: Relative position encoding is necessary for transformer-based networks because fine-grained positioninformation may be lost in high-level features with the deepening of the network.
Revised: The 3D point cloud feature generally contains the original coordinates information, which voxelswill inherit. However, the fine-grained location information may be blurred with the deepening ofthe network, so the relative position encoding is necessary.

Original: To make better useof position information to facilitate multi-scale feature learning in our case, we adopt a scale-awareadaptive relative position encoding strategy inspired by [30, 46, 52], which can generate the positionalbias dynamically with scales for different head groups.
Revised: Furthermore, since MsSVT can extractmulti-scale features, the position-coding should differ at different scales, even for the same relativeposition. In light of these, we adopt an adaptive, scale-aware relative position encoding strategyinspired by [25, 38, 44].

Original: Specifically,  we establish a learnable embedding table T k ∈ R D × R forthe k -th head group accordingto the size of the largest key window [21], where R is the number of possible relative position pairsand D is the feature dimension.
Revised: Specifically, according to the size of the maximum key window, we establish a learnable embeddingtable T k ∈ R D × R for k -th head group like [18], where R is the number of possible relativeposition pairs and D is the feature dimension.

Original: We get the  relative positional biasfor the queries as:
Revised: Then we can get the query relative position bias

Revised: B kQ ∈ R L × N Q × N K as:

Original: [Equation]
Revised: [Equation]

Original: where I
Revised: where I

Original: k ∈
Revised: k ∈

Original: Z L × N Q × N K represents the table indices corresponding to the actual relative positionbetween the queries andthe keys, G ( · ) is the operation of gathering features according to the indices.
Revised: Z L × N Q × N K means the table indices corresponding to the actual relative position of thequeries and keys, G ( · ) means the operation of gather features via indices.

Original: Similarly, we can ge therelative positional bias B Kk ∈ R L × N Q × N K for the keys.
Revised: We get the B kK in the sameway.

Original: Finally, the biasesact directly on the attention weights, and Eq.
Revised: Finally, the bias acts directly on the attention, and Eq.

Original: (2) can be rewritten as:
Revised: (2) can be rewrited as:

Original: [Equation]
Revised: [Equation]

Original: where σ ( · ) represents softmax.
Revised: where σ ( · ) represents softmax.

Original: This way, position embeddings can be adjusted adaptively accordingto different scales to guide the scale-ware head attention better.
Revised: In this way, position embeddings can be adjusted adaptively accordingto different scales to guide our scale-ware head attention better.

Original: 3.2 Sparse Implementation
Revised: 3.2 Chessboard Sampling

Revised: In Section 3.1, the prerequisite for us to sample key voxels to a specific number is that we do notneed to preserve all the key voxels and only need to select some representative ones.

Revised: However, thingsare different for queries.

Revised: On the one hand, we should maintain and update every query voxel after anattention layer; otherwise, the information could be irreversibly lost, and sampling queries seemsimpossible.

Revised: On the other hand, the computational cost grows cubically with window size, and itwould bring unacceptable memory costs if we can not reduce the number of queries.

Revised: However, since the non-empty voxel positions are not changed during a sequential of attentionblocks, an effective way is to iteratively sample a subset of voxels from all the queries andthen use the sampled voxels to update those unsampled.

Revised: In the light of this, we present Chessboard Sampling (CBS) as shown in Fig.

Revised: In an attention module, we select one " × ", " ⃝ "," △ ", " □ " as the sampling location.

Revised: The sampled non-empty voxels are sent to the subsequentnetwork for updating.

Revised: The updates of other non-empty voxels can be obtained by linear interpolation with their top k nearest neighbors (default by 3).

Revised: Four positions will be selected inturn in stacked modules.

Revised: In this way, we retain the original structure and cover all voxels ascomprehensively as possible.

Revised: Note that we can apply the interval to any axis to obtain the sampling rate of 1/2, 1/4, or 1/8.

Revised: Generally, we apply it to the horizontal xy plane (like Fig. 3).

Revised: 3.3 Sparse Implementation

Original: To leverage the natural sparsity of point clouds andfurther improve efficiency, we sparsely implementall our window center searching, window gathering, and balanced window sampling into CUDAoperations.
Revised: [Figure] We implement all our window center searching,window gathering, and balanced window samplingsparsely in cuda operations to leverage the natural sparsity of point clouds and improve efficiency.

Original: These operations are mainly based on a hash map that establishes the mapping fromcoordinate space to voxel index [23].
Revised: These operations are mainly based on a hash mapwhich establishes the mapping from coordinatespace to voxel index as in [20].

Original: Taking the window gathering operation as an example, wequery each possible position w.r.t. the given center within the window, and retrieve the correspondingfeatures if the position is a valid key in the pre-built hash map.
Revised: For example, forwindow gathering, we query each possible positionwrt the given center within the window and retrievethe feature if the position is a valid key in our prebuilt hash map.

Original: More details are available in thesupplementary materials.
Revised: More details are available in the supplementary materials.

Original: 3.3 Detector Establishment
Revised: 3.4 Detector Establishment

Original: We build our 3D backbone by stacking multiple MsSVT blocks, as shown in Fig.
Revised: As shown in Fig 2, we build our 3D backbone by stacking Mixed-scale Sparse Voxel Transformer(MsSVT) blocks.

Original: Noted thatweset both the query andthe key window size in the lastMsSVT block as (1 , 1 , ∞ )  to compress the 3Dvoxels intoa 2D feature map, where the query is the average  voxel features within the pillar window.
Revised: It is worth noting that both the query and key window size in the last block of MsSVT are set to (1 , 1 , ∞ ) so as to compressing the 3D voxels into 2D feature map, where the queryis the average of all the voxel features within the pillar window.

Original: We replace the 3D backbone  in SECOND [47] with MsSVT and keep the other network componentsunchanged. We discard the down sampling process because MsSVT can already capture features atdifferent scales.
Revised: We replace the 3D backbone withour MsSVT in SECOND [39] and keep the other architecture the same except for abandoning thedown-sampling process because MsSVT can already capture features at different scales.

Original: The input point cloud is first converted into regular voxels and fed into our MsSVTbackbone to get  mixed-scale voxel features.
Revised: The inputpoint cloud is first converted into regular voxels and fed into our MsSVT to get the mixed-scale voxelfeatures.

Original: The features arethen compressed vertically and sent tothe subsequent 2D RPN and detection head to get detection results.
Revised: Next, features are compressed vertically and sent to the following 2D RPN and detectionhead, outputting detection results.

Original:  Experiments
Revised:  Experiments

Original: In this section, wefirst provide architectural details of MsSVT, then compare our model with recentstate-of-the-art detectors on Waymo Open [38] and KITTI [8] datasets.
Revised: In this section, we provide architectural details of MsSVT and compare our model with recentstate-of-the-art detectors on Waymo Open Dataset [32] and KITTI [7].

Original: Thorough ablation studiesand in-depth analysis are further provided to validate our design choices.
Revised: Thorough ablation studiesand in-depth analysis are further provided to validate our design choices.

Original: 4.1 Architectural Details
Revised: 4.1 Architectural Details

Original: MsSVT comprises four regular MsSVT  blocks with a query window size of  (3 , 3 , 5)  and  key windowsizes of (3 , 3 , 5) and (7 , 7 , 7) , followed by a specialized MsSVT block where the windows are setas 1 × 1 pillar as mentioned in 3.3.
Revised: MsSVT includes four regular MsSVT attention blocks and an additional special one where thewindows are set as 1 × 1 pillars as mentioned in 3.4. The query window size of the normal MsSVTattention blocks is set to (3 , 3 , 5) , and the key windows are set to (3 , 3 , 5) and (7 , 7 , 7) .

Original: We divide 8 attention heads into 2  head groups.
Revised: We dividetotal 8 attention heads into 2 groups in the scale-aware head attention layer.

Original: The samplingrate of the chessboard sampling is 1/4 and the  maximum number ofsampled keys N K is 32 .
Revised: The queries are obtainedvia alternating chessboard sampling with 0.25 sampling ratio, and the keys are balanced sampledwith a maximum number of 32 in each window.

Original: Weuse the center head [53] to generate single-stage  bounding boxes.
Revised: We use center head [45] as the detection head togenerate a single-stage detection bounding box.

Original: In addition, we also provide atwo-stage version with CT3D [31].
Revised: In addition, we also provide a two-stage detectorwith CT3D [26].

Original: More details can be found in OpenPCDet [40] since we conductall experiments with this toolbox.
Revised: Please refer to OpenPCDet [34] for more details since we conduct our experimentswith this toolbox.

Original: 4.2 Results on Waymo
Revised: 4.2 Results on Waymo

Original: Setups.
Revised: Setup.

Original: We first evaluate our model on large-scale Waymo Open Dataset [38].
Revised: We first evaluate our model on large-scale Waymo Open Dataset [32].

Original: The input is asingle-frame point cloud with 150m × 150m detection range.
Revised: The input is a singleframe point cloud with 150m × 150m detection range.

Original: We set the detection range in thehorizontal and the vertical direction as [ − 75 .
Revised: We set the detection range in the horizontaland the vertical direction as [ − 75 .

Original: 2m , 75 . 2m] and [ − 2 .
Revised: 2m , 75 . 2m] and [ − 2 .

Original: 0m , 4 . 0m] , respectively.
Revised: 0m , 4 . 0m] , respectively.

Original: The voxelsize is (0 . 4m , 0 .
Revised: The voxel size is(0 . 4m , 0 .

Original: 4m , 0 .
Revised: 4m , 0 .

Original: 6m) .
Revised: 6m) .

Original: We follow the same training strategy as in [23].
Revised: We follow the same training strategy as in [20].

Original: Specifically, we train ourmodel for 80 epochs on 20% Waymo data using the Adam optimizer [13].
Revised: Specifically, we train the modelon Tesla-V100 using the Adam [12] optimizer for 80 epochs on 20% Waymo data.

Original: We apply the cyclic decayscheme [47], by which the learning rate is increased from 1 e- 4 to 1 e- 3 during the first 40% epochsand further decreased to 1 e- 5 in the remaining epochs.
Revised: We apply thecyclic decay scheme [39], by which the learning rate is increased from 1 e − 4 to 1 e − 3 during the first40% epochs and further decreased to 1e-5 in the remaining epochs.

Original: We also report the results oftraining for 30epochs  on 100%Waymo data using the same optimizer andlearning rate scheme.
Revised: We also provide the results of 30epochs training on 100% data with the same optimizer and scheme.

Original: The evaluationmetric is the 3D mean Average Precision (
Revised: The evaluation metric is the 3Dmean Average Precision (mAP) for difficulty levels of LEVEL 1 and LEVEL 2.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Main results.
Revised: Main results.

Original: We compare our model with state-of-the-art priors in Table 1 and Table 2 .
Revised: Our main results are shown in Table 1 and Table 2 .

Original: It is worthnoting that our model performs simultaneous detection of three object categories, which is morechallenging than detecting a single category.
Revised: It is worth noting that ourmodel detects three categories simultaneously, which is more challenging than detecting a singlecategory.

Original: Using 20%training data, our single-stage MsSVT-SSsignificantly surpasses other single-stage counterparts, even though some of them are  trained speciallyfor one specific category.
Revised: On 20% data, MsSVT-SS significantly outperforms other single-stage detectors, eventhough some methods are only trained on a single category.

Original: Noticeably, MsSVT-SS performs on par with the state-of-the-art two-stage  PV-RCNN++ [36] on Vehicle , and even better ( 0 . 8 - 1 . 5 mAP higher) on Pedestrian and Cyclist .
Revised: The single-stage results can almost matchthe state-of-the-art two-stage methods PV-RCNN++ [30] on the vehicle and even outperform 0.8-1.5mAP on pedestrians and cyclists.

Original: Inaddition, our two-stage MsSVT-TS performs the best on all categories, and significantly outperformsprevious best-performing PV-RCNN++ by 2 . 9 and 3 . 2 mAP on Pedestrian and Cyclist , respectively.
Revised: MsSVT-TS achieves the best performance in all categories and hasabsolute superiority over pedestrians and cyclists.

Original: Similarly, when using 100% data,both our single-stage MsSVT-SS and two-stage MsSVT-TS achievethe best results compared to their corresponding counterparts.
Revised: On 100% data, our model also achieves state-of-the-art performance.

Original: Noticeably, our MsSVT-SS surpassesthe recent transformer-based detector SST [7] dedicated to small object detection by large margins of4 . 3 and 4 . 5 mAP on Vehicle and Cyclist , respectively.
Revised: Compared with the recenttransformer work SST [6], which is adept at the small object detection, MsSVT not only takes intoaccount all categories but also has better performance on small objects.

Original: This clear evidences the superiority of our MsSVT in capturing mixed-scale information over conventional transformer designs.
Revised: This result proves MsSVT’ssuperiority over general transformers in capturing multi-scale features.

Original: Visualization.
Revised: Visualization.

Original: We provide some qualitative results in Fig. 
Revised: Fig. 4 shows some visualization of our results.

Original: Our MsSVT can still predict accuratebounding boxes in the scenes beyond the range of 50m where the points become extremely sparse(Fig. 4 (a)).
Revised: As shown in Fig. 4 (a), in the rangeof > 50 m that the points are very sparse, the bounding boxes are still accurate, which means that

Original: This evidences that MsSVT successfully captures valuable contextual information in theabsence of fine-grained details for long-range object detection.
Revised: MsSVT can rely on context information in the absence of fine-grained information. As shown in

Original: In addition, our model also performswell in  complex scenes in the presence ofdense objects with large scale variations (Fig. 4 (b). Itshould be owed to the flexibility and robustness of MsSVT.
Revised: Fig. 4 (b), even in a complex scene with many different categories of objects at the same time, themodel still perform well, which should be owed to the flexibility and robustness of MsSVT.

Original: Analysis.
Revised: Analysis.

Original: We further visualize some attention maps which indicate what have been focused on byour MsSVT to understand its behavior better.
Revised: We visualized some attention weights.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 5, thehead group with a smaller keywindow focuses on the local foreground information, while thehead group with a larger key windowattends more to longer-range context.
Revised: 5, the smaller window modelfocuses on the local foreground information, while the larger window model tends to focus on thecontext information.

Original: As a result, the two head groups complement each other, wellcapturing mixed-scale information to support the detection of objects of various scales.
Revised: The two windows can complement each other. This proves the ability of MsSVTto capture mixed-scale features.

Original: [Table]
Revised: [Table]

Original: 4.3 Results on KITTI
Revised: 4.3 Results on KITTI

Original: Setups.
Revised: Setup.

Original: We also evaluate our model on KITTI [8].
Revised: We further evaluate our model on KITTI [7].

Original: Given an input point cloud, we reserve thepoints within the range of [0m , 70 . 4m] , [ − 40 .
Revised: Given an input point cloud, we reserve thepoints within the range of [0m , 70 . 4m] , [ − 40 .

Original: 0m , 40 .
Revised: 0m , 40 .

Original: 0m] and [ − 3 . 0m , 1 . 0m] on the x , y , z axes,respectively.
Revised: 0m] and [ − 3 . 0m , 1 . 0m] in the X, Y, and Zdirections, respectively.

Original: The voxel size is set to (0 .
Revised: The voxel size is set to (0 .

Original: 32m , 0 . 32m , 0 .
Revised: 32m , 0 . 32m , 0 .

Original: 4m)
Revised: 4m)

Original: and other settings remain the same asin the experiments on Waymo.
Revised: and other settings are sameas 4.2.

Original: We train the model for 100 epochs with the Adam optimizer.
Revised: We train the model for 100 epochs with the Adam optimizer.

Original: Thelearning rate is 0 . 003 , decayed by the cyclic scheme [47].
Revised: The learning rate is 0 . 003 ,decayed by the cyclic scheme [39].

Original: The evaluation metric is 3D mAP for threedifficulty levels (easy, moderate, and hard).
Revised: The evaluation metric is 3D mAP for three difficulty levels (easy,moderate, and hard).

Original: Main results.
Revised: Main results.

Original: Table 3 depictsthat our model achieves competitive performance on all  threecategories.
Revised: Table 3 depicts our MsSVT achieves competitive performance on all the threecategories.

Original: Specifically, on Car ,our single-stage MsSVT-SS surpasses the superior VoTr [23] by0 . 5 mAP.
Revised: Specifically, on Car , MsSVT-SS beats the superior VoTr [20] by 0 . 5 mAP.

Original: Meanwhile, on Pedestrian and Cyclist , MsSVT-SS even outperforms some preeminenttwo-stage detectors.
Revised: Meanwhile, on Pedestrian and Cyclist , MsSVT-SS even outperforms some preeminent two-stage detectors.

Original: Moreover, our two-stage MsSVT-TS further increases the lead.
Revised: Moreover,our two-stage MsSVT-TS further increases the lead.

Original: These resultsdemonstrate that our MsSVT can be well generalized to various datasets.
Revised: The results on KITTI demonstrate that our MsSVT has a good generalization ability to adapt to different datasets.

Original: 4.4 Ablation Study
Revised: 4.4 Ablation Study

Revised: We perform ablation studies to validate our design choices.

Original: Allablation models are trained for 12 epochs on 20% Waymo data.
Revised: All models are trained for 12 epochs on20% Waymo data.

Original: More implementation details andablations on hyperparameters are available in supplementary materials.
Revised: More implementation details and ablations on hyperparameters are available insupplementary materials.

Original: [Table]
Revised: [Table]

Revised: [Table]

Original:  Balanced multi-window sampling.
Revised: [Table] Balanced [Table] window sampling

Revised: [Table]

Original: We first validate our balanced multiwindow sampling strategy in Table 4.
Revised: We first validate our balanced multiwindow sampling strategy in Table 4.

Original: The base model listed in the first rowemploys a 3D version of standardwindow-based attention [21] withwindow size (3 , 3 , 5) and without shiftwindow scheme, adopts dilated keysampling [23], and gathers all non-emptyvoxels within the window as queries.
Revised: The base model, listed in the firstrow, employs 3D version of standardwindow-based attention [18] (withwindow size (3 , 3 , 5) and without shiftwindow scheme), adopts dilated keysampling strategy [20], and gathers allnon-empty voxels within the windowas queries.

Original: We build a model variant by simplyreplacing the dilated key sampling withour proposed balanced multi-window sampling.
Revised: We build a model variantby simply replacing the dilated keysampling in the base model with our balanced multi-window sampling strategy.

Original: Results in the second row demonstrate that samplingkey voxels frommultiple windows of different sizes brings noticeable performance gains, i.e. ,71 .[-
-]
Revised: The results inthe second row shows sampling key voxels from windows of multiple sizes brings noticeableperformance gains, i.e. , 71 . 66 → 73 .[-
-]

Original: → 73 . 44 on Pedestrian and 63 .
Revised: 44 on Pedestrian and 63 .

Original: 31 → 66 . 88 on Cyclist .
Revised: 31 → 66 . 88 on Cyclist .

Original: Scale-aware head attention.
Revised: Scale-aware head attention.

Original: We also validate the effectiveness of ourproposed scale-aware headattention in Table 4.
Revised: Table 4 also validates the effectiveness of our scale-aware headattention module.

Original: By comparing theresults in the second and the third rows, one can find thatenabling multiple head groups  to capture information at different scales clearly boots the performance.
Revised: By comparing the second and the third rows, we find that making different attentionhead groups specialized in different scales helps to capture mixed-scale features and thus bringssignificant performance gains. We further visualize some attention maps by different head groups in Fig.

Revised: One can observe that the query point pays more attention to fine-grained details in a smallwindow and is more interested in long-range contextual information in a large window.

Original: Scale-aware relative position encoding.
Revised: Scale-aware relative position encoding.

Original: Table 4 showsthat incorporating scale-aware relativeposition encoding (listed in the last row) further improves the performance compared to the modelvariant that adopts scale-agnostic position encoding. It well supports our design motivation that theposition encoding should vary with different scales.
Revised: Table 4 shows compared with the model variant thatadopts scale-agnostic position encoding (listed in the second row), our scale-aware relative positionencoding gives better performance, which well supports our design motivation that the importance ofpositional information varies with different scales.

Original: [Table]
Revised: [Table]

Original: Chessboard sampling.
Revised: Chessboard sampling.

Original: Table 5 reportsthe performance of the model variants thatapply chessboard sampling with different sampling rates and of the model without performing any sampling.
Revised: Table 5 reportsthe performance of the model variants using chessboard sampling with differentsampling ratios, and of the model without performing any sampling.

Original: We measure the latency on Tesla-V100 GPU and

Original: Unbuntu-16.04, Python 3.7, Cuda-10.2,and Pytorch-1.8.

Original: Our model is not sensitive to varying sampling rates.
Revised: One can seethat our model is not sensitive to the variation of sampling ratio.

Original: Incorporating chessboard sampling with a samplingrate of 1/4 results in only negligible performance degradation compared to the model variant withoutsampling, yet largely improves the computational efficiency by a significant reduction  in memoryfootprint (up to 33% decrease) and in latency (up to 28% decrease).
Revised: The model variant with sampling ratio 1/4 performs evenon par with the model without sampling,while bringing a significant reduction (upto 6.1G) in memory footprint.

Original:  Conclusion
Revised:  Conclusion

Original: This paper presents MsSVT, a novel  sparse voxel transformer that is capable of capturing mixed-scaleinformation to boost 3D object detection.
Revised: In this paper, we propose MsSVT, a novel 3D sparse voxel transformer backbone for 3D detectionto capture mixed-scale information simultaneously.

Original: We begin with sampling the queries and the keys atdifferent scales through the novel chessboard and the balanced multi-window sampling strategy,respectively.
Revised: We first utilize the novel chessboard samplingand balanced multi-window sampling to obtain queries and a set of keys with different scales.

Original: Then, we explicitly divide attention heads into multiple scale-aware groups, each incharge of capturing information at a specific scale.
Revised: Then MsSVT explicitly divides attention heads into multiple scale-aware groups and each in charge ofcapturing information at a specific scale.

Original: Extensive experiments have shown that our MsSVT is superior in detecting objects of various scales and granularities thanks to the learndmixed-scale information.
Revised: Equipped with the scale-aware relative position encoding, MsSVT finally gets the mixed-scale features which is crucial for accurate detection.

Original: Limitations.
Revised: Limitations.

Original: MsSVT can capture mixed-scale information from multiple local windows and achievespromising performance. Yet, the sizes of the windows need to be pre-set manually. We will explorethe adaptive-window version of MsSVT in future work.
Revised: MsSVT considers both fine-grained details and long-range contextual information indifferent windows and achieves promising performance, but still needs to set hand-crafted windowsizes, we will explore adaptive window version of MsSVT in the future.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes](b) Did you describe the limitations of your work?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 5.(c)

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [No] Our work is only for academic research purpose. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [No] Our workis only for academic research purpose.(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]
Revised: [Yes]2.

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A](b)

Original: (b) Did you include complete proofs of all theoretical results?
Revised: Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]3.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] Our implemen- tation details are posted on Section 4.1 and the supplemental materials.
Revised: [Yes] Our implementation details are posted on Section 4.1 and the supplemental materials.

Original: The data arepublic datasets and our code will be available.
Revised: The data arepublic datasets and our code will be available.(b)

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
Revised: Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] See Section 4.1 and setup in 4.2 and 4.{+4.
+}
Revised: [Yes] See Section 4.1 and setup in 4.2 and {+4.
+}

Original: (c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] We report average performance of 5 trials by default.
Revised: [Yes] We report average performance of 5 trials by default.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Section 4.1.
Revised: [Yes] See Section 4.1.4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] Waymo [38] and
Revised: [Yes] Waymo [32] and

Original: KITTI [8]
Revised: KITTI [7]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: We don’t include any new assets.
Revised: We don’t include any new assets.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [Yes] Waymo [38] and KITTI [8] are two standard benchmarks.
Revised: [Yes] Waymo [32] and KITTI [7] are two standard benchmarks.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [Yes] Waymo [38] and KITTI [8] are two standardbenchmarks.
Revised: [Yes] Waymo [32] and KITTI [7] are two standard benchmarks.

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots,
Revised: (a) Did you include the full text of instructions given to participants and screenshots,

Original: if applicable?
Revised: if applicable?

Original: [N/A]
Revised: [N/A]

Original: We didn’t use crowdsourcing or conduct research with human
Revised: We didn’t use crowdsourcing or conduct research with human

Original: subjects...
Revised: subjects...

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: We didn’t use crowdsourcing or conduct
Revised: We didn’t use crowdsourcing or conduct

Original: research with human subjects...
Revised: research with human subjects...

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A] We didn’t use crowdsourcing or conduct
Revised: [N/A] We didn’t use crowdsourcing or conduct

Original: research with human subjects...
Revised: research with human subjects...