Original: DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries
Revised: DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract: We introduce a framework for multi-camera 3D object detection.
Revised: Abstract: We introduce a framework for multi-camera 3D object detection.

Original: Incontrast to existing works, which estimate 3D bounding boxes directly frommonocular images or use depth prediction networks to generate input for 3D object detection from 2D information, our method manipulates predictions directlyin 3D space.
Revised: In contrast to existing works, which estimate 3D bounding boxes directly from monocular images or use depth prediction networks to generate input for 3D object detection from 2D information, our method manipulates predictions directly in 3D space.

Original: Our architecture extracts 2D features from multiple camera imagesand then uses a sparse set of 3D object queries to index into these 2D features,linking 3D positions to multi-view images using camera transformation matrices.
Revised: Our architecture extracts 2D features from multiple camera images and then uses a sparse set of 3D object queries to index into these 2D features, linking 3D positions to multi-view images using camera transformation matrices.

Original: Finally, our model makes a bounding box prediction per object query, using aset-to-set loss to measure the discrepancy between the ground-truth and the prediction.
Revised: Finally, our model makes a bounding box prediction per object query, using a set-to-set loss to measure the discrepancy between the ground-truth and the prediction.

Original: This top-down approach outperforms its bottom-up counterpart in whichobject bounding box prediction follows per-pixel depth estimation, since it doesnot suffer from the compounding error introduced by a depth prediction model.
Revised: This top-down approach outperforms its bottom-up counterpart in which object bounding box prediction follows per-pixel depth estimation, since it does not suffer from the compounding error introduced by a depth prediction model.

Original: Moreover, our method does not require post-processing such as non-maximumsuppression, dramatically improving inference speed.
Revised: Moreover, our method does not require post-processing such as non-maximum suppression, dramatically improving inference speed.

Original: We achieve state-of-the-artperformance on the nuScenes autonomous driving benchmark.
Revised: We achieve state-of-the-art performance on the nuScenes autonomous driving benchmark.

Original:  Introduction
Revised:  Introduction

Original: 3D object detection from visual information is a long-standing challenge for low-cost autonomousdriving systems.
Revised: 3D object detection from visual information is a long-standing challenge for low-cost autonomous driving systems.

Original: While object detection from point clouds collected using modalities like LiDARbeneﬁts from information about the 3D structure of visible objects, the camera-based setting is evenmore ill-posed, since we must generate 3D bounding box predictions solely from the 2D informationcontained in RGB images.
Revised: While object detection from point clouds collected using modalities like LiDAR beneﬁts from information about the 3D structure of visible objects, the camera-based setting is even more ill-posed, since we must generate 3D bounding box predictions solely from the 2D information contained in RGB images.

Original: Existing methods [1, 2] typically build their detection pipelines purely from 2D computations.
Revised: Existing methods [1, 2] typically build their detection pipelines purely from 2D computations.

Original: Thatis, they predict 3D information like object pose and velocity using an object detection pipeline designed for 2D tasks (e.g., CenterNet [1], FCOS [3]), without considering 3D scene structure or sensorconﬁguration.
Revised: That is, they predict 3D information like object pose and velocity using an object detection pipeline designed for 2D tasks (e.g., CenterNet [1], FCOS [3]), without considering 3D scene structure or sensor conﬁguration.

Original: These methods require several post-processing steps to fuse predictions across cameras and to remove redundant boxes, yielding a steep trade-off between efﬁciency and effectiveness.
Revised: These methods require several post-processing steps to fuse predictions across cameras and to remove redundant boxes, yielding a steep trade-off between efﬁciency and effectiveness.

Original: As an alternative to these 2D-based methods, some methods incorporate more 3D computationsinto our object detection pipeline by applying a 3D reconstruction method like [4, 5, 6] to create apseudo-LiDAR or range input of the scene from camera images.
Revised: As an alternative to these 2D-based methods, some methods incorporate more 3D computations into our object detection pipeline by applying a 3D reconstruction method like [4, 5, 6] to create a pseudo-LiDAR or range input of the scene from camera images.

Original: Then, they could apply 3D objectdetection methods to this data as if it were collected directly from a 3D sensor.
Revised: Then, they could apply 3D object detection methods to this data as if it were collected directly from a 3D sensor.

Original: This strategy, however, is subject to compounding errors [7]: poorly-estimated depth values have a strongly negativeeffect on the performance of 3D object detection, which also can exhibit errors of its own. 
Revised: This strategy, however, is subject to compounding errors [7]: poorly-estimated depth values have a strongly negative effect on the performance of 3D object detection, which also can exhibit errors of its own. [Table]

Original: In this paper, we propose a more graceful transition between 2D observations and 3D predictions forautonomous driving, which does not rely on a module for dense depth prediction.
Revised: In this paper, we propose a more graceful transition between 2D observations and 3D predictions for autonomous driving, which does not rely on a module for dense depth prediction.

Original: Our framework,termed DETR3D (Multi-View 3D Detection), addresses this problem in a top-down fashion.
Revised: Our framework, termed DETR3D (Multi-View 3D Detection), addresses this problem in a top-down fashion.

Original: We link2D feature extraction and 3D object prediction via geometric back-projection with camera transformation matrices.
Revised: We link 2D feature extraction and 3D object prediction via geometric back-projection with camera transformation matrices.

Original: Our method starts from a sparse set of object priors, shared across the datasetand learned end-to-end.
Revised: Our method starts from a sparse set of object priors, shared across the dataset and learned end-to-end.

Original: To gather scene-speciﬁc information, we back-project a set of referencepoints decoded from these object priors to each camera and fetch the corresponding image featuresextracted by a ResNet backbone [8].
Revised: To gather scene-speciﬁc information, we back-project a set of reference points decoded from these object priors to each camera and fetch the corresponding image features extracted by a ResNet backbone [8].

Original: The features collected from the image features of the referencepoints then interact with each other through a multi-head self-attention layer [9].
Revised: The features collected from the image features of the reference points then interact with each other through a multi-head self-attention layer [9].

Original: After a series ofdistribute. self-attention layers, we read off bounding box parameters from every layer and use a set-to-set lossinspired by DETR [10] to evaluate performance.
Revised: After a series of self-attention layers, we read off bounding box parameters from every layer and use a set-to-set loss inspired by DETR [10] to evaluate performance.

Original: Our architecture does not perform point cloud reconstruction or explicit depth prediction from images, making it robust to errors in depth estimation.
Revised: Our architecture does not perform point cloud reconstruction or explicit depth prediction from images, making it robust to errors in depth estimation.

Original: Moreover, our method does not require anypost-processing, such as non-maximum suppression (NMS), improving efﬁciency and reducing reliance on hand-designed methods for cleaning its output.
Revised: Moreover, our method does not require any post-processing, such as non-maximum suppression (NMS), improving efﬁciency and reducing reliance on hand-designed methods for cleaning its output.

Original: On the nuScenes dataset, our method(without NMS) is comparable with prior art (with NMS).
Revised: On the nuScenes dataset, our method (without NMS) is comparable with prior art (with NMS).

Original: In the camera overlap regions, our methodsigniﬁcantly outperforms others.
Revised: In the camera overlap regions, our method signiﬁcantly outperforms others.

Original: Contributions.
Revised: Contributions.

Original: We summarize our key contributions as follows:
Revised: We summarize our key contributions as follows

Original:  • We present a streamlined 3D object detection model from RGB images.
Revised: : • We present a streamlined 3D object detection model from RGB images.

Original: Different from existingworks that combine object predictions from the different camera views in a ﬁnal stage, our methodfuses information from all the camera views in each layer of computation.
Revised: Different from existing works that combine object predictions from the different camera views in a ﬁnal stage, our method fuses information from all the camera views in each layer of computation.

Original: To the best of ourknowledge, this is the ﬁrst attempt to cast multi-camera detection as 3D set-to-set prediction.•
Revised: To the best of our knowledge, this is the ﬁrst attempt to cast multi-camera detection as 3D set-to-set prediction.

Original:  We introduce a module that connects 2D feature extraction and 3D bounding box prediction via backward geometric projection.
Revised: • We introduce a module that connects 2D feature extraction and 3D bounding box prediction via backward geometric projection.

Original: It does not suffer from inaccurate depth predictions from a secondary network, and seamlessly uses information from multiple cameras by back-projecting 3Dinformation onto all available frames.
Revised: It does not suffer from inaccurate depth predictions from a secondary network, and seamlessly uses information from multiple cameras by back-projecting 3D information onto all available frames.

Original: • Similarly to Object DGCNN [11], our method does not require post-processing such as per-imageor global NMS, and it is on par with existing NMS-based methods.
Revised: • Similarly to Object DGCNN [11], our method does not require post-processing such as per-image or global NMS and it is on par with existing NMS-based methods.

Original: In the camera overlap regions,our method outperforms others by a substantial margin.•
Revised: In the camera overlap regions, our method outperforms others by a substantial margin.

Original:  Wewill release our code to promote reproducibility and future research.
Revised: • We release our code to facilitate reproducibility and future research.

Original:  Related Work
Revised:  Related Work

Original: 2D object detection.
Revised: 2D object detection.

Original: RCNN [12] pioneered object detection using deep learning.
Revised: RCNN [12] pioneered object detection using deep learning.

Original: It feeds a setof pre-selected object proposals into a convolutional neural network (CNN) and predicts boundingbox parameters accordingly.
Revised: It feeds a set of pre-selected object proposals into a convolutional neural network (CNN) and predicts bounding box parameters accordingly.

Original: Although this method exhibits surprising performance, it is an order ofmagnitude slower than others because it performs a ConvNet forward pass for each object proposal.
Revised: Although this method exhibits surprising performance, it is an order of magnitude slower than others because it performs a ConvNet forward pass for each object proposal.

Original: To ﬁx this issue, Fast RCNN [13] introduces a shared learnable CNN to process the entire image at asingle forward pass.
Revised: To ﬁx this issue, Fast RCNN [13] introduces a shared learnable CNN to process the entire image at a single forward pass.

Original: To further improve performance and speed, Faster RCNN [13] includes a regionproposal network (RPN) that shares full-image convolutional features with the detection network,thus enabling nearly cost-free region proposals.
Revised: To further improve performance and speed, Faster RCNN [13] includes a region proposal network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals.

Original: Mask RCNN [14] incorporates a mask predictionbranch to enable instance segmentation in parallel.
Revised: Mask RCNN [14] incorporates a mask prediction branch to enable instance segmentation in parallel.

Original: These methods typically involve multi-stagereﬁnements and can be slow in practice.
Revised: These methods typically involve multi-stage reﬁnements and can be slow in practice.

Original: Different from these multi-stage methods, SSD [15] and
Revised: Different from these multi-stage methods, SSD [15] and

Original: YOLO [16] perform dense predictions in a single shot.
Revised: YOLO [16] perform dense predictions in a single shot.

Original: Although they are signiﬁcantly faster thanthe alternatives above, they still rely on NMS to remove redundant box predictions.
Revised: Although they are signiﬁcantly faster than the alternatives above, they still rely on NMS to remove redundant box predictions.

Original: These methodspredict bounding boxes w.r.t.
Revised: These methods predict bounding boxes w.r.t.

Original: pre-deﬁned anchors.
Revised: pre-deﬁned anchors.

Original: CenterNet [1] and FCOS [3] change the paradigmby shifting from per-anchor prediction to per-pixel prediction, signiﬁcantly simplifying the commonobject detection pipeline.
Revised: CenterNet [1] and FCOS [3] change the paradigm by shifting from per-anchor prediction to per-pixel prediction, signiﬁcantly simplifying the common object detection pipeline.

Original: Set-based object detection.
Revised: Set-based object detection.

Original: DETR [10] casts object detection as a set-to-set problem.
Revised: DETR [10] casts object detection as a set-to-set problem.

Original: It employs a
Revised: It employs a

Original: Transformer [9] to capture feature and object interactions.
Revised: Transformer [9] to capture feature and object interactions.

Original: DETR learns to assign predictions to a setof ground-truth boxes; thus, it does not require post-processing to ﬁlter out redundant boxes.
Revised: DETR learns to assign predictions to a set of ground-truth boxes; thus, it does not require post-processing to ﬁlter out redundant boxes.

Original: Onecritical drawback of DETR, however, is that it requires a signiﬁcant amount of training time.
Revised: One critical drawback of DETR, however, is that it requires a signiﬁcant amount of training time.

Original: Deformable DETR [17] analyzes DETR’s slow convergence and proposes a deformable self-attentionmodule to localize features and accelerate training.
Revised: Deformable DETR [17] analyzes DETR’s slow convergence and proposes a deformable self-attention module to localize features and accelerate training.

Original: Concurrently, [18] attributes the slow convergence of DETR to the set-based loss and the Transformer cross attention mechanism.
Revised: Concurrently, [18] attributes the slow convergence of DETR to the set-based loss and the Transformer cross attention mechanism.

Original: They propose two variants, TSP-FCOS and TSP-RCNN, to overcome these problematic aspects.
Revised: They pro- [Figure] pose two variants, TSP-FCOS and TSP-RCNN, to overcome these problematic aspects.

Original: SparseRCNN [19] incorporates set prediction into a RCNN-style pipeline; it outperforms multi-stage objectdetection without NMS.
Revised: SparseRCNN [19] incorporates set prediction into a RCNN-style pipeline; it outperforms multi-stage object detection without NMS.

Original: OneNet [20] studies an interesting phenomenon: dense-based object detectors can be made NMS-free after they are equipped with a minimum-cost set loss.
Revised: OneNet [20] studies an interesting phenomenon: dense-based object detectors can be made NMS-free after they are equipped with a minimum-cost set loss.

Original: For 3D domains,
Revised: For 3D domains,

Original: Object DGCNN [11] studies 3D object detection from point clouds.
Revised: Object DGCNN [11] studies 3D object detection from point clouds.

Original: It models 3D object detectionas message passing on a dynamic graph, generalizing the DGCNN framework to predict a set ofobjects.
Revised: It models 3D object detection as message passing on a dynamic graph, generalizing the DGCNN framework to predict a set of objects.

Original: Similar to DETR, Object DGCNN is also NMS-free.
Revised: Similar to DETR, Object DGCNN is also NMS-free.

Original: Monocular 3D object detection.
Revised: Monocular 3D object detection.

Original: An early method for 3D detection from RGB images is
Revised: An early method for 3D detection from RGB images is

Original: Mono3D [21], which uses semantic and shape cues to select from a collection of 3D proposals,using scene constraints and additional priors at training time.
Revised: Mono3D [21], which uses semantic and shape cues to select from a collection of 3D proposals, using scene constraints and additional priors at training time.

Original: [22] uses the birds-eye-view (BEV)[Figure] for monocular 3D detection, and [23] leverages 2D detections for 3D bounding box regression viathe minimization of 2D-3D projection error.
Revised: [22] uses the birds-eye-view (BEV) for monocular 3D detection, and [23] leverages 2D detections for 3D bounding box regression via the minimization of 2D-3D projection error.

Original: The use of 2D detectors as a starting point for 3Dcomputation recently has become a standard approach [24, 25].
Revised: The use of 2D detectors as a starting point for 3D computation recently has become a standard approach [24, 25].

Original: Other works also explore advancesin differentiable rendering [26] or 3D keypoint detection [27, 28, 1] to enable state-of-the-art 3D[Figure] object detection performance.
Revised: Other works also explore advances in differentiable rendering [26] or 3D keypoint detection [27, 28, 1] to enable state-of-the-art 3D object detection performance.

Original: All these methods operate in a monocular setting, and extensions tomultiple cameras are done by independently processing each frame before merging the outputs in a[Figure] post-processing stage.[Figure]
Revised: All these methods operate in a monocular setting, and extensions to multiple cameras are done by independently processing each frame before merging the outputs in a post-processing stage.

Original:  Multi-view 3D Object Detection 
Revised:  Multi-view 3D Object Detection 3.1 Overview

Original: [Figure] 3.1 Overview

Original: [Figure] Our architecture inputs RGB images collected from a set of cameras whose projection matrices (thecombination of intrinsics and relative extrinsics) are known, and it outputs a set of 3D bounding boxparameters for the objects in the scene.
Revised: Our architecture inputs RGB images collected from a set of cameras whose projection matrices (the combination of intrinsics and relative extrinsics) are known, and it outputs a set of 3D bounding box parameters for the objects in the scene.

Original: In contrast to past approaches, we build our architecture[Figure] based on a few high-level desiderata:
Revised: In contrast to past approaches, we build our architecture based on a few high-level desiderata:

Original: [Figure] • We incorporate 3D information into intermediate computations within our architecture, rather than[Figure] performing purely 2D computations in the image plane.
Revised: • We incorporate 3D information into intermediate computations within our architecture, rather than performing purely 2D computations in the image plane.

Original: [Figure] • We do not estimate dense 3D scene geometry, avoiding associated reconstruction errors.
Revised: • We do not estimate dense 3D scene geometry, avoiding associated reconstruction errors.

Original: • We avoid post-processing steps such as NMS
Revised: • We avoid post-processing steps such as NMS.

Original: We address these desiderata using a new set prediction module, which links 2D feature extraction and 3D box prediction by alternating between 2D and 3D computations.
Revised: We address these desiderata using a new set prediction module, which links 2D feature extraction and 3D box prediction by alternating between 2D and 3D computations.

Original: Our model containsthree critical components, illustrated in Figure 1.
Revised: Our model contains three critical components, illustrated in Figure 1.

Original: First, following common practice in 2D vision,it extracts features from the camera images using a shared ResNet [8] backbone.
Revised: First, following common practice in 2D vision, it extracts features from the camera images using a shared ResNet [8] backbone.

Original: Optionally, thesefeatures are enhanced by a feature pyramid network (FPN) [29] (§3.2).
Revised: Optionally, these features are enhanced by a feature pyramid network (FPN) [29] (§3.2).

Original: Second, a detection head(§3.3)—our main contribution—links the computed 2D features to a set of 3D bounding box predictions in a geometry-aware manner (§3.3).
Revised: Second, a detection head (§3.3)—our main contribution—links the computed 2D features to a set of 3D bounding box predictions in a geometry-aware manner (§3.3).

Original: Each layer of the detection head starts from a sparse setof object queries, which are learned from the data.
Revised: Each layer of the detection head starts from a sparse set of object queries, which are learned from the data.

Original: Each object query encodes a 3D location, whichis projected to the camera planes and used to collect image features via bilinear interpolation.
Revised: Each object query encodes a 3D location, which is projected to the camera planes and used to collect image features via bilinear interpolation.

Original: Similarly to DETR [10], we then use multi-head attention [9] to reﬁne the object queries by incorporatingobject interactions.
Revised: Similarly to DETR [10], we then use multi-head attention [9] to reﬁne the object queries by incorporating object interactions.

Original: This layer is repeated multiple times, alternating between feature sampling andobject query reﬁnement.
Revised: This layer is repeated multiple times, alternating between feature sampling and object query reﬁnement.

Original: Finally, we evaluate a set-to-set loss [30, 10] to train the network (§3.4).
Revised: Finally, we evaluate a set-to-set loss [30, 10] to train the network (§3.4).

Original: 3.2 Feature Learning
Revised: 3.2 Feature Learning

Original: Our model starts with a set of images
Revised: Our model starts with a set of images

Original: I =
Revised: I =

Original: { im 1 , . .
Revised: { im 1 , . .

Original: , im K } ⇢ R H
Revised: , im K } ⇢ R H

Original: im ⇥
Revised: im ⇥

Original: W im ⇥ 3 (captured by surrounding cameras), camera matrices T =
Revised: W im ⇥ 3 (captured by surrounding cameras), camera matrices T =

Original: { T 1 , . . .
Revised: { T 1 , . . .

Original: , T K } ⇢ R 3 ⇥ 4 , ground-truth bounding boxes
Revised: , T K } ⇢ R 3 ⇥ 4 , ground-truth bounding boxes

Original: B = { b 1 , . . .
Revised: B = { b 1 , . . .

Original: , b j , . . .
Revised: , b j , . . .

Original: , b M } ⇢ R 9 , and categorical labels C
Revised: , b M } ⇢ R 9 , and categorical labels C

Original: = { c 1 , . . .
Revised: = { c 1 , . . .

Original: , c j , . .
Revised: , c j , . .

Original: . , c M } ⇢ Z .
Revised: . , c M } ⇢ Z .

Original: Each b j128 contains position, size, heading angle, and velocity in the birds-eye view (BEV); our model aims to predict these boxes and their labels from the these images.
Revised: Each b j contains position, size, heading angle, and velocity in the birds-eye view (BEV); our model aims to predict these boxes and their labels from the these images.

Original: We do not use point clouds, which areusually captured by high-end LiDAR.
Revised: We do not use point clouds, which are usually captured by high-end LiDAR.

Original: These images are encoded with a ResNet [8] and a FPN [29] into four sets of features F 1 , F 2 , F 3 , F 4 .
Revised: These images are encoded with a ResNet [8] and a FPN [29] into four sets of features F 1 , F 2 , F 3 , F 4 .

Original: 132 Each set
Revised: Each set

Original: F k = { f k 1 , . . .
Revised: F k = { f k 1 , . . .

Original: , f k 6 } ⇢ R H ⇥ W ⇥ C corresponds to a level of features of the 6 images.
Revised: , f k 6 } ⇢ R H ⇥ W ⇥ C corresponds to a level of features of the 6 images.

Original: These multi-scale features provide rich information to recognize objects of different sizes.
Revised: These multi-scale features provide rich information to recognize objects of different sizes.

Original: Next, wedetail our approach to transform these 2D features into 3D using a novel set prediction module.
Revised: Next, we detail our approach to transform these 2D features into 3D using a novel set prediction module.

Original: 3.3 Detection Head
Revised: 3.3 Detection Head

Original: Existing methods for detecting objects from camera input typically employ a bottom-up approach,which predicts a dense set of bounding boxes per image, ﬁlters redundant boxes between the images,and aggregates predictions across cameras in a post-processing step.
Revised: Existing methods for detecting objects from camera input typically employ a bottom-up approach, which predicts a dense set of bounding boxes per image, ﬁlters redundant boxes between the images, and aggregates predictions across cameras in a post-processing step.

Original: This paradigm has two crucialdrawbacks: dense bounding box prediction requires accurate depth perception, which itself is achallenging problem; and NMS-based redundancy removal and aggregation are non-parallelizableoperations that introduce signiﬁcant inference overhead.
Revised: This paradigm has two crucial drawbacks: dense bounding box prediction requires accurate depth perception, which itself is a challenging problem; and NMS-based redundancy removal and aggregation are non-parallelizable operations that introduce signiﬁcant inference overhead.

Original: We address these issues using a top-downobject detection head described below.
Revised: We address these issues using a top-down object detection head described below.

Original: Analogously to [11, 17], DETR3D is iterative ; it uses L layers with set-based computations toproduce bounding box estimates from 2D feature maps.
Revised: Analogously to [11, 17], DETR3D is iterative ; it uses L layers with set-based computations to produce bounding box estimates from 2D feature maps.

Original: Each layer includes the following steps:
Revised: Each layer includes the following steps:

Original: 1. predict a set of bounding box centers associated with object queries; 2. project these centers into all the feature maps using the camera transformation matrices;  sample features via bilinear interpolation and incorporate them into object queries; and 
Revised: 1. predict a set of bounding box centers associated with object queries; 2. project these centers into all the feature maps using the camera transformation matrices; 3. sample features via bilinear interpolation and incorporate them into object queries; and 4.

Original: describe object interactions using multi-head attention.
Revised: describe object interactions using multi-head attention.

Original: Motivated by DETR [10], each layer ` 2 { 0 , . . .
Revised: Motivated by DETR [10], each layer ` 2 { 0 , . . .

Original: , L  1 } operates on a set of object queries Q ` ={ q ` 1 , . . .
Revised: , L  1 } operates on a set of object queries Q ` = { q ` 1 , . . .

Original: , q ` M ⇤ } ⇢ R C , producing a new set Q ` +1 .
Revised: , q ` M ⇤ } ⇢ R C , producing a new set Q ` +1 .

Original: A reference point c
Revised: A reference point c

Original: ` i 2 R 3 is decoded from a151 object query q ` i as follows:
Revised: ` i 2 R 3 is decoded from a object query q ` i as follows:

Original: [Equation]
Revised: [Equation]

Original: ) where  ref is a neural network.
Revised: where  ref is a neural network.

Original: c ` i can be thought of a hypothesis for the center of the i -th box.
Revised: c ` i can be thought of a hypothesis for the center of the i -th box.

Original: Next, we acquire image features corresponding to c ` i to reﬁne and predict the ﬁnal bounding box.
Revised: Next, we acquire image features corresponding to c ` i to reﬁne and predict the ﬁnal bounding box.

Original: Then, c ` i (or more accurately its homogeneous counterpart c ⇤ ` i ) is projected into each one of theimages using the camera transformation matrices:
Revised: Then, c ` i (or more accurately its homogeneous counterpart c ⇤ ` i ) is projected into each one of the images using the camera transformation matrices:

Original: [Equation]
Revised: [Equation]

Original: where  denotes concatenation, and c ` mi is the projection of the reference point onto the m -thcamera.
Revised: where  denotes concatenation, and c ` mi is the projection of the reference point onto the m -th camera.

Original: To remove the effects of the feature map size and gather features across different levels, wenormalize c ` mi to [  1 , 1] .
Revised: To remove the effects of the feature map size and gather features across different levels, we normalize c ` mi to [  1 , 1] .

Original: Next, the images features are collected by
Revised: Next, the images features are collected by

Original: [Equation]
Revised: [Equation]

Original: where f ` kmi is the feature for i -th point from k -th level of m -th camera at ` -th layer.
Revised: where f ` kmi is the feature for i -th point from k -th level of m -th camera at ` -th layer.

Original: A given reference point is not necessarily visible in all the camera images, so we need some heuristics to ﬁlter invalid points.
Revised: A given reference point is not necessarily visible in all the camera images, so we need some heuristics to ﬁlter invalid points.

Original: To that end, we deﬁne a binary value  ` kmi , which is determined basedon whether a reference point is projected outside an image plane.
Revised: To that end, we deﬁne a binary value  ` kmi , which is determined based on whether a reference point is projected outside an image plane.

Original: The ﬁnal feature f ` i and objectquery in next layer q ( ` +1) i are given by
Revised: The ﬁnal feature f ` i and object query in next layer q ( ` +1) i are given by

Original: [Equation]
Revised: [Equation]

Original: where ✏ is a small number to avoid division by zero.
Revised: where ✏ is a small number to avoid division by zero.

Original: Finally, for each object query q ` i , we predict abounding box ˆ b ` i and its categorical label ˆ c ` i with two neural networks  reg ` and  cls` :
Revised: Finally, for each object query q ` i , we predict a bounding box ˆ b ` i and its categorical label ˆ c ` i with two neural networks  reg ` and  cls

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: We compute the loss for the predictions ˆ B 
Revised: We compute the loss for the predictions ˆ B ` = { ˆ b ` 1 , . . .

Original: [Equation]
Revised: , ˆ b ` j , . . .

Original: [Equation]
Revised: , ˆ b ` M ⇤ } ⇢ R 9 and ˆ C ` = { ˆ c ` 1 , . . . , ˆ c ` j , . . . , ˆ c ` M } ⇢ Z from every layer during training.

Original: [Table]
Revised: During inference, we only use the outputs from the last layer.

Original: 3.4 Loss
Revised: 3.4 Loss

Original: Following [30, 10], we use a set-to-set loss to measure the discrepancy between the prediction set( ˆ B ` , ˆ C ` ) and the ground-truth set ( B , C ) .
Revised: Following [30, 10], we use a set-to-set loss to measure the discrepancy between the prediction set ( ˆ B ` , ˆ C ` ) and the ground-truth set ( B , C ) .

Original: This loss consists of two parts: a focal loss [31] for theclass labels and a L 1 loss for the bounding box parameters.
Revised: This loss consists of two parts: a focal loss [31] for the class labels and a L 1 loss for the bounding box parameters.

Original: For notational convenience, we drop the` subscript in ˆ B ` and ˆ C ` .
Revised: For notational convenience, we drop the ` subscript in ˆ B ` and ˆ C ` .

Original: The number of ground-truth boxes M is typically smaller than the numberof predictions M ⇤ , so we pad the set of ground-truth boxes with ?
Revised: The number of ground-truth boxes M is typically smaller than the number of predictions M ⇤ , so we pad the set of ground-truth boxes with ?

Original: s (no object) up to M ⇤ for ease ofcomputation.
Revised: s (no object) up to M ⇤ for ease of computation.

Original: We establish a correspondence between the ground-truth and the prediction via a bipartite matching problem:  ⇤ =
Revised: We establish a correspondence between the ground-truth and the prediction via a bipartite matching problem:  ⇤ =

Original: arg min
Revised: arg min

Original:  2 P

Original: P

Original: [Equation]
Revised: [Equation]

Original: L box ( [Equation] ˆ b  ( j ) ) ,where P denotes the set of permutations, ˆ p  ( j ) ( c j ) is the probability of class c j for the predictionwith index  ( j ) , and L box is the L 1 loss for bounding box parameters.
Revised: where P denotes the set of permutations, ˆ p  ( j ) ( c j ) is the probability of class c j for the prediction with index  ( j ) , and L box is the L 1 loss for bounding box parameters.

Original: We use the Hungarianalgorithm [32] to solve this assignment problem, as in [30, 10], yielding the set-to-set loss L sup =
Revised: We use the Hungarian algorithm [32] to solve this assignment problem, as in [30, 10], yielding the set-to-set loss L sup =

Original: P
Revised: P

Original: Nj =
Revised: Nj =

Original: [Equation]

Original:  log

Original: [Equation]

Original: p

Original: [Equation]

Original: c

Original: [Equation]

Original: +

Original: [Equation]

Original: { c j = ?

Original: [Equation]

Original: L box ( b

Original: [Equation] [Equation]  [Equation]
Revised: [Table]

Original: j ) ) .

Original:  Experiments
Revised:  Experiments

Original: We present our results as follows:
Revised: We present our results as follows:

Original: ﬁrst, we detail the dataset, metrics, and implementation in §4.1;then we compare our method to existing works in §4.2; we benchmark the performance of differentmodels in camera overlap regions in §4.3; we compare to a forward prediction model in §4.4; andwe provide additional analysis and ablations in §4.5.
Revised: ﬁrst, we detail the dataset, metrics, and implementation in §4.1; then we compare our method to existing works in §4.2; we benchmark the performance of different models in camera overlap regions in §4.3; we compare to a forward prediction model in §4.4; and we provide additional analysis and ablations in §4.5.

Original: 4.1 Implementation Details
Revised: 4.1 Implementation Details

Original: Dataset.
Revised: Dataset.

Original: We test our method on the nuScenes dataset [33].
Revised: We test our method on the nuScenes dataset [33].

Original: nuScenes consists of 1,000 se-
Revised: nuScenes consists of 1,000 sequences; each sequence is roughly 20s long, with a sampling rate of 20 frames/second. Each sample contains images from 6 cameras [front left, front, front right, back left, back, back right] . Camera parameters including intrinsics and extrinsics are available. nuScenes provides annotations every 0.5s; in total there are 28k, 6k, and 6k annotated samples for training, validation, and testing, respectively.

Revised: 10 from the total 23 classes are available to compute the metrics.

Original: Metrics.
Revised: Metrics.

Original: We follow the ofﬁcial evaluation protocol provided by nuScenes.
Revised: We follow the ofﬁcial evaluation protocol provided by nuScenes.

Original: We evaluate averagetranslation error (ATE), average scale error (ASE), average orientation error (AOE), average velocityerror (AVE), and average attribute error (AAE).
Revised: We evaluate average translation error (ATE), average scale error (ASE), average orientation error (AOE), average velocity error (AVE), and average attribute error (AAE).

Original: These metrics are true positive metrics (TP metrics)and computed in the physical unit.
Revised: These metrics are true positive metrics (TP metrics) and computed in the physical unit.

Original: In addition, we measure mean average precision (mAP).
Revised: In addition, we measure mean average precision (mAP).

Original: Tocapture all aspects of the detection task, a consolidated scalar metric–the nuScenes Detection Score(NDS) [33]–is deﬁned as NDS = 110 [5mAP + 
Revised: To capture all aspects of the detection task, a consolidated scalar metric–the nuScenes Detection Score (NDS) [33]–is deﬁned as NDS = 110 [5mAP + P [Equation]

Revised: , mTP

Original: [Equation]
Revised: [Equation]

Original: Model.
Revised: Model.

Original: Our model consists of a ResNet [8] feature extractor, a FPN, and a DETR3D detection head.
Revised: Our model consists of a ResNet [8] feature extractor, a FPN, and a DETR3D detection head.

Original: We use ResNet101 with deformable convolutions [34] in the 3rd stage and 4th stage.
Revised: We use ResNet101 with deformable convolutions [34] in the 3rd stage and 4th stage.

Original: The FPN [29]takes features output by the ResNet and produces 4 feature maps whose sizes are 1 / 8 , 1 / 16 , 1 / 32 , and 202/ 64 of the input image sizes.
Revised: The FPN [29] takes features output by the ResNet and produces 4 feature maps whose sizes are 1 / 8 , 1 / 16 , 1 / 32 , and/ 64 of the input image sizes.

Original: The DETR3D detection head consists of 6 layers, where each layer is acombination of a feature reﬁnement step and a multi-head attention layer.
Revised: The DETR3D detection head consists of 6 layers, where each layer is a combination of a feature reﬁnement step and a multi-head attention layer.

Original: The hidden dimension ofthe DETR3D detection head is 256.
Revised: The hidden dimension of the DETR3D detection head is 256.

Original: Finally, two sub-networks predict bounding box parameters anda class label per object query; each sub-network consists of two fully connected layers with hiddendimensions 256.
Revised: Finally, two sub-networks predict bounding box parameters and a class label per object query; each sub-network consists of two fully connected layers with hidden dimensions 256.

Original: We use LayerNorm [35] in the detection head.
Revised: We use LayerNorm [35] in the detection head.

Original: Training & inference.
Revised: Training & inference.

Original: We use AdamW [36] to train the whole pipeline.
Revised: We use AdamW [36] to train the whole pipeline.

Original: The weight decay is 10  4 .
Revised: The weight decay is 10  4 .

Original: We use an initial learning rate 10  4 , which is decreased to 10  5 and 10  6 at 8th and 11th epochs.
Revised: We use an initial learning rate 10  4 , which is decreased to 10  5 and 10  6 at 8th and 11th epochs.

Original: The model is trained for 12 epochs in total on 8 RTX 3090 GPUs and the per-GPU batch size is 1. 
Revised: The model is trained for 24 epochs in total on 8 RTX 3090 GPUs and the per-GPU batch size is 1. [Table]

Revised: [Table]

Original: The training procedure takes roughly 18 hours.
Revised: The training procedure takes roughly 18 hours.

Original: We do not use any post-processing such as NMSduring inference.
Revised: We do not use any post-processing such as NMS during inference.

Original: For evaluation, we use the nuScenes evalutation toolkit.
Revised: For evaluation, we use the nuScenes evalutation toolkit.

Original: 4.2 Comparison to Existing Works
Revised: 4.2 Comparison to Existing Works

Original: We compare to previous state-of-the-art methods CenterNet [1] and FCOS3D [2].
Revised: We compare to previous state-of-the-art methods CenterNet [1] and FCOS3D [2].

Original: CenterNet is ananchor-free 2D detection method that makes dense predictions in a high resolution feature map.
Revised: CenterNet is an anchor-free 2D detection method that makes dense predictions in a high resolution feature map.

Original: FCOS3D employs a FCOS [3] pipeline to make per-pixel predictions.
Revised: FCOS3D employs a FCOS [3] pipeline to make per-pixel predictions.

Original: These methods both turn 3Dobject detection into a 2D problem, and in doing so ignore scene geometry and sensor conﬁguration.
Revised: These methods both turn 3D object detection into a 2D problem, and in doing so ignore scene geometry and sensor conﬁguration.

Original: To perform multi-view object detection, these methods have to process each image independently,

Original: [Table] head is initialized randomly. [Table] and use both per-image and global NMS to remove redundant boxes in each view and in the overlapregions respectively.
Revised: To perform multi-view object detection, these methods have to process each image independently, and use both per-image and global NMS to remove redundant boxes in each view and in the overlap regions respectively.

Original: As shown in Table 1, our method is on a par with these methods even thoughwe do not use any post-processing.
Revised: As shown in Table 1, our method outperforms these methods even though we do not use any post-processing.

Original: Our method performs worse than FCOS3D in terms of mATE.
Revised: Our method performs worse than FCOS3D in terms of mATE.

Original: We suspect this is because FCOS3D directly predicts bounding box depth, which leads to strongsupervision on object translation.
Revised: We suspect this is because FCOS3D directly predicts bounding box depth, which leads to strong supervision on object translation.

Original: Also, FCOS3D uses disentangled heads for different boundingbox parameters, which can increase performance.
Revised: Also, FCOS3D uses disentangled heads for different bounding box parameters, which can increase performance.

Revised: On the test set (Table 2), our method outperforms all existing methods as of 10/13/2021; our method uses the same backbone as DD3D [37] for a fair comparison.

Original: 4.3 Comparison in Overlap Regions
Revised: 4.3 Comparison in Overlap Regions

Original: A great challenge lies in the overlap regions where objects are more likely to be cut off.
Revised: A great challenge lies in the overlap regions where objects are more likely to be cut off.

Original: Our methodconsiders all cameras simultaneously, while FCOS3D predicts bounding boxes per camera individually.
Revised: Our method considers all cameras simultaneously, while FCOS3D predicts bounding boxes per camera individually.

Original: To further demonstrate the advantages of fused inference, we calculate the metrics for boxesfalling into the camera overlaps.
Revised: To further demonstrate the advantages of fused inference, we calculate the metrics for boxes falling into the camera overlaps.

Original: To compute the metrics, we select boxes whose 3D center is visibleto multiple cameras.
Revised: To compute the metrics, we select boxes whose 3D center is visible to multiple cameras.

Original: On the validation set, there are 18,147 such boxes, 9.7% of the total.
Revised: On the validation set, there are 18,147 such boxes, 9.7% of the total.

Original: Table 2shows the results; our method outperforms FCOS3D remarkably in terms of NDS scores in thissetting.
Revised: Tableshows the results; our method outperforms FCOS3D remarkably in terms of NDS scores in this setting.

Original: This conﬁrms that our integrated prediction approach is more effective. 
Revised: This conﬁrms that our integrated prediction approach is more effective. [Table] [Table]

Original: 4.4 Comparison to pseudo-LiDAR Methods
Revised: 4.4 Comparison to pseudo-LiDAR Methods

Original: Another way to perform 3D object detection is by generating pseudo-LiDAR point clouds frommulti-view images using a depth prediction model.
Revised: Another way to perform 3D object detection is by generating pseudo-LiDAR point clouds from multi-view images using a depth prediction model.

Original: On the nuScenes dataset, there are no publiclyavailable pseudo-LiDAR works for us to make a direct comparison.
Revised: On the nuScenes dataset, there are no publicly available pseudo-LiDAR works for us to make a direct comparison.

Original: Hence, we implement a baseline ourselves to verify that our approach is more effective than explicit depth prediction.
Revised: Hence, we implement a baseline ourselves to verify that our approach is more effective than explicit depth prediction.

Original: We usea pre-trained PackNet [6] network to predict dense depth maps from all six cameras and then convert these depth maps into point clouds using the camera transformations.
Revised: We use a pre-trained PackNet [6] network to predict dense depth maps from all six cameras and then convert these depth maps into point clouds using the camera transformations.

Original: We also experimentedwith a self-supervised PackNet model with velocity supervision (as in the original paper), but wefound that ground-truth depth supervision yielded more realistic point clouds and therefore used asupervised model as baseline.
Revised: We also experimented with a self-supervised PackNet model with velocity supervision (as in the original paper), but we found that ground-truth depth supervision yielded more realistic point clouds and therefore used a supervised model as baseline.

Original: For 3D detection, we employ the recently-proposed CenterPoint architecture [38].
Revised: For 3D detection, we employ the recently-proposed CenterPoint architecture [41].

Original: Conceptually, this pipeline is a variant of pseudo-LiDAR [39].
Revised: Conceptually, this pipeline is a variant of pseudo-LiDAR [42].

Original: Table 3 shows theresults; we conclude that this pseudo-LiDAR method underperforms ours signiﬁcantly even whendepth estimates are generated by a state-of-the-art model.
Revised: Table 4 shows the results; we conclude that this pseudo-LiDAR method underperforms ours signiﬁcantly even when depth estimates are generated by a state-of-the-art model.

Original: One possible explanation is that pseudoLiDAR object detectors suffer from compounding errors introduced by inaccurate depth prediction,that in turn is known to overﬁt to training data and generalizes poorly to other distributions [7].[Table] [Table] [Figure]
Revised: One possible explanation is that pseudoLiDAR object detectors suffer from compounding errors introduced by inaccurate depth prediction, that in turn is known to overﬁt to training data and generalizes poorly to other distributions [7].

Original: 4.5 Ablation & Analysis
Revised: 4.5 Ablation & Analysis

Original: We provide a visualization of object query reﬁnement in Figure 2.
Revised: We provide a visualization of object query reﬁnement in Figure 2.

Original: We visualize bounding boxesdecoded from the object queries in each layer.
Revised: We visualize bounding boxes decoded from the object queries in each layer.

Original: The predicted bounding boxes get closer to theground-truth as we go into deeper layers in the model.
Revised: The predicted bounding boxes get closer to the ground-truth as we go into deeper layers in the model.

Original: Also, the leftmost ﬁgure shows the learnedobject query priors shared by all data.
Revised: Also, the leftmost ﬁgure shows the learned object query priors shared by all data.

Original: We also provide quantitative results in Table 4, which showsthat iterative reﬁnement indeed improves performance signiﬁcantly.
Revised: We also provide quantitative results in Table 5, which shows that iterative reﬁnement indeed improves performance signiﬁcantly.

Original: This suggests that iterativereﬁnement is both beneﬁcial and necessary to fully leverage our proposed architecture.
Revised: This suggests that iterative reﬁnement is both beneﬁcial and necessary to fully leverage our proposed architecture.

Original: Furthermore,we provide ablations on the number of object queries in Table 5; increasing the number queriesconsistently improves the performance until it gets saturated at 900.
Revised: Furthermore, we provide ablations on the number of object queries in Table 6; increasing the number queries consistently improves the performance until it gets saturated at 900.

Original: Finally, Table 6 shows theresults with different backbones.
Revised: Finally, Table 7 shows the results with different backbones.

Original: We also provide qualitative results in Figure 3 to facilitate an intuitive understanding of model performance.

Original: We project the predicted bounding boxes into 6 cameras as well as a BEV perspective.

Original: Ingeneral, our method generates reasonable results and even detects relatively small objects.

Original: However,our method still exhibits substantial translation error (in line with results in Table 4.2):

Original: Although ourmodel avoids explicit depth prediction, depth estimation is still a core challenging in this problem.

Original: [Table]

Original: [Table]

Original: P r e d i c ti o n

Original: P r e d i c ti on

Original: G r ound - t r u t h

Original: G r ound - t r u t h

Original: P r e d i c ti on

Original: G r ound - t r u t h

Original:  Conclusion
Revised:  Conclusion

Original: We propose a new paradigm to address the ill-posed inverse problem of recovering 3D informationfrom 2D images.
Revised: We propose a new paradigm to address the ill-posed inverse problem of recovering 3D information from 2D images.

Original: In this setting, the input signal lacks essential information for models to makeeffective predictions without priors learned from data.
Revised: In this setting, the input signal lacks essential information for models to make effective predictions without priors learned from data.

Original: While other methods either operate solely on2D computations or use additional depth networks to reconstruct the scene, ours operates in 3D spaceand uses backward projection to retrieve image features as needed.
Revised: While other methods either operate solely on 2D computations or use additional depth networks to reconstruct the scene, ours operates in 3D space and uses backward projection to retrieve image features as needed.

Original: The beneﬁts of our approach aretwo-fold: (1) it eliminates the need for middle-level representations (e.g., predicted depth mapsor point clouds), which can be a source of compounding errors; and (2) it uses information frommultiple cameras by projecting the same 3D point onto all available frames.
Revised: The beneﬁts of our approach are two-fold: (1) it eliminates the need for middle-level representations (e.g., predicted depth maps or point clouds), which can be a source of compounding errors; and (2) it [Table] [Table] [Table]

Original: Beyond the direct application of our work to 3D object detection for autonomous driving, thereare several venues that warrant future investigation.
Revised: Beyond the direct application of our work to 3D object detection for autonomous driving, there are several venues that warrant future investigation.

Original: For example, single point projection creates alimited receptive ﬁeld in the retrieved image feature maps, and sampling multiple points for eachobject query would incorporate more information for object reﬁnement.
Revised: For example, single point projection creates a limited receptive ﬁeld in the retrieved image feature maps, and sampling multiple points for each object query would incorporate more information for object reﬁnement.

Original: Furthermore, the new detection head is input-agnostic, and including other modalities such as LiDAR/RADAR would enhanceperformance and robustness.
Revised: Furthermore, the new detection head is input-agnostic, and including other modalities such as LiDAR/RADAR would enhance performance and robustness.

Original: Finally, generalizing our pipeline to other domains such as indoor navigation and object manipulation would increase its scope of application and reveal additional waysfor further improvement. 
Revised: Finally, generalizing our pipeline to other domains such as indoor navigation and object manipulation would increase its scope of application and reveal additional ways for further improvement. Acknowledgement

Revised: The MIT Geometric Data Processing group acknowledges the generous support of Army Research

Revised: Ofﬁce grants W911NF2010168 and W911NF2110293, of Air Force Ofﬁce of Scientiﬁc Research award FA9550-19-1-031, of National Science Foundation grants IIS-1838071 and CHS-1955697, from the CSAIL Systems that Learn program, from the MIT–IBM Watson AI Laboratory, from the Toyota–CSAIL Joint Research Center, from a gift from Adobe Systems, from an MIT.nano

Revised: Immersion Lab/NCSOFT Gaming Program seed grant, and from the Skoltech–MIT

Revised: Next Generation

Revised: Program.