Original:  3D Object Detection from Multi-view Images via 3D-to-2D Queries
Revised: DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract: We introduce a framework for multi-camera 3D object detection.
Revised: Abstract: We introduce a framework for multi-camera 3D object detection.

Original: Incontrast to existing works, which estimate 3D bounding boxes directly frommonocular images or use depth prediction networks to generate input for 3D object detection from 2D information, our method manipulates predictions directlyin 3D space.
Revised: Incontrast to existing works, which estimate 3D bounding boxes directly frommonocular images or use depth prediction networks to generate input for 3D object detection from 2D information, our method manipulates predictions directlyin 3D space.

Original: Our architecture extracts 2D features from multiple camera images and then uses a sparse set of 3D object queries to index into these 2D features, linking 3D positions to multi-view images using camera transformation matrices.
Revised: Our architecture extracts 2D features from multiple camera imagesand then uses a sparse set of 3D object queries to index into these 2D features,linking 3D positions to multi-view images using camera transformation matrices.

Original: Finally, our model makes a bounding box prediction per object query, using aset-to-set loss to measure the discrepancy between the ground-truth and the prediction.
Revised: Finally, our model makes a bounding box prediction per object query, using aset-to-set loss to measure the discrepancy between the ground-truth and the prediction.

Original: This top-down approach outperforms its bottom-up counterpart in whichobject bounding box prediction follows per-pixel depth estimation, since it doesnot suffer from the compounding error introduced by a depth prediction model.
Revised: This top-down approach outperforms its bottom-up counterpart in whichobject bounding box prediction follows per-pixel depth estimation, since it doesnot suffer from the compounding error introduced by a depth prediction model.

Original: Moreover, our method does not require post-processing such as non-maximumsuppression, dramatically improving inference speed.
Revised: Moreover, our method does not require post-processing such as non-maximumsuppression, dramatically improving inference speed.

Original: We achieve state-of-the-artperformance on the nuScenes autonomous driving benchmark.
Revised: We achieve state-of-the-artperformance on the nuScenes autonomous driving benchmark.

Original:  Introduction
Revised:  Introduction

Original: 3D object detection from visual information is a long-standing challenge for low-cost autonomousdriving systems.
Revised: 3D object detection from visual information is a long-standing challenge for low-cost autonomousdriving systems.

Original: While object detection from point clouds collected using modalities like LiDARbeneﬁts from information about the 3D structure of visible objects, the camera-based setting is evenmore ill-posed, since we must generate 3D bounding box predictions solely from the 2D informationcontained in RGB images.
Revised: While object detection from point clouds collected using modalities like LiDARbeneﬁts from information about the 3D structure of visible objects, the camera-based setting is evenmore ill-posed, since we must generate 3D bounding box predictions solely from the 2D informationcontained in RGB images.

Original: Existing methods [1, 2] typically build their detection pipelines purely from 2D computations.
Revised: Existing methods [1, 2] typically build their detection pipelines purely from 2D computations.

Original: Thatis, they predict 3D information like object pose and velocity using an object detection pipeline designed for 2D tasks (e.g., CenterNet [1], FCOS [3]), withoutdirectly considering 3D scene structureor sensor conﬁguration.
Revised: Thatis, they predict 3D information like object pose and velocity using an object detection pipeline designed for 2D tasks (e.g., CenterNet [1], FCOS [3]), without considering 3D scene structure or sensorconﬁguration.

Original: These methods require several post-processing steps to fuse predictionsacross cameras and to remove redundantbounding boxes, yielding a steep trade-off between efﬁciency and effectiveness.
Revised: These methods require several post-processing steps to fuse predictions across cameras and to remove redundant boxes, yielding a steep trade-off between efﬁciency and effectiveness.

Original: As an alternative to these 2D-based methods, some methods incorporatemore 3D computations into our object detection pipeline by applying a 3D reconstruction methodlike [4, 5, 6] to create a pseudo-LiDAR or range input of the scene from camera images.
Revised: As an alternative to these 2D-based methods, some methods incorporate more 3D computationsinto our object detection pipeline by applying a 3D reconstruction method like [4, 5, 6] to create apseudo-LiDAR or range input of the scene from camera images.

Original: Then,they could apply 3D object detection methods to this data as if it were collected directly from a 3Dsensor.
Revised: Then, they could apply 3D objectdetection methods to this data as if it were collected directly from a 3D sensor.

Original: This strategy, however, is subject to compounding errors [7]: poorly-estimated depth valueshave a strongly negative effect on the performance of 3D object detection, which also can exhibiterrors of its own.
Revised: This strategy, however, is subject to compounding errors [7]: poorly-estimated depth values have a strongly negativeeffect on the performance of 3D object detection, which also can exhibit errors of its own.

Original: In this paper, we propose a more graceful transition between 2D observations and 3D predictions forautonomous driving, which does not rely on a module for dense depth prediction.
Revised: In this paper, we propose a more graceful transition between 2D observations and 3D predictions forautonomous driving, which does not rely on a module for dense depth prediction.

Original: Our framework,termed MVOD (Multi-View 3D Detection), addresses this problem in a top-down fashion.
Revised: Our framework,termed DETR3D (Multi-View 3D Detection), addresses this problem in a top-down fashion.

Original: We link2D feature extraction and 3D object prediction via geometric back-projection with camera transformation matrices.
Revised: We link2D feature extraction and 3D object prediction via geometric back-projection with camera transformation matrices.

Original: Our method starts from a sparse set of object priors, shared across the datasetand learned end-to-end.
Revised: Our method starts from a sparse set of object priors, shared across the datasetand learned end-to-end.

Original: To gather scene-speciﬁc information, we back-project a set of referencepoints decoded from these object priors to each camera and fetch the corresponding image featuresextracted by a ResNet backbone [8].
Revised: To gather scene-speciﬁc information, we back-project a set of referencepoints decoded from these object priors to each camera and fetch the corresponding image featuresextracted by a ResNet backbone [8].

Original: The features collected from the image features of the reference points then interact with each other through a multi-head self-attention layer [9].
Revised: The features collected from the image features of the referencepoints then interact with each other through a multi-head self-attention layer [9].

Original: After a series ofself-attention layers, we read off bounding box parameters from every layer and use a set-to-set lossinspired by DETR [10] to evaluate performance.
Revised: After a series of distribute. self-attention layers, we read off bounding box parameters from every layer and use a set-to-set lossinspired by DETR [10] to evaluate performance.

Original: Our architecture does not perform point cloud reconstruction or explicit depth prediction from images, making it robust to errors in depth estimation.
Revised: Our architecture does not perform point cloud reconstruction or explicit depth prediction from images, making it robust to errors in depth estimation.

Original: Moreover, our method does not require anypost-processing, such as non-maximum suppression (NMS), improving efﬁciency and reducing reliance on hand-designed methods for cleaning its output.
Revised: Moreover, our method does not require anypost-processing, such as non-maximum suppression (NMS), improving efﬁciency and reducing reliance on hand-designed methods for cleaning its output.

Original: On the nuScenes dataset, our method(without NMS) is comparable with prior art (with NMS).
Revised: On the nuScenes dataset, our method(without NMS) is comparable with prior art (with NMS).

Original: In the camera overlap regions, our methodsigniﬁcantly outperforms others.
Revised: In the camera overlap regions, our methodsigniﬁcantly outperforms others.

Original: Contributions.
Revised: Contributions.

Original: We summarize our key contributions as follows:
Revised: We summarize our key contributions as follows:

Original: • We present a streamlined 3D object detection model from RGB images.
Revised: • We present a streamlined 3D object detection model from RGB images.

Original: Different from existingworks that combine object predictions from the different camera views in a ﬁnal stage, our method fuses information from all the camera views in each layer of computation.
Revised: Different from existingworks that combine object predictions from the different camera views in a ﬁnal stage, our methodfuses information from all the camera views in each layer of computation.

Original: To the best of ourknowledge, this is the ﬁrst attempt to cast multi-camera detection as 3D set-to-set prediction.•
Revised: To the best of ourknowledge, this is the ﬁrst attempt to cast multi-camera detection as 3D set-to-set prediction.•

Original: We introduce a module that connects 2D feature extraction and 3D bounding box prediction via
Revised: We introduce a module that connects 2D feature extraction and 3D bounding box prediction via

Original: backward geometric projection.
Revised: backward geometric projection.

Original: It does not suffer from inaccurate depth predictions from a sec- ondary network, and seamlessly uses information from multiple cameras by back-projecting 3D information onto all available frames.
Revised: It does not suffer from inaccurate depth predictions from a secondary network, and seamlessly uses information from multiple cameras by back-projecting 3Dinformation onto all available frames.

Original: • Similarly to Object DGCNN [11], our method does not require post-processing such as per-imageor global NMS, and it is on par with existing NMS-based methods.
Revised: • Similarly to Object DGCNN [11], our method does not require post-processing such as per-imageor global NMS, and it is on par with existing NMS-based methods.

Original: In the camera overlap regions,our method outperforms others by a substantial margin.•
Revised: In the camera overlap regions,our method outperforms others by a substantial margin.•

Original: We will release our code to promote reproducibility and future research.
Revised: We will release our code to promote reproducibility and future research.

Original:  Related Work
Revised:  Related Work

Original: 2D object detection.
Revised: 2D object detection.

Original: RCNN [12] pioneered object detection using deep learning.
Revised: RCNN [12] pioneered object detection using deep learning.

Original: It feeds a setof pre-selected object proposals into a convolutional neural network (CNN) and predicts boundingbox parameters accordingly.
Revised: It feeds a setof pre-selected object proposals into a convolutional neural network (CNN) and predicts boundingbox parameters accordingly.

Original: Although this method exhibits surprising performance, it is an order ofmagnitude slower than others because it performs a ConvNet forward pass for each object proposal.
Revised: Although this method exhibits surprising performance, it is an order ofmagnitude slower than others because it performs a ConvNet forward pass for each object proposal.

Original: To ﬁx this issue, Fast RCNN [13] introduces a shared learnable CNN to process the entire image at asingle forward pass.
Revised: To ﬁx this issue, Fast RCNN [13] introduces a shared learnable CNN to process the entire image at asingle forward pass.

Original: To further improve performance and speed, Faster RCNN [13] includes a regionproposal network (RPN) that shares full-image convolutional features with the detection network,thus enabling nearly cost-free region proposals.
Revised: To further improve performance and speed, Faster RCNN [13] includes a regionproposal network (RPN) that shares full-image convolutional features with the detection network,thus enabling nearly cost-free region proposals.

Original: Mask RCNN [14] incorporates a mask predictionbranch to enable instance segmentation in parallel.
Revised: Mask RCNN [14] incorporates a mask predictionbranch to enable instance segmentation in parallel.

Original: These methods typically involve multi-stagereﬁnements and can be slow in practice.
Revised: These methods typically involve multi-stagereﬁnements and can be slow in practice.

Original: Different from these multi-stage methods, SSD [15] and
Revised: Different from these multi-stage methods, SSD [15] and

Original: YOLO [16] perform dense predictions in a single shot.
Revised: YOLO [16] perform dense predictions in a single shot.

Original: Although they are signiﬁcantly faster thanthe alternatives above, they still rely on NMS to remove redundant box predictions.
Revised: Although they are signiﬁcantly faster thanthe alternatives above, they still rely on NMS to remove redundant box predictions.

Original: These methodspredict bounding boxes w.r.t.
Revised: These methodspredict bounding boxes w.r.t.

Original: pre-deﬁned anchors.
Revised: pre-deﬁned anchors.

Original: CenterNet [1] and FCOS [3] change the paradigmby shifting from per-anchor prediction to per-pixel prediction, signiﬁcantly simplifying the commonobject detection pipeline.
Revised: CenterNet [1] and FCOS [3] change the paradigmby shifting from per-anchor prediction to per-pixel prediction, signiﬁcantly simplifying the commonobject detection pipeline.

Original: Set-based object detection.
Revised: Set-based object detection.

Original: DETR [10] casts object detection as a set-to-set problem.
Revised: DETR [10] casts object detection as a set-to-set problem.

Original: It employs a
Revised: It employs a

Original: Transformer [9] to capture feature and object interactions.
Revised: Transformer [9] to capture feature and object interactions.

Original: DETR learns to assign predictions to a setof ground-truth boxes; thus, it does not require post-processing to ﬁlter out redundant boxes.
Revised: DETR learns to assign predictions to a setof ground-truth boxes; thus, it does not require post-processing to ﬁlter out redundant boxes.

Original: Onecritical drawback of DETR, however, is that it requires a signiﬁcant amount of training time.
Revised: Onecritical drawback of DETR, however, is that it requires a signiﬁcant amount of training time.

Original: Deformable DETR [17] analyzes DETR’s slow convergence and proposes a deformable self-attentionmodule to localize features and accelerate training.
Revised: Deformable DETR [17] analyzes DETR’s slow convergence and proposes a deformable self-attentionmodule to localize features and accelerate training.

Original: Concurrently, [18] attributes the slow convergence of DETR to the set-based loss and the Transformer cross attention mechanism.
Revised: Concurrently, [18] attributes the slow convergence of DETR to the set-based loss and the Transformer cross attention mechanism.

Original: They propose two variants, TSP-FCOS and TSP-RCNN, to overcome these problematic aspects.
Revised: They propose two variants, TSP-FCOS and TSP-RCNN, to overcome these problematic aspects.

Original: SparseRCNN [19] incorporates set prediction into a RCNN-style pipeline; it outperforms multi-stage objectdetection without NMS.
Revised: SparseRCNN [19] incorporates set prediction into a RCNN-style pipeline; it outperforms multi-stage objectdetection without NMS.

Original: OneNet [20] studies an interesting phenomenon: dense-based object detectors can be made NMS-free after they are equipped with a minimum-cost set loss.
Revised: OneNet [20] studies an interesting phenomenon: dense-based object detectors can be made NMS-free after they are equipped with a minimum-cost set loss.

Original: For 3D domains,
Revised: For 3D domains,

Original: Object DGCNN [11] studies 3D object detection from point clouds.
Revised: Object DGCNN [11] studies 3D object detection from point clouds.

Original: It models 3D object detectionas message passing on a dynamic graph, generalizing the DGCNN framework to predict a set ofobjects.
Revised: It models 3D object detectionas message passing on a dynamic graph, generalizing the DGCNN framework to predict a set ofobjects.

Original: Similar to DETR, Object DGCNN is also NMS-free.
Revised: Similar to DETR, Object DGCNN is also NMS-free.

Original: Monocular 3D object detection.
Revised: Monocular 3D object detection.

Original: An early method for 3D detection from RGB images is
Revised: An early method for 3D detection from RGB images is

Original: Mono3D [21], which uses semantic and shape cues to select from a collection of 3D proposals, [Figure] using scene constraints and additional priors at training time.
Revised: Mono3D [21], which uses semantic and shape cues to select from a collection of 3D proposals,using scene constraints and additional priors at training time.

Original: [22] uses the birds-eye-view (BEV)
Revised: [22] uses the birds-eye-view (BEV)

Original: [Figure]
Revised: [Figure]

Original: for monocular 3D detection, and [23] leverages 2D detections for 3D bounding box regression via [Figure] the minimization of 2D-3D projection error.
Revised: for monocular 3D detection, and [23] leverages 2D detections for 3D bounding box regression viathe minimization of 2D-3D projection error.

Original: The use of 2D detectors as a starting point for 3Dcomputation recently has become a standard approach [24, 25].
Revised: The use of 2D detectors as a starting point for 3Dcomputation recently has become a standard approach [24, 25].

Original: Other works also explore advances [Figure] in differentiable rendering [26] or 3D keypoint detection [27, 28, 1] to enable state-of-the-art 3D
Revised: Other works also explore advancesin differentiable rendering [26] or 3D keypoint detection [27, 28, 1] to enable state-of-the-art 3D

Original: [Figure]
Revised: [Figure]

Original: object detection performance.
Revised: object detection performance.

Original: All these methods operate in a monocular setting, and extensions to [Figure] multiple cameras are done by independently processing each frame before merging the outputs in a
Revised: All these methods operate in a monocular setting, and extensions tomultiple cameras are done by independently processing each frame before merging the outputs in a

Original: [Figure]
Revised: [Figure]

Original: post-processing stage.
Revised: post-processing stage.

Original: [Figure]
Revised: [Figure]

Original:  Multi-view 3D Object Detection
Revised:  Multi-view 3D Object Detection

Original: [Figure]
Revised: [Figure]

Original: 3.1 Overview
Revised: 3.1 Overview

Original: [Figure]
Revised: [Figure]

Original: Our architecture inputs RGB images collected from a set of cameras whose projection matrices (thecombination of intrinsics and relative extrinsics) are known, and it outputs a set of 3D bounding boxparameters for the objects in the scene.
Revised: Our architecture inputs RGB images collected from a set of cameras whose projection matrices (thecombination of intrinsics and relative extrinsics) are known, and it outputs a set of 3D bounding boxparameters for the objects in the scene.

Original: In contrast to past approaches, we build our architecturebased on a few high-level desiderata:
Revised: In contrast to past approaches, we build our architecture [Figure] based on a few high-level desiderata:

Original:  • We incorporate 3D information into intermediate computations within our architecture, rather than 
Revised: [Figure] • We incorporate 3D information into intermediate computations within our architecture, rather than [Figure]

Original: performing purely 2D computations in the image plane.
Revised: performing purely 2D computations in the image plane.

Revised: [Figure]

Original: • We do not estimate dense 3D scene geometry, avoiding associated reconstruction errors.
Revised: • We do not estimate dense 3D scene geometry, avoiding associated reconstruction errors.

Original: • We avoid post-processing steps such as NMS.
Revised: • We avoid post-processing steps such as NMS

Original: We address these desiderata using a new set prediction module, which links 2D feature extractionand 3D box prediction by alternating between 2D and 3D computations.
Revised: We address these desiderata using a new set prediction module, which links 2D feature extraction and 3D box prediction by alternating between 2D and 3D computations.

Original: Our model contains three critical components, illustrated in Figure 1.
Revised: Our model containsthree critical components, illustrated in Figure 1.

Original: First, following commonpractice in 2D vision, it extracts features from the camera images using a shared ResNet [8] backbone.
Revised: First, following common practice in 2D vision,it extracts features from the camera images using a shared ResNet [8] backbone.

Original: Optionally, these features are enhanced by a feature pyramid network (FPN) [29] (§3.2).
Revised: Optionally, thesefeatures are enhanced by a feature pyramid network (FPN) [29] (§3.2).

Original: Second, a detection head (§3.3)—our main contribution—links the computed 2D features to a setof 3D bounding box predictions in a geometry-aware manner (§3.3).
Revised: Second, a detection head(§3.3)—our main contribution—links the computed 2D features to a set of 3D bounding box predictions in a geometry-aware manner (§3.3).

Original: Each layer of the detectionhead starts from a sparse set of object queries, which are learned from the data.
Revised: Each layer of the detection head starts from a sparse setof object queries, which are learned from the data.

Original: Each object queryencodes a 3D location, which is projected to the camera planes and used to collect image featuresvia bilinear interpolation.
Revised: Each object query encodes a 3D location, whichis projected to the camera planes and used to collect image features via bilinear interpolation.

Original: Similarly to DETR [10], we then use multi-head attention [9] to reﬁne theobject queries by incorporating object interactions.
Revised: Similarly to DETR [10], we then use multi-head attention [9] to reﬁne the object queries by incorporatingobject interactions.

Original: This layer is repeated multiple times, alternatingbetween feature sampling and object query reﬁnement.
Revised: This layer is repeated multiple times, alternating between feature sampling andobject query reﬁnement.

Original: After the last layer, we evaluate a set-to-setloss [30, 10] to train the network (§3.4).
Revised: Finally, we evaluate a set-to-set loss [30, 10] to train the network (§3.4).

Original: 3.2 Feature Learning
Revised: 3.2 Feature Learning

Original: Our model starts with a set of images
Revised: Our model starts with a set of images

Original: I =
Revised: I =

Original: { im 1 , . .
Revised: { im 1 , . .

Original: , im K } ⊂ R H
Revised: , im K } ⇢ R H

Original: im 
Revised: im ⇥

Original: × W im × 3 (captured by surrounding cameras) , camera matrices T =
Revised: W im ⇥ 3 (captured by surrounding cameras), camera matrices T =

Original: { T 1 , . . .
Revised: { T 1 , . . .

Original: , T K } ⊂ R 3 × 4 , ground-truth bounding boxes
Revised: , T K } ⇢ R 3 ⇥ 4 , ground-truth bounding boxes

Original: B = { b 1 , . . .
Revised: B = { b 1 , . . .

Original: , b j , . . .
Revised: , b j , . . .

Original: , b M } ⊂ R 9 , and categorical labels C
Revised: , b M } ⇢ R 9 , and categorical labels C

Original: = { c 1 , . . .
Revised: = { c 1 , . . .

Original: , c j , . .
Revised: , c j , . .

Original:  c M } ⊂ Z .
Revised: . , c M } ⇢ Z .

Original: Each b jcontains position, size, heading angle, and velocity in the birds-eye view (BEV); our model aims topredict these boxes and their labels from the these images.
Revised: Each b j 128 contains position, size, heading angle, and velocity in the birds-eye view (BEV); our model aims to predict these boxes and their labels from the these images.

Original: We do not use point clouds, which areusually captured by high-end LiDAR.
Revised: We do not use point clouds, which areusually captured by high-end LiDAR.

Original: These images are encoded with a ResNet [8] and a FPN [29] into four sets of features F 1 , F 2 , F 3 , F 4 .
Revised: These images are encoded with a ResNet [8] and a FPN [29] into four sets of features F 1 , F 2 , F 3 , F 4 .

Original:  Each set F k = { f k 1 , . . .
Revised: 132 Each set F k = { f k 1 , . . .

Original: , f k 6 } ⊂ R H × W × C corresponds to a level of features of the 6 images output by the FPN.
Revised: , f k 6 } ⇢ R H ⇥ W ⇥ C corresponds to a level of features of the 6 images.

Original: These multi-scale features provide rich information to recognize objects of differentsizes.
Revised: These multi-scale features provide rich information to recognize objects of different sizes.

Original: Next, we detail our approach to transform these 2D features into 3D using a novel set prediction module.
Revised: Next, wedetail our approach to transform these 2D features into 3D using a novel set prediction module.

Original: 3.3 Detection Head
Revised: 3.3 Detection Head

Original: Existing methods for detecting objects from camera input typically employ a bottom-up approach,which predicts a dense set of bounding boxes per image, ﬁlters redundant boxes between the images,and aggregates predictions across cameras in a post-processing step.
Revised: Existing methods for detecting objects from camera input typically employ a bottom-up approach,which predicts a dense set of bounding boxes per image, ﬁlters redundant boxes between the images,and aggregates predictions across cameras in a post-processing step.

Original: This paradigm has two crucialdrawbacks: dense bounding box prediction requires accurate depth perception, which itself is achallenging problem; and NMS-based redundancy removal and aggregation are non-parallelizableoperations that introduce signiﬁcant inference overhead.
Revised: This paradigm has two crucialdrawbacks: dense bounding box prediction requires accurate depth perception, which itself is achallenging problem; and NMS-based redundancy removal and aggregation are non-parallelizableoperations that introduce signiﬁcant inference overhead.

Original: We address these issues using a top-downobject detection head described below.
Revised: We address these issues using a top-downobject detection head described below.

Original: Analogously to [11, 17], MVOD is iterative ; it uses L layers with set-based computations to producebounding box estimates from 2D feature maps.
Revised: Analogously to [11, 17], DETR3D is iterative ; it uses L layers with set-based computations toproduce bounding box estimates from 2D feature maps.

Original: Each layer includes the following steps:
Revised: Each layer includes the following steps:

Original: 1. predict a set of bounding box centers associated with object queries;
Revised: 1. predict a set of bounding box centers associated with object queries;

Original: 2. project these centers into all the feature maps using the camera transformation matrices;
Revised: 2. project these centers into all the feature maps using the camera transformation matrices;

Original: sample features via bilinear interpolation and incorporate them into object queries; and
Revised: sample features via bilinear interpolation and incorporate them into object queries; and

Original: describe object interactions using multi-head attention.
Revised: describe object interactions using multi-head attention.

Original: Motivated by DETR [10], each layer (cid:96) ∈ { 0 , . . .
Revised: Motivated by DETR [10], each layer ` 2 { 0 , . . .

Original: , L − 1 } operates on a set of object queries Q (cid:96) = { q (cid:96) 1 , . . .
Revised: , L  1 } operates on a set of object queries Q ` ={ q ` 1 , . . .

Original: , q (cid:96)M ∗ } ⊂ R C , producing a new set Q (cid:96) +1 .
Revised: , q ` M ⇤ } ⇢ R C , producing a new set Q ` +1 .

Original: A reference point c (cid:96)i ∈ R 3 is decoded from aobject query q (cid:96)i as follows:
Revised: A reference point c ` i 2 R 3 is decoded from a 151 object query q ` i as follows:

Original: [Equation]
Revised: [Equation]

Original:  where Φ ref is a neural network.
Revised: ) where  ref is a neural network.

Original: c (cid:96)i can be thought of a hypothesis for the center of the i -th box.
Revised: c ` i can be thought of a hypothesis for the center of the i -th box.

Original: Next, we acquire image features corresponding to c (cid:96)i to reﬁne and predict the ﬁnal bounding box.
Revised: Next, we acquire image features corresponding to c ` i to reﬁne and predict the ﬁnal bounding box.

Original: Then, c (cid:96)i (or more accurately its homogeneous counterpart c ∗ (cid:96)i ) is projected into each one of theimages using the camera transformation matrices:
Revised: Then, c ` i (or more accurately its homogeneous counterpart c ⇤ ` i ) is projected into each one of theimages using the camera transformation matrices:

Original: [Equation]
Revised: [Equation]

Original: where ⊕ denotes concatenation, and c (cid:96)mi is the projection of the reference point onto the m -thcamera.
Revised: where  denotes concatenation, and c ` mi is the projection of the reference point onto the m -thcamera.

Original: To remove the effects of the feature map size and gather features across different levels, wenormalize c (cid:96)mi to [ − 1 , 1] .
Revised: To remove the effects of the feature map size and gather features across different levels, wenormalize c ` mi to [  1 , 1] .

Original: Next, the images features are collected by
Revised: Next, the images features are collected by

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where f (cid:96)kmi is the feature for i -th point from k -th level of m -th camera at (cid:96) -th layer.
Revised: where f ` kmi is the feature for i -th point from k -th level of m -th camera at ` -th layer.

Original: A given reference point is not necessarily visible in all the camera images, so we need some heuristics to ﬁlter invalid points.
Revised: A given reference point is not necessarily visible in all the camera images, so we need some heuristics to ﬁlter invalid points.

Original: To that end, we deﬁne a binary value σ (cid:96)kmi , which is determined basedon whether a reference point is projected outside an image plane.
Revised: To that end, we deﬁne a binary value  ` kmi , which is determined basedon whether a reference point is projected outside an image plane.

Original: The ﬁnal feature f (cid:96)i and objectquery in next layer q ( (cid:96) +1) i are given by[Equation] and q
Revised: The ﬁnal feature f ` i and objectquery in next layer q ( ` +1) i are given by

Original: [Equation]
Revised: [Equation]

Original: 4) where (cid:15) is a small number to avoid division by zero.
Revised: where ✏ is a small number to avoid division by zero.

Original: Finally, for each object query q (cid:96)i , we predict abounding box ˆ b (cid:96)i and its categorical label ˆ c (cid:96)i with two neural networks Φ reg (cid:96) and Φ cls (cid:96) :ˆ b [Equation] = Φ reg (cid:96)
Revised: Finally, for each object query q ` i , we predict abounding box ˆ b ` i and its categorical label ˆ c ` i with two neural networks  reg ` and  cls ` :

Original: [Equation]
Revised: [Equation]

Original: q
Revised: and

Original: [Equation]
Revised: [Equation]

Original: and ˆ c (cid:96)i =

Original: Φ cls (cid:96) ( q (cid:96)i ) .

Original: We compute the loss for the predictions ˆ B (cid:96) =
Revised: We compute the loss for the predictions ˆ B [Equation]

Original: { ˆ b (cid:96) 1 , . . . , ˆ b (cid:96)j , . . . , ˆ b (cid:96)M ∗ } ⊂ R 9 and ˆ C (cid:96) = { ˆ c (cid:96) 1 , . . .
Revised: [Equation]

Original: , ˆ c (cid:96)j , . . . , ˆ c (cid:96)M } ⊂ Z from every layer during training. During inference, we only use theoutputs from the last layer.
Revised: [Table]

Original: 3.4 Loss
Revised: 3.4 Loss

Original: Following [30, 10], we use a set-to-set loss to measure the discrepancy between the prediction set( ˆ B (cid:96) , ˆ C (cid:96) ) and the ground-truth set ( B , C ) .
Revised: Following [30, 10], we use a set-to-set loss to measure the discrepancy between the prediction set( ˆ B ` , ˆ C ` ) and the ground-truth set ( B , C ) .

Original: This loss consists of two parts: a focal loss [31] for theclass labels and a L 1 loss for the bounding box parameters.
Revised: This loss consists of two parts: a focal loss [31] for theclass labels and a L 1 loss for the bounding box parameters.

Original: For notational convenience, we drop the(cid:96) subscript in ˆ B (cid:96) and ˆ C (cid:96) .
Revised: For notational convenience, we drop the` subscript in ˆ B ` and ˆ C ` .

Original: The number of ground-truth boxes M is typically smaller than the numberof predictions M ∗ , so we pad the set of ground-truth boxes with ∅ s (no object) up to M ∗ for easeof computation.
Revised: The number of ground-truth boxes M is typically smaller than the numberof predictions M ⇤ , so we pad the set of ground-truth boxes with ? s (no object) up to M ⇤ for ease ofcomputation.

Original: We establish a correspondence between the ground-truth and the prediction via abipartite matching problem: 
Revised: We establish a correspondence between the ground-truth and the prediction via a bipartite matching problem:  ⇤ = arg min  2 P P

Original: [Equation]
Revised: [Equation]

Original: where P denotes the set of permutations, ˆ p σ ( j ) ( c j ) is the probability of class c j for the predictionwith index σ ( j ) , and L box is the L 1 loss for bounding box parameters.
Revised: L box ( [Equation] ˆ b  ( j ) ) ,where P denotes the set of permutations, ˆ p  ( j ) ( c j ) is the probability of class c j for the predictionwith index  ( j ) , and L box is the L 1 loss for bounding box parameters.

Original: We use the Hungarianalgorithm [32] to solve this assignment problem, as in [30, 10], yielding the following set-to-setloss:
Revised: We use the Hungarianalgorithm [32] to solve this assignment problem, as in [30, 10], yielding the set-to-set loss L sup = P Nj = [Equation]

Revised:  log

Original: [Equation]
Revised: [Equation]

Revised: p

Revised: [Equation]

Revised: c

Revised: [Equation]

Revised: +

Revised: [Equation]

Revised: { c j = ?

Revised: [Equation]

Revised: L box ( b

Revised: [Equation]

Revised: [Equation]

Revised:

Revised: [Equation]

Revised: j ) ) .

Original:  Experiments
Revised:  Experiments

Original: We present our results as follows:
Revised: We present our results as follows:

Original: ﬁrst, we detail the dataset, metrics, and implementation in §4.1;then
Revised: ﬁrst, we detail the dataset, metrics, and implementation in §4.1;then

Original: we compare our method to existing works in §4.2; we benchmark the performance of differentmodels in camera overlap regions in §4.3; we compare to a forward prediction model in §4.4; andwe provide additional analysis and ablations in §4.5.
Revised: we compare our method to existing works in §4.2; we benchmark the performance of differentmodels in camera overlap regions in §4.3; we compare to a forward prediction model in §4.4; andwe provide additional analysis and ablations in §4.5.

Original: 4.1 Implementation Details
Revised: 4.1 Implementation Details

Original: Dataset.
Revised: Dataset.

Original: We test our method on the nuScenes dataset [33].
Revised: We test our method on the nuScenes dataset [33].

Original: nuScenes consists of 1,000 sequences; each sequence is roughly 20s long, with a sampling rate of 20 frames/second. Each sample contains images from 6 cameras [front left, front, front right, back left, back,back right] . Camera parameters including intrinsics and extrinsics are available. nuScenes provides annotations every 0.5s; in total there are 28k, 6k, and 6k annotated samples for training, validation, and testing, respectively.
Revised: nuScenes consists of 1,000 se-

Original: 10 from the total 23 classes are available to compute the metrics.

Original: Metrics.
Revised: Metrics.

Original: We follow the ofﬁcial evaluation protocol provided by nuScenes.
Revised: We follow the ofﬁcial evaluation protocol provided by nuScenes.

Original: We evaluate averagetranslation error (ATE), average scale error (ASE), average orientation error (AOE), average velocityerror (AVE), and average attribute error (AAE).
Revised: We evaluate averagetranslation error (ATE), average scale error (ASE), average orientation error (AOE), average velocityerror (AVE), and average attribute error (AAE).

Original: These metrics are true positive metrics (TP metrics)and computed in the physical unit.
Revised: These metrics are true positive metrics (TP metrics)and computed in the physical unit.

Original: In addition, we measure mean average precision (mAP).
Revised: In addition, we measure mean average precision (mAP).

Original: Tocapture all aspects of the detection task, a consolidated scalar metric–the nuScenes Detection Score(NDS)
Revised: Tocapture all aspects of the detection task, a consolidated scalar metric–the nuScenes Detection Score(NDS)

Original: [33]–is deﬁned as 
Revised: [33]–is deﬁned as NDS = 110 [5mAP +

Original: [Equation]
Revised: [Equation]

Original: Model.
Revised: Model.

Original: Our model consists of a ResNet [8] feature extractor, a FPN, and a MVOD detection head.
Revised: Our model consists of a ResNet [8] feature extractor, a FPN, and a DETR3D detection head.

Original: We use ResNet101 with deformable convolutions [34] in the 3rd stage and 4th stage.
Revised: We use ResNet101 with deformable convolutions [34] in the 3rd stage and 4th stage.

Original: The FPN [29]takes features output by the ResNet and produces 4 feature maps whose sizes are 1 / 8 , 1 / 16 , 1 / 32 , and 205/ 64 of the input image sizes.
Revised: The FPN [29]takes features output by the ResNet and produces 4 feature maps whose sizes are 1 / 8 , 1 / 16 , 1 / 32 , and 202/ 64 of the input image sizes.

Original: The MVOD detection head consists of 6 layers, where each layer is acombination of a feature reﬁnement step and a multi-head attention layer.
Revised: The DETR3D detection head consists of 6 layers, where each layer is acombination of a feature reﬁnement step and a multi-head attention layer.

Original: The hidden dimension ofthe MVOD detection head is 256.
Revised: The hidden dimension ofthe DETR3D detection head is 256.

Original: Finally, two sub-networks predict bounding box parameters anda class label per object query; each sub-network consists of two fully connected layers with hiddendimensions 256.
Revised: Finally, two sub-networks predict bounding box parameters anda class label per object query; each sub-network consists of two fully connected layers with hiddendimensions 256.

Original: We use LayerNorm [35] in the detection head.[Table] [Table]
Revised: We use LayerNorm [35] in the detection head.

Original: Training & inference.
Revised: Training & inference.

Original: We use AdamW [36] to train the whole pipeline.
Revised: We use AdamW [36] to train the whole pipeline.

Original: The weight decay is 10 − 4 .
Revised: The weight decay is 10  4 .

Original: We use an initial learning rate 10 − 4 , which is decreased to 10 − 5 and 10 − 6 at 8th and 11th epochs.
Revised: We use an initial learning rate 10  4 , which is decreased to 10  5 and 10  6 at 8th and 11th epochs.

Original: The model is trained for 12 epochs in total on 8 RTX 3090 GPUs and the per-GPU batch size is 1.
Revised: The model is trained for 12 epochs in total on 8 RTX 3090 GPUs and the per-GPU batch size is 1.

Original: The training procedure takes roughly 18 hours.
Revised: The training procedure takes roughly 18 hours.

Original: We do not use any post-processing such as NMSduring inference.
Revised: We do not use any post-processing such as NMSduring inference.

Original: For evaluation, we use the nuScenes evalutation toolkit.
Revised: For evaluation, we use the nuScenes evalutation toolkit.

Original: 4.2 Comparison to Existing Works
Revised: 4.2 Comparison to Existing Works

Original: We compare to previous state-of-the-art methods CenterNet [1] and FCOS3D [2].
Revised: We compare to previous state-of-the-art methods CenterNet [1] and FCOS3D [2].

Original: CenterNet is ananchor-free 2D detection method that makes dense predictions in a high resolution feature map.
Revised: CenterNet is ananchor-free 2D detection method that makes dense predictions in a high resolution feature map.

Original: FCOS3D employs a FCOS [3] pipeline to make per-pixel predictions.
Revised: FCOS3D employs a FCOS [3] pipeline to make per-pixel predictions.

Original: These methods both turn 3Dobject detection into a 2D problem, and in doing so ignore scene geometry and sensor conﬁguration.
Revised: These methods both turn 3Dobject detection into a 2D problem, and in doing so ignore scene geometry and sensor conﬁguration.

Revised: To perform multi-view object detection, these methods have to process each image independently,

Original: To perform multi-view object detection, these methods have to process each image independently,and use both per-image and global NMS to remove redundant boxes in each view and in the overlapregions respectively.
Revised: [Table] head is initialized randomly. [Table] and use both per-image and global NMS to remove redundant boxes in each view and in the overlapregions respectively.

Original: As shown in Table 1, our method is on a par with these methods even thoughwe do not use any post-processing.
Revised: As shown in Table 1, our method is on a par with these methods even thoughwe do not use any post-processing.

Original: Our method performs worse than FCOS3D in terms of mATE.
Revised: Our method performs worse than FCOS3D in terms of mATE.

Original: We suspect this is because FCOS3D directly predicts bounding box depth, which leads to strongsupervision on object translation.
Revised: We suspect this is because FCOS3D directly predicts bounding box depth, which leads to strongsupervision on object translation.

Original: Also, FCOS3D uses disentangled heads for different boundingbox parameters, which can increase performance.
Revised: Also, FCOS3D uses disentangled heads for different boundingbox parameters, which can increase performance.

Original: 4.3 Comparison in Overlap Regions
Revised: 4.3 Comparison in Overlap Regions

Original: A great challenge lies in the overlap regions where objects are more likely to be cut off.
Revised: A great challenge lies in the overlap regions where objects are more likely to be cut off.

Original: Our methodconsiders all cameras simultaneously, while FCOS3D predicts bounding boxes per camera individually.
Revised: Our methodconsiders all cameras simultaneously, while FCOS3D predicts bounding boxes per camera individually.

Original: To further demonstrate the advantages of fused inference, we calculate the metrics for boxesfalling into the camera overlaps.
Revised: To further demonstrate the advantages of fused inference, we calculate the metrics for boxesfalling into the camera overlaps.

Original: To compute the metrics, we select boxes whose 3D center is visibleto multiple cameras.
Revised: To compute the metrics, we select boxes whose 3D center is visibleto multiple cameras.

Original: On the validation set, there are 18,147 such boxes, 9.7% of the total.
Revised: On the validation set, there are 18,147 such boxes, 9.7% of the total.

Original: Table 2shows the results; our method outperforms FCOS3D remarkably in terms of NDS scores in thissetting.
Revised: Table 2shows the results; our method outperforms FCOS3D remarkably in terms of NDS scores in thissetting.

Original: This conﬁrms that our integrated prediction approach is more effective.
Revised: This conﬁrms that our integrated prediction approach is more effective.

Original: 4.4 Comparison to pseudo-LiDAR Methods
Revised: 4.4 Comparison to pseudo-LiDAR Methods

Original: Another way to perform 3D object detection is by generating pseudo-LiDAR point clouds frommulti-view images using a depth prediction model.
Revised: Another way to perform 3D object detection is by generating pseudo-LiDAR point clouds frommulti-view images using a depth prediction model.

Original: On the nuScenes dataset, there are no publiclyavailable pseudo-LiDAR works for us to make a direct comparison.
Revised: On the nuScenes dataset, there are no publiclyavailable pseudo-LiDAR works for us to make a direct comparison.

Original: Hence, we implement a baseline ourselves to verify that our approach is more effective than explicit depth prediction.
Revised: Hence, we implement a baseline ourselves to verify that our approach is more effective than explicit depth prediction.

Original: We usea pre-trained PackNet [6] network to predict dense depth maps from all six cameras and then convert these depth maps into point clouds using the camera transformations.
Revised: We usea pre-trained PackNet [6] network to predict dense depth maps from all six cameras and then convert these depth maps into point clouds using the camera transformations.

Original: We also experimentedwith a self-supervised PackNet model with velocity supervision (as in the original paper), but wefound that ground-truth depth supervision yielded more realistic point clouds and therefore used a [Table] [Figure] supervised model as baseline.
Revised: We also experimentedwith a self-supervised PackNet model with velocity supervision (as in the original paper), but wefound that ground-truth depth supervision yielded more realistic point clouds and therefore used asupervised model as baseline.

Original: For 3D detection, we employ the recently-proposed CenterPoint architecture [38].
Revised: For 3D detection, we employ the recently-proposed CenterPoint architecture [38].

Original: Conceptually, this pipeline is a variant of pseudo-LiDAR [39].
Revised: Conceptually, this pipeline is a variant of pseudo-LiDAR [39].

Original: Table 3 shows theresults; we conclude that this pseudo-LiDAR method underperforms ours signiﬁcantly even whendepth estimates are generated by a state-of-the-art model.
Revised: Table 3 shows theresults; we conclude that this pseudo-LiDAR method underperforms ours signiﬁcantly even whendepth estimates are generated by a state-of-the-art model.

Original: One possible explanation is that pseudoLiDAR object detectors suffer from compounding errors introduced by inaccurate depth prediction,that in turn is known to overﬁt to training data and generalizes poorly to other distributions [7]. 
Revised: One possible explanation is that pseudoLiDAR object detectors suffer from compounding errors introduced by inaccurate depth prediction,that in turn is known to overﬁt to training data and generalizes poorly to other distributions [7]. [Table] [Table] [Figure]

Original: 4.5 Ablation & Analysis
Revised: 4.5 Ablation & Analysis

Original: We provide a visualization of object query reﬁnement in Figure 2.
Revised: We provide a visualization of object query reﬁnement in Figure 2.

Original: We visualize bounding boxesdecoded from the object queries in each layer.
Revised: We visualize bounding boxesdecoded from the object queries in each layer.

Original: The predicted bounding boxes get closer to theground-truth as we go into deeper layers in the model.
Revised: The predicted bounding boxes get closer to theground-truth as we go into deeper layers in the model.

Revised: Also, the leftmost ﬁgure shows the learnedobject query priors shared by all data.

Original: We also provide quantitative results in Table 4, which shows that iterative reﬁnement indeed improves performance signiﬁcantly.
Revised: We also provide quantitative results in Table 4, which showsthat iterative reﬁnement indeed improves performance signiﬁcantly.

Original: Thissuggests that iterative reﬁnement is both beneﬁcial and necessary to fully leverage our proposedarchitecture.
Revised: This suggests that iterativereﬁnement is both beneﬁcial and necessary to fully leverage our proposed architecture.

Original: Also, the leftmost ﬁgure shows the learned object query priors shared by all data.
Revised: Furthermore,we provide ablations on the number of object queries in Table 5; increasing the number queriesconsistently improves the performance until it gets saturated at 900. Finally, Table 6 shows theresults with different backbones.

Original: We also provide qualitative results in Figure 3 to facilitate an intuitive understanding of model performance.
Revised: We also provide qualitative results in Figure 3 to facilitate an intuitive understanding of model performance.

Original: We project the predicted bounding boxes into 6 cameras as well as a BEV perspective.
Revised: We project the predicted bounding boxes into 6 cameras as well as a BEV perspective.

Original: Ingeneral, our method generates reasonable results and even detects relatively small objects.
Revised: Ingeneral, our method generates reasonable results and even detects relatively small objects.

Original: However,our method still exhibits substantial translation error (in line with results in Table 4.2):
Revised: However,our method still exhibits substantial translation error (in line with results in Table 4.2):

Original: Although ourmodel avoids explicit depth prediction, depth estimation is still a core challenging in this problem.
Revised: Although ourmodel avoids explicit depth prediction, depth estimation is still a core challenging in this problem.

Original: [Table]
Revised: [Table]

Original: P r e d i c ti on

Original: P r e d i c

Original: [Table]
Revised: [Table]

Original:  on
Revised: P r e d i c ti o n P r e d i c ti on

Original: G r oun d - t r u t h G r ound - t r u t h
Revised: G r ound - t r u t h G r ound - t r u t h

Original: P r e d i c ti on
Revised: P r e d i c ti on

Original: G r ound - t r u t h
Revised: G r ound - t r u t h

Original:  Conclusion
Revised:  Conclusion

Original: We propose a new paradigm to address the ill-posed inverse problem of recovering 3D informationfrom 2D images.
Revised: We propose a new paradigm to address the ill-posed inverse problem of recovering 3D informationfrom 2D images.

Original: In this setting, the input signal lacks essential information for models to makeeffective predictions without priors learned from data.
Revised: In this setting, the input signal lacks essential information for models to makeeffective predictions without priors learned from data.

Original: While other methods either operate solely on2D computations or use additional depth networks to reconstruct the scene, ours operates in 3D spaceand uses backward projection to retrieve image features as needed.
Revised: While other methods either operate solely on2D computations or use additional depth networks to reconstruct the scene, ours operates in 3D spaceand uses backward projection to retrieve image features as needed.

Original: The beneﬁts of our approach aretwo-fold: (1) it eliminates the need for middle-level representations (e.g., predicted depth mapsor point clouds), which can be a source of compounding errors; and (2) it uses information frommultiple cameras by projecting the same 3D point onto all available frames.
Revised: The beneﬁts of our approach aretwo-fold: (1) it eliminates the need for middle-level representations (e.g., predicted depth mapsor point clouds), which can be a source of compounding errors; and (2) it uses information frommultiple cameras by projecting the same 3D point onto all available frames.

Original: Beyond the direct application of our work to 3D object detection for autonomous driving, thereare several venues that warrant future investigation.
Revised: Beyond the direct application of our work to 3D object detection for autonomous driving, thereare several venues that warrant future investigation.

Original: For example, single point projection creates alimited receptive ﬁeld in the retrieved image feature maps, and sampling multiple points for eachobject query would incorporate more information for object reﬁnement.
Revised: For example, single point projection creates alimited receptive ﬁeld in the retrieved image feature maps, and sampling multiple points for eachobject query would incorporate more information for object reﬁnement.

Original: Furthermore, the new detection head is input-agnostic, and including other modalities such as LiDAR/RADAR would enhanceperformance and robustness.
Revised: Furthermore, the new detection head is input-agnostic, and including other modalities such as LiDAR/RADAR would enhanceperformance and robustness.

Original: Finally, generalizing our pipeline to other domains such as indoor navigation and object manipulation would increase its scope of application and reveal additional waysfor further improvement.
Revised: Finally, generalizing our pipeline to other domains such as indoor navigation and object manipulation would increase its scope of application and reveal additional waysfor further improvement.