Original: BEVD ISTILL : C ROSS -M ODAL BEV D ISTILLATION
Revised: BEVD ISTILL : C ROSS -M ODAL BEV D ISTILLATION

Original: FOR M ULTI -V IEW 3D O BJECT D ETECTION
Revised: FOR M ULTI -V IEW 3D O BJECT D ETECTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: 3D object detection from multiple image views is a fundamental and challenging task for visual scene understanding.
Revised: 3D object detection from multiple image views is a fundamental and challenging task for visual scene understanding.

Original: Owing to its low cost and high efficiency, multi-view 3D object detection has demonstrated promising application prospects.
Revised: Owing to its low cost and high efﬁciency, multi-view 3D object detection has demonstrated promising application prospects.

Original: However, accurately detecting objects through perspective views is extremely difficult due to the lack of depth information.
Revised: However, accurately detecting objects through perspective views is extremely difﬁcult due to the lack of depth information.

Original: Current approaches tend to adopt heavy backbones for image encoders, making them inapplicable for real-world deployment.
Revised: Current approaches tend to adopt heavy backbones for image encoders, making them inapplicable for real-world deployment.

Original: Different from the images, LiDAR points are superior in providing spatial cues, resulting in highly precise localization.
Revised: Different from the images, LiDAR points are superior in providing spatial cues, resulting in highly precise localization.

Original: In this paper, we explore the incorporation of LiDAR-based detectors for multi-view 3D object detection.
Revised: In this paper, we explore the incorporation of LiDAR-based detectors for multi-view 3D object detection.

Original: Instead of directly training a depth prediction network, we unify the image and LiDAR features in the Bird-Eye-View (BEV) space and adaptively transfer knowledge across non-homogenous representations in a teacher-student paradigm.
Revised: Instead of directly training a depth prediction network, we unify the image and LiDAR features in the Bird-Eye-View (BEV) space and adaptively transfer knowledge across non-homogenous representations in a teacher-student paradigm.

Original: To this end, we propose BEVDistill , a cross-modal BEV knowledge distillation (KD) framework for multi-view 3D object detection.
Revised: To this end, we propose BEVDistill , a cross-modal BEV knowledge distillation (KD) framework for multi-view 3D object detection.

Original: Extensive experiments demonstrate that the proposed method outperforms current KD approaches on a highly-competitive baseline, BEVFormer, without introducing any extra cost in the inference phase.
Revised: Extensive experiments demonstrate that the proposed method outperforms current KD approaches on a highly-competitive baseline, BEVFormer, without introducing any extra cost in the inference phase.

Original: Notably, our best model achieves 59.4 NDS on the nuScenes test leaderboard, achieving new state-of-the-arts in comparison with various image-based detectors.
Revised: Notably, our best model achieves 59.4 NDS on the nuScenes test leaderboard, achieving new state-of-the-arts in comparison with various image-based detectors.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: 3D object detection, aiming at localizing objects in the 3D space, is a crucial ingredient for 3D scene understanding.
Revised: 3D object detection, aiming at localizing objects in the 3D space, is a crucial ingredient for 3D scene understanding.

Original: It has been widely adopted in various applications, such as autonomous driving (Chen et al., 2022a; Shi et al., 2020; Wang et al., 2021b), robotic navigation (Antonello et al., 2017), and virtual reality (Schuemie et al., 2001).
Revised: It has been widely adopted in various applications, such as autonomous driving (Chen et al., 2022a; Shi et al., 2020; Wang et al., 2021b), robotic navigation (Antonello et al., 2017), and virtual reality (Schuemie et al., 2001).

Original: Recently, multi-view 3D object detection has drawn great attention thanks to its low cost and high efficiency.
Revised: Recently, multi-view 3D object detection has drawn great attention thanks to its low cost and high efﬁciency.

Original: As images offer a discriminative appearance and rich texture with dense pixels, detectors can easily discover and categorize the objects, even in a far distance.
Revised: As images offer a discriminative appearance and rich texture with dense pixels, detectors can easily discover and categorize the objects, even in a far distance.

Original: Despite the promising deployment advantage, accurately localizing instances from camera view only is extremely difficult, mainly due to the ill-posed nature of monocular imagery.
Revised: Despite the promising deployment advantage, accurately localizing instances from camera view only is extremely difﬁcult, mainly due to the ill-posed nature of monocular imagery.

Original: Therefore, recent approaches adopt heavy backbones ( e.g., ResNet-101-DCN (He et al., 2016), VoVNetV2 (Lee & Park, 2020)) for image feature extraction, making it inapplicable for real-world applications.
Revised: Therefore, recent approaches adopt heavy backbones ( e.g., ResNet-101-DCN (He et al., 2016), VoVNetV2 (Lee & Park, 2020)) for image feature extraction, making it inapplicable for real-world applications.

Original: LiDAR points, which capture precise 3D spatial information, provide natural guidance for camerabased object detection.
Revised: LiDAR points, which capture precise 3D spatial information, provide natural guidance for camerabased object detection.

Original: In light of this, recent works (Guo et al., 2021b; Peng et al., 2021) start to explore the incorporation of point clouds in 3D object detection for performance improvement.
Revised: In light of this, recent works (Guo et al., 2021b; Peng et al., 2021) start to explore the incorporation of point clouds in 3D object detection for performance improvement.

Original: One line of work (Wang et al., 2019b) projects each points to the image to form depth map labels, and subsequently trains a depth estimator to explicitly extract the spatial information.
Revised: One line of work (Wang et al., 2019b) projects each points to the image to form depth map labels, and subsequently trains a depth estimator to explicitly extract the spatial information.

Original: Such a paradigm generates intermediate products, i.e., depth prediction maps, therefore introducing extra computational cost.
Revised: Such a paradigm generates intermediate products, i.e., depth prediction maps, therefore introducing extra computational cost.

Original: Another line of work (Chong et al., 2021) is to leverage the teacher-student paradigm for knowledge transfer.
Revised: Another line of work (Chong et al., 2021) is to leverage the teacher-student paradigm for knowledge transfer.

Original: (Chong et al., 2021) projects the LiDAR points to the image plane, constructing a 2D input for the teacher model.
Revised: (Chong et al., 2021) projects the LiDAR points to the image plane, constructing a 2D input for the teacher model.

Original: Since the student and teacher models are exactly structurally identical, feature mimicking can be naturally conducted under the framework.
Revised: Since the student and teacher models are exactly structurally identical, feature mimicking can be naturally conducted under the framework.

Original: Although it solves the alignment issue across different modalities, it misses the opportunity of pursuing a strong LiDARbased teacher, which is indeed important in the knowledge distillation (KD) paradigm.
Revised: Although it solves the alignment issue across different modalities, it misses the opportunity of pursuing a strong LiDARbased teacher, which is indeed important in the knowledge distillation (KD) paradigm.

Original: Recently,
Revised: Recently,

Original: UVTR (Li et al., 2022a) propose to distill the cross-modal knowledge in the voxel space, while maintaining the structure of respective detectors.
Revised: UVTR (Li et al., 2022a) propose to distill the cross-modal knowledge in the voxel space, while maintaining the structure of respective detectors.

Original: However, it directly forces the 2D branch to imitate the 3D features, ignoring the divergence between different modalities.
Revised: However, it directly forces the 2D branch to imitate the 3D features, ignoring the divergence between different modalities.

Original: In this work, by carefully examining the non-homogenous features represented in the 2D and 3D spaces, we explore the incorporation of knowledge distillation for multi-view 3D object detection.
Revised: In this work, by carefully examining the non-homogenous features represented in the 2D and 3D spaces, we explore the incorporation of knowledge distillation for multi-view 3D object detection.

Original: Yet, there are two technical challenges.
Revised: Yet, there are two technical challenges.

Original: First, the views of images and LiDAR points are different, i.e., camera features are in the perspective view, while LiDAR features are in the 3D/bird’s-eye view.
Revised: First, the views of images and LiDAR points are different, i.e., camera features are in the perspective view, while LiDAR features are in the 3D/bird’s-eye view.

Original: Such a view discrepancy indicates that a natural one-to-one imitation (Romero et al., 2014) may not be suitable.
Revised: Such a view discrepancy indicates that a natural one-to-one imitation (Romero et al., 2014) may not be suitable.

Original: Second, RGB images and point clouds hold respective representations in their own modalities.
Revised: Second, RGB images and point clouds hold respective representations in their own modalities.

Original: Therefore, it can be suboptimal to mimic features directly, which is commonly adopted in the 2D detection paradigm (Yang et al., 2022a; Zhang & Ma, 2020).
Revised: Therefore, it can be suboptimal to mimic features directly, which is commonly adopted in the 2D detection paradigm (Yang et al., 2022a; Zhang & Ma, 2020).

Original: We address the above challenges by designing a cross-modal BEV knowledge distillation framework, namely BEVDistill .
Revised: We address the above challenges by designing a cross-modal BEV knowledge distillation framework, namely BEVDistill .

Original: Instead of constructing a separate depth estimation network or explicitly projecting one view into the other one, we convert all features to the BEV space, maintaining both geometric structure and semantic information.
Revised: Instead of constructing a separate depth estimation network or explicitly projecting one view into the other one, we convert all features to the BEV space, maintaining both geometric structure and semantic information.

Original: Through the shared BEV representation, features from different modalities are naturally aligned without much information loss.
Revised: Through the shared BEV representation, features from different modalities are naturally aligned without much information loss.

Original: After that, we adaptively transfer the spatial knowledge through both dense and sparse supervision: (i) we introduce soft foreground-guided distillation for non-homogenous dense feature imitation, and (ii) a sparse style instance-wise distillation paradigm is proposed to selectively supervise the student by maximizing the mutual information.
Revised: After that, we adaptively transfer the spatial knowledge through both dense and sparse supervision: (i) we introduce soft foreground-guided distillation for non-homogenous dense feature imitation, and (ii) a sparse style instance-wise distillation paradigm is proposed to selectively supervise the student by maximizing the mutual information.

Original: Experimental results on the competitive nuScenes dataset demonstrate the superiority and generalization of our BEVDistill.
Revised: Experimental results on the competitive nuScenes dataset demonstrate the superiority and generalization of our BEVDistill.

Original: For instance, we achieve 3.4 NDS and 2.7 NDS improvements on a competitive multi-view 3D detector, BEVFormer (Li et al., 2022c), under the single-frame and multi-frame settings, respectively.
Revised: For instance, we achieve 3.4 NDS and 2.7 NDS improvements on a competitive multi-view 3D detector, BEVFormer (Li et al., 2022c), under the single-frame and multi-frame settings, respectively.

Original: Besides, we also present extensive experiments on lightweight backbones, as well as detailed ablation studies to validate the effectiveness of our method.
Revised: Besides, we also present extensive experiments on lightweight backbones, as well as detailed ablation studies to validate the effectiveness of our method.

Original: Notably, our best model reaches 59.4 NDS on nuScenes test leaderboard, achieving new state-of-the-art results among all the published multi-view 3D detectors.
Revised: Notably, our best model reaches 59.4 NDS on nuScenes test leaderboard, achieving new state-of-the-art results among all the published multi-view 3D detectors.

Original:  R ELATED W ORK 2. V ISION - BASED 3D O BJECT D ETECTION
Revised:  R ELATED W ORK 2. V ISION - BASED 3D O BJECT D ETECTION

Original: Vision-based 3D object detection aims to detect object locations, scales, and rotations, which is of great importance in autonomous driving (Xie et al., 2022; Jiang et al., 2022; Zhang et al., 2022) and augmented reality (Azuma, 1997).
Revised: Vision-based 3D object detection aims to detect object locations, scales, and rotations, which is of great importance in autonomous driving (Xie et al., 2022; Jiang et al., 2022; Zhang et al., 2022) and augmented reality (Azuma, 1997).

Original: One line of work is to detect 3D boxes directly from single images.
Revised: One line of work is to detect 3D boxes directly from single images.

Original: Mono3D (Chen et al., 2016) utilizes traditional methods to lift 2D objects into 3D space with semantic and geometrical information.
Revised: Mono3D (Chen et al., 2016) utilizes traditional methods to lift 2D objects into 3D space with semantic and geometrical information.

Original: In the consideration that objects located at different distances appear on various scales, D4LCN (Ding et al., 2020) proposes to leverage depth prediction for convolutional kernel learning.
Revised: In the consideration that objects located at different distances appear on various scales, D4LCN (Ding et al., 2020) proposes to leverage depth prediction for convolutional kernel learning.

Original: Recently, FCOS3D (Wang et al., 2021a) extends the classical 2D paradigm FCOS (Tian et al., 2019b) into monocular 3D object detection.
Revised: Recently, FCOS3D (Wang et al., 2021a) extends the classical 2D paradigm FCOS (Tian et al., 2019b) into monocular 3D object detection.

Original: It transforms the regression targets into the image domain by predicting its 2D-based attributes.
Revised: It transforms the regression targets into the image domain by predicting its 2D-based attributes.

Original: Further, PGD (Wang et al., 2022a) introduces relational graphs to improve the depth estimation for object localization.
Revised: Further, PGD (Wang et al., 2022b) introduces relational graphs to improve the depth estimation for object localization.

Original: MonoFlex (Zhang et al., 2021) argues that objects located at different positions should not be treated equally and presents auto-adjusted supervision.
Revised: MonoFlex (Zhang et al., 2021) argues that objects located at different positions should not be treated equally and presents auto-adjusted supervision.

Original: Another line of work predicts objects from multi-view images.
Revised: Another line of work predicts objects from multi-view images.

Original: DETR3D (Wang et al., 2022b) first incorporates DETR (Carion et al., 2020) into 3D detection by introducing a novel concept: 3D reference points.
Revised: DETR3D (Wang et al., 2022c) ﬁrst incorporates DETR (Carion et al., 2020) into 3D detection by introducing a novel concept: 3D reference points.

Original: After that, Graph-DETR3D (Chen et al., 2022b) extends it by enriching the feature representations with dynamic graph feature aggregation.
Revised: After that, Graph-DETR3D (Chen et al., 2022b) extends it by enriching the feature representations with dynamic graph feature aggregation.

Original: Different from the above methods, BEVDet (Huang et al., 2021) leverages the Lift-splat-shoot (Philion & Fidler, 2020) to explicitly project images into the BEV space, followed by a traditional 3D detection head.
Revised: Different from the above methods, BEVDet (Huang et al., 2021) leverages the Lift-splat-shoot (Philion & Fidler, 2020) to explicitly project images into the BEV space, followed by a traditional 3D detection head.

Original: Inspired by the recently developed attention mechanism, BEVFormer (Li et al., 2022c) automates the cam2bev process with a learnable attention manner and achieves superior performance.
Revised: Inspired by the recently developed attention mechanism, BEVFormer (Li et al., 2022c) automates the cam2bev process with a learnable attention manner and achieves superior performance.

Original: PolarFormer (Jiang et al., 2022) introduces polar coordinates into the model construction of BEV space and greatly improves the performance.
Revised: PolarFormer (Jiang et al., 2022) introduces polar coordinates into the model construction of BEV space and greatly improves the performance.

Original: Moreover, BEVDepth (Li et al., 2022b) improves BEVDet by explicitly supervising depth prediction with projected LiDAR points and achieves state-of-the-art performance.
Revised: Moreover, BEVDepth (Li et al., 2022b) improves BEVDet by explicitly supervising depth prediction with projected LiDAR points and achieves state-of-the-art performance.

Original: 2. K NOWLEDGE D ISTILLATION IN O BJECT D ETECTION
Revised: 2. K NOWLEDGE D ISTILLATION IN O BJECT D ETECTION

Original: Most KD approaches for object detection focus on transferring knowledge among two homogenous detectors by forcing students’ predictions to match those of the teacher (Chen et al., 2017; Dai et al.,
Revised: Most KD approaches for object detection focus on transferring knowledge among two homogenous detectors by forcing students’ predictions to match those of the teacher (Chen et al., 2017; Dai et al.,

Original: ; Zheng et al., 2022).
Revised: ; Zheng et al., 2022).

Original: More recent works (Guo et al., 2021a; Wang et al., 2019a) find that imitation of feature representations is more effective for detection.
Revised: More recent works (Guo et al., 2021a; Wang et al., 2019a) ﬁnd that imitation of feature representations is more effective for detection.

Original: A vital challenge is to determine which feature regions should be distilled from the teacher model.
Revised: A vital challenge is to determine which feature regions should be distilled from the teacher model.

Original: FGFI (Wang et al., 2019a) chooses features that are covered by anchor boxes whose IoU with GT is larger than a certain threshold.
Revised: FGFI (Wang et al., 2019a) chooses features that are covered by anchor boxes whose IoU with GT is larger than a certain threshold.

Original: PGD (Yang et al., 2022a) only focuses on a few key predictive regions using a combination of classification and regression scores as a measure of quality.
Revised: PGD (Yang et al., 2022a) only focuses on a few key predictive regions using a combination of classiﬁcation and regression scores as a measure of quality.

Original: Despite a large number of works discussing the KD in object detection, only a few of them consider the multi-modal setting.
Revised: Despite a large number of works discussing the KD in object detection, only a few of them consider the multi-modal setting.

Original: MonoDistill (Chong et al., 2021) projects the points into the image plane and applies a modified image-based 3D detector as the teacher to distill knowledge.
Revised: MonoDistill (Chong et al., 2021) projects the points into the image plane and applies a modiﬁed image-based 3D detector as the teacher to distill knowledge.

Original: Such a paradigm solves the alignment issue naturally, however, it misses the chance to pursue a much stronger pointbased teacher model.
Revised: Such a paradigm solves the alignment issue naturally, however, it misses the chance to pursue a much stronger pointbased teacher model.

Original: LIGA-stereo (Guo et al., 2021b) leverages the information of LiDAR by supervising the BEV representation of the vision-based model with the intermediate output from the SECOND
Revised: LIGA-stereo (Guo et al., 2021b) leverages the information of LiDAR by supervising the BEV representation of the vision-based model with the intermediate output from the SECOND

Original: (Yan et al., 2018).
Revised: (Yan et al., 2018).

Original: Recently, UVTR (Li et al., 2022a) presents a simple approach by directly regularizing the voxel representations between the student and teacher models.
Revised: Recently, UVTR (Li et al., 2022a) presents a simple approach by directly regularizing the voxel representations between the student and teacher models.

Original: Both of them choose to mimic the feature representations across the models, while ignoring the divergence between different modalities.
Revised: Both of them choose to mimic the feature representations across the models, while ignoring the divergence between different modalities.

Original:  M ETHODS
Revised:  M ETHODS

Original: In this section, we introduce our proposed BEVDistill in detail.
Revised: In this section, we introduce our proposed BEVDistill in detail.

Original: We first give an overview of the whole framework in Figure 2 and clarify the model designs of the teacher and student model in
Revised: We ﬁrst give an overview of the whole framework in Figure 1 and clarify the model designs of the teacher and student model in

Original: Section 3.1.
Revised: Section 3.1.

Original: After that, we present the cross-modal knowledge distillation approach in Section 3.2, which consists of two modules: dense feature distillation and sparse instance distillation
Revised: After that, we present the cross-modal knowledge distillation approach in Section 3.2, which consists of two modules: dense feature distillation and sparse instance distillation

Original: . 3. B ASELINE M ODEL
Revised: . 3. B ASELINE M ODEL

Original: Student Model.
Revised: Student Model.

Original: We adopt current state-of-the-art camera-based detector, BEVFormer (Li et al., 2022c), as our student model.
Revised: We adopt current state-of-the-art camera-based detector, BEVFormer (Li et al., 2022c), as our student model.

Original: It consists of an image backbone for feature extraction, a spatial cross-attention module for cam2bev view transformation, and a transformer head for 3D object detection.
Revised: It consists of an image backbone for feature extraction, a spatial cross-attention module for cam2bev view transformation, and a transformer head for 3D object detection.

Original: Besides, it provides a temporal cross-attention module to perceive subsequential multiframe information for better predictions.
Revised: Besides, it provides a temporal cross-attention module to perceive subsequential multiframe information for better predictions.

Original: Teacher Model.
Revised: Teacher Model.

Original: In order to keep consistent with the student model, we select Object-DGCNN (Wang & Solomon, 2021) as our teacher model.
Revised: In order to keep consistent with the student model, we select Object-DGCNN (Wang & Solomon, 2021) as our teacher model.

Original: For simplicity and generality, we replace the
Revised: For simplicity and generality, we replace the

Original: DGCNN attention with a vanilla multi-scale attention module.
Revised: DGCNN attention with a vanilla multi-scale attention module.

Original: It first casts 3D points into the BEV plane and then conducts one-to-one supervision with transformer-based label assignment.
Revised: It ﬁrst casts 3D points into the BEV plane and then conducts one-to-one supervision with transformer-based label assignment.

Original: We train the model by initializing it from a pre-trained CenterPoint model and fix all parameters during knowledge distillation.
Revised: We train the model by initializing it from a pre-trained CenterPoint model and ﬁx all parameters during knowledge distillation.

Original: 3. BEVD ISTILL
Revised: 3. BEVD ISTILL

Original: BEVDistill follows the common knowledge distillation paradigm, with a 3D point cloud detector as teacher and an image detector as student.
Revised: BEVDistill follows the common knowledge distillation paradigm, with a 3D point cloud detector as teacher and an image detector as student.

Original: Different from previous knowledge distillation methods, which keep the same architecture for both student and teacher models (except for backbone),
Revised: Different from previous knowledge distillation methods, which keep the same architecture for both student and teacher models (except for backbone),

Original: BEVDistill explores a more challenging setting with non-homogenous representations.
Revised: BEVDistill explores a more challenging setting with non-homogenous representations.

Original: 3.2. D ENSE F EATURE D ISTILLATION
Revised: 3.2. D ENSE F EATURE D ISTILLATION

Original: In order to conduct dense supervision for feature distillation, we first need to determine the BEV features generated from both models.
Revised: In order to conduct dense supervision for feature distillation, we ﬁrst need to determine the BEV features generated from both models.

Original: For the student, we directly adopt the BEV features maps
Revised: For the student, we directly adopt the BEV features maps

Original: F 2 D produced by the BEVTransformerEncoder .
Revised: F 2 D produced by the BEVTransformerEncoder .

Original: In order to align the feature representations between the teacher and student, we choose the same BEV feature F 3 D outputted by the transformer encoder for the teacher model.
Revised: In order to align the feature representations between the teacher and student, we choose the same BEV feature F 3 D outputted by the transformer encoder for the teacher model.

Original: Different from UVTR (Li et al., 2022a) which directly mimics the  features at the voxel space, we postpone such supervision after the transformer encoder, which offers more opportunities for the network to align information across different modalities.
Revised: Different from LIGA-Stereo (Guo et al., 2021b) which directly mimics the BEV features extracted from the 3D backbone, we postpone such supervision after the transformer encoder (illustrated in Figure 1), which offers more opportunities for the network to align information across different modalities.

Original: Previous works (Romero et al., 2014; Li et al., 2022a) directly force the student to mimic the feature map from the teacher and achieve significant performance improvement:
Revised: Previous works (Romero et al., 2014; Li et al., 2022a) directly force the student to mimic the feature map from the teacher and achieve signiﬁcant performance improvement:

Original: [Equation]
Revised: [Equation]

Original: where H, W denotes the width and height of the distilled feature map, and || · || 2 is the L 2 norm.
Revised: where H, W denotes the width and height of the distilled feature map, and || · || 2 is the L 2 norm.

Original: However, such a strategy may not work well under the cross-modal feature setting.
Revised: However, such a strategy may not work well under the cross-modal feature setting.

Original: Although the view discrepancy is eliminated by projecting both features into the BEV plane, there is still domain discrepancy between different modalities: despite the same scenes being captured with the point clouds and images with careful alignment, the representation itself can be diverse under different modalities .
Revised: Although the view discrepancy is eliminated by projecting both features into the BEV plane, there is still domain discrepancy between different modalities: despite the same scenes being captured with the point clouds and images with careful alignment, the representation itself can be diverse under different modalities .

Original: For instance, images holds pixels for both foreground and backgrounds regions, however
Revised: For instance, images holds pixels for both foreground and backgrounds regions, however

Original: LiDAR points only appears when there exists objects to reflect the ray.
Revised: LiDAR points only appears when there exists objects to reﬂect the ray.

Original: Consider that the regional 3D features can hold meaningful information only if the point exists, we regularize the distillation within the foreground regions.
Revised: Consider that the regional 3D features can hold meaningful information only if the point exists, we regularize the distillation within the foreground regions.

Original: Besides, we notice that the boundary of foreground could also provides useful information, therefore, instead of performing hard supervision on each foreground regions, we introduce a soft-supervision manner similar in (Zhou et al., 2019).
Revised: Besides, we notice that the boundary of foreground could also provides useful information, therefore, instead of performing hard supervision on each foreground regions, we introduce a soft-supervision manner similar in (Zhou et al., 2019).

Original: Concretely, we draw Gaussian distribution for each ground truth center ( x i , y i ) in the BEV space with,
Revised: Concretely, we draw Gaussian distribution for each ground truth center ( x i , y i ) in the BEV space with,

Original: [Equation]
Revised: [Equation]

Original: where σ i is a constant, indicating the object size standard deviation.
Revised: where σ i is a constant, indicating the object size standard deviation.

Original: Since the feature maps are classagnostic, we merge all w i,x,y into one single mask W .
Revised: Since the feature maps are classagnostic, we merge all w i,x,y into one single mask W .

Original: For overlapping regions among different w i,x,y at the same position, we simply take the maximum value of them.
Revised: For overlapping regions among different w i,x,y at the same position, we simply take the maximum value of them.

Original: After that, we enforce the student to mimic the feature with the foreground-guided mask W for dense feature distillation:
Revised: After that, we enforce the student to mimic the feature with the foreground-guided mask W for dense feature distillation:

Original: [Equation]
Revised: [Equation]

Original: This foreground-guided re-weight strategy allows the model to focus on the foreground regions from the teacher, and at the same time avoid imitation of useless empty 3D features in the background regions.
Revised: This foreground-guided re-weight strategy allows the model to focus on the foreground regions from the teacher, and at the same time avoid imitation of useless empty 3D features in the background regions.

Original: Such a manner is counter to the finding in previous homogeneous knowledge distillation works (Yang et al., 2022c).
Revised: Such a manner is counter to the ﬁnding in previous homogeneous knowledge distillation works (Yang et al., 2022c).

Original: 3.2. S PARSE I NSTANCE D ISTILLATION
Revised: 3.2. S PARSE I NSTANCE D ISTILLATION

Original: Instance-level distillation can be easily conducted under the dense prediction setting, where only a pixel-to-pixel mapping is needed for supervision.
Revised: Instance-level distillation can be easily conducted under the dense prediction setting, where only a pixel-to-pixel mapping is needed for supervision.

Original: However, both the student and teacher models in
Revised: However, both the student and teacher models in

Original: BEVDistill act in a sparse-prediction style.
Revised: BEVDistill act in a sparse-prediction style.

Original: Therefore, a set-to-set mapping is required to ensure instance-level distillation.
Revised: Therefore, a set-to-set mapping is required to ensure instance-level distillation.

Original: To achieve the goal, we simply follow the practice in (Wang & Solomon, 2021) to construct the correspondence between the student and teacher predictions.
Revised: To achieve the goal, we simply follow the practice in (Wang & Solomon, 2021) to construct the correspondence between the student and teacher predictions.

Original: Specifically, suppose the categorical and localization predictions output by the i th query from the teacher model are c Ti and b Ti , and the student can be represented as c Si and b Si , a permutation of ˆ σ can be found between the output set of the teacher and the student with the lowest cost:
Revised: Speciﬁcally, suppose the categorical and localization predictions output by the i th query from the teacher model are c Ti and b Ti , and the student can be represented as c Si and b Si , a permutation of ˆ σ can be found between the output set of the teacher and the student with the lowest cost:

Original: [Equation]
Revised: [Equation]

Original: where L match ( y i , ˆ y i ) is a pair-wise matching cost, which is defined as:
Revised: where L match ( y i , ˆ y i ) is a pair-wise matching cost, which is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: However, we empirically find such a vanilla distillation approach works poorly in a cross-domain supervision manner.
Revised: However, we empirically ﬁnd such a vanilla distillation approach works poorly in a cross-domain supervision manner.

Original: There are two main issues that impede the model from further performance improvement.
Revised: There are two main issues that impede the model from further performance improvement.

Original: On one hand, not all predictions from the teacher should be treated equally as valuable cues, since most of the predictions are false positives with a low classification score.
Revised: On one hand, not all predictions from the teacher should be treated equally as valuable cues, since most of the predictions are false positives with a low classiﬁcation score.

Original: On the other hand, though the classification logits can represent rich knowledge (Hinton et al., 2015), it may not hold when the input data are different.
Revised: On the other hand, though the classiﬁcation logits can represent rich knowledge (Hinton et al., 2015), it may not hold when the input data are different.

Original: Directly distilling these prediction will introduce great noise to the model and deteriorate the performance.
Revised: Directly distilling these prediction will introduce great noise to the model and deteriorate the performance.

Original: To tackle the problem, we exploit reliable quality score derived from the teacher model to measure the importance of the instance-level pseudo labels and use the score as soft re-weighted factors to eliminate the potential noise predictions made by the teacher.
Revised: To tackle the problem, we exploit reliable quality score derived from the teacher model to measure the importance of the instance-level pseudo labels and use the score as soft re-weighted factors to eliminate the potential noise predictions made by the teacher.

Original: Specifically, we consider the classification score c i (categorical information) and the
Revised: Speciﬁcally, we consider the classiﬁcation score c i (categorical information) and the

Original: IoU between the prediction b predi and the ground truth b GTi (localization information) together to formulate the quality-score q i :
Revised: IoU between the prediction b predi and the ground truth b GTi (localization information) together to formulate the quality-score q i :

Original: [Equation]
Revised: [Equation]

Original: The quality score serves as an indicator to guide the student on which teacher’s predictions should be paid more weight.
Revised: The quality score serves as an indicator to guide the student on which teacher’s predictions should be paid more weight.

Original: Therefore, the final instance-level distillation can be written as:
Revised: Therefore, the ﬁnal instance-level distillation can be written as:

Original: [Equation]
Revised: [Equation]

Original: where L cls is the KL divergence loss, L box is the L 1 loss in (Wang & Solomon, 2021), and α , β are reweighted factors to balance the supervision.
Revised: where L cls is the KL divergence loss, L box is the L 1 loss in (Wang & Solomon, 2021), and α , β are reweighted factors to balance the supervision.

Original: Given that the prediction probability mass function over classes contains richer information than one-hot labels, student models are proven to been benefited from such extra supervision (Hinton et al., 2015).
Revised: Given that the prediction probability mass function over classes contains richer information than one-hot labels, student models are proven to been beneﬁted from such extra supervision (Hinton et al., 2015).

Original: However, this low dimensional prediction distribution (number of classes) means that only a few amount of knowledge is encoded, thus limiting the knowledge that can be transferred, especially under the cross-modal setting.
Revised: However, this low dimensional prediction distribution (number of classes) means that only a few amount of knowledge is encoded, thus limiting the knowledge that can be transferred, especially under the cross-modal setting.

Original: Besides, the representational knowledge is often structured, with implicit complex interdependencies across different dimensions, while the KL objective treats all dimensions independently.
Revised: Besides, the representational knowledge is often structured, with implicit complex interdependencies across different dimensions, while the KL objective treats all dimensions independently.

Original: To overcome the problem, instead of minimizing the KL divergence over two distributions, we choose to directly maximize the mutual information (MI) between the representations h S , h T at the penultimate layer (before logits) from the teacher and student networks:
Revised: To overcome the problem, instead of minimizing the KL divergence over two distributions, we choose to directly maximize the mutual information (MI) between the representations h S , h T at the penultimate layer (before logits) from the teacher and student networks:

Original: [Equation]
Revised: [Equation]

Original: where p ( h T , h S ) is the joint distribution and the marginal distributions µ ( h T ) , v ( h S ) .
Revised: where p ( h T , h S ) is the joint distribution and the marginal distributions µ ( h T ) , v ( h S ) .

Original: We then define a distribution q conditioned on η that captures whether the pair is congruent ( q ( η = 1) ) or incongruent ( q ( η = 0)):
Revised: We then deﬁne a distribution q conditioned on η that captures whether the pair is congruent ( q ( η = 1) ) or incongruent ( q ( η = 0)

Original: [Equation]
Revised: [Equation]

Original: ( h T , h S | η = 0)

Original: =

Original: µ ( h T ) v ( h S ) .

Original: With Bayes’ rule, we can obtain the posterior for η = 1 :
Revised: With Bayes’ rule, we can obtain the posterior for η = 1 :

Original: [Equation]
Revised: [Equation]

Original: By taking the logarithm of both sides, we have
Revised: By taking the logarithm of both sides, we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, the objective can be transformed to maximize the lower bound of MI.
Revised: Therefore, the objective can be transformed to maximize the lower bound of MI.

Original: Since there is no closed form for q ( η = 1 | h T , h S ) , we utilize a neural network g , called critic (Tian et al., 2019a) to approximate with NCE loss (see Appendix A for detailed implementation):
Revised: Since there is no closed form for q ( η = 1 | h T , h S ) , we utilize a neural network g , called critic (Tian et al., 2019a) to approximate with NCE loss (see Appendix A for detailed implementation):

Original: [Equation]
Revised: [Equation]

Original: To this end, the final sparse instance distillation loss can be represented as:
Revised: To this end, the ﬁnal sparse instance distillation loss can be represented as:

Original: [Equation]
Revised: [Equation]

Original:  E XPERIMENTS 4. D ATASET AND E VALUATION M ETRICS
Revised:  E XPERIMENTS 4. D ATASET AND E VALUATION M ETRICS

Original: We conduct the experiments on the NuScenes dataset (Caesar et al., 2020), which is one of the most popular datasets for 3D object detection.
Revised: We conduct the experiments on the NuScenes dataset (Caesar et al., 2020), which is one of the most popular datasets for 3D object detection.

Original: It consists of 700 scenes for training, 150 scenes for validation, and 150 scenes for testing.
Revised: It consists of 700 scenes for training, 150 scenes for validation, and 150 scenes for testing.

Original: For each scene, it includes 6 camera images (front, front left, front right, back left, back right, back) to cover the whole viewpoint, and a 360 ◦ LiDAR point cloud.
Revised: For each scene, it includes 6 camera images (front, front left, front right, back left, back right, back) to cover the whole viewpoint, and a 360 ◦ LiDAR point cloud.

Original: Camera matrixes including both intrinsic and extrinsic are provided, which establish a one-to-one correspondence between each 3D point and the 2D image plane.
Revised: Camera matrixes including both intrinsic and extrinsic are provided, which establish a one-to-one correspondence between each 3D point and the 2D image plane.

Original: There are 23 categories in total, and only 10 classes are utilized to compute the final metrics according to the official evaluation code.
Revised: There are 23 categories in total, and only 10 classes are utilized to compute the ﬁnal metrics according to the ofﬁcial evaluation code.

Original: 4. E VALUATION M ETRICS

Original: We adopt the official evaluation toolbox provided by nuScenes.
Revised: We adopt the ofﬁcial evaluation toolbox provided by nuScenes.

Original: We report nuScenes Detection Score (NDS) and mean Average Precision (mAP), along with mean Average Translation Error (mATE), mean Average Scale Error (mASE), mean Average Orientation Error (mAOE), mean Average Velocity Error (mAVE), mean Average Attribute Error (mAAE) in our experiments.
Revised: We report nuScenes Detection Score (NDS) and mean Average Precision (mAP), along with mean Average Translation Error (mATE), mean Average Scale Error (mASE), mean Average Orientation Error (mAOE), mean Average Velocity Error (mAVE), mean Average Attribute Error (mAAE) in our experiments.

Original: The NDS score is a weighted sum of mean Average Precision (mAP) and the aforementioned True Positive (TP) metrics, which is defined as NDS = 110 [5 × mAP + (cid:80) mTP ∈ TP (1 − min (1 , mTP ))] .
Revised: The NDS score is a weighted sum of mean Average Precision (mAP) and the aforementioned True Positive (TP) metrics, which is deﬁned as NDS = 110 [5 × mAP + (cid:80) mTP ∈ TP (1 − min (1 , mTP ))] .

Original: 4. I MPLEMENTATION D ETAILS
Revised: 4. I MPLEMENTATION D ETAILS

Original: Our codebase is built on MMDetection3D (Contributors, 2020) toolkit.
Revised: Our codebase is built on MMDetection3D (Contributors, 2020) toolkit.

Original: All models are trained on 8 NVIDIA A100 GPUs.
Revised: All models are trained on 8 NVIDIA A100 GPUs.

Original: We first train the 3D detector with a voxel size of (0 . 1 m, 0 .
Revised: We ﬁrst train the 3D detector with a voxel size of (0 . 1 m, 0 .

Original: 1 m, 0 .
Revised: 1 m, 0 .

Original: 2 m ) followed in (Yin et al., 2021).
Revised: 2 m ) followed in (Yin et al., 2021).

Original: During the distillation phase, the batch size is set to 1 per GPU with an initial learning rate of 2e-4.
Revised: During the distillation phase, the batch size is set to 1 per GPU with an initial learning rate of 2e-4.

Original: Unless otherwise specified, we train the models for 2 × schedule ( epochs) with a cyclic policy.
Revised: Unless otherwise speciﬁed, we train the models for 2 × schedule ( epochs) with a cyclic policy.

Original: The input image size is set to 1600 × 900 and the grid size of the BEV plane in BEVFormer is set to 128 × 128.
Revised: The input image size is set to 1600 × 900 and the grid size of the BEV plane in BEVFormer is set to 128 × 128.

Original: FP16 training is enabled for GPU memory management.
Revised: FP16 training is enabled for GPU memory management.

Original: 4. M AIN R ESULTS4.4. C OMPARISON WITH S TATE - OF - THE -A RT
Revised: 4. M AIN R ESULTS

Original: We first implement BEVFormer and BEVFormer-T on the nuScenes validation subset with ResNet50 backbone and compare BEVDistill with other state-of-the-art knowledge distillation methods.
Revised: We ﬁrst implement BEVFormer and BEVFormer-T on the nuScenes validation subset with ResNet50 backbone and compare BEVDistill with other state-of-the-art knowledge distillation methods.

Original: The final results are shown in Table 1.
Revised: The ﬁnal results are shown in Table 1.

Original: Our BEVDistill greatly boosts its vanilla student model,
Revised: Our BEVDistill greatly boosts its vanilla student model,

Original: [Table]

Original: BEVFormer, by more than 3.4/2.3 mAP and 3.4/2.7 NDS, respectively, without introducing any extra inference cost.
Revised: BEVFormer, by more than 3.4/2.3 mAP and 3.4/2.7 NDS, respectively, without introducing any extra inference cost.

Original: This validates the effectiveness and generalization of the proposed method under both single-frame and multiple-frame settings.
Revised: This validates the effectiveness and generalization of the proposed method under both single-frame and multiple-frame settings.

Original: Classical and effective distillation methods like FitNet (Romero et al., 2014), fail to bring any enhancement and even deteriorate the final performance, 
Revised: Classical and effective distillation methods like FitNet (Romero et al., 2014), fail to bring any enhancement and even deteriorate the ﬁnal performance, i.e., -1.2% NDS.

Original: i
Revised: The Set2Set (Wang & Solomon, 2021) also gets struck in the detection accuracy [Table]

Original: [Table]
Revised: [Table]

Original: e., -1.2% NDS. The Set2Set (Wang & Solomon, 2021) also gets struck in the detection accuracy (-1.3% NDS) due to the miss consideration of the divergence between different modality features.
Revised: 1.3% NDS) due to the miss consideration of the divergence between different modality features.

Original: UVTR (Li et al., 2022a), which is the first work to leverage 3D points for multi-view 3D object detection, manages to get an 0.8 NDS improvement.
Revised: UVTR (Li et al., 2022a), which is the ﬁrst work to leverage 3D points for multi-view 3D object detection, manages to get an 0.8 NDS improvement.

Original: However, it only focuses on feature-based distillation while missing the potential benefits in other aspects.
Revised: However, it only focuses on feature-based distillation while missing the potential beneﬁts in other aspects.

Original: We also reimplement MonoDistill (Chong et al., 2021) by extending its monocular version to a multi-view one, yielding an improvement of 0.7 NDS.
Revised: We also reimplement MonoDistill (Chong et al., 2021) by extending its monocular version to a multi-view one, yielding an improvement of 0.7 NDS.

Original: Besides, our best model based on BEVDepth reaches 59.4 NDS on nuScenes test leaderboard, achieving new state-of-the-art performance (See Appendix B.3).

Original: 4.4. D ISTILLING TO A L IGHTWEIGHT B ACKBONE
Revised: 4. C OMPARISON WITH S TATE - OF - THE -A RTS

Original: Knowledge distillation is actually one of the most common solutions to model compression.
Revised: In addition to ofﬂine results

Original: It is usually adopted to transfer useful information from a large stronger model to a lightweight weaker model, which is suitable for deployment on the edge. Considering that vision-based detectors are fed with a batch of images for surrounding view inputs, it is extremely important to reduce the size of the image backbone.
Revised: [Table]

Original: With this in mind, we conduct experiments with smaller backbones, including ResNet-18 and MobileNetV2, which is a more demanded setting in real applications.
Revised: we also report the detection performance on nuScenes test leaderboard compared to various multi-view 3D detection approaches to further explore the potential of BEVDistill.

Original: Thefinal results are shown in Table 2.
Revised: The results are shown in Table 2.

Original: BEVDistill improves the ResNet-18 and MobileNetV2 backbones by more than 2.1/1.7 NDS, pointing its great potential for resource-limited applications including autonomous driving and edge computing.
Revised: Our ﬁnal model is based on current state-of-the-art model BEVDepth with an input image size of 640 × 1600. We adopt ConvNeXt-base (Liu et al., 2022d) as our image encoder with a BEV space of 256 × 256. The model is trained 20 epochs with CBGS and no test-time augmentation is applied. It surpasses all other detectors, including the recently developed PETRv2, BEVFormer, and BEVDet4D by more than 1.0 NDS, achieving new state-of-the-art on this competitive benchmark.

Original: 4. A BLATION S TUDIES
Revised: 4. A BLATION S TUDIES

Original: In this section, we provide detailed ablation studies on different components of our model to gain a deeper understanding of BEVDistill.
Revised: In this section, we provide detailed ablation studies on different components of our model to gain a deeper understanding of BEVDistill.

Original: For efficiency, we adopt BEVFormer-R50 as our baseline model with 1 × schedule on 1/2 subset of the nuScenes training set if not specified.
Revised: For efﬁciency, we adopt BEVFormer-R50 as our baseline model with 1 × schedule on 1/2 subset of the nuScenes training set if not speciﬁed.

Original: 4.5. M AIN A BLATIONS
Revised: 4.5. M AIN A BLATIONS

Original: To understand how each module contributed to the final detection performance in BEVDistill, we test each component independently and report its performance in Table 3.
Revised: To understand how each module contributed to the ﬁnal detection performance in BEVDistill, we test each component independently and report its performance in Table 3.

Original: The overall baseline starts from 42.2 NDS.
Revised: The overall baseline starts from 42.2 NDS.

Original: When dense feature distillation is applied, the NDS is raised by 2.5 points, which[Table] [Table] validates the finding that disentangling the foreground regions with soft supervision can greatly promote the student’s detection ability.
Revised: When dense feature distillation is applied, the NDS is raised by 2.5 points, which validates the ﬁnding that disentangling the foreground regions with soft supervision can greatly promote the student’s detection ability.

Original: Then, we add the sparse instance distillation, which brings us a 1.7% NDS enhancement.
Revised: Then, we add the sparse instance distillation, which brings us a 1.7% NDS enhancement.

Original: Finally, the NDS score achieves 45.7 when all components are applied, yielding a 3.4 % absolute improvement, validating the effectiveness of BEVDistill. 
Revised: Finally, the NDS score achieves 45.7 when all components are applied, yielding a 3.4 % absolute improvement, validating the effectiveness of BEVDistill. [Table] Arxiv [Table]

Revised: PolarFormer

Original: [Table]
Revised: [Table]

Revised: Arxiv

Original: D
Revised: [Table] Arxiv2022b BEVDepth [Table]

Original:  Comparison with other feature-based distillation mechanisms.
Revised: Arxiv2022b + BEVDistill [Table] [Table] 4.5. A BLATIONS ON D ENSE F EATURE D ISTILLATION Comparison with other feature-based distillation mechanisms.

Original: Feature-based distillation methods have been extensively explored in both image classification (Hinton et al., 2015) and object detection tasks (Yang et al., 2022b;c).
Revised: Feature-based distillation methods have been extensively explored in both image classiﬁcation (Hinton et al., 2015) and object detection tasks (Yang et al., 2022b;c).

Original: Therefore, we compare our dense feature distillation module with other competitive feature-based distillation approaches.
Revised: Therefore, we compare our dense feature distillation module with other competitive feature-based distillation approaches.

Original: The results are shown in Table 4.
Revised: The results are shown in Table 4.

Original: FitNet distills all pixels across 2D and 3D features with equal supervision, yielding little improvement.
Revised: FitNet distills all pixels across 2D and 3D features with equal supervision, yielding little improvement.

Original: FGD (Yang et al., 2022b) is current state-of-the-art knowledge distillation approach in 2D object detection.
Revised: FGD (Yang et al., 2022b) is current state-of-the-art knowledge distillation approach in 2D object detection.

Original: However, it neglects the divergence between different modalities across features, therefore gets the worse performance.
Revised: However, it neglects the divergence between different modalities across features, therefore gets the worse performance.

Original: UVTR (Li et al., 2022a) also mimics the features directly but it only selects positive regions (query points) for feature imitation.
Revised: UVTR (Li et al., 2022a) also mimics the features directly but it only selects positive regions (query points) for feature imitation.

Original: Compared with the above methods, BEVDistill considers both modality divergence and region importance for feature imitation, providing the highest accuracy.
Revised: Compared with the above methods, BEVDistill considers both modality divergence and region importance for feature imitation, providing the highest accuracy.

Original: Strategies on foreground mask generation.
Revised: Strategies on foreground mask generation.

Original: As mentioned in Section 3.2.1, not all positions in BEV feature representation should be treated equally.
Revised: As mentioned in Section 3.2.1, not all positions in BEV feature representation should be treated equally.

Original: How to select the most informative regions for feature imitation is also non-trivial.
Revised: How to select the most informative regions for feature imitation is also non-trivial.

Original: Hence, we provide detailed experiments on how to generate the foreground imitation mask for distillation.
Revised: Hence, we provide detailed experiments on how to generate the foreground imitation mask for distillation.

Original: Intuitively, we can directly select the center points of the ground truth objects (GT-Center) or the query reference points (Query-Center) as imitation instances.
Revised: Intuitively, we can directly select the center points of the ground truth objects (GT-Center) or the query reference points (Query-Center) as imitation instances.

Original: However, such a strategy directly abandons most areas, which may contain useful information.
Revised: However, such a strategy directly abandons most areas, which may contain useful information.

Original: Therefore, we attempt to convert sucha ‘hard’ decision into the ‘soft’ manner with a heatmap mask, which can be the model classification
Revised: Therefore, we attempt to convert such

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original:  prediction from the teacher (Pred-Heatmap) or the ground truth heatmap (GT-Heatmap).
Revised: a ‘hard’ decision into the ‘soft’ manner with a heatmap mask, which can be the model classiﬁcation prediction from the teacher (Pred-Heatmap) or the ground truth heatmap (GT-Heatmap).

Original: The final results are shown in Table 5.
Revised: The ﬁnal results are shown in Table 5.

Original: The soft foreground mask generation is much better compared to the hard one.
Revised: The soft foreground mask generation is much better compared to the hard one.

Original: We choose the GT-Heatmap strategy in our experiments due to it highest score.
Revised: We choose the GT-Heatmap strategy in our experiments due to it highest score.

Original: 4.5. A BLATIONS ON S PARSE I NSTANCE D ISTILLATION
Revised: 4.5. A BLATIONS ON S PARSE I NSTANCE D ISTILLATION

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Ablations on the instance distillation loss.
Revised: Ablations on the instance distillation loss.

Original: Instance-level distillation is a necessary module in detection-based knowledge distillation.
Revised: Instance-level distillation is a necessary module in detection-based knowledge distillation.

Original: However, such a component has been seldom discussed in a transformer-based sparse prediction setting.
Revised: However, such a component has been seldom discussed in a transformer-based sparse prediction setting.

Original: In this ablation, we conduct detailed experiments on the loss selection in Table 6.
Revised: In this ablation, we conduct detailed experiments on the loss selection in Table 6.

Original: We first conduct the commonly adopted cross-entropy and L1 loss for the classification and regression branches.
Revised: We ﬁrst conduct the commonly adopted cross-entropy and L1 loss for the classiﬁcation and regression branches.

Original: The result is surprisingly lower than its vanilla baseline.
Revised: The result is surprisingly lower than its vanilla baseline.

Original: When adding them respectively, we figure out that the cross-entropy is the key issue leading to performance degradation.
Revised: When adding them respectively, we ﬁgure out that the cross-entropy is the key issue leading to performance degradation.

Original: To avoid the problem, we replace it with the InfoNCE loss along with the contrastive paradigm, and an immediate improvement is observed, which validates our assumption in Section 3.2.2.
Revised: To avoid the problem, we replace it with the InfoNCE loss along with the contrastive paradigm, and an immediate improvement is observed, which validates our assumption in Section 3.2.2.

Original: Strategies on selection of critic g .
Revised: Strategies on selection of critic g .

Original: Since most self-supervised learning approaches are based on homogenous representations, exploring the suitable contrastive paradigm for cross-modality is important.
Revised: Since most self-supervised learning approaches are based on homogenous representations, exploring the suitable contrastive paradigm for cross-modality is important.

Original: Note that features located in the same positions in the 2D and 3D BEV feature maps are considered positive pairs while the rest are negative pairs.
Revised: Note that features located in the same positions in the 2D and 3D BEV feature maps are considered positive pairs while the rest are negative pairs.

Original: We compare three prototypes of critic g and report the results in Table 7.
Revised: We compare three prototypes of critic g and report the results in Table 7.

Original: We first adopt the classic InfoNCE loss, which provides the most improvement.
Revised: We ﬁrst adopt the classic InfoNCE loss, which provides the most improvement.

Original: We also conduct experiments on positive pairs only with NCS (negative cosine similarity) and CE (cross-entropy) loss, while gets little enhancement, validating the proof in Section 3.2.2.
Revised: We also conduct experiments on positive pairs only with NCS (negative cosine similarity) and CE (cross-entropy) loss, while gets little enhancement, validating the proof in Section 3.2.2.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we introduce a cross-modal knowledge distillation framework for multi-view 3D object detection, namely BEVDistill.
Revised: In this paper, we introduce a cross-modal knowledge distillation framework for multi-view 3D object detection, namely BEVDistill.

Original: It unifies different modalities in the BEV space to ease the modal alignment and feature imitation.
Revised: It uniﬁes different modalities in the BEV space to ease the modal alignment and feature imitation.

Original: With the dense feature distillation module, it can effectively transfer rich teacher knowledge to the student across 2D and 3D data.
Revised: With the dense feature distillation module, it can effectively transfer rich teacher knowledge to the student across 2D and 3D data.

Original: BEVDistill also explores a sparse instance distillation approach under the transformer-based detection paradigm.
Revised: BEVDistill also explores a sparse instance distillation approach under the transformer-based detection paradigm.

Original: Without bells and whistles, it achieves new state-of-the-art performanceon the competitive nuScenes test leaderboard.
Revised: Without bells and whistles, it achieves new state-of-the-art performance

Original: We hope it can serve as a simple baseline for leveraging spatial information for high accurate multiview 3D object detectors.

Original: A I MPLEMENTATION OF C RITIC g
Revised: A I MPLEMENTATION OF C RITIC g

Original: Firstly, we adopt bilinear F bilinear to obtain the paired 2D/3D features
Revised: Firstly, we adopt bilinear F bilinear to obtain the paired 2D/3D features

Original: f 2 D i , f 3 D i on student and teacher BEV features maps F 2 D , F 3 D according to the reference location r i predicted by each student query q i :
Revised: f 2 D i , f 3 D i on student and teacher BEV features maps F 2 D , F 3 D according to the reference location r i predicted by each student query q i :

Original: [Equation]
Revised: [Equation]

Original: Note that we adopt query points from 6 stages which actually form 900 × 6 feature pairs.
Revised: Note that we adopt query points from 6 stages which actually form 900 × 6 feature pairs.

Original: Each 2D and 3D feature located at the same position are viewed as a positive pair.
Revised: Each 2D and 3D feature located at the same position are viewed as a positive pair.

Original: We use separate projection heads for 2D and 3D features since they originated from different modalities.
Revised: We use separate projection heads for 2D and 3D features since they originated from different modalities.

Original: Each projection head is a 3-layer MLP ( g 2 D , g 3 D ) and the final loss can be written as
Revised: Each projection head is a 3-layer MLP ( g 2 D , g 3 D ) and the ﬁnal loss can be written as

Original: [Equation]
Revised: [Equation]

Original: where τ is the temperature.
Revised: where τ is the temperature.

Original: Different from the original implementation in (He et al., 2020), we do not collate samples from other cards during distributed training since there are enough negative samples (5400-1=5399) for each card.
Revised: Different from the original implementation in (He et al., 2020), we do not collate samples from other cards during distributed training since there are enough negative samples (5400-1=5399) for each card.

Original: B M ORE E XPERIMENTAL R ESULTS B. E XPERIMENTS ON BEVF ORMER WITH L ARGER B ACKBONES
Revised: B M ORE E XPERIMENTAL R ESULTS B. E XPERIMENTS ON BEVF ORMER WITH L ARGER B ACKBONES

Original: Instead of the results conducted on ResNet-50 backbone, we also provide detailed experiments with stronger backbones, ResNet-101.
Revised: Instead of the results conducted on ResNet-50 backbone, we also provide detailed experiments with stronger backbones, ResNet-101.

Original: We keep all experimental settings the same in Section 4.5 except for the image backbone and the results are shown in Table 8.
Revised: We keep all experimental settings the same in Section 4.5 except for the image backbone and the results are shown in Table 8.

Original: From the table, we can conclude that
Revised: From the table, we can conclude that

Original: BEVDistill can still consistently improve the competitive baseline BEVFormer, even with a much stronger ResNet-101 backbone by more than 2.3 and 1.8 NDS.
Revised: BEVDistill can still consistently improve the competitive baseline BEVFormer, even with a much stronger ResNet-101 backbone by more than 2.3 and 1.8 NDS.

Original:  ResNet-101 DCN backbone, under both single-frame and multi-frame settings.
Revised: BEVFormer on ResNet-101 DCN backbone, under both single-frame and multi-frame settings.

Original: [Table]
Revised: [Table]

Original: B. E XPERIMENTS ON BEVD EPTH
Revised: B. D ISTILLING TO A L IGHTWEIGHT B ACKBONE

Revised: Knowledge distillation is actually one of the most common solutions to model compression.

Revised: It is usually adopted to transfer useful information from a large stronger model to a lightweight weaker model, which is suitable for deployment on the edge.

Revised: Considering that vision-based detectors are fed with a batch of images for surrounding view inputs, it is extremely important to reduce the size of the image backbone.

Revised: With this in mind, we conduct experiments with smaller backbones, including ResNet-18 and MobileNetV2, which is a more demanded setting in real applications.

Revised: The ﬁnal results are shown in Table 9.

Revised: BEVDistill improves the ResNet-18 and MobileNetV2 backbones by more than 2.1/1.7 NDS, pointing its great potential for resource-limited applications including autonomous driving and edge computing.

Revised: [Table]

Revised: [Table]

Revised: B. E XPERIMENTS ON BEVD EPTH

Original: To fully exploit the capability of this simple framework, we extend it to adapt the current stateof-the-art multi-view 3D detector, BEVDepth.
Revised: To fully exploit the capability of this simple framework [Table] we extend it to adapt the current stateof-the-art multi-view 3D detector, BEVDepth.

Original: Since BEVDepth and BEVFormer both share BEV space as the intermediate feature representation, we can directly apply dense feature distillation.
Revised: Since BEVDepth and BEVFormer both share BEV space as the intermediate feature representation, we can directly apply dense feature distillation.

Original: As for the instance distillation, BEVDepth conducts dense prediction similar to CenterPoint.
Revised: As for the instance distillation, BEVDepth conducts dense prediction similar to CenterPoint.

Original: Therefore, we directly choose CenterPoint as our teacher model and only conduct instance distillation on the foreground regions ( i.e., the center of ground truth objects).
Revised: Therefore, we directly choose CenterPoint as our teacher model and only conduct instance distillation on the foreground regions ( i.e., the center of ground truth objects).

Original: For fair comparison, we conduct experiments on ResNet-50 backbone on nuScenes 1/2 training subset and report the results on validation subset in Table 9.
Revised: We conduct experiments on ResNetbackbone on nuScenes 1/2 training subset and report the results on validation subset in Table 10.

Original: Surprisingly, BEVDistill can still improve its vanilla baseline by 2.2 NDS, even though BEVDepth has already leveraged the information from LiDAR data with explicit depth supervision. [Table] [Table] [Table]
Revised: Surprisingly, BEVDistill can still improve its vanilla baseline by 1.2 NDS, even though BEVDepth has already leveraged the information from LiDAR data with explicit depth supervision

Original: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: Arxiv

Original: G-DETR3D

Original: [Table]
Revised: [Table]

Original: [Table] [Table] [Table] PETR
Revised: E XPERIMENTS ON W AYMO O PEN D ATASET

Original: [Table] [Table] [Table] ECCV2022a
Revised: In this section

Original: UVTR

Original: [Table]
Revised: [Table]

Original: NeurIPS2022a

Original: [Table]

Original: Arxiv

Original: BEVFormer [Table] PolarFormer [Table]
Revised: we report BEVDistill on another competitive public dataset, Waymo Open Dataset (Sun et al., 2020) to further demonstrate the generalization of our approach.

Original: Arxiv

Original: PETRv [Table] Arxiv2022b BEVDepth ‡
Revised: Due to the characteristics of the camera-based detections, instead of adopting original mAP metrics, we choose the ofﬁcial released LET-mAP (Hung et al., 2022) to evaluate the model performance.

Original: [Table] Arxiv2022b + BEVDistill [Table] [Table]
Revised: Since most multi-view 3D detectors do not report their performances on Waymo dataset, we select another competitive multi-view 3D detector, MV-FCOS3D++ (Wang et al., 2022a), which achieves 2 rd place at the

Original: B.

Original: In addition to offline results, we also report the detection performance on nuScenes test leaderboard compared to various multi-view 3D detection approaches to further explore the potential of BEVDistill.
Revised: Waymo Open Dataset Challenge–Camera-Only Track, as our baseline. The follow the same experimental settings in the ofﬁcial MV-FCOS3D++ repository. We select our teacher model as SECOND to adapt with the Anchor3DHead used by MV-FCOS3D++.

Original: The  results are shown in Table 10.
Revised: The ﬁnal results are reported in Table 11.

Original: Our final model is based on current state-of-the-art model BEVDepth with an input image size of 640 × 1600. We adopt ConvNeXt-base (Liu et al., 2022d) as our image encoder with a BEV space of 256 × 256. The model is trained 20 epochs with CBGS andno test-time augmentation is applied.
Revised: It can be concluded that BEVDistill can consistently improves MV-FCOS3D++ and

Original: It surpasses all other detectors, including the recently developed PETRv2, BEVFormer, and BEVDet4D by more than 1.0 NDS, achieving new state-of-the-art on this competitive benchmark.
Revised: MV-FCOS4D++ (multi-frames input version of MV-FCOS3D++) by 1.7/1.1 mAPL, respectively, indicating the robustness and general of the proposed method.

Original: C D ISCUSSIONS C. W HY KL DIVERGENCE DISTILLATION ON SPARSE QUERY LOGITS FAILS TO IMPROVE ?
Revised: C D ISCUSSIONS C. W HY KL DIVERGENCE DISTILLATION ON SPARSE QUERY LOGITS FAILS TO IMPROVE ?

Original: KL logit distillation has been proven to be effective in plenty of works (Tian et al., 2019a; Hinton et al., 2015).
Revised: KL logit distillation has been proven to be effective in plenty of works (Tian et al., 2019a; Hinton et al., 2015).

Original: However, it fails to yield improvements under the cross-modal distillation circumstance.
Revised: However, it fails to yield improvements under the cross-modal distillation.

Original: By visualizing the training data and the predicted logits, we attribute one reason to this problem: for most previous distillations, both teacher and student are fed with the data in the same modality.
Revised: By visualizing the training data and the predicted logits, we attribute one reason to this problem: for most previous distillations, both teacher and student are fed with the data in the same modality.

Original: Therefore, both of them should behave in a similar manner in every region.
Revised: Therefore, both of them should behave in a similar manner in every region.

Original: Such a hypothesis may not hold under different modalities: due to the characteristics of the modal sensor, the information collected by each sensor can be diverse.
Revised: Such a hypothesis may not hold under different modalities: due to the characteristics of the modal sensor, the information collected by each sensor can be diverse.

Original: For instance, even if one person is occluded by another one in the image view, he may still be perceived by the LiDAR sensors.
Revised: For instance, even if one person is occluded by another one in the image view, he may still be perceived by the LiDAR sensors.

Original: Besides, some instances can be easily detected by one modal sensor, while it may be hard on the other one, leading to different logits distribution predicted by the model.
Revised: Besides, some instances can be easily detected by one modal sensor, while it may be hard on the other one, leading to different logits distribution predicted by the model.

Original: Directly forcing the student to mimic such distribution can deteriorate the inner knowledge learned by the student itself.
Revised: Directly forcing the student to mimic such distribution can deteriorate the inner knowledge learned by the student itself.

Original: Different from logit imitation, the contrastive paradigm only enforces the contexts encoded in each modality are the same, providing the freedom for the model to learn the representation encoded in the respective modality.
Revised: Different from logit imitation, the contrastive paradigm only enforces the contexts encoded in each modality are the same, providing the freedom for the model to learn the representation encoded in the respective modality.

Original: C. W HAT DOES THE MODEL LEARN FROM ITS CROSS - MODAL TEACHER ?
Revised: C. W HAT DOES THE MODEL LEARN FROM ITS CROSS - MODAL TEACHER ?

Original: Different from the previous knowledge distillation framework, BEVDistill leverages LiDAR models to transfer knowledge.
Revised: Different from the previous knowledge distillation framework, BEVDistill leverages LiDAR models to transfer knowledge.

Original: A natural question is: what does a multi-view vision-based student learn from its LiDAR-based 3D teacher?
Revised: A natural question is: what does a multi-view vision-based student learn from its LiDAR-based 3D teacher?

Original: Actually, it is hard to figure out the explicit knowledge learned by the student, however, we can get some hints to some extent by comparing the detection performance evaluated from various aspects.
Revised: Actually, it is hard to ﬁgure out the explicit knowledge learned by the student, however, we can get some hints to some extent by comparing the detection performance evaluated from various aspects.

Original: The detailed results are shown in Table 11. [Table]
Revised: The detailed results are shown in Table 12.

Original: BEVFormer

Original: [Table]
Revised: [Table]

Original: + BEVDistill

Original: [Table]

Original: [Table]

Original: [Table] [Table] [Table] From the table we can find that the most significant improvements are mainly related to object localization, orientation and velocity estimation.
Revised: From the table we can ﬁnd that the most signiﬁcant improvements are mainly related to object localization, orientation and velocity estimation.

Original: LiDAR point clouds provide accurate localization information, which is of great benefit to localizing the object, including the absolute 3D space and its orientation.
Revised: LiDAR point clouds provide accurate localization information, which is of great beneﬁt to localizing the object, including the absolute 3D space and its orientation.

Original: Both of them are more challenging to acquire from the camera view only.
Revised: Both of them are more challenging to acquire from the camera view only.

Original: Note that the velocity estimation gets a 10% enhancement since the LiDAR model introduces more spatial information ( i.e., the LiDAR model takes 10 consecutive sweeps of point clouds as inputs instead of a single frame).
Revised: Note that the velocity estimation gets a 10% enhancement since the LiDAR model introduces more spatial information ( i.e., the LiDAR model takes 10 consecutive sweeps of point clouds as inputs instead of a single frame).

Original: As for the attribute prediction, which is mostly bound up with the classification tasks, BEVDistill fails to gain satisfying performance.
Revised: As for the attribute prediction, which is mostly bound up with the classiﬁcation tasks, BEVDistill fails to gain satisfying performance.

Original: We attribute the reason to the deficiency of point clouds in semantic representation compared to the imagery data, which may also account for the failure in KL distillation on the classification branch discussed in Section 3.2.2.
Revised: We attribute the reason to the deﬁciency of point clouds in semantic representation compared to the imagery data, which may also account for the failure in KL distillation on the classiﬁcation branch discussed in Section 3.2.2.

Original: D Q UALITATIVE R ESULTS
Revised: D Q UALITATIVE R ESULTS

Revised: We provide some qualitative results in Figure 2.

Revised: More visualizations can refer to the supplementary materials, which are presented in a video format.