Original: M ONO D ISTILL : L EARNING S PATIAL F EATURES FOR M ONOCULAR 3D O BJECT D ETECTION
Revised: M ONO D ISTILL : L EARNING S PATIAL F EATURES FOR M ONOCULAR 3D O BJECT D ETECTION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: 3D object detection is a fundamental and challenging task for 3D scene understanding, and the monocular-based methods can serve as an economical alternative to the stereo-based or LiDAR-based methods.
Revised: 3D object detection is a fundamental and challenging task for 3D scene understanding, and the monocular-based methods can serve as an economical alternative to the stereo-based or LiDAR-based methods.

Original: However, accurately detecting objects in the 3D space from a single image is extremely difficult due to the lack of spatial cues.
Revised: However, accurately detecting objects in the 3D space from a single image is extremely difficult due to the lack of spatial cues.

Original: To mitigate this issue, we propose a simple and effective scheme to introduce the spatial information from LiDAR signals to the monocular 3D detectors, without introducing any extra cost in the inference phase.
Revised: To mitigate this issue, we propose a simple and effective scheme to introduce the spatial information from LiDAR signals to the monocular 3D detectors, without introducing any extra cost in the inference phase.

Original: In particular, we first project the LiDAR signals into the image plane and align them with the
Revised: In particular, we first project the LiDAR signals into the image plane and align them with the

Original: RGB images.
Revised: RGB images.

Original: After that, we use the resulting data to train a 3D detector (LiDAR
Revised: After that, we use the resulting data to train a 3D detector (LiDAR

Original: Net) with the same architecture as the baseline model.
Revised: Net) with the same architecture as the baseline model.

Original: Finally, this LiDAR Net can serve as the teacher to transfer the learned knowledge to the baseline model.
Revised: Finally, this LiDAR Net can serve as the teacher to transfer the learned knowledge to the baseline model.

Original: Experimental results show that the proposed method can significantly boost the performance of the baseline model and ranks the 1 st place among all monocularbased methods on the KITTI benchmark.
Revised: Experimental results show that the proposed method can significantly boost the performance of the baseline model and ranks the 1 st place among all monocularbased methods on the KITTI benchmark.

Original: Besides, extensive ablation studies are conducted, which further prove the effectiveness of each part of our designs and illustrate what the baseline model has learned from the LiDAR Net.
Revised: Besides, extensive ablation studies are conducted, which further prove the effectiveness of each part of our designs and illustrate what the baseline model has learned from the LiDAR Net.

Original: Our code will be publicly available.
Revised: Our code will be released at https://github.com/monster-ghost/MonoDistill .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: 3D object detection is an indispensable component for 3D scene perception, which has wide applications in the real world, such as autonomous driving and robotic navigation.
Revised: 3D object detection is an indispensable component for 3D scene perception, which has wide applications in the real world, such as autonomous driving and robotic navigation.

Original: Although the algorithms with stereo (Li et al., 2019b; Wang et al., 2019; Chen et al., 2020a) or LiDAR sensors (Qi et al., 2018; Shi et al., 2019; 2020) show promising performances, the heavy dependence on the expensive equipment restricts the application of these algorithms.
Revised: Although the algorithms with stereo (Li et al., 2019b; Wang et al., 2019; Chen et al., 2020a) or LiDAR sensors (Qi et al., 2018; Shi et al., 2019; 2020) show promising performances, the heavy dependence on the expensive equipment restricts the application of these algorithms.

Original: Accordingly, the methods based on the cheaper and more easy-to-deploy monocular cameras (Xu & Chen, 2018; Ma et al., 2019; 2021;
Revised: Accordingly, the methods based on the cheaper and more easy-to-deploy monocular cameras (Xu & Chen, 2018; Ma et al., 2019; 2021;

Original: Brazil & Liu, 2019; Ding et al., 2020) show great potentials and have attracted lots of attention.
Revised: Brazil & Liu, 2019; Ding et al., 2020) show great potentials and have attracted lots of attention.

Original: As shown in Figure 1 (a), some prior works (Brazil & Liu, 2019; Simonelli et al., 2019; Chen et al., 2020b) directly estimate the 3D bounding boxes from monocular images.
Revised: As shown in Figure 1 (a), some prior works (Brazil & Liu, 2019; Simonelli et al., 2019; Chen et al., 2020b) directly estimate the 3D bounding boxes from monocular images.

Original: However, because of the lack of depth cues, it is extremely hard to accurately detect the objects in the 3D space, and the localization error is the major issue of these methods (Ma et al., 2021).
Revised: However, because of the lack of depth cues, it is extremely hard to accurately detect the objects in the 3D space, and the localization error is the major issue of these methods (Ma et al., 2021).

Original: To mitigate this problem, an intuitive idea is to estimate the depth maps from RGB images, and then use them to augment the input data (Figure 1 (b))
Revised: To mitigate this problem, an intuitive idea is to estimate the depth maps from RGB images, and then use them to augment the input data (Figure 1 (b))

Original: (Xu & Chen, 2018; Ding et al., 2020) or directly use them as the input data (Figure 1 (c))
Revised: (Xu & Chen, 2018; Ding et al., 2020) or directly use them as the input data (Figure 1 (c))

Original: (Wang et al., 2019; Ma et al., 2019).
Revised: (Wang et al., 2019; Ma et al., 2019).

Original: Although these two strategies have made significant improvement in performance, the drawbacks of them can not be ignored: (1) These methods generally use an off-the-shelf depth estimator to generate the depth maps, which introduce lots of computational cost ( e.g. the most commonly used depth estimator (Fu et al., 2018) need about 400ms to process a standard KITTI image).
Revised: Although these two strategies have made significant improvement in performance, the drawbacks of them can not be ignored: (1) These methods generally use an off-the-shelf depth estimator to generate the depth maps, which introduce lots of computational cost ( e.g. the most commonly used depth estimator (Fu et al., 2018) need about 400ms to process a standard KITTI image).

Original: (2) The depth estimator and detector are trained separately, which may lead to a sub-optimal optimization.
Revised: (2) The depth estimator and detector are trained separately, which may lead to a sub-optimal optimization.

Original: Recently, Reading et al.
Revised: Recently, Reading et al.

Original: (2021) propose an end-to-end framework (Figure 1 (d)) for monocular 3D detection, which can also leverage depth estimator to provide depth cues.
Revised: (2021) propose an end-to-end framework (Figure 1 (d)) for monocular 3D detection, which can also leverage depth (b) (c) RGB-basedmethod Depth-basedmethod

Revised: Ours

Original: Specifically, they introduce a sub-network to estimate the depth distribution and use it to enrich the RGB features.
Revised: RGB Data Depth Data 3D Box Pseudo-LiDAR T (d) 3D Box RGB Data

Revised: Depth Data (a)

Revised: RGB Data 3D Box

Revised: D

Revised: D

Revised: DE

Revised: DE

Revised: D

Revised: RGB Data 3D Box

Revised: Depth Distribution

Revised: Image Feature B

Revised: DE

Revised: D

Revised: RGB Data

Revised: LiDAR Depth 3D Box 3D Box

Revised: Feature Guidance

Revised: Response Guidance (e)

Revised: D

Revised: D

Revised: Inference Training

Revised: T Data Transformation

Revised: OuterProduct

Revised: DE

Revised: D DepthEstimator

Revised: Detector

Revised: B Backbone estimator to provide depth cues.

Revised: Specifically, they introduce a sub-network to estimate the depth distribution and use it to enrich the RGB features.

Original: Although this model can be trained end-to-end and achieves better performance, it still suffer from the low inference speed (630ms per image), mainly caused by the depth estimator and the complicated network architecture.
Revised: Although this model can be trained end-to-end and achieves better performance, it still suffer from the low inference speed (630ms per image), mainly caused by the depth estimator and the complicated network architecture.

Original: Note that the welldesigned monocular detectors, like Ma et al.
Revised: Note that the welldesigned monocular detectors, like Ma et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: (2021b)
Revised: (2021b)

Original: ; Lu et al.
Revised: ; Lu et al.

Original: (2021), only take about 40ms per image.
Revised: (2021), only take about 40ms per image.

Original: In this paper, we aim to introduce the depth cues to the monocular 3D detectors without introducing any extra cost in the inference phase.
Revised: In this paper, we aim to introduce the depth cues to the monocular 3D detectors without introducing any extra cost in the inference phase.

Original: Inspired by the knowledge distillation (Hinton et al.), which can transfer the learned knowledge from a well-trained CNN to another one without any changes in the model design, we propose that the spatial cues may also be transferred by this way from the
Revised: Inspired by the knowledge distillation (Hinton et al.), which can transfer the learned knowledge from a well-trained CNN to another one without any changes in the model design, we propose that the spatial cues may also be transferred by this way from the

Original: LiDAR-based models.
Revised: LiDAR-based models.

Original: However, the main problem for this proposal is the difference of the feature representations used in these two kinds of methods (2D images features vs. 3D voxel features).
Revised: However, the main problem for this proposal is the difference of the feature representations used in these two kinds of methods (2D images features vs. 3D voxel features).

Original: To bridge this gap, we propose to project the LiDAR signals into the image plane and use the 2D CNN, instead of the commonly used 3D CNN or point-wise CNN, to train a ‘image-version’ LiDAR-based model.
Revised: To bridge this gap, we propose to project the LiDAR signals into the image plane and use the 2D CNN, instead of the commonly used 3D CNN or point-wise CNN, to train a ‘image-version’ LiDAR-based model.

Original: After this alignment, the knowledge distillation can be friendly applied to enrich the features of our monocular detector.
Revised: After this alignment, the knowledge distillation can be friendly applied to enrich the features of our monocular detector.

Original: Based on the above-mentioned motivation and strategy, we propose the distill ation based mono cular 3D detector ( MonoDistill ): We first train a teacher net using the projected LiDAR maps (used as the ground truths of the depth estimator in previous works), and then train our monocular 3D detector under the guidance of the teacher net.
Revised: Based on the above-mentioned motivation and strategy, we propose the distill ation based mono cular 3D detector ( MonoDistill ): We first train a teacher net using the projected LiDAR maps (used as the ground truths of the depth estimator in previous works), and then train our monocular 3D detector under the guidance of the teacher net.

Original: We argue that, compared with previous works, the proposed method has the following two advantages: First, our method directly learn the spatial cues from the teacher net, instead of the estimated depth maps.
Revised: We argue that, compared with previous works, the proposed method has the following two advantages: First, our method directly learn the spatial cues from the teacher net, instead of the estimated depth maps.

Original: This design performs better by avoiding the information loss in the proxy task.
Revised: This design performs better by avoiding the information loss in the proxy task.

Original: Second, our method does not change the network architecture of the baseline model, and thus no extra computational cost is introduced.
Revised: Second, our method does not change the network architecture of the baseline model, and thus no extra computational cost is introduced.

Original: Experimental results on the most commonly used KITTI benchmark, where we rank the 1 st place among all monocular based models by applying the proposed method on a simple baseline, demonstrate the effectiveness of our approach.
Revised: Experimental results on the most commonly used KITTI benchmark, where we rank the 1 st place among all monocular based models by applying the proposed method on a simple baseline, demonstrate the effectiveness of our approach.

Original: Besides, we also conduct extensive ablation studies to present each design of our method in detail.
Revised: Besides, we also conduct extensive ablation studies to present each design of our method in detail.

Original: More importantly, these experiments clearly illustrate the improvements are achieved by the introduction of spatial cues, instead of other unaccountable factors in CNN.
Revised: More importantly, these experiments clearly illustrate the improvements are achieved by the introduction of spatial cues, instead of other unaccountable factors in CNN.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: 3D Detection from only monocular images.
Revised: 3D detection from only monocular images.

Original: 3D detection from only monocular image data is challenging due to the lack of reliable depth information.
Revised: 3D detection from only monocular image data is challenging due to the lack of reliable depth information.

Original: To alleviate this problem, lots of scholars propose their solutions in different ways, including but not limited to network design (Roddick et al., 2019; Brazil & Liu, 2019;Zhou et al.
Revised: To alleviate this problem, lots of scholars propose their solutions in different ways, including but not limited to network design (Roddick et al., 2019; Brazil & Liu, 2019;

Original: ; Liu et al., 2020; Luo et al., 2021), loss formulation (Simonelli et al., 2019; Ma et al., 2021), 3D prior (Brazil & Liu, 2019), geometric constraint (Mousavian et al., 2017; Qin et al., 2019; Li et al., 2019a; Chen et al., 2020b), or perspective modeling (Zhang et al., 2021a; Lu et al., 2021; Shi et al., 2021).
Revised: Zhou et al., 2019; Liu et al., 2020; Luo et al., 2021), loss formulation (Simonelli et al., 2019; Ma et al., 2021), 3D prior (Brazil & Liu, 2019), geometric constraint (Mousavian et al., 2017; Qin et al., 2019; Li et al., 2019a; Chen et al., 2020b), or perspective modeling (Zhang et al., 2021a; Lu et al., 2021; Shi et al., 2021).

Original: Depth augmented monocular 3D detection.
Revised: Depth augmented monocular 3D detection.

Original: To provide the depth information to the 3D detectors, several works choose to estimate the depth maps from RGB images.
Revised: To provide the depth information to the 3D detectors, several works choose to estimate the depth maps from RGB images.

Original: According to the usage of the estimated depth maps, these methods can be briefly divided into three categories.
Revised: According to the usage of the estimated depth maps, these methods can be briefly divided into three categories.

Original: The first class of these methods use the estimated depth maps to augment the RGB images (Figure 1 (b)).
Revised: The first class of these methods use the estimated depth maps to augment the RGB images (Figure 1 (b)).

Original: In particular,
Revised: In particular,

Original: Xu & Chen (2018) propose three fusion strategies of the RGB images and depth maps, while Ding et al.
Revised: Xu & Chen (2018) propose three fusion strategies of the RGB images and depth maps, while Ding et al.

Original: (2020) and Wang et al.
Revised: (2020) and Wang et al.

Original: (2021a) focus on how to enrich the RGB features with depth maps in the latent feature space.
Revised: (2021a) focus on how to enrich the RGB features with depth maps in the latent feature space.

Original: Besides, Wang et al.
Revised: Besides, Wang et al.

Original: ; Ma et al.
Revised: ; Ma et al.

Original: (2019) propose another pipeline (Figure 1 (c)): They back-project the depth maps into the 3D space, and then train a LiDAR-based model and use the resulting data (pseudo-LiDAR signal) to predict the 3D boxes.
Revised: (2019) propose another pipeline (Figure 1 (c)): They back-project the depth maps into the 3D space, and then train a LiDAR-based model and use the resulting data (pseudo-LiDAR signal) to predict the 3D boxes.

Original: This framework shows promising performance, and lots of works (Weng & Kitani, 2019;
Revised: This framework shows promising performance, and lots of works (Weng & Kitani, 2019;

Original: Cai et al., 2020; Wang et al., 2020a; Chu et al., 2021) are built on this solid foundation.
Revised: Cai et al., 2020; Wang et al., 2020a; Chu et al., 2021) are built on this solid foundation.

Original: Recently, Reading et al.
Revised: Recently, Reading et al.

Original: (2021) propose another way to leverage the depth cues for monocular 3D detection (Figure 1 (d)).
Revised: (2021) propose another way to leverage the depth cues for monocular 3D detection (Figure 1 (d)).

Original: Particularly, they first estimate the depth distribution using a sub-network, and then use it to lift the 2D features into 3D features, which is used to generate the final results.
Revised: Particularly, they first estimate the depth distribution using a sub-network, and then use it to lift the 2D features into 3D features, which is used to generate the final results.

Original: Compared with the previous two families, this model can be trained in the end-to-end manner, avoiding the sub-optimal optimization.
Revised: Compared with the previous two families, this model can be trained in the end-to-end manner, avoiding the sub-optimal optimization.

Original: However, a common disadvantage of these methods is that they inevitably increase the computational cost while introducing depth information.
Revised: However, a common disadvantage of these methods is that they inevitably increase the computational cost while introducing depth information.

Original: Unlike these methods, our model chooses to learn the feature representation under the guidance of depth maps, instead of integrating them.
Revised: Unlike these methods, our model chooses to learn the feature representation under the guidance of depth maps, instead of integrating them.

Original: Accordingly, the proposed model not only introduces rich depth cues but also maintains high efficiency.
Revised: Accordingly, the proposed model not only introduces rich depth cues but also maintains high efficiency.

Original: Knowledge distillation.
Revised: Knowledge distillation.

Original: Knowledge distillation (KD) is initially proposed by Hinton et al.
Revised: Knowledge distillation (KD) is initially proposed by Hinton et al.

Original: for model compression, and the main idea of this mechanism is transferring the learned knowledge from large CNN models to the small one.
Revised: for model compression, and the main idea of this mechanism is transferring the learned knowledge from large CNN models to the small one.

Original: This strategy has been proved in many computer vision tasks, such as 2D object detection (Dai et al., 2021; Chen et al., 2017; Gupta et al., 2016), semantic semantic segmentation (Hou et al., 2020b; Liu et al., 2019).
Revised: This strategy has been proved in many computer vision tasks, such as 2D object detection (Dai et al., 2021; Chen et al., 2017; Gupta et al., 2016), semantic semantic segmentation (Hou et al., 2020; Liu et al., 2019).

Original: However, few work explore it in monocular 3D detection.
Revised: However, few work explore it in monocular 3D detection.

Original: In this work, we design a KD-based paradigm to efficiently introduce depth cues for monocular 3D detectors.
Revised: In this work, we design a KD-based paradigm to efficiently introduce depth cues for monocular 3D detectors.

Original: LIGA Stereo.
Revised: LIGA Stereo.

Original: We found a recent work LIGA Stereo
Revised: We found a recent work LIGA Stereo

Original: (Guo et al., 2021) (submitted to arXiv on
Revised: (Guo et al., 2021) (submitted to arXiv on

Original: Aug. 2021) discusses the application of KD for stereo 3D detection under the guidance of LiDAR signals.
Revised: Aug. 2021) discusses the application of KD for stereo 3D detection under the guidance of LiDAR signals.

Original: Here we discuss the main differences of LIGA Stereo and our work.
Revised: Here we discuss the main differences of LIGA Stereo and our work.

Original: First, the tasks and underlying data are different (monocular vs. stereo), which leads to different conclusions.
Revised: First, the tasks and underlying data are different (monocular vs. stereo), which leads to different conclusions.

Original: For example, Guo et al.
Revised: For example, Guo et al.

Original: (2021) concludes that using the predictions of teacher net as ‘soft label’ can not bring benefits.
Revised: (2021) concludes that using the predictions of teacher net as ‘soft label’ can not bring benefits.

Original: However, our experimental results show the effectiveness of this design.
Revised: However, our experimental results show the effectiveness of this design.

Original: Even more, in our task, supervising the student net in the result space is more effective than feature space.
Revised: Even more, in our task, supervising the student net in the result space is more effective than feature space.

Original: Second, they use an off-the-shelf LiDAR-based model to provide guidance to their model.
Revised: Second, they use an off-the-shelf LiDAR-based model to provide guidance to their model.

Original: However, we project the LiDAR signals into image plane and use the resulting data to train the teacher net.
Revised: However, we project the LiDAR signals into image plane and use the resulting data to train the teacher net.

Original: Except for the input data, the teacher net and student net are completely aligned, including network architecture, hyper-parameters, and training schedule.
Revised: Except for the input data, the teacher net and student net are completely aligned, including network architecture, hyper-parameters, and training schedule.

Original: Third, to ensure the consistent shape of features, LIGA Stereo need to generate the cost volume from stereo images, which is time-consuming (it need about 350ms to estimate 3D boxes from a KITTI image) and hard to achieve for monocular images.
Revised: Third, to ensure the consistent shape of features, LIGA Stereo need to generate the cost volume from stereo images, which is time-consuming (it need about 350ms to estimate 3D boxes from a KITTI image) and hard to achieve for monocular images.

Original: In contrast, our method align the feature representations by adjusting the LiDAR-based model, instead of the target model.
Revised: In contrast, our method align the feature representations by adjusting the LiDAR-based model, instead of the target model.

Original: This design makes our method more efficient (about 35ms per image) and can generalize to all kinds of image-based models in theory.
Revised: This design makes our method more efficient (about 35ms per image) and can generalize to all kinds of image-based models in theory.

Original:  M ETHOD 3. O VERVIEW
Revised:  M ETHOD 3. O VERVIEW

Original: Figure 3 presents the framework of the proposed MonoDistill, which mainly has three components: a monocular 3D detector, an aligned LiDAR-based detector, and several side branches which build  the bridge to provide guidance from the LiDAR-based detector to our monocular 3D detector.
Revised: Figure 3 presents the framework of the proposed MonoDistill, which mainly has three components: a monocular 3D detector, an aligned LiDAR-based detector, and several side branches which build [Figure] the bridge to provide guidance from the LiDAR-based detector to our monocular 3D detector.

Original: We will introduce the how to build these parts one by one in the rest of this section.
Revised: We will introduce the how to build these parts one by one in the rest of this section.

Original: 3. B ASELINE M ODEL
Revised: 3. B ASELINE M ODEL

Original: Student model.
Revised: Student model.

Original: We use the one-stage monocular 3D detector MonoDLE (Ma et al., 2021) as our baseline model.
Revised: We use the one-stage monocular 3D detector MonoDLE (Ma et al., 2021) as our baseline model.

Original: Particularly, this baseline model adopts DLA-34 (Yu et al., 2017) as the backbone and uses several parallel heads to predict the required items for 3D object detection.
Revised: Particularly, this baseline model adopts DLA-34 (Yu et al., 2017) as the backbone and uses several parallel heads to predict the required items for 3D object detection.

Original: Due to this clean and compact design, this model achieves good performance with high efficiency.
Revised: Due to this clean and compact design, this model achieves good performance with high efficiency.

Original: Besides, we further normalize the confidence of each predicted object using the estimated depth uncertainty (see
Revised: Besides, we further normalize the confidence of each predicted object using the estimated depth uncertainty (see

Original: Appendix A.1 for more details), which brings about 1 AP improvement 1 .
Revised: Appendix A.1 for more details), which brings about 1 AP improvement * .

Original: Teacher model.
Revised: Teacher model.

Original: Existing LiDAR-based models are mainly based on the 3D CNN or point-wise
Revised: Existing LiDAR-based models are mainly based on the 3D CNN or point-wise

Original: CNN.
Revised: CNN.

Original: To align the gap between the feature representations of the monocular detector and the
Revised: To align the gap between the feature representations of the monocular detector and the

Original: LiDAR-based detector
Revised: LiDAR-based detector

Original: , we project the LiDAR points into the image plane to generate the sparse depth map.
Revised: , we project the LiDAR points into the image plane to generate the sparse depth map.

Original: Further, we also use the interpolation algorithm
Revised: Further, we also use the interpolation algorithm

Original: (Ku et al., 2018) to generate the dense depth, and see Figure 2 for the visualization of generated data.
Revised: (Ku et al., 2018) to generate the dense depth, and see Figure 2 for the visualization of generated data.

Original: Then, we use these ‘image-like
Revised: Then, we use these ‘image-like

Original: LiDAR maps’ to train a LiDAR-based detector using the identical network with our student model.
Revised: LiDAR maps’ to train a LiDAR-based detector using the identical network with our student model.

Original: 3. M ONO D ISTILL
Revised: 3. M ONO D ISTILL

Original: In order to transfer the spatial cues from the well-trained teacher model to the student model, we design three complementary distillation schemes to provide additional guidance to the baseline model.
Revised: In order to transfer the spatial cues from the well-trained teacher model to the student model, we design three complementary distillation schemes to provide additional guidance to the baseline model.

Original: Scene-level distillation in the feature space.
Revised: Scene-level distillation in the feature space.

Original: First, we think directly enforcing the image-based model learns the feature representations of the LiDAR-based models is sub-optimal, caused by the different modalities.
Revised: First, we think directly enforcing the image-based model learns the feature representations of the LiDAR-based models is sub-optimal, caused by the different modalities.

Original: The scene level knowledge can help the monocular 3D detectors build a highlevel understanding for the given image by encoding the relative relations of the features, keeping the knowledge structure and alleviating the modality gap.
Revised: The scene level knowledge can help the monocular 3D detectors build a highlevel understanding for the given image by encoding the relative relations of the features, keeping the knowledge structure and alleviating the modality gap.

Original: Therefore, we train our student model under the guidance of the high-level semantic features provided by the backbone of the teacher model.
Revised: Therefore, we train our student model under the guidance of the high-level semantic features provided by the backbone of the teacher model.

Original: To better model the structured cues, we choose to learn the affinity map (Hou et al., 2020a) of high-level features, instead of the features themselves.
Revised: To better model the structured cues, we choose to learn the affinity map (Hou et al., 2020) of high-level features, instead of the features themselves.

Original: Specifically, we first generate the affinity map, which encodes the similarity of each feature vector pair, for both the teacher and student network, and each element A i , j in this affinity map can be computed by:
Revised: Specifically, we first generate the affinity map, which encodes the similarity of each feature vector pair, for both the teacher and student network, and each element A i , j in this affinity map can be computed by:

Original: [Equation]
Revised: [Equation]

Original: where f j and f j denote the i th and j th feature vector.
Revised: where f j and f j denote the i th and j th feature vector.

Original: After that, we use the L1 norm to enforce the student net to learn the structured information from the teacher net:
Revised: After that, we use the L1 norm to enforce the student net to learn the structured information from the teacher net:

Original: [Equation]
Revised: [Equation]

Original: where K is the number of the feature vectors.
Revised: where K is the number of the feature vectors.

Original: Note the computational/storage complexity is quadratically related to K. To reduce the cost, we group all features into several local regions and generate the affinity map using the features of local regions.
Revised: Note the computational/storage complexity is quadratically related to K. To reduce the cost, we group all features into several local regions and generate the affinity map using the features of local regions.

Original: This makes the training of the proposed model more efficient, and we did not observe any performance drop caused by this strategy.
Revised: This makes the training of the proposed model more efficient, and we did not observe any performance drop caused by this strategy.

Original: Object-level distillation in the feature space.
Revised: Object-level distillation in the feature space.

Original: Second, except for the affinity map, directly using the features from teacher net as guidance may also provide valuable cues to the student.
Revised: Second, except for the affinity map, directly using the features from teacher net as guidance may also provide valuable cues to the student.

Original: However, there is much noise in feature maps, since the background occupies most of the area and is less informative.
Revised: However, there is much noise in feature maps, since the background occupies most of the area and is less informative.

Original: Distilling knowledge from these regions may make the network deviate from the right optimization direction.
Revised: Distilling knowledge from these regions may make the network deviate from the right optimization direction.

Original: To make the knowledge distillation more focused, limiting the distillation area is necessary.
Revised: To make the knowledge distillation more focused, limiting the distillation area is necessary.

Original: Particularly, the regions in the ground-truth 2D bounding boxes are used for knowledge transfer to mitigate the effects of noise.
Revised: Particularly, the regions in the ground-truth 2D bounding boxes are used for knowledge transfer to mitigate the effects of noise.

Original: Specifically, given the feature maps of the teacher model and student model { F t , F s } , our second distillation loss can be formulated as.
Revised: Specifically, given the feature maps of the teacher model and student model { F t , F s } , our second distillation loss can be formulated as.

Original: [Equation]
Revised: [Equation]

Original: where M of is the mask generated from the center point and the size of 2D bounding box and N pos is the number of valid feature vectors.
Revised: where M of is the mask generated from the center point and the size of 2D bounding box and N pos is the number of valid feature vectors.

Original: Object-level distillation in the result space.
Revised: Object-level distillation in the result space.

Original: Third, similar to the traditional KD, we use the predictions from the teacher net as extra ‘soft label’ for the student net.
Revised: Third, similar to the traditional KD, we use the predictions from the teacher net as extra ‘soft label’ for the student net.

Original: Note that in this scheme, only the predictions on the foreground region should be used, because the predictions on the background region are usually false detection.
Revised: Note that in this scheme, only the predictions on the foreground region should be used, because the predictions on the background region are usually false detection.

Original: As for the definition of the ‘foreground regions’, inspired by CenterNet (Zhou et al.), a simple baseline is regarding the center point as the foreground region.
Revised: As for the definition of the ‘foreground regions’, inspired by CenterNet (Zhou et al., 2019), a simple baseline is regarding the center point as the foreground region.

Original: Furtherly, we find that the quality of the predicted value of the teacher net near the center point is good enough to guide the student net.
Revised: Furtherly, we find that the quality of the predicted value of the teacher net near the center point is good enough to guide the student net.

Original: Therefore, we generate a Gaussian-like mask (Tian et al., 2019;
Revised: Therefore, we generate a Gaussian-like mask (Tian et al., 2019;

Original: Wang et al., 2021b)
Revised: Wang et al., 2021b)

Original: based on the position of the center point and the size of 2D bounding box and the pixels whose response values surpass a predefined threshold are sampled, and then we train these samples with equal weights (see Figure 4 for the visualization).
Revised: based on the position of the center point and the size of 2D bounding box and the pixels whose response values surpass a predefined threshold are sampled, and then we train these samples with equal weights (see Figure 4 for the visualization).

Original: After that, our third distillation loss can be formulated as:
Revised: After that, our third distillation loss can be formulated as:

Original: [Equation]
Revised: [Equation]

Original: where M or is the mask which represents positive and negative samples, y k is the output of the k th detection head and N is the number of detection heads.
Revised: where M or is the mask which represents positive and negative samples, y k is the output of the k th detection head and N is the number of detection heads.

Original: Additional strategies.
Revised: Additional strategies.

Original: We further propose some strategies for our method.
Revised: We further propose some strategies for our method.

Original: First, for the distillation schemes in the feature space ( i.e. L sf and L of ), we only perform them on the last three blocks of the backbone.
Revised: First, for the distillation schemes in the feature space ( i.e. L sf and L of ), we only perform them on the last three blocks of the backbone.

Original: The main motivation of this strategy is: The first block usually is rich in the low-level features (such as edges, textures, etc.).
Revised: The main motivation of this strategy is: The first block usually is rich in the low-level features (such as edges, textures, etc.).

Original: The expression forms of the low-level features for LiDAR and image data may be completely different, and enforcing the student net to learn these features in a modality-across manner may mislead it.
Revised: The expression forms of the low-level features for LiDAR and image data may be completely different, and enforcing the student net to learn these features in a modality-across manner may mislead it.

Original: Second, in order to better guide the student to learn spatial-aware feature representations, we apply the attention based fusion module (FF in Table 1) proposed by Chen et al.
Revised: Second, in order to better guide the student to learn spatial-aware feature representations, we apply the attention based fusion module (FF in Table 1) proposed by Chen et al.

Original: (2021b)
Revised: (2021b)

Original: in our distillation schemes in the feature space ( i.e. L sf and L of ).
Revised: in our distillation schemes in the feature space ( i.e. L sf and L of ).

Original: Loss function.
Revised: Loss function.

Original: We train our model in an end-to-end manner using the following loss function:
Revised: We train our model in an end-to-end manner using the following loss function:

Original: [Equation]
Revised: [Equation]

Original: where L src denotes the loss function used in the MonoDLE (Ma et al., 2021).
Revised: where L src denotes the loss function used in the MonoDLE (Ma et al., 2021).

Original: λ 1 , λ 2 , λ 3 are the hyper-parameters to balance each loss.
Revised: λ 1 , λ 2 , λ 3 are the hyper-parameters to balance each loss.

Original: For the teacher net, only L src is adopted.
Revised: For the teacher net, only L src is adopted.

Original:  E XPERIMENTS 4. S ETUP
Revised:  E XPERIMENTS 4. S ETUP

Original: Dataset and metrics.
Revised: Dataset and metrics.

Original: We conduct our experiments on the KITTI (Geiger et al., 2012), which is most commonly used dataset in 3D detection task.
Revised: We conduct our experiments on the KITTI (Geiger et al., 2012), which is most commonly used dataset in 3D detection task.

Original: Specifically, this dataset provides 7,481 training samples and 7,518 testing samples, and we further divide the training data into a train set (3, samples) and a validation set (3,769 samples), following prior works (Chen et al., 2015).
Revised: Specifically, this dataset provides 7,481 training samples and 7,518 testing samples, and we further divide the training data into a train set (3, samples) and a validation set (3,769 samples), following prior works (Chen et al., 2015).

Original: Both 3D detection and Bird’s Eye View (BEV) detection are evaluated using AP | R 40 (Simonelli et al., 2019) as metric.
Revised: Both 3D detection and Bird’s Eye View (BEV) detection are evaluated using AP | R 40 (Simonelli et al., 2019) as metric.

Original: We report our final results on the testing set, while the ablation studies are conducted on the validation set.
Revised: We report our final results on the testing set, while the ablation studies are conducted on the validation set.

Original: Besides, we mainly focus on the Car category, while also present the performances of Pedestrian and Cyclist in Appendix A.2 for reference.
Revised: Besides, we mainly focus on the Car category, while also present the performances of Pedestrian and Cyclist in Appendix A.2 for reference.

Original: Implementation.
Revised: Implementation.

Original: We provide the implementation details in Appendix A.1.
Revised: We provide the implementation details in Appendix A.1.

Original: Besides, our code will be open-sourced for the reproducibility.
Revised: Besides, our code will be open-sourced for the reproducibility.

Original: 4. M AIN R ESULTS
Revised: 4. M AIN R ESULTS

Original: Ablation studies.
Revised: Ablation studies.

Original: Table 1 shows the ablation studies of the proposed methods.
Revised: Table 1 shows the ablation studies of the proposed methods.

Original: Specifically, we found that all three distillation schemes can improve the accuracy of the baseline model, and the improvements of them are complementary.
Revised: Specifically, we found that all three distillation schemes can improve the accuracy of the baseline model, and the improvements of them are complementary.

Original: Besides, the feature fusion strategy can also boost the accuracy.
Revised: Besides, the feature fusion strategy can also boost the accuracy.

Original: Compared with the baseline, our full model improves 3D detection performance by 3.34 , 5.02 , 2.98 and improve BEV performance by 5.16 , 6.62 , 3.87 on the moderate, easy and hard settings respectively. 
Revised: Compared with the baseline, our full model improves 3D detection performance by 3.34 , 5.02 , 2.98 and improve BEV performance by 5.16 , 6.62 , 3.87 on the moderate, easy and hard settings respectively. SF OF OR

Revised: FF

Original: [Table]
Revised: [Table]

Original: @IOU=0.
Revised: IOU=0.

Original: BEV@IOU=0.
Revised: BEV@IOU=0.

Original: Mod.
Revised: Mod.

Original: Easy
Revised: Easy

Original: Hard
Revised: Hard

Original: Mod.
Revised: Mod.

Original: Easy
Revised: Easy

Original: Hard a. b.
Revised: Hard a. b.

Original: ✓ 16.
Revised: ✓ 16.

Original: c.
Revised: c.

Original: ✓ 16.
Revised: ✓ 16.

Original: d.
Revised: d.

Original: ✓ 17.
Revised: ✓ 17.

Original: e.
Revised: e.

Original: ✓ ✓ 17.
Revised: ✓ ✓ 17.

Original: f.
Revised: f.

Original: ✓ ✓ 17.
Revised: ✓ ✓ 17.

Original: g.
Revised: g.

Original: ✓ ✓ 17.
Revised: ✓ ✓ 17.

Original: h.
Revised: h.

Original: ✓ ✓ ✓ 18.
Revised: ✓ ✓ ✓ 18.

Original: i.
Revised: i.

Original: ✓ ✓ ✓ ✓ 18.
Revised: ✓ ✓ ✓ ✓ 18.

Original: Detailed design choice.
Revised: Detailed design choice.

Original: We provide additional experiments in Table 2 for our method.
Revised: We provide additional experiments in Table 2 for our method.

Original: First, as for object-level distillation in the feature space, we investigate the different effects of applying distillation on the whole image and foreground regions.
Revised: First, as for object-level distillation in the feature space, we investigate the different effects of applying distillation on the whole image and foreground regions.

Original: Due to the noise in the background, guiding the foreground regions is more effective than the whole image, which improves the accuracy by 0.72 on the moderate settings in 3D detection.
Revised: Due to the noise in the background, guiding the foreground regions is more effective than the whole image, which improves the accuracy by 0.72 on the moderate settings in 3D detection.

Original: Second, as for object-level distillation in the result space, we compare the different effects of point label and region label.
Revised: Second, as for object-level distillation in the result space, we compare the different effects of point label and region label.

Original: It can be observed that the generated region can significantly increase performance while guiding only in sparse point label brings limited improvements.
Revised: It can be observed that the generated region can significantly increase performance while guiding only in sparse point label brings limited improvements.

Original: Our proposed label diffusion strategy can increase the number of positive samples for supervision, thus improving performance.
Revised: Our proposed label diffusion strategy can increase the number of positive samples for supervision, thus improving performance.

Original: [Table]
Revised: [Table]

Original: Comparison with state-of-the-art methods.
Revised: Comparison with state-of-the-art methods.

Original: Table 3 and Table 4 compare the proposed method with other state-of-the-art methods on the KITTI test and validation sets.
Revised: Table 3 and Table 4 compare the proposed method with other state-of-the-art methods on the KITTI test and validation sets.

Original: On the test set, the proposed method outperforms existing methods in all metrics.
Revised: On the test set, the proposed method outperforms existing methods in all metrics.

Original: We note that, compared with previous best results, we can obtain 1.83, 0.50, 1.53 improvements on the moderate, easy and hard settings in 3D detection.
Revised: We note that, compared with previous best results, we can obtain 1.83, 0.50, 1.53 improvements on the moderate, easy and hard settings in 3D detection.

Original: Furthermore, our method achieves more significant improvements in BEV detection, increasing upon the prior work by 2.51, 1.21, 2.46 on the moderate, easy and hard settings.
Revised: Furthermore, our method achieves more significant improvements in BEV detection, increasing upon the prior work by 2.51, 1.21, 2.46 on the moderate, easy and hard settings.

Original: Moreover, compared with the depth-based methods, our method outperforms them in performance by a margin and is superior to theirs in the inference speed.
Revised: Moreover, compared with the depth-based methods, our method outperforms them in performance by a margin and is superior to theirs in the inference speed.

Original: By contrast, our method only takes 40ms to process a KITTI image, tested on a single NVIDIA GTX 1080Ti, while the Fastest of the depth-based methods (Ma et al., 2019; 2020; Ding et al., 2020; Wang et al., 2021a; Reading et al., 2021) need 180ms.
Revised: By contrast, our method only takes 40ms to process a KITTI image, tested on a single NVIDIA GTX 1080Ti, while the Fastest of the depth-based methods (Ma et al., 2019; 2020; Ding et al., 2020; Wang et al., 2021a; Reading et al., 2021) need 180ms.

Original: On the validation set, the proposed also performs best, both for the 0.7 IoU threshold and 0.5 IoU threshold.
Revised: On the validation set, the proposed also performs best, both for the 0.7 IoU threshold and 0.5 IoU threshold.

Original: Besides, we also present the performance of the baseline model to better show the effectiveness of the proposed method.
Revised: Besides, we also present the performance of the baseline model to better show the effectiveness of the proposed method.

Original: Note that we do not report the performances of some depth-based methods (Ma et al., 2019; 2020; Ding et al., 2020; Wang et al., 2021a) due to the data leakage problem 2 .
Revised: Note that we do not report the performances of some depth-based methods (Ma et al., 2019; 2020; Ding et al., 2020; Wang et al., 2021a) due to the data leakage problem * .

Original: 4. M ORE D ISCUSSIONS
Revised: 4. M ORE D ISCUSSIONS

Original: What has the student model learned from the teacher model?
Revised: What has the student model learned from the teacher model?

Original: To locate the source of improvement, we use the items predicted from the baseline model to replace that from our full model, and
Revised: To locate the source of improvement, we use the items predicted from the baseline model to replace that from our full model, and

Original: Table 5 summarizes the results of the cross-model evaluation.
Revised: Table 5 summarizes the results of the cross-model evaluation.

Original: From these results, we can see that the teacher model provides effective guidance to the location estimation (b → f), and improvement of dimension part is also considerable (c → f).
Revised: From these results, we can see that the teacher model provides effective guidance to the location estimation (b → f), and improvement of dimension part is also considerable (c → f).

Original: Relatively, the teacher model provides limited valuable cues to the classification and orientation part.
Revised: Relatively, the teacher model provides limited valuable cues to the classification and orientation part.

Original: This phenomenon suggests the proposed methods boost the performance of the baseline model mainly by introducing the spatial-related information, which is consistent with our initial motivation.
Revised: This phenomenon suggests the proposed methods boost the performance of the baseline model mainly by introducing the spatial-related information, which is consistent with our initial motivation.

Original: Besides, we also show the errors of depth estimation, see Appendix A.3 for the results.
Revised: Besides, we also show the errors of depth estimation, see Appendix A.3 for the results.

Original: Is the effectiveness of our method related to the performance of the teacher model?
Revised: Is the effectiveness of our method related to the performance of the teacher model?

Original: An intuitive conjecture is the student can learn more if the teacher network has better performance.
Revised: An intuitive conjecture is the student can learn more if the teacher network has better performance.

Original: To explore this problem, we also use the sparse LiDAR maps to train a teacher net to provide guidance to the student model (see Figure 2 for the comparison of the sparse and dense data).
Revised: To explore this problem, we also use the sparse LiDAR maps to train a teacher net to provide guidance to the student model (see Figure 2 for the comparison of the sparse and dense data).

Original: As shown in Table 6, the performance of the teacher model trained from the sparse LiDAR maps is largely behind by that from dense LiDAR maps (drop to 22.05% from 42.45%, moderate setting), while both of them provides comparable benefits to the student model.
Revised: As shown in Table 6, the performance of the teacher model trained from the sparse LiDAR maps is largely behind by that from dense LiDAR maps (drop to 22.05% from 42.45%, moderate setting), while both of them provides comparable benefits to the student model.

Original: Therefore, for our task, the performance of the teacher model is not directly related to the performance improvement, while the more critical factor is whether the teacher network contains complementary information to the student network.
Revised: Therefore, for our task, the performance of the teacher model is not directly related to the performance improvement, while the more critical factor is whether the teacher network contains complementary information to the student network.

Original: Do we need depth estimation as an intermediate task?
Revised: Do we need depth estimation as an intermediate task?

Original: As shown in Figure 1, most previous methods choose to estimate the depth maps to provide depth information for monocular 3D detection (information flow: LiDAR data → estimated depth map → 3D detector).
Revised: As shown in Figure 1, most previous methods choose to estimate the depth maps to provide depth information for monocular 3D detection (information flow: LiDAR data → estimated depth map → 3D detector).

Original: Compared with this
Revised: Compared with this

Original: [Table]
Revised: [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: 120 ms
Revised: 120 ms

Original: DDMP-3D* (Wang et al., 2021a)
Revised: DDMP-3D* (Wang et al., 2021a)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 180 ms
Revised: 180 ms

Original: CaDDN* (Reading et al., 2021)
Revised: CaDDN* (Reading et al., 2021)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 630 ms
Revised: 630 ms

Original: [Table]
Revised: [Table]

Original: Zhou et al., 2021)
Revised: Zhou et al., 2021)

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: 30 ms

Original: [Table]
Revised: [Table]

Original: (Zhang et al., 2021b
Revised: (Zhang et al., 2021b

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: 30 ms
Revised: 30 ms

Original: Autoshape (Liu et al., 2021)
Revised: Autoshape (Liu et al.

Original: [Table]
Revised: [Table]

Original: [Table][Table] [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 50 ms
Revised: 50 ms

Original: [Table]
Revised: [Table]

Original: (Lu et al.,2021)
Revised: (Lu et al.,

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 35 ms
Revised: 35 ms

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]

Original: +0.
Revised: +0.

Original: +1.
Revised: +1.

Original: +2.
Revised: +2.

Original: +1.
Revised: +1.

Original: +2.
Revised: +2.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: d.
Revised: d.

Original: [Table]
Revised: [Table]

Original: O

Original: [Table]

Original: O 18.

Original: e.
Revised: e.

Original: O O
Revised: O O

Original: O
Revised: O

Original: B 18.
Revised: B 18.

Original: f.
Revised: f.

Original: O O
Revised: O O

Original: O
Revised: O

Original: O 18.
Revised: O 18.

Original: scheme, our method directly learns the depth cues from LiDAR-based methods (information flow:
Revised: scheme, our method directly learns the depth cues from LiDAR-based methods (information flow:

Original: LiDAR data → 3D detector), avoiding the information loss in the depth estimation step.
Revised: LiDAR data → 3D detector), avoiding the information loss in the depth estimation step.

Original: Here we quantitatively show the information loss in depth estimation using a simple experiment.
Revised: Here we quantitatively show the information loss in depth estimation using a simple experiment.

Original: Specifically, we use DORN (Fu et al., 2018)
Revised: Specifically, we use DORN (Fu et al., 2018)

Original: (same as most previous depth augmented methods) to generate the depth maps, and then use them to train the teacher net.
Revised: (same as most previous depth augmented methods) to generate the depth maps, and then use them to train the teacher net.

Original: Table 7 shows the results of this experi-
Revised: Table 7 shows the results of this experi-

Original: [Table]
Revised: [Table]

Original: ment.
Revised: ment.

Original: Note that, compared with setting c, setting b’s teacher net is trained from a larger training set (23,488 vs. 3,712) with ground-truth depth maps (ground truth depth maps vs. noisy depth maps).
Revised: Note that, compared with setting c, setting b’s teacher net is trained from a larger training set (23,488 vs. 3,712) with ground-truth depth maps (ground truth depth maps vs. noisy depth maps).

Original: Nevertheless, this scheme still lags behind our original method, which means that there is serious information loss in monocular depth estimation (stereo image performs better, which is discussed in
Revised: Nevertheless, this scheme still lags behind our original method, which means that there is serious information loss in monocular depth estimation (stereo image performs better, which is discussed in

Original: Appendix A.4).
Revised: Appendix A.4).

Original: [Table]
Revised: [Table]

Original: 4. Q UALITATIVE R ESULTS
Revised: 4. Q UALITATIVE R ESULTS

Original: In Figure 5, we show the qualitative comparison of detection results.
Revised: In Figure 5, we show the qualitative comparison of detection results.

Original: We can see that the proposed method shows better localization accuracy than the baseline model.
Revised: We can see that the proposed method shows better localization accuracy than the baseline model.

Original: See Appendix A.7 for more detailed qualitative results.
Revised: See Appendix A.7 for more detailed qualitative results.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we propose the MonoDistill, which introduces spatial cues to the monocular 3D detector based on the knowledge distillation mechanism.
Revised: In this work, we propose the MonoDistill, which introduces spatial cues to the monocular 3D detector based on the knowledge distillation mechanism.

Original: Compared with previous schemes, which share the same motivation, our method avoids any modifications on the target model and directly learns the spatial features from the model rich in these features.
Revised: Compared with previous schemes, which share the same motivation, our method avoids any modifications on the target model and directly learns the spatial features from the model rich in these features.

Original: This design makes the proposed methods perform wellboth in  performance and efficiency.
Revised: This design makes the proposed method perform well in both performance and efficiency.

Original: To show an all-around display of our model, extensive experiments are conducted on the KITTI dataset, where the proposed method ranks 1 st at
Revised: To show an all-around display of our model, extensive experiments are conducted on the KITTI dataset, where the proposed method ranks 1 st at

Original: FPS among all monocular 3D detectors.
Revised: FPS among all monocular 3D detectors.

Revised: A CKNOWLEDGEMENTS

Revised: This work was supported in part by the National Natual Science Foundation of China (NSFC) under

Revised: Grants No.61932020, 61976038, U1908210 and 61772108.

Revised: Wanli Ouyang was supported by the

Revised: Australian Research Council Grant DP200103223, FT210100228, and Australian Medical Research

Original:  A A PPENDIX A. M ORE DETAILS OF THE BASELINE MODEL
Revised: Future Fund MRFAI000085. A A PPENDIX A. M ORE DETAILS OF THE BASELINE MODEL

Original: Network architecture.
Revised: Network architecture.

Original: The baseline network is extended from the anchor-free 2D object detection framework, which consists of a feature extraction network and seven detection subheads.
Revised: The baseline network is extended from the anchor-free 2D object detection framework, which consists of a feature extraction network and seven detection subheads.

Original: We employ
Revised: We employ

Original: DLA-34 (Yu et al., 2017) without deformable convolutions as our backbone.
Revised: DLA-34 (Yu et al., 2017) without deformable convolutions as our backbone.

Original: The feature maps are downsampled by 4 times and then we take the image features as input and use 3x3 convolution,
Revised: The feature maps are downsampled by 4 times and then we take the image features as input and use 3x3 convolution,

Original: ReLU, and 1x1 convolution to output predictions for each detection head.
Revised: ReLU, and 1x1 convolution to output predictions for each detection head.

Original: Detection head branches include three for 2D components and four for 3D components.
Revised: Detection head branches include three for 2D components and four for 3D components.

Original: Specifically, 2D detection heads include heatmap, offset between the 2D key-point and the 2D box center, and size of 2D box.
Revised: Specifically, 2D detection heads include heatmap, offset between the 2D key-point and the 2D box center, and size of 2D box.

Original: 3D components include offset between the 2D key-point and the projected 3D object center, depth, dimensions, and orientations.
Revised: 3D components include offset between the 2D key-point and the projected 3D object center, depth, dimensions, and orientations.

Original: As for objective functions, we train the heatmap with focal loss.
Revised: As for objective functions, we train the heatmap with focal loss.

Original: The other loss items adopt L1 losses except for depth and orientation.
Revised: The other loss items adopt L1 losses except for depth and orientation.

Original: The depth branch employs a modified L1 loss with the assist of heteroscedastic aleatoric uncertainty.
Revised: The depth branch employs a modified L1 loss with the assist of heteroscedastic aleatoric uncertainty.

Original: Common MultiBin loss is used for the orientation branch.
Revised: Common MultiBin loss is used for the orientation branch.

Original: Besides, we propose a strategy to improve the accuracy of baseline.
Revised: Besides, we propose a strategy to improve the accuracy of baseline.

Original: Inspire by (Lu et al., 2021), estimated depth uncertainty can provide confidence for each projection depth.
Revised: Inspire by (Lu et al., 2021), estimated depth uncertainty can provide confidence for each projection depth.

Original: Therefore, we normalize the confidence of each predicted box using depth uncertainty.
Revised: Therefore, we normalize the confidence of each predicted box using depth uncertainty.

Original: In this way, the score has capability of indicating the uncertainty of depth.
Revised: In this way, the score has capability of indicating the uncertainty of depth.

Original: Training details.
Revised: Training details.

Original: Our model is trained on 2 NVIDIA 1080Ti GPUs in an end-to-end manner for 150 epochs.
Revised: Our model is trained on 2 NVIDIA 1080Ti GPUs in an end-to-end manner for 150 epochs.

Original: We employ the common Adam optimizer with initial learning rate 1.25 e − 4 , and decay it by ten times at 90 and 120 epochs.
Revised: We employ the common Adam optimizer with initial learning rate 1.25 e − 4 , and decay it by ten times at 90 and 120 epochs.

Original: To stabilize the training process, we also applied the warm-up strategy (5 epochs).
Revised: To stabilize the training process, we also applied the warm-up strategy (5 epochs).

Original: As for data augmentations, only random random flip and center crop are applied.
Revised: As for data augmentations, only random random flip and center crop are applied.

Original: Same as the common knowledge distillation scheme, we first train teacher network in advance, and then fix the teacher network.
Revised: Same as the common knowledge distillation scheme, we first train teacher network in advance, and then fix the teacher network.

Original: As for student network, we simply train the detection model to give a suitable initialization.
Revised: As for student network, we simply train the detection model to give a suitable initialization.

Original: We implemented our method using PyTorch.
Revised: We implemented our method using PyTorch.

Original: And our code is based on Ma et al.
Revised: And our code is based on Ma et al.

Original: A. P EDESTRIAN /C YCLIST DETECTION .
Revised: A. P EDESTRIAN /C YCLIST DETECTION .

Original: Due to the small sizes, non-rigid structures
Revised: Due to the small sizes, non-rigid structures

Original: [Table]
Revised: [Table]

Original: and limited training samples, the pedestrians and cyclists are much more challenging to detect than cars.
Revised: and limited training samples, the pedestrians and cyclists are much more challenging to detect than cars.

Original: We first report the detection results on test set in Table 8.
Revised: We first report the detection results on test set in Table 8.

Original: It can be seen that our proposed method is also competitive with current state-of-the-art methods on the KITTI test set, which increases 0.69 AP on hard difficulty level of pedestrian category.
Revised: It can be seen that our proposed method is also competitive with current state-of-the-art methods on the KITTI test set, which increases 0.69 AP on hard difficulty level of pedestrian category.

Original: Note that, the accuracy of these difficult categories fluctuates greatly compared with Car detection due to insufficient training samples (see Table 10 for the details).
Revised: Note that, the accuracy of these difficult categories fluctuates greatly compared with Car detection due to insufficient training samples (see Table 10 for the details).

Original: Due the access to the test server is limited, we conduct more experiments for pedestrian/cyclist on the validation set for general conclusions (we run the proposed method three times with different random seeds)
Revised: Due the access to the test server is limited, we conduct more experiments for pedestrian/cyclist on the validation set for general conclusions (we run the proposed method three times with different random seeds)

Original: [Table]
Revised: [Table]

Original: and the experimental results are summarized in Table 9.
Revised: and the experimental results are summarized in Table 9.

Original: According to these results, we can find that the proposed method can effectively boost the accuracy of the baseline model for pedestrian/cyclist detection
Revised: According to these results, we can find that the proposed method can effectively boost the accuracy of the baseline model for pedestrian/cyclist detection

Original: [Table]
Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: D EPTH E RROR A NALYSIS

Original: As shown in Figure
Revised: As shown in Figure

Original: [Table]
Revised: [Table]

Original: we compare the depth error between baseline and our method.
Revised: we compare the depth error between baseline and our method.

Original: Specifically, we project all valid samples of the Car category into the image plane to get the corresponding predicted depth values.
Revised: Specifically, we project all valid samples of the Car category into the image plane to get the corresponding predicted depth values.

Original: Then we fit the depth errors between ground truths and predictions as a linear function by least square method.
Revised: Then we fit the depth errors between ground truths and predictions as a linear function by least square method.

Original: According to the experimental results, we can find that our proposed method can boost the accuracy of depth estimation at different distances
Revised: According to the experimental results, we can find that our proposed method can boost the accuracy of depth estimation at different distances

Original: [Table]
Revised: [Table]

Original: A. T HE E FFECTS OF S TEREO D EPTH
Revised: A. T HE E FFECTS OF S TEREO D EPTH

Original: We also explored the changes in performance under the guidance of estimated stereo depth (Chang &
Revised: We also explored the changes in performance under the guidance of estimated stereo depth (Chang &

Original: Chen, 2018), and show the results in Table 11.
Revised: Chen, 2018), and show the results in Table 11.

Original: Stereo depth estimation exploits geometric constraints in stereo images to obtain the absolute depth value through pixel-wise matching, which is more accurate compared with monocular depth estimation.
Revised: Stereo depth estimation exploits geometric constraints in stereo images to obtain the absolute depth value through pixel-wise matching, which is more accurate compared with monocular depth estimation.

Original: Therefore, under the guidance of stereo depth, the model achieves almost the same accuracy as LiDAR signals guidance at 0.5 IoU threshold, and there is only a small performance drop at 0.7 IoU threshold.
Revised: Therefore, under the guidance of stereo depth, the model achieves almost the same accuracy as LiDAR signals guidance at 0.5 IoU threshold, and there is only a small performance drop at 0.7 IoU threshold.

Original: A. G ENERALIZATION OF THE PROPOSED METHOD
Revised: A. G ENERALIZATION OF THE PROPOSED METHOD

Original: In the main paper, we introduced the proposed method based on MonoDLE (Ma et al., 2021).
Revised: In the main paper, we introduced the proposed method based on MonoDLE (Ma et al., 2021).

Original: Here we discuss the generalization ability of the proposed method.
Revised: Here we discuss the generalization ability of the proposed method.

Original: Generalizing to other baseline models.
Revised: Generalizing to other baseline models.

Original: To show the generalization ability of the proposed method, we apply our method on another monocular detector GUPNet (Lu et al., 2021), which is a two-stage detection method.
Revised: To show the generalization ability of the proposed method, we apply our method on another monocular detector GUPNet (Lu et al., 2021), which is a two-stage detection method.

Original: Experimental results are shown in the Table 12.
Revised: Experimental results are shown in the Table 12.

Original: We can find that the proposed
Revised: We can find that the proposed

Original: [Table]
Revised: [Table]

Original: method can boosts the  of our method.
Revised: method performances confirms the generalization of our method.

Original: [Table]
Revised: [Table]

Original: Generalizing to sparse LiDAR signals.
Revised: Generalizing to sparse LiDAR signals.

Original: We also explore the changes in performance under different resolution of LiDAR signals.
Revised: We also explore the changes in performance under different resolution of LiDAR signals.

Original: In particular, following Pseudo-LiDAR++ (You et al., 2020)
Revised: In particular, following Pseudo-LiDAR++ (You et al., 2020)

Original: [Table]
Revised: [Table]

Original: we generate the simulated 32-beam/16-beam LiDAR signals and use them to train our teacher model (in the ‘sparse’ setting).
Revised: we generate the simulated 32-beam/16-beam LiDAR signals and use them to train our teacher model (in the ‘sparse’ setting).

Original: We show the experimental results, based on MonoDLE, in the Table 13.
Revised: We show the experimental results, based on MonoDLE, in the Table 13.

Original: We can see that, although the improvement is slightly reduced due to the decrease of the resolution of
Revised: We can see that, although the improvement is slightly reduced due to the decrease of the resolution of

Original: LiDAR signals, the proposed method significantly boost the performances 
Revised: LiDAR signals, the proposed method significantly boost the performances of

Original: [Table]
Revised: [Table]

Original: More discussion.
Revised: More discussion.

Original: Besides
Revised: Besides

Original: [Table]
Revised: [Table]

Original: note that the camera parameters of the images on the KITTI test set are different from these of the training/validation set
Revised: note that the camera parameters of the images on the KITTI test set are different from these of the training/validation set

Original: [Table]
Revised: [Table]

Original: and the good performance on the test set suggests the proposed method can also generalize to different camera parameters.
Revised: and the good performance on the test set suggests the proposed method can also generalize to different camera parameters.

Original: However, generalizing to the new scenes with different statistical characteristics is a hard task for existing 3D detectors (Yang et al.
Revised: However, generalizing to the new scenes with different statistical characteristics is a hard task for existing 3D detectors (Yang et al.

Original: ; Wang et al., 2020b), including the image-based models and LiDAR-based models
Revised: ; Wang et al., 2020b), including the image-based models and LiDAR-based models

Original: [Table]
Revised: [Table]

Original: and deserves further investigation by future works.
Revised: and deserves further investigation by future works.

Original: We also argue that the proposed method can generalize to the new scenes better than other monocular models because ours model learns the stronger features from the teacher net.
Revised: We also argue that the proposed method can generalize to the new scenes better than other monocular models because ours model learns the stronger features from the teacher net.

Original: These results and analysis will be included in the revised version.
Revised: These results and analysis will be included in the revised version.

Original: A. C OMPARISON WITH D IRECT D ENSE D EPTH S UPERVISION .
Revised: A. C OMPARISON WITH D IRECT D ENSE D EPTH S UPERVISION .

Original: According to the ablation studies in the main paper
Revised: According to the ablation studies in the main paper

Original: [Table]
Revised: [Table]

Original: we can find that depth cues are the key factor to affect the performance of the monocular 3D models.
Revised: we can find that depth cues are the key factor to affect the performance of the monocular 3D models.

Original: However, dense depth supervision in the student model without KD may also introduce depth cues to the monocular 3D detectors.
Revised: However, dense depth supervision in the student model without KD may also introduce depth cues to the monocular 3D detectors.

Original: Here we conduct the control experiment by adding a new depth estimation branch
Revised: Here we conduct the control experiment by adding a new depth estimation branch

Original: [Table]
Revised: [Table]

Original: which is supervised by the dense LiDAR maps.
Revised: which is supervised by the dense LiDAR maps.

Original: Note that, this model is trained without KD.
Revised: Note that, this model is trained without KD.

Original: Table 14 compares the performances of the baseline model, the new control experiment
Revised: Table 14 compares the performances of the baseline model, the new control experiment

Original: [Table]
Revised: [Table]

Original: and the proposed method.
Revised: and the proposed method.

Original: From these results, we can get the following conclusions: (i) additional depth supervision can introduce the spatial cues to the models, thereby improving the overall performance; (ii) the proposed KDbased method significantly performs better than the baseline model and the new control experiment, which demonstrates the effectiveness of our method.
Revised: From these results, we can get the following conclusions: (i) additional depth supervision can introduce the spatial cues to the models, thereby improving the overall performance; (ii) the proposed KDbased method significantly performs better than the baseline model and the new control experiment, which demonstrates the effectiveness of our method.

Original: [Table]
Revised: [Table]

Original: A. M ORE Q UALITATIVE R ESULTS
Revised: A. M ORE Q UALITATIVE R ESULTS

Original: In Figure 7, we show more qualitative results on the KITTI dataset.
Revised: In Figure 7, we show more qualitative results on the KITTI dataset.

Original: We use orange box, green, and purple boxes for cars, pedestrians, and cyclists, respectively.
Revised: We use orange box, green, and purple boxes for cars, pedestrians, and cyclists, respectively.

Original: In Figure 8, we show comparison of detection results in the 3D space.
Revised: In Figure 8, we show comparison of detection results in the 3D space.

Original: It can be found that our method can significantly improve the accuracy of depth estimation compared with the baseline.
Revised: It can be found that our method can significantly improve the accuracy of depth estimation compared with the baseline.

Revised: [Figure]