Paper: MixSup: Mixed-grained Supervision for Label-efficient LiDAR-based 3D Object Detection (ICLR-2024)

Reviewer comment (Presentation):
The illustration of distinct properties of point clouds compared to images in the introduction section seems not very relevant to the motivation and the concept of the study (The paper is not about multi-modality fusion).

Author response:
We sincerely apologize for any potential confusion caused by the introduction. In the upcoming revised version, we will present it in a clearer and more precise manner. It's important to clarify that the images in the introduction are intended to offer readers a more vivid and intuitive depiction of the distinctive properties of point clouds.

Reviewer comment (Novelty):
The motivation "good detector needs massive semantic labels for difficult semantic learning but only a few accurate labels for geometry estimationhis" is simply another way of expressing a common phenomenon in object detection, which is the high recall often contribute more on high average precision (AP).

Author response:
We acknowledge that this phenomenon is reasonable in 2D object detection, and we truly thank the reviewer for pointing it out. But it's crucial to note that 3D detection is more sensitive to regression quality compared to its 2D counterpart, due to the following two reasons:

- In the 3-dimensional space, the additional spatial dimension makes achieving high 3D IoU more challenging than achieving high 2D IoU.
- 3D object detection requires the consideration of orientation angles, which introduces an additional freedom in the matching process between predictions and GTs.

Taking the widely used autonomous driving dataset Waymo as an example, Waymo sets the official IoU threshold as 0.7 for vehicle, and proposes the metric APH that incorporates heading as a weighted factor, which is a strict metric for evaluating a 3D object detector.

To vividly illustrate the rigorous evaluation of geometry attributes in 3D object detection, we conduct experiments with a pre-trained CenterPoint model. Specifically, we introduce different noises to the predicted boxes.

| Noise in location (m) | Noise in dimension (m) | Noise in heading (degree) | Vehicle L1 AP | Vehicle L1 APH | Vehicle L2 AP | Vehicle L2 APH |
|---|---|---|:---:|:---:|:---:|:---:|
| - | - | - | 73.43 | 72.88 | 65.42 | 64.92 |
| 0.05 | 0.05 | 5 | 61.39 | 59.62 | 54.13 | 52.57 |
| 0.1 | 0.1 | 7.5 | 23.82 | 22.84 | 20.66 | 19.81 |

The introduced noise (e.g., 0.05m) is minor because vehicles are usually more than 4 meters long.
Experimental results demonstrate that even minor geometry attribute deviations can lead to significant performance drops under the strict metric. This highlights the need for a good 3D detector with strong geometry estimation capabilities, as high recall alone may not guarantee good performance.

Reviewer comment (Experiments):
The author proposed to create cluster-level supervision as the parallelogram of three click. If one can generate massive pseudo-boxes from clicks. Why not just using a region-based network (e.g. the second stage PointNet in LiDAR-RCNN) to refine these boxes as precise box-level supervision?

Author response:
We greatly appreciate your insightful suggestion. It is an insightful suggestion, but there are some potential drawbacks:

- **Pretrained region-based detectors cannot be directly utilized.** There are domain gaps between the human-labeled cluster labels and the input regions of the region-based detector. For example, the input of LiDAR-RCNN comes from the output proposals of other detectors with being enlarged, which is quite different from our cluster labels. So it is infeasible to directly utilize an off-the-shelf pretrained region-based detector. Moreover, the diverse data distributions of different datasets make it more infeasible to use pretrained region-based detectors from other datasets. How to improve the transferring performance of region-based detectors remains open.

- **Transforming coarse labels to precise boxes for supervision introduces inherent errors, reducing the performance upper bound.**  This is because the predicted precise boxes cannot be as good as human annotations. In contrast, although our coarse cluster labels do not have accurate geometry information, the semantic label for each cluster remains precise. Furthermore, the regression learning in MixSup only relies on accurate box-level labels. Thus, although we adopt coarse labels, the supervision in our method does not have such inherent errors and potentially offers a higher performance upper bound.

Although there are these drawbacks, your suggestion is indeed very insightful. We will try to make it more feasible in the future, for example by improving the transferability of region-based detectors.

Reviewer comment (Evaluation):
From author's experiment in Table 4, previous methods like MV-JAR can achieve very close performance to author's with 10% anotations. Considering author use massive cluster-level semantic supervision, does this indicate the massive semantic label is not that important?

Author response:
Thank you for expressing your concerns, and we appreciate the opportunity to clarify this point. It's essential to note that in Table 4, MixSup adopts SECOND as the base detector rather than SST, which is a better detector than SECOND. Therefore, the marginal improvement of MixSup over MV-JAR in Table 4 is reasonable.

To further address your concerns, we also take SST as the base detector to compare with MV-JAR fairly. As shown in the following table, MixSup outperforms MV-JAR with the same base detector, showing that using extra coarse labels leads to improvement.

| Detector | Label-efficient method | Annotation | Mean | Vehicle | Pedestrian | cyclist |
|---|---|:---:|:---:|:---:|:---:|:---:|
| SST | MV-JAR | 10% annotation cost | 54.06 | 58.00 | 54.66 | 49.52 |
| SECOND | MixSup | 10% annotation cost | 54.23 | 55.02 | 49.61 | 58.06 |
| SST | MixSup | 10% annotation cost | **60.74** | **59.10** | **60.00** | **63.13** |

Reviewer comment (Experiments):
The author use cluster center to substitue the object center for semantic learning. I think this will introduce instability of training due to the cluster containing noise as it is generated from clicking/SAM model. I suggest author can perform another experiment by training the semantic head by leveraging voxel-to-point interpolation (eg. used in PVCNN, SASSD) and imposing point-level supervision.

Author response:
We sincerely appreciate your insightful suggestion. Your proposal is indeed quite reasonable.

In fact, our method has already employed such supervision in certain situations, such as PV-RCNN [R5]  (tested on KITTI validation split with moderate difficulty). Specifically, PV-RCNN generates features for key points through the voxel set abstraction module, which is very similar to voxel-to-point interpolation, as you pointed out, from neighboring voxels.

Moreover, we supplement a point-based method FSD [R2], which adopts a segmentation network in the first stage to obtain foreground points with **point-level supervision** and test it on Waymo validation split with L2 APH. 

Indeed, as you pointed out, MixSup incorporating point-level supervision has shown excellent performance. We sincerely appreciate your valuable suggestion!

| Detector | Car | Pedestrian | Cyclist |
|---|---|---|---|
| PV-RCNN (100% frames) | 83.61 | 57.90 | 70.47 |
| PV-RCNN (MixSup)* | 76.09 (91.01%) | 54.33 (93.83%) | 65.67 (93.19%) |

| Detector | Mean | Vehicle | Pedestrian | Cyclist |
|---|---|---|---|---|
| FSD (100% frames) | 71.27 | 70.09 | 69.79 | 73.93 |
| FSD (MixSup)* | 68.57 (96.21%) | 66.08 (94.28%) | 66.53 (95.33%) | 73.09 (98.86%) |

*: Using **coarse cluster-level labels and 10% box-level labels**. The percentage in parentheses indicates the performance ratio to the fully supervised counterpart

Reviewer comment (Theory):
Several design decisions appear to be largely heuristic in nature. A number of assertions lack rigorous justification or analytical backing.

Author response:
Thanks for your insightful suggestion regarding the need for rigorous justification and analytical backing. In response, we will enhance the manuscript to ensure a more comprehensive and transparent presentation of the underlying rationale. The revision will be reflected in both the responses to Questions and the upcoming edited version of the manuscript.

Reviewer comment (Reproducibility):
The paper omits certain pivotal implementation specifics. This omission hampers a clear assessment of the proposed method's true efficacy.

Author response:
Due to the page limit, we couldn't elaborate on all implementation details. However, we appreciate your suggestion, and we will carefully include detailed implementation of crucial steps. In addition, we commit to open-sourcing the code for reference in the future.

Reviewer comment (Experiments):
The pilot study's design and outcomes could gain depth with a more granular breakdown of the experimental setup and subsequent insights. Furthermore, could you elucidate the term “data size”? Is it suggestive of the detectors' adaptability to varying “object/instance sizes”?

Author response:
We sincerely apologize for the confusion caused by the term “data size”. We will replace it with “data amount” to convey that it refers to the amount of cropped regions used in training the SparseConv Detector, rather than the size of objects or instances. By changing the amount of training data, we demonstrate that geometry estimation does not necessitate a large amount of data.

Moreover, we supplement experiments to enhance the claim that “massive data is necessary for semantic learning”. More details can be found in [Response to Reviewer yY5M (Part 1) W2](https://openreview.net/forum?id=Q1vkAhdI6j&noteId=rAHhKGSTQF).

Reviewer comment (Reproducibility):
The mechanics of the “clicking annotation” remain somewhat elusive. How were the annotators guided in executing this task? Was there a specific strategy adopted for different instance types? Was this manual labeling extended to all point clouds during training? If not, how were the crucial clicking points discerned? This reviewer believes a more thorough exposition on this subject would be beneficial in the rebuttal phase.

Author response:
We will address your questions regarding the mechanics of the “clicking annotation” as follows.

- “Clicking annotation” for label-efficient 3D detection is first proposed by Meng et al. [R4], [R5]. They require annotators to click object centers in the BEV perspective, which are used to supervise the center prediction.
- In the process of labeling, there is no difference made for different instance types.
- For the last question about “Was manual labeling extended to all point clouds and how were the crucial clicking points discerned”. We are not sure we understand the reviewer’s question accurately. Our understanding is that the reviewer wants to know if the “click annotation” annotates all the objects in the training dataset. If we understand accurately, the answer is no. Using their implementation as an example, they randomly choose 500 frames from KITTI, where the centers of all instances are labeled in each frame. If our understanding is not accurate, we are open to any further discussions at anytime.

Below we attach the comparison with Meng et al. [R4], [R5] to showcase our superiority. The evaluation results on KITTI val set (Car) with 534 box-level labels along with weakly labeled annotations.

| Label-efficient method | Annotation | Easy | Moderate | Hard |
|:---:|:---:|:---:|:---:|:---:|
| WS3D [R4] | 534 boxes + weak labels | 84.04 | 75.10 | 73.29 |
| WS3D [R5] | 534 boxes + weak labels | 85.04 | 75.94 | 74.38 |
| MixSup | 534 boxes + weak labels | 86.37 | 76.20 | 72.36 |

Reviewer comment (Writing):
The acronym “SAR” makes its debut in Sec. 4.3 without any contextual introduction. The authors are suggested to revise this.

Author response:
Thank you for raising questions about contents that were not clearly elucidated. We sincerely apologize for any inconvenience caused by this. Since SAM's over-segmentation and nuScenes' imprecise point-pixel projection result in mediocre 3D instance masks, we leverage the spatial separability of objects in point clouds to refine the segmentation with Separability-Aware Refinement (SAR), which is demonstrated in the response to Q7. In SAR, the Connected Components Labeling (CCL) is a conventional operation using Euclidean distance for connectivity as outlined in https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csgraph.connected_components.html. Our code will be open-sourced for reference.

Reviewer comment (Reproducibility):
The narrative on connected components labeling (CCL) in Sec. 4.3 is quite succinct, leaving readers wanting more details on its practical implementation.

Author response:
Thank you for raising questions about contents that were not clearly elucidated. We sincerely apologize for any inconvenience caused by this. Since SAM's over-segmentation and nuScenes' imprecise point-pixel projection result in mediocre 3D instance masks, we leverage the spatial separability of objects in point clouds to refine the segmentation with Separability-Aware Refinement (SAR), which is demonstrated in the response to Q7. In SAR, the Connected Components Labeling (CCL) is a conventional operation using Euclidean distance for connectivity as outlined in https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csgraph.connected_components.html. Our code will be open-sourced for reference.

Reviewer comment (Reproducibility):
It remains unclear which pre-trained semantic segmentation model was harnessed to create the 2D semantic masks. The possible implications of such a model leading to inadvertent data leakage need to be addressed. This information is indispensable to ensure that the experimental comparisons are grounded in validity.

Author response:
The model we employed is the HTC [R6]  pre-trained on nuImages dataset. In automated coarse labeling, we solely utilized its semantic segmentation head to obtain semantics for each pixel. There is no need for concern regarding data leakage, as the author of nuScenes have explicitly stated that the overlap between nuImages and nuScenes is negligible in  https://github.com/nutonomy/nuscenes-devkit/issues/550#issuecomment-769571712

Reviewer comment (Experiments):
Camera-LiDAR calibrations might not always achieve perfection. In scenarios where calibration discrepancies exist, there's potential for the SAR-derived coarse instance masks to be tainted with errors. The authors are suggested to consider conducting auxiliary experiments to probe whether calibration inaccuracies compromise the efficacy of the proposed method.

Author response:
We sincerely appreciate your suggestion for experiments regarding the inaccurate calibration. Throughout our experimentation, we identified inherent inaccuracies in the calibration of nuScenes, leading to discrepancies in the pixel-point projection of foreground objects. Hereby, we introduce SAR module to mitigate the degradation in segmentation caused by projection error.

To further investigate the impact of projection discrepancies on automated coarse labeling, we introduced random noise to each camera's position. The performance for foreground objects on nuScenes val split panoptic segmentation is summarized in the table below. The results indicate that SAR-derived coarse instance masks exhibit a certain degree of resilience to calibration inaccuracies, attributed to the refinement introduced by the SAR module.

| Noise of Camera Position (cm) | SAR | PQ | SQ | RQ |
|---|:---:|:---:|:---:|:---:|
| 0 | x | 44.9 | 74.6 | 59.8 |
| -5~5 | x | 43.8 | 74.6 | 58.3 |
| -10~10 | x | 41.4 | 74.5 | 55.2 |
| 0 | √ | 63.7 | 82.6 | 76.9 |
| -5~5 | √ | 62.8 | 82.3 | 76.1 |
| -10~10 | √ | 60.6 | 81.4 | 74.3 |

Reviewer comment (Experiments):
This paper could be enriched with comparative analyses against some of the latest entrants in the domain of weakly and semi-supervised 3D object detectors.

Author response:
We highly value your suggestion and have enriched the comparisons with relevant works. To incorporate more entrants, we find a new work CoIn [R7] published in ICCV2023, with a 5% annotation cost on the KITTI validation split, evaluating Car-3D AP R40. It's worth noting that the 3.3% annotation cost for MixSup in the experiment is borrowed from other experiments. Despite having a lower annotation cost than CoIn, we achieve higher AP in both moderate and hard difficulties. We will include the comparison in the revised manuscript.

| Annotation cost | Method | Detector | Easy | Moderate | Hard |
|:---:|:---:|:---:|:---:|:---:|:---:|
| 5% | CoIn[R7] | CenterPoint | 81.64 | 67.48 | 58.32 |
| 3.3% | MixSup | CenterPoint | 80.02 | 69.12 | 65.62 |

- It's also worthwhile to note that MixSup is orthogonal and complementary to semi-supervised learning, such as a simple self-training as highlighted in Sec 5.5 Table 6 in the paper. We will delve into the combination of MixSup and other semi-supervised training schemes in future work.

Reviewer comment (Presentation):
A minor formatting suggestion: For improved readability, perhaps anchor tables such as Table 2, Table 4, Table 5, Table 7, and Table 10 to the top of their respective pages.

Author response:
Thank you for your generous suggestion. We will modify tables in the subsequent manuscript.

Reviewer comment (Presentation):
This paper could benefit from having a standalone paragraph discussing the limitations and potential negative social impact of this work.

Author response:
Regarding the limitations of MixSup, as a novel label-efficient paradigm, it is orthogonal to other label-efficient methods. Our work has not explored integration with semi-supervised methods, providing an avenue for potential performance enhancements. Moreover, our proposed automated coarse labeling, when combined with other exceptional 3D segmentation models, could yield higher-quality coarse labels. We believe MixSup holds significant promise in reducing annotation costs for the community, contributing to the conservation of both human and environmental resources. However, due to the cost savings in annotation come with a certain performance trade-off, practical deployment may raise risks of compromising driving safety.

We are more than willing to continue engaging in in-depth discussions. If you have any further questions, please feel free to comment at any time.

Reviewer comment (Novelty):
Discussing and comparing with [Meng et al. 2020; 2021]. As pointed out in the related works in the paper, there are a bunch of existing works trying to reduce the labeling cost in the 3D detection task, and one of the closest works to this paper is probably [Meng et al. 2020; 2021], where they explored a similar setup: training models with many weakly labeled and few fully labeled 3D data. As a reader, I would expect to see further discussion and comparison between the proposed work and this work (e.g. the difference in labeling cost, the performance difference, and why the proposed method is a better approach). I do read that in Sec 4.5 and the main results, the authors did compare the proposed method with some of the label-efficient frameworks, but it looks like the comparison with [Meng et al. 2020; 2021] is missed out.

Author response:
We apologize for unintentionally overlooking the comparison as our focus was primarily on Waymo and nuScenes. Here we supplement a discussion and experimental comparison with **WS3D** [Meng et al. 2020/2021] as follows.

- WS3D has a specifically designed detection pipeline and cannot be generalized to other detectors. In contrast, MixSup can be integrated with various detectors.
- The coarse cluster-level labels proposed in our method can be obtained through foundational models such as SAM, as we demonstrated in Automated Coarse Labeling.
- For the annotation cost, we are truly sorry that we cannot provide a rigorous quantitative comparison with center-click in WS3D because the cost largely depends on labeling tools and protocols, but we do not have the labeling tools in WS3D. However, we provide the analysis and discussion in the following **W3**.
- At the same annotation cost, our method has better performance. In the following Table, we report the performance of Car detection in KITTI val split since WS3D does not use Waymo and nuScenes dataset.

| Label-efficient method | Annotation | Easy | Moderate | Hard |
|:---:|:---:|:---:|:---:|:---:|
| WS3D [R1] | 534 boxes + weak labels | 84.04 | 75.10 | 73.29 |
| WS3D [R2] | 534 boxes + weak labels | 85.04 | 75.94 | 74.38 |
| MixSup | 534 boxes + weak labels | 86.37 | 76.20 | 72.36 |

Reviewer comment (Experiments):
The hypothesis is not fully backed up: in the last paragraph of Sec 3 (pilot study), the authors concluded: "This phenomenon suggests that LiDAR-based detectors indeed only need a very limited number of accurate labels for geometry estimation. Massive data is only necessary for semantic learning." If I understand correctly, the pilot experiments only provide evidence to some degree in the first half, but the latter half "Massive data is only necessary for semantic learning" seems not fully grounded. An evaluation of how well the model can perform with different amounts of semantic labels could better support the claim.

Author response:
We greatly appreciate your insightful feedback. We supplement the experiments to enhance this claim.

Specifically, we decrease the data amount (#. training frames) from 100% to 10% in Waymo and train a popular detector CenterPoint [R7]. Such a decrease in data amount has a negative impact on **both** geometry learning and semantic learning. To mitigate the impact on geometry learning and only focus on semantic learning, we aggressively relax the IoU thresholds in evaluation from 0.7 / 0.5 / 0.5 to 0.5 / 0.25 / 0.25 for vehicle/pedestrian/cyclist, respectively. 

Let’s first look at the following table, there are huge performance gaps between using 100% data and using 10% data. Obviously, here these gaps are caused by the degradation of **both** semantic learning and geometry estimation.

| Data amount | IoU thresholds | Vehicle L2 AP / APH | Pedestrian L2 AP / APH | Cyclist L2 AP / APH |
|---|:---:|:---:|:---:|:---:|
| 100% | 0.7 / 0.5 / 0.5 (Normal) | 65.42 / 64.92 | 66.49 / 60.53 | 69.28 / 68.12 |
| 10% | 0.7 / 0.5 / 0.5 (Normal) | 55.92 / 55.39 | 56.97 / 49.07 | 51.52 / 50.46 |
| Performance Gaps | - | 9.50 / 9.53 | 9.52 / 11.46 | 17.76 / 17.66 |

Then let’s look at the following table, where we **relax the IoU thresholds for evaluation** to mitigate the negative impact introduced by degradation of geometry estimation. The overall performance becomes much higher after relaxing IoU thresholds. However, there are still significant gaps between using 100% data and using 10% data. Especially, for Pedestrian and Cyclist, the gaps (between 10% 
 and 100%) **do not even get smaller** after we relax the IoU thresholds. Thus, the performance gaps between 100% data and 10% data **should be mainly caused by the degradation of semantic learning.** In other words, semantic learning is sensitive to the data amount.

| Data amount | IoU thresholds | Vehicle L2 AP / APH | Pedestrian L2 AP / APH | Cyclist L2 AP / APH |
|---|:---:|:---:|:---:|:---:|
| 100% | 0.5 / 0.25 / 0.25 (relaxed) | 87.15 / 86.20 | 82.99 / 74.62 | 74.01 / 72.70 |
| 10% | 0.5 / 0.25 / 0.25 (relaxed) | 81.99 / 80.67 | 73.61 / 62.14 | 56.50 / 55.20 |
| Performance Gaps | - | 5.16 / 5.53 | 9.38 / 12.48 | 17.51 / 17.50 |

From another point of view, in the main paper, we have decreased the data amount of well-classified patches from 100% to 10% to reveal the impact on geometry estimation, as shown in the following table. Compared with the aforementioned performance change caused by semantic learning degradation, **the performance change in the following table is negligible**. Thus, we draw the conclusion that “LiDAR-based detectors indeed only need a very limited number of accurate labels for geometry estimation. Massive data is only necessary for semantic learning”.

*The following table: Performances with varying data amounts on the well-classified dataset (Copy from main paper).*

| Data amount | IoU thresholds for three classes | Vehicle L2 AP / APH | Pedestrian L2 AP / APH | Cyclist L2 AP / APH |
|---|:---:|:---:|:---:|:---:|
| 100% | 0.7 / 0.5 / 0.5 | 64.19 / 63.74 | 65.23 / 58.02 | 67.04 / 65.99 |
| 10% | 0.7 / 0.5 / 0.5 | 63.37 / 62.89 | 64.78 / 57.96 | 66.26 / 65.14 |
| Performance Gaps | - | 0.82 / 0.85 | 0.45 / 0.06 | 0.78 / 0.85 |

The discussion above will be included in the edited version.

Reviewer comment (Experiments):
The study on the labeling costs is not fully grounded. In Sec 5.3, the authors claim "The average time cost of a coarse cluster label is only 14% of an accurate box label." In Sec 4.2, the authors also claim "In addition, it is also non-trivial for annotators to make an accurate center click." However, evidence is missing for these claims. Where does the "14% cost" come from? How much harder / more inaccurate is it for the annotator to click the center? Do you perform a user study with a reasonable number of annotators? How similar are the coarse labels from the annotators to the simulated coarse labels used in the experiments? I would suggest the authors include more details to back the claim.

Author response:
We highly value your questions and try to resolve your reasonable concerns from the following aspects.

- As described in Sec 5.3, we conducted a study by asking **three** experienced annotators to label 100 frames from different sequences of nuScenes. “The average time cost of a coarse cluster label is only 14% of an accurate box label” is based on the statistics gathered from the annotators.
- In the original paper, we simulate the coarse labels by randomly expanding GT boxes 0~10% in each dimension without evaluating the similarity with human coarse annotation. Thank you so much for pointing this out! To be more rigorous, here we first evaluate the similarity between human-annotated coarse labels (100-frame subset) and ground-truth labels, **which is 81.22% in terms of segmentation mIoU**. Then we add noise to ground-truth labels to simulate more rigorous coarse labels, controlling coarse labels possessing **80% to 82%** mIoU with ground-truth labels, just like the human-annotated coarse labels. In particular, we use the following two noises:
    - Randomly shifting center locations $-0.1\sim 0.1$ m and expanding 0%$\sim$50% in each dimension (Noise ①). This noise has 81.91% mIoU with the ground-truth label.
    - Randomly shifting center locations $-0.2\sim 0.2$ m, expanding 0%$\sim$ 20%in each dimension, and rotating $-15^\circ\sim15^\circ$ in heading (Noise ②). This noise has 80.60% mIoU with the ground-truth label.
    
    We then conduct experiments on nuScenes using such noisy coarse labels, results shown in the following table. Performance with noise ①, ② is similar to the performance in the original paper. Thus, our simulated coarse labels are reliable for the experiments in our paper.

| Detector | Noise | mAP | NDS |
|---|---|---|---|
| CenterPoint (100%) | - | 62.41 | 68.20 |
| CenterPoint (MixSup) | original paper | 60.73 | 66.46 |
| CenterPoint (MixSup) | Noise ① | 60.23 | 65.99 |
| CenterPoint (MixSup) | Noise ② | 60.21 | 66.28 |

- As for the center-click annotation, we unintentionally overlooked the human labeling study because it is not a core part of our method. We cannot launch and finish a labeling study during the limited rebuttal period without the labeling tools in WS3D. We are truly sorry for that. However, we demonstrate that center-click annotation has some limitations.
    - For the inevitable instances with very partial point clouds, accurately clicking the center is challenging.  Conversely, MixSup's cluster-level label only requires clicking around the visible part of an object with three points, ensuring that the generated parallelogram encompasses the partial point clouds.
    - Some methods need more than centers, such as the recent state-of-the-art FSD [R3]. It adopts a segmentation network in the first stage to obtain foreground points, **requiring point-level supervision**. Here we attach the performance on Waymo validation split with L2 APH using FSD as the base detector to show the effectiveness of MixSup, where the percentage in parentheses indicates the performance ratio to the fully supervised counterpart.

| Detector | Mean | Vehicle | Pedestrian | Cyclist |
|---|---|---|---|---|
| FSD (100% frames) | 71.27 | 70.09 | 69.79 | 73.93 |
| FSD (MixSup) | 68.57 (96.21%) | 66.08 (94.28%) | 66.53 (95.33%) | 73.09 (98.86%) |

Reviewer comment (Writing):
Presentation — The authors should be more specific about "semantic" and "geometry" from the beginning of the paper. If I understand correctly, by "semantic" the authors meant to coarsely identify object locations and types, and by "geometry" they meant to accurately regress the discovered object's location, dimensions, and heading. It is a bit confusing when reading the 3rd paragraph of the introduction.

Author response:
We sincerely appreciate your meticulous reading of our work. Your understanding of "semantic," "geometry," and "assignment" is highly accurate. We definitely believe good papers necessitate high-quality writing and are truly sorry for the unclear expressions. We highly value your suggestions and will make revisions in the upcoming manuscript.

Reviewer comment (Reproducibility):
In Table 4, it looks the PointContrast and the ProposalContrast have negative performance gains compared with training with 10% frames. But from the GCC-3D paper [Liang et al. 2021], the PointContrast usually can improve over the from-scratch baseline. The results reported here look inconsistent with previous literature. I am wondering how are these baselines trained?

Author response:
We apologize for any confusion caused. In our paper, we took a recent method MV-JAR [R4] (CVPR 23) as a compared method. The results of PointContrast and ProposalContrast in our paper are reported by MV-JAR.  For a direct comparison, we compare MixSup with the results in the **original ProposalConstrast paper** as follows. Using **1% boxes and coarse labels**, MixSup is slightly better than them with **20% frame annotations**. Due to the different settings, we cannot perfectly align the annotation cost with them, but the following results suggest MixSup could achieve promising results.
| Detector | annotation cost |Mean L2 APH (Waymo val) | 
|---|---|---|
| PVRCNN + GCC 3D [R6] | 20% frames |58.18 |
| PVRCNN + ProposalContrast |  20% frames | 59.28 |
| PVRCNN + MixSup (ours) | 1% boxes + coarse labels | 59.74 |