Paper: Dual-level Adaptive Self-Labeling for Novel Class Discovery in Point Cloud Segmentation (ICLR-2024)

Reviewer comment (Writing):
The motivation is not straightforward. More specifically, the relation between self-labeling and dual-level representation is not clear. It seems that this study addresses two individual problems of the point-wise clustering method. The background of the point-wise clustering should be more informative to improve the readability.

Author response:
## Motivation is not straightforward.

In the second paragraph of the introduction, we have highlighted the shortcomings of point-wise clustering based on equal-size constraints. To reiterate, these include 1) overlooking the inherently imbalanced data distribution and 2) neglecting the rich spatial context information of objects. Those two weaknesses are closely related to novel class discovery for the point cloud segmentation problem and are both important issues. Therefore, we believe our motivation is clear, and we respectfully disagree that solving a problem from two important perspectives is a weakness.

The point-wise clustering means we segment novel classes in the point cloud relying on each individual point, ignoring structured information, such as superpoint or region[3,4]. The existing point-wise clustering methods [3,4] usually employ equal-size constraints on cluster size to avoid degenerate solutions, where all classes are assigned to a single cluster.

Reviewer comment (Writing):
It is unclear what the "degenerate solution" means for the point-cloud segmentation. It is also unclear why a semi-relaxed optimal transport problem definition is needed from the introduction. The role of data-dependent annealing in the self-labeling process and dual-level representation is also not clear in the introduction.

Author response:
## degenerate solution and semi-relaxed optimal transport problem

In unsupervised clustering, the degenerate solution means we learn a single cluster [1,2] for all classes. Specifically in point cloud segmentation, it means we segment all unlabeled points to a single class. To avoid the degenerate solution, they [1, 3] usually introduce a uniform distribution constraint and formulate the pseudo-label generation problem as an optimal transport problem (Eq 3 in manuscript). **However, such a uniform and equality constraint contradicts with point cloud segmentation problem, where the class distribution is inherently imbalanced, thus generating a noisy pseudo label (Figure 2 in manuscript).** To mitigate this issue, we propose a semi-relaxed optimal transport problem, which relaxes the equality constraint into a KL constraint (Eq 4) and enables us to generate high-quality pseudo-labels (Figure 2 in manuscript). Note that our manuscript has clearly motivated the semi-relaxed optimal transport in the second paragraph, and we have updated the illustration of degenerate solutions for better understanding.

The data-dependent annealing strategy is specifically designed for the self-labeling process and orthogonal with dual-level representation. It adaptively adjusts the regularization strength of KL constraint in semi-relaxed optimal transport problems, enhancing the flexibility of our formulation (see the last sentence of Paragraph 4 in the Introduction).

Reviewer comment (Presentation):
The presentation of Figure 1 is confusing and contains errors. For example, it is unclear what the different colors represent, which component represents the novel pseudo-label generation process, and where the dual-level representation is shown. The arrows among p and y on the right-most part of the figure are difficult to understand, and the "Encoder f_theta" has been occluded.

Author response:
## Figure 1

We appreciate your suggestion and apologize for any misunderstanding. Please refer to the General Response Part 1 for clarification.

Reviewer comment (Presentation):
The paper's content lacks a smooth organization and its order creates confusion regarding the key method modules. Figure 1 is incomplete, with unexplained symbols.

Author response:
We appreciate your suggestion and have carefully considered your suggestions. However, we believe that our organizational structure is reasonable, as we employ a standard format. We present an overview of our method in Section 3.1, followed by a description of our network architecture, encompassing dual-level representation and a prototype-based classifier. Subsequently, we outline the overall self-labeling framework and provide details on imbalanced pseudo-label generation and the adaptive regularization strategy. To improve clarity, we have reorganized Section 3.1 to ensure coherence. It's notable that other reviewers (cd9p, 74h9) have also praised that "it is well-written and easy to follow".  We would appreciate it if you could specify which part lacks a smooth organization. 

We appreciate your advice on Figure 1, and please refer to the General Response Part 1 for details.

Reviewer comment (Evaluation):
In the ablation results, the application of region-level learning results in a decrease in the IoU value for 'Building.' Why does this phenomenon occur, and is this method more beneficial for smaller objects?

Author response:
**Explanation of the decrease of "Building"**

We appreciate your feedback. As depicted in Table 4, the inclusion of region-level features results in only a relatively small decrease in the "Building" class (53.1% -> 51.5%), while the average mIoU increases by 4.2%. We note that the performance of "Building" is improved on the training data after incorporating region-level learning, as shown in Figure 3. Therefore, we speculate that the marginal decrease in the building class may be due to the model overfitting to the training set.

Distinct from other classes, such as plants, cars, and grounds, "Building" is positioned farther away from the sensor, leading to a sparse spatial distribution and relatively small regions generated from DBSCAN. Therefore, we suspect that this phenomenon may arise from the prototypes capturing the bias of the relatively small region associated with the "Building" in the training data, which is inconsistent with the test data. This issue can be addressed by introducing additional regularization. For instance, as demonstrated in the table in General Response 2, configuring DBSCAN to generate relatively larger regions (e.g., with epsilon=0.7) significantly increases the results for the building class.

**Is this method more beneficial for smaller objects?**

Our method is more beneficial to dense objects instead of smaller objects. Specifically, we utilize DBSCAN, which is a density-based clustering algorithm, to generate regions for point clouds with relatively high density. Therefore, our method is beneficial for objects relative to density, rather than size.

Reviewer comment (Theory):
The utility of the novel class discovery setting, where the number of novel classes is predetermined, is a valid point for consideration. In an open-world scenario, the number of novel classes is often dynamic and not known in advance.

Author response:
We agree that the utility of the novel class discovery setting is a valid point for consideration. In our manuscript, we divide novel class discovery in point clouds into two sub-problems. The first is to estimate the number of novel classes. The second problem is to discover novel classes after determining the number of novel classes, which is challenging due to imbalanced class distribution. Additionally, to ensure a fair comparison with other methods [2], we use the number of novel classes as a prior and focus on the second problem.

And it is more realistic to consider unknown $|C_u|$. To estimate $|C_u|$, we extend the classic estimation method [1] in NCD to point cloud semantic segmentation. Specifically, we set the candidate range of the total number of categories (seen classes number $\textless |C_{all}|\textless$ max classes), and apply Kmeans to cluster the labeled and unlabeled point clouds in the training data, using the representation extracted from a known-class pre-trained model. Then, we evaluate the clustering performance of known classes under different $|C_{all}|$, and select $|C_{all}|$  with the highest clustering performance as the estimated $|C_{all}|$. In practice, for computational simplicity, we randomly sample 800,000 points from all scenes to cluster. We conduct experiments on splits 0 of SemanticPOSS, setting max classes to 50, which is a large class number in point cloud semantic segmentation. Consequently, the estimated $|C_u|$ is 3, which is close to the ground truth value (GT is 4).

In addition, we conduct experiments in SemanticPOSS split 0 to compare the results of our method and previous sota[2] when the estimated value deviates from the true value. As shown in the table below, our method can still significantly outperform NOPS with estimated $|C_u|$.

|      | split 0  |      |        |        |           |
| ---- | -------- | ---- | ------ | ------ | --------- |
|      | Building | Car  | Ground | Plants | mIoU      |
| NOPS | 25.54    | 0.00 | 68.15  | 34.12  | 31.95     |
| Ours | 64.05    | 0.00 | 82.22  | 67.63  | **53.47** |

In conclusion, our sub-problem division is reasonable, and our method shows significant improvement compared to the previous sota, even when the estimated novel class deviates slightly from the real value. 

In addition, we find that under the estimated $|C_u|$, our method achieves better results than the ground truth $|C_u|=4$. As shown in Figure 2 in the paper, when $C_u=4$, many plants and buildings are misclassified as the car class (a minor class), resulting in poorer learning for buildings and plants. In contrast, at $C_u=3$, the model ignores the car class, which is relatively small compared to plants and buildings, allowing for better learning of buildings and plants, thereby achieving superior results.

[1] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Generalized category discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7492–7501, 2022.

[2] Luigi Riz, Cristiano Saltori, Elisa Ricci, and Fabio Poiesi. Novel class discovery for 3d point cloud semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9393–9402, 2023.

Reviewer comment (Experiments):
Discussing the applicability of the proposed framework in indoor scenarios, where there are typically more novel classes to discover, would enhance this paper's practical relevance.

Author response:
Our framework is general and can be applied to indoor scenarios with little modification. The key difference between indoor scenarios and outdoor scenarios is that Indoor scenarios 1) have more classes; 2) are more denser; 3) include additional RGB information.

As for more novel classes, our results in Table 2 show we can achieve satisfactory results compared to NOPS on harder splits, where half (6 or 7) classes are novel.  As for more dense points, we have to adjust the DBSCAN to generate appropriate regions. For the additional RGB information, we can utilize it to enhance the quality of regions and adjust the backbone to process the input RGB information.

In conclusion, our method can be relatively easily extended to indoor scenarios.

Reviewer comment (Experiments):
Conducting an ablation study on a split of the SemanticKITTI dataset, which is known to be more challenging than SemanticPOSS, would strengthen this paper's findings.

Author response:
We appreciate your suggestion and conduct additional ablation on split 0 of SemanticKITTI. The results are shown below:

|      |      |        |          |      | split 0  |         |            |      |
| :--: | :--: | :----: | :------: | :--: | :------: | :-----: | :--------: | :--: |
| ISL  |  AR  | Region | Building | Road | Sidewalk | Terrain | Vegetation | Avg  |
|      |      |        |   46.7   | 43.2 |   17.9   |  21.7   |    23.9    | 30.7 |
|  ✓   |      |        |   57.4   | 32.1 |   25.2   |  18.9   |    37.2    | 34.1 |
|  ✓   |  ✓   |        |   70.8   | 34.7 |   23.2   |  16.8   |    57.9    | 40.7 |
|  ✓   |  ✓   |   ✓    |   74.6   | 41.4 |   22.5   |  23.6   |    66.4    | 45.7 |

Similar to SemanticPOSS, each component enhances performance. Specifically, adaptive regularization and region-level learning individually contribute to a 6.6% and 5.0% improvement in mIoU for the model.

Reviewer comment (Presentation):
In Table 3, adding a column that indicates the split index, similar to Table 1, would improve clarity.

Author response:
We appreciate your advice. But the table is too wide to insert another column. To make it clear, we add an explanation of the split index to the caption.