Paper: Hyperbolic Active Learning for Semantic Segmentation under Domain Shift (ICLR-2024)

Reviewer comment (Experiments):
The correlation between hyperbolic radius and class complexity is only supported with some experimental evidence on GTAV->Cityscapes dataset. Actually, as the two aspects of class complexity considered in the paper, i.e., class difficulty and scarcity, are correlated by itself, the hyperbolic radius may be mostly affected by label scarcity. The coefficient factor (-0.605 vs. -0.899) also indicates that the hyperbolic radius is more correlated with label scarcity. More convincing support (i.e., theoretical analysis, more experimental evidence) for the correlation between hyperbolic radius and class complexity is needed.

Author response:
Dear Reviewer 3ouZ,
Your commitment to reviewing our paper is greatly appreciated. Kindly find our responses to your questions below:

- **(Weakness 1)** This question plays a crucial role in understanding the novel interpretation of the hyperbolic radius and the proposed acquisition strategy, so we have addressed it in our **message to all reviewers**. Please see the answer on **Correlation Analysis**.
Additionally, following the reviewer's advice, we have computed the correlation between the class accuracy and the percentage of total pixels, which yields 0.539. This correlation is moderate, but lower than that between the hyperbolic radius Vs.\ the class accuracy and total number of pixels, $-0.605$ and $-0.899$ respectively. In our answer on **Correlation Analysis**, as further support to the use of the hyperbolic radius as a data acquisition strategy, please also see the larger correlations in the case of Cityscapes $\rightarrow$ ACDC, -0.759 and -0.868 respectively.

Reviewer comment (Writing):
The writing needs to be improved, some sentences are broken, e.g., "So more difficult classes such as pole which have lower accuracy, also have the larger Riemannian variance, so the largest effective volume available." (available is not a verb), "Atigh et al. (2022) has been the first to demonstrate a performance of hyperbolic SS on par with Euclidean." (Euclidean is not a noun)

Author response:
Dear Reviewer 3ouZ,
Your commitment to reviewing our paper is greatly appreciated. Kindly find our responses to your questions below:

- **(Weakness 2)** Thank you for your comment. As highlighted in the **message to all reviewers**, we value your feedback and have diligently attended to the noted writing issues, conducting additional proofreading to enhance the overall quality of the paper. Check out the **Paper's writing and detailing** for the answer.

Reviewer comment (Reproducibility):
The paper is not that easy to follow as there are missing details like how to get the embeddings of the pixels? Directly in the pixel space or get the feature first? How to plot the Figure 2? How to get the accuracy in Figure 2?

Author response:
We are sorry for the confusion caused by insufficient pipeline description and we appreciate the opportunity to clarify the details. Following [1,2], HALO comprises a Euclidean Segmenter (DeepLab-v2 or DeepLab-v3plus), a hyperbolic projection layer (expmap), and a hyperbolic multinomial logistic regression (HyperMLR) layer. The Euclidean Segmenter produces a _d_-dimensional embedding (d=64 for HALO) for each pixel of the image, which is projected onto the Poincaré ball via expmap, thus yielding the hyperbolic embedding of each pixel. The parameters to estimate the hyperbolic embedding for each pixel are end-to-end learned, based a pixel classification loss (cf. the HyperMLR layer.) To answer your question: embeddings are end-to-end learned in a hyperbolic manifold for each pixel. We have updated these details in the HALO pipeline in Sec. 5.1.
Fig. 2 is generated by calculating the _Class Average Radius_, which involves computing the mean hyperbolic radius for all pixel embeddings within each class in the target dataset. Additionally, the _Class Accuracy_ in Fig. 2 (left) represents the average classification accuracy for each class in the target dataset. Finally, the _\% of Total Pixels_ in Fig. 2 (right) corresponds to the percentage of acquired pixel annotations at the end of the active learning process for each class.

Reviewer comment (Evaluation):
Setting a new state-of-the-art across "all ADA benchmarks for SS" is overclaimed.

Author response:
To the best of our knowledge, RIPU [6] is the state-of-the-art on ADA for SS on the two benchmarks of GTAV $\rightarrow$ Cityscapes and SYNTHIA $\rightarrow$ Cityscapes. The two benchmarks were originally adopted by MADA [7] (Ning et al. ICCV'21), which proposes the task of ADA for SS for the first time. Our claim is supported by surpassing RIPU on both benchmarks and on a third one, which we additionally introduce, Cityscape $\rightarrow$ ACDC.

Reviewer comment (Evaluation):
The conclusion seems problematic: We conclude that the hyperbolic radius indicates the difficulty in recognizing a class, as a consequence of the class complexity and its label scarcity. Like in the previous papers, the hard pixels are more relative to their locations, like pixels on the boundary, even though there are lots of pixels in the dataset for such a category.

Author response:
Your inquiry is much appreciated as it regards the specific reason why HALO achieves state-of-the-art performance, in our view. Specifically, HALO's strategy involves the selection of boundary pixels for "easier" classes such as car or road, while also incorporating internal pixels for more complex classes such as person, rider, and bicycle. HALO showcases the importance of boundary pixels while acknowledging the significance of inner parts of objects in comprehending the scene. Please refer to Section A.6 in the appendix and Fig. 7 for more insights. To validate this, we conduct an oracle study involving the exclusive selection of boundary pixels during active learning stages. However, the resulting performance of 69.8\% mIoU was inferior to both HALO and RIPU. Please take a look at the "**Baseline selection comparison**" response in the **message to all reviewers** for more details about HALO's pixel selection.

Reviewer comment (Evaluation):
The paper mentioned that 'the pixels at the class boundaries are not necessarily the most informative and annotating only those degrade performance, as we confirm with an oracular study.' Then what should be the most informative ones? I ask because seems that the 'acquisition map' also indicates boundaries are the most informative ones?

Author response:
Your inquiry is much appreciated as it regards the specific reason why HALO achieves state-of-the-art performance, in our view. Specifically, HALO's strategy involves the selection of boundary pixels for "easier" classes such as car or road, while also incorporating internal pixels for more complex classes such as person, rider, and bicycle. HALO showcases the importance of boundary pixels while acknowledging the significance of inner parts of objects in comprehending the scene. Please refer to Section A.6 in the appendix and Fig. 7 for more insights. To validate this, we conduct an oracle study involving the exclusive selection of boundary pixels during active learning stages. However, the resulting performance of 69.8\% mIoU was inferior to both HALO and RIPU. Please take a look at the "**Baseline selection comparison**" response in the **message to all reviewers** for more details about HALO's pixel selection.

Reviewer comment (Evaluation):
How can you ensure that the hyperbolic NN is measuring the scarcity of labels for certain class prototypical appearances and the intrinsic complexity of classes? As you know the uncertainty will also lead to such a state.

Author response:
Thank you for this question, as it allows us to remark the interesting insight of the proposed analysis. Usually, in hyperbolic space, when the (1 - radius) is interpreted as the uncertainty, a low radius indicates high uncertainty. However, in our case, classes with lower radii, such as road, sky, building, are actually easier to classify by the model (higher accuracy and mIoU), leading to a mismatch with the canonical interpretation of hyperbolic uncertainty. Effectively, our work had originated by the wish to leverage the relation between the hyperbolic radius and uncertainty, but it transformed into a novel geometric interpretation of the hyperbolic radius when we realized a novel interpretation was needed. The nuanced interpretation of the hyperbolic radius is extensively discussed in Sec. 4, where we also provide a comparison with existing viewpoints (see Sec. 4.2). While a theoretical proof remains still an open research question, our empirical evidence currently supports this novel interpretation.

Reviewer comment (Experiments):
How to make sure the manually defined class hierarchies are reliable? Why not learn from the data?

Author response:
Thank you for your question. Our approach is not dependent on manually defined class hierarchies. In fact, this study originates from retraining the architecture of [1] and finding that the interpretation of the hyperbolic radius changes when not enforcing hierarchies.

Reviewer comment (Evaluation):
It seems this is not true from Figure 2 left: classes with larger hyperbolic radii have lower performance and are likely more difficult to recognize, and more complex. (BTW, there is no Fig (a) and (b), only left and right)

Author response:
Thank you for this question, as it allows us to remark the interesting insight of the proposed analysis. Usually, in hyperbolic space, when the (1 - radius) is interpreted as the uncertainty, a low radius indicates high uncertainty. However, in our case, classes with lower radii, such as road, sky, building, are actually easier to classify by the model (higher accuracy and mIoU), leading to a mismatch with the canonical interpretation of hyperbolic uncertainty. Effectively, our work had originated by the wish to leverage the relation between the hyperbolic radius and uncertainty, but it transformed into a novel geometric interpretation of the hyperbolic radius when we realized a novel interpretation was needed. The nuanced interpretation of the hyperbolic radius is extensively discussed in Sec. 4, where we also provide a comparison with existing viewpoints (see Sec. 4.2). While a theoretical proof remains still an open research question, our empirical evidence currently supports this novel interpretation.

Reviewer comment (Experiments):
The authors are suggested to have a discussion regarding the number of the parameters of the proposed method with the leveraged baselines.

Author response:
3. Your query holds significant importance and offers valuable insight into comparing parameters between Euclidean and Hyperbolic Neural Networks. To delve deeper into this comparison, we conduct an assessment of the parameter count between our method HALO and the prior state-of-the-art, RIPU [Xie et al.]. Both employ the DeepLab-v3+ architecture but with some distinctions:
    1. RIPU operates with a pixel embedding dimension of 512, resulting in a parameter count of 60.1M.
    2. In contrast, HALO operates with a reduced pixel embedding dimension of 64, which the adoption of a hyperbolic learning enables. Moreover, the HyperMLR requires fewer parameters than the Euclidean Linear layer used for classification due to the reduced embedding dimension. This results in a slightly lower total parameter count than RIPU's (10k fewer params). Additionally, HALO introduces the HFR module, consisting of two linear layers separated by a BatchNorm layer and a ReLU. Thanks to the lower embedding dimensions, the input and output sizes of the HFR module are only 64-dimensional, adding less than 10k additional parameters. This roughly matches the number of parameters removed from the segmenter. These modifications result in the parameter count being nearly identical between the two methods (60.1M), aligning with other studies leveraging the DeepLab-v3+ architecture.
Here's a table for a better visual comparison between the models:
   
    | Method | Segmenter | Dim. | HFR (params) | Total Params |
    |-------|-------|-------|-------|-------|
    | RIPU | DeepLab-v3+ | 512 | Not used | 60.1M |
    | HALO | Hyper-DeepLab-v3+ | 64 | 10k | 60.1M |

Reviewer comment (Presentation):
In Figure 8, the authors only provided the predictions of the proposed method, the predictions from the baselines are also interesting to deliver more comparison with the proposed method.

Author response:
4. We appreciate your comment, as it prompted us to enhance the comprehensiveness of our paper. In Appendix A.8, we have included additional comparisons with RIPU, that comprehend a qualitative assessment of the selection process and present quantitative results from the oracle experiment to elucidate the limitations of RIPU's boundary selection. Please refer to the "**Baseline Selection Comparison**" section in the **message to all reviewers** for the complete answer.

Reviewer comment (Experiments):
In Section 5.3, the authors claim that "However, these approaches often yield suboptimal or comparable performances when compared to the Euclidean counterpart.". Will it be possible to provide quantitative comparison between these approaches ((Guo et al., 2022; Franco et al., 2023; van Spengler et al., 2023)) and the HFR?

Author response:
5. Thank you for raising this point, as it prompted us to conduct a more comprehensive evaluation of these approaches. In response, we test Guo et al., 2022's Feature Clipping method in our framework for comparison with our HFR. As shown in the table below, while Feature Clipping works and produces better results than the baseline RIPU, it still falls short of our HFR method (-1.2\%). Guo et al. utilize Feature Clipping to prevent vanishing gradients during backpropagation. Despite its simplicity, this technique restricts the model's representational capacity by clipping features, resulting in inferior performance compared to HFR. In the allotted rebuttal time, we have not tested the curriculum learning of [Franco et al., 2023] and the initialization approach of [van Spengler et al., 2023], because their adaptation to the ADA task is not straightforward.
The curriculum learning in Franco et al. is specifically tailored for metric learning scenarios involving a hyperbolic loss, enabling training in hyperbolic space by utilizing cosine distance for improved initialization, gradually transitioning to the Poincaré loss. Our method does not involve comparing embeddings and leveraging the Poincaré loss. Similarly, the initialization approach in van Spengler et al. is designed explicitly for fully hyperbolic ResNets, particularly hyperbolic convolutions. As we do not employ hyperbolic convolutional layers, their initialization approach is not immediately suitable for our model.

    | Method | mIoU |
    |--------|-----------|
    | **Hyperbolic Feature Reweighting (ours)** | **74.5** |
    | Feature Clipping (Guo et al., 2022) | 73.3 |
    | Initialization (van Spengler et al., 2023) | not compatible |
    | Curriculum Learning (Franco et al., 2023) | not compatible |

Reviewer comment (Evaluation):
The method uses softmax score to serve as uncertainty. However, in model calibration field, pure softmax can barely estimate a good uncertainty score of the model without any open-set techniques, e.g., Monte-Carlo Dropout, OpenMax, Deep evidential learning,..., will the unsatisfied uncertainty prediction from the softmax score limit the performance of the proposed method? More discussion is expected toward this concern.

Author response:
6. We genuinely appreciate your question and the opportunity it provides to clarify certain aspects of our proposed method. In our paper, we utilize the entropy of the softmax probability array as a measure of uncertainty (ref. Eq. 7), drawing from the existing literature on ADA [Gal et al., 2017; Wang & Shang, 2014; Wang et al., 2016; Sinha et al., 2019; Xie et al., 2022b; Prabhu et al., 2021; Xie et al., 2022a], which currently yields state-of-the-art results. We acknowledge that relying solely on the pure softmax value may not accurately estimate model uncertainty, as you point out. However, please consider that our contribution regards the introduction of the hyperbolic radius to complement uncertainty as the data acquisition strategy. Maintaining the entropic softmax formulation of earlier literature and esp. of RIPU enables a fair comparison. We agree that uncertainty deserves more consideration in future research.