Paper: Hyperbolic Visual-Semantic Alignment for Structural Visual Recognition (ICLR-2024)

Reviewer comment (Writing):
The definition of q_θ in Eq 7 and 8 are missing.

Author response:
Sorry about this. Here we follow the conventional notation in VAE [ref1]:  $q_{\phi}(z|x)$ represents the variational posterior distribution that  is an
approximation to the intractable true posterior $p_\theta(z|x)$. It typically refers to a probabilistic encoder, which produces a distribution (in our paper is a Gaussian) of latent representation $z$ given a feature $x$. Relevant parts have been carefully revised to reflect this. 

[ref1] Auto-Encoding Variational Bayes. ICLR 2014

Reviewer comment (Writing):
Typo: Section 3.2, Task Setting: “An undirected edges (vi, vj) ∈ E indicates that the class i is a superclass of label i” It should be “label j” instead of “label i”.

Author response:
Thank you so much for your careful review! They are fixed and a thorough proofreading is made.

“An undirected edge $(v_i, v_j) \in E$ indicates that the class $i$ is a superclass of label $i$” $\rightarrow$ “An undirected edge $(v_i, v_j) \in E$ indicates that the class $j$ is a superclass of _label $i$_”

“we seek to align visual embedding with visual embedding to yield a shared representation space.” $\rightarrow$ “we seek to align visual embedding with _label_ embedding to yield a shared representation space.”

Reviewer comment (Writing):
Typo: Section 3.2.1 Hyperbolic Gaussian Mixture VAE: “we seek to align visual embedding with visual embedding to yield a shared representation space.” Is it “align visual embedding with LABEL embedding”?

Author response:
Thank you so much for your careful review! They are fixed and a thorough proofreading is made.

“An undirected edge $(v_i, v_j) \in E$ indicates that the class $i$ is a superclass of label $i$” $\rightarrow$ “An undirected edge $(v_i, v_j) \in E$ indicates that the class $j$ is a superclass of _label $i$_”

“we seek to align visual embedding with visual embedding to yield a shared representation space.” $\rightarrow$ “we seek to align visual embedding with _label_ embedding to yield a shared representation space.”

Reviewer comment (Reproducibility):
Are the authors planning to release the source code for this work?

Author response:
Definitely! We will release the code and model weights to the public.

Reviewer comment (Reproducibility):
The logic flow is broken, one can hardly see the connection between 3.1, 3.2. It will be very difficult (even for a hyperbolic learning researcher) to reproduce the method by reading the ``Method'' section.

Author response:
> The logic flow is broken, one can hardly see the connection between 3.1, 3.2. It will be very difficult (even for a hyperbolic learning researcher) to reproduce the method by reading the ``Method'' section.

Our apologies. We agree that the writing of mentioned parts can be improved upon rereading.  We have worked to improve our writing. 

- _connection between 3.1 and 3.2_: Sec. 3.1 is only to present the basic formulations for hyperbolic learning. It helps readers to understand basic concepts, and the formulations are also reused in Sec. 3.2 (_e.g.,_ Eq. 5, Eq. 6, Eq. 8). There are no inherent connections between them. To avoid confusion, we take an action to move Sec. 3.1 into a new Sec. 3 and the methodology part into a new Sec. 4. Thanks.

- writing improvements in the methodology part: 

  (1) in Sec. 4.1, we have added missed details, including an explicit definition of covariance, i.e., $\Sigma_l=diag(\sigma_l^2)$, as well as the definition of the reconstruction loss. 

  (2) in Sec. 4.2, we re-written most parts, and make the section more logical. Concretely, we compile hierarchy-aware semantic learning into two major parts, one is hyperbolic metric learning, and the other is how to construct samples for metric learning. 

We believe that these improvements make the methodology more logical and easier to understand. Beyond this, we promise that we will release the code to help  reproducibility. Thanks.

Reviewer comment (Theory):
Many important descriptions are missing or even incorrect, e.g., " ... the mean of the wrapped normal mixture distribution is calculated by Möbius addition." is not logically correct; " ... resulting in $y_{l,i} = \mathbf{\mu}_l + \mathbf{\Sigma}_l$ ...", you cannot add a mean vector to a covariance matrix; the reconstruction loss is neither explained nor linked to a reference.

Author response:
> a) " ... the mean of the wrapped normal mixture distribution is calculated by Möbius addition. " is not logically correct.

We agree with the reviewer: this statement is not reasonable and is redundant; we removed it in the revised version. 

> b) " ... resulting in $y_{l,i} = \mu_l + \epsilon\Sigma_l \in \mathcal{Y}$...", you cannot add a mean vector to a covariance matrix.

Thank you for your careful review. In the original writing, we accidentally omitted the explicit definition of $\Sigma_l$, which is a diagonal matrix: $\Sigma_l=diag(\sigma_l^2)$. Here $\mu_l\in\mathbb{R}^d$ and $\sigma_l\in\mathbb{R}^d$ are derived from the semantic encoder. With this respect, the formula of the reparameterization trick should be more strictly revised to $y_{l,i} = {\mu}_l + \epsilon\sigma_l$. The relevant parts have been carefully revised. 

> c) The reconstruction loss is neither explained nor linked to a reference

The reconstruction loss is a standard negative log-likelihood. We provide its formal definition in Eq. 8 in the updated article. Thanks.

Reviewer comment (Evaluation):
The tables basically show "we have better numbers", but the analysis lacks in-depth understanding of why each part works.

Author response:
> The tables basically show "we have better numbers", but the analysis lacks in-depth understanding of why each part works.

Thanks for pointing this out. Following your comments, we have carefully improved the experimental parts to offer more in-depth discussions on the results. Please refer to Sec. 5.2 and Sec. 5.3 for details.

Reviewer comment (Theory):
What is the aim of hierarchy-agnostic alignment when we already have the hierarchy information? Why is the ``agnostic'' part important?

Author response:
> What is the aim of hierarchy-agnostic alignment when we already have the hierarchy information? Why is the ``agnostic'' part important?

Sorry for the confusion. Please bear with our clarifications below.   Our main idea is to learn a shared, hierarchical, multimodal feature space. Our model achieves this based on two major insights:  (1) each visual entity (e.g., image, pixel) can be represented through a composition of multiple semantic labels, and (2) these labels are correlated in nature, collectively forming a tree-shaped hierarchy. Based on these insights, we develop two components: hierarchy-agnostic visual-semantic alignment addresses (1), and hierarchy-aware semantic learning addresses (2). 

**Difference and relation between the two components:** the hierarchy-agnostic component __handles multimodal alignment, but ignores hierarchical information__; the hierarchy-aware component __only interprets the hierarchical structure of semantic concepts (visual information is not taken into account)__. These two objectives operate on different granularity and synergistically contribute visual-semantic structural alignment. Notably, direct aligning the hierarchical structures between two modalities is very challenging, and our decoupled design alleviates this and helps model converge.

Reviewer comment (Experiments):
The paper primarily discusses the advantages of conducting feature learning in hyperbolic space compared to Euclidean space in a theoretical manner, highlighting the inherent exponential growth characteristic of hyperbolic embeddings in capturing hierarchical structures. However, there is a lack of experimental evidence to support this design choice. A potential baseline approach could involve performing all the loss terms in Euclidean space, such as triplet loss in Euclidean space.

Author response:
Sorry for the confusion.  We had already provided the experiments to examine the choice of latent space. Please refer to **Table 7** (copied below), in which `Hyperbolic` indicates our approach, while `Euclidean` means that all losses are computed in Euclidean space. The results show that using a hyperbolic space in our approach yields **consistently** and **notably** improvements against using a Euclidean space, across four datasets. This supports our theoretical analysis and evidences the efficacy of our algorithmic design. Thanks.

|Geometric space | City (mIoU$^1$, mIoU$^2$) | PASCAL (mIoU$^1$, mIoU$^2$, mIoU$^3$)  | ImCLEF07A (mAP, CmAP, CV) | ImCLEF07D (mAP, CmAP, CV) | 
|:----------|:------------:|:------------:|:---------:|:------------:|
| Euclidean  | 83.04, 92.41 | 74.17, 87.11, 96.42 | 88.47, 90.14, 5.47 | 87.61, 88.47, 6.98 |
| Hyperbolic | **84.63**, **94.27** | **76.37**, **88.94**, **97.88** | **92.21**, **92.44**, **3.18** | **90.76**, **91.42**, **5.71** |

Reviewer comment (Experiments):
The paper does not provide explicit details on how the taxonomy of labels in different datasets is obtained. It would be beneficial to include information regarding the acquisition of label taxonomy and potentially include tree structures to visually illustrate the hierarchical relationships within the taxonomy.

Author response:
Thanks for pointing this out.  All the hierarchical structures we used are officially defined in respective datasets. We directly use them without any modification. Following your suggestion, we have illustrated detailed semantic hierarchies on the datasets in Appendix Fig. 2 and Fig. 3. Thanks.

Reviewer comment (Experiments):
Since the paper demonstrates the benefits of incorporating taxonomy as prior knowledge for recognition tasks in a close-vocabulary setting, it raises an intriguing question about the scalability of this approach to an open-vocabulary setting. Considering the superior performance achieved by visual-semantic alignment methods like CLIP on in-the-wild data, it is worth questioning whether this work can provide similar advantages in the open-vocabulary domain.

Author response:
This is an interesting question and our insights are in two aspects:

First, theoretically, our algorithm can be promoted to open-vocabulary domain, since its basic idea is consistent with CLIP. However, in practice, there are challenges: 1) models like CLIP require very large-scale training data like image-text pairs, and explicit modelling the hierarchical structure within the data (as our method does) is hard if not impossible. Hence, implicit hierarchal structure learning appears to be more favorable and the proposed triplet loss that relies on pre-defined label taxonomy might be not suitable. 2) in comparison with contrastive-like loss used in CLIP, end-to-end optimization of VAE for large models is more difficult and expensive. Therefore, greedy optimization algorithms seem to be crucial as well. 

Second, though not closely relevant to your question, we highlight that our model can serve as a more trustworthy system to handle open-vocabulary cases, as compared to current recognition models. Consider a simple label taxonomy: `Animal` $\rightarrow$ `Okapi`. For an Okapi image, even we might be not sufficiently trained to identify rare animal `Okapi`, our model can still hold a high confidence to interpret the image belonging to a broader category of `Animal'.  This feature makes our algorithm fit to safety-critical applications (e.g., autonomous driving), even it itself is not purposely trained for open-vocabulary recognition.  Thanks.