Paper: Shadow Cones: A Generalized Framework for Partial Order Embeddings (ICLR-2024)

Reviewer comment (Theory):
Why the experimental results by Entailment Cone and Penumbral-Poincaré-ball are different while they are equivalent according to Theorem 3.2?

Author response:
In addition, we have a different view regarding the experimental results. Specifically, as summarized in Table 1 of the current manuscript, different formulations have different properties, such as geodesic convexity, light source positions and size. We cannot conclude that one cone formulation is always better than the rest. The optimal choice depends on certain types of data or task objectives.

Reviewer comment (Theory):
What do the shape and size of the light source and the size of embedding in the umbral cases imply in the context of partial order or semantics?

Author response:
**What do the shape and size of the light source and the size of embedding in the umbral cases imply in the context of partial order or semantics?**

This is a very interesting question that we would like to explore with an additional experiment. Specifically, we modified our existing framework to learn source radius from data. **Figure 3 and 4 of the supplementary rebuttal pdf** show the learned embedding as well as light source radius. For the mammal dataset, the source radius quickly expands from its initial size, resulting in a converged radius that is very large compared to the distribution of learned embeddings. We think this might be a result of the connectivity of the underlying data. The mammal graph has only four disconnected components, and is thus highly interconnected. A large light source can cast large penumbral shadows, making it easier to model highly connected DAGs. In follow-up work, we would be interested in exploring whether different data can induce radius of different sizes, and quantify if there is a relation between source radius and graph connectivity.

Reviewer comment (Evaluation):
How do we select one of the 4 proposed methods depending on the situation?

Author response:
**How to select one of the 4 proposed methods depending on the situation?**

In general, we recommend using shadow cones in the half-space model as there’s no $\epsilon$-hole issue compared to the Poincaré-ball model. If the goal is to represent partial order data with maximal accuracy as represented by the F1 score, then our experiments show Umbral-half-space to be the best method. 

If there are requirements or potential benefits from geodesic convexity, we recommend using the Penumbral-half-space cone. For example, if one were to perform interpolations between nodes, then perhaps Penumbral-half-space is superior as it achieves a relatively high F1 score, and its cones are geodesically convex. 

For multi-relation embeddings, where one desires to learn the sizes and positions of multiple light sources, each casting a different colored shadow, we recommend the Penumbral or Umbral-Poincaré-ball, because the half-space formulations restrict the size and position of the light source. In umbral-half-space, the light source is placed at infinity, while in penumbral-half-space, the light source is infinitely large. We discussed multi-relation embeddings in Appendix H.

Reviewer comment (Experiments):
At the end of Section 4, the authors make claims about ease of training, but do not justify claims beyond just a statement by fiat.

Author response:
**Claim on the Ease of Training**

Our training routine is less complicated than that of the Entailment Cone, in the following two ways. 
1) While the entailment cone uses pretrained 100-epoch embedding as initialization in order to avoid the $\epsilon$-hole issue in the Poincaré-ball, we simply use random initialization around the origin as a one-stage approach.
2) Our half-space based cones and embeddings have no $\epsilon$-hole issue at the origin, and need no extra steps to project the “stray” points out of the hole.

Reviewer comment (Theory):
What is the relationship between convexity and performance? The non-convex method seems to perform better. Why is this the case?

Author response:
**Relationship between convexity and performance**

This is an interesting question. As discussed in Appendix H, future works, we don’t have a clear answer to it now. We expect geodesic convexity to be beneficial for certain types of data and the task objectives, which we would like to explore in follow-up works.

Reviewer comment (Theory):
What are the potential challenges to learning representations in the half-space formulation?

Author response:
**Challenges to learning representation in the half-space formulation**
There are always precision-related challenges in learning representations in any models of hyperbolic space. Such problems occur when the embedding is near the boundary of the model, i.e., $||x|| = 1$ in the Poincaré-ball model and $x_n=0$ in the Poincaré-half-space model. The limited precision of underlying floating-point computations (imprecision issue) can lead to unbounded error. Yet, the half-space model is already less prone to this issue compared to other models of hyperbolic space like the Poincaré-ball and Lorentz model, as studied in some previous work.

Reviewer comment (Experiments):
In my opinion, the authors should include in their comparison methods that do not rely on hyperbolic spaces (but rather on Euclidean ones for example or with manifold with positive curvature).

Author response:
**Comparison with methods that do not rely on hyperbolic spaces**

We would like to refer the reviewer to our general response, where we compare our proposed shadow cones against box-embeddings, which is a type of Euclidean embedding.

Reviewer comment (Experiments):
I would be interested to know if the method could be used in more general settings. In particular, how the method could be used for real world graphs that are not (exactly) DAGs?

Author response:
**How the method could be used for real world graphs that are not (exactly) DAGs?**

We propose shadow cones as a novel partial order embedding framework in hyperbolic space. Thus, we experiment on datasets with partial orders. Graphs that are not (exactly) DAGs deviate from the field of partial order embedding. For example, knowledge graphs, edge relations in KGs contain symmetric and other more complicated relations, which are not partial orders. Naively encoding them is likely to fail. It’s an interesting and growing area to embed general (not necessarily DAG) graphs within a partial order embedding framework. An example is embedding with minimal spanning trees. Such methods are beyond the scope of this paper.

Reviewer comment (Experiments):
I would be interested to know if the method could be used in more general settings. In particular, how the method could be used for multiclass hierarchical problems?

Author response:
**How the method could be used for multiclass hierarchical problems?**

We assume the reviewer is referring to multi-relation embeddings. This can be achieved using multiple light sources, with each light source casting shadows of one color that captures one type of relation. We briefly illustrated this idea in Appendix H. Multi-relation embedding is an exciting direction that we are interested in exploring in follow-up works.

Reviewer comment (Writing):
At the end of the first paragraph of the introduction of Section 3, I think there is an error. Should the last sentence rather be "Specifically, $v$ is in the shadow cone of $u$ iff $v \subset$ the shadow of $u$." ?

Author response:
We thank the reviewer for pointing out the missing definition of $P$ (the set of positive relations) and $N$ (the set of negative relations), the error in section 3 paragraph 1, and after theorem 4.2. We will make sure to fix them in the updated manuscript.

Reviewer comment (Writing):
After Theorem 4.2, I think the first "their" should be removed from the sentence "which is designed to draw child nodes v closer to their the cones of their patent nodes".

Author response:
We thank the reviewer for pointing out the missing definition of $P$ (the set of positive relations) and $N$ (the set of negative relations), the error in section 3 paragraph 1, and after theorem 4.2. We will make sure to fix them in the updated manuscript.

Reviewer comment (Novelty):
The first weakness for me is the context for the work. I think some more discussion to related concepts such as DAG learning, which is learning DAGs from data [1,2,3], other hyperbolic hierarchical learning like hyperbolic tree learning, which is about learning trees to represent hyperbolic data or embed in hyperbolic apace [4,5], graph embeddings in hyperbolic space [6,7,8], and hyperbolic link prediction [9] would be great.

Author response:
**Broader Contexts:**
We sincerely thank the reviewer for kindly suggesting broader literatures to contextualize our work within. We will incorporate discussions and comparisons with the suggested papers in section 2 of our updated manuscript.

Reviewer comment (Experiments):
I think the experimental setup could be further expanded upon. First, I think it should be clarified that we can do the link prediction without the embedding step (this would have a 100% accuracy) and then mention that you do the embedding step to understand the how well the embedding performs. As part of the experiment, it would be good to have a baseline for Euclidean poset embedding. However, I admit I do not know of one. Maybe one of the DAG learning papers has a baseline that could be used.

Author response:
We thank the reviewer for pointing out the missing definition of $P$ (the set of positive relations) and $N$ (the set of negative relations). We also agree with the suggestion on further explanation of the experimental setup. We will make sure to clarify and explain them in the updated manuscript. We would also like to refer the reviewer to our general response section, where we compare our proposed shadow cones against box-embeddings, which is a type of Euclidean embedding.

Reviewer comment (Writing):
In the loss function $P$ and $N$ haven't been defined.

Author response:
We thank the reviewer for pointing out the missing definition of $P$ (the set of positive relations) and $N$ (the set of negative relations). We also agree with the suggestion on further explanation of the experimental setup. We will make sure to clarify and explain them in the updated manuscript.

Reviewer comment (Theory):
For the cones I do not understand the boundary computation. Could the authors please expand on that? I did not see anything in the appendix either.

Author response:
**Boundary computation and intuitive explanations why umbral cones are not geodesically convex**

As illustrated in the last question, Penumbral shadow, which results from partial eclipse of a light source with volume, is produced by a point object. Therefore, the boundaries of penumbral shadow are geodesics passing through $u$ while being tangent to the light source. Note that the $\text{Penumbral Shadow}(v) \subset \text{Penumbral Shadow}(u)$ for any $v\in \text{Penumbral Shadow}(u)$, therefore, all possible children of $u$, i.e., penumbral cone of $u$ coincides with the penumbral shadow of $u$, with geodesics boundary described above. 

On the other hand, Umbral shadow, which results from total eclipse of a point light source, is produced by an object with volume (a ball around $u$). Therefore, the boundary of umbral shadow is geodesics passing through the point light source, while being tangent to the ball around $u$. Now, let’s look at all possible children $v$ of $u$, i.e., umbral cone of $u$. Since the objects $u$ and $v$ have volumes, then for $u \preceq v$ to be true, it must be that $v$ and its ball are entirely immersed in $u$’s shadow. Otherwise, the portion of $v$’s ball outside of $u$’s shadow can cast shadows outside. In **Figure 1 of the supplementary rebuttal pdf**, we sketch a case where $v$’s center is in $u$’s shadow, but nevertheless $u \npreceq v$, because part of $v$’s ball is outside.

Reviewer comment (Theory):
Why do we need the ball around $y$ to be in the shadow for $y$ to be in the umbral cone, but only need the point $y$ to be in the shadows for the penumbral cone? I see that the umbral method seems to perform the best. I imagine based on how the loss in formulated this pushes the point to the interior of the cone rather than leaving it on the boundary where we might have numerical issues.

Author response:
**Question 3:**

First we would like to remind the reviewer that all of our cone formulations are motivated by the same overarching idea: $u \preceq v$ if and only if the (penumbral or umbral) shadow cast by $v$ is a subset of the shadow cast by $u$. The shape of the object (single point, or ball with volume) is dictated by the physicality of different types of shadows, and is not an arbitrary choice. Penumbral shadow, which results from partial eclipse of a light source with volume, is produced by a point object. 

On the other hand, Umbral shadow, which results from total eclipse of a point light source, is produced by an object with volume. In order to satisfy the definition of the subset relation of (umbral and penumbral) shadows, we need the ball around objects $y$ to be in the shadow in the umbral cone, but only need the point $y$ to be in the shadow for the penumbral cone. 

The Umbral-half-space cone performs the best in general. The reason can be multi-folds, including loss function and optimization process. However, we attribute its main advantage to the infinite-height light source, and lack of $\epsilon$-hole at the origin.

Reviewer comment (Presentation):
Compared with the novel and interesting motivation, the description, especially the figures, lacks explanation, almost all the symbols in the figures are not mentioned in the captions or corresponding text.

Author response:
We kindly thank the reviewer for the suggestions on the descriptions. We will update the manuscript to explain all symbols in the figures.

Reviewer comment (Evaluation):
The experiment is not very supportive of the claim. For example, In the experiment, Poincaré ball based cones performed worse than Poincaré half-space based cones. While in the important baseline "hyperbolic entailment cones", the Poincaré ball is used, if the proposed cone generalizes "hyperbolic entailment cone", it should be as good as the "hyperbolic entailment cone".

Author response:
We believe the reviewer misunderstood our claim: shadow cones generalize hyperbolic entailment cones (Poincaré-ball based) to a broad class of formulations. The hyperbolic entailment cones are mathematically equivalent to a special case of shadow cones, namely, Penumbral-Poincaré-Ball. We understand that the reviewer might be puzzled by the performance difference of hyperbolic entailment cones and Penumbral-Poincaré-Ball, given that the two are mathematically equivalent. We would like to refer the reviewer to our explanation in the general responses. 

Poincaré half-space based cones are another category of cones within the shadow cones, which are distinct from the Poincaré-Ball based formulations. We analyzed and discussed why Poincaré half-space based formulations perform better than Poincaré-Ball based formulations in section 5, page 9, [Discussion on the impact of $\epsilon$-hole].

Reviewer comment (Writing):
The use of different definitions for light source in the Penumbral cone and the Umbral cone is somewhat inconsistent.

Author response:
We wanted to emphasize that whether the light source has volume is dictated by the physicality of different types of shadows, and is not an arbitrary choice. Penumbral shadow results from partial eclipse of the light source, and could only be produced by light source with volume. Umbral shadow, on the other hand, results from total eclipse of a point light source, and can be produced by both point-like light sources and objects with volume. When both the light source and objects have volume, a third type of shadow is produced - antumbral shadow, which we show in Appendix G to be mathematically equivalent to penumbral shadow formulations.

Reviewer comment (Experiments):
While the authors tested their framework on four datasets, it would be beneficial to see how the framework performs on a wider variety of datasets, including those from different domains or those with different characteristics (like KG, recommender system datasets and so on). This would provide a more comprehensive evaluation of the framework's performance and versatility.

Author response:
**Experiments on a wider variety of datasets from different domains or those with different characteristics (like KG etc.)**

We propose shadow cones as a novel partial order embedding framework in hyperbolic space. Thus, we experiment on datasets with partial orders. We augment our experiments with MCG and Hearst dataset beyond the standard Wordnet benchmark. Datasets from different domains or those with different characteristics are likely to deviate from the partial order embedding area. For example, knowledge graphs, edge relations in KGs contain symmetric and other more complicated relations, which are not partial orders. Naively encoding them is likely to fail. In fact, it’s an interesting and growing area of embedding general relations in a partial order embedding framework, for example, with minimal spanning tree. Hence, we only consider datasets with partial order relations and it’s beyond the scope of this paper to experiment on datasets and tasks from other domains.

Reviewer comment (Novelty):
The work builds on the entailment cone but a few partial order methods have been proposed and a thorough comparison or discussion is needed.

Author response:
We kindly thank the reviewer for suggesting further comparisons. We would like to refer the reviewer to our general response section, where we compare our proposed shadow cones against box-embeddings.

Reviewer comment (Presentation):
Captions for the figures are brief and ambiguous, hindering their readability.

Author response:
We will update the captions of figures to improve the readability.

Thank you for your thoughtful feedback! We will update the manuscript shortly incorporating the responses and improving the readability of figures and their captions.