Paper: Binder: Hierarchical Concept Representation through Order Embedding of Binary Vectors (ICLR-2024)

Reviewer comment (Theory):
As a starting motivation of the research, the paper criticizes hyperbolic embedding, pointing out that "learning in hyperbolic space is challenging because optimization algorithms, like gradient descent and its variants, are not well studied for hyperbolic space." Indeed, the gradient descent methods on hyperbolic space have been well-studied theoretically, e.g., [A-E]. Although the convergence to the global optimum cannot be guaranteed, as not in Euclidean space, they are not by far worse than the author's theoretical guarantee on the proposed algorithm. The author mentioned that the problem is a NP-complete problem as a decision problem, but it is not a practically positive result unless P=NP. In this sense, the current draft gives readers impression that the author has not solved the original motivation. If it is difficult to provide a theoretical guarantee of the proposed algorithm, the author should criticize the hyperbolic embedding in another way.

Author response:
WK2: As a starting motivation of the research, the paper criticizes hyperbolic embedding, pointing out that "learning in hyperbolic space is challenging ...  are not well studied for hyperbolic space." Indeed, the gradient descent methods on hyperbolic space have been well-studied theoretically, e.g., [A-E]. Although the convergence 
  .... hyperbolic embedding in another way.

A:  After reading the references that you have listed, we admit our lack of knowledge about the latest advances of ML in hyperbolic space. What we meant to say in our comment is that optimization in hyperbolic space is more challenging that real-space and this is how we want to change the above sentence in a revised manuscript. We thank you for reading our paper including the Appendix to learn about BINDER's convergence claim, and we agree with your comment entirely. Nevertheless, without criticizing competitors framework, we like to point you towards the superior experimental results of BINDER over its competitors.

Reviewer comment (Theory):
This item is about another important motivation of the paper, "logical operation." The explanation regarding the logical operation on the binary representations does not seem correct. The logical "not" operator does not seem to work like the semantic "not." Assume that "living thing" is [0, 0], "cat" is [0, 1], and "dog" is [1, 0]. This does not self-contradict since a cat is a living thing and a dog is a living thing, too. Let's apply the logical "not" to the living thing. According to your explanation, "not living thing" is [1, 1]. Now, according to the rule, we conclude that "a not living thing is a cat", and "a not living thing is a dog." This is obviously wrong. Hence, the proposed boolean representations are not intuitive as the authors claim.

Author response:
WK3: This item is about another important motivation of the paper, "logical operation." The logical "not" operator does not seem to work like the semantic "not." Assume that "living thing" is [0, 0], "cat" is [0, 1], and "dog" is [1, 0]. This does not self-contradict since a cat is a living thing and a dog is a living thing, too. Let's apply the logical "not" to the living thing. According to your explanation, "not living thing" is [1, 1]. Now, according to the rule, we conclude that "a not living thing is a cat", and "a not living thing is a dog." This is obviously wrong. Hence, the proposed boolean representations are not intuitive as the authors claim.

A: We refer you to the answer that we provided to Q4 of Reviewer VtjX as both your question and that question are similar. We only claim that we can obtain an embedding of an entity outside our domain by bit operation (bit operation when applied on a bit vector yield another bit vector, due to closeness property), but we cannot guaranty logical implication of an outside domain entity as those entity are not part of the training data. Going with your example: yes "not living thing" has a representation of [1, 1], and we stop right there, rather than making further implication, based on this embedding. We hypothesize, for real-life purpose, such embedding built by boolean operation will suffice for downstream knowledge discovery task, but validation of that claim is outside the scope of this paper. Please note, we clearly noted (sec. 5) that out method is transductive, just like SOTA methods proposed for this task.

Reviewer comment (Presentation):
Citation does not include which year it is published, which makes it extremely difficult to see the flow of the existing methods.

Author response:
WK4: Citation does not include which year it is published.

A: We are sorry about this and thank you very much for noticing this. We have fixed it already in our manuscript.

Reviewer comment (Presentation):
The page limitation is violated.

Author response:
WK5: The page limitation is violated.

A. We are within the page limit (9 pages of main text). References, Appendix, and Reproducibilty does not count towards page limit.

Reviewer comment (Evaluation):
Why did you not show the memory (RAM) to store the representation in the numerical experiments? I thought you could emphasize the advantage of the proposed method clearer this way, since the proposed method's type is boolean, and existing method's types are float or double.

Author response:
Q1: Why did you not show the memory (RAM) to store the representation in the numerical experiments? I thought you could emphasize the advantage of the proposed method clearer this way, since the proposed method's type is boolean, and existing method's types are float or double.

A: We had to compromise lot of content of this paper due to ICLR page limit, as you can see from our 5 pages of Appendix. In Appendix F.3, we mentioned that for $n$ entities and $d$ dimension, we only need $\dfrac{nd}{8}$ bytes, whereas competitors take $ndK$ bytes where $K$ is the number of bytes needed for a floating point (4 or 8). We will emphasize that in the main part of our revised manuscript.

Reviewer comment (Theory):
Why did you not show the order of time complexity of the proposed algorithm? If I understand it correctly, each step's time and space complexity is linear to the dimension, which seems to be an advantage of the proposed algorithm.

Author response:
Q2: Why did you not show the order of time complexity of the proposed algorithm? If I understand it correctly, each step's time and space complexity is linear to the dimension, which seems to be an advantage of the proposed algorithm.

A. Yes your understanding is correct. The complexity equation is provided in Appendix B along with Pseudo-Code. Quoting from our paper: "The overcall computational complexity is $O(ndT(|P|+|N|))$, for $n$ words, and $d$ dimensions, which is linear in each variable.". We will revise the manuscript to bring this line in the main part of the paper from the Appendix.

Reviewer comment (Writing):
Generalization to Unseen Concepts: While BINDER claims to generate embeddings for unseen concepts by using logical functions over existing concepts, it would be beneficial to provide more detailed explanations and examples of how this generalization is achieved. A concrete illustration of how BINDER generates embeddings for unseen concepts could strengthen the paper.

Author response:
A: All the existing works that we compare against, along with BINDER follows a transductive learning setup, so they cannot generate embedding of arbitrary unseen objects. This is the reason, in our paper and in earlier works, link prediction is evaluated over transitive closure edges, for which the entities of the predicted edges are already present in the training data. However, BINDER being binary
vector based representation, is able to provide embedding of some entities, which are direct logical derivative of existing entities in the
training data. One example that we gave in the paper is, if BINDER generate an embedding of "living-thing", and "non-living-thing" is not in the training data, we can simply inverse the bits of "living-thing" to give an embedding of non-living thing. In general, if an entity
$X$ can be logically derived by a boolean formula over entities in the training data, Say, $X = formula(S)$, where $S$ is a subset of entities in the training data, BINDER can obtain the embedding of $X$ as following: $Emb(X) = formula(Emb(S))$. This is how, BINDER can obtain limited generalization. Although this is not full generalization, it is better than the competitors method, which provide no generalization at all.

Reviewer comment (Experiments):
Hyperparameter Sensitivity: The paper discusses hyperparameters like the learning rate and bias but does not delve into their sensitivity analysis. A study on how these hyperparameters affect BINDER's performance and convergence would provide a better understanding of its behavior.

Author response:
A: In appendix C we have discussed about hyperparameter tuning and listed the optimum hyperparameters for our experiments. However, we agree with the reviewer that hyperparameter sensitivity experiment would provide a better understanding of BINDER's behavior. If the paper is accepted, we will add another result section in Appendix, discussing "hyperparameter sensitivity".

Reviewer comment (Evaluation):
Comparative Discussion: While BINDER's strengths are well-discussed, it would be beneficial to have a comparative discussion with competing methods, highlighting where BINDER outperforms them in more depth. This would provide additional context for readers.

Author response:
A: BINDER outperforms competing models particularly in transitive closure experiments. BINDER only needs direct edges to learn all entity embeddings in the training set. This is validated in the  0% transitive closure experiments; in these experiments, BINDER can easily achieve close to 99% accuracy, whereas the competing models perform poorly, with accuracy in the range of 60% to 90%. We will update our manuscript to highlight BINDER's strength.

Reviewer comment (Theory):
The complexity. It seems that the model requires very high dimensionalty but there is no such discussion. The worst case is that the concepts are fully disjoint, then you need N dimension, which makes the model not scalable.

Author response:
A: Since binary space has only two possible values at each dimension, it is expected that binary embedding will need many bits to capture the desired is-a relationships, specifically for large datasets. However, do note that, BINDER works on bit domain, whereas existing SOTA works on real-vector domain. For BINDER, a dimension of 200 is simply 200/8 = 25 bytes, which is space taken by  6 real numbers.

Reviewer comment (Evaluation):
Why does the accuracy (acc) decrease when the number of transitive edges increases? How can this phenomenon be explained?

Author response:
A:  General trend is that when the transitive edges in the training data increase, the performance of the learning task increase. This is indeed the case in Table 4 as we gradually increase Transitive closure from 0% to 50%. There is exactly 3 exceptions in Table 4. Those  
are:  (1) Medical dataset (10% →  25%), (2) Music (0% →  10%) and Nouns (25% →  50%); in these three cases there are slight drops in accuracy, which can probably be attributed to  BINDER's randomized algorithm for learning embedding.

Reviewer comment (Theory):
In equation (1), you specify that a $\neq$ b, meaning when a = b, it should be considered a negative example. However, in Section 2.3, the negative pairs (n) involving reflexivity constraints, where a is-a a, are excluded. Does this mean that reflexivity constraints are neither treated as positive nor negative examples? I understand that in the experiments and method, you've avoided dealing with the self-relation, but in practice, it might occur. How do you plan to handle cases where the reflexivity constraint is present?

Author response:
A:  If we allow $a=b$, then an all-zero vector for all entities will satisfy all constraints $a \cap b = b, \forall (a, b) \in P$, trivially. $a \ne b$ is needed solely to avoid that. By explicitly ensuring that $a \ne b$, we guaranty that every entity has a distinct bit-vector. Yes, you are right, reflexivity constraints are neither treated as positive nor negative examples. In practice, if $a = b$ occurs, we will first pre-process the dataset to cluster all equal entities to create super-entities; this will be like contraction of the edge $(a, b)$ (as is done in Karger's min-cut algorithm). After such contraction, all remaining entities are distinct and BINDER can work as usual.

Reviewer comment (Presentation):
In Table 3 (Reconstruction Results Acc(\%) (dim)) and similar cases, what is the purpose of including "(dim)" in the table header?

Author response:
A: Since, BINDER uses bit-vector, it provides massive saving in terms of memory footprint of the embedding vectors. To compare BINDER's memory footprint with other competing methods, we were really interested to see at what dimension other competing methods reach their best results. As we have harvested those results, we thought to include this in this Table
for the readers to see that not all methods reach their best for similar dimension value. This could instigate a follow-up work
by us or others to investigate how the dimension value affects the performance of different embedding models, which may
use different kind of geometric space. For instance, a general observation from this results is that HEC (Hyperbolic Entailment Cone) embedding generally requires higher dimension value to achieve best result.

Reviewer comment (Theory):
Could the finite permutation space of binary vectors result in a loss of expressivity, limiting the model's ability to capture complex relationships?

Author response:
A:  It is possible, and that is why a randomized algorithm is the best to obtain a solution for such an embedding framework, like ours. We believe by using randomization, we are able to exploit the best out of a finite permutation space of binary vectors. More research is needed to understand BINDER's limit for capturing complex relationship. In this work we confined ourself only to is-a relationship, which is transitive, and easy to work with (non-complex). But, we will claim that BINDER can be generalized to extend to other relationships, which are more complex and in that scenario BINDER's ability can be put into test.

Reviewer comment (Novelty):
The Introduction section is also missing an important recent work on two-view knowledge graph embeddings, which jointly embed both the ontological and instance view spaces: [KDD 2022] Dual-Geometric Space Embedding Model for Two-View Knowledge Graphs. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD '22). Association for Computing Machinery, New York, NY, USA, 676–686. https://doi.org/10.1145/3534678.3539350

Author response:
Our work is not a general knowledge graph embedding paper, so it is not clear why the reviewer is considering "two-view knowledge graph embedding" an important recent work related to our contribution. In our entire manuscript (except the references) the phrase "knowledge graph" came only twice (both in second paragraph of introduction); just to clarify that we are concerned with only hierarchical concept representation, not for embedding of RDF like knowledge graph entities. So, it is not clear to us in what context we will cite or discuss the paper that the reviewer is referring to. We would appreciate if the reviewer kindly clarify this to us.

Just to reiterate: given a collection of entities where the entities are related by is-a relation, we are designing order embedding representation of these entities so that the entity vectors can capture the order relation. This is our task, which is also the task of the works that we are comparing against in our experiment section. The KDD 2022 paper is neither related to us, nor can we use the dataset used in that paper (Yago, DbPedia) to compare against.

Reviewer comment (Experiments):
Further, the model fails to include important baseline models such as standard knowledge graph embedding model in the hyperbolic space e.g., RefH/RotH/AttH, hyperbolic GCN (HGCN), and the product space (M2GNN).

Author response:
This work is not a model for "knowledge graph embedding in the hyperbolic space". Could the reviewer kindly explain why and how the listed works are important baselines?

Reviewer comment (Experiments):
Moreover, the size of the datasets also seem to be relatively small-scale with number of nodes and edges on the scale of thousands as opposed to million node/billion edge graphs indicative of real world KGs e.g., DBPedia & YAGO.

Author response:
We used the largest labeled datasets that is publicly available (WordNet Nouns) for the order-embedding. Earlier works also use this dataset. In fact, several of the earlier works use only one dataset, we are using 5 datasets to produce a comprehensive result.

Reviewer comment (Theory):
Why is only the hyperbolic space being considered? Entities can form cyclic relations as well, which is better modeled in the spherical space. Perhaps the authors need to more clearly denote the distinction between entities and concepts.

Author response:
Hyperbolic space is being considered because hyperbolic cones have been used in earlier works for performing order embedding of entities exhibiting is-a relationship. Is-a relation is by definition non-cyclic, so cyclic relations do not occur in our dataset and our problem formulation. Spherical space is not ordered space, nor is it used for ordered embedding, so it is not considered.

We are using entity and concept synonymously in this paper, as both are relevant when someone refer to hypernym-hyponym (is-a) relationship. For example, we can think of "classification model" as a concept in Machine Learning, a sub-concept of this can be "Neural Networks"; Neural Networks is-a Classification model, so they exhibits an ordered is-a relation, which is clearly not cyclic. On the other hand "cat" is an entity, and "animal" is another entity. Cat is-a animal, and they exhibit an ordered is-a relation, which is also non cyclic.

Reviewer comment (Theory):
The claim that Binder embeddings have unique capabilities such as well-defined complement, union, or intersection is incorrect or no more true than for other embeddings. An example is provided showing how reversing bits to represent a complement leads to logical inconsistencies, including overlap between sets and elements belonging to neither a set nor its complement. It is shown that no single bit vector can represent the full complement due to the inability to express certain logical conditions, and similar reasoning applies to the lack of closure under union.

Author response:
Q4: You mention unique capabilities of Binder embeddings, such as well-defined complement, union, or intersection operations. Given the issues highlighted with these operations, how do you respond to the concerns about this?

A: Complement, union, intersection are well-defined for binary bits, so using BINDER embeddings, one can extrapolate embedding of entities which are not in the training data by using logical operations over BINDER's embedding vectors. Our claim extends only up to this. We are not claiming that BINDER's embedding will ensure logical inference and is-a relation consistency over entity-pairs, which are not in the training data. We claimed that other continuous domain models do not have well defined complements. For example, in a continuous domain, the “inverse” cone of (0.5, 0.5, ..., 0.5) is itself, which makes little sense when such a vector represents a concept in a domain. The scope of our work is to highlight the benefits of binary vector embedding to capture order relationships over the competing models in terms of simplicity of the optimization algorithm, compactness of the embeddings, time-efficiency, and generalization over unseen entity up to certain capacity.

Considering reviewer's example, say, based on BINDER, living-thing has a representation of $[0, 1, 1]$. Then based on our claim, non-living thing has an embedding of $[1, 0, 0]$. Our claim extends only up to this. Any transitivity and is-a extension over the non-living entities is out of scope of BINDER as the method is transductive and has no capability to ensure consistency over relations between unseen pairs (again, all SOTA methods on this task are transductive). So, when the reviewer 
extends is-a relation over the set of not living and suggests that the set of non-living things are: $\{[1, 0, 0], [1, 1, 0], [1, 0, 1], [1, 1, 1]\}$, BINDER does not endorse that claim. 

We are sorry if we over-claimed in our write-up which created the confusion. We will clarify our claim in revised manuscript.

Reviewer comment (Experiments):
There are several problems with the experiments. The authors use a reweighted accuracy statistic instead of the more conventional F1 metric for imbalanced data. The test set negatives are created by random perturbation, which can yield coarse and biased evaluations. For example, in the Order Embeddings paper test set, very high F1 can be achieved with a trivial heuristic, motivating more comprehensive evaluations such as using the full adjacency matrix. Comparable baselines are available with implementations in existing code repositories.

Author response:
Q6: Why was a re-weighted accuracy statistic chosen over the conventional F1 metric in your experiments, especially in the context of data imbalances? 

A: We agree that for classification over imbalance data, F1 measure is a good metric; however some argue (and we agree with them) that balanced_accuracy (BA) is actually a better metric, which is defined as the average of positive class recall = $\frac{TP}{TP+FN}$, and negative class recall=$\frac{TN}{TN+FP}$. Balanced accuracy is also defined as $\frac{sensitivity + specificity}{2}$. Balanced accuracy is a better metric than F1-score for imbalanced dataset, because F1-score does not care about how many true negatives are being classified. This is because F1-score uses precision and recall which together use only three entries of the confusion matrix (TP, FP, FN); on the other hand, balanced accuracy uses all four entries of the confusion matrix. For an example, say a dataset has 1000 (-ve) and 10 (+ve) examples. If the model predicts there are 15 positive (TP=5, FP = 10), and predicts the rest as negative (TN = 990, FN = 5), we get, $Precision=\frac{5}{15}= 0.33$, and $Recall = \frac{5}{10} = 0.5$, it yields, $F1-score = 0.4$. The model gets absolutely not much credit for correctly predicting 990 out of 1000 examples as negative. However, the balanced accuracy $= \frac{1}{2} (\frac{5}{10} + \frac{990}{1000}) = 0.745$, which provides somewhat a more realistic picture. 

We actually used F1-measure, but found it to be very harsh for the competitors. We want to be nicer to the competitors and used Balanced accuracy. In the Table below, we show Precision, Recall, and F1-score (left to right) results for 0% transitive link prediction. As can be seen BINDER's F1-Scores (bold numbers) are significantly better than all other methods on all datasets. The competitor methods suffers severely due their poor precision. 

| Models   | Medical                | Music                  | Shwartz Lex            | Shwartz Random         | Nouns                  | 
| -------- | ---------------------- | ---------------------- | ---------------------- | ---------------------- | ---------------------- | 
| OE       | 70.1, 97.6, 81.6       | 43.8, 94.6, 59.9  | 33.1, 79.3, 46.7           | 23.6, 83.4, 36.7       | 49.6, 48.2, 48.9     | 
| Poincare | 35.2, 95.2, 51.4 | 19.6, 66.2, 30.3 | 21.1, 80.6, 33.5 | 24.4, 59, 34.5  | 40.1, 28.1, 33.1 | 
| HEC      | 44.9, 83.3, 58.3     | 27.7, 61.5, 38.2    | 23.3, 80.6, 36.1        | 31.1, 61, 41.2          | 43.7, 36.2, 39.6  | 
| T-Box    | 18, 88, 29.8       | 17.5, 93.1, 29.5    | 14.8, 94.2, 25.5         | 13.2, 83.2, 22.8          | 14.8, 97.8, 25.7 | 
| BINDER   | 93.3, 100.0, **96.6** | 78.3, 100.0, **87.8**  | 96.8, 100.0, **98.4** | 95.1, 100.0, **97.5**  | 85.2, 99.2, **91.7**  |

Reviewer comment (Novelty):
Several characterizations or claims made in the introduction are incorrect. The assertion that optimization algorithms are not well studied for hyperbolic space is false, as Riemannian gradient descent and related approaches are well understood, with methods existing for training in hyperbolic space using standard optimizers. The claim that box embeddings have more degrees of freedom than point embeddings is incorrect; experiments compare boxes to vectors with matched numbers of parameters. The claim that bit vectors are more interpretable is unsupported by experiments and lacks a clear theoretical basis, with interpretability being no greater than that of other embedding methods such as order, probabilistic order, or box embeddings.

Author response:
Q7: Please correct or respond to my assertions above regarding the inaccuracies in characterizing other baselines. (For example, the assertion that optimization algorithms in hyperbolic space are not well-studied, or that box embeddings have more degrees of freedom than point embeddings.) After correcting these claims, please address what specific benefits this embedding provides beyond those provided by the baselines. 

A: We will correct the claims in the revised manuscript, as you suggested. However, after correcting these claims, BINDER still holds the following specific benefits over competitors: (1)  better performance, (2) lower memory footprint of the algorithm, and (3) Ability to get embedding of out-of-training data in a limited setting (through binary operation over bit vectors). 

Q8: The interpretability of bit vectors is claimed to be superior in your paper. Can you provide empirical evidence or a theoretical framework that supports this claim, in a setting where equivalent effort is also given to order, probabilistic order, or box embeddings? 

A: BINDER is inspired from Formal Concept Analysis (FCA), which uses intent-extent ideas to represent relationships between a concept and subconcept. We tried to draw a parallel with FCA and alluded that '1' entries in a BINDER embedding corresponding to "presence of a latent property". This gives us a relative idea of how many latent properties are shared by two concepts and how strong is their relationships. We want to refer to Appendix D where we show a case study experiment on a toy dataset. At present, we do not have any theoretical framework to support this claim. This is a hypothesis only, and we will address BINDER embedding explainability in future works.

Reviewer comment (Experiments):
For reconstruction task, OE achieves better performance than BINDER with fewer dimension. Thus, BINDER does not show superiority over OE.

Author response:
You pointed out that for reconstruction task, OE achieves better performance than BINDER for fewer dimensions and concluded that BINDER does not show superiority over OE. Your claim may be coming from the observation that on two smaller datasets (Edges: Medical: 4.3k, Music:6.5k), OE achieves 100% accuracy by using d = 10, and 20, but binder achieves 99.9% accuracy by using 50 bits. 
As a counter argument we want to clarify than BINDER uses bits, whereas other methods operates in real number domain. We need at least 4 bytes (32 bits) to represent a real number. So, the claim that OE is using fewer dimension is untrue, because OE is using d=10 (10X 32 = 320 bits), and d = 20 (20 X 32 = 640 bits), whereas BINDER is using only 50 bits. If BINDER is allowed only 50 bits, for a fair comparison other methods should be allowed $\lceil 50/32 \rceil = 2 bytes$. From our experiments we observed that even with d=5 (5x32=160 bits) and d=10 (10x32=320 bits), OE achieves 99.4% and 98.6% accuracy respectively for medical and music, which is poorer than BINDER. Most importantly, on the largest dataset (Nouns) which has 100 times more edges than these smaller datasets, BINDER achieves 99.7% accuracy with 100 bits, whereas OE achieved 96.7% accuracy by using 200 x 32 = 6400 bits.

Reviewer comment (Theory):
BINDER may still suffer from the limitation of optimization, leading to inferior performance.

Author response:
The reconstruction performance on 5 datasets, where BINDER achieved 99.9, 99.9, 100, 100, and 99.7 percent of accuracy clearly validates that the optimization scheme that BINDER uses is excellent to find representation vector for is-a relation. We have further provided BINDER's proof of convergence in Appendix A. If you have any specific concern regarding BINDER's optimization, can you kindly elaborate so that we can alleviate your concern.

Reviewer comment (Evaluation):
It is better to report the mean results, rather than the best results.

Author response:
We actually reported mean results in Table 5 along with standard deviation for all BINDER experiments. Based on Table 5 results, our best results are well within a reasonable standard deviation from the mean results.

Reviewer comment (Novelty):
What are the differences between the proposed optimization method and randomized local search algorithm? What is the novelty of the proposed optimization method?

Author response:
Randomized local search does not utilize gradient of the objective, whereas BINDER's optimization uses gradient of each bit position to construct a probability of flipping that bit. The idea of using gradient to flip bit probabilistically to optimize is binary space is certainly novel. The novelty of BINDER's optimization method from methodology perspective consists of: (1) compute a proxy of gradient for a binary space (Section 2.4), Compute flip probability (Section 2.5). Substantial thinking has been put forth to come up with Table 1 and Table 2 along with gradient binary logic formulas for gradient computation shown in Eq. 5-10.

Reviewer comment (Theory):
How to ensure the convergence of the proposed optimization algorithms?

Author response:
Please see the Proof of Convergence in Appendix (Lemma 1-3 and Theorem 4)

Reviewer comment (Evaluation):
Why BINDER is better than OE in WordNet Nouns dataset?

Author response:
Commenting on the performance on a specific dataset is not easy, however, we can comment based on the trend that we have observed. Nouns is the most difficult dataset due to it having 82k entities, and 743k edges; which are multiple order of magnitude larger than other datasets. Performance of all methods drops on this dataset for both the tasks. But, other methods (as well as OE) suffer more severely than BINDER. BINDER loses only one to two percent in terms of balanced accuracy, whereas other methods loses 5% to 10%, or even more on some cases.

Reviewer comment (Reproducibility):
How is the dimension (in parenthesis) in Table 3 set? Why is the dimension of OE smaller than that of BINDER?

Author response:
We made extensive hyper-parameter tuning for all the methods: BINDER and competitors, and reported best results for all. OE uses the smallest dimension than other competitors probably because it uses real-number space which is less constrained, whereas other competitors use hyperbolic space, which is more constrained. Comparison to BINDER, OE actually uses more dimension than BINDER, as OE uses real space and BINDER uses binary bits.

Reviewer comment (Experiments):
Can you provide the experimental results of box embedding?

Author response:
Yes, we have it now: On the five datasets from Left to Right:

Recon. (100, 100, 100, 100, 99.8) 
LP (0 tran:   70.4, 78.0, 74.6, 71.4, 70.8)
LP (10 tran: 80.0, 80.0, 74.4, 74.4, 80.1)
LP (25 tran: 80.8, 80.3, 74.6, 74.6, 86.8)
LP (50 tran: 83.8, 84.3, 74.4, 74.5, 91.2)

On LP (Link Prediction), BINDER is substantially better than Box. However, On Recon (Reconstruction), Box is marginally better than BINDER. However, LP is prediction on unknown data, but Recon is prediction on train data. So, performance on LP is more significant for a ML task.