Box-To-Box Transformations for Modeling Joint Hierarchies (2021)

Abstract: Learning representations of entities and relations in structured knowledge bases is an active area of research, with much emphasis placed on choosing the appropriate geometry to capture the hierarchical structures exploited in, for example, isa or haspart relations. Box embeddings (Vilnis et al., 2018; Li et al., 2019; Dasgupta et al., 2020), which represent concepts as n-dimensional hyperrectangles, are capable of embedding hierarchies when training on a subset of the transitive closure. In Patel et al., (2020), the authors demonstrate that only the transitive reduction is required and further extend box embeddings to capture joint hierarchies by augmenting the graph with new nodes. While it is possible to represent joint hierarchies with this method, the parameters for each hierarchy are decoupled, making generalization between hierarchies infeasible. In this work, we introduce a learned box-to-box transformation that respects the structure of each hierarchy. We demonstrate that this not only improves the capability of modeling cross-hierarchy compositional edges but is also capable of generalizing from a subset of the transitive reduction.

## introduction representation learning for hierarchical relations is crucial in natural language processing because of the hierarchical nature of common knowledge, for example, <bird isa @xmath0. the isa relation represents meaningful hierarchical relationships between concepts and plays an essential role in generalization for other relations, such as the generalization @xmath1, @xmath2. the fundamental nature of the isa relation means that it is inherently involved in a large amount of compositional reasoning involving other relations. modeling hierarchies is essentially the problem of modeling a poset, or partially ordered set. the task of inferring missing edges that requires learning a transitive relation, was introduced in @xcite . the authors also introduce a model based on the reverse product order on r n , which essentially models concepts as infinite cones. region-based representations have been effective in representing hierarchical data, as containment between regions is naturally transitive. @xcite introduced axis-aligned hyperrectangles (or boxes) that are provably more flexible than cones, and demonstrated state-of-the-art performance in multiple tasks. thus far, not as much effort has been put into modeling joint hierarchies. @xcite proposed to simultaneously model isa and haspart hierarchies from wordnet @xcite . in order to do so, they effectively augmented the graph by duplicating the nodes to create a single massive hierarchy. their model assigns two separate box embeddings b isa and b haspart for each node n, where these two do not share any common parameter between them, and therefore misses out on a large amount of semantic relatedness between isa and haspart . in this paper we propose a box-to-box transformation which translates and dilates box representations between hierarchies. our proposed model shares information between the isa and haspart hierarchies via this transformation as well as crosshierarchy containment training objectives. we compare box-transform model with multiple strong baselines under different settings. we substantially outperform the prior two-box model while training with only the transitive reduction (which is informally the minimal graph with the same connectivity as the original hierarchy) of both hierarchies and predicting inferred composition figure 1 : an example box embedding representation of the isa hierarchy where edges. as mentioned above, our model's shared learned features should allow for more generalization, and we test this by training on a subset of the transitive reduction, where we find we are able to outperform strong baselines. finally, we perform a detailed analysis of the model's capacity to predict compositional edges and transitive closure edges, both from an overfitting and generalization standpoint, identifying subsets where further improvement is needed. the source code for our model and the dataset can be found in https://github.com/ iesl/box-to-box-transform.git . ## related work recent advances in representing one single hierarchy mainly fall in two categories: 1) representing hierarchies in non-euclidian space (eg. hyperbolic space, due to the curvature's inductive bias to model tree-like structures) 2) using region-based representations instead of vectors for each node in the hierarchy @xcite . hyperbolic space has been shown to be efficient in representing hierarchical relations, but also encounters difficulties in training @xcite @xcite . categorization models in psychology often represent a concept as a region @xcite @xcite . @xcite and @xcite use gaussian distributions to embed each word in the corpus, the latter of which uses thresholded divergences which amount to region representations. @xcite and @xcite make use of the reverse product order on r n + , which effectively results in cone representations. @xcite further extend this cone representation to axis-aligned hyper-rectangles (or boxes), and demonstrate state-of-theart performance on modeling hierarchies. various training improvement methods for box embeddings have been proposed @xcite , the most recent of which, gumbelbox, use a latent noise model where box parameters are represented via gumbel distributions to improve on the loss landscape by making the gradient smooth for the geometric operations involved with box embeddings. region representations are also used for tasks which do not require modeling hierarchy. in @xcite , the authors also model conditional probability distributions using box embeddings. abboud et al. (2020) and ren et al. (2020 ) take a different approach, using boxes for their capacity to contain many vectors to provide slack in the loss function when modeling knowledge base triples or representing logical queries, respectively. ren et al. (2020) 2020 ), which our work is most similar to, represent joint hierarchies using box embeddings. however, they represent each concept with two boxes ignoring the internal semantics of the concepts. modeling joint hierarchies shares some similarities with knowledge base completion, however the goals of the two settings are different. when modeling joint hierarchies you are attempting to learn simultaneous transitive relations, and potentially learn relevant compositional edges involving these relations. for knowledge base completion, on the other hand, you may be learning many different relations, and primarily seek to recover edges which were removed rather than inferring new compositional edges. still, the models which perform knowledge base completion can be applied to this task, as the data can be viewed as knowledge base triples with only 2 relations. there have been multiple works that aim to build better knowledge representation @xcite ; trouillon @xcite @xcite . most relevant, @xcite recently proposed kg embedding methods which embeds entities in the poincaré ball model of hyperbolic space. these models are intended to capture relational patterns present in multi-relational graphs, with a particular emphasis on hierarchical relations. ## conclusion we proposed a box-to-box transformation that facilitates sharing of learned features across hierarchies when modeling joint hierarchies. we demonstrate the box-transform model is capable of achieving state-of-the-art performance compared with other strong baseline models when predicting compositional edges across a joint hierarchy. furthermore, the model also outperforms other models when modeling the transitive closure of each relation independently. in the future, we aim to extend the current model from two relations to multiple relations in order to obtain more generalization from hierarchical isa edges.