A Fully Hyperbolic Neural Model for Hierarchical Multi-Class Classification (2020)

Abstract: Label inventories for fine-grained entity typing have grown in size and complexity. Nonetheless, they exhibit a hierarchical structure. Hyperbolic spaces offer a mathematically appealing approach for learning hierarchical representations of symbolic data. However, it is not clear how to integrate hyperbolic components into downstream tasks. This is the first work that proposes a fully hyperbolic model for multi-class multi-label classification, which performs all operations in hyperbolic space. We evaluate the proposed model on two challenging datasets and compare to different baselines that operate under Euclidean assumptions. Our hyperbolic model infers the latent hierarchy from the class distribution, captures implicit hyponymic relations in the inventory, and shows performance on par with state-of-the-art methods on fine-grained classification with remarkable reduction of the parameter size. A thorough analysis sheds light on the impact of each component in the final prediction and showcases its ease of integration with Euclidean layers.

## introduction entity typing classifies textual mentions of entities, according to their semantic class, within a set of labels (or classes) organized in an inventory. the task has progressed from recognizing a few coarse classes @xcite , to extremely large inventories, with hundreds @xcite or thousands of labels @xcite . therefore, exploiting inter-label correlations has become critical to improve performance. large inventories tend to exhibit a hierarchical structure, either by an explicit tree-like arrangement of the labels (coarse labels at the top, finegrained at the bottom), or implicitly through the figure 1 : tree embedded in hyperbolic space. items at the top of the hierarchy are placed near the origin of the space, and lower items near the boundary. moreover, the hyperbolic distance (eq. 1) between sibling nodes resembles the one through the common ancestor, analogous to the distance in the tree. that is in this work, we propose a fully hyperbolic neural model for fine-grained entity typing. noticing a perfect match between hierarchical label inventories in the linguistic task and the benefits of hyperbolic spaces, we endow a classification model with a suitable geometry to capture this fundamental property of the data distribution. by virtue of the hyperbolic representations, the proposed approach automatically infers the latent hierarchy arising from the class distribution and achieves a meaningful and interpretable organization of the label space. this arrangement captures implicit hyponymic @xcite in the inventory and enables the model to excel at fine-grained classification. to the best of our knowledge, this work is the first to apply hyperbolic geometry from beginning to end to perform multi-label classification on real nlp datasets. recent work has proposed hyperbolic neural components, such as word embeddings @xcite , recurrent neural networks @xcite and attention layers @xcite . however, researchers have incorporated these isolated components into neural models, whereas the rest of the layers and algorithms operate under euclidean assumptions. this impedes models from fully exploiting the properties of hyperbolic geometry. furthermore, there are different analytic models of hyperbolic space, and not all previous work operates in the same one, which hinders their combination, and hampers their adoption for downstream tasks (e.g. @xcite learn embeddings in the poincaré model, @xcite aggregate points in the klein model, or @xcite perform optimization in the lorentz model). we address these issues. our model encodes textual inputs, applies a novel attention mechanism, and performs multi-class multilabel classification, executing all operations in the poincaré model of hyperbolic space ( §4). we evaluate the model on two datasets, namely ultra-fine @xcite and ontonotes @xcite , and compare to euclidean baselines as well as to state-of-the-art methods for the task @xcite . the hyperbolic system has competitive performance when compared to an elmo model @xcite and a bert model @xcite on very fine-grained types, with remarkable reduction of the parameter size ( §6). instead of relying on large pre-trained models, we impose a suitable inductive bias by choosing an adequate metric space to embed the data, which does not introduce extra burden on the parameter footprint. by means of the exponential and logarithmic maps (explained in §2) we are able to mix hyperbolic and euclidean components into one model, aiming to exploit their strengths at different levels of the representation. we perform a thorough ablation that allows us to understand the impact of each hyperbolic component in the final performance of the system ( §6.1.1 and §6.1.2), and showcases its ease of integration with euclidean layers. ## hyperbolic neural networks in this section we briefly recall the necessary background on hyperbolic neural components. the terminology and formulas used throughout this work follow the formalism of möbius gyrovector spaces @xcite , and the definitions of hyperbolic neural components of @xcite . for more information about riemannian geometry and möbius operations see appendix a and b. in the following, •, • and • are the inner product and norm inherited from the euclidean space. hyperbolic space: it is a non-euclidean space with constant negative curvature. we adopt the poincaré ball model of hyperbolic space @xcite . in the general n-dimensional case, it becomes möbius matrix-vector multiplication: given a linear map m : r @xmath0, which we identify with its matrix representation, and a point @xmath1, m @xmath2, it is defined as: pointwise non-linearity: if we model it as ϕ : r @xmath3, then its möbius version ϕ ⊗ can be applied using the same formulation of the matrixvector multiplication. a visualization of the aforementioned operations can be seen in figure 2 . by combining these operations we obtain a onelayer feed-forward neural network (ffnn) in hyperbolic space, described as @xmath4. note that the parameter b lies in the hyperbolic space, thus its updates during training need to be corrected for this geometry. exponential and logarithmic maps: for each point @xmath5, let t x d n denote the associated tangent space, which is always a subset of euclidean space @xcite . we make use of the exponential map exp x : t x d @xmath6: d @xmath7, and viceversa. at the origin of the space, they are given for @xmath8: to map a point @xmath9. conversely, to map a point @xmath10, we assume r @xmath11. this allows to mix hyperbolic and euclidean neural layers as shown in §6.1.2. ## fine-grained entity typing given a context sentence s containing an entity mention m, the goal of entity typing is to predict the correct type labels t m that describe m from a type inventory t . the ground-truth type set t m may contain multiple types, making the task a multiclass multi-label classification problem. for fine-grained entity typing the type inventory t tends to contain hundreds to thousands of labels. encoding hierarchical information from large type inventories has been proven critical to improve performance @xcite . thus we hypothesize that our proposed hyperbolic model will benefit from this representation. ## hyperbolic classification model in this section we propose a general hyperbolic neural model for classification with sequential data as input. the building blocks are defined in a generic manner such that they can be applied to different tasks, or integrated with regular euclidean layers. our proposed architecture resembles recent neural models applied to entity typing @xcite . for the encoders we employ the neural networks introduced in @xcite , we propose a novel attention mechanism operating entirely in the poincaré model, and we extend the hyperbolic classifier to multi-class multi-label setups. an overview of the model can be seen in figure 3 . ## experiments we evaluate the proposed hyperbolic model on two different datasets for fine-grained entity typing, and compare to euclidean baselines as well as state-ofthe-art models. ## results and discussion following previous work @xcite , we report results on the devel-opment set in the reduction of the parameter size is also remarkable: 70% and 91% versus denoised and bert respectively. this emphasizes the importance of choosing a suitable metric space that fits the data distribution (hierarchical in this case) as a powerful and efficient inductive bias. through adequate tools and formulations, we are able to exploit this bias without introducing an overload correspondence of results between hy base and labelgcn suggest that both models capture similar information. labelgcn requires label co-occurrence statistics represented as a weighted graph, from where a hierarchy can be easily derived @xcite . the similarity of results indicates that the hyperbolic model is able to implicitly encode the latent hierarchical information in the label co-occurrences without additional inputs or the burden of the graph layer. to shed light on this aspect, we inspect the points p k learned by hy base to define the hyperplanes of equation 10 . ## related work type inventories for the task of fine-grained entity typing @xcite have grown in size and complexity @xcite . researchers have tried to incorporate hierarchical information on the type distribution in different manners @xcite . @xcite encode the hierarchy through a sparse matrix. xu ## conclusions incorporating hierarchical information from the label inventory into neural models has become critical to improve performance. hyperbolic spaces are an exciting approach since they are naturally equipped to model hierarchical structures. however, previous work integrated isolated components into neural systems. in this work we propose a fully hyperbolic model and showcase its effectiveness on challenging datasets. our hyperbolic model automatically infers the latent hierarchy from the class distribution, captures implicit hyponymic relations in the inventory and achieves a performance comparable to state-of-the-art systems on very finegrained labels with a remarkable reduction of the parameter size. this emphasizes the importance of choosing a metric space suitable to the data distribution as an effective inductive bias to capture fundamental properties, such as hierarchical structure. moreover, we illustrate ways to integrate different components with euclidean layers, showing their strengths and drawbacks. an interesting future direction is to employ hyperbolic representations in combination with contextualized word embeddings. we release our implementation with the aim to ease the adoption of hyperbolic components into neural models, yielding lightweight and efficient systems.