Joint Learning of Hyperbolic Label Embeddings for Hierarchical Multi-label Classification (2021)

Abstract: We consider the problem of multi-label classification, where the labels lie on a hierarchy. However, unlike most existing works in hierarchical multi-label classification, we do not assume that the label-hierarchy is known. Encouraged by the recent success of hyperbolic embeddings in capturing hierarchical relations, we propose to jointly learn the classifier parameters as well as the label embeddings. Such a joint learning is expected to provide a twofold advantage: i) the classifier generalises better as it leverages the prior knowledge of existence of a hierarchy over the labels, and ii) in addition to the label co-occurrence information, the label-embedding may benefit from the manifold structure of the input datapoints, leading to embeddings that are more faithful to the label hierarchy. We propose a novel formulation for the joint learning and empirically evaluate its efficacy. The results show that the joint learning improves over the baseline that employs label co-occurrence based pre-trained hyperbolic embeddings. Moreover, the proposed classifiers achieve state-of-the-art generalization on standard benchmarks. We also present evaluation of the hyperbolic embeddings obtained by joint learning and show that they represent the hierarchy more accurately than the other alternatives.

## introduction the problem of multi-label text classification is well known and extensively studied in literature @xcite @xcite . the fundamental assumption is that a document is associated with multiple labels from a fixed vocabulary of labels. often, these labels are organised in a hierarchical structure. for ex. consider a sample headline from the nyt (newyork * equal contribution times) corpus "voice recognition is improving, but don't stop the elocution lessons" for which labels are "top/news/technology". here, labels are arranged in a hierarchy, hereafter referred as label hierarchy. we undertake the task of labelling documents with classes that are hierarchically organised; this problem is popularly known as hierarchical multi-label text classification (hmc). hmc methods have found several applications in online advertising systems @xcite , bio-informatics @xcite ), text classification @xcite . the main challenge in hmc is in modelling classification of the document into a large, imbalanced and structured output space. in hmc, the label taxonomy is a partially ordered set (l, ≺) where l is a finite set of all class labels. relation ≺ refers to is-a relationship between labels, which is asymmetric, anti-reflexive and transitive @xcite . hierarchical structures can provide important insights for learning and classification tasks. however, explicit knowledge of hierarchy is not available in several domains, for instance, extreme classification datasets @xcite . in this paper, we consider the problem of structured prediction from unstructured text, in which label hierarchy is not known apriori. we infer hierarchies from classification judgements on the outputs that are readily available. we focus on discovering relationships between the labels in a hyperbolic space, which has natural capacity to encode hierarchical structures. in our approach, hidden (hyperbolic label embeddings for hierarchical multi-label classi-fication), the labels are represented in a hyperbolic space to help respect their latent hierarchical organisation. we use this intuition to learn label embeddings for hmc without explicit supervision on the label hierarchy. apart from employing hyperbolic embeddings, another key aspect of our methodology is that the parameters of the classifier as well as of the label embedding are learnt jointly. we next explain the advantage in doing so. in the absence of any partial information regarding the hierarchy, label embeddings are typically learnt using the weak supervision available in label co-occurrences @xcite . this weak form of supervision can be complemented if the label embedding learning is also aware of the manifold structure of the input (documents). for e.g., similar documents may have similar labels, etc. such a strengthening is possible only if learning happens in a joint fashion. moreover, the generalization of the classifier also improves because of the improved embeddings @xcite . our contributions can be summarised as follows: 1. we present an approach hidden, that models the implicit hierarchical organisation of labels for improved classification. it leverages properties of hyperbolic geometry to help learn embeddings for the hierarchically organised labels. 2. we present a novel formulation for jointly learning the parameters of the classifier as well as the label embedding, which can be trained solely using the supervision from the training data, and without using any explicit information regarding the label hierarchy. 3. we evaluate hidden on real-world as well as synthetic datasets and show: (a) significant improvement over classical multi-label classification methods as well as baselines that employ hyperbolic label embeddings learnt in isolation solely based on label co-occurrence information (b) hidden sometimes generalizes even better than state-of-the-art hierarchical multi-label classifiers that have complete access to the true label hierarchy (c) label embeddings learnt using the joint optimisation approach correlate better with the ground truth than other alternatives. ## related work several conventional classification methods are capable of handling classification in multi-label settings. however, relatively fewer of these are designed to incorporate the possibly hierarchical organisation of the class labels. these include both traditional methods @xcite as well as deep learning methods @xcite across varied domains such as news articles, web content, etc. some approaches @xcite have also attempted to identify a subset of class labels from the classification hierarchy that effectively represents most instances from the training dataset. traditional or flat classification approaches typically perform prediction assuming that all the classes are independent of each other, ignoring the class hierarchy. whereas 'local' classification approaches @xcite train a set of classifiers at each level of the hierarchy. however, it has also been argued @xcite that it is impractical to train separate classifiers at each level. on the other hand, 'global' approaches @xcite ) train a single classifier that factors in the complete class hierarchy, while often also explicitly factoring in the label-label correlation @xcite . unlike the local approach, 'global' approaches do not suffer from the error propagation problem, although they are prone to under-fit by not considering local information in the hierarchy. some recent papers have proposed a mix of local and global approaches for hmc. @xcite propose an objective that leverages both local and global information while introducing global hierarchical violation penalty. @xcite employ a reinforcement learning framework to learn a label assignment policy. they model hmc as a markov decision process, wherein, the agent takes an action of label assignment on the tree hierarchy and receives scalar rewards as feedback for the actions. @xcite embed both document and label hierarchy in the same hyperbolic space and use interactions between these embeddings for hmc. our approach differs from these in two important ways: (i) we embed only labels into the hyperbolic space and (ii) label hierarchy is not known apriori -all we assume is that there is some hidden hierarchy. recently, the use of hyperbolic geometry has been found to be promising in machine learning and network sciences to model data with latent hierarchies. @xcite showed that properties of complex networks, namely heterogeneous degree distribution and strong clustering, naturally manifest in hyperbolic geometry. they showed that if a network has some heterogeneous degree distribution and metric structure, the network can be mapped effectively to the hyperbolic space (since euclidean distance has limitations in approximating the distance between nodes in a tree). @xcite have shown that any finite tree structure can be embedded into a finite hyperbolic space while preserving the distance between nodes. @xcite learnt hierarchical representations of symbolic data by embedding them into an n-dimensional poincaré ball by leveraging the distance property of hyperbolic spaces. instead of relying on the true hierarchy to learn embeddings, @xcite inferred hierarchies from real-valued similarity scores using the lorentz model of hyperbolic geometry. we used a similar formulation in our model hidden to build our hmc model by leveraging the co-occurrence count of labels for each document, but additionally (and more importantly), in a joint manner, learn the parameters of the classifier. ## hyperbolic geometry & the poincaré model in this section, we give an overview of hyperbolic geometry and the poincaré model for embedding in hyperbolic spaces @xcite . a hyperbolic space is a non-euclidean riemannian manifold of constant negative curvature. though there are several fundamental differences between the euclidean and the hyperbolic geometry, the most interesting characteristic of hyperbolic spaces is their ability to naturally represent hierarchical relations @xcite . in the poincaré ball model, which is one of the standard models for hyperbolic geometry, the euclidean distances between equidistant points, according to the inherent manifold metric d, fall exponentially as one moves from origin towards the surface of the ball. this interesting property is the key for enabling learning of continuous embeddings of hierarchies. for example, one can imagine root node of hierarchy at origin and leaf nodes near the ball's surface. then, this model can easily accommodate exponentially growing number of equidistant siblings at deeper levels of the hierarchy. whereas, such an accommodation is not possible using euclidean geometry. below we provide some details of this model. let b @xmath0, where . is the euclidean 2 norm. the poincaré ball model is a riemannian manifold (b n , g x ), the open unit ball equipped with the riemannian metric tensor the euclidean metric tensor. the geodesic distance between two points u, @xmath1 ## problem formulation and approach in this section, we present present details of our model, training, as well as inference. ## experiments in this section, we compare our approach against the baseline models and other state-of-the-art hmc approaches. first we describe the evaluation metrics and illustrating results in a synthetic setting. ## conclusion we propose a novel approach to hierarchical multilabel classification based on joint learning of document classifier and label embeddings in hyperbolic space. with state-of-the-art hierarchy aware methods. we performed extensive experiments on three datasets and demonstrate effectiveness of the learned embeddings. the poincaré model and the lorentz model are equivalent in isometry. therefore points in lorentz manifold can be mapped into poincaré ball as, p : a point x in the euclidean space r n can be projected onto the lorentz manifold h n using the transformation ω(x) = 1 + x 2 , x . this transformation ensures that the minkowski innerproduct, ω(x), ω(x) l , is equal to -1, and that the first component, ω(x) 0 ≡ 1 + x 2 is positive, as required for membership in the lorentz manifold. now using the isometry between poincaré and lorentz models @xcite , we have π : r @xmath2 we describe the details of the datasets used in our experiments. for rcv1 dataset @xcite , we use the original training/test split and use 10% of the training set as the validation set. we introduce an extra root label in addition to the 103 labels present in the dataset. each document in the dataset is labelled with this label. the details for the other datasets used are same as in @xcite and we refer the readers to the same.