Hyperbolic Capsule Networks for Multi-Label Classification (2020)

Abstract: Although deep neural networks are effective at extracting high-level features, classification methods usually encode an input into a vector representation via simple feature aggregation operations (e.g. pooling). Such operations limit the performance. For instance, a multi-label document may contain several concepts. In this case, one vector can not sufficiently capture its salient and discriminative content. Thus, we propose Hyperbolic Capsule Networks (HyperCaps) for Multi-Label Classification (MLC), which have two merits. First, hyperbolic capsules are designed to capture fine-grained document information for each label, which has the ability to characterize complicated structures among labels and documents. Second, Hyperbolic Dynamic Routing (HDR) is introduced to aggregate hyperbolic capsules in a label-aware manner, so that the label-level discriminative information can be preserved along the depth of neural networks. To efficiently handle large-scale MLC datasets, we additionally present a new routing method to adaptively adjust the capsule number during routing. Extensive experiments are conducted on four benchmark datasets. Compared with the state-of-the-art methods, HyperCaps significantly improves the performance of MLC especially on tail labels.

## introduction the main difference between multi-class classification (mcc) and multi-label classification (mlc) is that datasets in mcc have only serval mutually exclusive classes, while datasets in mlc contain much more correlated labels. mlc allows label co-occurrence in one document, which indicates that the labels are not disjointed. in addition, a large fraction of the labels are the infrequently occurring tail labels @xcite , which is also referred as the power-law label distribution. figure 1 illustrates the label distribution of eur-lex57k @xcite . a multi-label document usually has serval head and tail labels, and hence contain several concepts about both its head and tail labels simultaneously. recent works for text classification, such as cnn-kim @xcite and fasttext @xcite , focus on encoding a document into a fixed-length vector as the distributed document representation @xcite . these encoding based deep learning methods use simple operations (e.g. pooling) to aggregate features extracted by neural networks and construct the document vector representation. a fully-connected (fc) layer is usually applied upon the document vector to predict the probability of each label. and each row in its weight matrix can be interpreted as a label vector representation @xcite . in this way, the label probability can be predicted by computing the dot product between label and document vectors, which is proportional to the scalar projection of the label vector onto the document vector as shown in figure 2 . for example, label "movie" should have the largest scalar projection onto a document about "movie". however, even the learned label representation of "music" can be distinguished from "movie", it may also have a large scalar projection onto the document. moreover, multi-label documents always contain several concepts about multiple labels, such as a document about "sport movie". whereas the document vector representation is identical to all the labels, and training instances for tail labels are inadequate compared to head labels. the imbalance between head and tail labels makes it hard for the fc layer to make prediction, especially on tail labels. in this case, one vector can not sufficiently capture its salient and discriminative content. therefore, the performance of constructing the document vector representation via simple aggregation operations is limited for mlc. capsule networks @xcite has recently proposed to use dynamic routing in place of pooling and achieved better performance for classification tasks. in fact, capsules are fine-grained features compared to the distributed document representation, and dynamic routing is a label-aware feature aggregation procedure. @xcite improves the scalability of capsule networks for mlc. however, they only use cnn to construct capsules, which capture local contextual information @xcite . effectively learning the document information about multiple labels is crucial for mlc. thus we propose to connect cnn and rnn in parallel to capture both local and global contextual information, which would be complementary to each other. nevertheless, euclidean capsules necessitate designing a non-linear squashing function. inspired by the hyperbolic representation learning methods which demonstrate that the hyper-bolic space has more representation capacity than the euclidean space @xcite , hyperbolic capsule networks (hypercaps) is proposed. capsules are constrained in the hyperbolic space which does not require the squashing function. hyperbolic dynamic routing (hdr) is introduced to aggregate hyperbolic capsules in a label-aware manner. moreover, in order to fit the large label set of mlc and improve the scalability of hypercaps, adaptive routing is presented to adjust the number of capsules participated in the routing procedure. the main contributions of our work are therefore summarized as follows: ## preliminaries in order to make neural networks work in the hyperbolic space, formalism of the möbius gyrovector space is adopted @xcite ). an n-dimensional poincaré ball b n is a riemannian manifold defined as b @xmath0, with its tangent space around @xmath1:= 2 1-p 2 . the exponential map exp p : to work with hyperbolic capsules, möbius operations in the poincaré ball also need to be formulated. möbius scalar multiplication for @xmath2 and k ⊗ @xmath3. the definition of möbius matrix-vector multiplication for @xmath4 and m ⊗ @xmath5. hdr is developed based on these operations. ## local and global hyperbolic capsules neural networks are generally used as effective feature extractors for text classification. kernels of cnn can be used to capture local n-gram contextual information at different positions of a text sequence, while hidden states of rnn can represent global long-term dependencies of the text @xcite . hence, we propose to obtain the combination of local and global hyperbolic capsules by connecting cnn and rnn in parallel, which would be complementary to each other. given a text sequence of a document with t word tokens @xmath6, . . . , x t ], pre-trained wdimensional word embeddings (e.g. glove @xcite ) are used to compose word vector representations @xmath7, . . . , e t ] ∈ r t ×w , upon which cnn and rnn connected in parallel are used to construct local and global hyperbolic capsules in the poincaré ball. figure 3 illustrates the framework for hypercaps. ## hyperbolic dynamic routing the purpose of hyperbolic dynamic routing (hdr) is to iteratively aggregate local and global hyperbolic capsules into label-aware hyperbolic capsules, whose activations stand for probabilities of the labels. ## experiments the proposed hypercaps is evaluated on four benchmark datasets with various label number from 54 to 4271. we compare with the state-of-the-art methods in terms of widely used metrics. performance on tail labels is also compared to demonstrate the superiority of hypercaps for mlc. an ablation test is also carried out to analyse the contribution of each component of hypercaps. ## conclusion we present the hyperbolic capsule networks (hypercaps) with hyperbolic dynamic routing (hdr) and adaptive routing for multi-label classification (mlc). the proposed hypercaps takes advantage of the parallel combination of finegrained local and global contextual information and label-aware feature aggregation method hdr to dynamically construct label-aware hyperbolic capsules for tail and head labels. adaptive routing is additionally applied to improve the scalability of hypercaps by controlling the number of capsules during the routing procedure. extensive experiments are carried out on four benchmark datasets. results compared with the state-of-the-art methods demonstrate the superiority of hypercaps, especially on tail labels. as recent works explore the superiority of hyperbolic space to euclidean space for serval natural language processing tasks, we intend to couple with the hyperbolic neural networks @xcite and the hyperbolic word embedding method such as poincar églove @xcite in the future.