Concept-Based Label Embedding via Dynamic Routing for Hierarchical Text Classification (2021)

Abstract: Hierarchical Text Classification (HTC) is a challenging task that categorizes a textual description within a taxonomic hierarchy. Most of the existing methods focus on modeling the text. Recently, researchers attempt to model the class representations with some resources (e.g., external dictionaries). However, the concept shared among classes which is a kind of domain-specific and fine-grained information has been ignored in previous work. In this paper, we propose a novel concept-based label embedding method that can explicitly represent the concept and model the sharing mechanism among classes for the hierarchical text classification. Experimental results on two widely used datasets prove that the proposed model outperforms several state-of-the-art methods. We release our complementary resources (concepts and definitions of classes) for these two datasets to benefit the research on HTC.

## introduction text classification is a classical natural language processing (nlp) task. in the real world, the text classification is usually cast as a hierarchical text classification (htc) problem, such as patent collection @xcite , web content collection @xcite and medical record coding @xcite . in these scenarios, the htc task aims to categorize a textual description within a set of labels that are organized in a structured class hierarchy @xcite . lots of researchers devote their effort to investigate this challenging problem. they have proposed various htc solutions, which are usually categorized into flat @xcite , local @xcite , global @xcite and combined approaches @xcite . in most of the previous htc work, researchers mainly focus on modeling the text, the labels are simply represented as one-hot vectors @xcite . actually, the one-hot vectors act as ids without any semantic information. how to describe a class is also worthy of discussion. there is some work that embeds labels into a vector space which contains more semantic information. compared with one-hot representations, label embeddings have advantages in capturing domain-specific information and importing external knowledge. in the field of text classification (includes the htc task), researchers propose several forms of label embeddings to encode different kinds of information, such as 1) anchor points @xcite , 2) compatibility between labels and words @xcite @xcite , 3) taxonomic hierarchy @xcite and 4) external knowledge @xcite . although the external knowledge has been proven effective for htc, it comes from a dictionary or knowledge base that humans constructed for entity definition, and it doesn't focus on the class explanations of a certain htc task. in this sense, external knowledge is a type of domainindependent information. the taxonomic hierarchy encoding can capture the structural information of classes, which is a sort of domain-specific information for htc. however, actually it only models the hypernym-hyponym relations in the class hierarchy. the process is implicit and difficult to be interpreted. besides the structural connections between classes, we find that the information of concept shared between adjacent levels of classes is ignored by previous work. for instance, there is a parent node named "sports" in a concrete class hierarchy @xcite . its subclasses "surfing" and "swimming" are "water" related sports. the subclasses "basketball" and "football" are "ball" related sports. the "water" and "ball" are a type of abstract concept included in the parent class "sports" and can be shared by the subclasses. as shown in figure 1 , we have a similar observation in wos @xcite , which is a widely used public dataset (details in our experiments). the concept "design" of the parent class "computer science" is shared by the child classes "soft engineering" and "algorithm design". the concept "distributed" is shared by "network security" and "distributed computing". the concept information can help to group the classes and measure the correlation intensity between parent and child classes. compared with the information of node connections in the class hierarchy, the concept is more semantic and fine-grained, but rarely investigated. although @xcite have noticed the concept in htc, they define the concept in a latent way and the process of represent learning is also implicit. additionally, few of previous work investigates how to extract the concepts or model the sharing interactions among class nodes. to further exploit the information of concept for htc, we propose a novel concept-based label embedding method which can explicitly represent the concepts and model the sharing mechanism among classes. more specifically, we first construct a hierarchical attention-based framework which is proved to be effective by @xcite and @xcite . there is one conceptbased classifier for each level. the prior level classification result (i.e. predicted soft label embedding) is fed into the next level. a label embedding attention mechanism is utilized to measure the compatibility between texts and classes. then we design a concept sharing module in our model. it firstly extracts the concepts explicitly in the corpus and represents them in the form of embeddings. inspired by the capsnet @xcite , we employ the dynamic routing mechanism. the iterative routing helps to share the information from the lower level to the higher level with the agreement in capsnet. taking into account the characters of htc, we modify the dynamic routing mechanism for modeling the concepts sharing interactions among classes. in detail, we calculate the agreement between concepts and classes. an external knowledge source is taken as an initial reference of the child classes. different from the full connections in capsnet, we build routing only between the class and its own child classes to utilize the structured class hierarchy of htc. then the routing coefficients are iteratively refined by measuring the agreement between the parent class concepts embeddings and the child class embeddings. in this way, the module models the concept sharing process and outputs a novel label representation which is constructed by the concepts of parent classes. finally, our hierarchical network adopts such label embeddings to represent the input document with an attention mechanism and makes a classification. in summary, our major contributions include: ## model in this section, we detailedly introduce our model cled (figure 2 ). it is designed for hierarchical text classification with concept-based label embeddings via a modified dynamic routing mechanism. firstly, we construct a hierarchical attentionbased framework. then a concept sharing module is designed for extracting concepts and modeling the sharing mechanism among classes. the module learns a novel label representation with concepts. finally, the model takes the concept-based label embeddings to categorize a textual description. ## conclusion in this paper, we investigate the concept which is a kind of domain-specific and fine-grained information for the hierarchical text classification. we propose a novel concept-based label embedding model. compared with several competitive stateof-the-art methods, the experimental results on two widely used datasets prove the effectiveness of our proposed model. the visualization of the concepts and the learnt concept-based label embeddings re-