Hierarchy-Aware Global Model for Hierarchical Text Classification (2020)

Abstract: Hierarchical text classification is an essential yet challenging subtask of multi-label text classification with a taxonomic hierarchy. Existing methods have difficulties in modeling the hierarchical label structure in a global view. Furthermore, they cannot make full use of the mutual interactions between the text feature space and the label space. In this paper, we formulate the hierarchy as a directed graph and introduce hierarchy-aware structure encoders for modeling label dependencies. Based on the hierarchy encoder, we propose a novel end-to-end hierarchy-aware global model (HiAGM) with two variants. A multi-label attention variant (HiAGM-LA) learns hierarchy-aware label embeddings through the hierarchy encoder and conducts inductive fusion of label-aware text features. A text feature propagation model (HiAGM-TP) is proposed as the deductive variant that directly feeds text features into hierarchy encoders. Compared with previous works, both HiAGM-LA and HiAGM-TP achieve significant and consistent improvements on three benchmark datasets.

## introduction text classification is widely used in natural language processing (nlp) applications, such as sentimental analysis @xcite , information retrieval @xcite , and document categorization @xcite . hierarchical text classification (htc) is a particular multi-label text classification (mlc) problem, where the classification result corresponds to one or more nodes of a taxonomic hierarchy. the taxonomic hierarchy is commonly modeled as a tree or a directed acyclic graph, as depicted in figure 1 . existing approaches for htc could be categorized into two groups: local approach and global figure 1 : this short sample is tagged with news, sports, football, features and books. note that htc could be either a single-path or a multi-path problem. in this paper, we formulate the hierarchy as a directed graph and utilize prior probabilities of label dependencies to aggregate node information. a hierarchy-aware global model (hiagm) is pro-posed to enhance textual information with the label structural features. it comprises a traditional text encoder for extracting textual information and a hierarchy-aware structure encoder for modeling hierarchical label relations. the hierarchy-aware structure encoder could be either a treelstm or a hierarchy-gcn where hierarchical prior knowledge is integrated. moreover, these two structure encoders are bidirectionally calculated, allowing them to capture label correlation information in both top-down and bottom-up manners. as a result, hiagm is more robust than previous top-down models and is able to alleviate the problems caused by exposure bias and imbalanced data. to aggregate text features and label structural features, we present two variants of hiagm, a multi-label attention model hiagm-la and a text feature propagation model hiagm-tp. both variants extract hierarchy-aware text features based on the structure encoders. hiagm-la extracts the inductive label-wise text features while hiagm-tp generates hybrid information in a deductive manner. specifically, hiagm-la updates the label embedding across the holistic hierarchy and then employs node outputs as the hierarchy-aware label representations. finally, it conducts multi-label attention for label-aware text features. on the other hand, hiagm-tp directly utilizes text features as the input of the structure encoder in a serial dataflow. hence it propagates textual information throughout the overall hierarchy. the hidden state of each node in the entire hierarchy represents the class-specific textual information. the major contributions of this paper are: existing works for htc could be categorized into local and global approaches. local approaches could be subdivided into local classifier per node (lcn) (banerjee et al., 2019), local classifier per parent node (lcpn) @xcite , and local classifier per level (lcl) @xcite @xcite . the global approach improves flat mlc models with the hierarchy information. @xcite modifies svm to hierarchical-svm by decomposition. @xcite proposes a simple recursive regularization of parameters among adjacent classes. deep learning architectures are also employed in global models, such as sequence-to-sequence @xcite , metalearning @xcite , reinforcement learning @xcite , and capsule network @xcite . those models mainly focus on improving decoders based on the constraint of hierarchical paths. in contrast, we propose an effective hierarchy-aware global model, hiagm, that extracts label-wise text features with hierarchy encoders based on prior hierarchy information. moreover, the attention mechanism is introduced in mlc by @xcite for icd coding. @xcite trains label representation through basic graphcnn and conducts mutli-label attention with residual shortcuts. at-tentionxml @xcite converts mlc to a multi-label attention lcl model by label clusters. @xcite improves hmcn @xcite with label attention per level. our hiagm-la, however, employs multi-label attention in a single model with a simplified structure encoder, reducing the computational complexity. recent works, in semantic analysis @xcite , semantic role labeling @xcite and machine translation @xcite ), shows the improvement on sentence representation of syntax encoder, such as tree-based rnn @xcite and graphcnn @xcite . we modify those structure encoders for htc with fine-grained prior knowledge in both top-down and bottom-up manners. ## problem definition hierarchical text classification (htc), a subtask of text classification, organizes the label space with a predefined taxonomic hierarchy. the hierarchy is predefined based on holistic corpus. the hierarchy groups label subsets according to class relations. the taxonomic hierarchy mainly contains the treelike structure and the directed acyclic graph (dag) structure. note that dag can be converted into a tree-like structure by distinguishing each label node as a single-path node. thus, the taxonomic hierarchy can be simplified as a tree-like structure. as illustrated in figure 2 , we formulate a taxonomic hierarchy as a directed graph @xmath0, -→ e , ← -e ) where v refers to the set of label nodes @xmath1, v 2 , . . . , v c } and c denotes the number of label nodes. as depicted in figure 1 , each sample x i corresponds to a label set l i that includes multiple classes. those corresponding classes belong to either one or more sub-paths in the hierarchy. note that the sample belongs to the parent node v i in the condition pertaining to the child node v @xmath2. ## hierarchy-aware global model as depicted in figure 3 , we propose a hierarchy-aware global model (hiagm) that leverages the fine-grained hierarchy information and then aggregates label-wise text features. hiagm consists of a traditional text encoder for textual information and a hierarchy-aware structure encoder for hierarchical label correlation features. we present two variants of hiagm for hybrid information aggregation, a multi-label attention model (hiagm-la) and a text feature propagation model (hiagm-tp). hiagm-la updates label representations with the structure encoder and generates label-aware text features with multi-label attention mechanism. hiagm-tp propagates text representations throughout the holistic hierarchy, thus obtaining label-wise text features with the fusion of label correlations. ## experiment in this section, we introduce our experiments with datasets, evaluation metrics, implementation details, comparison, ablation study, and analysis of experimental results. ## conclusion in this paper, we propose a novel end-to-end hierarchy-aware global model that extracts the label structural information for aggregating label-wise text features. we present a bidirectional treel-stm and a hierarchy-gcn as the hierarchy-aware structure encoder. furthermore, our framework is extended into a parallel variant based on multilabel attention and a serial variant of text feature propagation. our approaches empirically achieve significant and consistent improvement on three distinct datasets, especially on the low-frequency labels. specifically, both variants outperform the state-of-the-art model on the rcv1-v2 benchmark dataset. and our best model obtains a macro-f1 score of 63.35% and a micro-f1 score of 83.96%.