Exploiting Global and Local Hierarchies for Hierarchical Text Classification (2022)

Abstract: Hierarchical text classification aims to leverage label hierarchy in multi-label text classification. Existing methods encode label hierarchy in a global view, where label hierarchy is treated as the static hierarchical structure containing all labels. Since global hierarchy is static and irrelevant to text samples, it makes these methods hard to exploit hierarchical information. Contrary to global hierarchy, local hierarchy as a structured labels hierarchy corresponding to each text sample. It is dynamic and relevant to text samples, which is ignored in previous methods. To exploit global and local hierarchies, we propose Hierarchy-guided BERT with Global and Local hierarchies (HBGL), which utilizes the large-scale parameters and prior language knowledge of BERT to model both global and local hierarchies. Moreover, HBGL avoids the intentional fusion of semantic and hierarchical modules by directly modeling semantic and hierarchical information with BERT. Compared with the state-of-the-art method HGCLR, our method achieves significant improvement on three benchmark datasets.

## introduction hierarchical text classification (htc) focuses on assigning one or more labels from the label hierarchy to a text sample @xcite . as a special case of multi-label text classification, htc has various applications such as news categorization @xcite and scientific paper classification @xcite . the methods in htc aim to improve prediction accuracy by modeling the large-scale, imbalanced, and structured label hierarchy @xcite . to model the label hierarchy, recent methods @xcite @xcite view hierarchy as a directed acyclic graph and model label hierarchy based on graph encoders. however, the input of graph encoders is static, considering that all htc text samples share the same hierarchical structure, which leads graph encoders to model the same graph redundantly. to solve this problem, @xcite directly discards the graph encoder during prediction, but this method still suffers from the same problem during training. moreover, since the target labels corresponding to each text sample could be either a single-path or a multi-path in htc @xcite , recent methods only consider the graph of the entire label hierarchy and ignore the subgraph corresponding to each text sample. this subgraph can contain structured label co-occurrence information. for instance, a news report about france travel is labeled "european" under the parent label "world" and "france" under a different parent label "travel destinations". there is a strong correlation between the labels "france" and "european". but these labels are far apart on the graph, making it difficult for graph encoders to model this relationship. under such observation, we divide the label hierarchy into global and local hierarchies to take full advantage of hierarchical information in htc. we define global hierarchy as the whole hierarchical structure, referred to as hierarchical information in previous methods. then we define local hierarchy as a structured label hierarchy corresponding to each text sample, which is the subgraph of global hierarchy. moreover, global hierarchy is static and irrelevant to text samples, while local hierarchy is dynamic and relevant to text samples. considering the characteristics of two hierarchies, our method models them separately to avoid redundantly modeling static global hierarchy and fully exploit hierarchical information with dynamic local hierarchy. to model semantic information along with hierarchical information, @xcite proposes hierarchy-aware multi-label attention. @xcite reformulates it as a matching problem by encouraging the text representation to be similar to its label representation. although, these methods can improve the performance of text encoders by injecting label hierarchy with the graph encoder , the improvement on pretrained language model bert @xcite is limited @xcite . compared to previous text encoders such as cnn or rnn, bert has large-scale parameters and prior language knowledge. it enables bert to roughly grasp hierarchical information with multi-label text classification. therefore, hg-clr @xcite is proposed to improve the bert performance on htc, and the hierarchy is embedded into bert based on contrastive learning during training. for prediction, hgclr directly uses bert as a multi-label classifier. specifically, the hierarchy in hgclr is represented by positive samples in contrastive learning, which is implemented by scaling the bert token embeddings based on a graph encoder. however, representing hierarchy by simply scaling token embeddings is inefficient, which may also lead to a gap between training and prediction. to efficiently exploit bert in htc, we leverage the prior knowledge of bert by transforming both global and local hierarchy modeling as mask prediction tasks. moreover, we discard the auxiliary graph encoder and utilize bert to model hierarchical information to avoid the intentional fusion of bert and graph encoder. for global hierarchy, we propose a label mask prediction task to recover masked labels based on the label relationship in global hierarchy. since global hierarchy is irrelevant to text samples, we only fine-tune label embeddings and keep bert frozen. for local hierarchy, we combine text samples and labels as the input of bert to directly fuse semantic and hierarchical information according to the attention mechanism in bert. in summary, the contributions of this paper are following: ## related work hierarchical text classification (htc) is a special multi-label text classification problem that requires constructing one or more paths from the taxonomic hierarchy in a top-down manner @xcite . compared to multi-label text classification, htc focuses on leveraging hierarchical information to achieve better results. there are two groups of existing htc methods based on treating the label hierarchy: local and global approaches. the local approaches leverage hierarchical information by constructing one or more classifiers at each level or each node in hierarchy. generally speaking, a text sample will be classified top-down according to its hierarchy. @xcite applies a cnn with a fine-tuning technique to utilize the data in the upper levels. @xcite initials parameters of the child classifier by the finetuned parent classifiers. the global approaches leverage hierarchical information by directly treating htc as multi-label text classification with hierarchy information as input. many methods like recursive regularization @xcite , reinforcement learning @xcite , capsule network @xcite , and meta-learning @xcite has been proposed to capture hierarchical information. to better represent hierarchical information, @xcite formulates the hierarchy as a directed graph and introduces hierarchy-aware structure encoders. @xcite formulates the text-label semantics relationship as a semantic matching problem. with the development of pretrained language model (plm), plm outperforms previous methods even without using hierarchical information. compared to text encoders like rnn or cnn, plm is strong enough to learn hierarchical information without hierarchy-aware structure encoders. under this observation, @xcite proposes hgclr to embed the hierarchical information into the plm directly. however, hgclr still requires the hierarchy-aware structure encoder like graphormer @xcite to incorporate hierarchical information during training. ## problem definition given a training set {(x i , y i )} n @xmath0, and y @xmath1, 1} l is the label of x i represented by l dimensional multi-hot vector. the goal of hierarchical text classification (htc) is to predict a subset labels for x i with the help of hierarchical information, which can be organized as although labels in y i follow the labels hierarchy, htc could be either a single-path or a multi-path problem @xcite . ## methodology in this section, we provide the technical details of the proposed hbgl. figure 1 shows the overall framework of the model. the left part corresponds to global hierarchy-aware label embeddings, while the right part corresponds to local hierarchy-aware text encoder. we first inject global hierarchy into label embeddings in global hierarchy-aware label embeddings. then we leverage these label embeddings with our local hierarchy-aware text encoder. ## conclusion in this paper, we proposed a bert-based htc framework hbgl. hbgl avoids the intentional fusion of semantic and hierarchical modules by utilizing bert to model both semantic and hierarchical information. moreover, hbgl takes full advantage of hierarchical information by modeling global and local hierarchies, respectively. considering that global hierarchy is static and irrelevant to text samples, we propose global hierarchy-aware label embeddings to inject global hierarchy into label embeddings directly. considering that local hierarchy is dynamic and relevant to text samples, we propose local hierarchy-aware text encoder to deeply combine semantic and hierarchical information according to the attention mechanism in bert. compared to existing methods, hbgl achieves significant improvements on all three datasets, while only parameters corresponding to label embeddings are required except bert.