Peer-Label Assisted Hierarchical Text Classification (2023)

Abstract: Hierarchical text classification (HTC) is a challenging task, in which the labels of texts can be organized into a category hierarchy. To deal with the HTC problem, many existing works focus on utilizing the parent-child relationships that are explicitly shown in the hierarchy. However, texts with a category hierarchy also have some latent relevancy among labels in the same level of the hierarchy. We refer to these labels as peer labels, from which the peer effects are originally utilized in our work to improve the classification performance. To fully explore the peer-label relationship, we develop a PeerHTC method. This method innovatively measures the latent relevancy of peer labels through several metrics and then encodes the relevancy with a Graph Convolutional Neural Network. We also propose a sample importance learning method to ameliorate the side effects raised by modelling the peer label relevancy. Our experiments on several standard datasets demonstrate the evidence of peer labels and the superiority of PeerHTC over other state-of-the-art HTC methods in terms of classification accuracy.

## introduction hierarchical text classification (htc) is a multilabel text classification problem which aims to classify texts into categories that can be organized into a taxonomic hierarchy. it is an important problem in natural language processing and has attracted increasing attention in both industrial and academic fields. typical htc problems include patent categorization @xcite , medical record coding @xcite , and product categorization @xcite . due to the complexity of category hierarchy, the problem of hierarchical text classification is more challenging than plain text classification. the parent-child relationships between categories in adjacent levels of the hierarchy are usually defined in advance. then a natural way to solve the htc problem is to incorporate this prior knowledge into the model, i.e., making the model aware of the hierarchy. building "hierarchy-aware" models is beneficial for htc, which is particularly true for categories with few samples. therefore, it has long been the main focus in htc to figure out the most effective way of utilizing the category hierarchy to improve the classification performance. in the past literature, existing approaches for htc can be generally categorized into three groups: local approaches, global approaches, and local-global-combined ones (also known as hybrid approaches). the local approaches train local classifiers for every child label, every parent label or every level in the hierarchy @xcite . the parameters of local classifiers are initialized in a top-down fashion according to the category hierarchy. however, these approaches usually contain a large number of parameters, and the whole hierarchy cannot be fully captured merely by parameter initialization. global approaches, which are popular in recent years, aim to flatten htc into a multi-label classification problem, and then incorporate the information of category hierarchy in various ways, such as using regularization terms @xcite , modeling the architecture of category hierarchy @xcite , and using contrastive learning @xcite . the local-global-combined approaches can be seen as an improvement on local approaches, which construct the information flow between local classifiers in more effective ways, and meanwhile utilize a global classifier to coordinate local ones @xcite . however, these models might still suffer from error propagation @xcite , because the classification of child layers are dependent on that of their parents. to the best of our knowledge, existing methods only exploit category relevancy that is explicitly reflected in the hierarchy. for example, @xcite used a recursive regularization term in which the parameters of parent labels are expected to be similar to those of their children. @xcite used a structure encoder for labels, in which the information of parent and child labels are integrated into each label's representation. however, there could still exist some latent relevancy among the labels in the same level, which could also be beneficial to the htc problem. take the blurb-genrecollection dataset @xcite as an example, which consists of descriptions and genres of books. in this dataset, two third-level book categories "world history" and "travel: asia" belong to different second-level categories "history" and "travel", respectively. however, these two thirdlevel categories both involve geographical and cultural contents. therefore, intuitively they should share some common characteristics, and the classification of one category could benefit that of the other. the phenomenon that labels in the same level possess latent relevancy is similar to the "peer effect" existing among peer friends. thus we call these labels as "peer labels" in this work. to utilize the latent relevancy of peer labels to improve the htc performance, we develop a peer-htc method. it incorporates two types of label relationships: the parent-child relationship explicitly reflected in the hierarchy, and the peer-label relationship implicitly hidden in the hierarchy. we propose several measures to learn the relevancy structure among peer labels, and then utilize the graph convolutional neural network (gcn) to realize "feature sharing" among peer labels. to address the possible side effect caused by modeling peer labels, we also develop a measure to evaluate the degree of confusion between labels in the same level, and then assign different weights to training samples according to their contribution in alleviating label confusion. the peerhtc method is realized through an embedded two-stage training approach, in which valuable information about latent relevancy of peer labels and the label confusion can be harvested from the first round of warm-up training and then enhances the second round for final classification. the rest of this article is organized as follows: section 2 introduces related works. section 3 defines the htc problem. section 4 introduces the peerhtc method in detail. section 5 presents the experimental results on three datasets. section 6 concludes the article. we share our codes on github foot_0 for reproducibility. ## problem formulation we define the htc problem in this section. specifically, we first give the definition of a category hierarchy and its properties, and then define the htc problem mathematically. definition 1. (category hierarchy) assume there exists an h-level category hierarchy γ. all possible labels in γ are denoted by @xmath0, c 2 , • • • , c h }, where the category hierarchy γ is then defined to be a partially order set (c, ≺), where ≺ represents the superior-subordinate relationship between labels and it satisfies the following three properties: definition 2. ## methodology in this section, we introduce the peerhtc method in detail. we first introduce peer label learning and sample importance learning, and then propose a two-stage training procedure. the overall architecture of peerhtc is illustrated in figure 1 . ## conclusion in this work, we originally propose the concept of "peer labels" to characterize the phenomenon that labels in the same level have latent relevancy. to utilize these peer labels to enhance htc, we develop the peerhtc method. we exploit gcn as an encoder for latent relevancy among peer labels, and reinforce feature sharing among these closely related peer labels. we also use a novel technique to assign training sample weights based on their importance in alleviating label confusion. the above procedures are embedded in a two-stage training approach. our experimental results demonstrate the evidence of peer labels in real datasets and the generally better performance of peerhtc against other state-of-the-art htc methods. in terms of application, one would expect a higher lift in performance from peerhtc when the granularity of categorization is relatively low, as demonstrated by our experimental results. we also suggest that one should carry out exploratory analysis or take into account domain knowledge, in order to decide the extent of peer-label relevancy for a specific dataset.