Exploring Label Hierarchy in a Generative Way for Hierarchical Text Classification (2022)

Abstract: Hierarchical Text Classification (HTC), which aims to predict text labels organized in hierarchical space, is a significant task lacking in investigation in natural language processing. Existing methods usually encode the entire hierarchical structure and fail to construct a robust label-dependent model, making it hard to make accurate predictions on sparse lower-level labels and achieving low Macro-F1. In this paper, we explore the level dependency and path dependency of the label hierarchy in a generative way for building the knowledge of upper-level labels of current path into lower-level ones, and thus propose a novel PAAM-HiA-T5 model for HTC: a hierarchy-aware T5 model with path-adaptive attention mechanism. Specifically, we generate a multi-level sequential label structure to exploit hierarchical dependency across different levels with Breadth-First Search (BFS) and T5 model. To further improve label dependency prediction within each path, we then propose an original path-adaptive attention mechanism (PAAM) to lead the model to adaptively focus on the path where the currently generated label is located, shielding the noise from other paths. Comprehensive experiments on three benchmark datasets show that PAAM-HiA-T5 greatly outperforms all state-of-the-art HTC approaches especially in Macro-F1.

## introduction hierarchical text classification (htc), where text labels are predicted within a hierarchical structure, is a challenging task that has not yet received due attention within the field of multi-label classification. htc methods have been extensively applied in industry domains, e.g., news article classification @xcite , product classification in ecommerce @xcite , bidding strategy in paid search marketing @xcite . in htc tasks, labels at lower-level are inevitably sparse due to the hierarchical structure. many studies @xcite @xcite completely or partially neglect such hierarchical structure and fail to accurately predict those lower-level labels, achieving low macro-f1 score. existing studies @xcite have proved that introducing structure information can boost the predictive power on low-level labels and thus improve the overall task performance. a number of studies @xcite @xcite propose to construct multi-level classifiers that are trained independently and predicted sequentially, where only local maximum is achieved and propagation of error negatively impacts model prediction. some studies design end-to-end models that introduce various strategies (such as tree-lstm/gcn @xcite , label semantics matching network @xcite , graph-cnn @xcite and hierarchical fine-tuning based cnn @xcite ) to encode the overall hierarchy information (as depicted in figure 1 (a) ), where label dependency across different levels (as depicted in figure 1 (b) ) is not captured in a more principled way and unnecessary noises are introduced. such models tend to predict all labels simultaneously and independently with sigmoid function, and they could cause serious label inconsistencies (one label is predicted positive but its ancestors are not. for example, in figure 1 (b), the "football" is predicted while the "sports" is not.) and require post-processing to rectify these contradictions @xcite . although one recent study @xcite develops label-dependent models with reinforcement learning, it still fails to address label dependency within each path (as depicted in figure 1 (c) ) and fails to fully integrate labels and text information. this paper seeks to close the gap by proposing the paam-hia-t5. we are not only the first to capture lower-level label dependency on upper-level ones with generation model, but also the first to accurately explore the level dependency within each path. in each step of prediction phase, our model generates next label based on the text sequence and labels previously generated on current path. as illustrated in figure 1 (c), our model sequentially predicts "features", "news", "arts", "sports", "music", "football" labels. in the process where label "football" is generated, our model pays more attention on "news" and "sports" labels on its own path instead of "features" and "arts" labels on another path. therefore, our model is less likely to cause label inconsistency. for example, when labels "news" and "sports" are known to the model, it is easier to predict the label "football" as figure 1 (b) shows. our paam-hia-t5 model follows a two-step design. hierarchy-aware t5 (hia-t5), a variant of t5 that is fully aware of the level dependency in a multi-level sequential generative manner. inspired by the idea that conventional classification routines often follow the order that from coarsegrained to fine-grained to predict labels, we firstly use breadth-first search (bfs) to flatten hierarchical labels into multi-level sequential label structure, transforming the hierarchy to sequence. t5 model is applied to map the text sequence to label sequence, where the text sequence and upper-level labels generated earlier are then integrated in order to determine the next label. path-adaptive attention mechanism (paam), a mechanism to exploit the label correlation within each path and shield the noise from other paths. through the paam, the model can adaptively obtain a more reasonable attention distribution belonging to the path where the currently generated label is located. paam is implemented by means of a regularization method. this study makes the following major contributions: ## related work hierarchical text classification (htc) is a critical task with numerous applications @xcite @xcite . by methods of hierarchical information modeling, htc approaches can be categorized into flat, local and global approaches @xcite . flat approaches @xcite @xcite completely or partially ignore the label hierarchy and each label is independently predicted. some of them simply ignore the invaluable hierarchical information and achieve poor performance. some others predict leaf nodes first and then mechanically add their ancestor labels, which is only applicable where different paths in the label hierarchy share the same length. local approaches @xcite @xcite @xcite construct multiple local classifiers so that the misclassification at a certain level is propagated downwards the hierarchy, easily leading to the exposure of bias @xcite . specifically, @xcite proposes deep graph convolutional neural networks with hierarchical regularization. @xcite utilizes a multi-label neural network architecture with local and global optimization. to address the lower-level labels sparsity problem, @xcite takes advantage of a cnn-based model with the fine-tuning method. @xcite proposes to transfer the parameters of parent classifiers to initialize child classifiers for htc task. global approaches @xcite @xcite @xcite , where the entire structural information is encoded and all labels are simultaneously predicted, has become recent mainstream due to its better performance. @xcite handles htc task with reinforcement-learning-based label assignment method. @xcite uses metalearning to model the label interaction for multilabel classification. @xcite utilizes the bi-treelstm and gcn to model hierarchical relationship and makes flat predictions for hierarchical labels. @xcite combines cnn, rnn, gcn, and capsnet to model hierarchical labels. @xcite formulate htc as a semantic matching problem to mine the text-label semantics relationship. although recent researchers have managed to introduce hierarchical information in different fashions, most of them still regard flat multi-label classification as the backbone of htc where all labels are predicted simultaneously and independently. their exploitation of hierarchical structure is far from sufficient. ## problem definition for htc, we define the overall label hierarchy as a tree-like structure, denoted by @xmath0, e), where @xmath1, l 2 , . . . , l k } refers to the set of all label nodes in the corpus and k is the total number of them. e refers to the set of edges indicating the nodes' parent-child relations. formally, we denote text objects as @xmath2, x 2 , . . . , x n } and their labels as @xmath3, l 2 , . . . , l n }. each text object is represented by a text sequence x @xmath4, x 2 , . . . , x j ], where x j is a word and j is the number of words in the text object. meanwhile, each text object x i is mapped to a original label set l @xmath5, l 2 , . . . , l k , 1 ≤ @xmath6. we then define a set of special symbols @xmath7, /, eos} to identify special hierarchical relationships in the hierarchy. all labels l in the corpus constitute the overall label hierarchy t . the original label set l @xmath8, l 2 , . . . , l k , 0 ≤ @xmath9. we aim to train a model to predict corresponding label set l i for each text object x i , where the label set l i is constrained by the hierarchy t i . ## background the t5 model consists of an encoder-decoder architecture, which mainly includes the multi-head attention mechanism, the feed-forward network and so on @xcite , as depicted in the figure 2 . attention mechanism attn(•) is calculated as: t5 encoder is composed of a stack of "encoder blocks" and we define the number of blocks as b. each block contains a self-attention sublayer and a feed-forward sublayer. the input sequence of encoder is mapped to the embedding q encoder , k encoder , v @xmath10, which are then passed into the encoder. the output of encoder is denoted as o encoder . 1 0 0 0 0 1 1 0 0 0 0 0 1 0 0 0 0 1 1 0 1 1 0 0 the structure of the decoder looks similar to that of the encoder, except that it has an additional encoder-decoder attention sublayer that attends to the output of the encoder stack, following each casual self-attention sublayer. the causal attention mechanism of decoder only permits the model attend to past outputs. in the end, we obtain the decoder output denoted as o decoder . ## hierarchy-aware twith path-adaptive attention mechanism as depicted in figure 3 , we propose a paam-hia-t5 model for htc: a hierarchy-aware t5 model with path-adaptive attention mechanism. paam-hia-t5 consists of the hia-t5 for level-dependent label generation and the paam for path-specific label generation. ## conclusion for htc task, we explicitly define the concepts of "level dependency" and "path dependency" for the first time. furthermore, in order to build the knowledge of upper-level labels into lower-level ones in htc task, we devise an innovative paam-hia-t5 methodology by exhaustively exploring level dependency and path dependency of hierarchy in a generative manner. comprehensive experiments on three benchmark datasets show that our model greatly outperforms all state-of-the-art htc approaches especially in macro-f1.