HPT : Hierarchy-aware Prompt Tuning for Hierarchical Text Classification (2022)

Abstract: Hierarchical text classification (HTC) is a challenging subtask of multi-label classification due to its complex label hierarchy.Recently, the pretrained language models (PLM)have been widely adopted in HTC through a fine-tuning paradigm. However, in this paradigm, there exists a huge gap between the classification tasks with sophisticated label hierarchy and the masked language model (MLM) pretraining tasks of PLMs and thus the potential of PLMs cannot be fully tapped.To bridge the gap, in this paper, we propose HPT, a Hierarchy-aware Prompt Tuning method to handle HTC from a multi-label MLM perspective.Specifically, we construct a dynamic virtual template and label words that take the form of soft prompts to fuse the label hierarchy knowledge and introduce a zero-bounded multi-label cross-entropy loss to harmonize the objectives of HTC and MLM.Extensive experiments show HPT achieves state-of-the-art performances on 3 popular HTC datasets and is adept at handling the imbalance and low resource situations. Our code is available at https://github.com/wzh9969/HPT.

## introduction hierarchical text classification (htc) aims to categorize a text into a set of labels with a structured class hierarchy (commonly modeled as a tree) @xcite . htc is a multi-label text classification problem, where the classification result corresponds to one or more paths of the hierarchy @xcite . the major challenge of htc is to model the large-scale, imbalanced, and structured label hierarchy @xcite . as shown in figure 1 (a), existing state-of-theart htc models @xcite @xcite separately extract text and label hierarchy features by utilizing text and graph encoders, and then fuse the two sources of features into a final representation for text classification. specifically, @xcite takes advantage of powerful pretrained language models (plms) in htc through a finetuning paradigm, where they use plms as the text encoder. in this paradigm, the plms are trained to infer with complex label hierarchy. despite the success of the fine-tuning paradigm, some recent studies suggest that it may suffer from distinct training strategies in the pretraining and fine-tuning stages, which restrains the finetuned models to take full advantage of knowledge in plms @xcite . therefore, a new paradigm known as prompt tuning is proposed to bridge the gap between the downstream tasks and the pretraining tasks of plms, which can tap the full potential of plms. by warping the text (e.g., "x") into the model input (e.g., "x is [mask]" ) and taming the plms to complete the masked cloze test, prompt tuning has achieved promising performances on the flat text classification where labels have no hierarchy @xcite . how about the performances of the prompt tuning in htc? in the pilot study, we test flat prompt tuning methods on htc and surprisingly find that they are even comparable with the state-of-theart models in htc. this result suggests that the expressive power of plms has been undermined in the prior htc methods due to the pretrainingfinetuning gap. although the flat prompt tuning methods have somewhat narrowed the gap, there still remain two challenges while combining plms with htc. to bridge these two gaps, as shown in figure 1(b) , we propose a hierarchy-aware prompt tuning (hpt) method that solves htc from a multi-label mlm perspective. in detail, to bridge the hierarchy and flat gap, we incorporate the label hierarchy knowledge into soft prompt with continuous representation. specifically, we incorporate the depth and width information in the label hierarchy into different virtual template words, which is helpful to alleviate the label imbalance problem as verified by our experiments. to bridge the multi-label and multi-class gap, we transform htc into a multilabel mlm problem by a zero-bounded multi-label cross-entropy loss which continually seeks to increase the score of the correct label and decrease the score of the incorrect labels. we summarize our contributions as follows: to bridge these gaps, we transform htc into a hierarchy-aware multi-label mlm problem. ## methodology in this section, we introduce a hierarchy-aware prompt tuning method to solve htc from a multilabel mlm perspective. ## conclusion in this paper, we propose a hierarchy-aware prompt tuning (hpt) method to bridge the gaps between htc and mlm. to bridge the hierarchy and flat gap, hpt incorporates the label hierarchy knowledge into a virtual template and label words. to bridge the multi-label and multi-class gap, hpt introduces a zero-bounded multi-label cross-entropy loss to harmonize the objectives of htc and mlm. hpt transforms htc into a hierarchy-aware multilabel mlm task, which can better tap the potential of the pretrained language model in htc. extensive experiments show that our method achieves state-of-the-art performances on 3 popular htc datasets, and is adept at handling the imbalance and low resource situations.