Hierarchical Verbalizer for Few-Shot Hierarchical Text Classification (2023)

Abstract: Due to the complex label hierarchy and intensive labeling cost in practice, the hierarchical text classification (HTC) suffers a poor performance especially when low-resource or few-shot settings are considered. Recently, there is a growing trend of applying prompts on pre-trained language models (PLMs), which has exhibited effectiveness in the few-shot flat text classification tasks. However, limited work has studied the paradigm of prompt-based learning in the HTC problem when the training data is extremely scarce. In this work, we define a path-based few-shot setting and establish a strict path-based evaluation metric to further explore few-shot HTC tasks. To address the issue, we propose the hierarchical verbalizer (“HierVerb”), a multi-verbalizer framework treating HTC as a single- or multi-label classification problem at multiple layers and learning vectors as verbalizers constrained by hierarchical structure and hierarchical contrastive learning. In this manner, HierVerb fuses label hierarchy knowledge into verbalizers and remarkably outperforms those who inject hierarchy through graph encoders, maximizing the benefits of PLMs. Extensive experiments on three popular HTC datasets under the few-shot settings demonstrate that prompt with HierVerb significantly boosts the HTC performance, meanwhile indicating an elegant way to bridge the gap between the large pre-trained model and downstream hierarchical classification tasks.

## introduction hierarchical text classification (htc) is a longstanding research problem due to the wide range of real applications @xcite . however, prior works could still suffer poor performance in practice due to the nature of its sophisticated label hierarchy as well as the requirement of largescale data annotation before training the model. therefore, solving the htc under the low-resource @xcite or few-shot setting becomes an urgent research topic. existing state-of-the-art htc models focus on inserting label hierarchy features through graph encoders and then fuse the features into the input layer @xcite or output layer @xcite of a text encoder such as bidirectional lstm or pre-trained language models (plms), as shown in figure 1 @xcite . and there is a trend of taking advantage of plms @xcite as the backbone of the text encoder through a fine-tuning paradigm. despite the success of plms @xcite in extensive nlp-related tasks, recently, a series of studies @xcite @xcite suggest that it's helpful to elicit the knowledge contained in plms and point out the fine-tuning paradigm is suboptimal in few-shot settings due to distinct training strategies between the pre-training and fine-tuning stages. inspired by "incontext learning" proposed by @xcite , lots of prompt-based @xcite @xcite methods were proposed to bridge the gap between pre-training and downstream tasks via stimulating pre-trained model knowledge with a few hard or soft prompts. in prompt-based tuning, the input is usually wrapped through a natural language template and the tasks are converted as masked language modeling (mlm) for plm. for example, in the sentiment classification task, the original input x will be wrapped as "x. it was [mask]". the objective is to utilize mlm to predict the word that fills the [mask], and subsequently employ a verbalizer to map the predicted word to the final classification (e.g. "positive" -> label "positive"). although remarkable performances have been achieved via prompt tuning on the flat text classification where labels have no hierarchy, its effects on htc problems remain unclear, as discussed in hpt @xcite . as shown in figure 1 (b), hpt proposed a hierarchy-aware prompt tuning method that incorporates the label hierarchy knowledge into soft prompts through graph representation and achieves the new state-of-the-art results on several htc popular datasets. however, even though the low-resource setting experiment was considered in hpt, the commonly used k-shot setting was not investigated. the limitation lies in the absence of a uniform definition of the k-shot setting in htc. besides, the way to utilize plms in few-shot settings through soft prompts and fuse hierarchy by graph encoder into the plms harms tapping the full potential of plms. hence, it is crucial to exploit a new method to elicit knowledge from plms in a hierarchy-aware manner for few-shot learning. inspired by the prior works on verbalizer design @xcite between model outputs and labels, as shown in figure 4 (a) and 4(b), which makes promising improvements over prompt-based tuning, it is natural to raise this question: is there any verbalizer design method specific to the htc problems? the most current works can be mainly divided into three kinds of verbalizers: manual verbalizers, search-based verbalizers, and soft verbalizers. however, the main difference between previous works on verbalizers is the way of embedding the semantic space and they are all based on a strong assumption that there is no hierarchical dependency between downstream task labels, which raises a gap between rich flat prior knowledge in plm and downstream task hierarchies. thus these verbalizers are not suitable for hierarchical classification tasks, lacking awareness of hierarchy in their architectural design. to address these issues, we introduce a hierarchical-aware verbalizer (hierverb) combined with the prompt tuning method to fully exploit the hierarchical knowledge within labels. the major contributions of this paper can be summarized as follows: ## hierarchical verbalizer in this section, we will introduce the proposed hierarchy-aware verbalizer in detail. we incorporate hierarchical information through our multiverbalizer framework with prompt templates to elicit rich prior knowledge within the plms. figure 3 shows the overall architecture of our proposed hierverb. we first obtain the hidden states of the multiple mask tokens to represent the sentence and then project it to the verbalizer's space of different label layers. ## conclusion in this paper, we define the few-shot settings on htc tasks and a novel evaluation method based on the perspective of path correctness, which is valuable in practical applications. we propose a novel approach to adapt flat prior knowledge in plm to downstream hierarchical tasks. the proposed hierverb learns hierarchical-aware verbalizers through flat contrastive learning and constraint chain, which elegantly leverages the prior knowledge of plms for better few-shot learning. we perform few-shot settings on htc tasks and extensive experiments show that our method achieves state-of-the-art performances on 3 popular htc datasets while guaranteeing excellent consistency performance.