Dual Prompt Tuning based Contrastive Learning for Hierarchical Text Classification (2024)

Abstract: Hierarchical text classification aims at categorizing texts into a multi-tiered tree-structured hierarchy of labels. Existing methods pay more attention to capture hierarchy-aware text feature by exploiting explicit parent-child relationships, while interactions between peer labels are rarely taken into account, resulting in severe label confusion within each layer. In this work, we propose a novel Dual Prompt Tuning (DPT) method, which emphasizes identifying discrimination among peer labels by performing contrastive learning on each hierarchical layer. We design an innovative hand-crafted prompt containing slots for both positive and negative label predictions to cooperate with contrastive learning. In addition, we introduce a label hierarchy self-sensing auxiliary task to ensure cross-layer label consistency. Extensive experiments demonstrate that DPT achieves significant improvements and outperforms the current state-of-the-art methods on BGC and RCV1-V2 benchmark datasets.

## introduction as a specialized sub-task of text classification, hierarchical text classification (htc) has a wide range of applications in the realistic scenarios, such as intent recognition in dialogue system, commodity and book management @xcite , where a large number of categories are organized into a tree-structured hierarchy. the ultimate goal of htc is to categorize texts or documents from the topmost level to the finest granularity precisely. due to the challenges of large-scale, imbalanced and complex label hierarchy @xcite , simply transferring flat multi-label text classification algorithms to deal with htc tasks often fails to achieve sufficient performance. 1 code is publicly available at https://github.com/ ccx06/dual-prompt-tuning-for-htc . full use of the hierarchical structure of labels is the key to achieving well-performing classification in htc tasks, which facilitates the model in predicting labels that align with hierarchical relationship. as shown in figure 1 , current mainstream methods @xcite @xcite apply a dualencoders framework to model the text and hierarchical structure separately, and then fuse them to obtain hierarchy-enhanced text features. @xcite first proposes a hierarchy-aware prompttuning method, which incorporates the label hierarchy encoded by the graph attention network into a soft prompt to bridge hierarchy and flat gap. however, most studies pay close attention to exploit relations that explicitly displayed in the hierarchy, while internal interactions between "peer labels", which refer to a group of labels at the same hierarchical layer, are often neglected. peerhtc @xcite recently tries to explore latent relevancy among peer labels with a complicated two-stage training procedure, in which peer and adjacent level-wise features are separately extracted by graph convolutional neural networks. nevertheless, alleviating confusion within peer labels, especially fine-grained ones that share the same parent node at lower level, still remains challenging and highly valued. to this end, we propose a novel dual prompt tuning (dpt) method, aimed at alleviating label confusion between peer labels. we put forward a hierarchy-aware peer-label contrastive learning (hierpcl) approach to extract discriminative pair-wise representations. in detail, we create an original dual prompt template containing both positive and negative label slots, and then perform label-wise contrastive learning on the embeddings of both two types of slots. dual prompt is multifunctional, targeted for predicting positive labels and recognizing incorrect but confused negatives at each hierarchical layer. we furthermore incorporate a rank loss component into the contrastive loss function to enhance label consistency. moreover, we design an adaptive hard negative sampling strategy and a hierarchy-injected label representation encoding method to further boost the performance of hierpcl. besides, we introduce a simple and effective label hierarchy self-sensing auxiliary task to keep our model in the best sense of holistic hierarchical structure. the basic idea is to internalize structural hierarchy knowledge to ensure the cross-layer consistency of the final path predictions. instead of directly injecting label hierarchy into text semantics, we perform multi-task learning collaborating with label prediction to identify the consistency and correctness of each candidate label path. the contributions are summarized as follows: ## methodology in this section, we present a detailed description of our proposed dpt model to address htc tasks. as shown in figure 2 , our model is based on prompttuning framework with multi-verbalizer. the principal innovations of dpt are twofold, including (1) the implementation of hierarchy-aware peerlabel contrastive learning to obtain rich discriminative features, and (2) the incorporation of label hierarchy self-sensing auxiliary task to enhance encoder's ability for an in-depth understanding of label hierarchy structure. ## conclusion in this paper, we present a novel dual prompt tuning method for htc tasks. firstly, we propose a hierarchy-aware peer-label contrastive learning approach to alleviate confusion between peer labels. an original dual prompt template is created with slots for both positive and negative label, on which contrastive learning is performed at each layer. secondly, to further strengthen knowledge understanding of label hierarchy structure, we design a label hierarchy self-sensing auxiliary task to identify consistency and correctness of model predictions. experimental results illustrate that our proposed dpt method achieves significant improvements on popular htc datasets. it excels in precisely recognizing negative labels and contributes to obtaining hierarchy-aware discriminative features. notably, dpt exhibits remarkable effectiveness in preserving label path consistency and addressing imbalanced hierarchy challenge.