Developing Prefix-Tuning Models for Hierarchical Text Classification (2022)

Abstract: Hierarchical text classification (HTC) is a key problem and task in many industrial applications, which aims to predict labels organized in a hierarchy for given input text. For example, HTC can group the descriptions of online products into a taxonomy or organizing customer reviews into a hierarchy of categories. In real-life applications, while Pre-trained Language Models (PLMs) have dominated many NLP tasks, they face significant challenges too—the conventional fine-tuning process needs to modify and save models with a huge number of parameters. This is becoming more critical for HTC in both global and local modelling—the latter needs to learn multiple classifiers at different levels/nodes in a hierarchy. The concern will be even more serious since PLM sizes are continuing to increase in order to attain more competitive performances. Most recently, prefix tuning has become a very attractive technology by only tuning and saving a tiny set of parameters. Exploring prefix turning for HTC is hence highly desirable and has timely impact. In this paper, we investigate prefix tuning on HTC in two typical setups: local and global HTC. Our experiment shows that the prefix-tuning model only needs less than 1% of parameters and can achieve performance comparable to regular full fine-tuning. We demonstrate that using contrastive learning in learning prefix vectors can further improve HTC performance.

## introduction hierarchical text classification (htc) is a key task in many industrial applications. typically, a large number of labels are defined and organized in a taxonomic tree. how to accurately and efficiently predict texts into label paths in the label hierarchies is an important capacity in high demand. for example, many e-commerce applications need to assign an online product to a path in the label hierarchy, e.g., @xmath0. identifying these la-bel paths allows the information to be easily accessed by down-stream applications and human users. in the past few years, pre-trained language models (plms) have become a dominant solution for most natural language processing (nlp) applications. however, plm models often contain a very large number of parameters, and the model sizes keep increasing, which can put a heavy burden on htc applications. as an example, htc often benefits from building a number of local models to fully utilize label hierarchies. instead of training one model as in global htc modelling, local htc models rely on and leverage several inner classifiers @xcite . figure 1 shows that when building a local htc model for separating various types of drinks into the beverage category, the model sizes dramatically increase along with the increase of hierarchy levels. (more discussion about local and global htc models can be found in section 2). most recently, prefix tuning (li and liang, 2021; @xcite has become a very attractive technology by only tuning and saving a tiny set of parameters compared that of a fully fine-tuned model. exploring prefix turning for htc is hence highly desirable and has timely impact. in this paper, we investigate prefix tuning on htc in two typical setups: local and global htc. our experiment shows that the prefix-tuning model only needs less than 1% of parameters and can achieve performance comparable to regular full fine-tuning. we demonstrate that using contrastive learning in learning prefix vectors can further improve htc performance. in brief, our contributions are summarized as follows: beverage coffee tea instant coffee starbuck lipton oolong plm model size hierarchy level local models on each level local models on many internal nodes 1 2 3 100% 200% 300% figure 1 : an illustration to show that local htc model will face a size issue when using plm models to be classifiers. ## related work there are two major means of handling label hierarchies for htc, i.e., the local and global approach @xcite . the local approach builds a number of classifiers on different label levels or on many internal nodes in a label hierarchy but the global approach develops a single classifier to predict all labels that are flattened from the label hierarchy. @xcite developed convolution neural network (cnn) based local models at each level of label hierarchies and proposed to use the trained cnn at the higher level to initialize the cnn at a lower level. this transferring approach that considers inter-connections among the cnn models in a hierarchy showed to improve htc performance. regarding the global htc, a straightforward method is flattening labels' hierarchical structure into a flat list and modelling the htc simply to a multi-label classification task. recently, a trending method is utilizing a structure encoder to retain the label hierarchy to better utilize mutual information among labels. @xcite used a structure encoder, either a tree lstm or a graph convolution network (gcn), to consider labels' prior hierarchy information when learning label representations. plms have become a foundational paradigm on building various nlu tasks. for example, bert @xcite has been applied to tackle the htc task @xcite . in parallel, contrastive learning (cl) has been found to be effective in providing high-quality encoders in a simple self-learning way. for example, in computer vision, simclr @xcite uses the consistence between an anchor image and its transformed version and the in-consistence between the anchor and other instances in a batch (inbatch negative instances) to guide encoder training. inspired by the success of simclr in computer vision, cl-based textual representation learning has become a hot research topic in nlp. simcse @xcite uses dropout operations existing in transformer @xcite to provide selfaugmentation and can learn effective text representations. the cl training has been applied on the htc task. @xcite embeds both text inputs and labels (in a hierarchy) into an unified semantics space and solving the htc via vector matching. when training the text encoder, a cl setup is used and it considers label hierarchy information when forming contrastive pairs. @xcite uses a cl setup to train a high-quality text encoder. label hierarchy is firstly encoded by a graphormer @xcite and the encoded label information is used to generate text variations for providing positive pairs in the cl. although fine-tuning plm models enable many down-stream natural language understanding (nlu) tasks to achieve high performance, this paradigm faces a challenge in real deployment compared to other light weight models, e.g., cnn. also, plms contain a large number of parameters and the model sizes have been exploding in recent years for reaching more competitive performance and the trend is continuing. when deploying the fine-tuned plm models, all model parameters (updated in the fine-tuning process) need be stored. when many such plm models need be stored, for example, for local htc modelling, the required models sizes can be very large. to use plms in a more spaceefficient way, previous efforts, such as fine-tuning only several top layers in a plm or fine-tuning an adapter, are proposed @xcite . @xcite only learns prefix vectors to trigger a plm, which is frozen and cannot be tuned, to output the text representations fitting to the targeted domain better. in addition, @xcite extended the prefix-tuning on nlu tasks by using prefix vectors on each plm layer and dropped several components in a conventional prompt-tuning, e.g., verbalizer. ## exploring effective prefix tuning for hierarchical classification let x denote the text input, y a label hierarchy and y a specific category label path in y . htc aims to solve a multi-label categorization task: given textual input x, htc learns to predict possible label paths y in the hierarchy y . as discussed above, when developing htc in industrial applications, plm-based models face model-size issues, which is becoming more serious as plm sizes are continuing to increase. to tackle the challenge, we investigate soft prefix prompt (spp) tuning on htc. we propose to explore the models in two typical approaches. in section 3.1 below, we explore a transferring approach to better train spp vectors among different label levels in the local htc modelling. section 3.2 explores global htc models in which we propose to add a cl-based self-training step. tiple transformer layers in a plm model. built on that, the hidden output h [cls] at the final layer serves as the representation for x. the h [cls] ## conclusions and future work htc is a key task in many industrial applications. the conventional fine-tuning process needs to modify and save models that have a large number of parameters. this has become a more significant concern as plm sizes are continuing to increase in the foreseeable future. in this paper, we investigate prefix tuning on htc in two typical setups: local and global htc. to the best of our knowledge, this is the first systematic study towards developing prefix-tuning for htc in these typical architectures. in local htc modelling, we examine different architectures to leverage prefix vectors learned at different levels of label hierarchies and provide results about our best practice. we found that spp vectors trained at a higher level can be utilized to initialize a portion of spp vectors at a lower level of the hierarchy and such a vector transferring strategy is beneficial. for spp vectors, using them intact works better than using their transformed version. in the global htc strategy, we propose to add a self-training step built on a contrastive learning (cl) loss. on both wos and amazon datasets, such a cl pre-training is found to be helpful on improving model performance. for future work, we will extend the current work to study long-tailed labels which is very common in many applications. also, how to use labels' hierarchical information that can be represented by structural encoders is worth studying in spptuning.