Making Pretrained Language Models Good Long-tailed Learners (2022)

Abstract: Prompt-tuning has shown appealing performance in few-shot classification by virtue of its capability in effectively exploiting pre-trained knowledge. This motivates us to check the hypothesis that prompt-tuning is also a promising choice for long-tailed classification, since the tail classes are intuitively few-shot ones. To achieve this aim, we conduct empirical studies to examine the hypothesis. The results demonstrate that prompt-tuning makes pretrained language models at least good long-tailed learners. For intuitions on why prompt-tuning can achieve good performance in long-tailed classification, we carry out in-depth analyses by progressively bridging the gap between prompt-tuning and commonly used finetuning. The summary is that the classifier structure and parameterization form the key to making good long-tailed learners, in comparison with the less important input structure. Finally, we verify the applicability of our finding to few-shot classification.

## introduction pretrained language models (plms) with clstuning (i.e., finetuning a plm by applying a classifier head over the [cls] representation) have achieved strong performance in a wide range of downstream classification tasks @xcite @xcite . however, they have been less promising in the long-tailed scenario @xcite . the long-tailed scenario is different from common scenarios due to the long tail phenomenon exhibited in the class distribution, as illustrated in figure 1 . the long-tailed class distribution prevents plms from achieving good performance, especially in 0 20 40 60 80 100 120 class index (from head to tail). head tail figure 1 : an example long-tailed class distribution from iflytek dataset @xcite , where we can distinguish tail classes from head classes. recent advances on prompt-tuning foot_0 have witnessed a surge of making plms better few-shot learners @xcite @xcite . prompt-tuning requires plms to perform classification in a cloze style, thus is superior to cls-tuning in two aspects: 1) it aligns the input structure with that of masked language modeling (mlm); and 2) it reuses the classifier structure and parameterization from the pretrained mlm head. these two merits of prompt-tuning equip plms to better exploit pretrained knowledge and hence gain a better few-shot performance than that with cls-tuning. the success of prompt-tuning in few-shot scenarios motivates us to hypothesize that prompttuning can relieve the tail bottleneck and thus make plms good long-tailed learners. the reason why we make such a hypothesis is that the tail classes are intuitively few-shot ones. however, long-tailed classification is different from few-shot classification to a certain extent, as it allows the possibility to transfer knowledge from head classes to tail ones. we empirically examine the hypothesis by conducting empirical evaluations on three long-tailed classification datasets. the comparison results show that plms can be good long-tailed learners with prompt-tuning, which outperforms plms with cls-tuning by large margins. besides, prompttuning even exhibits a better performance than that of cls-tuning with appropriate calibrations (e.g., focal loss @xcite . the widely accepted decoupling property @xcite claims that a good long-tailed learner should enjoy a nearly uniform distribution in terms of weight norms, otherwise the norms of head classes can be way much larger than the tail ones'. it is therefore expected that the weights tuned with prompt-tuning own a flat distribution. we validate the property of prompttuning by visualizing the norms of trained classification weights across classes. with the compelling results, we put that our hypothesis is valid. we also provide further intuitions by asking why prompt-tuning could be so promising, as shown in the above empirical investigations. through indepth analyses, we uncover that re-using the classifier structure and parameterization from the mlm head is a key for attaining the good long-tailed performance, largely outweighing the importance of aligning the input structure with that of mlm. cls-tuning, with classifier structure derived and parameters partly initialized from the mlm head, approximates the performance of prompt-tuning. we believe that this observation would as well shed light on related work that aims to improve prompttuning itself. as such, we finally present the comparison results of the improved cls-tuning and prompt-tuning in the few-shot scenario, suggesting the applicability of the improved cls-tuning to few-shot classification. ## analyses although we have verified that our hypothesis is valid, we are more of thirst to see why prompttuning can be so promising and provide further intuitions. to this demand, we consider three research questions here and carry out in-depth analyses to answer them. we also hope the analyses can shed light on the design of prompt-tuning itself in related areas. ## applicability to few-shot classification from the analyses above, we come up with an alternative to cls-tuning that owns a comparable effectiveness with prompt-tuning for long-tailed classification, by equipping cls-tuning with the structure and the layer normalization affinities of the pretrained mlm head. with this much simpler surrogate classifier, we prefer to retrospectively explore how the classifier would perform in the few-shot scenario, especially when compared with prompt-tuning. the intuition is that, while the layer normalization can not transfer knowledge from the head to tail any longer in few-shot classification, its innate friendliness to the tail can probably be applied to few-shot classes for generalization. therefore, we conduct experiments on three few-shot datasets, ecom from fewclue @xcite , and rte, boolq from super-glue @xcite . ecom is a review sentiment classification dataset, rte @xcite ) is a two-way entailment dataset, and boolq @xcite is a yes-or-no question answering dataset. while ecom has already been designed for fewshot learning, the others are not originally. for the latter two, we treat the original development set as the test set following @xcite , and randomly sample 32 examples uniformly from the original training set as the training set following @xcite . to strictly following a true few-shot setting @xcite , 32 examples that do not overlap with those in our training set are used to form the development set. the implementation is akin to the one we used for the long-tailed experiments. the batch size is reduced to 2 due to the small scale of training data. bert-base-uncased is leveraged as the backbone for english datasets. the maximum length is set to 64, 128, and 256 respectively for ecom, rte, and boolq. the statistics of these datasets are listed in the example templates and verbalizers are listed in appendix b. moreover, since few-shot experiments are sensitive to the choice of hyperparameters, we again take average accuracy scores over 5 runs as the results, attached with variances. we can observe from ## related work plms have brought classification tasks to a brandnew stage where the solutions to these tasks are way much simpler than ever @xcite . however, plms are still suboptimal for some corner cases, such as few-shot classification @xcite and long-tailed classification @xcite . for few-shot classification, prompt-tuning, which finetunes models in a language modeling fashion, is increasingly taking a central role in the mainstream methods @xcite . since plms are mostly trained with language modeling objectives, prompttuning becomes, as water is to fish, the key to unearthing the few-shot or even zero-shot learning capabilities of plms @xcite @xcite . instantly after the success, the engineering of prompts drives related research on prompt search/generation @xcite @xcite and various downstream applications such as text generation @xcite , relation extraction @xcite , and entity typing @xcite . recently, prompt-tuning also serves as an alternative way for parameter-efficient finetuning by only finetuning parameters of the inserted continuous prompts @xcite , in place of previously adopted adaptertuning @xcite . the parameter efficiency brought by prompt-tuning has blazed a trail for increasingly large language models. in contrast, little work has been investigated to make plms good long-tailed learners. intuitively, the tail classes are essentially few-shot ones. thus, we speculate that prompt-tuning is also a promising choice to make plms good long-tailed learners via transferring knowledge of head classes. as longtailed classification is a long-standing problem in the general area of machine learning @xcite @xcite @xcite and the long-tailed phenomenon also exists in the domain of natural language processing, we believe that a systematic exploration on whether and why prompt-tuning can make plms good long-tailed learners will facilitate further advances in the related areas. ## conclusions inspired by the success of prompt-tuning in fewshot learning, we empirically examine whether prompt-tuning can make plms good long-tailed learners in this work. the results validate the hypothesis. we also conduct in-depth analyses on why prompt-tuning benefits plms for long-tailed classification, from the perspectives of coupling, classifier, and input respectively, to offer further intuitions. based on the analyses, we summarize that the classifier structure and parameterization are crucial for enhancing long-tailed performance of plms, in contrast to other factors. extended empirical evaluation results on few-shot classification show that our finding would shed light on related work that seeks to boost prompt-tuning.