Fine-tuning Pre-trained Language Models for Few-shot Intent Detection: Supervised Pre-training and Isotropization (2022)

Abstract: It is challenging to train a good intent classifier for a task-oriented dialogue system with only a few annotations. Recent studies have shown that fine-tuning pre-trained language models with a small set of labeled utterances from public benchmarks in a supervised manner is extremely helpful. However, we find that supervised pre-training yields an anisotropic feature space, which may suppress the expressive power of the semantic representations. Inspired by recent research in isotropization, we propose to improve supervised pre-training by regularizing the feature space towards isotropy. We propose two regularizers based on contrastive learning and correlation matrix respectively, and demonstrate their effectiveness through extensive experiments. Our main finding is that it is promising to regularize supervised pre-training with isotropization to further improve the performance of few-shot intent detection. The source code can be found at https://github.com/fanolabs/isoIntentBert-main .

## introduction intent detection is a core module of task-oriented dialogue systems. training a well-performing intent classifier with only a few annotations, i.e., few-shot intent detection, is of great practical value. recently, this problem has attracted considerable attention @xcite @xcite but remains a challenge. to tackle few-shot intent detection, earlier works employ induction network @xcite , generation-based methods @xcite , metric learning @xcite , and selftraining @xcite , to design sophisticated algorithms. recently, pre-trained language models (plms) have emerged as a simple yet promising solution to a wide spectrum of natural language processing (nlp) tasks, triggering the surge of plm- * corresponding author. a plm-based fine-tuning method @xcite , called intentbert, utilizes a small amount of labeled utterances from public intent datasets to fine-tune plms with a standard classification task, which is referred to as supervised pre-training. despite its simplicity, supervised pre-training has been shown extremely useful for few-shot intent detection even when the target data and the data used for fine-tuning are very different in semantics. however, as will be shown in section 3.2, intentbert suffers from severe anisotropy, an undesirable property of plms @xcite @xcite . anisotropy is a geometric property that semantic vectors fall into a narrow cone. it has been identified as a crucial factor for the sub-optimal performance of plms on a variety of downstream tasks @xcite @xcite , which is also known as the representation degeneration problem @xcite . fortunately, isotropization techniques can be applied to adjust the embedding space and yield significant performance improvement in many tasks @xcite . hence, this paper aims to answer the question: many isotropization techniques have been developed based on transformation @xcite , contrastive learning @xcite , and top principal components elimination @xcite . however, these methods are designed for off-the-shelf plms. when applied on plms that have been fine-tuned on some nlp task such as semantic textual similarity or intent classification, they may introduce an adverse effect, as observed in rajaee and pilehvar (2021c) and our pilot experiments. in this work, we propose to regularize supervised pre-training with isotropic regularizers. as shown in fig. 1 , we devise two regularizers, a contrastive-learning-based regularizer (cl-reg) and a correlation-matrix-based regularizer (cor-reg), each of which can increase the isotropy of the feature space during supervised training. our empirical study shows that the regularizers can significantly improve the performance of standard supervised training, and better performance can often be achieved when they are combined. the contributions of this work are three-fold: ## pilot study before introducing our approach, we present pilot experiments to gain some insights into the interaction between isotropization and fine-tuning plms. ## method the pilot experiments reveal the anisotropy of a plm fine-tuned on intent detection tasks and the 1 we refer the reader to the original paper for details. specifically, we propose two regularizers to make the feature space more isotropic during fine-tuning. before presenting our method, we first introduce supervised pre-training. ## experiments to validate the effectiveness of the approach, we conduct extensive experiments. ## conclusion in this work, we have identified and analyzed the anisotropy of the feature space of a plm finetuned on intent detection tasks. further, we have proposed a joint training framework and designed two regularizers based on contrastive learning and correlation matrix respectively to increase the insotropy of the feature space during fine-tuning, which leads to notably improved performance on few-shot intent detection. our findings and solutions may have broader implications for solving other natural language understanding tasks with plm-based models.