Label Semantic Aware Pre-training for Few-shot Text Classification (2022)

Abstract: In text classification tasks, useful information is encoded in the label names. Label semantic aware systems have leveraged this information for improved text classification performance during fine-tuning and prediction. However, use of label-semantics during pre-training has not been extensively explored. We therefore propose Label Semantic Aware Pre-training (LSAP) to improve the generalization and data efficiency of text classification systems. LSAP incorporates label semantics into pre-trained generative models (T5 in our case) by performing secondary pre-training on labeled sentences from a variety of domains. As domain-general pre-training requires large amounts of data, we develop a filtering and labeling pipeline to automatically create sentence-label pairs from unlabeled text. We perform experiments on intent (ATIS, Snips, TOPv2) and topic classification (AG News, Yahoo! Answers). LSAP obtains significant accuracy improvements over state-of-the-art models for few-shot text classification while maintaining performance comparable to state of the art in high-resource settings.

## introduction large pre-trained language models have enabled better performance on many nlp tasks-especially in few-shot settings @xcite @xcite . more informative representations of textual inputs often leads to much higher downstream performance on nlp applications, which explains the rapid and general adoption of models such as (ro)bert(a) @xcite , gpt-2 @xcite , and t5 @xcite . however, while these models are often used to effectively encode inputs, fewer works have attempted to give models access to informative representations of labels as well. figure 1 : overview of our approach, label semantic aware pre-training (lsap). we collect utterance-intent pairs and create new pairs from unlabeled reddit and twitter data, convert the intents to natural language, concatenate the utterance and intent, noise the concatenated sequence, and train a sequence-to-sequence model to denoise the sequence. most discriminative approaches to text classification only give the model access to label indices. a recent stream of work has obtained significant improvements in structured prediction tasks by using sequence-to-sequence (seq2seq) models to generate labels @xcite . yet these generative approaches make use of label semantics-the meaning of class label names-only during fine-tuning and prediction. thus, we propose label semantic aware pretraining (lsap) to incorporate label semantics as well as input-label associations into the pre-training step (figure 1 ). our experiments show that lsap yields higher performance with fewer fine-tuning examples in a variety of domains. our contributions include the following: ## approach our approach, lsap, performs a secondary pretraining step with t5 on a large scale collection of (pseudo-)labeled examples; see figure 1 for an overview. in addition to training on existing labeled datasets ( §3.1), we (1) filter unlabeled data using a dialogue act classifier ( §3.1.1); (2) pseudo-label the utterances that pass the filter using an intent generator ( §3.1.1); and (3) perform secondary pretraining on the labeled and pseudo-labeled data using t5 ( §3.2). §3.1 details how we collect, select, and label pre-training data. §3.2 describes the pretraining formats we test in our experiments. ## experimental setup we aim to understand whether lsap can improve text classification performance over a variety of state-of-the-art (sota) baselines. we evaluate lsap on two text classification tasks: intent classification (ic) and topic classification (tc). ic is the most similar task to our pre-training objective. our ic evaluation datasets include snips @xcite , atis @xcite , and the low-resource reminder and weather domains provided in topv2 @xcite . snips' intents are each from different domains. atis focuses on the flight domain; some intents appear only once or not at all in the training set, and some utterances are tagged with multiple intents (we count these as separate intents). for multi-class examples, we separate intents with the character "#" (e.g., "book flight # airfare"). topv2 (reminder) focuses on the creation and modification of personal reminders, while topv2 (weather) focuses on queries regarding the weather and time of sunrise/sunset. for tc, we evaluate on yahoo! answers (ya) foot_7 and ag news. 9 ya consists of conversational user questions labeled with the general category of the question (e.g., "health", "sports"). ag news consists of formal-register news articles labeled with their category (e.g., "business", "world"). ## results which pre-training format is best? we present 1shot, 4-shot, and full-resource ic accuracies on each evaluation set in ## conclusions we have proposed a pre-training approach for leveraging the semantic information inherent in labels. our method improves few-shot text classification performance across domains while maintaining high performance in full-resource settings. this approach is fairly general and could potentially be extended to more structured semantic parsing tasks by annotating some of the pre-training examples, or perhaps by simply including diverse labeled examples from a variety of tasks in the pre-training step. future work could investigate extending this method to pre-training from scratch, as well as tuning the utterance and label formats. one could also use demonstrations to achieve better performance with fewer gradient updates.