In-Context Learning for Text Classification with Many Labels (2023)

Abstract: In-context learning (ICL) using large language models for tasks with many labels is challenging due to the limited context window, which makes it difficult to fit a sufficient number of examples in the prompt. In this paper, we use a pre-trained dense retrieval model to bypass this limitation, giving the model only a partial view of the full label space for each inference call. Testing with recent open-source LLMs (OPT, LLaMA), we set new state of the art performance in few-shot settings for three common intent classification datasets, with no fine-tuning. We also surpass fine-tuned performance on fine-grained sentiment classification in certain cases. We analyze the performance across number of in-context examples and different model scales, showing that larger models are necessary to effectively make use of larger context lengths for ICL. By running several ablations, we analyze the model’s use of: a) the similarity of the in-context examples to the current input, b) the semantic content of the class names, and c) the correct correspondence between examples and labels. We demonstrate that all three are needed to varying degrees depending on the domain, contrary to certain recent works.

## introduction in-context learning (icl) using large language models (llms) has recently exploded in popularity. models pre-trained on massive amounts of textual data are able to reach reasonable performance on a wide variety of tasks with only a few examples of input and output for a given task provided in the model's input prompt in natural language @xcite @xcite . in this work, we study whether icl can handle challenging classification tasks with many possible labels, by augmenting the lm with a secondary pre-trained retrieval model. the main problem with applying icl to tasks involving classification with many labels is the lim-ited context window these models have. ordinarily with icl, at minimum one example from each class is provided in-context to allow the model to make a choice between all the labels of the task. because of this limitation, icl has not been directly applied to these sorts of problems. in this work we relax this requirement, allowing the model to see only a subset of the most relevant labels for the given datapoint we are performing inference on. by testing on intent classification (upwards of 50 classes) and fine-grained sentiment analysis (upwards of 25 classes), we demonstrate that the resulting performance with this method can reach sota. by coupling the llm with an external pre-trained dense retriever model @xcite , we can dynamically retrieve a set of examples to provide to the lm in-context, that reflects only the most relevant labels to the current example in the label space. most existing work on augmenting lms with retrieval models @xcite focuses on tuning the retrieval and/or lm. we demonstrate that even without tuning either, when the pre-trained models are strong enough we can still achieve sota across various tasks using icl. we evaluate llms in this setting with three intent classification datasets: banking77 @xcite , hwu64 @xcite , and clinc150 @xcite , as well as one fine-grained sentiment classification dataset: goemotions @xcite . experiments are done using the llama models @xcite and the opt models @xcite as llms. we compare the performance achieved against adapter-based fine-tuning of mlm models (deberta-v2-xxlarge with the "pfeiffer" bottleneck-style adapter @xcite implemented with adapterhub @xcite ) and the previous sota for intent detection (convfit; @xcite , as well as comparing against setfit @xcite , a recent lightweight method involving contrastive training of small mlm models. the contributions of this work are: ## experimental setup specific retrieval model: for our sentence encoder/retriever, we use the sentencetransformers library @xcite , and use the pre-trained "all-mpnet-base-v2" model (a 110m parameter model pre-trained on over 1 billion training pairs). the setfit results are based on contrastively tuning the same pre-trained model trained by microsoft through the setfit library foot_0 . ## results example ordering: we provide a brief study regarding how to order examples in-prompt by similarity, since previous work has been inconclusive on this front, suggesting that the ideal ordering is dataset dependent @xcite . as seen from sota performance: tables 1 and 2 shows the performance comparison of all methods. performance of the retrieval+icl pipeline on banking, hwu and clinc is state of the art in both the 5 and 10-shot settings. not only this, but to significantly surpass the previous state of the art for all three intent classification datasets only llama-2 7b is necessary, which with 8-bit quantization can be run on consumer hardware. in the most challenging evaluation setting (the highly-specialized intent classes of the banking dataset in the most data-scarce 5-shot setting), the margin between de-berta and llama-2 70b is 7.49%. in general the deberta model showed lower performance in the 5-shot scenarios, likely due to the extremely limited data. in the case of goemotions ( ## ablation studies several ablations studies are done to test what aspects of the retrieved examples the llm is using to make the predictions. the ablation studies were done on a random split of the hwu dataset and the goemotions dataset. ablation results for hwu are shown visually in figure 3 and for goemotions in figure 4 . ## retriever and lm generalization one interesting result from our experiments is the fact that generic retrievers seem to be able to quite effectively generalize across domains and tasks. using the same exact retriever model across 3 different intent detection datasets (which according to the taxonomy of @xcite constitutes cross-task generalization) as well as a sentiment classification dataset (according to the previous taxonomy, a cross-domain generalization) demonstrates sota or better performance in almost all cases. the distribution shift locus, for both the retriever and the language model generating the final prediction, is from pretraining to testing time. this is because they are both pre-trained on massive generic data before being tested in a zero-shot setting. nearest neighbor selection of in-context examples: one of the earliest studies of the role of example selection in icl is "kate" @xcite . in this paper, the authors probe the performance of gpt-3 on nlp tasks using knn retrieval (roberta) for example selection. they compare this method against random selection and using the retrieval model directly (plain knn). they also examine the effect of example ordering on performance and conclude that the most performant ordering (least-to-most and most-to-least similar orderings are tested) depends on the dataset. in our work, we also experiment with example ordering, and conclude that least-to-most ordering is the most effective across all datasets tested. works demonstrating order instability: several recent works have demonstrated that the order of in-context examples makes a larger difference in performance, including @xcite ; @xcite . these works demonstrate such order instability that certain permutations bring near sota performance on tasks while others perform at near random guessing. fine-tuned retrieval: several works employ the use of fine-tuned retrievers, re-rankers, and/or lms, including @xcite ; @xcite ; @xcite . some, like replug @xcite , use lm feedback in the form of using the lm to score documents to train the retriever. the goal of both @xcite and @xcite is to improve language modeling and not icl ability. @xcite uses a similar lm-scorebased feedback to train a retriever (like replug) but for icl. the difference between all of these works and this work is that we demonstrate that an off-the-shelf retriever is sufficient out-of-the-box for sota performance with no additional tuning. works calling into question efficacy of icl: certain recent works have called into question the efficacy of icl and models' ability to learn tasks they were not exposed to during pre-training @xcite . in @xcite authors show that randomly perturbing input-label pairings for some tasks can still lead to reasonably good performance, calling into question whether any "learning" is happening at all with icl. the work in @xcite demonstrates that models perform better on data instances they have seen frequently during pre-training, implying that models are primarily memorizing and that their generalization capabilities in terms of icl remain limited. @xcite suggests that icl ability emerges due to the specific structure of the training data, specifically long-range dependencies. use of long contexts: several works have demonstrated that long contexts are difficult for lms to handle and show certain peculiarities. @xcite investigates the relationship between length generalization and positional embedding types, showing that in certain cases no positional embeddings can perform better. this work is closely related to use of long contexts for icl, as it demonstrates the difficulty involved in gener-alizing to long context lengths, as well as providing an explanation for lms' sensitivity to ordering (positional embeddings). in @xcite , the authors investigate the impact of long contexts on document question answering, finding that the positions of the answers within the context matter greatly for performance, and generally demonstrating that longer contexts cause lower performance. in this work we show that larger models are needed to effectively take advantage of long contexts for icl. few-shot intent detection: the current state of the art in few-shot intent detection is the convfit method @xcite . convfit uses a pretrained lm in a dual-encoder configuration (e.g. bert or roberta) with two training stages. the first stage is a conversational fine-tuning stage using a generic conversational corpus with a retrieval task (using tuples of (context, response) retrieve the correct response for each context). the second stage is fine-tuning on the specific intent classification dataset with a contrastive loss, allowing the resulting lm to be used in a knn fashion. ## conclusion in this work, we show that icl with off-the-shelf frozen pre-trained retriever models can provide strong performance for text classification tasks with many labels. we show state of the art performance across three different intent classification datasets, and competitive performance with fine-grained sentiment classification. we also show that larger models are necessary to make use of more in-context examples, whereas small models mostly plateau or even show decreasing performance after a point. through several ablation experiments, we demonstrate that lms make use of all aspects of the input examples: semantically significant label names, correct input-label correspondences, as well as the similarity between the in-context demonstrations and the current input point, however to varying degrees depending on the dataset and domain.