Large Language Models Can be Lazy Learners: Analyze Shortcuts in In-Context Learning (2023)

Abstract: Large language models (LLMs) have recently shown great potential for in-context learning, where LLMs learn a new task simply by conditioning on a few input-label pairs (prompts). Despite their potential, our understanding of the factors influencing end-task performance and the robustness of in-context learning remains limited. This paper aims to bridge this knowledge gap by investigating the reliance of LLMs on shortcuts or spurious correlations within prompts. Through comprehensive experiments on classification and extraction tasks, we reveal that LLMs are “lazy learners” that tend to exploit such shortcuts. Additionally, we uncover a surprising finding that larger models are more likely to utilize shortcuts in prompts during inference. Our findings provide a new perspective on evaluating robustness in in-context learning and pose new challenges for detecting and mitigating the use of shortcuts in prompts.

## introduction large language models have shown great potential on downstream tasks by simply conditioning on a few input-label pairs (prompts), referred to as in-context learning @xcite @xcite . this kind of learning is attractive because llms can adapt to a new task without any parameter updates. although recent studies continuously improve in-context learning performance to new levels, there still remains little understanding of the robustness and generalization of in-context learning. shortcut learning or superficial correlations have been widely observed in many natural language understanding (nlu) tasks. fine-tuned language models are known to learn or even amplify biases in the training datasets, leading to poor performance on downstream tasks @xcite @xcite @xcite . for instance, recent studies figure 1 : performance drops on sst2 in three llms: opt-2.7b, opt-6.7b, and opt-13b. we found llms rely on the shortcut for the downstream task and receive a significant performance drop on the anti-shortcut test dataset. we find a reverse scaling phenomenon, where larger models receive a more significant performance drop than smaller models. on natural language inference tasks demonstrate that language models heavily rely on simple words or phrases, such as "is", "not", and "can not", for making inferences @xcite . similarly, in the question-answering tasks, language models are shown to rely on the lexical matching of words between the input passage and question without understanding the underlying linguistic semantics @xcite . shortcut learning has been identified as a major cause of the low robustness in large language models and has become a benchmark for evaluating models' generalization ability @xcite @xcite . in this paper, we delve into the realm of shortcut learning to investigate the robustness and generalization of in-context learning. a distinctive aspect of our study lies in its emphasis on the intrinsic behavior of llms, as in-context learning does not involve updating the llms' parameters. to the best of our knowledge, this is the first study to examine shortcut learning in a non-training setting, as previous literature has primarily focused on short-cut learning during the fine-tuning process. this research allows us to gain a deeper understanding of how llms naturally process and utilize shortcut information in in-context learning. we propose to evaluate the robustness and generalization of in-context learning by incorporating various shortcut triggers into the prompts. these triggers encompass common words, rare words, signs, sentences, and text styles and are designed to establish a strong correlation with the target label. this approach allows us to equip llms with two types of knowledge during in-context learning: non-robust knowledge and robust knowledge @xcite . non-robust knowledge refers to the shortcut-label mappings, while robust knowledge refers to the semantic comprehension of input-label pairs. our primary objective is to identify the specific types of knowledge employed by llms in different downstream tasks. to achieve this, we follow previous studies @xcite and create an antishortcut test set, where llms relying on shortcuts will receive a significant performance drop. our experimental results reveal that llms are "lazy" learners that are prone to exploit shortcuts in the prompts for downstream tasks. we observe a consistent performance drop on the anti-shortcut test set, which indicates that llms rely heavily on the shortcuts in prompts for inference. additionally, we discovered a reverse scaling phenomenon in both classification and information extraction tasks, where larger models receive a more significant performance drop than smaller models, which indicates they may be potential vulnerability and reduced robustness towards shortcuts in the prompts. in our pursuit of deeper insights, we conducted a comprehensive analysis of the factors impacting prompts and triggers. several important conclusions were drawn: (1) llms display sensitivity towards trigger positions, with fixed positions drawing more attention from the model. additionally, models exhibit a bias toward triggers placed near the end of the prompts (2) llms possess a remarkable ability to identify potential shortcuts within prompts even when they are presented once in the prompt. (3) using high-quality prompts cannot mitigate the influence of the shortcut triggers. in conclusion, our paper makes the following contributions: in-context learning. recently, scaling improvements through the larger dataset @xcite and larger model size @xcite have significantly improved the semantic understanding and reasoning ability of pre-trained language models. @xcite first proposed to use a concatenation of training examples (prompts) for few-shot learning. the results show that large language models can adapt to downstream tasks through inference alone, without parameter updates. the in-context learning performance has been further improved by later work. researchers have proposed advanced prompt formats @xcite @xcite @xcite , reasoning procedure @xcite @xcite , meta-training with an in-context learning objective @xcite , showing great potential for a variety of downstream tasks @xcite . ## framework to generate shortcuts in-context learning can be regarded as a conditional text generation problem. given a prompt p that contains k input-label pairs x 1 , y 1 , x 2 , y 2 , ..., x k , y k and a source text x, llms will generate a probability of target y conditioning on the prompt p, which can be written as: to evaluate if the model is using the shortcut mapping, @xmath0, for inference, we follow previous literature @xcite and create an anti-short test set. the idea is to inject a shortcut into a test example x, which has a label ĉ, where ĉ ̸ = c. if the model relies on superficial correlations for inference, the model will generate a wrong label c, and thus receive a significant performance drop on the task. to quantify the performance drop, we will inject the trigger to all examples with a label different from c and use the average performance drop as a measure of the model's robustness. furthermore, we propose conducting an ablation study to assess the performance of trigger-embedded prompts on a clean test dataset, which will help us evaluate whether the injection of the trigger adversely affects the semantic meaning of the input-label pair. ## shortcut detection previous sections of this study have demonstrated that large language models are highly efficient in utilizing shortcuts in training prompts for downstream tasks, which can have a substantial impact on performance. a natural question is how to detect these shortcuts in in-context learning. to address this question, we adopted the approach lime @xcite and leveraged model interpretation to detect potential shortcuts in the training prompts. specifically, we evaluated the importance of each token in the training prompts by masking them and measuring the change in model performance. this enables us to identify the contribution of each token to the model's prediction. we present the attention visualization results in figure 11 , alongside the word importance score on the anti-shortcut test data foot_1 . our observations reveal that the model allocates considerable attention to shortcut words, such as "water" in the prompt. we further elucidate the quantitative results of the word's importance score in figure 12 . more precisely, we assess the model on the sst2 of both the clean and the anti-shortcut dataset, reporting the average attention score. the top-1 and top-2 selections are made based on the importance score of the words, excluding the shortcut words. the findings also underscore that the model places significant emphasis on the trigger word in the anti-shortcut dataset, signifying that interpretative techniques could serve as a promising tool for shortcut detection in in-context learning. ## limitations effectiveness of task and model scopes. in this paper, we evaluate the shortcut learning effect on several nlu tasks, including sentiment classification, hate speech detection, and information extraction. our task selection is mainly based on the robustness and effectiveness of in-context learning on certain tasks. therefore, we do not adopt tasks such as natural language inference, where in-context learning exhibits sub-optimal performance @xcite . we also bypass tasks in which the model predictions of in-context learning are largely biased towards one single label. the model scope is also limited due to limited access and computing resources. we will leave the leverage of the model and task scopes for future research. calibration of shortcut learning effect. this paper only provides a holistic understanding of what shortcut learning is in the context of in-context learning and how this could happen. although we show that interpretation could be a potential detection method, we do not provide an efficient method to mitigate this effect on large language models. we will leave it for future research. ## conclusion in this paper, we uncover the propensity of large language models to leverage shortcuts within prompts for downstream tasks, even in the absence of parameter updates. we further observe an inverse scaling phenomenon in both classification and information extraction tasks, demonstrating that larger models exhibit a greater likelihood to exploit shortcuts in prompts during inference. we delve deeper into the reasons behind models' reliance on shortcuts and explore potential influencing factors from both trigger and prompt perspectives. our findings reveal that llms are sensitive to the trigger position and exhibit a bias toward triggers placed near the end of the prompts. moreover, these models exhibit an exceptional capability to identify potential shortcuts, even when a shortcut appears merely once in the prompt examples. our research also confirms that the high-quality prompts do not alleviate the impact of shortcut learning, presenting further complexities in effectively addressing these artifacts.