A Survey on In-context Learning (2024)

Abstract: With the increasing capabilities of large language models (LLMs), in-context learning (ICL) has emerged as a new paradigm for natural language processing (NLP), where LLMs make predictions based on contexts augmented with a few examples. It has been a significant trend to explore ICL to evaluate and extrapolate the ability of LLMs. In this paper, we aim to survey and summarize the progress and challenges of ICL. We first present a formal definition of ICL and clarify its correlation to related studies. Then, we organize and discuss advanced techniques, including training strategies, prompt designing strategies, and related analysis. Additionally, we explore various ICL application scenarios, such as data engineering and knowledge updating. Finally, we address the challenges of ICL and suggest potential directions for further research. We hope that our work can encourage more research on uncovering how ICL works and improving ICL.

## introduction with the scaling of model size and data size @xcite @xcite , large language models (llms) demonstrate the in-context learning (icl) ability, that is, learning from a few examples in the context. many studies have shown that llms can perform a series of complex tasks through icl, such as solving mathematical reasoning problems @xcite . these strong abilities have been widely verified as emerging abilities for large language models @xcite . the key idea of in-context learning is to learn from analogy. figure 1 gives an example that describes how language models make decisions via icl. first, icl requires a few demonstration examples to form a prompt context. these examples are usually written in natural language templates. then, icl concatenates a query question and the piece of prompt context together to form the input, which is then fed into the language model for prediction. different from supervised learning, which requires a training stage that uses backward gradients to update model parameters, icl does not perform parameter updates. the model is expected to learn the pattern hidden in the demonstration and accordingly make the right prediction. as a new paradigm, icl has multiple attractive advantages. first, since the demonstration is written in natural language, it provides an interpretable interface to communicate with llms @xcite . this paradigm makes it much easier to incorporate human knowledge into llms by changing the demonstration and templates @xcite @xcite . second, in-context learning is similar to the decision process of human beings by learning from analogy @xcite . third, compared to supervised training, icl is a training-free learning framework. this could not only greatly reduce the computational costs for adapting the model to new tasks, but also make language-model-as-aservice @xcite possible and can be easily applied to large-scale real-world tasks. despite being promising, there are also interesting questions and intriguing properties that require further investigation in icl. although a range of vanilla gpt models show excellent icl capability, several studies have found that this capability can be significantly improved through adaptation during pretraining @xcite . moreover, the performance of icl is sensitive to specific settings, including the prompt template, the selection and order of demonstration examples, and other factors @xcite . additionally, optimizing the conciseness of demonstration examples and improving the computational efficiency of icl are critical areas of ongoing research @xcite . furthermore, despite preliminary explanations @xcite , the underlying working mechanism of icl remains unclear and requires further investigation. with the rapid growth of studies in icl, our survey aims to sensitize the community toward the current progress. in the following sections, we delve into an in-depth discussion of related studies, and we summarize the taxonomy in figure 2 and the key findings in appendix a. we highlight the challenges and potential directions and hope our work provide a useful roadmap for beginners interested in this area and shed light on future research. ## definition and formulation following brown et al. (2020), we here provide a formal definition of in-context learning: in-context learning is a paradigm that allows language models to learn tasks given only a few examples in the form of demonstration. formally, given a query input text x and a set of candidate answers @xmath0, . . . , y m }, a pretrained language model m takes the candidate an-swer with the maximum score as the prediction, 1 conditioned a demonstration set c. c contains an optional task instruction i and k demonstration examples, thus @xmath1, s(x 1 , y 1 ), . . . , s(x k , y k )} or @xmath2, y 1 , i), . . . , s ′ (x k , y k , i)}, where s ′ (x i , y i , i) is an example written in natural language according to the task. depending on whether k and the demonstration examples belong to the same task, it can be categorized as task-specific icl and cross-task icl. in the latter, different examples have their own instructions. the likelihood of a candidate answer y j comes from a scoring function f on the whole input sequence: the final predicted label ŷ is the candidate answer with the highest probability: according to the definition, we can see that icl differs from related concepts as follows: (1) prompt learning: prompts can be discrete templates or soft parameters that encourage the model to predict the desired output. icl can be regarded as a subclass of prompt tuning where the demonstration examples are part of the prompt. @xcite made a thorough survey on prompt learning, but icl was not included in their study. (2) few-shot learning: few-shot learning is a general machine learning approach that involves adapting model parameters to perform a task with a limited number of supervised examples @xcite . in contrast, icl does not require parameter updates and is directly performed on pretrained llms. ## model training although llms have demonstrated promising icl capability directly, many studies revealed that these icl capabilities can be further enhanced through specialized training before inference @xcite @xcite . ## prompt designing in this section, we focus on the principles of icl during inference, including demonstration organization ( §4.1) and instruction formatting ( §4.2) . ## analysis to understand icl, recent studies attempt to investigate what influence icl performance @xcite @xcite and why icl works @xcite . in this section, we present a detailed elaboration of influencing factors ( §5.1) and learning mechanisms ( §5.2) of icl, as illustrated in figure 4 . ## application given its user-friendly interface and lightweight prompting method, icl has broad applications on traditional nlp tasks @xcite @xcite . particularly, by using demonstrations that explicitly guide the reasoning process, icl manifests remarkable effects on tasks requiring complex reasoning @xcite @xcite and compositional generalization @xcite . we explore several emerging and prevalent applications of icl, including data engineering, model augmentation, and knowledge updating. 1) data engineering: unlike traditional methods such as human annotation and noisy automatic annotation, icl generates relatively high-quality data at a lower cost, leading to improved performance. @xcite @xcite . 2) model augmentation: the context-flexible nature of icl shows promise in model augmentation. it can enhance retrievalaugmented methods by prepending grounding documents to the input @xcite . additionally, icl for retrieval demonstrates potential in steering models toward safer outputs @xcite . 3) knowledge updating: llms often contain outdated or incorrect knowledge @xcite . icl has demonstrated efficacy in revising such knowledge through carefully crafted demonstrations, yielding higher success rates compared to gradient-based methods @xcite . as mentioned above, icl has yielded significant benefits on both traditional and emergent nlp applications. the tremendous success of icl in nlp has inspired researchers to explore its potential in various modalities beyond text (elaborated in appendix d), including vision @xcite , vision-language @xcite , as well as speech applications @xcite . ## challenges and future directions in this section, we review existing challenges and discuss future directions for icl. ## conclusion in this paper, we comprehensively review the existing literature on icl, examining advanced techniques, conducting analytical studies, discussing relevant applications, and identifying critical challenges and potential directions for future research. to our knowledge, this is the first comprehensive survey dedicated to icl. we aim to highlight the current state of research in icl and provide insights to guide future work in this promising area.