The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysis (2024)

Abstract: Understanding in-context learning (ICL) capability that enables large language models (LLMs) to excel in proficiency through demonstration examples is of utmost importance. This importance stems not only from the better utilization of this capability across various tasks, but also from the proactive identification and mitigation of potential risks, including concerns regarding truthfulness, bias, and toxicity, that may arise alongside the capability. In this paper, we present a thorough survey on the interpretation and analysis of in-context learning. First, we provide a concise introduction to the background and definition of in-context learning. Then, we give an overview of advancements from two perspectives: 1) a theoretical perspective, emphasizing studies on mechanistic interpretability and delving into the mathematical foundations behind ICL; and 2) an empirical perspective, concerning studies that empirically analyze factors associated with ICL. We conclude by discussing open questions and the challenges encountered, and suggesting potential avenues for future research. We believe that our work establishes the basis for further exploration into the interpretation of in-context learning. To aid this effort, we have created a repository containing resources that will be continually updated.

## introduction the concept of in-context learning (icl) was originally introduced by @xcite , defined as 'the model is conditioned on a natural language instruction and/or a few demonstrations of the task and is then expected to complete further instances of the task simply by predicting what comes next'. icl is receiving increasing attention due to its remarkable adaptability and parameter-free nature. as shown in figure 1 , llms such as @xcite , llama3 (ai@meta, 2024), and qwen2 (qwenteam, 2024) have exhibited proficiency across various tasks, such as machine translation, sentiment analysis, and question answering, with a minimal set of task-oriented examples, all without re-training. while icl has been dominantly deployed in the natural language processing (nlp) community, our understanding of icl remains limited. recently, an increasing number of studies have attempted to interpret and analyze icl. garg et al. (2022), dai et al. (2023), and akyürek et al. (2023) explained icl through the lens of linear regression formulation. xie et al. (2022), wang et al. (2023b), and hahn and goyal (2023) provided an interpretation of icl rooted in latent variable models. meanwhile, a distinct line of research has aimed to understand the influential factors affecting icl through empirical analyses. min et al. (2022), wei et al. (2023), wang et al. (2023a), and yoo et al. 2022 demonstrated that the icl performance is influenced by task-specific characteristics and multiple facets of icl instances, including quantities, order, and flipped labels. consequently, it is essential to systematically categorize and summarize these studies, not only for a deeper understanding and more effective utilization of icl across various tasks, but also to assist in anticipating and mitigating potential risks. these risks encompass concerns related to truthfulness, bias, and toxicity, that may arise alongside icl. in this paper, we present a thorough and organized survey of the research on the interpretation and analysis of icl. first, we provide a brief introduction of the background and offer the definition of icl. then, we present a comprehensive overview of advancements, from two distinct viewpoints: 1) the theoretical perspective, encapsulating studies focused on mechanistic interpretability and mathematical investigations into the foundations of icl; and 2) the empirical perspective, pertaining to studies that prioritize empirical analysis by probing factors associated with icl. in conclusion, we highlight the existing challenges and suggest potential avenues for further research. ## background and notation in this section, we define the icl paradigm using the following notations. a task t d consists of two components: a demonstration space d, which encompasses all possible demonstrations, and a joint probability distribution p (x,y ) . a task demonstration @xmath0, y i )} n @xmath1. in nlp, these example pairs could be question-answering pairs for qa tasks, parallel text for machine translation tasks, or sentence-label pairs for text classification tasks. for example, for machine translation, an example pair (x, y) could be (english: happy new year, french: bonne année). in contrast to traditional supervised learning approaches which aim to generalize from a fixed training dataset to predict future instances, icl leverages continual exposure to demonstration examples and is guided by task query to adapt the model dynamically to different contexts. in this survey, we denote the query as t @xmath2, p q }, consisting a query space q and marginal distribution p q . a task query @xmath3. for example, q could be "english: thank you french:" in the machine translation task. additionally, we define @xmath4. let an llm be defined as a function f θ pretrained on large-scale text corpora. icl can be defined as follows: definition (in-context learning) in the context of a task query q, in-context learning refers to the capability of f θ to predict the correct answer a, conditioned on a task demonstration d. based on the above definition, the process of icl can be formally expressed as follows: the performance of icl can be measured by: â denotes the model-predicted output, and m is an evaluation metric chosen based on the task query q and its gold label a. based on our definition, we organize the existing literature on the interpretation and analysis of icl into theoretical and empirical perspectives, as summarized in ## open questions despite ongoing endeavours to interpret and analyse icl, we are still far from fully understanding it due to the open-ended nature of some questions. for example, while @xcite and @xcite contribute to our understanding of icl by probing the internal architecture of llms, it is important to note that their findings represent initial steps towards the comprehensive reverseengineering of llms. it becomes particularly intricate when dealing with llms characterized by complex structures comprising hundreds of layers and spanning billions to trillions of parameters. this complexity introduces significant challenges. similarly, although studies have provided theoretical proofs and empirical evidence on the relation of icl and regression function learning @xcite @xcite , gradient descent or meta-optimization @xcite , and bayesian inference @xcite @xcite @xcite , their conclusions are limited to simplified model architectures and controlled synthetic experimental settings. this raises the open question of whether these findings hold in the context of standard model architectures without approximations and if they can be directly applied to real-world scenarios. on the other hand, there are contradictory findings from researchers analysing the factors affecting icl. for instance, while @xcite and @xcite identified a positive relation between icl performance and term frequency in pre-training data, @xcite found that pre-training data with long-tail and rarely occurring tokens contribute more significantly to icl. additionally, while @xcite suggested that icl exhibits low sensitivity to labels in the demonstrations, other studies revealed that accurate input-label mappings play an important role in performing icl @xcite @xcite . one of the core challenges in analysing icl empirically lies in the necessity of controlling for numerous relevant variables, leading most existing conclusions to typically rely on correlations rather than causal relations. this raises open questions about the reliability of these findings and the extent to which confounding factors may influence the results. evaluation current research efforts typically measure icl by assessing task performance or optimizing criteria such as gradient @xcite and token loss @xcite during the pre-training stage. however, @xcite have recently argued that emergent abilities (e.g., icl) discussed in some prior studies appear to be mirages, due to the researchers' choice of evaluation metrics. their hypothesis is that choosing a metric that nonlinearly or discontinuously deforms per-token error rates and the limited size of test datasets may not provide an accurate estimation of the performance of smaller models. the core challenge is that aggregated performance metrics do not adequately assess icl ability across various scenarios or predict their behaviour in new tasks alongside data distributions drifting. dedicated criteria explicitly designed for the assessment of icl are currently lacking. in addition, the evaluation process typically encompasses multiple models with a distinct objective in existing llm paradigms, such as reinforcement learning from human feedback (rlhf) @xcite . a potential solution involves identifying and addressing specific capability gaps to enhance predictions of model performance on novel tasks. for example, @xcite incrementally inferred the capability of llms with subsets of tasks before assessing more complex dependencies. demonstration selection while research exploring the relationship between demonstration and icl is expanding @xcite , it is mostly limited to interpretation based on a finite number of demonstrations. the core challenge lies in the exponential growth of possible demonstrations with the increase in examples. a potential approach is to formalize the demonstration selection as a sequential decision-making problem @xcite , aiming to learn an approximation of the expected reward from demonstrations to identify those most beneficial for control experiments. other potential solutions can include disentangling features supportive for performing icl on downstream tasks, and controlling for these features in demonstrations @xcite . trustworthiness trustworthiness issues such as fairness, truthfulness, robustness, bias, and toxicity are significant concerns for llms. exploring these properties within icl in llms is particularly challenging due to their unanticipated nature @xcite . it is difficult to analyze the relationship between various aspects of icl and factors relating to trustworthiness when llm training objectives and downstream tasks are inconsistent. safety concerns have also become one of the most pressing issues. studies @xcite have shown that llms can be manipulated to perform harmful and dangerous icl through exposure to toxic demonstrations. understanding icl could play a crucial role in addressing the trustworthiness and safety issues associated with llms. for example, knowledge of how llms incorporate biases during icl can guide the development of debiasing models. furthermore, insights gained from studying how llms respond to toxic demonstrations can inform the design of countermeasures aimed at detecting and filtering out harmful input. ## conclusion this paper presents a comprehensive review of current research efforts focused on interpreting and analyzing icl in llms. we organize these advancements into theoretical and empirical perspectives, highlighting existing challenges and discussing potential avenues for further research in this area. we believe this survey will serve as a valuable resource for encouraging further exploration into the interpretation of icl of llms.