Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning (2023)

Abstract: In-context learning (ICL) emerges as a promising capability of large language models (LLMs) by providing them with demonstration examples to perform diverse tasks. However, the underlying mechanism of how LLMs learn from the provided context remains under-explored. In this paper, we investigate the working mechanism of ICL through an information flow lens. Our findings reveal that label words in the demonstration examples function as anchors: (1) semantic information aggregates into label word representations during the shallow computation layers’ processing; (2) the consolidated information in label words serves as a reference for LLMs’ final predictions. Based on these insights, we introduce an anchor re-weighting method to improve ICL performance, a demonstration compression technique to expedite inference, and an analysis framework for diagnosing ICL errors in GPT2-XL. The promising applications of our findings again validate the uncovered ICL working mechanism and pave the way for future studies.

## introduction in-context learning (icl) has emerged as a powerful capability alongside the development of scaledup large language models (llms) @xcite . by instructing llms using few-shot demonstration examples, icl enables them to perform a wide range of tasks, such as text classification @xcite and mathematical reasoning @xcite . since icl does not require updates to millions or trillions of model parameters and relies on human-understandable natural language instructions @xcite , it has become a promising approach for harnessing the full potentiality of llms. despite its significance, the inner working mechanism of icl remains an open question, garnering considerable interest from research communities @xcite @xcite . in this paper, we find that the label words serve as anchors that aggregate and distribute information in icl. we first visualize the attention interactive pattern between tokens with a gpt model @xcite ) on sentiment analysis (figure 1 ). initial observations suggest that label words aggregate information in shallow layers and distribute it in deep layers. 2 to draw a clearer picture of this phenomenon, we design two metrics based on saliency scores to portray the information flow in icl and further propose the following hypothesis: information flow with labels as anchors h 1 : in shallow layers, label words gather the information of demonstrations to form semantic representations for deeper layers. h 2 : in deep layers, the model extracts the information from label words to form the final prediction. two experiments are designed to validate the hypothesis using gpt2-xl @xcite and gpt-j @xcite across several text classification benchmarks. (1) by blocking the information aggregation path to label words in certain layers, we find that such isolation in shallow layers significantly impairs model performance. this indicates that label words collect useful information during forward propagation in shallow layers. (2) we investigate the relationship between the attention distributions on the label words of the target position and the model's final prediction. our results illustrate a strong positive correlation, where a candidate label's probability increases with more attention weight on its corresponding label token. in summary, these experimental findings suggest that our hypothesis holds well with large language models on real-world datasets. drawing on insights from the information flow perspective, we explore three approaches to enhance icl's effectiveness, efficiency, and interpretability. (1) an anchor re-weighting method is introduced, which employs a learnable vector to adjust the significance of different label words in demonstrations, leading to a 16.7% average accuracy boost compared to standard icl baselines. (2) for quicker icl inference, inputs are compressed into pre-calculated anchor representations since model predictions primarily rely on label word activations. testing shows a 1.8 × speedup in inference with only a minimal performance trade-off. (3) an error analysis of icl on gpt2-xl demonstrates that the label confusion matrix aligns closely with the distance distribution of anchor key vectors, implying that errors might result from similar anchor representations. these promising applications further validate our hypothesis and shed light on future icl studies for better transparency of llms. ## label words are anchors this section confirms the intuitive findings using two saliency score-based metrics as discussed in § 2.1. the quantitative results lead to a proposed hypothesis for the icl working mechanism: h 1 : in shallow layers, label words aggregate information from demonstration examples to form semantic representations for later computations. h 2 : in deep layers, the model makes predictions by extracting information from label words. the validation for these hypotheses is presented in § 2.2 and § 2.3, respectively. ## applications of our anchor-based understanding with insights from the validated hypothesis, we propose strategies to boost icl's accuracy and inference speed. we propose an anchor re-weighting method in § 3.1 to adjust the demonstrations' contributions and improve accuracy. in § 3.2, we explore a context compression technique that reduces original demonstrations to anchor hidden states to speed up icl inference. besides, in § 3.3, we utilize anchor distances to perform an analysis to understand the errors icl made in real-world scenarios. these approaches corroborate our hypothesis, pointing to potential paths for future icl enhancements. ## related work the existing literature on in-context learning analysis can be broadly divided into two streams, each focusing on different aspects. the first stream explores the influencing factors of icl based on input perturbation, such as the order @xcite , the formatting @xcite , and the selection of the demonstration @xcite . designing proper demonstration construc-tion strategies @xcite and calibration techniques @xcite could bring clear boosts to the icl performance. the second stream investigates the inner working mechanism of icl through different conceptual lenses, such as making an analogy of icl to gradient descent @xcite and viewing the process of icl as a bayesian inference @xcite . in this paper, we provide a novel perspective by examining the information flow in language models to gain an understanding of icl. our approach offers new insights and demonstrates the potential for leveraging this understanding to improve the effectiveness, efficiency, and interpretability of icl. ## conclusion in this paper, we propose a hypothesis that label words serve as anchors in in-context learning for aggregating and distributing the task-relevant information flow. experimental results with attention manipulation and analysis of predictions correlation consolidate the hypothesis holds well in gpt2-xl and gpt-j models. inspired by the new understanding perspective, we propose three practical applications. first, an anchor re-weighting method is proposed to improve icl accuracy. second, we explore a demonstration compression technique to accelerate icl inference. lastly, we showcase an analysis framework to diagnose icl errors on a real-world dataset. these promising applications again verify the hypothesis and open up new directions for future investigations on icl.