Inference and Verbalization Functions During In-Context Learning (2024)

Abstract: Large language models (LMs) are capable of in-context learning from a few demonstrations (example-label pairs) to solve new tasks during inference. Despite the intuitive importance of high-quality demonstrations, previous work has observed that, in some settings, ICL performance is minimally affected by irrelevant labels (Min et al., 2022). We hypothesize that LMs perform ICL with irrelevant labels via two sequential processes: an inference function that solves the task, followed by a verbalization function that maps the inferred answer to the label space. Importantly, we hypothesize that the inference function is invariant to remappings of the label space (e.g., “true”/“false” to “cat”/“dog”), enabling LMs to share the same inference function across settings with different label words. We empirically validate this hypothesis with controlled layer-wise interchange intervention experiments. Our findings confirm the hypotheses on multiple datasets and tasks (natural language inference, sentiment analysis, and topic classification) and further suggest that the two functions can be localized in specific layers across various open-sourced models, including GEMMA-7B, MISTRAL-7B-V0.3, GEMMA-2-27B, and LLAMA-3.1-70B.

## introduction large language models (lms) are capable of incontext learning (icl)-the ability to solve novel tasks from solely a handful of demonstration examples provided in-context during inference @xcite . previous work has found that, in certain settings, icl performance is minimally affected by using demonstrations with irrelevant label words @xcite . how do lms manage to perform in-context learning with irrelevant and even misleading label words? this research seeks to offer a causal explanation for such model * equal contribution. ## methods we aim for an explanation that reveals the causal structure employed by the model. successful causal explanation is marked by its ability to allow us to make counterfactual predictions, that is, answering the "what-if-things-had-been-different questions" should we perform certain interventions @xcite . typically, direct observation of outcomes from distinct interventions on the same unit is not possible @xcite . but, it is not an issue for us, because we can let the model generate outputs under any conceivable intervention on the units. moreover, we can make counterfactual statements with mere interventional information with deterministic neural network models @xcite , which is also typically not possible @xcite . these enable a concrete operationalization of our hypothesis testing. ## experiments models. we choose open-sourced base models of various sizes and families, including gemma-7b @xcite , mistral-7b-v0.3 @xcite , gemma-2-27b (team et al., 2024), and llama-3.1-70b @xcite . ## discussion validation of h 1 : causal mechanism. this hypothesis is confirmed if we can "concatenate" an inference function with another verbalization function and observe the generation of the counterfactual output that reflects the answer obtained by the inference function and the corresponding label space of the verbalization function. we achieve such a scenario with the experiment with remapped label spaces, as manifested by the high flip rates we observe during the middle layers of the model. concern may be raised that the intervened model just re-solves the task on its own during the downstream processes, that is, both inference function and verbalization function are performed in the middle and the late layers. our experiment with alternative tasks shows that this is not true. this experiment ensures that only the source model knows the answer that leads to the hypothetical counterfactual output, which means high flip rates can only be achieved if the plugged-in representation involves a sufficiently developed representation of the answer. therefore, the high flip rates we observe during and after the middle layers support the hypothesis that the inference function is implemented by the upstream processes (before the layers with effective interventions). validation of h 2 : invariant to label remapping. from the results of the experiment with remapped label spaces, we know that intervening on a sequence of middle layers in the model can successfully achieve the counterfactual scenario, where the inference function can be concatenated with another verbalization function to produce the hypothetical counterfactual output. this indicates that the inference function can play its causal role invariant to the downstream verbalization function. validation of h 3 : separate localization and consistent localization. this hypothesis is a necessary condition for us to use interchange intervention to realize the counterfactual scenario, and thus it gets validated if our interventions are effective. it can be further supported by the observation that for each model, the trend of flip rates is similar across different datasets and tasks. for example, flip rates surge at the same or very close layers (figure 3 and 4 ). we summarize the start and the end of the layers enabling effective interventions in ## related work understanding icl. a variety of work has sought to better understand how language models perform in-context learning. @xcite show that icl performance on a variety of text classification tasks is minimally affected by randomly changing the demonstration labels, hypothesizing that demonstrations may primarily help with determining the label space (i.e., the verbalization function), the distribution of the input examples, and how the output text should be formatted. @xcite hypothesize that lms learn latent tasks during pre-training, primarily using icl demonstrations to identify which latent task is most pertinent to the provided icl input. other work @xcite @xcite hypothesizes that lms perform implicit gradient descent on a latent model during icl. @xcite and @xcite confirmed these observations and further indicated that this consistency depends on the size of the model. our study offers a deeper analysis of these behaviors at the representational level. @xcite study how model size affects whether lms prefer to rely on semantic priors about a task, as opposed to the example-label mappings provided in the demonstrations. they find that smaller models tend to rely on semantic priors during pre-training, since they ignore flipped labels presented in the icl demonstrations. in contrast, larger models make use these flipped labels, indicating that they rely less on their semantic task priors. @xcite disentangle task learning and task recognition in in-context learning; intuitively, lms use their priors to perform novel tasks during icl (since a few demonstrations are unlikely to provide complete information about a complex task), but they also do not completely rely on their task priors (since they can handle icl settings with arbitrary label mappings). implicit functions in language models. another line of prior work has sought to characterize what functions might be implicitly implemented by lms. @xcite identify "induction heads", a unique type of attention head that replicates repeating patterns from prior contexts. @xcite argue icl compresses the demonstrations into a task vector, which is then used to generate the output for the icl input. @xcite provide evidence that transformer lms perform icl in three layer-wise stages: argument formulation, function application, and saturation. @xcite show that icl creates function vectors of the demonstrated task, and these function vectors can trigger execution of the task in other settings (e.g., zero-shot prediction or natural text). ## conclusion we hypothesize that when lms perform icl with irrelevant or misleading label words, they first apply an inference function to obtain a representation of the answer and then apply a verbaliza tion function to verbalize the answer as one of the label words specified in the demonstrations (h 1 ). in addition, the inference function is invariant to remappings of label words (h 2 ), and both the functions can be localized separately and consistently within certain layers of the model (h 3 ). to validate our hypotheses, we design experiments based on interchange intervention to realize counterfactual scenarios that would only occur if all of our hypotheses hold true. our experiments across a variety of tasks, datasets and models indicate that our hypotheses hold across a variety of settings. our findings contribute to a growing body of work on mechanistically understanding how language models perform in-context learning.