Exploring the Relationship between In-Context Learning and Instruction Tuning (2024)

Abstract: In-Context Learning (ICL) and Instruction Tuning (IT) are two primary paradigms of adopting Large Language Models (LLMs) to downstream applications. However, they are significantly different. In ICL, a set of demonstrations is provided at the inference time, but the LLM’s parameters are not updated. In IT, a set of demonstrations is used to adjust the parameters of the LLM during training, but no demonstrations are provided at the inference time. Although a growing body of literature has explored ICL and IT, studies on these topics have largely been conducted in isolation, leading to a disconnect between these two paradigms. In this work, we explore the relationship between ICL and IT by examining how the hidden states of LLMs change in these two paradigms. Through carefully designed experiments conducted with LLaMA-2 and LLaMA-2-Chat (7B and 13B), we find that ICL and IT converge in LLM hidden states despite their apparent differences in implementation. Specifically, ICL changes an LLM’s hidden states as if its accompanying demonstrations were used to instructionally tune the model. Furthermore, the convergence between ICL and IT is largely contingent upon several factors related to the demonstration. Overall, this work offers a unique perspective to explore the connection between ICL and IT and sheds light on understanding the behaviors of LLMs.

## introduction large language models (llms), such as chatgpt 1 , @xcite , palm @xcite , and llama-2 @xcite , have significantly changed the paradigm of natural language processing and have great potential for artificial general intelligence @xcite . in real-world applications, the success of deploying llms can largely be attributed to the effectiveness of two important learning paradigms: 1) 1 https://openai.com/chatgpt in-context learning (icl) and 2) instruction tuning (it). icl, a paradigm introduced in the gpt-3 paper @xcite , involves utilizing a set of demonstrations provided at the inference time to guide the model's responses, but the model's parameters are not updated during this process. in contrast, it refers to the process of further training llms on ("input", "output") pairs, along with instructions in a supervised fashion. it has been shown to be effective in enhancing an llm's generalizability on unseen tasks @xcite and a viable strategy for llm alignment @xcite . we illustrate icl and it using sentiment analysis in figure 1 . a growing body of literature has examined the inner workings of icl and it, such as looking for the conditions under which icl emerges @xcite @xcite @xcite , and identifying the training data or tasks for effective instruction tuning to enhance the zero-shot generalizability of llms @xcite . although icl and it are both effective strategies for improving the capability of llms, studies on icl and it have been conducted in isolation. this has led to a research question: what are the connections between icl and it, and in which way do they enhance an llm's capability? this question further extends a more recent discussion on representation convergence across training objectives @xcite and the platonic representation hypothesis @xcite . in this work, we empirically examine the connections between icl and it from a hidden state perspective. in an autoregressive model, the hidden state of the last input token summarizes the information of the entire input sequence and determines the logit vector for sampling the following token. in the context of icl and it, three situations arise, each producing a different hidden state compared to the others. the first situation involves zero-shot learning for an llm. in this case, the hidden state of the last token in the input sequence is determined by the llm, conditioned on the inference example. since this is the basic case -where no demonstrations are provided and the llm's parameters are not updated -we denote this hidden state as the anchor hidden state, h anchor . the second situation is icl, where demonstrations are provided to guide the llm's responses. since icl does not modify the llm's parameters, the hidden state is determined by the llm, conditioned on the provided demonstrations and the inference example. we denote this hidden state as h icl . the third situation is it, where demonstrations are used to adjust the llm's parameters, transforming the llm into a tuned-llm. here, the hidden state is determined by the tuned-llm, conditioned on the inference example, and we denote this hidden state as h it . comparing the similarity between h anchor and h icl allows to quantify the effect of the demonstrations through icl, while comparing the similarity between h anchor and h it allows to assess the impact of it on such demonstrations. if a demonstration is effective for icl and it, we would observe little similarity between h anchor and h icl , as well as between h anchor and h it because this demonstration gears the llm to produce a guided (either through icl or through it) response. moreover, measuring the similarity between h icl and h it allows us to quantify the extent to which icl and it converge in hidden states. figure 2 illustrates our analysis framework involving the above three situations. in our main experiment, we choose llama-2 (7b) @xcite as the foundation llm and sentiment analysis as the downstream task. we compile a data set d consisting of tuples like (instruction, text, label). we then apply icl and it to the llm using same demonstrations and inference example drawn from d and examine the similarities among h anchor , h icl , and h it . we repeat the experiment multiple times with variations in the wording of instructions and demonstrations by sampling d repeatedly using different random seeds. the results reveal a high similarity between h icl and h it , while the similarity of each of the two hidden states with h anchor is low. this suggests that icl and it essentially guide the llm to a similar status, although it tunes the llm's parameters while icl does not. to further investigate, we manipulate the demonstrations used in icl and it; for example, we vary the number of demonstrations (from one-shot icl to few-shot icl), alter the semantic similarity between the demonstration and the inference example, demonstrate the llm with incorrect labels, and ask the llm to perform a task different from the one shown in demonstrations. the results consistently support the finding that using a demonstration in icl has a similar effect as using this demonstration to instructionally tune the llm. in the additional analyses examining the robustness and generalizability of our findings, we change the downstream task to machine translation and text summarization, apply different tuning strategies, and replace llama-2 (7b) with llama-2 (13b) and llama-2-chat (7b); the results remain consistent. in summary, this work makes three important contributions. first, we provide empirical evidence that icl and it are closely related. although icl does not alter model parameters -unlike itthe instructions and demonstrations they employ drive the model towards convergent hidden states. second, we introduce a generic analysis framework that facilitates conducting controlled experiments to study llm behaviors empirically. third, our findings shed light on compiling dataset for efficient icl and it (as discussed in later sections), potentially advancing the alignment of foundation models. foot_0 ## analysis framework we illustrate the analysis framework in figure 2 . building upon this framework, we examine the impact of different demonstrations (zero-shot vs. few-shot icl) and learning paradigms (icl vs. it) on the model's hidden states separately. although llms maintain hidden states for every input token, we primarily focus on the hidden states associated with the last input token of the sequence in this study. this is consistent with the majority of work studying how a decoder-only-llm's internal states (activations) affect its behavior @xcite . we denote the instruction as x (such as, what is the sentiment of this review?), demonstration as @xmath0, label a) (such as, review: this is a wonderful movie. sentiment: positive), and inference example as @xmath1, review: i like this movie.). we then consider the following three situations. basic situation. this is the basic zero-shot learning setting where no demonstrations are provided to guide the model's inference. in this situation, we concatenate the instruction with the inference example (i.e., instruction x + text b) and feed it through the llm. we collect the final hidden state of the last input token, denoted as h anchor . icl situation. in icl, demonstration, along with the inference example (i.e., instruction x + text a + label a + text b), are provided as input to the llm, which then directly infers the distribution of the next token. similarly, we collect the final hidden state of the last token, denoted as h icl . comparing the similarity between h anchor and h icl allows us to examine the effect of the provided demonstration through icl. if the similarity is low, it indicates that the demonstration information is incorporated by the llm so that the final hidden state is geared away. it situation. in it, unlike the icl situation where the demonstration is used as a part of the input sequence, we instead use the demonstration (i.e., instruction x + text a + label a) to instructionally tune the llm, leading to a tuned llm. we then send the inference example (i.e., instruction x + text b) through the tuned llm, and the associated hidden state is denoted as h it . note that the input sequences to the llm are exactly the same (i.e., instruction x + text b) in both the basic situation and the it situation. the only difference is that the basic situation involves the vanilla llm while the it situation involves the instruction-tuned llm. therefore, by comparing h anchor with h it , we can quantify the effect of it with the demonstration. since same demonstrations are used in both icl and it, measuring the similarity between h icl and h it allows us to precisely quantify the convergence between them. moreover, by manipulating the demonstrations, we can examine how closely icl is connected to it in different situations. in the following analyses, we denote the cosine similarity between h anchor and h icl as s anchor-icl , and denote that between h anchor and h it as s anchor-it . similarly, we calculate the similarity between h icl and h it , denoted as s icl-it , which quantifies how much icl and it aligns from the hidden state perspective. if s icl-it is very high, it suggests that icl and it guide the model status towards the same direction although the model parameters are not updated in icl but adjusted in it. ## related work in-context learning (icl) is a phenomenon emerged in large language models @xcite . a growing body of literature has investigated the icl phenomenon in llms. some studies have focused on identifying the conditions under which icl emerges in llms, predominantly by finding good demonstrations @xcite @xcite and identifying pre-training data distributions that can lead to the emergence of icl @xcite . another line of research aims to explain icl through building the relationship with the model training stage @xcite @xcite . for instance, @xcite find icl implicitly updates smaller models encoded in the activations. @xcite provide evidence that the so-called "induction heads" contribute to the majority of the icl behaviors in llms. our work differs from existing studies in two ways. first, we attempt to understand icl by investigating its connection with it, which is new and opens up the possibilities for harnessing the complementary advantages of icl and it. second, we empirically study off-the-shelf llms with more complex model structures (llama-2 7b and 13b), whereas most prior works conduct experiments using simplified models @xcite . instruction tuning (it) is an efficient technique to adapt llms to downstream tasks by further tuning the model on ("input", "output") pairs with instructions in a supervised manner. the intuition behind it is to bridge the gap between the language modeling objective in pre-training and the users' objective in downstream tasks, such that the model can follow the instructions from users. the effectiveness of it is well-demonstrated by a variety of instruction-tuned llms, with representatives such as instructgpt @xcite , alpaca @xcite , flan-t5 @xcite , and vicuna foot_7 . a growing body of literature focuses on designing tasks and datasets for effective instruction tuning. for example, lima @xcite shows that a small set of high-quality instruction data is sufficient for foundation model alignment. our work aims to provide empirical evidence to further understand it, through the lens of its connection with icl. ## conclusions in this work, we explore the connection between in-context learning (icl) and instruction tuning (it). through carefully designed experiments, we provide strong evidence suggesting icl and it implicitly converge in llm hidden states. that is, icl changes an llm's hidden states as if the demonstrations were used in it. this finding sheds light on the behaviors of the two very different learning paradigms (icl vs. it), potentially benefiting the development and alignment of foundation llms to downstream real-world applications.