Learning vs Retrieval: The Role of In-Context Examples in Regression with Large Language Models (2025)

Abstract: Generative Large Language Models (LLMs) are capable of being in-context learners. However, the underlying mechanism of in-context learning (ICL) is still a major research question, and experimental research results about how models exploit ICL are not always consistent. In this work, we propose a framework for evaluating in-context learning mechanisms, which we claim are a combination of retrieving internal knowledge and learning from in-context examples by focusing on regression tasks. First, we show that LLMs can solve real-world regression problems and then design experiments to measure the extent to which the LLM retrieves its internal knowledge versus learning from in-context examples. We argue that this process lies on a spectrum between these two extremes. We provide an in-depth analysis of the degrees to which these mechanisms are triggered depending on various factors, such as prior knowledge about the tasks and the type and richness of the information provided by the in-context examples. We employ three LLMs and utilize multiple datasets to corroborate the robustness of our findings. Our results shed light on how to engineer prompts to leverage meta-learning from in-context examples and foster knowledge retrieval depending on the problem being addressed.

## introduction the emergence of transformers @xcite has revolutionized natural language processing, leading to the development of llms such as gpts @xcite . in addition to their impressive zero-shot performance, these models demonstrated the capability of in-context learning (icl), by which they learn a task from examples provided in the context of the prompt @xcite . in tasks where both zero-shot and icl settings are applicable, icl consistently outperforms the zero-shot setting @xcite . however, the inner mechanism of icl in llms remains an open area of research. broadly speaking, current research identifies two main approaches to explain the icl mechanism @xcite : 1) meta-learning, 2) knowledge retrieval. with metalearning @xcite , the llm learns a new unseen pattern from the in-context examples, allowing it to generalize to new inputs that follow the same pattern. in contrast, with knowledge retrieval, the llm uses the in-context examples as cues to access and apply information already stored within its pre-trained parameters without inferring new patterns from the provided examples. for instance, given examples of input/output number pairs in the context, meta-learning allows the model to learn their relationship and apply it to new inputs. however, if the prompt informs the model that the pairs are the earth's population given a year, the llm utilizes its knowledge retrieval and infers that the function is exponential and plateaus at around 8-10 billion after the year 2000. contrary to current research, we propose a different approach, arguing that icl is not merely learning or retrieving knowledge, but rather, its behavior lies on a spectrum between the two, which can be adjusted depending on various factors. before exploring our proposed approach, we elaborate on the hypotheses mentioned above. 1 the first hypothesis suggests that transformers are effective meta-learners, and llms can generate accurate predictions based solely on the given input-output pairs. @xcite theoretically proves and practically tests the capability of an encoder-based transformer to implement generalized linear models in-context. @xcite uses a decoder-based model, a gpt-2 architecture modified for regression, that uses the input features inside the embeddings (instead of using tokens) and outputs a number. their model's performance surpasses a 2-layer multi-layer perceptron (mlp) and a decision tree by meta-learning. further, @xcite directly uses llms to test regression capabilities using a limited number of features (less than 3) and concludes that llms are capable regressors based on in-context examples. however, according to our findings, their claim of avoiding data contamination is not wellsupported. we note that none of these research works use realistic datasets or consider the combination of meta-learning with knowledge retrieval. the second approach emphasizes knowledge retrieval while downplaying the learning aspect. for example, @xcite examines 12 llms across various classification tasks, concluding that altering output labels, which are needed for learning, has no impact on performance. they argue that the crucial elements of icl are limited to defining the label space, input distribution, and overall task format. meanwhile, @xcite argues that llms utilize the labels but in an arbitrary manner when experimenting with the same classification tasks. @xcite further dismisses the significance of output labels and instead proposes two knowledge retrieval approaches. these include the retrieval of pre-existing solutions from training data @xcite @xcite , and a novel solution composition approach, inspired by @xcite , which proposes that learned solutions can be combined. while this work offers valuable insights into icl mechanisms, its findings are limited to a narrow selection of datasets and models. here, the exclusive use of llama 2 @xcite , which is known to struggle with long token contexts @xcite results in some different findings compared to our work. for instance, they report that remapping the inputs of in-context input-output pairs to alternative text reduces the accuracy to that of a random model (even with up to 40 in-context examples). on the other hand, under similar conditions, our experiments indicate that in newer models, learning occurs with optimal performance. we refrain from making specific claims about task selection or composition in our work. instead, we categorize these processes under the broader umbrella of knowledge retrieval foot_1 . similarly, to investigate icl, @xcite uses simple classification tasks and older models (e.g., gpt-3 ada), and as a result, its findings are partially different from ours. we propose a different hypothesis that resolves the contradictions in the research community and support our claims with extensive empirical testing. we argue that icl is not merely learning or retrieving knowledge but uses a combination of the two, which lies on a spectrum determined by factors we can manipulate. in this regard, we propose an evaluation framework and conduct a comparative study of different llms and datasets, focusing specifically on regression problems as our testing ground. in our evaluation framework, we query the llm to estimate an output number based on a set of (feature, value) pairs given a set of icl examples, as shown in figure 1(a) . we opted for regression tasks for the following reasons: this choice aligns our work with the majority of related icl meta-learning research, thus facilitating direct comparisons and building upon existing findings. these include works that alter the output module of llms, such as @xcite and those that use the token generation output of llms as is, such as @xcite . moreover, @xcite demonstrates that the icl capability of llms is independent of their input/output modules and is present in the intermediary layers of the llms. finally, while llms are capable of regression (even when inputs are given to them in a textual format @xcite ), this task still has a complex output space (e.g., continuous or unbounded), which is challenging for llms @xcite . we show that llms can perform regression on realistic datasets and measure the extent to which the llm retrieves its internal knowledge versus learning from in-context examples. we provide an in-depth analysis of the degrees to which these mechanisms are triggered depending on the factors we use in our framework: the number of (feature, value) pairs, the number of in-context examples, and the prompting strategies. in summary, our contributions are as follows: 1) we demonstrate that llms can effectively learn from regression examples of realistic datasets incontext, extending previous work on synthetic data to more practical scenarios. 2) we propose a new hypothesis about the icl mechanism that combines both learning and knowledge retrieval during the llm inference, reconciling the research community's findings. 3) we introduce an evaluation framework that allows for systematic comparison of icl mechanisms across different llms, datasets, and prompt configurations 3 . 4) we provide a comprehensive analysis of how llms balance internal knowledge retrieval and learning from in-context examples and propose prompt engineering techniques to control them. ## problem setting our study focuses on a regression task where we use llms to predict numerical outputs based on the given inputs. we assume a regression dataset, @xmath0, y 1 ), (x 2 , y 2 ), . . . , (x n , y n )}, comprising input-output pairs, is given. each x i comprises a set of pairs (f ij , v ij ) where f ij represents the fea3 the code and analytical tools are available at . ture name and v ij is the numerical value of f ij . the target variable y i is the numerical value of the output. for example, if we consider a dataset providing the price of used cars given their fuel economy and mileage, we could have an input with (feature, value) pairs, (fuel economy,16) and (mileage, 0), and an output, 95595, indicating the price. to resemble a regression problem setting, we present the llm with a subset of m input-output examples {(x 1 , y 1 ), (x 2 , y 2 ), . . . , (x m , y m )} taken from the in-context split of d for learning. then, we query the llm with the feature pairs of x taken from the test split of d to obtain the answer y as the output. 4 figure 1 (a) depicts our problem setting in the context of a regression task that involves predicting the "price of a used toyota or maserati in 2019", which is the target variable name, given three (feature, value) pairs with a set of provided in-context examples. when m equals 0, we obtain the zero-shot setting without prior examples. ## experiments this section presents our experimental findings, highlighting the interplay between various factors that influence the performance of llms across regression tasks and datasets. ## discussion our evaluation framework and extensive experiments provide insights into the icl mechanisms of llms. in this section, we discuss the implications of our findings, address limitations, and suggest directions for future research. ## conclusion in this study, we have explored the icl capabilities of llms through an empirical study on regression tasks. our results demonstrate that llms utilize a blend of retrieved internal knowledge and learned information from in-context examples. our findings extend the evaluations of prior hypotheses on icl. for example, we evaluate the usage of outputs in in-context learning examples and show how to manipulate their effectiveness. this work not only advances our understanding of llms' in-context learning phenomenon but also offers practical insights for optimizing their application through careful prompt engineering.