A Comparative Study of Learning Paradigms in Large Language Models via Intrinsic Dimension (2025)

Abstract: The performance of Large Language Models (LLMs) on natural language tasks can be improved through both supervised fine-tuning (SFT) and in-context learning (ICL), which operate via distinct mechanisms. SFT updates the model’s weights by minimizing loss on training data, whereas ICL leverages task demonstrations embedded in the prompt, without changing the model’s parameters. This study investigates the effects of these learning paradigms on the hidden representations of LLMs using Intrinsic Dimension (ID). We use ID to estimate the number of degrees of freedom between representations extracted from LLMs as they perform specific natural language tasks. We first explore how the ID of LLM representations evolves during SFT and how it varies due to the number of demonstrations in ICL. We then compare the IDs induced by SFT and ICL and find that ICL consistently induces a higher ID compared to SFT, suggesting that representations generated during ICL reside in higher dimensional manifolds in the embedding space.

## introduction large language models (llms) have transformed the field of natural language processing through their general natural language understanding capabilities, which can be applied to a broad range of tasks. the performance of an llm on a specific task can be improved through two primary learning paradigms: supervised fine-tuning (sft) and in-context learning (icl). sft adapts pretrained models to specific tasks by updating their parameters, while icl requires no parameter updates, relying instead on task-specific demonstrations within the model's context window. despite their widespread success, how these methods influence a model's internal representation space is still not fully understood. 1 code is available at the following github repo. intrinsic dimension (id) is a useful metric for assessing the geometric complexity of a model's representations. it quantifies the number of degrees of freedom in the representation space, serving as a measure of the complexity of the underlying manifolds where the embeddings reside. in this work, we analyze the intrinsic dimension (id) of hidden representations across model layers during task execution under both supervised fine-tuning (sft) and in-context learning (icl). specifically, we explore: our findings reveal that (1) the id sometimes decreases during the early stages of fine-tuning but generally increases in the later stages, and (2) the id increases initially with more demonstrations in icl, then either plateaus or decreases as the number of demonstrations continues to rise. we then conduct experiments directly comparing the intrinsic dimensions of icl and fine-tuning across several models and datasets. we find that the intrinsic dimensions of representations from fine-tuned models are generally lower than those from models using icl, even though the fine-tuned models achieve higher accuracy than the icl models. additionally, our results suggest that id may serve as a practical heuristic for selecting the optimal number of demonstrations in icl to maximize performance while minimizing input length. these findings shed light on the differing impacts that the two learning paradigms have on the representation space of llms. 2 background ## methods we perform experiments using subsets from the following datasets: ag news (zhang et al., 2015), cola @xcite , commonsenseqa @xcite , mmlu @xcite , @xcite , qnli @xcite , qqp @xcite , and sst2 @xcite . for these experiments, we utilize the following open-source llms: llama-3-8b @xcite , llama-2-13b, llama-2-7b @xcite , and mistral-7b-v0.3 @xcite , running them on 6 nvidia a6000s. for each dataset, we created a training set of 1000 examples and a validation set of 5000 examples. we use the 5000 validation examples to ensure stability of the twonn estimator. details regarding dataset creation can be found in appendix g. details of split generations and prompt templates are provided in appendix g. we calculate the accuracy of model responses using the logit probabilities assigned to the tokens corresponding to the possible answers for each question. we mark a response as correct if the probability corresponding to the first token of the correct answer label is the highest. ## summary we present a detailed analysis of the intrinsic dimension (id) induced by the sft and icl learning paradigms. our experiments reveal that the normalized auc of id curves sometimes decreases during the initial stages of sft but generally increases during the later stages. additionally, we observe that the normalized auc of id curves in icl initially increases for small values of k (the number of demonstrations) but plateaus or slightly decreases as k increases further. notably, the k value corresponding to the highest normalized auc also achieves peak (or near-peak) accuracy, suggesting that id may serve as a useful indicator for selecting the optimal number of demonstrations during icl. finally, our direct comparison of id curves from icl and sft reveals that representations generated during icl consistently yield higher id curves compared to those from sft on 1k samples, even though sft with 1k samples achieves the highest overall performance. this analysis provides evidence that the two learning paradigms induce distinct representational structures in the embedding space, with icl representations occupying higher-dimensional manifolds.