Latent Traits and Cross-Task Transfer: Deconstructing Dataset Interactions in LLM Fine-tuning (2025)

Abstract: Large language models are increasingly deployed across diverse applications. This often includes tasks LLMs have not encountered during training.This implies that enumerating and obtaining the high-quality training data for all tasks is infeasible. Thus, we often need to rely on transfer learning using datasets with different characteristics, and anticipate out-of-distribution requests.Motivated by this practical need, we propose an analysis framework, building a transfer learning matrix and dimensionality reduction, to dissect these cross-task interactions.We train and analyze 10 models to identify latent abilities (e.g., Reasoning, Sentiment Classification, NLU, Arithmetic)and discover the side effects of the transfer learning.Our findings reveal that performance improvements often defy explanations based on surface-level dataset similarity or source data quality. Instead, hidden statistical factors of the source dataset, such as class distribution and generation length proclivities, alongside specific linguistic features, are actually more influential.This work offers insights into the complex dynamics of transfer learning, paving the way for more predictable and effective LLM adaptation.

## introduction large language models (llms) demonstrate remarkable capabilities across diverse tasks, yet their deployment in real-world applications faces significant practical constraints. cost and latency considerations render giant all-purpose models impractical for many use cases, driving widespread adoption of task-specific fine-tuning. however, this approach encounters a fundamental challenge: high- † corresponding author. ‡ this work was partly done when the author was at anthropic. quality training data for target tasks is often unavailable or proprietary. moreover, deployed llms routinely face out-of-distribution (ood) requests that extend beyond their fine-tuning scope. this is especially true for agentic systems, which rely heavily on cross-domain skill transfer to perform diverse sequences of tasks. these realities necessitate a deeper understanding of transfer learning. traditional transfer learning research has pri-marily focused on scenarios where source and target tasks share the same domain, assuming that similar or related in-domain data provides useful signal for the target task. however, the diverse task landscape that modern llms navigate demands deeper understanding of ood transfer effects. our experiments using low-rank adaptation (lora) reveal counterintuitive transfer behaviors: fine-tuning on one dataset can yield surprising performance improvements or degradations on seemingly unrelated tasks, often defying expectations based on semantic similarity (illustrated conceptually in figure 1 ). this unpredictability creates significant challenges for practitioners selecting optimal source datasets for fine-tuning, particularly in resource-constrained environments where training efficiency is paramount, or when acquiring pre-trained lora adapters from service providers without clear transferability guarantees. in this paper, we propose a framework to analyze how the source fine-tuning dataset influences the performances on the target datasets in transfer learning and use this framework to systematically characterize the ood generalization of an llm using multiple lora adapters. our analysis framework first constructs a performance matrix across different source and target tasks. we apply principal component analysis (pca) to this matrix to uncover latent abilities or "traits" that fine-tuned llms acquire from the transfer learning. we demonstrate that straightforward factors like source data quality or simple source-target similarity often fail to explain observed transfer learning effects. instead, we highlight the critical role of more subtle, "hidden" statistical properties of the source training data (e.g., output length distribution, label imbalance) and learned sensitivities to specific linguistic features. our work aims to provide actionable insights into the selection of the source dataset for finetuning, fostering a deeper understanding of the interactions among the datasets and guiding the development of more robust llm adaptation strategies. in our experiments, we fine-tune the llama 3.2 3b base model @xcite using lora and systematically evaluate models fine-tuned on one dataset across datasets for math, coding, natural language inference, sentiment, and toxicity detection tasks to map diverse data interactions. through analyzing the fine-tuned llm and datasets, we observe several surprising cross-domain interactions, including: (1) the impact of source data generation length on fine-tuned model outputs; (2) asymmetric enhancement through out-of-domain fine-tuning datasets; and (3) the profound effects of source label imbalance on both in-domain and ood performance. ## related work the transfer learning of fine-tuning language models is investigated by several existing works @xcite @xcite @xcite @xcite @xcite . most studies focus on identifying similar tasks for positive transfer effect through fully fine-tuning small language models. instead, our work focuses on modeling the impact of lora fine-tuning and discovering the often-overlooked side effect of the source training datasets including out-of-domain and out-ofdistribution datasets. compared to the full finetuning, @xcite ; @xcite find lora "learns less and forgets less", which potentially preserves out-of-domain base model capabilities better. this is one of the main reasons behind lora's effectiveness and popularity. nevertheless, we demonstrate that loras, which are fine-tuned on many source datasets, could still cause several types of undesirable side effects when being evaluated on a wide range of target tasks. methodologically, our analysis framework is related to @xcite , which employs pca to analyze observational scaling laws and the predictability of llm performance across different model sizes and tasks. some recent findings also support our discoveries of hidden factors. for example, @xcite report that instruction fine-tuning with coding data can sometimes negatively impact mathematical reasoning. @xcite find that the length distribution of the instruction tuning training data could affect the llms' code generation ability. @xcite ; @xcite ; @xcite discover that the format of the fine-tuning data might be more important than its content or correctness. our work confirms their findings and provides a more comprehensive list of latent traits that influence loras' performance. ## methodology in our framework, we first prepare n representative tasks/datasets of interest and fine-tune llms effects: on these n datasets to acquire i fine-tuned llm variants. in this paper, we use lora to fine-tune llms, but the framework could be applied to any fine-tuning method (e.g., full fine-tuning, prompt tuning @xcite , bitfit (zaken et al., 2022), etc.) or any variants of llms (e.g., incontext learning or chain of thoughts). viewing each fine-tuned lora adapter as a specialized piece of equipment in a practitioner's toolkit, a crucial challenge is selecting the right tool for a new task. the conventional approach assumes a tool's effectiveness is dictated by its labeled domain, for instance, using a 'sentiment' adapter for a sentiment task. however, these tools might come with unexpected side-effects and hidden capabilities driven by the latent statistical properties of their training data, not just their domain. to solve this issue, we evaluate the i fine-tuned llms on the n datasets on their accuracy, and organize the pairwise results into a i × n performance matrix as shown in figure 2 . note that since our goal is to measure the impact on out-of-domain tasks, we focus more on relative gains, rather than the absolute performance numbers. throughout this paper, we denote the base llm as m; a model fine-tuned on a dataset d as m["d"]; and the output performance of such a model on the evaluation data x is denoted m["d"](x). for example, m["flipkart"](gsm8k) refers to the score of the model fine-tuned on the flipkart dataset and tested on gsm8k. in the performance matrix, m["flipkart"](gsm8k) corresponds to the row for flipkart and column for gsm8k. to understand the overall characteristics and transfer learning impact across these datasets, we decompose the performance matrix using pca. each principal component corresponds to a group and the tasks with high values in the corresponding eigenvector belong to the group. in this way, similar evaluation tasks whose llm scores have high correlations will cluster together. we can then use the common attribute of the tasks in a group as its name -a standard practice to make the abstract mathematical components interpretable. guided by the pca results, we discover the transfer learning patterns among the tasks of interest and further investigate the outliers in the performance matrix. we then conduct analyses to identify the factors that could explain the patterns and outliers. ## experimental setup we curate a diverse set of datasets spanning mathematical reasoning (metamath @xcite , gsm8k @xcite , and goat @xcite ), code generation (magicoder (ise-uiuc, 2023)), natural language inference (nli) (paws @xcite and mnli @xcite ), sentiment analysis (flipkart sentiment (kayee), amazon reviews @xcite , and imdb reviews @xcite ), and toxicity detection (pile @xcite ). for more information about the datasets refer to ## results and analyses in this section, we show the analysis on how the statistical properties drive transfer learning regardless of the domain similarity. we show various statistical properties and their effects on performance for both in-domain transfer and out-of-domain transfer. the overall cross-task performance matrix is summarized in ## conclusion our investigation into the cross-task dynamics of low-rank adaptation (lora) confirms that transfer learning behavior is often unintuitive and defies explanations based on task domains or surface level dataset similarity. this work introduced a systematic framework to dissect these interactions, revealing that performance on target tasks is more influenced by the transfer of latent statistical and linguistic traits learned from a source dataset. we established the existence of these phenomena, such as asymmetric skill transfer and the impact of the class distribution, providing a new lens through which to view the fine-tuning process. our work paves the way for a more predictable and engineering-driven discipline around llm adaptation. the logical next step is to move toward a modular approach, creating a "tool-belt" of skill adapters for agentic systems. an agent could then dynamically load a "conciseness adapter" for summarization or a "syntactic-parser adapter," like the one we observed emerging from classification data, for complex instruction understanding. while we demonstrated these dynamics on the llama 3 architecture with lora, a critical next step is to validate and expand these findings across a broader range of models and adaptation methods to assess their scalability. by pursuing these avenues of modularity and prediction, we can build more robust and capable ai agents.