Data Doping or True Intelligence? Evaluating the Transferability of Injected Knowledge in LLM s (2025)

Abstract: As the knowledge of large language models (LLMs) becomes outdated over time, there is a growing need for efficient methods to update them, especially when injecting proprietary information. Our study reveals that comprehension-intensive fine-tuning tasks (e.g., question answering and blanks) achieve substantially higher knowledge retention rates (48%) compared to mapping-oriented tasks like translation (17%) or text-to-JSON conversion (20%), despite exposure to identical factual content. We demonstrate that this pattern persists across model architectures and follows scaling laws, with larger models showing improved retention across all task types. However, all models exhibit significant performance drops when applying injected knowledge in broader contexts, suggesting limited semantic integration. These findings show the importance of task selection in updating LLM knowledge, showing that effective knowledge injection relies not just on data exposure but on the depth of cognitive engagement during fine-tuning.

## introduction llms demonstrated proficiency in possessing factual knowledge across a wide variety of domains @xcite . despite their impressive capabilities, these models face a fundamental limitation: their knowledge is bounded by the cutoff date of their pre-training data. while extended pre-training offers a potential solution to incorporate new knowledge, it demands substantial computational resources, often involving thousands of gpu hours and millions of tokens, making it expensive and impractical for most researchers and organizations @xcite . this economic barrier has pushed researchers to explore more efficient methods to inject new knowledge into llms. recently, knowledge injection has emerged as an alternative to extended pre-training, as supervised fine-tuning (sft) on curated datasets has been shown to inject knowledge into an llm @xcite . however, these approaches have predominantly focused on question-answering tasks, leaving unexplored how the nature of the fine-tuning task itself influences knowledge retention and accessibility. this gap is particularly significant given that recent work has demonstrated differential impacts on security when fine-tuning llms across varied task types @xcite . this observation raises a question: whether comprehension-intensive tasks like question answering (qa) (requiring deep understanding of information) yield different knowledge injection outcomes compared to mapping-oriented tasks like translation. the main question is that, in tasks such as translation, where the model theoretically needs only to perform @xmath0, is the model actually internalizing the factual content contained within that data? this distinction is important for understanding how different fine-tuning approaches affect a model's overall knowledge representation. building on the variability of task-based knowledge injection, we ask the following research questions: ## related work to keep llms up-to-date with new information without incurring high costs of full retraining, some widely studied approaches are supervised fine-tuning (sft) @xcite , retrieval-augmented generation @xcite , and continual pre-training (cpt) @xcite . in this paper, we focus on sft as our injection mechanism. unlike rag, sft embeds knowledge directly into model parametersessential for evaluating true "learning" rather than deferred lookup-works offline, and requires fewer compute resources than cpt @xcite . existing studies hint at task-dependent finetuning outcomes. @xcite explored sft to inject new out-of-domain facts, i.e., recent sporting results, into llms, and demonstrated that fact-based scaling yields more effective injected knowledge than token-based scaling. however, a significant challenge in knowledge injection is determining whether an llm has truly internalized new information or merely memorized it. two widely used evaluation paradigms are probing and benchmarking. probing methods, such as those by @xcite , extract latent facts into structured artifacts (e.g., knowledge graphs) to inspect internal representations. benchmarking relies on standardized test suites: mmlu, truth-fulqa, to measure factual recall and reasoning accuracy but suffers from data contamination and hallucination artifacts. recognizing these shortcomings, @xcite advocates a shift towards capability-based assessments that better isolate retention versus surface performance. additionally, researchers employ task variation probes to reveal superficial learning. for instance, @xcite demonstrated that llms struggle with symbolic versions of familiar tasks, revealing a reliance on pattern matching over genuine understanding. in this paper, we investigate how shifting the fine-tuning objective from comprehension-based tasks to mapping-based tasks affects the depth and transferability of injected knowledge. ## conclusion our study reveals that effective knowledge injection in llms depends not just on data exposure but on the depth of semantic engagement, favoring comprehension-based tasks over token-mapping ones. scaling trends suggest that knowledge integration improves with model size, yet the gap between recall and broader application points to lingering limitations in current fine-tuning methods. these insights highlight the importance of task selection for efficient and meaningful model updates, emphasizing the need to prioritize task formats that require deeper semantic processing, especially for critical factual information.