Diversification Catalyzes Language Models’ Instruction Generalization To Unseen Semantics (2025)

Abstract: Instruction-tuned language models excel in knowledge, reasoning, and instruction-following. While knowledge and reasoning are well-explored, the factors enabling generalization to unseen instructions remain underexplored due to challenges in isolating instruction-following dynamics.In this work, we model instruction-following as a computational process and design controlled experiments inspired by the Turing-complete Markov algorithm to disentangle its dynamics. Our findings reveal that the ability to generalize to instructions with unseen semantics emerges only when training data is strategically diversified across rich semantics. This finding gives us the hammer that breaks down the wall separating training instructions from unseen ones encountered in the wild. For specialist models, a balanced mix of in-domain and diverse out-of-domain tasks enhances performance more effectively than simply increasing in-domain data. For generalist models, domain diversification consistently outweighs the costs of reduced task-specific data, regardless of data budgets. Furthermore, we show that proper diversification with a lower data budget can outperform simply scaling up data volume. These findings highlight strategic data diversification as key to optimizing instruction-following and improving model performance across applications.

## introduction the rapid advancements in large language models (llms) have revolutionized a wide range of tasks, including language comprehension @xcite , generation @xcite , knowledgebased question answering @xcite , and solving complex reasoning problems in fields such as mathematics @xcite and programming @xcite @xcite . instruction-tuning plays an important role in making language models performing these tasks. the current research landscape regarding instruction tuning data has yielded varied and sometimes contradictory findings on composition @xcite and scaling @xcite of instruction tuning data. these inconsistencies suggest that the best practices to craft an instruction tuning dataset is not fully understood @xcite . a key limitation is the absence of controlled studies that systematically examine different aspects of llm capabilities, where the effects of instruction tuning data may follow different patterns. while reasoning (multi-step deduction, like math problem-solving) and knowledge retrieval (accessing information stored in the parameters) have been extensively studied, instruction following-essential for practical interaction with users @xcite -remains poorly understood. the role of instruction-tuning @xcite @xcite ) data composition and diversity in enabling llms to generalize to unseen tasks is particularly unclear. our work addresses this gap by focusing explicitly on instruction following capabilities of llms. to address this gap, our work presents a systematic investigation into instruction-following capabilities, leveraging controlled experiments inspired by the turing-complete markov algorithm @xcite .by isolating instruction-following, we reveal that diverse cross-domain instruction data is decisive for enabling llms to generalize effectively to unseen semantic tasks. our findings reveal that robust generalization emerges only when instructions span diverse, cross-domain semantics. limited domain diversification, in contrast, fails to achieve this goal, underscoring the necessity of training strategies that prioritize broader instruction diversity. our findings reveal that diversification confined to limited domains does not guarantee robust generalization. in contrast, cross-domain diversification significantly enhances the model's adaptability to new instructions, highlight- ## rewriting with abstraction real-world instruction-following often requires abstracting high-level concepts and grounding them in specific contexts. to simulate this, we extend the string-rewriting experiment by introducing abstraction: abstract rules describes the general patterns we design a mathematical deduction task as a context-sensitive generalization of the markov string replacement task (section 2). the task involves simplifying algebraic expressions using specified deduction rules, mimicking instruction-following across semantic domains. we ensure it remains challenging for pre-trained models (see zs and fs performances in figure 5a ) to eliminate confounding factors. in this experiment, we present the model with a randomly generated mathematical expression i and an abstract mathematical deduction rule ι abs :(@xmath0.g. an example abstract rule would be a 2 -b 2 = (a+b)×(a-b), to assess the model's ability to identify the relevant sub-expression in ξ (e.g. in (2x+5) 2 -(3y-6) 2 , here @xmath1,@xmath2. we observe how well the model generalizes when trained on datasets of varying rule diversity. a full example will be: output: ( (2x + 5 + 3y -6) × (2x + 5-(3y -6)) ) 3 +(log(5t)-cos4k) ## specialist instruction follower -case study of code generation in this section, we investigate the specialist training scenario, where the objective is to train a language model to adapt to a certain task domain. specifically, we consider code generation task as an example since it features instruction following aspectrequiring conversion of natural language prompts into executable code, demanding precise interpretation and faithful execution of input instructions over complex reasoning or knowledge. ## fine-tuning generalist llms in this section, we evaluate the impact of cross-domain instruction diversification on general reasoning tasks and investigate the optimal high-level strategy to improve the quality of a dataset. ## conclusion this work systematically studies instruction-following of language models by modeling it as a computation process and careful controlled experiments. we isolate and study models' instruction-following abilities, offering a new perspective on this fundamental capability. our experiments further demonstrate that instruction diversity, even within a fixed data budget, plays a critical role in improving model generalization. this finding underscores the value of diverse instruction semantics over large dataset size, in enhancing performance across both specialized and generalized llm applications. finally, we offer practical insights into dataset collation strategies, highlighting that proper diversification can significantly outperform dataset expansion.