Revealing the Inherent Instructability of Pre-Trained Language Models (2025)

Abstract: Instruction tuning—supervised fine-tuning using instruction-response pairs—is a key step in making pre-trained large language models (LLMs) instructable. Meanwhile, LLMs perform multitask learning during their pre-training, acquiring extensive knowledge and capabilities. We hypothesize that the pre-training stage can enable them to develop the ability to comprehend and address instructions. To verify this, we propose Response Tuning (RT), which removes the instruction and its corresponding mapping to the response from instruction tuning. Instead, it focuses solely on establishing a response distribution. Our experiments demonstrate that RT models, trained only on responses, can effectively respond to a wide range of instructions akin to their instruction-tuned counterparts. In addition, we observe that the models can recognize and reject unsafe queries after learning a safety policy only from the response data. Furthermore, we find that these observations extend to an in-context learning setting. These findings support our hypothesis, highlighting the extensive inherent capabilities of pre-trained LLMs.

## introduction large language models (llms) are pre-trained to predict the next token using massive amounts of web-crawled text, implicitly learning a wide range of tasks @xcite @xcite . to make the pre-trained models instructable, instruction tuning (it) @xcite @xcite )a supervised fine-tuning process using instructionresponse paired data-is widely performed, thereby enhancing their usability and facilitating real-world applications @xcite @xcite @xcite @xcite . however, how llms achieve such a transition remains unclear @xcite . we hypothesize that llms can acquire the ability to process instructions-that is, to comprehend and address them-during pre-training, in addition to the extensive knowledge required to perform specific tasks @xcite @xcite . for instance, they may implicitly learn instruction-response dynamics from q&a threads on stack exchange-a common source in typical pre-training datasets @xcite . under this assumption, the models may be able to respond appropriately to instructions once their response distributions are established. previous studies have reported similar phenomena in canonical nlp tasks, demonstrating that supervision from the output space is crucial for performing such tasks @xcite . to test our hypothesis, we propose response tuning (rt), which omits the instruction and its associated instruction-response mapping from it (see figure 1 ). this omission prevents the model from learning to produce responses according to instructions; rather, it focuses on establishing a response distribution. we evaluate whether rt models can understand and respond appropriately to diverse instructions. in our experiments, we consider four recent llms and three different datasets: llama-3.1-8b @xcite , gemma-2-2b and gemma-2-9b @xcite , and mistral-7b-v0.3 @xcite , utilizing only the responses from alpaca @xcite , dolly @xcite , and lima @xcite . our human and automatic evaluations, based on test instructions derived from multiple sources @xcite , show that rt models can appropriately respond to a wide range of instructions and exhibit helpfulness close to that of their it counterparts. these findings show that establishing a response distribution alone can yield instructionfollowing models, suggesting that pre-training allows the models to learn how to process instructions. furthermore, we find that rt models can identify and reject unsafe queries. specifically, we incorporate explanatory refusals for unsafe queriesresponses that decline to fulfill requests while specifying the underlying safety policy-into the training data. when tested with diverse unsafe instructions derived from multiple safety evaluation datasets @xcite @xcite , these models invoke their learned safety rules to the unsafe queries, exhibiting refusal rates approaching those of it counterparts trained with paired examples. these results also support our hypothesis that pre-training enables the model to develop the ability to process instructions. finally, we extend our investigation to an incontext learning setting to test if our prior observations hold. we find that response-only demonstrations can enable the models to appropriately handle instructions, further supporting the hypothesis. overall, our findings provide a deeper understanding of how llms become instructable agents and suggest the potential of extensive inherent capabilities acquired during pre-training. in summary: 1. we hypothesize that llms might acquire the ability to process instructions during pretraining. to verify this, we propose response tuning (rt), which omits both the instruction and its corresponding mapping to the response, focusing solely on establishing a response distribution. recent studies have shifted their focus from crosstask generalization in canonical nlp tasks @xcite @xcite to generalization for unseen user instructions. notable contributions include synthetic data generation frameworks @xcite @xcite , human-involved conversation collection methods @xcite @xcite @xcite . recently, a line of research has explored the superficiality of the it stage, questioning the necessity of large-scale data @xcite @xcite or parameter updates @xcite . however, what the models learn from it and how they become instructable agents remains unclear @xcite . learning from input-output pairs. @xcite highlights the crucial role of label space information in in-context learning of canonical nlp tasks (e.g., classification). @xcite examines the role of task definitions in the prompts of the it dataset and demonstrates that models trained with misleading task definitions or only on the label space exhibit similar task generalization performance. however, these studies focus on generalization across canonical tasks with a finite label space, rather than on how models handle natural language instructions with an open-ended response space. our approach. to address these gaps, we employ rt to investigate how well llms can process given instructions, without additional fine-tuning on the paired data, but with an established response distribution. a concurrent study by hewitt et al. (2024) examines whether training only on responses can yield instruction-following. in this work, we focus on the inherent ability to process instructions, employing various evaluation methods (including tests for unsafe query identification) across both fine-tuning and in-context learning settings to carefully assess them. ## response tuning (rt) we propose response tuning (rt), a simple ablation of it. it prevents further learning of instruction-response mappings by omitting the instructions, thereby allowing us to investigate the model's capability to process instructions acquired during pre-training. training data. we adopt the chatbot-style template proposed by @xcite , which separates user instructions and assistant responses using special tokens: 
 @xmath0. however, in rt, we omit both @xmath1. therefore, the training data consists only of @xmath2. training. rt employs standard teacher forcing and computes the loss only on the response tokens that appear after @xmath3. the loss function for the autoregressive language model is defined as: here, θ represents the model parameters, l is the total number of tokens in a training example, and t i denotes the i-th token in the sequence. the indicator function δ i equals 1 if the i-th token belongs to the response set r (i.e., the assistant's response), and 0 otherwise. this formulation mirrors the loss function used in it, where loss masking is applied to instruction tokens @xcite @xcite . however, unlike it, rt does not condition the response tokens on the paired instructions, which precludes the model from learning to generate responses according to instructions. rather, it focuses on learning the response distribution. inference. we input a sequence that starts with the instruction delimiter (
), followed by the user's instruction, and then the response delimiter (
). the model then generates the assistant's response after @xmath4. although rt provides no explicit supervision for instruction-processing, we find that the models can effectively leverage their inherent capabilities to do so. we demonstrate this in the following experimental sections. ## instructability of rt models in this section, we assess whether rt models can process diverse instructions (e.g., creative writing, trip planning, and general question-answering) and, if so, how their performance compares to that of it models. for reliable verification, we conduct both human and automatic evaluations across four models and three datasets. ## rejecting unsafe instructions in this section, we investigate whether rt models can understand instructions and correspondingly decide how to handle them. to test this, we incorporate refusals for unsafe queries into rt training data and then evaluate whether they can recognize and reject unsafe instructions. unlike it, which utilizes explicit unsafe instruction-rejection mappings to allow models to identify them, rt omits such direct mappings. ## in-context response learning we further validate our hypothesis in an in-context learning setting. to this end, we test whether untuned base llms can appropriately respond to user queries when provided only with demonstrations of responses. experimental setup. we remove instructions and the associated instruction-response mappings from urial @xcite , which consists of 4 instruction-response pairs including one pair of unsafe instruction and refusal. we refer to this new version as urial-r. we then evaluate it using two different base llms, llama-3.1-8b, and gemma-2-9b, with the justeval benchmark. we employ greedy decoding with a maximum generation length of 2,048 tokens. we also evaluate the zero-shot template prompting baseline @xcite to further investigate the effect of learning a response distribution. the prompts and details of the setup can be found in appendix a. results. figure 6 presents the evaluation results. the results show that using response demonstrations alone enables the base models to effectively handle both benign and unsafe instructions. across all metrics, the scores of the models prompted with urial-r are similar to those of the models prompted with urial, which includes instructions. additionally, while the zero-shot prompting baseline generates outputs relevant to the instructions, urial-r-which includes demonstrations of coherent responses-substantially outperforms it. these results further reinforce our earlier conclusion that the ability to handle instructions is inherent in pre-trained llms and that establishing an adequate output distribution helps the models use them effectively. ## conclusion we hypothesize that the pre-training stage enables llms to develop the ability to process instructions. to test this, we propose response tuning (rt), a method that removes instructions from it and focuses solely on learning a response distribution. our extensive experiments demonstrate that rt models-trained only on responses without paired instructions-can effectively respond to a wide range of user queries. moreover, we observe that they can identify and correspondingly reject unsafe requests by invoking a safety policy learned solely from response data. these observations also extend to an in-context learning setting. such results show that establishing an adequate output distribution alone can yield instructable models, supporting our hypothesis. taken together, our work contributes to understanding how llms become instructable agents and suggests the potential of extensive inherent capabilities developed during pre-training.