Instruction Position Matters in Sequence Generation with Large Language Models (2024)

Abstract: Large language models (LLMs) are capable of performing conditional sequence generation tasks, such as translation or summarization, through instruction fine-tuning. The fine-tuning data is generally sequentially concatenated from a specific task instruction, an input sentence, and the corresponding response. Considering the locality modeled by the self-attention mechanism of LLMs, these models face the risk of instruction forgetting when generating responses for long input sentences. To mitigate this issue, we propose enhancing the instruction-following capability of LLMs by shifting the position of task instructions after the input sentences. Theoretical analysis suggests that our straightforward method can alter the model’s learning focus, thereby emphasizing the training of instruction-following capabilities. Concurrently, experimental results demonstrate that our approach consistently outperforms traditional settings across various model scales (1B / 7B / 13B) and different sequence generation tasks (translation and summarization), without any additional data or annotation costs. Notably, our method significantly improves the zero-shot performance on conditional sequence generation, e.g., up to 9.7 BLEU points on WMT zero-shot translation tasks. Further analysis reveals that our method can significantly improve the tranditional model’s instruction following ability by 1x over traditional approch.

## introduction in recent years, there has been a rapid emergence of large language models (llms) like chatgpt and gpt-4 1 , which have shown promising performance in various traditional natural language processing tasks @xcite @xcite @xcite . meanwhile, there is also a growing interest in open source mediumsized language models, such as the llama model with 13 billion parameters @xcite and the bloomz language model with 7.1 billion parameters @xcite , to meet research and hardware deployment requirements. to align the outputs of language models with human intentions and unlock their full potential, instructgpt @xcite constructs a small amount of instruction-following data for finetuning llms and conducts reinforcement learning to align the model with human preferences. this approach to instruction following has gained widespread attention from both the academic and industrial communities @xcite @xcite @xcite . generally, the instruction-following data consists of three parts. taking the machine translation task as an example, these parts include a specific task instruction (e.g., "please translate the following paragraph from english to french"), an input sentence (the english sentence to be translated), and the final response (the corresponding french translation). since most large language models are based on the decoder-only structure of the transformer @xcite @xcite , with a training objective of predicting the next token. generally, these three parts of the instruction-following data are sequentially concatenated into a long nature sentence as the training data for language models. given that the self-attention mechanism in transformer decoders tends to focus more on nearby words i.e., the locality of self-attention modeling @xcite @xcite , there is a considerable risk of instruction forgetting when predicting responses for long input sentences. for example, when performing long text summarization tasks, the input sentence may contain thousands of tokens. consequently, the model may be at risk of forgetting the initial task instruction when predicting responses, leading to the generation of responses that do not fully comply with the user's intent. in this paper, we refer to this risk as the instruction forgetting issue. to alleviate the above issue for llms during instruction fine-tuning, we first observe that the relative position of the input sentence and the task instruction is crucial. therefore, we propose a simple and straightforward solution, namely, placing the task instruction at the end of the input sentence (referred to as 'post-ins'). in this way, when the model predicts the final response, it naturally attends to the nearest preceding sequence, which is just the task instruction indicating what content should be generated next. for comparison, we refer to the data format in existing studies where the task instruction is concatenated to the front of the input sentence as pre-instruction (abbreviated as 'pre-ins'). to verify whether post-ins improves the instruction-following ability of language models and alleviates the instruction forgetting issue on long sentences compared to pre-ins, we first analyze the conditional probability characteristics of the models under both data formats with the trinomial bayes formula. through appropriate assumptions and formula derivations, we draw the following conclusions: (1) pre-ins tends to model a reverse conditional probability (e.g., reverse translation probability), emphasizing the coverage of the input sentence while insufficiently modeling the task instruction. (2) post-ins is more inclined to model a conditional probability about the task instruction (e.g., predicting the task instruction given inputs and outputs), emphasizing the modeling of task instruction-following ability. in addition to the theoretical analysis, we conduct extensive experiments based on two widely used large language models, llama and bloomz, with various parameter sizes ranging from 1.7 billion to 13 billion. we select two common sequence generation tasks as specific downstream tasks, namely, machine translation and long text summarization. the experimental results show that post-ins consistently outperforms pre-ins across various settings without using any additional supervised data. furthermore, due to the superior modeling ability of task instruction, post-ins exhibits stronger task instruction generalization capabilities, resulting in significant performance gains in zero-shot translation tasks (e.g., up to a 9.7 bleu score improvement). further analysis reveals that our method can substantially enhance the model's instruction-following ability by 1x compared to the traditional approach, and about 4x improvements on specific instructions. our contribution can be summarized as follows: 2 ## related work instructgpt @xcite is the first to unveil the immense potential of instruction learning. remarkably, an instructgpt model with 1.3 billion parameters can outperform the 175b gpt-3, despite having 100x fewer parameters. then stanford releases the alpaca instruction-following dataset @xcite , which is constructed by the self-instruction data generation pipline (wang instruction: translate the following sentences from [src] to [tgt]. input: at the start of this week the wada had already raised the issue of tampering and demanded that russia reply to the accusations within three weeks. russia is now faced with the rusada again being excluded and in the worst case being banned from the 2020 olympic games in tokyo. response: zu beginn dieser woche hatte bereits die wada manipulationen in den raum gestellt und russland aufgefordert, binnen drei wochen auf die vorwürfe zu antworten. russland droht nun die erneute sperre der rusada und im schlimmsten fall auch der ausschluss von den olympischen spielen 2020 in tokio. (a) pre-ins instruction: translate the above sentences from [src] to [tgt]. input: at the start of this week the wada had already raised the issue of tampering and demanded that russia reply to the accusations within three weeks. russia is now faced with the rusada again being excluded and in the worst case being banned from the 2020 olympic games in tokyo. response: zu beginn dieser woche hatte bereits die wada manipulationen in den raum gestellt und russland aufgefordert, binnen drei wochen auf die vorwürfe zu antworten. russland droht nun die erneute sperre der rusada und im schlimmsten fall auch der ausschluss von den olympischen spielen 2020 in tokio. et al., 2022). in the field of machine translation, parrot (jiao et al., 2023a) builds contrastive and errorguided instructions to align the translation results of llms with humman preferences. subsequently, @xcite further extends the error-guided instructions with token-level direct preference optimization @xcite . to better transfer the capabilities of sequence generation of llms, bayling @xcite proposes to conduct interactive translation task through instructing fine-tuning. although the aforementioned methods have made considerable progress, we argue that the pre-ins data format ultilized in existing studies face the potential risk of instruction forgetting, what we aims to address in this paper. ## main results and analysis in this section, we first list the detailed experimental results of both the machine translation and text summarization tasks in section 5.1 and section 5.2. subsequently, we show the analysis of the distributions of self-attention in section 5.3, and human evaluation results in appendix a. ## conclusion this paper highlights the importance of task instruction positioning in the instruction fine-tuning process of llms for conditional sequence generation tasks. we propose a simple yet effective method, post-ins, which relocates the task instruction after the input sequence to enhance the instructionfollowing ability of llms. our theoretical analysis and experimental results demonstrate that post-ins effectively shifts the model's learning focus, leading to improved performance across various model scales and different tasks, such as machine translation and long text summarization. notably, our method significantly boosts zero-shot performance without additional data or annotation costs.