Better Alignment with Instruction Back-and-Forth Translation (2024)

Abstract: We propose a new method, instruction back-and-forth translation, to improve the quality of instruction-tuning data used for aligning large language models (LLMs). Given preprocessed texts from an initial web corpus (e.g. Dolma (Soldaini et al., 2024)), we generate synthetic instructions using the backtranslation approach proposed by Li et al., (2023), filter the generated data and rewrite the responses to improve their quality further based on the initial texts. Given similar quantities of instructions, fine-tuning Llama-2 on our (synthetic instruction, rewritten response) pairs yields better AlpacaEval win rates than using other common instruction datasets such as Humpback, ShareGPT, Open Orca, Alpaca-GPT4 and Self-instruct, at both 7B and 70B parameter scales. We also demonstrate that rewriting the responses with an LLM is different from direct distillation: the former process yields better win rate at 70B scale, and the two text distributions exhibit significant distinction in the embedding space. Besides, we provide analyses showing that our backtranslated instructions are of higher quality than other sources of synthetic instructions, while our responses are more diverse and complex than what can be obtained from distillation. Overall we find that instruction back-and-forth translation combines the best of both worlds—making use of the information diversity and quantity found on the web, while ensuring the quality of the responses which is necessary for effective alignment.

## introduction in recent years, it is increasingly common for large language models (llms) to be deployed through a chat interface to interact with users' queries. this capability is achieved by taking models that have been pre-trained on massive amounts of webcrawled text and fine-tuning them on a relatively smaller set of instruction-response pairs or preferences @xcite . popular instructiontuning corpora are often constructed by (i) human annotation and curation @xcite @xcite , (ii) converting existing texts, e.g. from other nlp tasks @xcite or web crawls @xcite to instruction-response pairs, and (iii) distilling knowledge from a model @xcite . there are benefits and disadvantages to each of these approaches. while (i) can offer strong baselines @xcite , the reliance on human annotations makes scaling up these datasets difficult. the success of (i) and (ii) is also more dependent on having an effective filtering mechanism to remove noisy data from humans and the web. on the other hand, (iii) is more costeffective and scalable; many current state-of-the-art instruction datasets employ distillation. however, distillation alone has also been claimed to be a "false promise"- @xcite shows that specific capability improvements depend on the coverage of the distilled data and that the performance gains from distilling can plateau quickly. these findings suggest that less noisy and more diverse data is crucial for instruction-tuning to close the gap between open and closed llms. given the knowledge breadth that can be found on the internet, generating instruction-response data with backtranslation has been shown to be a promising alternative to distillation @xcite . in particular, the authors parse the clueweb corpus @xcite for selfcontained text segments, train a model specifically for instruction generation on open assistant data @xcite , and generate instructions to go with the text segments. however, this approach relies on high-quality text data provided by clueweb, a paid-access corpus containing only the most popular web pages from search results, as well as a list of manually specified rules for parsing raw html files into structured responses (which are then used directly for fine-tuning). in this work, we also adopt the instruction backtranslation technique but make it more accessible and scalable. we make direct use of preprocessed documents from a large-scale open-source corpus like dolma @xcite and generate instructions via backtranslation accordingly. we find that the quality of our instructions are comparable to those backtranslated from clueweb. to make up for the lack of manually designed rules for structuring the response, we experiment with using an llm to rewrite the response based on the generated instruction and the initial web text. this also allows us to avoid directly distilling and overfitting to an llm's knowledge. an overview of our pipeline, which we call back-and-forth translation, can be found in figure 1 . given the same data quantity, fine-tuning llama-2-70b on the instruction-response pairs from our data generation pipeline improves the alpacaeval win rate by 3.6% compared to using the backtranslation data from previous work @xcite , and by at least 3.2% compared to using other existing distillation datasets such as openorca @xcite , sharegpt @xcite , evol-instruct @xcite , alpaca-gpt4 @xcite and self-instruct @xcite . by asking a model to rewrite responses based on initial texts extracted from dolma, we obtain a distribution of responses that interpolates between the original web text distribution and the distribution of outputs distilled from the same model. fine-tuning on the rewritten responses in turn outperforms fine-tuning on the distilled responses corresponding to the same instructions (section 5.1). in addition, we offer some insights into how instruction backtranslation and response rewriting affect the quality of instructions and responses respectively, especially in comparison with existing data generation methods (sections 5.2 and 5.3). overall our results suggest that back-andforth translation offers an effective way to generate instruction-tuning data enriched with diverse information found on the web, while ensuring the quality of the response annotations by having aligned llms in the loop. ## method figure 1 shows an overview of our pipeline. here we describe each step in more detail. ## fine-tuning results we validate the effectiveness of our data generation method, by examining the alpacaeval performance of llama-2 models fine-tuned on our data versus on other datasets, given the same instruction quantity. ## understanding rewritten data quality given the performance benefits of rewritten data, we analyze how the outputs obtained from rewriting are different from those obtained from distillation. we also analyze the characteristics of the instruction-response pairs resulting from our pipeline compared to other existing datasets. ## related work we discuss related papers that construct new instruction-tuning datasets or propose methods to improve existing ones. more in-depth review can be found in @xcite . human-crafted data. open assistant @xcite , dolly @xcite and super-ni @xcite are some examples of datasets that contain solely human-generated and human-annotated conversations, covering a range of topics and nlp tasks. these datasets tend to be relatively small in scale due to the expensive costs of manual annotation and verification. other papers do not explicitly ask humans to create questions and answer them, but instead repurpose existing datasets. for example, flan @xcite and natural instructions @xcite transform inputs and outputs of more than 60 nlp tasks into instruction-tuning data. this suffers from the same scalability issue as human-annotated datasets. ## discussion we propose instruction back-and-forth translation: combining instruction backtranslation method from li et al. (2023a) with response rewriting, in order to benefit from both the information diversity found on the internet and the quality of model annotations, while enabling scalability owing to the size of the web corpus where we source initial responses from. future work. our findings motivate a number of interesting future directions. one concrete question is whether applying other existing curation techniques-e.g. quality filters proposed by @xcite -to our pool of (synthetic instructions, rewritten response) pairs would lead to further performance gains. in addition, we also look forward to scaling up our data generation pipeline and studying the implication of the rewritten data on the pre-training process, given concurrent work @xcite that explores paraphrasing pre-training data into the question-answering format. limitations. although we try to control for confounding factors (e.g. data quantity), our findings are only obtained from using one model family, i.e. llama-2 @xcite . besides, our pipeline revolves around general-purpose english instructions, with limited coding or science-related tasks. nevertheless, it is possible to extend our method to more domain-specific data, e.g. by crawling texts from stackoverflow, generating instructions and rewriting the responses with code llama @xcite .