Reformatted Alignment (2024)

Abstract: The quality of finetuning data is crucial for aligning large language models (LLMs) with human values. Current methods to improve data quality are either labor-intensive or prone to factual errors caused by LLM hallucinations. This paper explores elevating the quality of existing instruction data to better align with human values, introducing a simple and effective approach named ReAlign, which reformats the responses of instruction data into a format that better aligns with pre-established criteria and the collated evidence. This approach minimizes human annotation, hallucination, and the difficulty in scaling, remaining orthogonal to existing alignment techniques. Experimentally, ReAlign significantly boosts the general alignment ability, math reasoning, factuality, and readability of the LLMs.Encouragingly, without introducing any additional data or advanced training techniques, and merely by reformatting the response, LLaMA-2-13B’s mathematical reasoning ability on GSM8K can be improved **from 46.77% to 56.63%** in accuracy. Additionally, a mere 5% of ReAlign data yields a 67% boost in general alignment ability measured by the Alpaca dataset. This work highlights the need for further research into the science and mechanistic interpretability of LLMs. We have made the associated code and data publicly accessible to support future studies at https://github.com/GAIR-NLP/ReAlign.

## introduction alignment has been witnessed to be an effective technique for aligning large language models (llms) to human values and human intent @xcite , which usually requires fine-tuning on a large amount of synthetic data derived from llms @xcite in this paper, instead of focusing on the creation of instruction data from scratch, we investigate how existing instruction data can be made higher quality and better aligned with human values. we propose a simple and effective method, named re-align, which is orthogonal to the above existing approaches. specifically, realign necessitates a base instruction dataset, which can be sourced from extensive existing supervised datasets (e.g., gsm8k (cobbe et al., 2021)), or publicly available instruction data compiled through various methods (e.g., self-instruct @xcite , evol-instruct (xu et al., 2023), and self-alignment (li et al., 2023b)). the realign process unfolds in three main steps. the first step involves criteria definition ( §3.1), where humans define their preferences (e.g., the preferred format of responses) in various scenarios in the form of natural language. in this paper, we meticulously define criteria for 46 distinct scenarios. the second step, retrieval augmentation ( §3.2), broadens the knowledge base for knowledge-intensive tasks like open-domain qa and fact verification. this is achieved by incorporating additional information, thereby improving the factuality and informativeness of responses. the final step, reformatting ( §3.3), aims to re-align the responses with the pre-established criteria and the collated evidence, guaranteeing outputs that are both structured and substantiated. as demonstrated in fig. 1 , the realigned response provides a better format and a clearer chain of thoughts. the underlying philosophy of realign is to re-coordinate the roles of humans and llms in the alignment process, leveraging their complementary strengths -humans articulate their preferences, and llms, in turn, reconstruct instructions based on their generative power (e.g., instructionfollowing ability), without directly using distilled llm knowledge. through this collaborative synergy, we expect the generated instruction data to be not only more contextually precise but also more closely aligned with human preferences. we operationalize this idea on five types of existing instruction data, where three are general datasets (i.e., open-platypus ## realign given a base instruction dataset @xmath0, r 1 ), • • • , (q n , r n )}, where q and r are the input query and response respectively, re align aims to improve the quality of responses by three steps as shown in fig. 2: (1) criteria definition: defining the criteria including tasks and formats for each task, (2) retrieval augmentation: retrieving relevant external information for the knowledge-intensive tasks, and (3) reformatting: reformatting the original response based on the guidance consisting of hand-written format and the retrieved information. an overview of our method is shown in fig. 2 . ## conclusion in this work, we propose realign, a simple and effective method for alignment, which automatically improves the quality of the existing instruction datasets while minimizing labor costs and hallucinations. we create five new high-quality datasets from open-platypus