C odec LM : Aligning Language Models with Tailored Synthetic Data (2024)

Abstract: Instruction tuning has emerged as the key in aligning large language models (LLMs) with specific task instructions, thereby mitigating the discrepancy between the next-token prediction objective and users’ actual goals. To reduce the labor and time cost to collect or annotate data by humans, researchers start to explore the use of LLMs to generate instruction-aligned synthetic data. Recent works focus on generating diverse instructions and applying LLM to increase instruction complexity, often neglecting downstream use cases. It remains unclear how to tailor high-quality data to elicit better instruction-following abilities in different target instruction distributions and LLMs. To this end, we introduce CodecLM, a general framework for adaptively generating high-quality synthetic data for LLM alignment with different downstream instruction distributions and LLMs. Drawing on the Encode-Decode principles, we use LLMs as codecs to guide the data generation process. We first encode seed instructions into metadata, which are concise keywords generated on-the-fly to capture the target instruction distribution, and then decode metadata to create tailored instructions. We also introduce Self-Rubrics and Contrastive Filtering during decoding to tailor data-efficient samples. Extensive experiments on four open-domain instruction following benchmarks validate the effectiveness of CodecLM over the current state-of-the-arts.

## introduction large language models (llms) have exhibited remarkable capabilities across a wide array of natural language processing (nlp) tasks @xcite @xcite . in particular, llms can be trained for improved instruction-following through various methods, including fine-tuning on human-annotated data @xcite or extracted knowledge from stronger llms @xcite @xcite . recent progress in this area highlights the critical role of high-quality data in enhancing llms' instruction-following capabilities @xcite @xcite . however, acquiring such data through human annotation remains cost-prohibitive and difficult to scale, hindering further progress. as an alternative solution to human annotation, recent work explores generating instructionresponse pairs for llm alignment by prompting them with example data or prompts and iteratively refining the results @xcite @xcite . while these methods are effective at generating diverse and complex instructions for llm alignment broadly, real-world applications often prioritize tailoring the llm to specific downstream tasks such as individual enterprise applications or personal assistant agents (openai, 2023b), which of-ten involve different instruction distributions. this desideratum for task-specific alignment brings us to a core question for data synthesis: how can we tailor synthetic data to align llms for different instruction-following tasks? specifically, current data synthesis approaches fall short of providing effective solutions for taskspecific llm alignment. while prior works by @xcite and @xcite emphasize diversity and complexity as hallmarks of highquality data, these approaches stumble when facing different downstream tasks that may involve specific instruction distributions. a diverse dataset for one task might not effectively cover the instruction distribution for another. furthermore, the definition of "complex" instructions can be subjective and vary across tasks. to complicate matters further, an llm might excel at some seemingly complex instructions while struggling with others that appear simple according to human-crafted criteria. these limitations underscore the need for a unified data synthesis framework that can generate tailored data to align llms on specific downstream tasks. in this work, we present a novel framework, codeclm, which systematically generates tailored high-quality data to align llms for different downstream tasks. a high-level overview of codeclm is shown in figure 1 . inspired by the principles of encode-decode process @xcite , we leverage a strong llm as a codec to "encode" seed instructions from our target task into instruction metadata and then "decode" the metadata into tailored synthetic instructions. the metadata serves as a word-level abstraction of the input instruction distribution, including the use case and skills for effective instruction following. it can be automatically generated by encoding seed instructions, or directly provided by users with a high-level anticipation of the downstream task. once the metadata is extracted, we then "decode" them to generate tailored instructions. we begin by prompting a llm with the metadata as constraints, creating basic instructions. to elevate the instruction quality, we introduce self-rubrics. it samples appropriate actions from strong llms to make the basic instruction more complex or challenging based on the rubrics it generates for different metadata. intuitively, a general knowledge qa instruction about math would differ in complexity rubrics from one in creative writing about sports. with self-generated rubrics and actions based on metadata, the strong llm crafts instructions that better align the target llm with specific knowledge required for the downstream task. we can run self-rubrics iteratively to control the instruction complexity, similar to @xcite , and finally generate the corresponding responses. we also introduce contrastive filtering during decoding to further identify the most effective instruction-response pairs by leveraging the quality discrepancy between the target and a stronger llm. this strategy identifies two key instruction sets: (a) those the target llm struggles with, pushing it to improve in its weak areas for more significant gains, and (b) those the target llm excels at, feeding them back into the self-rubrics process for improved data efficiency. contrastive filtering serves as a response-level analogy of contrastive decoding @xcite . codeclm sets a new state-of-the-art on four open-domain instruction-following benchmarks with various llm choices, demonstrating its effectiveness in llm alignment for diverse instruction distributions. ## problem statement we study the open-domain instruction following problem @xcite @xcite , where instructions vary in input format and tasks. specifically, we consider two practical scenarios: (1) starting with a given set of n seed instructions d @xmath0, each drawn from some underlying distribution p i . for our experiments, we create a set of seed instructions using a held-out validation set. practically, such instructions can be collected from the usage traffic of users. (2) in the absence of seed instructions, but with prior knowledge of downstream tasks, we directly start with a given set of instruction metadata m (see section 4.1 for definition). the latter scenario is especially useful for end users who lack existing instruction data but wish to jumpstart llm tailored to specific applications, similar to the concept of gpts @xcite . we focus on the first scenario for clarity, though the second can be derived similarly by leveraging an llm as the encoder (section 4.1). our goal is to generate a set of high-quality instruction-response pairs d @xmath1, r j )} m @xmath2, using a strong llm f s , and then use d g to fine-tune the target llm f t . we evaluate the performance of the fine-tuned llm f t on test instructions from the target distribution p i , to which we are aligning. ## codeclm we propose codeclm, a general framework for generating high-quality instruction-response pairs tailored to different downstream tasks and llms, eliminating the need for human annotation. see figure 2 for method overview. ## experiments we conduct comprehensive experiments to evaluate codeclm using different llms on multiple representative benchmarks, closely following wellestablished evaluation settings for open-domain instruction following in prior work @xcite . we also conduct a case study in appendix a.8 to illustrate how codeclm tailors an instruction step by step. ## conclusion in this work, we propose codeclm to tailor synthetic data for llm alignment with different target instruction distributions and llms. we show that codeclm effectively captures the underlying instruction distribution via instruction metadata, and further tailor the most effective instructionresponse pairs through self-rubrics and contrastive filtering. codeclm provides a potent solution towards adapting llms for customized uses, without the necessity of human annotation. we believe codeclm serves as a general framework for targeted llm alignment, which opens the door to multiple promising research directions within the framework, such as richer metadata definition, better prompt design, and more reliable llm-based scorer. codeclm can also benefit from orthogonal research fields, and we continue the discussion in ethical considerations and limitations sections.