Evolutionary Contrastive Distillation for Language Model Alignment (2024)

Abstract: The ability of large language models (LLMs) to execute complex instructions is essential for their real-world applications. However, several recent studies indicate that LLMs struggle with challenging instructions. In this paper, we propose Evolutionary Contrastive Distillation (ECD), a novel method for generating high-quality synthetic preference data designed to enhance the complex instruction-following capability of language models. ECD generates data that specifically illustrates the difference between a response that successfully follows a set of complex instructions and a response that is high-quality, but nevertheless makes some subtle mistakes. This is done by prompting LLMs to progressively evolve simple instructions to more complex instructions. When the complexity of an instruction is increased, the original successful response to the original instruction becomes a “hard negative” response for the new instruction, mostly meeting requirements of the new instruction, but barely missing one or two. By pairing a good response with such a hard negative response, and employing contrastive learning algorithms such as DPO, we improve language models’ ability to follow complex instructions. Empirically, we observe that our method yields a 7B model that exceeds the complex instruction-following performance of current SOTA 7B models and is competitive even with open-source 70B models.

## introduction large language models (llms) have demonstrated impressive capabilities in a wide range of tasks ranging from creative writing to code generation. in light of these achievements, there has been a surge of interest in building complex data processing systems with llm-based components @xcite . complex instruction-following capability, which is the capability of llms to generate outputs consistent with multiple interdependent specifications in the prompt, is critically important for such systems to operate reliably. consequently, multiple benchmarks have been proposed to capture various aspects of complex instructionfollowing @xcite @xcite . unfortunately, these studies find that there still is a significant gap between proprietary llms and open source models on these benchmarks. this raises the question: how shall we effectively distill the complex instruction-following ability of stronger llms into smaller language models? after the seminal work by @xcite , the usage of proprietary llms for the generation of alignment data has been actively studied. as a typical alignment pipeline consists of supervised fine-tuning (sft) methods and preference fine-tuning (pft foot_0 ) stages @xcite , these data generation methods are also largely categorized into sft data generation methods (e.g., ultra-chat @xcite ) and pft data generation methods (e.g., ultrafeedback @xcite ). among a wide variety of sft data generatin methods, evol-instruct @xcite and conifer @xcite are specifically designed to improve complex instruction-following. they share the common goal: generate highly complex instructions. and they also share the same evolutionary strategy: they first prompt proprietary llms to evolve simple seed instructions from sharegpt to have progressively more complex requirements. then, proprietary llms are again prompted to generate demonstrations on these complex instructions. figure 1 : example of ecd preference pair construction. the original instruction is to provide three examples of sports while the evolved instruction adds the additional requirement to sort the examples in alphabetical order. since the original response violates this requirement, it can be used as a rejected response in the preference pair. on the other hand, the evolved response satisfies all the requirements and can be used as the chosen response. on the other hand, while a wide variety of pft data generation methods have been proposed in the literature @xcite @xcite there has been little attention on leveraging pft stage for complex instructionfollowing. we claim, however, that pft stage has distinctive advantages in teaching complex instruction-following. first, sft training places an equal weight on every token in the response, and this can be less than ideal for instruction-following. when a tricky requirement such as "make sure every sentence rhymes with each other" is imposed in the instruction, certain tokens play a more important role in meeting this requirement than other tokens. in the pft stage, we can leverage preference pairs which are similar in content but differ in these instruction-critical tokens to more directly illustrate the causality between the token generation and the successful instruction-following. second, while the sft stage can utilize positive examples only, pft stage can also leverage negative examples to highlight the fine difference between a successful instruction-following and a subtle failure. to this end, we propose a novel technique, evolutionary contrastive distillation (ecd), for generating high-quality, synthetic preference data targeting complex instruction-following. following @xcite ; @xcite , we prompt proprietary llms to progressively increase the complexity of instructions. instead of separating the original example and the evolved example as two independent examples for sft, however, we connect them as a preference pair for pft: the proprietary llm's response on the evolved instruction is considered as a positive example, and its response on the original instruction is considered a negative example. the key observation is that when an instruction is evolved to have a new requirement, the original good response for the original instruction is not a good response for the evolved instruction anymore, and therefore can be used as the negative example to the evolved instruction. since the evolution of instructions is gradual, however, the original response still satisfies most of the requirements of the evolved instruction, and therefore can be used as a "hard" negative for the evolved instruction. see figure 1 for a concrete example. this method has the desireable properties that (i) it does not rely on llms to annotate preferences or generate undesirable responses, which can be unreliable @xcite and (ii) it is effective at creating hard negative examples, which have are crucial in contrastive learning @xcite @xcite . to ensure that each step of instruction evolution provides a high-quality example of subtle nuance in instruction-following, we devise a fine-grained hierarchical taxonomy of evolution operations, which we discuss in section 4. we demonstrate the data generated from this taxonomy is not only effective at improving complex instruction-following, but can also be combined with previous evolutionary @xcite as well as non-evolutionary @xcite methods to yield even stronger results. therefore, we believe the proposed taxonomy will serve as a useful resource on its own for future research on complex instruction-following. we validate the effectiveness of our approach in an extensive set of experiments, benchmarking our models on three recent instruction-following benchmarks: ifeval, followbench, and in-fobench. our approach yields a state-of-the-art 7b model, improving on prior sota (state-of-the-art) instruction-following 7b models by 7pp on ifeval and is competitive with popular open-source models at the 70b scale. furthermore, we develop a recipe to build sota instruction-following models that also achieve highly competitive performance on popular conversational quality benchmarks such as mt-bench and alpacaeval. finally, we perform ablations illustrating the advantages of ecd over other synthetic preference data generation methods such as rlaif and rlcd, and the importance of using contrastive learning (e.g., dpo) instead of sft to learn from preference pairs. contrastive learning. the effectiveness of contrastive learning @xcite has been shown across numerous modalities @xcite @xcite . in the contrastive learning literature, the importance of the quality of negative samples has been well-established @xcite . numerous "hard" negative mining techniques have been introduced to improve the quality of negative samples @xcite @xcite . while many alignment methods such as instructgpt @xcite and dpo @xcite leverage negative samples to facilitate learning @xcite , how to obtain sufficiently hard negative samples has not been actively studied. @xcite em-ploys hard negative mining techniques to improve the robustness of the model. our work instead targets improving complex instruction-following, and uses llm prompting to generate negative examples rather than mining samples from a fixed dataset. data generation methods for pft. much attention has been given to how to generate preference data for fine-tuning llms. pioneering works focused on collecting feedback from humans @xcite @xcite . under this approach, human annotators would observe a set of responses to an instruction and rank them according to their preferences. as human annotation is expensive and difficult to scale, however, follow-up works have proposed alternative synthetic data generation approaches. the most popular method of generating pft data from llms is to sample two independent responses on the same instruction, and then ask an llm to judge which one is of higher quality. this method is often called rlaif (reinforcement learning from ai feedback) @xcite . while rlaif methods have shown encouraging results @xcite @xcite , obtaining high-quality preference annotations from llms have been found challenging @xcite . this is because the difference between two responses on the identical instruction can be minor, and llm-based preference annotation can be subject to many confounding factors, such as self-bias and verbosity bias @xcite . to avoid quality issues from llm-based preference annotation, reinforcement learning from contrastive distillation (rlcd) @xcite proposes to employ two different prompt templates rather than using the same one. one prompt template is designed to elicit desirable responses, and another is designed to elicit undesirable response. such a deliberate design of templates allows rlcd to bypass preference annotation. as we discuss in section 5, however, we find it challenging to prompt-engineer proprietary llms to generate undesirable responses. as proprietary llms are already aligned to generate helpful and harmless responses @xcite , very often, their response from the undesirable response template are either as helpful as the response from the desirable response template, or trivially unhelpful ("no, i can't answer that question."), limiting their value as a hard negative example. in contrast to rlhf or rlaif, ecd does not require preference annotation by neither humans nor llms. this circumvents issues with preference annotation quality. in contrast to easy negative examples from rlcd, negative examples from ecd are hard, as they are helpful responses on a similar but subtly different instruction. ## evolutionary contrastive distillation first, we propose the evolutionary data generation process to synthesize preference data for complex instruction-following. then, we discuss how contrastive learning methods shall be employed to train on this generated data. ## fine-grained evolution taxonomy in order to ensure the evolutionary process generates diverse variations of instructions, we define a fine-grained hierarchical taxonomy of evolution operations. each evolution operation samples a leaf node from this taxonomy to determine the type of operation and the corresponding prompt template. the top level of the taxonomy consists of five categories: then, for each of the top-level category, we define around five fine-grained evolution types. see for illustration, consider the following example of an evolution step: the letter should have the following format: see figure 2 for a graphical illustration. ## conclusion in this paper, we proposed a novel approach to generate high-quality synthetic preference data for complex instruction-following, evolutionary contrastive distillation. using this approach, we trained models at the 7b scale that achieved stateof-the-art performance in instruction-following as measured by benchmarks such as ifeval, follow-bench, and infobench and achieved competitive performance on conversational quality benchmarks like alpacaeval and mt-bench. for example, one of our checkpoints improves over the prior sota, conifer-7b-dpo, at the 7b scale on ifeval loose prompt accuracy by 7pp while also achieving a score of 7.35 on mt-bench and a length-controlled win-rate against gpt-4-turbo of 22%. due to the generality of our approach, we believe that there are opportunities to apply it to many other domains beyond complex instruction-following. indeed, recent research has already produced evolved ift data for mathematics and coding @xcite with impressive results and our approach offers a way to convert these datasets into preference data so that models can learn from both positive and negative feedback. as part of our future work, we plan to explore these other domains.