C o E vol: Constructing Better Responses for Instruction Finetuning through Multi-Agent Cooperation (2024)

Abstract: In recent years, instruction fine-tuning (IFT) on large language models (LLMs) has garnered considerable attention to enhance model performance on unseen tasks. Attempts have been made on automatic construction and effective selection for IFT data. However, we posit that previous methods have not fully harnessed the potential of LLMs for enhancing data quality. The responses within IFT data could be further enhanced by leveraging the capabilities of LLMs themselves.In this paper, we propose CoEvol, an LLM-based multi-agent cooperation framework for the improvement of responses for instructions. To effectively refine the responses, we develop an iterative framework following a _debate-advise-edit-judge_ paradigm. A two-stage multi-agent debate strategy is further devised to ensure the diversity and reliability of editing suggestions within the framework. Empirically, models equipped with CoEvol outperform competitive baselines evaluated by MT-Bench and AlpacaEval, demonstrating its effectiveness in enhancing instruction-following capabilities for LLMs.

## introduction instruction fine-tuning (ift) is an effective approach for enhancing the performance of language models in zero-shot and few-shot scenarios on previously unseen tasks. improving the instructionfollowing capabilities of large language models (llms) has received increasing attentions from the natural language processing (nlp) community @xcite @xcite . recent research has been focusing on constructing substantial quantities of ift data with minimal human effort @xcite , where data construction is highlighted by researchers from multiple perspectives, including diversity, instruction complexity, and the quality of responses to instructions @xcite . to address the issue of easy or moderately difficult human-crafted instructions, several approaches @xcite @xcite have been developed to generate instructions of varying complexity levels. by emphasizing both complexity and diversity of the instructions, @xcite annotate ift data using advanced llms and introduce a complexity-focused diverse sampling method for data selection. since lima @xcite suggested that the quality of ift data is more important than its quantity, a series of data selection methods have been proposed, focusing on extracting high-quality samples from existing datasets of uneven distribution of qualities @xcite @xcite . however, we observe that due to the inherent characteristics of causal language modeling, llms sometimes fail to deliver the most comprehensive and reasonable answers they can produce. we posit that previous data construction approaches have not fully harnessed the potential of these llms. the responses within ift data could be further refined by leveraging the capabilities of llms themselves. due to the diversity and complexity of instructions within ift data, refining the present response is not a trivial task. consequently, we are attempting to introduce multi-agents to cooperate in this endeavor. although multi-agent debate (mad) has been proven effective in answer improvement @xcite and evaluation @xcite by prompting the diversity of thought, strengthening the divergent thinking of agents cost-effectively remains challenging. we categorize these approaches into two types based on their debate strategies: free debate ## method in this section, we introduce our proposed framework for efficiently editing ift data. figure 1 shows the architecture of coevol, where five llmbased agents: two counter-part debaters, an advisor, an editor, and a judge are assigned in one pipeline to finish the task altogether. ## analysis 4.1 ablation study to verify the validity of each component within coevol, we conduct an ablation study of the framework. continuing from experiments in section 3.1, we evolve the 9k randomly sampled data from d alpaca under the following settings: with other settings held constantly, we respectively fine-tune llama2-7b on these evolved data. evaluation results of the ablated models on mt-bench and alpacaeval are shown in we provide more detailed examples of data evolution using ablation models in appendix e. ## related work instruction fine-tuning for llms in recent years, instruction fine-tuning (ift) has become a prevalent approach for enhancing the applicability of pre-trained language models (plms) and improving their generalization capabilities on unseen tasks @xcite . for instruction fine-tuning (ift) on llms, attempts @xcite @xcite have been made to construct ift data with the help of human annotators, yet these methods are time-consuming and labor-intensive. to tackle this issue, pipelines @xcite are proposed to automatically generate data instances from seed tasks. in addition to single-turn instruction following data @xcite , largescale multi-turn dialogues for ift are also constructed to further enhance llms on chat scenarios @xcite @xcite . concurrently, a series of studies show that the complexity, diversity, and quality of ift data significantly influence model alignment. concerning the complexity of instructions, approaches @xcite @xcite are designed to create large amounts of instruction data with different levels of complexity. aware of both complexity and diversity of instructions, @xcite tag data with a strong llm and introduce a complexity-focus diverse sampling method for data selection. in the pursuit of data with high-quality response, chen et al. ( 2024) diverging from prior work, we focus on how to further improve the quality of responses in ift data. through our proposed framework coevol, the potential of llm-based multi-agents is unleashed in collaboration to automatically edit responses, thereby generating high-quality data for fine-tuning superior llms. ## conclusion in this paper, we introduce coevol, an innovative framework for efficient quality improvement on ift data through multi-agent cooperation. to fully exploit the potential of llms for response editing, we propose a two-stage mad strategy to maximize the diversity of perspectives within debate while minimizing the cost of agents. following a debateadvise-edit-judge paradigm, we establish a pipeline to harness the collective power of agents with distinct roles. experimental results substantiate the efficacy of our proposed framework, showcasing its superiority in evolving better ift data through response augmentation. codes, datasets, and model weights developed in this paper are publicly available. we hope this work can offer new perspectives and references for the automatic construction of high-quality data.