Review-Instruct: A Review-Driven Multi-Turn Conversations Generation Method for Large Language Models (2025)

Abstract: The effectiveness of large language models (LLMs) in conversational AI is hindered by their reliance on single-turn supervised fine-tuning (SFT) data, which limits contextual coherence in multi-turn dialogues. Existing methods for generating multi-turn dialogue data struggle to ensure both diversity and quality in instructions. To address this, we propose Review-Instruct, a novel framework that synthesizes multi-turn conversations through an iterative “Ask-Respond-Review” process involving three agent roles: a Candidate, multiple Reviewers, and a Chairman. The framework iteratively refines instructions by incorporating Reviewer feedback, enhancing dialogue diversity and difficulty. We construct a multi-turn dataset using the Alpaca dataset and fine-tune the LLaMA2-13B model. Evaluations on MT-Bench, MMLU-Pro, and Auto-Arena demonstrate significant improvements, achieving absolute gains of 2.9% on MMLU-Pro and 2% on MT-Bench compared to prior state-of-the-art models based on LLaMA2-13B. Ablation studies confirm the critical role of the Review stage and the use of multiple Reviewers in boosting instruction diversity and difficulty. Our work highlights the potential of review-driven, multi-agent frameworks for generating high-quality conversational data at scale.

## introduction the rapid advancement of large language models (llms) @xcite has revolutionized conversational ai systems. while these models demonstrate impressive capabilities in single-turn interactions, their performance in multi-turn dialogues remains suboptimal due to limitations in existing supervised fine-tuning (sft) data @xcite . current approaches predominantly rely on single-turn conversational data for sft @xcite @xcite , which fundamentally constrains the models' ability to maintain contextual coherence in extended dialogues @xcite . this limitation stems from two primary challenges in multi-turn data acquisition. first, simple concatenation of single-turn dialogues produces artificial conversations lacking natural flow @xcite . second, while human-ai interaction logs provide authentic data @xcite , they raise privacy concerns and require substantial collection efforts. recent attempts to automate dialogue generation through dual-agent "ask-respond" frameworks @xcite have shown promise but face inherent limitations. specifically, the asymmetric capabilities of llms in question generation versus answering, coupled with data scarcity for training specialized query generators @xcite , hinder the production of high-quality multi-turn dialogues at scale. to address these challenges, we propose review-instruct, a novel framework that introduces structured feedback mechanisms into dialogue generation. drawing inspiration from the evaluation methodologies in auto-arena @xcite , our approach extends the conventional "ask-respond" paradigm with a critical review stage, creating an "ask-respond-review" pipeline. as illustrated in figure 1 , the framework employs three distinct ai agents: a candidate, multiple reviewers, and a chairman. the process begins with the chairman selecting an instruction from a predefined instruction dataset. following the candidate's response, multiple reviewers conduct parallel evaluations using criteria including relevance, coherence, and depth. the chairman then aggregates these assessments to generate contextually appropriate follow-up instructions, creating an iterative refinement loop that enhances both dialogue quality and complexity. our experimental results demonstrate significant improvements over existing approaches. fine-tuning a llama2-13b model 1 with review-instruct generated data achieves state-of-the-art performance on key benchmarks: 2.9% absolute improvement on mmlu-pro @xcite , 2.0% absolute gain on mt-bench @xcite and superior performance in auto-arena pairwise comparisons @xcite . ablation studies reveal two critical success factors: (1) the review stage increases instruction diversity by 18.6% and difficulty by 33.4% compared to ask-respond paradigm, and (2) multi-reviewer configurations generate 7.5% more diversity instructions and 19.6% more difficulty instructions than singlereviewer setups. the principal contributions of this work are summarized as follows: the remainder of this paper is organized as follows: section 2 details the review-instruct methodology. section 3 presents experimental results and benchmark comparisons. sections 4 and 5 analyze the critical role of the review stage through comprehensive data-driven investigations. section 6 reviews related work in sft data generation. sections 7 and 8 discuss future research directions and examine potential limitations of our approach, respectively. ## method this chapter presents our proposed review-instruct method. section 2.1 provides a detailed explanation of this process, while section 2.2 outlines the responsibilities of each role. ## dataset analysis as previously mentioned, the common method for synthesizing multi-turn conversations relies on the ask-respond paradigm. building upon this foundation, our proposed review-instruct framework introduces a novel review stage. this section provides a data-driven analysis of the review stage's impact, specifically examining its influence on instruction diversity and difficulty. these metrics are crucial for assessing the quality of instructiontuning data @xcite @xcite . ## case study we conclude with a case analysis illustrating how instructions evolve throughout the review-instruct process. two primary methods of instruction evolution, breadth evolution and depth evolution, were discussed in evol-instruct @xcite . these methods utilize specifically crafted prompts to 3 https://huggingface.co/ofa-sys/instagger chairman: summarize the events of the 1787 constitutional convention. candidate: the 1787 constitutional convention was convened to amend the articles of ... and the "three-fifths compromise" regarding the issue of slavery. on september 17... chairman: please explain the long-term impact of the "three-fifths compromise" on the u.s. constitution and political system. an example illustrating breadth evolution is in an example of depth evolution is provided in ## related work the development of advanced multi-turn dialogue and instruction-following capabilities in llms relies heavily on quality supervised fine-tuning data. while models such as vicuna @xcite and realchat @xcite effectively utilize user-llm interaction logs from platforms like sharegpt and chatbot arena @xcite , the manual creation of instructions is both labor-intensive and time-consuming, not to mention fraught with privacy concerns. in light of recent advancements in llm technology, there has been a surge in research focused on the automated generation of instruction datasets. various methods have emerged for this purpose. for example, self-instruct @xcite begins with a small collection of seed instructions and expands these using few-shot prompt. evol-instruct @xcite refines or develops new instructions through a systematic process of indepth and breadth evolution. humpback @xcite uses instruction backtranslation to generate instructions based on web content. magpie @xcite ) employs a self-synthesis approach, utilizing chat model to automate instruction generation. persona-hub @xcite introduces a persona-driven strategy, guiding llms to create instructions from diverse perspectives. although these methods demonstrate effectiveness in synthesizing a wide range of high-quality instructions, they typically focus on single-turn conversations, which limits their applicability in multi-turn conversations scenarios. some recent approaches have begun to explore the generation of multi-turn conversations. baize @xcite and ultrachat @xcite , for instance, leverage two chatgpt apis to simulate interactions between users and assistants. pla-tolm @xcite and parrot @xcite adopt a different strategy, training user simulators to interact with chatgpt. these methods can all be categorized under the ask-respond pattern. in contrast to existing approaches, our proposed review-instruct method augments the ask-respond paradigm with a novel review stage. the incorporation of this review stage, and the utilization of the reviews generated therein, facilitates the creation of instructions that demonstrate a marked improvement in both diversity and difficulty. furthermore, the ability to synthesize multi-turn conversations from existing instruction datasets broadens the applicability of our approach. our proposed method can be effectively integrated with existing approaches such as self-instruct @xcite , evol-instruct @xcite , magpie @xcite , and persona-hub @xcite . these methods can generate extensive instruction datasets, which can then be synthesized into multi-turn dialogue data using our approach. ## conclusion and future work this study explores methods for synthesizing multiturn conversational data. we introduce the review-instruct, a novel approach based on the ask-respond-review framework, which transforms instructions into multi-turn conversations. experimental results demonstrate that this method is of high quality and significantly enhances model performance. further analysis reveals that the review process substantially increases both the diversity and difficulty of the instructions compared to a basic ask-respond pattern. beyond its capability in synthesizing textual instruction data, the proposed framework exhibits considerable potential for generating multi-modal instruction data. in multi-modal settings, the iterative process initiates with a single image presented to the chairman. the chairman then generates a question based on this visual stimulus, which the candidate is required to address. the remaining steps of the procedure align with the methodology outlined in the preceding sections. a case illustrating is provided in appendix a.4, highlighting directions for our future research.