Parrot: Enhancing Multi-Turn Instruction Following for Large Language Models (2024)

Abstract: Humans often interact with large language models (LLMs) in multi-turn interaction to obtain desired answers or more information. However, most existing studies overlook the multi-turn instruction following ability of LLMs, in terms of training dataset, training method, and evaluation benchmark. In this paper, we introduce Parrot, a solution aiming to enhance multi-turn instruction following for LLMs. First, we introduce an efficient but effective method for collecting multi-turn instructions that feature human-like queries, such as anaphora and ellipsis. Second, we propose a context-aware preference optimization strategy to further enhance LLMs for complex queries in multi-turn interaction. Moreover, to quantitatively evaluate LLMs in multi-turn instruction following, we manually build a multi-turn benchmark derived from existing ones. Extensive experiments show that Parrot improves current LLMs by up to 7.2% in multi-turn instruction following. Our dataset and codes will be open-sourced to facilitate future research.

## introduction large language models (llms) @xcite @xcite @xcite @xcite @xcite have demonstrated their strong capability in understanding a range of human instructions. by leveraging synthetic or human-created instructions to fine-tune the llama model @xcite , a series of studies @xcite @xcite @xcite achieve promising results, even in some benchmarks @xcite performing close to chatgpt and gpt-4. however, most of these studies and benchmarks overlook the multi-turn instruction following ability of llms, which is a more common demand in real-world scenarios. developing llms capable of multi-turn interaction presents significantly greater challenges compared to single-turn interaction. in terms of collecting instruction tuning data, as the number of turns increases, the cost of manual annotation for data collection rises considerably, making it economically unaffordable. existing user logs with chat-gpt, such as sharegpt, are limited in quantity, and approximately 60% of the data does not exceed three turns, which is still not enough to achieve satisfactory performance @xcite . some llm-based multi-turn instruction generation methods, even after employing specific prompts to let chatgpt act as a user @xcite , usually produce information-complete queries that lack common features found in human queries, such as anaphoras and ellipses, as shown in fig. 1 . in terms of training strategies, most of the current models depend on straightforward supervised finetuning (sft) methods @xcite @xcite , without specific optimization design for complex queries involving anaphora and ellipsis in multi-turn interactions. it may mislead the llm to neglect or hallucinate the context when generating responses. to overcome the above challenges, we propose parrot, which aims to facilitate the development of llms with stronger multi-turn instruction following capabilities. first, we introduce an efficient but effective approach for automatically collecting multi-turn instruction tuning data. instead of designing complicated prompts to make powerful llms (such as chatgpt or gpt-4) act as users and generate queries, we propose training a parrot-ask model based on a smaller llm to learn features found in human queries from a small number of real user dialogues, and then it can be used for generating human-like queries. second, to enhance the capability of llms in handling complex queries during multi-turn interactions, we propose a context-aware preference optimization (capo) strategy. we first select queries that heavily rely on contextual information to obtain accurate responses, and then construct negative responses by simulating common error scenarios such as ignoring context or misunderstanding anaphora. subsequently, we leverage these pairs to optimize the llm's preferences so that they can better exploit contextual information when generating responses. moreover, for a quantitative assessment of llms' multi-turn abilities, we ask annotators to expand the mt-bench benchmark @xcite that originally consists of two-turn queries, to an eight-turn mt-bench++ benchmark. mt-bench++ includes complex queries like anaphoras, ellipses, and topic transitions, better reflecting real multiturn interaction. we conduct extensive experiments on mt-bench and mt-bench++. compared with previous approaches, our proposed method can generate higher-quality multi-turn instruction tuning data with more human-like queries. our multi-turn instruction dataset combined with our proposed capo strategy improves the current llms by 7.2% in multi-turn instruction following evaluation. we summarize our contributions as follows: ## approach we describe our proposed parrot framework in this section. as illustrated in fig. 2 (a) , we first train a parrot-ask model to mimic the asking style of humans in generating multi-turn queries and then use it for collecting a multi-turn instruction tuning dataset. then we design three strategies to construct negative responses, involving context neglect, context hallucination, or context misunderstanding as shown in fig. 2 (b) , to compose context awareness preferences. finally, we use the collected data to train the parrot-chat model by instruction tuning (see fig. 2 (c )) and context-aware preference optimization (see fig. 2 (d )) to enhance its multi-turn instruction following capability. ## conclusion in this paper, we propose parrot for enhancing the multi-turn instruction following capability of llms, including an automatic method for collecting multi-turn instruction tuning data with humanlike queries, a specifically designed context-aware preference optimization strategy to further enhance llms for complex queries in multi-turn interaction. we also build an eight-turn mt-bench++ evaluation benchmark to qualitatively evaluate multi-turn instruction following ability. we demonstrate the collected parrot-40k dataset is superior to existing multi-turn instruction-tuning datasets on the number of turns and resemblance to human queries. with the help of such a high-quality dataset and proposed capo strategy, our parrot-chat model significantly outperforms other 13b open-source models on mt-bench and our constructed mt-bench++ benchmarks. we will make all codes and datasets publicly available to facilitate further advancements in this area.