Instruct Once, Chat Consistently in Multiple Rounds: An Efficient Tuning Framework for Dialogue (2024)

Abstract: Tuning language models for dialogue generation has been a prevalent paradigm for building capable dialogue agents. Yet, traditional tuning narrowly views dialogue generation as resembling other language generation tasks, ignoring the role disparities between two speakers and the multi-round interactive process that dialogues ought to be. Such a manner often leads to unsatisfactory chat consistency for the built agent. In this work, we emphasize the interactive, communicative nature of dialogue and argue that it is more feasible to model the speaker roles of agent and user separately, enabling the agent to adhere to its role consistently. With this in mind, we propose an efficient Multi-round Interactive Dialogue Tuning (Midi-Tuning) framework. It models the agent and user individually with two adapters built upon large language models. The adapters make use of respective utterances round by round in alternating order and they are tuned via a round-level memory caching mechanism. Extensive experiments demonstrate that, our framework performs superior to traditional fine-tuning and harbors the tremendous potential for improving dialogue consistency.

## introduction building human-like intelligent dialogue agents is a long-standing ambition for the research community of dialogue systems. recently, we have witnessed a substantial revolution in advanced conversational agents such as chatgpt @xcite and @xcite , which are fundamentally built upon large language models (llms) @xcite . similar efforts have also been made by academia and open-source communities, leading to a variety of notable chat language models, such as vicuna @xcite , koala @xcite , and llama 2-chat @xcite . these chat language models can be attained by instruction fine-tuning on downstream dialogue data, demonstrating promising performance in generating natural and comprehensive responses. tuning llms for dialogue generation has been the de-facto mainstream practice towards creating capable dialogue agents. traditional dialogue tuning narrowly views dialogue generation as resembling other language generation tasks without distinction. it performs in either one-dialogue-n-sample (see figure 1(a) ) or one-dialogue-one-sample (see figure 1(b) ) man-ner. the former transforms dialogue model training into general language generation via splitting each multi-round dialogue into multiple single-round samples, yet results in non-independent distributions among those samples. the latter enhances training efficiency by utilizing each multi-round dialogue at once, which computes the prediction loss for the agent's responses through causal masks, such as vicuna @xcite and ultral-lama @xcite . however, these methods simply concatenate utterances from two speakers (e.g., user and agent) together (and instructions for the agent, if any) and mix their content in the same language model space, ignoring the role disparities between two speakers and the multi-round interactive process that dialogues ought to be. such tuning methods inevitably hinder a built dialogue agent from maintaining the chat consistency @xcite , requiring that the agent always adhere to its role even with the dialogue rounds moving forward. it remains urgent to solve for many consistency-demanding scenarios. one of the primary challenges for improving dialogue consistency lies in the disparity modeling of the two speaker roles. it is because the inconsistency issue in real-world human communication @xcite @xcite is often caused by various types of speaker disparities, such as background knowledge, cognitive level, personalities, and goals. we emphasize that it is more feasible to model the roles of agent and user separately (see figure 1(c )), such that the agent and user models can consistently adhere to their respective roles and interact with each other round by round, similar to humans. when tuning llms for conversation, we have a similar motivation towards consistent dialogue generation. we propose a general, simple, and effective framework, namely multi-round interactive dialogue tuning (midi-tuning). it employs two language model adapters (e.g., @xcite ) built upon llms, to represent the agent and user, respectively. the two adapters are tuned by utilizing respective utterances round by round in alternating order, with each adapter learning to distinguish language model distribution about its role. however, such separate modeling is non-trivial in tracking the complete dialogue context. considering that the foundation architecture of mainstream llms is transformer @xcite , we propose a round-level memory caching mechanism to address it efficiently, which reuses previousround cached keys and values as ongoing context when processing present-round utterance. in summary, our main contributions are as follows: (1) to the best of our knowledge, this is the first work investigating how a new way of tuning could affect dialogue consistency in the era of llms. (2) we propose midi-tuning, a general, simple, and efficient framework to tune llms for dialogue generation, which can be applied in broad downstream dialogue scenarios. (3) extensive experiments demonstrate that midi-tuning outperforms traditional fine-tuning over various llms, especially in maintaining consistency for multi-round dialogues. ## our method we first provide the necessary background about general dialogue generation and lora-based tuning (see §3.1). then, we dive into the details of the proposed multi-round interactive dialogue tuning (midi-tuning) framework (see §3.2). ## conclusion this work explores how the way of tuning can improve the consistency of dialogue generation over multiple rounds. we highlight the importance of separately modeling agents and users due to their role disparities. we propose a general, efficient tuning framework called midi-tuning, which rep-resents the agent and user using two adapters and tunes them via round-level memory caching. empirical experiments show that our framework outperforms traditional dialogue tuning significantly.