Inductive-Deductive Strategy Reuse for Multi-Turn Instructional Dialogues (2024)

Abstract: Aligning large language models (LLMs) with human expectations requires high-quality instructional dialogues, which can be achieved by raising diverse, in-depth, and insightful instructions that deepen interactions. Existing methods target instructions from real instruction dialogues as a learning goal and fine-tune a user simulator for posing instructions. However, the user simulator struggles to implicitly model complex dialogue flows and pose high-quality instructions. In this paper, we take inspiration from the cognitive abilities inherent in human learning and propose the explicit modeling of complex dialogue flows through instructional strategy reuse. Specifically, we first induce high-level strategies from various real instruction dialogues. These strategies are applied to new dialogue scenarios deductively, where the instructional strategies facilitate high-quality instructions. Experimental results show that our method can generate diverse, in-depth, and insightful instructions for a given dialogue history. The constructed multi-turn instructional dialogues can outperform competitive baselines on the downstream chat model.

## introduction large language models (llms) @xcite have demonstrated emergent capabilities across a wide range of language-related tasks naturally and interactively. this mainly benefits from the alignment of llms with humans. the alignment requires high-quality multi-turn instructions for fine-tuning llms in a dialogue-based setting @xcite . such instructional dialogues usually require instructions that are diverse and in-depth @xcite . specifically, diverse instructions help instructionfollowing llms adapt to different situations and user needs. in-depth instructions help llms follow the logical flows of dialogues and deepen the understanding of dialogue histories to answer users' instructions. however, manually collecting such instructional dialogue data is usually labor-intensive and time-consuming @xcite . a feasible approach to automatic collection is leveraging two llms to interact: one simulating a user to pose instructions, and the other acting as a system agent to respond. user simulators are currently implemented by employing roleplayed llms to simulate humans through instruction prompts @xcite or fine-tuning llms to learn instruction generation from real instructional dialogues @xcite . however, dialogue data allows for different dialogue flows for the given dialogue history, i.e., various instructions can be posed @xcite . llms in these user simulators implicitly learn the underlying rules guiding these different flows from training data. due to the complexity and variability of these rules, llms struggle to capture various possible rules without explicit guidance so that the user simulators tend to capture frequently co-occurring patterns and generate more generic instructions @xcite . this results in diminished diversity and insufficient depth of multi-turn instructions. in this paper, we propose to explicitly model the rules behind how dialogues can flow in different directions, posing diverse and in-depth instructions. inspired by the cognitive abilities in human learning @xcite , we observe that humans can gain general rules from various real dialogues via inductive reasoning @xcite , which guides the dialogue flows in certain directions. as shown in figure 1 , humans induce rules such as "ask for specific information to clarify detail" and "information validation and illustration". we refer to these rules as instruction strategies. afterward, humans can pose various instructions in new dialogue scenarios by choosing different possible strategies via deductive reasoning @xcite . specifically, different strategies focus on different aspects, such as the "detail" or "correctness" of information, which directs the dialogue flows in various directions. motivated by this, we propose an inductive-deductive strategy reuse method, ideas for short, to generate diverse and in-depth instructions for building multi-turn instructional dialogues. ideas is composed of an induction stage and a deduction stage. in the induction stage, we prompt gpt-4 to extract the instruction strategy for each pair of the dialogue history and the corresponding instruction in real human-machine dialogues. those similar instruction strategies are further abstracted into a high-level strategy, which reveals the generic nature regardless of the details relevant to the specific dialogues. in the deduction stage, the user simulator is asked to sample an appropriate instruction strategy to guide instruction generation based on the current dialogue history in each new dialogue scenario. afterward, the user simulator and a system agent interact iteratively to build multi-turn instructional dialogues. experiment results indicate that ideas can generate diverse and in-depth instructions, and our constructed multi-turn instructional dialogues contribute to the performance improvement of the downstream chat model. our contributions are summarized as follows: (1) to the best of our knowledge, this is the first study to propose an inductive-deductive strategy reuse method to generate diverse and in-depth instructions in new dialogue scenarios. (2) experimental results show that ideas produces higher-quality instructions, which can be further used to improve the performance of the downstream chat models. (3) extensive experiments show that providing more instructional dialogues with high-quality instructions can further improve performance. ## preliminary this section describes task definitions and reviews the concepts of inductive and deductive reasoning. ## ideas we aim to reuse instruction strategies to generate diverse and in-depth instructions for building multiturn instructional dialogues. to this end, we introduce inductive reasoning for gaining high-level strategies from human-machine instructional dialogues d ins in section 3.1. in section 3.2, we describe how the user simulator chooses an appropriate strategy to generate instruction. in section 3.3, we show the implementation of the involved models. the overall architecture of building instructional dialogues is shown in figure 2 . ## conclusion this paper presents an inductive-deductive strategy reuse method, ideas, to generate diverse, indepth instructions for building multi-turn instructional dialogues. specifically, ideas first induces high-level instruction strategies from real dialogues via inductive reasoning. afterward, ideas deductively applies these strategies to new dialogue scenarios, where the strategies serve as underlying rules to guide the dialogue flow in different directions, i.e., posing various instructions. experimental results show that ideas can generate diverse and in-depth instructions. the constructed instructional dialogues can be used to improve the performance of the downstream chat model. we hope ideas will inspire more human-inspired methods to build higher-quality instructional dialogues.