CESAR : Automatic Induction of Compositional Instructions for Multi-turn Dialogs (2023)

Abstract: Instruction-based multitasking has played a critical role in the success of large language models (LLMs) in multi-turn dialog applications. While publicly available LLMs have shown promising performance, when exposed to complex instructions with multiple constraints, they lag against state-of-the-art models like ChatGPT. In this work, we hypothesize that the availability of large-scale complex demonstrations is crucial in bridging this gap. Focusing on dialog applications, we propose a novel framework, CESAR, that unifies a large number of dialog tasks in the same format and allows programmatic induction of complex instructions without any manual effort. We apply CESAR on InstructDial, a benchmark for instruction-based dialog tasks. We further enhance InstructDial with new datasets and tasks and utilize CESAR to induce complex tasks with compositional instructions. This results in a new benchmark called InstructDial++, which includes 63 datasets with 86 basic tasks and 68 composite tasks. Through rigorous experiments, we demonstrate the scalability of CESAR in providing rich instructions. Models trained on InstructDial++ can follow compositional prompts, such as prompts that ask for multiple stylistic constraints.

## introduction instruction tuning is a popular multi-tasking method for fine-tuning large language models (llms). in this setup, llms are trained over a range of tasks specified by instructions, which lets them generalize over new task descriptions at ease @xcite . although instruction tuning enables individual task performance at scale, a language model's practical usage often requires high performance on compositions of these tasks. for example, in fig. 1 , the prompt requires the ## related work the term instruction-tuning was popularized by @xcite ; @xcite and has gained popularity among various researchers-for example, @xcite optimized outputs based on user preferences by incorporating human feedback. @xcite , on the other hand, demonstrated that scaling tasks and model size improved performance across benchmarks. finally, instructdial @xcite focused on instruction tuning for downstream dialog tasks but lacked curated compositional tasks. to address this gap, cesar enables large-scale training of compositional tasks. please refer to ## motivation in this section, we first explore the performance disparity between closed-access and open-access models foot_1 on complex compositional tasks. we then investigate the impact of including compositions in the training data on task performance. closed-vs. open-access models in complex dialog tasks. we begin by checking the disparity between open and closed-access models. in fig. 2 , we notice that chatgpt (a closed-access model) can produce satisfactory results for simple composite tasks whereas publicly available dial-t0 @xcite struggles. this demonstrates that open-access models require additional resources to improve on complex dialog tasks. ## cesar for any given dialog interaction, we first define the notions of dialog items and dialog components: ## instructdial++ instructdial is an instruction tuning benchmark for dialogue, which consists of a repository of diverse dialogue tasks in a unified text-to-text format created from openly available dialogue datasets @xcite . in this section, we first discuss new tasks and datasets added to this benchmark, resulting in the updated version, in-structdial++. we then discuss how we map each task in instructdial++ to the cesar framework. ## conclusion we propose cesar to fill the compositional capability gap between public models and irreproducible state-of-the-art, chatgpt. cesar modularizes downstream dialog tasks under one format amined the flan collection and saw that out of 23 datasets it incorporates we only use dailydialog within the test set. moreover we saw that there is minimal intersection between the version of dailydialog used by flan (niv2 corpus) and the original version we used. thus we conclude there is only minimal chance of contamination, where in the worst case 100 test instance are seen by the model.