C ant T alk A bout T his: Aligning Language Models to Stay on Topic in Dialogues (2024)

Abstract: Recent advancements in instruction-tuning datasets have predominantly focused on specific tasks like mathematical or logical reasoning. There has been a notable gap in data designed for aligning language models to maintain topic relevance in conversations - a critical aspect for deploying chatbots to production. We introduce the CantTalkAboutThis dataset to help language models remain focused on the subject at hand during task-oriented interactions. It consists of synthetic dialogues on a wide range of conversation topics from different domains. These dialogues are interspersed with distractor turns that intentionally divert the chatbot from the predefined topic. Fine-tuning language models on this dataset helps make them resilient to deviating from the assigned role and improves their ability to maintain topical coherence compared to general-purpose instruction-tuned LLMs like gpt-4-turbo and Mixtral-Instruct. Additionally, preliminary observations suggest that training models on this dataset also enhance their performance on fine-grained instruction following tasks, including safety alignment.

## introduction recent progress in natural language processing (nlp) has been significantly driven by instructiontuning of language models @xcite . although the scaling of models is an importantt factor in the emergent properties of language models @xcite , an increasing number of studies mphasize that the quality of training data is equally vital @xcite . this emphasis on data quality is pertinent not only during the pretraining phase but also throughout the instruction-tuning process. the primary goal of instruction-tuning in language models is to ensure their outputs align with human expectations across various tasks @xcite . alignment of these models can be achieved through various methods, including supervised fine-tuning and reinforcement learning @xcite . generally, alignment encompasses two somewhat contradictory objectives: ensuring the llm is helpful and accurately responds to user prompts while concurrently avoiding engagement in harmful topics @xcite . this balance between helpfulness and the avoidance of certain topics, particularly focusing on safety alignment concerning toxicity and bias, bears resemblance to content moderation practices @xcite . while aligning language models (llms) for helpfulness is crucial, it is also essential for these models to have the capability to disengage from certain topics, particularly for practical applications in real-world scenarios. to address this, we introduce the notion of topic-following, a combination of instruction-tuning and moderation. this concept revolves around defining complex instructions that dictate how an intelligent assistant (chatbot) should interact with users. in topic-following, we include not only examples of user interactions that the assistant should engage in but also scenarios where the user deviates from the given instructions, denoted as distractors. in these cases, the chatbot should either decline to respond or tactfully guide the conversation back to the relevant topic. topic-following instructions distinctly define when an assistant should respond and when it should steer away during extended dialogues with users. unlike safety alignment, where models are trained to avoid toxic or biased responses through predefined behaviours, implementing topic-following in llms introduces a form of moderation that is user-defined. this concept shares similarities with the programmable guardrails proposed by nemo guardrails @xcite . however, unlike their approach, which utilizes a domain-specific language, the rules for topic-following can be articulated in natural language, offering a more accessible and intuitive means for guiding the responses of the chatbot. as a first effort to empower chatbots in topicfollowing we present canttalkaboutthis foot_0 , a small dataset consisting of 1080 synthetic dialogues designed to train models to stay on topic. our dataset creation process involves a three-step prompting-based approach that focuses on both diversity and quality: 1) we generate topic-following prompts across a variety of scenarios; 2) we create dialogues adhering to these topical instructions, employing a technique akin to dialogue inpainting @xcite ; 3) we integrate distractors into these dialogues to test topic following. our findings using this dataset reveal that even stateof-the-art general-purpose llms initially struggle with staying on topic, often engaging with distractors. we demonstrate that models fine-tuned on our dataset significantly improve performance in following complex dialogue instructions. however, investigation into the nature of our synthetic distractors reveals a limitation: they tend to be off-topic but simplistic. to address this, we develop an additional dataset with human-crafted distractors, offering a more rigorous benchmark for assessing topicfollowing in language models, akin to the role of mt-bench in evaluating instruction-tuning @xcite . we firmly believe that topic-following is a crucial yet overlooked aspect in current instructiontuning datasets @xcite . integrating canttalkaboutthis into instruction-tuning blends significantly enhances an llm's capacity to comprehend and follow complex system instructions about the interaction topic. this is particularly advantageous for developing task-oriented dialogue systems (tods) @xcite similar to the custom gpts from openai @xcite , which primarily use natural language to define dialogue systems. our approach to teaching llms topic-following has two key side benefits. firstly, it aids in improving overall instruction-following abilities. secondly, models trained for topic-following exhibit comparable efficacy to llamaguard @xcite in safety alignment, even in a zero-shot setting. in conclusion, our work presents three significant contributions. firstly, we highlight the necessity of incorporating topic-following as a fundamental task in the instruction-tuning phase for llms. secondly, we introduce a method for developing topic-following datasets and release multiple versions of the canttalkaboutthis dataset, including one suitable for commercial use. finally, we show that training on topic-following data not only enhances an llm's capability to navigate away from distractors but also improves its overall ability to follow instructions and allows it to effectively perform zero-shot safety alignment. ## background topic-following behaviour for llms is a specific case of instruction tuning. ideally, given a complex instruction describing the topic of the task to be solved by the assistant, the llm should be able to respect these topical instructions when generating its responses. to achieve this behaviour, we build a new synthetic dataset with conversations adhering to the instruction using a form of dialogue inpainting. in this section we will investigate these two aspects and also the few recent works that are similar to topic-following. ## method in this section, we detail the steps involved in creating the canttalkaboutthis dataset. its main aim is to improve the ability of llms and chatbots powered by llms to maintain topical focus during interactions. the data pipeline involves sev-eral stages: identifying specific scenarios, formulating system instructions describing the topic of the conversation, generating dialogues that respect the topical system instructions, and producing distractors that would make the chatbot go off-topic. the prompts used for creating the dataset can be found in appendix §b. ## experimental settings 4.1 dataset creation to develop the canttalkaboutthis dataset, we conduct experiments using two llm back-bones: openai gpt-4-turbo-1106 and the mixtral-instruct 8x7b model @xcite . using two different llms has several advantages. first, we can compare whether the models, including baselines, are biased towards distractors generated by a specific llm. second, we show that we can obtain high-quality topic-following data with smaller open models. third, the dataset generated with the mixtral model should be commercially usable. ## conclusion we introduce topic-following, a task that helps language models understand complex topical system instructions while allowing them to steer away from off-topic user requests. this behavior is most useful for building task-oriented bots that only respond to messages that respect the topical instructions specified in natural language and which are part of the system prompt. to demonstrate the relevance of topic-following, we propose canttalkaboutthis dataset -a small synthetic dataset consisting of on-topic conversations and distractors. even if this data has limitations, we show that it can be used to finetune models with remarkable performance compared to strong llms on a small human-annotated distractor dataset. the dataset has benefits to several out-of-domain tasks and we demonstrate that topic-following can also be used for safety alignment, content moderation, and improving the general helpfulness of instruction-tuned models. the latter is also supported by the fact the our dataset has been recently used as part of the instructiontuning alignment blend by powerful models from the nemotron family @xcite .