Plan-Grounded Large Language Models for Dual Goal Conversational Settings (2024)

Abstract: Training Large Language Models (LLMs) to follow user instructions has shown to supply the LLM with ample capacity to converse fluently while being aligned with humans. Yet, it is not completely clear how an LLM can lead a plan-grounded conversation in mixed-initiative settings where instructions flow in both directions of the conversation, i.e. both the LLM and the user provide instructions to one another. In this paper, we tackle a dual goal mixed-initiative conversational setting where the LLM not only grounds the conversation on an arbitrary plan but also seeks to satisfy both a procedural plan and user instructions. The LLM is then responsible for guiding the user through the plan and, at the same time, adapting to new circumstances, answering questions, and activating safety guardrails when needed. We propose a novel LLM that grounds the dialogue on a procedural plan, can take the dialogue initiative, and enforces guardrails on the system’s behavior, while also improving the LLM’s responses to unexpected user behavior. Experiments in controlled settings and with real users show that the best-performing model, which we call PlanLLM, achieves a 2.1x improvement over a strong baseline. Moreover, experiments also show good generalization to unseen domains.

## introduction guiding users through manual tasks, such as cooking or diy @xcite , is a novel and difficult setting for current large language models (llms). the problem is challenging because recent llms are only trained to follow user instructions, while in this new setting, instructions flow in both directions of the conversation. solving it requires addressing two objectives: (i) following a plan of procedures, and (ii) answering arbitrary user instructions. to tackle these joint objectives, 1 https://huggingface.co/dmgcsilva/planllm figure 1 : an example of a dual goal conversational setting where the user is executing a manual task with the guidance of an llm assistant. llms need to be aligned with both a procedural plan and user instructions in the context of the plan, as illustrated in figure 1 . in this work, we investigate llms with the ability to steer dialogues through a plan of procedures in an end-to-end fashion, while simultaneously addressing the user's changing needs as they move from step to step. there is sufficient evidence that llms can follow a single instruction and generate procedural plans @xcite . moreover, recent work has also explored prompt engineering methods to turn llms into tutors @xcite . however, prompt-based solutions may answer the question without sufficient guardrails and then fail to steer the conversation back to the plan. other approaches explore neural dialogue tutoring systems @xcite , demonstrate instruction-grounded tutoring @xcite , and explore llms as math tutors @xcite , but the llm's apparent lack of control over the course of a conversation remains a problem. in this paper, we diverge from current work and investigate how llms can guide users through a plan of procedures, avoid conversation detours, and proactively try to bring the user back to the plan, figure 2 . in particular, the large language model will that work for you? a: "great!! let's continue. u: "that's a good idea." u: "how soft should the batter be?" a: "a cake batter should be fluffy. this means that the batter is aerated and will rise when baked." u: "can i double the oven temperature to reduce the cooking time?" a: "please, no! that will burn your food and cause a fire hazard." that we propose encloses four key contributions. first, the proposed model can ground its behavior in procedural plans, label 1 of figure 2 . given a procedural plan, the proposed model can navigate through it and keep track of the dialogue state, label 2 of figure 2 . the second contribution concerns following user questions that are grounded on the plan of procedures. as the conversation advances, user questions will emerge and the llm needs to answer them. this is a non-trivial problem as the answer may be present in the previous conversation turns, the plan's steps, or external general knowledge, label 4 of figure 2 . for the third contribution, the llm can answer open-ended requests that have a human preference implied, e.g., suggest a replacement to a missing resource or suggest a plan-related fun fact, label 3 of figure 2. the fourth, and final, contribution aligns the model with conversational norms to steer users away from unsafe or unethical requests while being polite @xcite , label 5 of figure 2 . through automatic and human evaluation, we show that the proposed model, planllm, is capable of addressing most situations, even when they require external information. moreover, the zeroshot capabilities of the model are demonstrated within the plan of procedures in an unseen domain, i.e., trained in the cooking domain and tested in the unseen domain of diy. ## related work training large language models to follow instructions @xcite has garnered significant attention, such as flan-t5 @xcite , in-structgpt @xcite and alpaca @xcite . @xcite showed that instruction tuning substantially improves zero-shot performance on unseen tasks. later, instructgpt trained llms with reinforcement learning with human feedback (rlhf) and was able to improve their alignment with human preferences and maintain performance on nlp benchmarks. alpaca presents a fine-tuning of the llama @xcite foundation model on instruction data created using self-instruct @xcite . in a conversational setting, the vicuna @xcite model is trained on user-shared conversational data. when following a procedural plan through a dialogue, there are many dependencies and unexpected events that may occur during its execution. tutoring systems try to cover all possible events and define a complex mesh of dependencies and actions @xcite . in control theory, the revision and generation of a new procedural plan are now being tackled with llms and neuralsymbolic methods @xcite . generating data for instruction datasets using general-purpose llms is an active research topic. task2dial @xcite is especially relevant, as it contains realistic dialogues centered around recipes, and makes use of commonsense knowledge throughout. more recently, wizard of tasks @xcite has attempted to mimic how real users interact with conversational task assistants @xcite , while also focusing on document-grounded question-answering. @xcite and @xcite utilize a limited set of initial examples to generate new instructions and prompt the llms to extrapolate novel ones. models trained on these instructions display promising results @xcite . we build on these ideas to fill the gap of generating conversational data over procedural plans. ## learning to reason about procedural plans in this section, we investigate methods for providing language models with the ability to reason about procedural plans and to assist users in completing manual tasks. the key functional properties that the language model needs to acquire are (i) navigation of a plan, (ii) answering plan-grounded questions, (iii) solving open-ended requests, and (iv) being polite and safe. ## generating plan-grounded dialogues to generate real-user-driven dialogues that simulate user-system interactions, in a dual goal setting, we adopt a generation pipeline that leverages realworld conversational data @xcite through data augmentation techniques. the resulting dialogues go beyond the scope of choi et al. ( 2022 ) by adding contextual requests and requests not related to the task, thus better replicating real user behavior @xcite . ## conclusions assisting users in the execution of complex manual tasks is a challenging problem that requires a system to be able to understand and follow complex instructions, provide accurate answers to user questions, and adapt to new user requests. in this paper, we proposed a methodology to train llms for such dual goal conversational settings, tailored to assist users in following plans of procedures, i.e., cooking, and diy. representing this novel setting, we introduce a large-scale dataset of user-system dialogues covering key dual goal dialogue patterns, grounded on real user-system dialogues. the evaluation of the trained models' capabilities shows their ability to assist users in a variety of tasks, including recipe navigation, ingredient substitutions, question answering, and more, all while remaining safe and respectful, and rejecting any dangerous requests. finally, our user study with planllm, a vicuna model trained with dpo, on a novel domain showed it is able to generalize to a new domain with similar dexterity as observed in its training domain.