Improved Instruction Ordering in Recipe-Grounded Conversation (2023)

Abstract: In this paper, we study the task of instructional dialogue and focus on the cooking domain. Analyzing the generated output of the GPT-J model, we reveal that the primary challenge for a recipe-grounded dialog system is how to provide the instructions in the correct order. We hypothesize that this is due to the model’s lack of understanding of user intent and inability to track the instruction state (i.e., which step was last instructed). Therefore, we propose to explore two auxiliary subtasks, namely User Intent Detection and Instruction State Tracking, to support Response Generation with improved instruction grounding. Experimenting with our newly collected dataset, ChattyChef, shows that incorporating user intent and instruction state information helps the response generation model mitigate the incorrect order issue. Furthermore, to investigate whether ChatGPT has completely solved this task, we analyze its outputs and find that it also makes mistakes (10.7% of the responses), about half of which are out-of-order instructions. We will release ChattyChef to facilitate further research in this area at: https://github.com/octaviaguo/ChattyChef .

## introduction historically, work on conversational agents has mostly fallen into one of two categories: opendomain chatbots @xcite @xcite or goaldirected dialogue systems within narrow domains @xcite . however, recent advances in large language models have paved the way for the exploration of dialog agents that can engage in conversations with users to accomplish open-ended objectives, such as learning about a new topic @xcite @xcite , interpreting bureaucratic policies to answer questions @xcite , or negotiating within strategy games @xcite . correct response 49.6% incorrect response wrong order 22.9% irrelevant response 10.7% lack of information 8.4% wrong information 8.4% in this paper, we explore the task of recipe-grounded conversation, where the dialogue agent is expected to converse with a user to walk him/her through the cooking procedure of a recipe, while answering any questions that might arise along the way (see examples in figure 1 ). although many types of dialogue tasks have been proposed and explored, very little prior work has focused on providing instructions to a user to complete a task. in contrast to other dialogue tasks, such as documentgrounded conversation @xcite , accurately tracking the conversation state is more crucial in recipe-grounded dialogue. this is because the agent needs to know which step in the recipe the user is currently working on in order to answer questions, such as: what is the next step? to investigate what challenges may arise in recipe-grounded conversation, we have collected a dataset by crowdsourcing (see §2). as an initial baseline model, we used this data to fine-tune gpt-j following a similar protocol to @xcite . specifically, the conversation history, the grounded recipe, and the gold system response are concatenated as a long input sequence to the model (see §5 for details). we show that fine-tuning gpt-j on a few recipe-grounded conversations works surprisingly well, however, the model makes a significant number of mistakes based on our manual inspection what do i do next? ## dataset construction to collect a corpus of recipe-grounded conversations for fine-tuning and evaluating models, we first obtain wikihow foot_2 articles under the recipes category from the data compiled by @xcite . we control the qualities of the recipes by only selecting articles that have a helpful vote rating greater than 75% and have at least 5 votes. we retain the images from the recipes in our dataset, but experiments in this paper only make use of recipe texts. moreover, in order to improve the conversation quality and avoid crowd workers quitting in the middle of a long conversation, we remove recipes with more than 8 steps. ## user intent detection in this section, we discuss the user intent detection subtask. formally, the task is to predict a set of user intent indices i (as one utterance may contain multiple intents), given the t th user utterance u usr t and the conversation history @xmath0, u usr 1 , ..., u sys t-1 }. we hypothesize that providing information about the user's intents may help the response generation model better provide information to the user in the correct order. for example, if the user asks for ingredient substitutions, the system should not respond by providing information based on the current step of the recipe. ## instruction state tracking we study the second subtask to support the instruction ordering of the generation module -instruction state tracking. the goal of this task is to predict the current state of the instruction, or in other words, the last instructed recipe step. formally, given the t th system response u sys t , the previous instruction state t t-1 (i.e., an index of a recipe step), and the recipe with a list of n r steps @xmath1, r 2 , . . . , r nr }, the expected output of this subtask is t t . ## response generation given the conversation history and the grounded recipe, the response generation task aims to generate the instruction response to the user. formally, given the history @xmath2, u usr 1 , ..., u sys t-1 , u usr t-1 }, the recipe r, the dialog system is expected to generate the next utterance u sys t . ## related work the task of recipe-grounded conversation is close to the conversational question answering (cqa) task. in cqa, given a reference text, the system needs to engage in a multi-turn conversation to answer questions from users. compared to singleturn question answering, cqa raises new challenges (e.g., co-reference resolution, contextual reasoning) due to the dependency between question answering turns. there exist multiple datasets in this area, such as coqa @xcite , quac @xcite , and sharc @xcite . there are several differeces between instructional dialogue and conversational question answering. firstly, in the dialogue setting, the message from the system can also be a question, such as verification. secondly, while the goal of cqa is seeking information, instructional dialogue focuses on supporting users to complete a procedure; therefore, there is additional order-related relationship between the system's responses and the instructions that needs to be managed by the dialog agent. recent work has investigated issues that arise in chatbots based on large language models. for instance, they are known to sometimes generate toxic language @xcite , make factual errors in their statements @xcite @xcite , and be overly confident @xcite . in this work, we focus on addressing a specific problem related to instruction-grounded dialogue, which is presenting information in the wrong order to a user. a small amount of prior work @xcite has started to explore the problem of recipe-grounded conversation, which makes these papers the two closest to ours. both of these papers focused primarily on dataset creation. @xcite included experiments on response generation, but as their focus was on building a new dataset, they did not conduct extensive experiments or perform a human evaluation of their system's outputs. they did propose baselines and evaluate the tasks of user question understanding and agent action frame prediction, which are similar to our user intent detection and instruction state tracking. although these tasks have similar goals, our work is different in the sense that we focus on tackling the problems in the lowresource setting, by transferring knowledge from existing dialogue corpora such as multiwoz. finally, besides providing additional recipe-grounded conversations as in these two prior works, our main focuses are on analyzing challenges of current large langauge models (i.e., gpt-j and chatgpt) on this task and addressing the specific challenge of instruction ordering. ## conclusion in this paper, we have proposed to explore two additional subtasks, namely user intent detection and instructional state tracking, to mitigate the problem of incorrect instruction order in instructional dialogue. we analyze these two auxiliary subtasks with different methods in low-resource settings. even though the performance of the modules for the two subtasks is still low, experiment results show that incorporating the user intent or state information does help to mitigate the wrongorder instructions issue, with the intent information having a greater impact. however, combining the two pieces of information does not lead to improvement over using each individual one of them alone. therefore, we believe that further research for the two subtasks is still needed, and also more effective ways of incorporating the information into the response generation module need to be investigated. finally, we release chattychef, a new cooking instructional dataset, to promote future research in this direction.