Leveraging Explicit Procedural Instructions for Data-Efficient Action Prediction (2023)

Abstract: Task-oriented dialogues often require agents to enact complex, multi-step procedures in order to meet user requests. While large language models have found success automating these dialogues in constrained environments, their widespread deployment is limited by the substantial quantities of task-specific data required for training. The following paper presents a data-efficient solution to constructing dialogue systems, leveraging explicit instructions derived from agent guidelines, such as company policies or customer service manuals. Our proposed Knowledge-Augmented Dialogue System (KADS) combines a large language model with a knowledge retrieval module that pulls documents outlining relevant procedures from a predefined set of policies, given a user-agent interaction. To train this system, we introduce a semi-supervised pre-training scheme that employs dialogue-document matching and action-oriented masked language modeling with partial parameter freezing. We evaluate the effectiveness of our approach on prominent task-oriented dialogue datasets, Action-Based Conversations Dataset and Schema-Guided Dialogue, for two dialogue tasks: action state tracking and workflow discovery. Our results demonstrate that procedural knowledge augmentation improves accuracy predicting in- and out-of-distribution actions while preserving high performance in settings with low or sparse data.

## introduction for many real-world applications, it is crucial for task-oriented dialogue (tod) systems to complete user requests while strictly adhering to established procedures. for example, consider a customer service agent who must first verify a client's details before changing their password. although large language models have demonstrated potential in modeling such dialogues, they require large figure 1 : the knowledge-augmented dialogue system (kads) is composed of two modules: a knowledge retriever and a language model. the knowledge retriever takes the inner product as a measure of similarity between an embedded dialogue and each document in a provided knowledge base containing procedural instructions. the most similar document is then passed to a language model which attends over both the dialogue and retrieved document to generate the agent's next action. in prior work, retrieval-enhanced language models have achieved success integrating external knowledge from internet searches into conversational agents @xcite . however, a more controllable approach is necessary for instruction retrieval in task-oriented dialogue. rather than querying the open web, it's more suitable to perform retrieval over a closed set of documents, like in @xcite . however, while the training schemes utilized in these works sufficiently prime a model for question-answering tasks, they are not as effective for action prediction. following the lines of @xcite , which introduces a unique pre-training objective for slot-labeling, our method leverages custom objectives suited for action prediction tasks. we employ a specialized warm-up task where dialogues are matched with corresponding procedural instructions to ensure that the knowledge retrieval module is initialized with reasonable dialogue and document embeddings. then, the system is trained on an special case of masked language modeling in which masked actions are predicted from customeragent dialogues. finally, we found it necessary to encourage our system to incorporate signal from retrieved procedures by routinely freezing the language model's weights during training. we evaluated this approach on two dialogue tasks-action state tracking and workflow discovery-using two task-oriented dialogue datasets: action-based conversations dataset and schema-guided dialogue. our results suggest that kads yields improved action prediction accuracy against several baselines, including an unaugmented language model and a language model augmented with static guidelines, on both in-and out-of-distribution procedures. furthermore, we demonstrate that knowledge augmentation bolsters our system's ability to predict actions that occur infrequently in the training data. action state tracking (ast) aims to predict the next action performed by an agent during an interaction with a customer @xcite . formally, we represent an interaction as a sequence of turns x belonging to one of three categories: agent utterances x a ([agent]), agent actions x b ([action]), or customer utterances x c ([customer]). the model receives an interaction between a customer and agent up to turn t where prefix tokens p indicate the turn category: @xmath0... p t x t with @xmath1, [action], @xcite . see appendix b for an example. the model then predicts the following agent action x b t+1 which consists of a button, or b-slot, and any corresponding slot values if they are present: x b @xmath2: v 00 t , v 01 t . the goal of workflow discovery (wd) is to recover the workflow-the set of ordered actions taken by an agent-given a complete dialogue between a customer and agent @xcite . formally, we represent a dialogue as a sequence of turns belonging to one of two categories: agent utterances or customer utterances. the model receives a dialogue of length t between a customer and agent where prefix tokens indicate the turn category: @xmath3... p t x t with @xmath4 we evaluate kads on two tod datasets: action-based conversations dataset and schema-guided dialogue. both consist of multi-domain customer service interactions that loosely follow a set of predefined company policies which specify the actions to be taken by an agent to satisfy a particular customer inquiry. the core differences between these two datasets are their action and document structures. in action-based conversations dataset (abcd) @xcite , actions are composed such that the b-slot belongs to a predefined set of b-slots which describe the action being taken (e.g., "pull up account") and slot values consist of any corresponding information provided by the user (e.g., "johndoe@gmail.com"). in a given interaction, an average of 4 actions are taken. the documents provided within abcd are composed of a plain text description of a possible customer inquiry followed by an ordered set of action b-slots that should be performed by the agent. in schema-guided dialogue (sgd) @xcite , we take action b-slots to be the description of how the agent will interact with a piece of information (e.g., "inform", "confirm", or "request") and values as the type of information in question (e.g., "departure times"). in this dataset, the average number of actions per interaction is significantly longer at 21 actions, and the documents corresponding to sgd consist of a customer inquiry followed by all of the information types, or values, that can be acquired to fulfill the given inquiry. we use the train/dev/test splits presented in the original datasets (8034/1004/1004 and 16142/2482/4201 interactions per split for abcd and sgd respectively), and hold out a randomlyselected subset of 10% of actions during training for out-of-distribution testing. see appendix b for more details, including dialogue and corresponding document examples. ## results the evaluation of our tod system begins with bslot and value prediction accuracy for both known and novel actions. we also examine the data efficiency of our approach by reporting these metrics for progressively reduced training pools. we compare our model's performance against a base t5 model and t5 with static guidelines-a comprehensive list of agent actions-appended to the input sequence (t5 + guide) foot_0 . then, we assess the efficacy of our knowledge retriever in selecting relevant documents. finally, an ablation study of our pre-training routine highlights the importance of our custom training procedure. see appendix a for details of our experimental setup. ## conclusion while large language models make for effective tod systems in constrained settings, real-world applications often present insufficient data to train these models. kads offers a method of learning workflows with minimal or sparse supporting data and presents a more controllable and performant solution to low-resource tod automation. while our results offer a promising outlook for action prediction given dynamic guidance from structured procedural documents, future work should investigate the use of unstructured company guidelines and multi-document retrieval.