Conversation Graph: Data Augmentation, Training, and Evaluation for Non-Deterministic Dialogue Management (2021)

Abstract: Task-oriented dialogue systems typically rely on large amounts of high-quality training data or require complex handcrafted rules. However, existing datasets are often limited in size con- sidering the complexity of the dialogues. Additionally, conventional training signal in- ference is not suitable for non-deterministic agent behavior, namely, considering multiple actions as valid in identical dialogue states. We propose the Conversation Graph (ConvGraph), a graph-based representation of dialogues that can be exploited for data augmentation, multi- reference training and evaluation of non- deterministic agents. ConvGraph generates novel dialogue paths to augment data volume and diversity. Intrinsic and extrinsic evaluation across three datasets shows that data augmentation and/or multi-reference training with ConvGraph can improve dialogue success rates by up to 6.4%.

## introduction dialogue systems research focuses on the natural language interaction between a user and an artificial conversational agent. current trends lean towards end-to-end models @xcite while modular systems tend to be the preferred approach in industrial applications @xcite . at the core of a modular conversational agent is dialogue management (dm), whose function is to exchange information with a user, update the agent's internal state, and plan its next action according to a policy @xcite . the dialogue manager then collects the user's response and the process repeats until the conversation ends. more specifically, task-oriented dialogue managers are aiming towards the completion of some specific task or tasks (e.g., booking services or buying products). machine-learned policies for dm require large amounts of high-quality data to generalize to a variety of conversational scenarios @xcite @xcite . however, given the complexity of some tasks, the datasets are often limited in size. reinforcement learning approaches @xcite @xcite @xcite can replace the need for explicit training data by exploiting a custom-designed environment to infer the training signal for the policy. unfortunately, such custom-designed environments may not be representative of how a user would interact with a conversational agent and their manual development is time-consuming and domain-specific. furthermore, modern conversational agents exhibit non-deterministic behavior, that is, they are able to take different but equally valid actions in identical dialogue states. conventional agent training and evaluation do not support the non-deterministic nature of conversational datasets as only a single fixed target is considered per inference, penalizing valid model predictions. to address these problems, we propose the conversation graph (convgraph), a dm framework for data augmentation, which also enables the training and evaluation of non-deterministic policies. if we consider each dialogue as a sequence of dialogue states, alternating between the agent and the user, we assemble a graph structure by unifying matching states across all conversations in the dataset. new dialogue paths can then be traversed in the graph resulting in novel training instances that can improve the dm policy. the unification simultaneously collects all valid actions at each point in the dialogue to facilitate the training and evaluation of non-deterministic policies. we consider the contribution of this paper to be threefold. first, we explore several augmentation baselines as well as variants of convgraph augmentation to show their impact on policy through intrinsic and extrinsic evaluation. we show that augmentation with convgraph leads to improvements of up to 6.4% when applied in an end-toend dialogue system. in addition, we propose a loss function that takes advantage of convgraph's unified dialogue states to increase the success rate by up to 2.6%. finally, we exploit convgraph to introduce a multi-reference evaluation metric for non-deterministic dialogue management. ## background the idea of using graphs in dialogue systems is not new @xcite @xcite , but it is limited to graphs representing flow charts where each node is an action step in a sequence, without a specific semantic importance assigned to the nodes themselves @xcite . on the other hand, the convgraph dialogue state information is encoded in a structured way for each node, allowing the unification of nodes across conversations. it is primarily this structured representation of dialogues that enables the use of the graph for data augmentation by traversal of new dialogue paths. it also allows for nondeterministic training and evaluation by referencing the graph to validate model predictions. ## the conversation graph this section describes how dialogues are unified into a graph representation useful for data augmentation, model training, and evaluation. ## results and analysis tables 2 and 3 show intrinsic evaluation results for dm policies with a history of 4, trained with bce and sbce loss, respectively. the h-f1 columns denote hardf1 scores, and the s-f1 columns denote softf1 scores. ## conclusions we have introduced the conversation graph for dialogue management, an approach that unifies conversations based on matching nodes (dialogue states). exploiting the structure of convgraph can be effectively used for (1) data augmentation with our most frequent sampling method, (2) training non-deterministic policies with sbce, our soft loss function, and (3) a more complete and fair evaluation of nondeterministic agents with our softf1 score. we conducted a thorough analysis of convgraph on three conversational datasets and showed that they can have markedly different properties. extrinsic evaluation with a user simulator as well as intrinsic evaluation supports that convgraph can successfully augment datasets by generating novel paths through the graph. the soft training loss sbce lets the agent choose which actions to learn in each dialogue state, leading to consistent policy improvements. finally, the soft evaluation has extended the conventional ''hard'' evaluation, which was insufficient for non-deterministic agents, leading to unfair penalties for correct predictions. we hope that our methodology as well as suggestions for future work will inspire further research in this topic area.