Improving Dialogue Act Recognition with Augmented Data (2022)

Abstract: We present our work on augmenting dialog act recognition capabilities utilizing synthetically generated data. Our work is motivated by the limitations of current dialog act datasets, and the need to adapt for new domains as well as ambiguity in utterances written by humans. We list our observations and findings towards how synthetically generated data can contribute meaningfully towards more robust dialogue act recognition models extending to new domains. Our major finding shows that synthetic data, which is linguistically varied, can be very useful towards this goal and increase the performance from (0.39, 0.16) to (0.85, 0.88) for AFFIRM and NEGATE dialog acts respectively.

## introduction virtual assistants have been deployed towards helping users perform various tasks, such as setting up a credit card. behind the scenes, most dialogue systems powering these virtual assistants are built of various components which facilitate natural language understanding (nlu). one such critical component is dialogue state tracking (dst), which helps systems recognize the current state and intent of the user in the conversation. dst often consists of three main sub-components -intent classification, slot filling and dialogue act recognition (dar). dialogue acts describe how the dialogue state should be modified from a system perspective, whereas the intents and slots help identify the user's intent in an utterance. these sub-components are usually built separately for industrial applications, since dar could be generalizable, while intents and slots vary with the intended task or service. since dst can be subjective, large-scale industrial applications need to rise to many challenges, including supporting heterogeneous services and apis. the schema-guided dialogue (sgd) state * work conducted during an internship at servicenow, inc. tracking task at the eighth dialogue system technology challenge (dstc8) @xcite introduced a dataset which could help handle these challenges, towards being able to handle multiple services and apis while not requiring the collection of new data or retraining models. the sgd dataset includes various dialogue acts as well as intents, one of the first to allow multiple apis with overlapping functionality in each domain. out of the 3 sub-components for dst, we observe that training models separately towards dialogue act recognition allows better internal utilization, since dialogue acts are similar across virtual assistant tasks and customers, whereas intent recognition and slot filling can vary across customers as well as customer specific tasks. keeping this in mind, we focus our research towards developing robust, generalizable dar models. since sgd is one of the most dialogue act-rich datasets, we explore its applications towards training dialogue act recognition models for confidential internal data. however, during our experiments, we observe that the performance drops significantly (from 0.98 to 0.39 f1 for 'affirm'). digging deeper, we observe that the form of responses for certain dialogue acts could be improved with adding variations. for example, majority of 'af-firm' utterances include or start with the word 'yes'. we conduct more experiments for 'affirm' and 'negate', and present our observations further details in section 5. we study the limitations of the dialogue act recognition models trained on sgd and tested on confidential internal data. we focus our study on understanding the performance for affirm and negate, and looking for the existence of similar patterns in existing data which could lead to overfitting. to bolster the generalizability of the model to new domains, we explore and implement data augmentation strategies which help add more variety to the form of the utterances in the dataset. we present all our findings in this paper, focusing mainly on our data augmentation techniques which utilize synthetic text generation methods. overall, our main contributions thus focus around the following studies: ## related work dialogue state tracking, and consequently dialogue act recognition, are integral components of taskoriented dialogue systems. recent research often focuses on utilizing neural methods towards approaching these tasks @xcite , and both surveys find that generalizability in dialogue state tracking is understudied. they both also present various strategies towards data augmentation, including but not limited to training on resource-rich domains and applying to unseen domains (similar to sgd), using weak supervision to identify slots, reformulating dialogue state tracking as dialogue summarization to leverage external annotated data, using reinforcement learning towards generating relevant data, and prompting generative models to address unseen domains. many of the aforementioned methods focus on the intent slot values, and not the dialogue acts. since our work focuses mainly on dialogue act recognition, we include relevant work towards this task in this section. ## datasets this section details the datasets we utilize in our experiments, and provides aggregate details for each without revealing private and protected information for confidential internal data. for the purposes of our research, we focus mainly on a few dialogue acts relevant towards confidential internal applications -namely affirm and negate. each section describes how we build the utterance and associated dialogue act pairs towards dialogue act recognition. the final statistics for each dataset is presented in ## synthetic data generation we aim to augment our dataset using synthetic data generated by prompting gpt-3, as described in section 2. we detail our experiments and their results in this section.