A Co-Attentive Cross-Lingual Neural Model for Dialogue Breakdown Detection (2020)

Abstract: Ensuring smooth communication is essential in a chat-oriented dialogue system, so that a user can obtain meaningful responses through interactions with the system. Most prior work on dialogue research does not focus on preventing dialogue breakdown. One of the major challenges is that a dialogue system may generate an undesired utterance leading to a dialogue breakdown, which degrades the overall interaction quality. Hence, it is crucial for a machine to detect dialogue breakdowns in an ongoing conversation. In this paper, we propose a novel dialogue breakdown detection model that jointly incorporates a pretrained cross-lingual language model and a co-attention network. Our proposed model leverages effective word embeddings trained on one hundred different languages to generate contextualized representations. Co-attention aims to capture the interaction between the latest utterance and the conversation history, and thereby determines whether the latest utterance causes a dialogue breakdown. Experimental results show that our proposed model outperforms all previous approaches on all evaluation metrics in both the Japanese and English tracks in Dialogue Breakdown Detection Challenge 4 (DBDC4 at IWSDS2019).

## introduction in today's world, we often use interactive virtual conversational agents in our daily lives, such as siri, google assistant, alexa, etc. such chat-oriented dialogue systems assist users in their daily lives through interaction in the form of conversations. through information obtained via a conversation, a system can understand a user's need and generate the corresponding response, such as reporting weather forecasts, booking tickets, or entertaining the user by chit-chatting. to facilitate hassle-free communication, the system needs to generate proper and rational responses without any ambiguity. furthermore, while generating a response, the system should consider not only the last turn in the conversation but also all previous utterances so that the response generated is coherent with the entire conversation history. the user experience degrades when a dialogue breakdown occurs, where the system generates an inappropriate response which breaks the conversation flow. dialogue breakdown detection is one of the major user utterances system utterances breakdown? u 1 : what do you like to do? s 1 : i like to eat out and eat. no u 2 : i like to eat too. hamburgers and french fries are my favorite. s 4 : i have a lot of friends. yes dialogue breakdown detection challenge 4 (dbdc4) foot_0 is a shared task dedicated to dialogue breakdown detection. in this paper, we use the dataset released in dbdc4 for our experiments. since whether a system utterance causes a dialogue breakdown is somewhat subjective, the task is modeled as a classification task with three classes: breakdown (b), possible breakdown (pb), and not a breakdown (nb). every single instance in the dataset is annotated by multiple annotators. each instance is assigned a gold-standard (majority) class, and a probability distribution based on all annotators' predictions. the task has two tracks involving two different languages: japanese and english. the evaluation metrics consist of both classification-related metrics and distribution-related metrics. we will introduce them in detail in section 4.3. prior work has exploited feature-engineered machine learning approaches like random forests, neural network architectures such as lstm @xcite @xcite , and the monolingual pretrained language model bert @xcite . most prior work treats the dialogue history and the last system response in the same manner. in feature-based approaches @xcite , the features are extracted from the concatenation of both the dialogue history and the last system response, while in other models using pretrained word embeddings @xcite @xcite , the interaction between the dialogue history and the last system response is also not explicitly captured. recently, cross-lingual language models such as xlm @xcite and xlm-r @xcite demonstrate strong performance on many cross-lingual natural language processing (nlp) tasks @xcite . they also outperform pretrained monolingual language models on tasks with low-resource languages. by utilizing shared word embeddings over different languages and multilingual parallel texts, cross-lingual language models encode input texts into one single representation space shared by all languages. this removes the cost of language-specific training. in this paper, we utilize pretrained cross-lingual language models to benefit from the multilingual training data. to the best of our knowledge, ours is the first work to incorporate pretrained cross-lingual language models in dialogue breakdown detection. typically, pretrained language models are not trained in a task-specific setting. when we utilize them for a particular task, it might not perform well if the training data is not too large. to better capture the interaction between the previous dialogue history and the last utterance, we propose to integrate a coattention network with the cross-lingual language model. experimental results show that our co-attention network significantly improves the performance of our model on the dbdc4 dataset. the source code and trained models of this paper are available at https://github.com/nusnlp/cxm . ## task overview a dialogue history h consists of a sequence of alternating user and system utterances. the target utterance t for dialogue breakdown detection is the succeeding system utterance. each instance (h, t ) is assigned one of three candidate classes: breakdown (b), possible breakdown (pb), and not a breakdown (nb). the output of a model includes two components: a predicted class from one of the three candidates {b, pb, nb}, and a probability distribution over the three classes. dbdc4 includes two tracks in two different languages (japanese and english) with the same task setting. ## model description we propose a co-attentive cross-lingual neural model (cxm), which is based on a pretrained crosslingual language model and a co-attention network to tackle the task of dbdc4. we give a detailed description of cxm in this section. we present the overall architecture of the proposed model in figure 1 . ## results in this section, we present the comparison results and an ablation study to better understand our model. ## conclusion in this paper, we have proposed a novel model based on a cross-lingual language model and a co-attention network for dialogue breakdown detection. our model achieves new state-of-the-art scores in dialogue breakdown detection challenge 4. our proposed model is the first to achieve the best scores on all the evaluation metrics, significantly outperforming all previous models. we have also observed that our model outperforms previous monolingual models on this task. we exploit transfer learning using a crosslingual language model to utilize training data from other languages and further improve the performance of our model on this task. the co-attention network built on top of the cross-lingual language model better captures the relation between the current utterance and the dialogue history. this helps to reduce the probability of a system generating an undesired response, so that communication and user experience are further improved.