BETOLD : A Task-Oriented Dialog Dataset for Breakdown Detection (2022)

Abstract: Task-Oriented Dialog (TOD) systems often suffer from dialog breakdowns - situations in which users cannot or do not want to proceed with the conversation. Ideally TOD systems should be able to detect dialog breakdowns to prevent users from quitting a conversation and to encourage them to interact with the system again. In this paper, we present BETOLD, a privacy-preserving dataset for breakdown detection. The dataset consists of user and system turns represented by intents and entity annotations, derived from NLU and NLG dialog manager components. We also propose an attention-based model that detects potential breakdowns using these annotations, instead of the utterances’ text. This approach achieves a comparable performance to the corresponding utterance-only model, while ensuring data privacy.

## introduction task-oriented dialog (tod) systems @xcite enable users to complete specific tasks, such as booking a reservation at a restaurant. unlike open-domain dialog systems @xcite , where the aim is to maximize user engagement, in tod systems it is crucial to optimally assist a user to fulfill the task at hand. detecting dialog breakdowns due to miscommunications potentially paves the way to intervene and rescue the dialog, improve customer satisfaction and motivate the user to continue interacting with the system @xcite . a dialog breakdown is often defined as a point in the dialog where the user gives up the conversation without completing the task, often due to not understanding the intended meaning of user's utterance @xcite . if conversational system engineers can understand when and why a conversation is likely to break down, they can build systems that pre-vent broken dialogs, or design conversational breakdown recovery strategies @xcite . the dialog breakdown detection challenge (dbdc) has motivated the academic community's interest in the breakdown detection problem, which is the goal of predicting the occurrence of a breakdown at some point in the conversation @xcite . this challenge also came with the release of english and japanese datasets for addressing this task. despite the great value of these proposed datasets, they only provide the sequence of user and system utterances. utterances are indeed useful to detect a breakdown, however, in certain contexts, especially in industry, a stakeholder may decide not to share and release the texts for privacy preserving purposes @xcite . the utterances produced by a user during a task-oriented conversation often contain privacysensitive information. let us consider a dialog system in a company that handles issues relating to human resources as an example. the system may receive data regarding an employee's health status or compensation, i.e., data that a company is unwilling to share. in this paper, we demonstrate that even without access to the text of the conversation, it is still possible to identify a breakdown. in fact, traditional dialog systems often provide synthetic annotation of the user and system utterances as the intents and entities, originating from the natural language understanding (nlu) and natural language generation (nlg) components @xcite . in particular, the nlu component classifies the user utterances into intents (book_appointment) and extracts entities (@xmath0. the nlg component consists of a closed set of possible system utterances (each defined by a unique intent), often parameterized by or supplemented with entities (e.g. are two entities). in our study, we propose a simple yet effective way to automatically create a new breakdown detection dataset, given an existing tod system. in particular, we consider a phone-call scenario, in which a customer talks to a digital assistant to schedule a service appointment for their mobile phone. in our experience, two central events indicating caller's frustration with the current dialog state and projected outcomes are: a) hang-ups, b) requests to talk to a human agent. therefore, we consider caller hang-ups and transfer to human requests as dialog breakdowns. we also release a novel task-oriented dialog dataset betold (breakdown expectation for task-oriented long dialogues) with the proposed annotation schema. the dataset contains real human-agent conversations, between customers and our modular dialog system. the system automatically annotates the user utterances with nlu intents and entities, and generates appropriate nlg responses which contain nlg intents and accompanying entities. finally, we propose an attention-based model, capable of taking these features into account. our results show that, instead of relying only on the word tokens of the utterances, the use of nlu and nlg intents is sufficient to confidently predict a breakdown in a task-oriented conversation, therefore reaching satisfactory results while guaranteeing the privacy of the data. ## an attention-based model for luhf detection the task in betold dataset is to classify if a conversation between a human and the system is a luhf or not. in this setting, it is fundamental to keep track of what happened in the past, represented by the dialog history. we therefore investigate an attention-based architecture @xcite , that has proved to perform well in several dialog-related tasks @xcite @xcite , including dialog breakdown detection @xcite . ## limitations as mentioned in section 3.1, the labels in betold are automatically assigned to each conversation. therefore, it is possible that a conversation where everything goes smoothly but suddenly the user decides to hang up is classified as luhf. similarly, the user may decide to reach the end of the conversation even if they are extremely unsatisfied with the call. this case is not considered a dialog breakdown. in addition to this issue, a luhf annotation applies to the overall call and is not an indication of what happened during at each step of the conversation. a model that generalizes well should be able to predict whether a luhf happened at each step of the call. the process of data augmentation described in section 5.3 is an attempt to address this issue. this is, however, limited to not luhf calls, given that we have no guarantees on when a breakdown happened in a conversation. instead, we are quite confident that, if a call was successful, it was also successful in the previous steps. figure 3 shows a density histogram of the probability by class of predicting a luhf, averaged over all the conversation steps, for the test set conversations. as we can observe in the plot, the average probability of predicting a luhf in a not luhf call is low in general. we recall that we have more data available for the successful calls (the not luhfs are two times more than the luhfs and, in addition, we add 30% more successful calls through augmentation). therefore, it is not surprising to see that the not luhf distribution is more skewed towards to 0 than the luhf distribution. moreover, the results shown in the plot correspond to the average score across many steps of the conversation. a breakdown may happen very late in the conversation, thus resulting in an overall low average score. however, this is hard to determine through an automatic investigation and would require a manual inspection. ## conclusions in this paper, we proposed a simple way to automatically generate a breakdown detection dataset in task-oriented dialogs, where the breakdown labels are extracted by user-initiated events. this dataset guarantees the privacy of the data by only keeping the annotations from nlu and nlg components. we proposed an attention-based model which uses these types of annotations. as a result, we demonstrated that a model does not necessarily require textual utterances to predict a breakdown; yet, it can benefit from the nlg and nlu intents and entities, automatically provided by a classical dialog system. we also discussed some possible limitations of the model and the dataset, connected to the annotation schema. an automatic annotation, as it often happens, results in noisy data. however, we do believe that the proposed dataset is a promising starting point, which can save resources and could be further improved through manual annotations. as highlighted by the qualitative analysis, it is worth further investigating the results to understand which elements play a role in the detection of a luhf. the implementation of explainability methods can be an important tool in this context @xcite . given the transformer-based architecture of our proposed model, current explainability tools @xcite can enrich our investigation of the role of each attention head in the breakdown prediction. for that, gradient-based methods can give an overview of the importance of individual features as well as of the interactions.