Integrated taxonomy of errors in chat-oriented dialogue systems (2021)

Abstract: This paper proposes a taxonomy of errors in chat-oriented dialogue systems. Previously, two taxonomies were proposed; one is theory-driven and the other data-driven. The former suffers from the fact that dialogue theories for human conversation are often not appropriate for categorizing errors made by chat-oriented dialogue systems. The latter has limitations in that it can only cope with errors of systems for which we have data. This paper integrates these two taxonomies to create a comprehensive taxonomy of errors in chat-oriented dialogue systems. We found that, with our integrated taxonomy, errors can be reliably annotated with a higher Fleiss’ kappa compared with the previously proposed taxonomies.

## introduction from their social aspects, chat-oriented dialogue systems have been attracting much attention in recent years @xcite @xcite . neural-based methods have been extensively studied and have yielded promising results @xcite @xcite @xcite ). yet, the performance of these systems is still unsatisfactory, causing dialogues to often break down. one way to reduce the errors made by the systems is to understand what kinds of errors the systems are making and find solutions to counter them. for such a purpose, a taxonomy of errors will be useful. for task-oriented dialogue systems, several taxonomies have been proposed @xcite @xcite , leading to effective analyses for improving system performance. for dialogue systems that * currently mainly affiliated with nagoya university. are chat-oriented, such taxonomies have also been proposed. @xcite proposed two taxonomies; one is theory-driven and the other data-driven. however, the former suffers from the fact that dialogue theories for human conversation on which the taxonomy is based, such as grice's maxims @xcite and adjacency pairs @xcite , are often not appropriate for categorizing errors made by chat-oriented dialogue systems. the latter has limitations in that it can only cope with errors for which we have data. because of such shortcomings, these taxonomies suffer from low interannotator agreements, failing to successfully conceptualize the errors @xcite . this paper aims to create a new taxonomy of errors in chat-oriented dialogue systems. on the basis of the two taxonomies previously proposed, we discuss their merits and demerits, and we integrate the two into a comprehensive one. we verify the appropriateness of the integrated taxonomy by its inter-annotator agreement. we found that the kappa values were reasonable at 0.567 and 0.488 when expert annotators and crowd workers were used for annotation, respectively, and these values were much better than those of the previous taxonomies. this indicates that the errors have successfully been conceptualized, and we can safely use them to analyze errors made by chat-oriented dialogue systems. ## previous taxonomies and integration higashinaka et al. proposed two taxonomies of errors in chat-oriented dialogue systems: theorydriven @xcite and data-driven @xcite ). 1 the theory-driven taxonomy is based on principles in dialogue theories that explain the cooperative behavior in human dialogues. the taxonomy uses the deviations from such principles as error types. in contrast, the data-driven taxonomy uses the dialogue data of chat-oriented systems in order to identify typical errors made by such systems. the taxonomy was created by first collecting comments (textual descriptions) describing errors made by systems and then clustering the comments; each resulting cluster corresponds to an error type. ## integrated taxonomy ## evaluation we evaluated the integrated taxonomy by annotating dialogues with error types and calculating the inter-annotator agreement. the same dialogues were annotated with the theory-and data-driven taxonomies by the same annotators for comparison. ## summary and future work this paper proposed a new taxonomy of errors in chat-oriented dialogue systems. we integrated previously proposed theory-and data-driven taxonomies to create an integrated taxonomy. we evaluated the integrated taxonomy with fleiss' kappa and found that our taxonomy was better than the previously proposed ones. although there still remains some confusion between some error types, the reasonable kappa values of our taxonomy verify its validity. as future work, we want to test the language independence because we only worked in japanese, although we consider our taxonomy to be generally language-independent. another possible use of the taxonomy will be to use it as a guideline for artificially generating errors so as to improve dialogue modeling in unlikelihood training @xcite . although the proposed taxonomy will be useful for reducing errors by systems, it will be also interesting to consider ways to recover from dialogue breakdowns after they have occurred @xcite . various studies have been done on understanding how people react during miscommunication, such as by making repairs @xcite and clarification requests @xcite @xcite . we aim to expand our work to deal with various phenomena centering around dialogue breakdown. finally, we have released the annotation manual 6 (japanese version and its english translation) so that it can be used for the analysis of various chatoriented dialogue systems in the community.