Improving Bot Response Contradiction Detection via Utterance Rewriting (2022)

Abstract: Though chatbots based on large neural models can often produce fluent responses in open domain conversations, one salient error type is contradiction or inconsistency with the preceding conversation turns. Previous work has treated contradiction detection in bot responses as a task similar to natural language inference, e.g., detect the contradiction between a pair of bot utterances. However, utterances in conversations may contain co-references or ellipsis, and using these utterances as is may not always be sufficient for identifying contradictions. This work aims to improve the contradiction detection via rewriting all bot utterances to restore co-references and ellipsis. We curated a new dataset for utterance rewriting and built a rewriting model on it. We empirically demonstrate that this model can produce satisfactory rewrites to make bot utterances more complete. Furthermore, using rewritten utterances improves contradiction detection performance significantly, e.g., the AUPR and joint accuracy scores (detecting contradiction along with evidence) increase by 6.5% and 4.5% (absolute increase), respectively.

## introduction latest chatbots powered by large pre-trained neural models have shown decent capabilities to maintain fluent and interesting conversations with human users @xcite @xcite . however, they are still prone to various kinds of annoying mistakes @xcite @xcite . one such error is contradiction or inconsistency, as illustrated in in order to reduce contradiction errors, one approach is to develop a detection model to identify such problems after a system produces response candidates. to this end, @xcite characterized the modeling of personarelated consistency as a natural language inference (nli) problem and constructed a dialog nli dataset based on persona-chat. to cover a broader range of consistency types (e.g., persona, logic, causality, etc), @xcite collected de-code, a dataset containing human written dialogues with self-contradictory utterances. besides the in-distribution human-human dialogues test set, they collected an out-of-distribution set containing dialogues between human and different chatbots. this human-bot test set can better evaluate models' performance in detecting contradiction in conversations between human and chatbots, which is the focus of this work. we find one failure of the state-of-the-art (sota) contradiction detection model is due to the frequent anaphora and ellipses in chatbot utterances. one typical example is shown in ## conclusion in this work, we aim to improve contradiction detection in chatbot utterances via rewriting to restore anaphora and ellipsis. to develop such an utterance rewriting model, we curated a dataset by crowd-sourcing and demonstrated that the rewriting quality is satisfactory. with such a rewriting technique, we are able to significantly improve the contradiction detection performance.