M u T ual: A Dataset for Multi-Turn Dialogue Reasoning (2020)

Abstract: Non-task oriented dialogue systems have achieved great success in recent years due to largely accessible conversation data and the development of deep learning techniques. Given a context, current systems are able to yield a relevant and fluent response, but sometimes make logical mistakes because of weak reasoning capabilities. To facilitate the conversation reasoning research, we introduce MuTual, a novel dataset for Multi-Turn dialogue Reasoning, consisting of 8,860 manually annotated dialogues based on Chinese student English listening comprehension exams. Compared to previous benchmarks for non-task oriented dialogue systems, MuTual is much more challenging since it requires a model that be able to handle various reasoning problems. Empirical results show that state-of-the-art methods only reach 71%, which is far behind human performance of 94%, indicating that there is ample room for improving reasoning ability.

## introduction building an intelligent conversational agent is one of the longest running goals in ai. existing conversational agents can be categorized into taskoriented dialogue systems @xcite and non-task-oriented chatbot systems @xcite . owing to the rise of deep learning techniques and the large amount of conversation data for training @xcite @xcite , we are now witnessing promising results of chatbots both in academia and industry @xcite . neural dialogue systems are trained over a large dialogue corpus and used to predict responses given a context. there are two lines of methods. retrievebased methods and generation based methods rely figure 1 : b is incorrect because there is no reason to apologize. c and d can be excluded because the relationship between two speakers are waiter and customer based on the context. on matching scores and perplexity scores, respectively. due to the development of text matching and pre-training models @xcite , a machine is able to achieve highly competitive results on these datasets, even close to human performance. for instance, esim @xcite achieves 88% on the dialogue nli @xcite , and bert achieves 85.8%, 93.1% and 98.5% in terms of r 10 @1, r 10 @2 and r 10 @5 on the ubuntu corpus @xcite . however, there is still a huge gap between high performance on the leader-board and poor practical user experience. chatbot engines often generate responses that are logically incorrect or violate commonsense knowledge @xcite . a likely reason is that current dialogue systems do not have strong reasoning skills, and most of the cases in previous benchmarks can be tackled by linguistic information matching. previous work has demonstrated that neural encoders capture a rich hierarchy of syntactic and semantic information @xcite . however, reasoning capability and commonsense knowledge are not captured sufficiently @xcite . one important research question is how we can evaluate reasoning ability in chatbots, which can potentially allow us to bridge the gap between high performance on leader-board and unsatisfactory practical performance. to this end, we develop " narrative " ## related work dialogue: the ubuntu dialogue corpus is a large retrieval-based dataset @xcite , extracted from ubuntu chat logs. persona-chat @xcite considers consistent personality in dialogue. crowd workers are required to act the part of a given provided persona, and chat naturally. dialogue nli @xcite is a natural language inference dataset modified from persona-chat. it demonstrates that nli can be used to improve the consistency of dialogue models. coqa @xcite is collected by pairing two annotators to chat about a passage in the form of questions and answers. each question is dependent on the conversation history. there are also several large-scale datasets in chinese, such as sina weibo @xcite , douban conversation corpus @xcite and e-commerce dialogue corpus @xcite . as shown in ma'am, you forgot your phone. oh, thanks, i couldn't live without this little thing. i know what you mean. it is of great significance to you. so did you enjoy your dinner? oh yes, everything was just perfect. it's so hard to take the whole family out to eat, but your restaurant was perfect. johnny had his own place to play in and i had time to talk with my sisters and their husbands. i'm glad to hear it. our kids area is always popular. well, you can be sure we'll be back. m m m f f f question & answer what is the probable relationship between the speakers? ✓ a. waiter and customer. ✘ b. brother and sister. ✘ c. husband and wife. ma'am, you forgot your phone. oh, thanks, i couldn't live without this little thing. i know what you mean. it is of great significance to you. so did you enjoy your dinner? oh yes, everything was just perfect. it's so hard to take the whole family out to eat, but your restaurant was perfect. johnny had his own place to play in and i had time to talk with my sisters and their husbands. comprehension. it relies on an external question to test the model's understanding capability. in contrast to the above dataset, our dataset is a next utterance prediction task, which is the fundamental problem in retrieval-based chatbots. in addition, our dataset requires various specific reasoning abilities, such as algebraic reasoning, intention prediction and so on, which is the main characteristic of our dataset. reasoning: recently, efforts have been made to develop benchmarks and tasks to address reasoning for language understanding. winograd schema challenge @xcite is a reasoningbased coreference resolution task. each pair of sentences differs by only one phrase. swag @xcite is derived from pairs of consecutive video captions, including 113k short context each with four candidates endings. commonsenseqa @xcite is a question answering dataset extracted from conceptnet @xcite . utilizing conceptnet to construct the dataset ensures that questions directly target commonsense reasoning. race is a machine reading comprehension dataset collected from english exams for chinese students. ai2 reasoning challenge @xcite ) contains 7,787 genuine grade-school level science questions with a corpus of 14m science reference sentences. drop @xcite and cosmos @xcite focus on factual understanding and commonsense comprehension, respectively. despite their success, these datasets can hardly help chatbots directly. following the traditional dia-logue response selection setting, we deeply modify english listening comprehension conversation to form an utterance prediction task. ## experiments we split the data into training, development and test sets, with an 80%, 10% and 10% ratio. we pack instances constructed from the same conversation during splitting to avoid data leakage. following the standard dialogue setting @xcite , we consider our task as a response selection task and employ traditional information retrieval evaluation methods, including recall at position 1 in 4 candidates (r@1), recall at position 2 in 4 candidates (r@2) and mean reciprocal rank (mrr) @xcite . we compare the performance of several response selection models as well as pre-training models. we simply introduce these works as follows: ## conclusion we introduced mutual, a high-quality manually annotated multi-turn dialogue reasoning dataset, which contains 8,860 dialogues and aims to test reasoning ability of dialogue models. we describe the process for generating mutual, and perform a detailed analysis. we find that various state-ofthe-art models show poor performance in mutual. the best model roberta only obtains 71.3% r@1. there is a large gap between the model performance and human performance. we hope that this dataset facilitates future research on multi-turn conversation reasoning problem.