Conversational Machine Comprehension: a Literature Review (2020)

Abstract: Conversational Machine Comprehension (CMC), a research track in conversational AI, expects the machine to understand an open-domain natural language text and thereafter engage in a multi-turn conversation to answer questions related to the text. While most of the research in Machine Reading Comprehension (MRC) revolves around single-turn question answering (QA), multi-turn CMC has recently gained prominence, thanks to the advancement in natural language understanding via neural language models such as BERT and the introduction of large-scale conversational datasets such as CoQA and QuAC. The rise in interest has, however, led to a flurry of concurrent publications, each with a different yet structurally similar modeling approach and an inconsistent view of the surrounding literature. With the volume of model submissions to conversational datasets increasing every year, there exists a need to consolidate the scattered knowledge in this domain to streamline future research. This literature review attempts at providing a holistic overview of CMC with an emphasis on the common trends across recently published models, specifically in their approach to tackling conversational history. The review synthesizes a generic framework for CMC models while highlighting the differences in recent approaches and intends to serve as a compendium of CMC for future researchers.

## introduction developing open-domain, intelligent dialog systems that can satisfactorily interact like humans, perform complex tasks and/or answer on a range of topics has been one of the most ambitious and difficult goals in artificial intelligence (ai). the study of such systems, called conversational ai (convai), is at the confluence of natural language processing (nlp), information retrieval (ir), and machine learning (ml), attracting significant research from both academia and industry. the recent developments in deep learning (dl) @xcite and reinforcement learning (rl) @xcite have further boosted research in the domain, making it one of the most sought after research topics in ai. based on the nature of problems, a convai system is expected to solve three major research problems @xcite . question answering (qa) involves providing answers to user queries through conversation, using the knowledge drawn from various data sources like a snippet from a text, a collection of web documents, or an entire knowledge base. task completion expects the conversational agent to accomplish task/s for the user, using the information acquired through conversation. finally, social chat makes the agent emulate humans and converse seamlessly and appropriately with users, as in the turing test @xcite . each of these fields has its own set of challenges to tackle. challenges in qa can vary depending on the source of knowledge, the answer extraction strategy employed, and the domain of the question. machine reading comprehension (mrc) is one such challenge in qa, that requires the conversational qa (convqa) agent to understand a given open-domain text and thereafter answer question/s in conversation about it. these questions are often not paraphrased and may co-reference previous queries. the required answer may be a span of the given text or freeform. when the machine comprehension dialog involves multiple co-referenced questions such that a latter question may be a logical successor of the former, the challenge is termed as conversational machine comprehension (cmc). a lot of research in mrc revolves around single-turn qa, but multi-turn cmc also holds major relevance because humans seek information conversationally by asking follow-up questions for additional information based on what they have already learned. still, the inherent complexity involved in dealing with text comprehension and reasoning over dialogs and context had kept cmc as a far-fetched goal. however, the recent success in achieving at-par-with-human performance on single-turn mrc models @xcite due to the advancement in natural language understanding and modeling @xcite @xcite , and the introduction of large-scale conversational datasets coqa @xcite and quac @xcite have made information-seeking dialogs possible. as a consequence, cmc has seen a significant surge in research in recent years. in less than 2 years since the introduction of these datasets, there have been 40 submissions foot_0 to coqa leaderboard 2 and 22 submissions 1 to quac leaderboard 3 . many of these models are unpublished, indicating active ongoing research on these datasets. besides, the current state-of-the-art in quac lags behind human performance f1 benchmark by a margin of 6.7 1 , suggesting significant scope for improvement. almost simultaneously, there have been breakthroughs in nlp @xcite @xcite which the researchers have tried to leverage in their upcoming models @xcite @xcite . since many models are being published concurrently, there have been inconsistencies and/or overlap in their methodologies and research directions. this makes it difficult to compare different approaches against each other and weigh their pros and cons. this prevailing scenario has blurred the bigger picture and made it difficult for researchers to attend to novel research in this field. moreover, there is no singular summarized view on cmc models, except the individual literature studies of these publications which can be highly localized and inconsistent with the global view. thus, the current scenario motivates the need for organizing the scattered knowledge across these publications into a consolidated overview, so that future research in this field can be streamlined. this literature review, therefore, provides a bird-eye overview of conversational machine comprehension. we commence with an introduction to cmc, acquainting the reader with the challenges that make cmc unique, and the large-scale conversational datasets that spurred research in this field. to develop a general understanding of the cmc approaches, we shift the focus from comprehending individual models to observing the common trends that mark these models, synthesizing a generic framework for a cmc model in the process. we finally end our review with a discussion on the current trends and suggest advancements in the future. ## related work there have been several published literature reviews on mrc in recent years. @xcite provides an extensive review of conversational ai with a detailed account of the neural approaches being employed in each of its dialog systems (qa, task completion, and social chat). it briefly discusses the problem of cmc and its datasets but does not comment upon the recent advancements and prevalent approaches in this domain. @xcite provides a summary of all the recent single-turn mrc datasets and approaches, however, it briefly discusses coqa but does not touch upon any approaches for cmc. @xcite summarizes the classic models of single-turn mrc with a focus on deriving a common architecture and suggesting improvements based on the analysis. cmc is mentioned as an emerging research direction in this survey. the latest review by @xcite provides an overview of mrc along with the statistical analysis of datasets and the various problems in this domain. it mentions cmc as an mrc challenge but does not provide any further details. this review differs from its predecessors as it focuses primarily on conversational (multi-turn) machine comprehension which has not been detailed in the previous literature. cmc has its own set of challenges and an active research community around it. this calls for considering cmc as a separate research direction from single-turn mrc and review its rapid developments in terms of its general trends. ## what is conversational machine comprehension? the task of cmc is defined as: given a passage p , the conversation history in the form of questionanswer pairs {q 1 , a 1 , q 2 , a 2 , ..., q i-1 , a i-1 } and a question q i , the model needs to predict the answer a i . the answer a i can either be a text span (s i , e i ) @xcite or a free-form text {a i,1 , a i,2 , ..., a i,j } with evidence r i @xcite . single-turn mrc models cannot directly cater to cmc, as the latter is much more challenging to address. the major challenges being: ## multi-turn conversational datasets the surge in cmc research is credited to the emergence of large-scale multi-turn conversational datasets: coqa @xcite and quac @xcite . ## common trends across cmc models instead of describing each cmc model separately, we categorized them under the approaches they employ in their components (section 5) or other model characteristics. this will help in developing a high-level understanding of the cmc models. a model-wise summary of the cmc models is provided in appendix c. ## discussion how does the research progress in cmc, a constrained setup, benefit the more into-the-wild domain of conversational search? as stated by @xcite , conversational qa (and cmc) is a simplified setting of conversational search (convsearch), an information-seeking, "system ask, user respond" paradigm @xcite , that does not focus on asking proactively. cmc, specifically, tries to address the challenges of nlu, via contextual encoding, reasoning, and handling conversational history, via history selection and modeling. in that aspect, cmc is a concrete enough setting for ir researchers to understand the change of information needs and interactivity between conversational cycles. could commonsense reasoning improve cmc? commonsense reasoning (cr) is based on the set of background information or world knowledge that an individual is intended to know or assume, and may be missing from context. on the other hand, pragmatic reasoning, which the current cmc models cater to, is based on the derivation of explicit and implicit meanings within the context. the current mrc systems are nearing human performance on most datasets, however, they still perform poorly on single-turn cr based questions @xcite . while there is recently increasing interest in cr in the single-turn mrc setting @xcite @xcite , cmc remains relatively untouched. this may probably be due to the lack of foreknowledge requiring unanswerable questions (e.g. in squad 2.0 @xcite ) in current cmc datasets @xcite , suggesting a need for more complex cmc datasets that incorporate cr. however, humans annotators may often apply common-sense reasoning involuntarily while answering questions or comprehending, thus leaving room for incorporating cr in models. there seems to be no recent work that invalidates, experimentally, the role of cr in cmc. quac, for example, is drawn from articles on personalities, and current models still lag behind the human benchmark. it may be worth experimenting if adding domain knowledge or attributes about the context, like location and gender, help improve answering these questions. why did the paper focus on common trends across each component rather than a single overarching classification of cmc models? the study of common trends in modeling, rather than a single overarching classification, helped in providing a multi-faceted view of cmc that can generalize on future models, and identify possible open-ended research questions, such as (a) for history selection, ham @xcite ) has proved to be both effective and intuitive in selecting relevant history turns. the application of this history selection approach on previous techniques that considered immediate k turns could be experimented with. (b) as mentioned in training methodology (section 6.4), roberta-based cmc model @xcite that used knowledge distillation and adversarial training achieved state-of-the-art coqa results @xcite . this suggests that different training approach along with multi-task learning improves the performance of base models. these procedures could be experimented with more advanced models such as ham @xcite and flowdelta (yeh and chen, 2019). ## conclusion in this paper, we provide a holistic overview of conversational machine comprehension (cmc), which has seen a surge of research in recent years, owing to advancements in neural language modeling and the introduction of large-scale conversational datasets such as coqa @xcite and quac @xcite . we discuss the challenges that make cmc different from machine reading comprehension (mrc) and compare the multi-turn conversational datasets: coqa and quac, based on different cmc characteristics. to develop a high-level understanding of all the existing approaches to tackle cmc, we synthesize a general model framework and analyze the common trends across all the published models, loosely based on the components outlined in the framework. finally, we discuss some open questions that emerged during our research and which, in our view, can be explored further. this review could serve as a compendium for researchers in this domain and help streamline research in cmc. the dataset statistics are provided in passages collected from 7 diverse domains e.g. children stories from mctest, news articles from cnn, wikipedia articles, etc. sections from wikipedia articles filtered in the "people" category associated with subcategories like culture, animal, geography, etc.