MMC o QA : Conversational Question Answering over Text, Tables, and Images (2022)

Abstract: The rapid development of conversational assistants accelerates the study on conversational question answering (QA). However, the existing conversational QA systems usually answer users’ questions with a single knowledge source, e.g., paragraphs or a knowledge graph, but overlook the important visual cues, let alone multiple knowledge sources of different modalities. In this paper, we hence define a novel research task, i.e., multimodal conversational question answering (MMCoQA), aiming to answer users’ questions with multimodal knowledge sources via multi-turn conversations. This new task brings a series of research challenges, including but not limited to priority, consistency, and complementarity of multimodal knowledge. To facilitate the data-driven approaches in this area, we construct the first multimodal conversational QA dataset, named MMConvQA. Questions are fully annotated with not only natural language answers but also the corresponding evidence and valuable decontextualized self-contained questions. Meanwhile, we introduce an end-to-end baseline model, which divides this complex research task into question understanding, multi-modal evidence retrieval, and answer extraction. Moreover, we report a set of benchmarking results, and the results indicate that there is ample room for improvement.

## introduction the ever-increasing variety of information leads to the current information explosion. question answering (qa) systems play an important role in alleviating information overload by providing users brief and accurate answers. towards this end, a great many qa systems have been developed by utilizing external knowledge sources to obtain the correct answer, including knowledgebased qa @xcite , document-based qa @xcite , and community-based qa @xcite . recently, as the rapid which city features a green copper statue of a woman holding a torch? new york city. what is the largest catholic church in it? st. patrick's cathedral. on what dates was the better with u tour in the city? march 9, 2012 development of conversational assistants, there is growing interest in all matters conversational. conversational qa, aiming to satisfy users' complex information needs via multi-turn conversations, attracts a lot of attention. the existing conversational qa systems usually rely on a single knowledge source, e.g., paragraphs or a knowledge graph, and assume it contains enough evidence to extract answers to users' questions. however, these conversational qa systems are limited in real-world qa scenarios due to the following reasons. on the one hand, the important visual cues are overlooked in the existing conversational qa systems. as an old saying goes, "a picture is worth a thousand words", namely a picture can often vividly express a lot of information. for example, as shown in figure 1 , the question "which city features a green copper statue of a woman holding a torch?" can be naturally answered by looking at the related picture. on the other hand, the series of questions in a conversation may dynamically require multiple knowledge sources that encompass different modalities rather than only one constant knowledge source. as shown in figure 1 , three questions in the conversation involve images, passages, and structured tables respectively to extract the correct answers. in fact, although qa systems have been well studied thus far, conversational question answering with multiple knowledge sources of multi-modalities is still untapped. in this paper, we hence define this novel research task, i.e., multimodal conversational question answering (mmcoqa), aiming to answer users' questions with multimodal knowledge sources via multiturn conversations. mmcoqa is indeed non-trivial due to the following research challenges. 1) priority of multimodal knowledge. for a specific question, one modality may be more suitable for locating its corresponding answer than the others. for example, questions about numerical inquiries like date or statistics are better answered by utilizing tables. different from the previous conversational qa tasks, the most appropriate modality that can be used to answer the current question is not given in mmcoqa. given the conversation context, how to correctly determine the appropriate modality for the current question is a challenge. 2) consistency of multimodal knowledge. different modalities may provide consistent evidence to extract the correct answer for a question. for example, for the first question in figure 1 , the visual modality provides intuitional and direct information, while the related paragraph "the statue of liberty ..., off the coast of new york city. she holds a torch in her raised right hand ..." also reveals a certain of cues to indicate the correct answer. how to utilize the consistency among different modalities to verify the answer is another challenge. 3) complementarity of multimodal knowledge. some questions may require evidences of different modalities to reason the final answer. for example, the question "billy slater played for the nrl team in 2006 with a character holding what on the logo?" must be answered based on both the table about billy slater's career and the image of his team logo. therefore, to answer these questions, the system is required to have the ability of reasoning across multiple modalities. more importantly, the aforementioned three issues are not standalone but interweaved as conversation goes. thus, mmcoqa is not the simple combi-nation of multimodal qa and conversational qa but requires deep multimodal understanding and reasoning abilities across multi-turn conversations, which leaves ample room to study. to advance the progress of building mmcoqa systems using data-driven approaches, we construct the mmconvqa dataset, the first dataset for mm-coqa (see the contributions of this work are threefold: ## related work conversational qa is a relatively new topic in the qa community. benefiting from the released dataset @xcite ), text-based conversational qa has been greatly developed. researchers proposed to model and filter the conversation context via binary term classification @xcite and question rewriting @xcite @xcite . recently, some efforts @xcite @xcite task dataset conversational modality retrieval dq conversational qa quac @xcite ) text coqa @xcite text qrecc @xcite text csqa @xcite kb multimodal qa mmqa @xcite multi -manymodal qa @xcite multi - ## dataset construction in fact, there are few websites or applications where we can directly obtain a huge amount of questions that are answered with multimodal knowledge sources, let alone in the conversational form. fortunately, we notice that the mmqa @xcite dataset contains a number of complex questions answered with multiple modalities of knowledge. considering that an important intention of developing conversational qa systems is to gradually satisfy users' complex information needs via multi-turn conversations @xcite , we intuitively propose to decompose these complex questions into conversational questions @xcite . for example, as shown in figure 2 , the complex question "the player not wearing a helmet in 2019-20 buffalo sabres season free agents was on what team?" can be better presented in a conversation "q1: which player not wear a helmet in 2019-20 buffalo sabres season free agents" and "q2: he was on what team in that season?". however, if we obtain conversational questions only by decomposing complex questions, the number of questions in a conversation is rather limited since a complex question can only be decomposed into two or three questions. therefore, we automatically generate potential conversations as references for annotators to refine. ## dataset analysis mmconvqa contains 1,179 conversations and 5,753 qa pairs. there are 4.88 qa pairs on average for each conversation, as summarized in ## mae model we introduce a multimodal conversational qa system with adaptive extractors, mae for short, as a baseline model. as figure 5 illustrates, mae divides the mmcoqa task into three steps: conversational question understanding, multimodal evidence retrieval, and adaptive answer extraction. ## conclusion we define a novel and practical task, i.e., mm-coqa, and identify its research challenges, including priority, consistency, and complementarity of multimodal knowledge. we construct the mmcon-vqa dataset, containing multiple supervised labels to facilitate related researches in this community. we also report a set of results and analyze the current bottleneck.