QAC onv: Question Answering on Informative Conversations (2022)

Abstract: This paper introduces QAConv, a new question answering (QA) dataset that uses conversations as a knowledge source. We focus on informative conversations, including business emails, panel discussions, and work channels. Unlike open-domain and task-oriented dialogues, these conversations are usually long, complex, asynchronous, and involve strong domain knowledge. In total, we collect 34,608 QA pairs from 10,259 selected conversations with both human-written and machine-generated questions. We use a question generator and a dialogue summarizer as auxiliary tools to collect and recommend questions. The dataset has two testing scenarios: chunk mode and full mode, depending on whether the grounded partial conversation is provided or retrieved. Experimental results show that state-of-the-art pretrained QA systems have limited zero-shot performance and tend to predict our questions as unanswerable. Our dataset provides a new training and evaluation testbed to facilitate QA on conversations research.

## introduction having conversations is one of the most common ways to share knowledge and exchange information. recently, many communication tools and platforms are heavily used with the increasing volume of remote working, and how to effectively retrieve information and answer questions based on past conversations becomes more and more important. in this paper, we focus on qa on conversations such as business emails (e.g., gmail), panel discussions (e.g., zoom), and work channels (e.g., slack). different from daily chit-chat @xcite and task-oriented dialogues @xcite , these conversations are usually long, complex, asynchronous, multi-party, and involve strong domain knowledge. we refer to them as informative conversations and an example is shown in figure 1 . however, qa research mainly focuses on document understanding (e.g., wikipedia) not dialogue understanding, and dialogues have significant differences with documents in terms of data format and wording style, and important information is scattered in multiple speakers and turns @xcite . moreover, existing work related to qa and conversational ai focuses on conversational qa @xcite instead of qa on conversations. conversational qa has sequential dialogue-like qa pairs that are grounded on a short document paragraph, but what we are more interested in is to have qa pairs grounded on conversations, treating past dialogues as a knowledge source. qa on conversation has several unique challenges: 1) information is distributed across multiple speakers and scattered among dialogue turns; 2) harder coreference resolution problem of speakers and entities, and 3) missing supervision as no training data in such format is available. the most related work to ours is the friendsqa dataset @xcite and the molweni dataset @xcite . however, the former is built on chit-chat transcripts of tv shows with only one thousand dialogues, and the latter has short conversations in a specific domain (i.e., ubuntu). the dataset comparison is shown in therefore, we introduce qaconv dataset, sampling 10,259 conversations from email, panel, and channel data. the longest dialogue sample in our data has 19,917 words (or 32 speakers), coming from a long panel discussion. we segment long conversations into shorter conversational chunks to collect human-written (hw) qa pairs or to modify machine-generated (mg) qa pairs from amazon mechanical turk (amt). we train a multi-hop question generator and a dialogue summarizer to generate qa pairs. we use qa models to identify uncertain samples and conduct an additional human verification stage. the data collection flow is shown in figure 1 . in total, we collect 34,608 qa pairs. we construct two testing scenarios: 1) in the chunk mode, a conversational chunk is provided to answer questions, similar to the squad dataset @xcite ; 2) in the full mode, a conversational-retrieval stage is required before answering questions, similar to the open-domain qa dataset @xcite . we explore several state-of-the-art qa models such as the span extraction roberta-large model @xcite trained on squad 2.0 dataset, and the generative unifiedqa model @xcite trained on eight different qa datasets and showed its generalization ability to 12 unseen qa corpora. we investigate the statistic-based bm25 @xcite retriever and the neural-based dense passage retriever @xcite trained on wikipedia (dpr-wiki). we show zero-shot and finetuning performances in both modes and conduct improvement study and error analysis. the main contributions of our paper are threefold: 1) qaconv provides a new testbed for qa on informative conversations including emails, panel discussions, and work channels. we show the po-tential of treating long conversations as a knowledge source, and point out a performance gap between qa on documents and qa on conversations; 2) we incorporate question generation (qg) model into the qa data collection, and we show the effectiveness of such approach in human evaluation. 3) we introduce chunk mode and full mode settings for qa on conversations, and our training data enables existing qa models to perform better on dialogue understanding. ## error analysis we further check the results difference between answerable and unanswerable questions in the unifiedqa t5 models outperform span-based models among the answerable questions, however, they are not able to answer any unanswerable questions and keep predicting some "answers". more interestingly, we observe that those span-based models perform poorly on answerable questions, as they can achieve a high recall but a low f1 score on unanswerable questions with a binary setting (predict answerable or unanswerable). this implies that existing span-based models tend to predict our task as unanswerable, revealing their weakness of dialogue understanding ability. then we check what kinds of qa samples in the test set are improved the most while finetuning on our training data using roberta-large. we find that 75% of such samples are incorrectly predicted to be unanswerable, which is consistent with the results in in addition, we sample 100 qa pairs from the errors which have an fz-r score lower than 50% and manually check and categorize these predicted answers. we find out that 20% of such examples are somehow reasonable and may be able to count as correct answers (e.g., ucla v.s. university of california, jay sonneburg v.s. jay), 31% are predicted wrong answers but with correct entity type (e.g., eurasia v.s. china, susan flynn v.s. sara shackleton), 38% are wrong answers with different entity types (e.g., prison v.s. drug test, thanksgiving v.s., fourth quarter), and 11% are classified as unanswerable questions wrongly. this finding reveals the weakness of current evaluation metrics that they cannot measure semantic distances between two different answers. ## related work qa datasets can be categorized into four groups. the first one is cloze-style qa where a model has to fill in the blanks. for example, the children's book test @xcite and the who-did-what dataset @xcite . the second one is reading comprehension qa where a model picks the answers for multiple-choice questions or a yes/no question. for examples, race @xcite and dream @xcite datasets. the third one is span-based qa, such as squad @xcite and ms marco @xcite dataset, where a model extracts a text span from the given context as the answer. the fourth one is open-domain qa, where the answers are selected and extracted from a large pool of passages, e.g., the wikiqa @xcite and natural question @xcite datasets. conversation-related qa tasks have focused on asking sequential questions and answers like a conversation and are grounded on a short passage. doqa @xcite ) is collected based on stack exchange, coqa @xcite and quac @xcite are the two most representative conversational qa datasets under this category. coqa contains conversational qa pairs, free-form answers along with text spans as rationales, and text passages from seven domains. quac collected data by a teacher-student setting on wikipedia sections and it could be open-ended, unanswerable, or context-specific questions. closest to our work, dream @xcite is a multiple-choice dialogue-based reading comprehension examination dataset, but the conversations are in daily chit-chat domains between two people. friendsqa @xcite is compiled from transcripts of the tv show friends, which is also chit-chat conversations among characters and only has around one thousand dialogues. molweni @xcite is built on top of ubuntu corpus @xcite for machine-reading comprehension tasks, but its conversations are short and focused on one single domain, and their questions are less diverse due to their data collection strategy (10 annotators). in general, our task is also related to conversations as a knowledge source. the dialogue state tracking task in task-oriented dialogue systems can be viewed as one specific branch of this goal as well, where tracking slots and values can be reframed as a qa task @xcite . moreover, extracting user attributes from open-domain conversations @xcite , getting to know the user through conversations, can be marked as one of the potential applications. the very recently proposed query-based meeting summarization dataset, qmsum @xcite , can be viewed as one application of treating conversations as databases and conduct an abstractive question answering task. ## conclusion qaconv is a new dataset that conducts qa on informative conversations such as emails, panels, and channels. we show the unique challenges of our tasks in both chunk mode with oracle partial conversations and full mode with a retrieval stage. we find that state-of-the-art qa models have limited dialogue understanding and tend to predict our answerable qa pairs as unanswerable. we provide a new testbed for qa on conversation tasks to facilitate future research.