T opi OCQA : Open-domain Conversational Question Answering with Topic Switching (2022)

Abstract: In a conversational question answering scenario, a questioner seeks to extract information about a topic through a series of interdependent questions and answers. As the conversation progresses, they may switch to related topics, a phenomenon commonly observed in information-seeking search sessions. However, current datasets for conversational question answering are limiting in two ways: 1) they do not contain topic switches; and 2) they assume the reference text for the conversation is given, that is, the setting is not open-domain. We introduce TopiOCQA (pronounced Tapioca), an open-domain conversational dataset with topic switches based on Wikipedia. TopiOCQA contains 3,920 conversations with information-seeking questions and free-form answers. On average, a conversation in our dataset spans 13 question-answer turns and involves four topics (documents). TopiOCQA poses a challenging test-bed for models, where efficient retrieval is required on multiple turns of the same conversation, in conjunction with constructing valid responses using conversational history. We evaluate several baselines, by combining state-of-the-art document retrieval methods with neural reader models. Our best model achieves F1 of 55.8, falling short of human performance by 14.2 points, indicating the difficulty of our dataset. Our dataset and code are available at https://mcgill-nlp.github.io/topiocqa .

## introduction people often engage in information-seeking conversations to discover new knowledge @xcite . in such conversations, a questioner (the seeker) asks multiple rounds of questions to an answerer (the expert). as the conversation proceeds, the questioner becomes inquisitive of new but related topics based on the information pro-vided in the answers @xcite . such topic switching behaviour is natural in information-seeking conversations and is commonly observed when people seek information through search engines @xcite . according to @xcite , people switch from one to ten topics with a mean of 2.11 topic switches per search session. for example, a person can start a search session about tennis, and then land on roger federer, and after learning a bit about him may land on his country switzerland, and spend more time learning about other swiss athletes. thanks to tremendous progress in question answering research @xcite , we are coming close to enabling information-seeking conversations with machines (as opposed to just using keywords-based search). in order to realize this goal further, it is crucial to construct datasets that contain information-seeking conversations with topic switching, and measure progress of conversational models on this task, the two primary contributions of this work. in the literature, a simplified setting of information-seeking conversation known as conversational question answering (cqa) has been deeply explored @xcite . in this task, the entire conversation is based on a given reference text of a topic/entity. while the cqa task is challenging, it still falls short of the real-world setting, where the reference text is not known beforehand (first limitation) and the conversation is not restricted to a single topic (second limitation). @xcite and @xcite have attempted to overcome the first limitation by adapting existing cqa datasets to the opendomain setting. they do so by obtaining contextindependent rewrites of the first question to make the question independent of the reference text. for example, if the reference text is about augusto pinochet and the conversation starts with a question "was he known for being intelligent?", the question is re-written to "was augusto pinochet known for being intelligent?". however, as the entire question sequence in the conversation was collected with a given reference text of a topic, all the turns still revolve around a single topic. in this work, we present topiocqa 1 -topic switching in open-domain conversational question answering-a large-scale dataset for informationseeking conversations in open-domain based on the wikipedia corpus. we consider each wikipedia document to be a separate topic. the conversations in topiocqa start with a real information-seeking question from natural questions @xcite in order to determine a seed topic (document), and then the questioner may shift to other related topics (documents) as the conversation progresses. 2 throughout the conversation, the questioner is never shown the content of the documents (but only the main title and section titles) to simulate an information-seeking scenario, whereas the answerer has full access to the content along with the hyperlink structure for navigation. in each turn, both questioner and answerer use free-form text to converse (as opposed to extractive text spans as is common for an answerer in many existing datasets). figure 1 shows an example of a conversation from our dataset. the first question leads to the seed topic byzantine empire, and after two turns switches to mehmed the conqueror in q 4 , based on part of the answer (a 3 ) that contains reference to mehmed. note that the answers a 1 , a 3 , and a 4 are free-form answers that do not occur as spans in either the seed document or the follow up document. the topic then switches to anatolia based on part of the previous answer (a 4 ). the topics change in further turns to turkey and ankara. because of the conversational nature, topiocqa contains questions rife with complex coreference phenomena, for instance, q 9 relies on entities mentioned in a 7 , a 8 and q 1 . topiocqa contains 3,920 conversations and 50,574 qa pairs, based on wikipedia corpus of 5.9 million documents. on average, a conversation has 13 question-answer turns and involves 4 topics. twenty-eight percent of turns in our data-1 topiocqa is pronounced as tapioca. 2 a portion of the training data also contains conversations where the questioner asks the first question given a seed topic. figure 1 : a conversation from topiocqa. our dataset has information-seeking questions with free-form answers across multiple topics (documents). the consecutive turns from the same topic (document) have been excluded for brevity. to investigate the difficulty of the topiocqa dataset, we benchmark several strong retrieverreader neural baselines, considering both sparse and dense retrievers, as well as extractive and generative readers @xcite . inspired by previous work, we explore two ways to represent the question: (1) concatenating the entire conversation history @xcite , and (2) self-contained rewrites of the conversational question @xcite . the best performing model-fusion-indecoder @xcite trained on concatenated conversation history-is 14.2 f1 points short of human performance, indicating significant room for improvement. we also evaluate gpt-3 to estimate the performance in a closedbook zero-shot setting, and its performance is 38.2 f1 points below the human performance. ## dataset collection each conversation in topiocqa is an interaction between two annotators-a questioner and an answerer. the details about the annotator selection are provided in appendix a. ## dataset analysis we collected a total of 3,920 conversations, consisting of 50,466 turns. the annotators were encouraged to complete a minimum of 10 turns. conversations with fewer than 5 turns were discarded. we split the data into train, development, and test splits. ## experimental setup the task of open-domain information-seeking conversation can be framed as follows. given previous questions and ground truth answers {q 1 , a 1 , q 2 , a 2 , . . . , q i-1 , a i-1 } and current question q i , the model has to provide the answer a i . this can be considered as an oracle setting, as the gold answers of previous questions are provided. the models can optionally use a corpus of documents @xmath0, d 2 , . . . , d n }. ## results and discussion we report the end-to-end performance of all systems in ## conclusion we introduced topiocqa, a novel open-domain conversational question answering dataset with topic switching. in this work, we described our data collection effort, analyzed its topic switching behavior, and established strong neural baselines. the best performing model (dpr retriever + fid) is 6.9 points em and 14.2 points f1 below human performance, suggesting that advances in modeling are needed. we hope our dataset will be an important resource to enable more research on conversational agents that support topic switches in information-seeking scenarios.