F ocus QA : Open-Domain Question Answering with a Context in Focus (2022)

Abstract: We introduce question answering with a cotext in focus, a task that simulates a free interaction with a QA system. The user reads on a screen some information about a topic, and they can follow-up with questions that can be either related or not to the topic; and the answer can be found in the document containing the screen content or from other pages. We call such information context. To study the task, we construct FocusQA, a dataset for answer sentence selection (AS2) with 12,165011unique question/context pairs, and a total of 109,940 answers. To build the dataset, we developed a novel methodology that takes existing questions and pairs them with relevant contexts. To show the benefits of this approach, we present a comparative analysis with a set of questions written by humans after reading the context, showing that our approach greatly helps in eliciting more realistic question/context pairs. Finally, we show that the task poses several challenges for incorporating contextual information. In this respect, we introduce strong baselines for answer sentence selection that outperform the precision of state-of-the-art models for AS2 up to 21.3% absolute points.

## introduction as more and more information-seeking activities are moving to visual interfaces, the way of interacting with qa systems is changing. an example is given by screen-based virtual assistants (e.g., google assistant, alexa and siri), where the interaction with the user can be conditioned by the information on the screen. to study this modeling, we introduce question answering with a context in focus, a task where an information-seeking user interacts with a qa system and, after reading some information shown on the screen, they ask a follow-on question. the information may represent the result of a previous exploratory interaction (e.g., tell me about lady gaga) or by some input provided by the system (e.g., news feeds and daily contents). we refer to this information as the context and, for the scope of this work, we define it as the title and the first paragraph of a web page shown on the screen. previous works approached this problem as a machine reading (mr) task, limiting the application to questions focused on a document (a questioner creates them reading that document) containing the answer @xcite . in contrast, our scenario allows for a free interaction with the system, where users can decide whether to use the context or not. they can ask questions that are (i) grounded in the context, e.g., related to entities in the context, (ii) based on additional knowledge, i.e., the user has a prior knowledge about the topic and uses it to formulate the question, or (iii) self-standing, i.e., related or unrelated to the context, and containing all the information needed to identify the correct answer. all these questions can be answered with in-page text, i.e., from the document, and/or off-page text, i.e., from other documents. figure 1 shows an example of this free information-seeking interaction. the task poses two key challenges: (i) understanding when and if the question is related to the context, and (ii) ensuring that the answers are contextually-relevant when required. in this scenario, ambiguity arises frequently and the answers to the same question may change depending on the context in which they are asked. even though mr is a common approach to answer extraction for open-domain question answering (odqa), we model this problem as an answer sentence selection (as2) task, where the answer is selected by ranking over all possible candidates @xcite . we believe that as2 is more relevant to a production scenario since humans converse in compact and complete sentences. current qa datasets have limitations that do not allow us to study question answering with a context in focus. they either (i) do not rely on the context at all @xcite @xcite , (ii) prompt users to write probing questions based on a given text @xcite , or (iii) assume a conversational user engagement at each turn @xcite @xcite . to overcome such limitations, we construct focusqa. 1 our dataset contains 12,165 unique question, context pairs and a total of 109,940 annotated answers. instead of asking annotators to write questions, we developed a new methodology that takes existing ones and pairs them automatically with multiple contexts. we sampled questions from naturalquestions @xcite (nq), qrecc @xcite , and answer-sentence natural questions (asnq) @xcite , a dataset for answer sentence selection derived from nq. to show the benefits of our dataset construction methodology, we compare its questions against a sample of 100 question, context pairs, where the questions are written by humans after reading the context. the analysis shows that this common way of crowdsourcing questions leads to examples where all the questions are only related to entities in the context. in contrast, with our approach, we obtain a mix of questions that are grounded in the context and questions that contain additional knowledge with respect to the context. to assess the initial performance on our 1 https://github.com/alexa/focusqa task/data, we experimented with models for as2. we show that the task sets several challenges: (i) when models do not use contextual information, state-of-the-art systems only achieve precision of 29.37%. (ii) when we introduce a set of strong baselines that incorporate the context and additional context from the answer page into transformer models @xcite , the combination of these contexts achieves a precision of 50.7%. however, the task requires further research exploration to get closer to humans' precision of 81.4% on the test set. in brief, our contribution is threefold: recently, the advance of various smart devices allows users to interact with the system in a much richer way. this has influenced the creation of cqa datasets leveraging a wider forms of context such as newspapers @xcite , web pages @xcite , visual information @xcite , and conversational history @xcite @xcite . @xcite is the closest to our research in that we use the same form of context. however, they collect questions by showing context to annotators explicitly, which leads to shallow, superficial questions far from real scenarios. ## data collection our approach to build the focusqa dataset is divided into two stages: (i) question/context pairing, and (ii) answer collection. to collect candidate answers we implemented an open-domain question answering (odqa) system for as2. for this work we used a bm25/lexical-based sentence retriever and a tranformer-based as2 reader, as this combination already implements a reliable qa system. we retrieve documents using an open-domain index containing ∼100m web pages from the open repository of common crawl. foot_0 see appendix a for more implementation details. all the annotation tasks are performed by a team of professional annotators with a project lead 3 ## contextual models contextual sentence selection (cs22) models are transformer models with multiple token-type (sentence) embeddings @xcite and as in input a single textual sequence of question, answer, and context. they are very suitable to be used for tasks where the context is required to find the answer. to adapt such models to the context available in focusqa, we introduce two new input sequences, namely, readout and cross-context, which embed question/answer pairs along with text from the answer page and the textual context from the screen, respectively. ## error analysis in this section we further analyse models behavior depending on the type of question and context. do models performance vary when context is not needed? results from ## conclusion we introduced question answering with context in focus, a task where there is a context in focus and users can ask questions that can be answered from in-page sentences, i.e., from the document in focus, or from off-page sentences, i.e., from other documents. to study the task, we constructed fo-cusqa, a dataset with 12,165 unique question, context pairs and a total of 109,940 answers. in order to elicit more realistic questions, i.e., annotators ask a question only based on that text information, we proposed a new methodology that can take any existing question and automatically pair it with a context in focus. we also introduced new input sequences for cs2 models. our experiments show their effectiveness in learning from our data, as they greatly outperform state-of-the-art models for as2 that do not make use of context. focusqa highlights challenges of modeling realistic information-seeking scenarios and invites further research into this area. for example the retrieval struggles to extract contextually-relevant passages from an open-domain index. future research can be devoted to study new approaches to model the context in focus in the retrieval stage. while we studied how to inject cross-contexts into qa models, learning when to use or not the context is another future research direction to explore. furthermore, context is limited to text (i.e., title and first paragraph) and future research may include extending the task to a multi-modal or multi-turn scenario.