S ituated QA : Incorporating Extra-Linguistic Contexts into QA (2021)

Abstract: Answers to the same question may change depending on the extra-linguistic contexts (when and where the question was asked). To study this challenge, we introduce SituatedQA, an open-retrieval QA dataset where systems must produce the correct answer to a question given the temporal or geographical context. To construct SituatedQA, we first identify such questions in existing QA datasets. We find that a significant proportion of information seeking questions have context-dependent answers (e.g. roughly 16.5% of NQ-Open). For such context-dependent questions, we then crowdsource alternative contexts and their corresponding answers. Our study shows that existing models struggle with producing answers that are frequently updated or from uncommon locations. We further quantify how existing models, which are trained on data collected in the past, fail to generalize to answering questions asked in the present, even when provided with an updated evidence corpus (a roughly 15 point drop in accuracy). Our analysis suggests that open-retrieval QA benchmarks should incorporate extra-linguistic context to stay relevant globally and in the future. Our data, code, and datasheet are available at https://situatedqa.github.io/ .

## introduction language reflects our ever-changing world; thus, the meaning of a sentence depends on a variety of extra-linguistic contexts, including when it was stated, who stated it, and where it was stated. figure 1 depicts examples of this, where the answer to a question varies based on the temporal and geographical contexts. in this work, we study such questions by exploring the affects of temporal and geographical contexts in open-retrieval qa. ideally, qa systems should provide up-to-date, localized answers to questions like those in figure 1 that are situated in the inquirer's temporal and geographical contexts. the existing paradigm for evaluating qa systems, however, makes implicit assumptions about time and location and does not measure how well qa models adapt to new contexts. recent work studying other nlp tasks have shown that this static paradigm misrepresents how models perform in practical settings @xcite @xcite . to address this challenge, we introduce situatedqa, a dataset comprised of questions from existing openretrieval qa datasets @xcite that have been re-annotated for their their contextdependence and answers across different situations (i.e., temporal or geographical contexts). 1 as we will see in section 3, a significant proportion of information-seeking questions are sensitive to the two extra-linguistic contexts we study. we annotate 9k questions from four existing datasets @xcite @xcite with their temporal dependence and 2k questions for their geographical dependence. annotators find that up to 30% of questions in existing datasets have answers that change over time, and rule-based heuristics using an ner tagger identify that about 5% of questions specify a location. we collect answers from different temporal and geographic contexts for such context-dependent questions (see using our collected data, we evaluate whether existing open-retrieval qa systems adapt to new temporal or geographical contexts by providing answers that are up-to-date or from new locations. while it is often assumed that retrieval based systems for qa @xcite can adapt to updated facts during inference by simply updating the retrieval corpus, we find that this is not the case. state-of-the-art retrieval based systems with updated corpora are 15 percentage points less accurate on questions whose answers have been updated versus those that have remained constant since the time when their largescale training dataset was collected. we also observe that models fail to generalize to answering questions from new locations @xcite , with accuracy dropping by 10 percentage points when asked to provide the answer from a rare location versus a common one. to support future research developing openretrieval qa systems which situate questions within the inquirer's extra-linguistic contexts, we propose two tasks for modeling what facts change across different extra-linguistic contexts and how those facts change. we also provide fine-grained evaluations for measuring how well models adapt to new temporal and geographical contexts. we establish initial performance levels on our tasks by adapting state-of-the-art methods for open-retrieval qa @xcite @xcite . finally, we provide rich analysis of existing qa systems, which suggest that benchmark construction should incorporate extralinguistic contexts to remain relevant globally and in the future. ## data collection here we describe the process for identifying context-dependent questions and collecting answers from alternate contexts. we split up data collection into three stages which are depicted in figure 2 : identification, {context / answer} collection, and validation. additional data collection details with interface screenshots can be found in appendix a. ## models in this section, we describe our baseline models for context-dependent question identification and situated qa. details on our learning settings and hyperparameters are in appendix d. ## analysis can qa systems differentiate and recall previous answers to a question? to answer this, we study a new setting where we query models for the current and previous answer to a given question. following the same steps as above for the temp and geo context types, we develop a suite of retrieval-based and closed-book baselines for this new setting. our query-modified baselines in this setting consist of prepending the word "previously" to the beginning of each question to query for the previous answer. in the existing framing of qa, all questions are looking for the current answer we do no augmentations to query for the current answer. for our finetuned baselines, we train separate reader models and closed book models to query for current and previous answers. we report our results for these experiments in can qa systems trained on outdated answers adapt to the present? we investigate whether qa systems that are pretrained on outdated corpora and that are trained on large-scale qa datasets containing outdated answers can adapt to answering questions situated in the present. @xcite shows that updating the corpera of retrieval based models can be effective method for updating simple facts, such as current heads of state. we test whether this extends to a broader range of temporally-dependent facts by evaluating our baselines introduced above on their ability to predict the current answer in our dataset, which are up-to-date as of feb 2021. we split the current answers into two categories, stable where answer did not change since 2018, and updated where answer changed after 2018. we chose 2018 as a threshold as it is when natural questions @xcite was collected and roughly matches the timestamp of the most recent data our closed book model @xcite was pretrained on (feb 2019). ## related work comparison to other qa datasets while a few prior studies mention that answers to a question can change over time @xcite , no prior work investigated the performance of existing models on temporally dependent queries and their prevalence. addressing ambiguous questions @xcite overlaps with our study, as ambiguity can arise from extra-linguistic contexts. their study found 13% of examples were ambiguous due to temporal deixis. however, we found that contextdependence can co-occur with inherent ambiguities in the question which cannot be resolved with context. thus, this figure underestimates the true proportion of temporally dependent questions. furthermore, humans don't always consider contextdependent questions as ambiguous when there is only one answer given the context, which is normally assumed to be present time and location. thus, ambiguities that arise due to lack of context should, therefore, be modeled separately from semantic ambiguities of the question. recent work has also studied generating temporally-dependent questions with timestamped answers from temporal knowledge-bases @xcite . similarly, @xcite generates temporally-dependent cloze-style prompts and present a temporally aware language model to address them. these works, however, synthetically generate examples and are therefore limited in scope and diversity. in contrast, we manually annotate temporally and geographically dependent questions. temporal understanding temporal understanding in nlp has been typically studied in an intradocument setting, i.e., ordering events or finding temporal relationship between two events in the same document @xcite @xcite @xcite . dynamic evaluation dynamic evaluation has been studied under the language modeling @xcite , topic modeling @xcite and entity linking @xcite . recent work @xcite studied temporal drift in the context of large scale pretrained language model, hinting reusing models from previous snapshot can cause performance decay, as we observe in our study as well. adversarial, phased data collection has been proposed @xcite @xcite to drive model development, constantly feeding models examples that current models are unable to address. we suggest collecting questions dynamically for slightly different goal, to accurately reflect the changing world by identifying temporally dependent facts from such benchmarks and continuously updating them. building a benchmark based on a fixed snapshot @xcite can be a viable alternative. researchers studied keeping knowledge sources up-to-date. @xcite looked at the constantly changing facts from the perspective of automatically extracting knowledgebase revisions. @xcite presents contrastive sentences based on wikipedia revisions, showing how entailment decision can change over time. ## conclusion & future work we present the first study of how extra-linguistic contexts affect open retrieval qa. our study reveals that current systems fail to adapt to shifts in the temporal or geographical context. we, therefore, propose tasks and create a dataset for training and evaluating qa systems on modeling how facts change across contexts. our dataset will support ample future work for developing models which can gracefully update its predictions based on new temporal and geographical contexts. future research may address incorporating in temporally and geographically dependent source documents, such as news articles, or considering other extralinguistic contexts such as who is asking the question, taking individual's preferences into account. inter annotator agreement inter annotator agreement during identification phase can be found in agreement with natural questions the answer span exact match between our collected answers and the original answers from nq where the annotated start and end transition dates overlap with nq's creation (2018) was around 40%, similar to the average agreement rate on nq-open test data is 49.2% in the original study @xcite . our manual analysis on randomly sampled 50 errors and find that only 12% of the errors can be attributed to annotation errors. 70% of other errors are due to ambiguities in the question resulting in multiple possible answers (e.g. "three movies made from agatha christie's novels"), or the same answer being given in different forms (e.g. "chamberlain, wilt" vs. "wilt chamberlain"). the remaining errors are the result of inconsistencies in nq annotations (18%).