C onditional QA : A Complex Reading Comprehension Dataset with Conditional Answers (2022)

Abstract: We describe a Question Answering (QA) dataset that contains complex questions with conditional answers, i.e. the answers are only applicable when certain conditions apply. We call this dataset ConditionalQA. In addition to conditional answers, the dataset also features:(1) long context documents with information that is related in logically complex ways;(2) multi-hop questions that require compositional logical reasoning;(3) a combination of extractive questions, yes/no questions, questions with multiple answers, and not-answerable questions;(4) questions asked without knowing the answers. We show that ConditionalQA is challenging for many of the existing QA models, especially in selecting answer conditions. We believe that this dataset will motivate further research in answering complex questions over long documents.

## introduction many reading comprehension (rc) datasets have been recently proposed @xcite @xcite @xcite . in a reading comprehension task, models are provided with a document and a question and asked to find the answers. questions in existing reading comprehension datasets generally have a unique answer or a list of answers that are equally correct, e.g. "who was the president of the us?" with the answers "george washington", "thomas jefferson", etc. we say that these questions have deterministic answers. however, questions in the real world do not always have deterministic answers, i.e. answers to the questions are different under different conditions. for example, in figure 1 , the document figure 1 : an example of question and document in condi-tionalqa dataset. the left side is a snapshot of the document discussing the eligibility of the benefit "funeral expense payment". the text span "her husband" satisfies the requirement on the "relationship with the decease" (in yellow). text pieces in green and red are requirements that must be satisfied and thus selected as conditions for the "yes" and "no" answers. a conditional answer consists of an answer and a list of conditions. an answer is only true if its conditions apply. in the example above, "you are arranging a funeral in the uk" is the condition for the answer "yes". an answer can have multiple conditions. conditional answers are commonly seen when the context so complex so asking a complete question with a deterministic answer is impractical; for example, when a person asks a question with some prior knowledge in mind but cannot enumerate all necessary details. a practical way to answer incomplete questions is to find all possible answers to the question -and if some answers are only true under certain conditions, the conditions should be output as well. answering such questions generally requires the models to understand the complex logic in the context and perform extensive reasoning to identify the answers and conditions. we present the conditionalqa dataset, which contains questions with conditional answers. we take documents from the uk government website foot_1 as our corpus. documents in this corpus discuss public policies in the uk and were first used in the sharc dataset @xcite . it is partically interesting for constructing the condition-alqa dataset because it contains complex contents with complex internal logic such as conjunction, disjunction, and exception (see the example in figure 1 ). questions in conditionalqa are asked by human annotators. each example contains a question, a scenario when the question is asked, and a document that discusses the policy that the question asks about. the task is to find all possible answers to the questions that apply to the user's scenario. if an answer is only true under certain conditions, the model should return the list of conditions along with the answer. answers and conditions are annotated by human annotators with the exact input, i.e. the question, the scenario, and the associated document. we provide supporting evidences labeled by human annotators as additional supervision. in addition to having conditional answers, con-ditionalqa also features the following properties. first, the documents in conditionalqa have complex structure. as opposed to wikipedia pages, where most sentences or paragraphs contain standalone information, documents in conditionalqa usually have complex internal logic that is crucial for answering the questions. second, many questions in the dataset are naturally multi-hop, as illustrated in the example on figure 1 , e.g. being "the partner of the deceased" satisfied the requirement on "your relationship with the deceased" which is one of high-level requirements to obtain the benefit. answering those question requires models that understand the internal logic within the document and reason over the it to find correct answers. third, we decouple the asking and answering process when annotating questions, as suggested by @xcite ; @xcite ; @xcite , so questions are asked without knowing the answers. forth, conditionalqa contains various types of questions including yes/no ques-tions and extractive questions. questions can have one or multiple answers, or can be not answerable, as a result of the decoupled annotation process. we experimented with several strong baseline models on conditionalqa @xcite @xcite . the best performing model achieves only 64.9% accuracy on yes/no questions, marginally better than the majority baseline (62.2% if always predicting "yes"), and 25.2% exact match (em) on extractive answers. we further measure the accuracy of jointly predicting answers and conditions, in which case the accuracy drops to 49.1% and 22.5%. the best metrics with conditions are obtained if no condition is predicted, showing how challenging it is for existing models to predict conditional answers. ## related works many question answering datasets have been proposed in the past few years @xcite @xcite @xcite and research on these has significantly boosted the performance of qa models. as large pretrained language models @xcite @xcite @xcite achieved better performance on traditional reading comprehension and question answering tasks, efforts have been made to make the questions more complex. several multi-hop qa datasets were released @xcite @xcite to test models' ability to solve complex questions. however, most questions in these datasets are answerable by focusing on a small piece of evidence at a time, e.g. a sentence or a short passage, leaving reasoning through long and complex contents a challenging but unsolved problem. some datasets have been recently proposed for question answering over long documents. qasper @xcite ) contains questions asked from academic papers, e.g. "what are the datasets experimented in this paper?". to answer those questions, the model should read several sections and collect relevant information. narrativeqa @xcite requires reading entire books or movie scripts to answer questions about their characters or plots. other datasets, e.g. hybridqa @xcite , can also be viewed as question answering over long documents if tables with hyper-linked text from the cells are flattened into a hierarchical document. sharc @xcite is a conversational qa dataset that also use uk government websites as its corpus. however, the sharc dataset only contains yes/no questions and the conversation history is generated by annotators with the original rule text in hand, making the conversation artificial. the length of context in sharc is usually short, such as a few sentences or a short paragraph. while using the same corpus, conditionalqa contains completely different questions and new types of answers. it focuses on a new problem that has not been previously studied. most of the existing datasets, including the ones discussed above, contain questions with unique answers. answers are unique because questions are well specified, e.g. "who is the president of the us in 2010?". however, questions can be ambiguous if not all information is provided in the question, e.g. "when was the harry potter movie released?" does not specify which harry potter movie. ambigqa @xcite ) contains questions that are ambiguous, and requires the model to find all possible answers of an ambiguous question and rewrite the question to make it well specified. similar datasets temp-lama @xcite , timeqa @xcite and situat-edqa (zhang and choi, 2021) have been proposed that include questions that require resolving temporal or geographic ambiguity in the context to find the answers. they are similar to conditionalqa in that questions are incomplete, but conditionalqa focuses on understanding documents with complex logic and answering questions with conditions. it's usually not possible to disambiguate questions in conditionalqa as rewriting the questions (or scenarios) to reflect all conditions of answers to make the questions deterministic is impractical. we create conditionalqa in the public policy domain. there are some existing domain specific datasets, including pubmedqa and bioasq @xcite in medical domain, udc @xcite in computer software domain, qasper @xcite in academic paper domain, privacyqa and policyqa @xcite in legal domain, etc. privacyqa and policyqa have similar context as conditionalqa, but the questions do not require compositional reasoning and the answers are short text spans. we use a corpus in the public policy domain because it is easy to understand by non-experts while being complex enough to sup-port challenging questions. conditionalqa is not designed to be a domain specific dataset. ## the task in our task, the model is provided with a long document that describes a public policy, a question about this document, and a user scenario. the model is asked to read the document and find all answers and their conditions if any. ## conclusion we propose a challenging dataset conditionalqa that contains questions with conditional answers. the dataset requires models to understand complex logic in a document in order to find correct answers and conditions to the questions. experiments on state-of-the-art qa models show that their overall performance on conditionalqa is relatively poor. this also suggests that current qa models lack the reasoning ability to understand complex documents and answer hard questions with answers beyond single span extraction. we hope that this dataset will stimulate further research in building nlp models with better reasoning abilities.