When Retriever-Reader Meets Scenario-Based Multiple-Choice Questions (2021)

Abstract: Scenario-based question answering (SQA) requires retrieving and reading paragraphs from a large corpus to answer a question which is contextualized by a long scenario description. Since a scenario contains both keyphrases for retrieval and much noise, retrieval for SQA is extremely difficult. Moreover, it can hardly be supervised due to the lack of relevance labels of paragraphs for SQA. To meet the challenge, in this paper we propose a joint retriever-reader model called JEEVES where the retriever is implicitly supervised only using QA labels via a novel word weighting mechanism. JEEVES significantly outperforms a variety of strong baselines on multiple-choice questions in three SQA datasets.

## introduction scenario-based question answering (sqa) is the task of answering a question which is contextualized by a scenario describing a concrete case @xcite , e.g., predicting a judgment based on a legal scenario @xcite , answering a multiple-choice question in china's national college entrance examination (i.e., gaokao) based on a geographical scenario @xcite . sqa is a challenging task as it combines the difficulties of open-domain question answering (odqa) @xcite and machine reading comprehension (mrc) @xcite . indeed, sqa not only relies on accurately retrieving relevant paragraphs from a large corpus, but also requires thoroughly reading the retrieved paragraphs and the provided scenario to fuse their knowledge and infer an answer. for example, figure 1 shows a scenario and a multiple-choice question sampled from geography exams in gaokao. for machines to answer such a question, it would be insufficient to only read the given scenario but is necessary to retrieve from a corpus (e.g., textbooks, wikipedia) to acquire supporting knowledge. in this example, the scenario figure 1 : top: a scenario and a multiple-choice question about geography; in the scenario we highlight some keyphrases for retrieval to support answering the question. bottom: some supporting paragraphs in two articles retrieved from an online encyclopedia. arrows: two reasoning paths that connect the area described in the scenario with a similar area in the real world. research challenge. sqa, similar to odqa, is commonly solved by a retriever-reader framework. however, retrieval for sqa is fundamentally more difficult than for odqa because a scenario typically gives a long paragraph containing both keyphrases for retrieval and much noise, as illustrated in figure 1 . hence, in sqa, the paragraphs found by unsupervised retrievers such as bm25 are often not truly useful for answering the question. retrieval errors have become a bottleneck in sqa @xcite . supervised retrievers may be more powerful but they cannot directly apply to sqa due to the lack of relevance labels of paragraphs for training. only qa labels (i.e., correct answers) are available. re-cent dense passage retrievers (dpr) @xcite ) can be distantly supervised by qa labels, using paragraphs that contain the goldstandard answer as positives. while such distant supervision is effective for odqa where answers are short, it could not generate high-quality positives for most questions in sqa where the goldstandard answer is sentence-long and not explicitly contained in any paragraph. as a result, dpr shows unsatisfying performance on sqa datasets, as our experimental results will show. how can we effectively supervise a retriever for sqa? our approach-jeeves. to meet the challenge, we propose a novel retriever-reader method called jeeves where we employ only qa labels to implicitly train our retriever. it is realized by learning a word weighting for distinguishing keyphrases from noise in the scenario to improve the accuracy of retrieval and hence of the downstream qa which is supervised. since our retriever and reader are both supervised by qa labels, we train them jointly to further improve the overall accuracy. we summarize our contributions as follows. our code and data are on github: https://github.com/nju-websoft/ jeeves-gkmc . outline. we discuss related work in section 2, describe jeeves in section 3, present experiments in section 4, and conclude the paper in section 5. ## related work sqa has found application in many domains @xcite @xcite @xcite @xcite . among others, @xcite published the geosqa dataset containing multiplechoice questions in high-school geography exams, and they evaluated a variety of methods, mostly adopting a retriever-reader framework. all the tested methods performed close to random guess. the authors pointed out that a major cause of error is imprecise retrieval due to the long description of a scenario which is critical to retrieval but noisy. jeeves mainly aims at addressing this issue and it outperformed state-of-the-art methods on three sqa datasets in our experiments. retrievers for sqa as well as odqa are mainly unsupervised (e.g., bm25) due to the lack of relevance labels of paragraphs. although word matching based retrievers are still useful @xcite , they could not effectively exploit long and noisy scenario descriptions in sqa and became a bottleneck @xcite . supervised retrievers such as rerankers @xcite and dense retrievers @xcite @xcite need labeled paragraphs for training. while dense retrievers have been effectively trained with labels from distant supervision on odqa datasets, in our experiments they were less effective on sqa datasets such as geosqa where answers (i.e., options of multiple-choice questions) are sentencelong and often not explicitly contained in any paragraph. as a result, distant supervision by exact matching failed to generate positives, while approximate matching generated false positives. another way to handle the lack of labels is by transfer learning. for example, ar @xcite trains its retriever with labeled paragraphs from another similar dataset. however, we are not aware of any datasets that contain labeled paragraphs and are sufficiently similar to sqa datasets like geosqa. training with labeled paragraphs from a very different dataset would influence the accuracy of the learned retriever on the target dataset. indeed, ar showed unsatisfying performance on sqa datasets in our experiments. jeeves does not rely on distant supervision but only employs qa labels in the target sqa dataset to implicitly yet effectively train a retriever. readers have been extensively studied in mrc research. state-of-the-art methods use pre-trained language models (plms) @xcite @xcite . some readerretriever frameworks for odqa jointly train their retriever and reader @xcite . jeeves incorporates a reader which also builds on plms and fuses the knowledge in multi-ple paragraphs. jeeves also performs joint training. these design choices showed effectiveness in our ablation study, although they are not included in our main research contributions. ## experiments all the experiments were performed on rtx 3090. ## conclusion scenario-based multiple-choice questions have posed a great challenge to the retriever-reader framework: keyphrases for retrieving relevant paragraphs are blended with much noise in a long scenario description. in the absence of relevance labels of paragraphs, we devised a novel word weighting mechanism to implicitly train our retriever only using qa labels. it significantly outperformed existing unsupervised, distant supervision based, and transfer learning based retrievers. based on that, our joint end-to-end model jeeves exceeded a variety of strong baseline methods on three sqa datasets. while our experiments in the paper are focused on multiple-choice questions in high-school exams, jeeves has the potential to be adapted to other sqa tasks where scenarios are also very long, such as legal judgment predication. it would be interesting to explore this direction in the future. error analysis has revealed some shortcomings of jeeves, which we will address in future work. among others, we will incorporate knowledge graphs @xcite @xcite and enhance reasoning capabilities @xcite .