R 4 C : A Benchmark for Evaluating RC Systems to Get the Right Answer for the Right Reason (2020)

Abstract: Recent studies have revealed that reading comprehension (RC) systems learn to exploit annotation artifacts and other biases in current datasets. This prevents the community from reliably measuring the progress of RC systems. To address this issue, we introduce R4C, a new task for evaluating RC systems’ internal reasoning. R4C requires giving not only answers but also derivations: explanations that justify predicted answers. We present a reliable, crowdsourced framework for scalably annotating RC datasets with derivations. We create and publicly release the R4C dataset, the first, quality-assured dataset consisting of 4.6k questions, each of which is annotated with 3 reference derivations (i.e. 13.8k derivations). Experiments show that our automatic evaluation metrics using multiple reference derivations are reliable, and that R4C assesses different skills from an existing benchmark.

## introduction reading comprehension (rc) has become a key benchmark for natural language understanding (nlu) systems, and a large number of datasets are now available @xcite @xcite . however, it has been established that these datasets suffer from annotation artifacts and other biases, which may allow systems to "cheat": instead of learning to read and comprehend texts in their entirety, systems learn to exploit these biases and find answers via simple heuristics, such as looking for an entity with a particular semantic type @xcite (e.g. given a question starting with who, a system finds a person entity found in a document). to address this issue, the community has introduced increasingly more difficult question answering (qa) problems, for example, so that answer-title: return to olympus [1] return to olympus is the only album by the alternative rock band malfunkshun. ## analysis effect of multiple references do crowdsourced multiple golden derivations help us to evaluate output derivations more accurately? to verify this, we evaluated oracle derivations using one, two, or all three references. the derivations were written by qualified workers for 100 dev instances. the larger improvement of the relation-level performance (+14.5) compared to that of the entitylevel performance (+8.0) also suggests that relations are linguistically more diverse than entities, as we expected (e.g. is in, is a town in, and is located in are annotated for a locational relation). ## conclusions towards evaluating rc systems' internal reasoning, we have proposed r 4 c that requires systems not only to output answers but also to give their derivations. for scalability, we have carefully developed a crowdsourced framework for annotating existing rc datasets with derivations. our experiments have demonstrated that our framework produces high-quality derivations, and that automatic evaluation metrics using multiple reference derivations can reliably capture oracle derivations. the experiments using two simple baseline models highlight the nature of r 4 c, namely that the derivation generation task is not simply the sf detection task. we make the dataset, automatic evaluation script, and baseline systems publicly available at https://naoya-i.github.io/r4c/ . one immediate future work is to evaluate stateof-the-art rc systems' internal reasoning on our dataset. for modeling, we plan to explore recent advances in conditional language models for jointly modeling qa with generating their derivations.