Collecting high-quality adversarial data for machine reading comprehension tasks with humans and models in the loop (2022)

Abstract: We present our experience as annotators in the creation of high-quality, adversarial machine-reading-comprehension data for extractive QA for Task 1 of the First Workshop on Dynamic Adversarial Data Collection (DADC). DADC is an emergent data collection paradigm with both models and humans in the loop. We set up a quasi-experimental annotation design and perform quantitative analyses across groups with different numbers of annotators focusing on successful adversarial attacks, cost analysis, and annotator confidence correlation. We further perform a qualitative analysis of our perceived difficulty of the task given the different topics of the passages in our dataset and conclude with recommendations and suggestions that might be of value to people working on future DADC tasks and related annotation interfaces.

## introduction we present quantitative and qualitative analyses of our experience as annotators in the machine reading comprehension shared task for the first workshop on dynamic adversarial data collection. 1 . the shared task was a collection of three sub-tasks focused on the selection of excerpts from unstructured texts that best answer a given question (extractive question-answering). the sub-tasks included: (a) the manual creation of question-answer pairs by human annotators, (b) the submission of novel training data (10,000 training examples), and (c) the creation of better extractive question-answering models. in this paper, we focus on our participation in the the manual creation of question-answer pairs task dubbed as "track 1: better annotators". machine reading comprehension (mrc) is a type of natural language processing task that relies in the understanding of natural language and knowledge about the world to answer questions about a given text @xcite . in some cases, state-of-the-art mrc systems are close to or have already started outperforming standard human benchmarks @xcite . however, models trained on standard datasets (i.e., collected in non-adversarial conditions) do not perform as well when evaluated on adversarially-chosen inputs @xcite . to further challenge models and make them robust against adversarial attacks, researchers have started creating adversarial datasets which continuously change models as they grow stronger. dynamic adversarial data collection (dadc) is an emergent data collection paradigm explicitly created for the collection of such adversarial datasets. in dadc, human annotators interact with an adversary model or ensemble of models in real-time during the annotation process @xcite to create examples that elicit incorrect predictions from the model @xcite . dadc allows for the creation of increasingly more challenging data as well as improved models and benchmarks for adversarial attacks @xcite @xcite . there is evidence that data collected through adversarial means is distributionally different from standard data. from a lexical point of view, @xcite note that "what-" and "how-" questions dominate in adversarial data collection (adc) as opposed to "who-" and "when-" questions in the standard datasets. in the context of reading comprehension, dadc has been championed by @xcite , who observe that dadc qa datasets are generally syntactically and lexically more diverse, contain more paraphrases and comparisons, and often require multi-hop inference, especially implicit inference. apart from corpus analyses, researchers have also noted certain limitations of the dadc paradigm. for instance, @xcite note that annotators overfitting on models might lead to cyclical progress and that the dynamically collected data might rely too heavily on the model used, which can potentially be mitigated by mixing in standard data. similarly, @xcite find that dadc models do not respond well to distribution shifts and have problems generalizing to non-dadc tests. contributions in this paper, we present our experience as annotators in the reading comprehension shared task for the first workshop on dynamic adversarial data collection. through quantitative and qualitative analyses of a quasi-experimental annotation design, we discuss issues such as cost analysis, annotator confidence, perceived difficulty of the task in relation to the topics of the passages in our dataset, and the issues we encountered while interacting with the system, specifically in relationship with the commonly-used f1 word-overlap metric. we conclude with recommendations and suggestions that might be of value to people working on future dadc tasks and related annotation interfaces. ## task description track 1 of the first workshop on dynamic adversarial data collection consisted in generating 100 reading comprehension questions from a novel set of annotation passages while competing against the current state-of-the-art qa model @xcite , which would remain static throughout the task. through dynabench @xcite , 2 an annotation platform specialized in dadc, annotators would create model-fooling questions that could be answered with a continuous span of text. successful attacks required the annotators to pro-2 https://dynabench.org/ vide explanations of the question and a hypothesis for the model's failure. these were then subject to a post hoc human validation. ## methodology our annotator roster consisted of three annotators with postgraduate degrees in linguistics and natural language processing. one of the annotators spoke english as a first language, while the other two were proficient speakers of english as a second language who completed their graduate degrees in english-speaking universities. for the annotation process, we set up a quasi-experimental design using convenience sampling where approximately half of the annotations would be performed by a single annotator (@xmath0, and the other half would be performed synchronously by a group of two or more annotators (@xmath1. because the annotators live in different time zones, annotator groups did not remain consistent across group sessions. during the annotation task, the platform randomly picked a passage, usually of the length of a short paragraph (of about 160 words on average) from different topics. annotators could then choose to create questions for that passage or skip it entirely. annotators skipped passages when we agreed that it would be difficult to create even a single question to fool the model. 4 model fooled: items marked by dynabench as successful annotations. model not fooled: items marked by dynabench as unsuccessful annotations. false negatives: instances where the model was fooled, but dynabench marked them as not fooled. 5 false positives: items marked by dynabench as successful annotations but deemed unsuccessful by the annotators. 6 even though the limited number of examples does not allow us to draw any strong conclusions about the annotation task, we find our analyses worth presenting as a preliminary step for other annotators to further reflect on the annotation process during the planning stages of any dadc task. foot_5 ## qualitative analysis the relative difficulty of a dynamic adversarial dataset creation task may vary partly as a function of the genre and specific topic of the text passages from which question-answer pairs are drawn. during the shared task, dynabench randomly assigned passages for the creation of question-answer pairs, revealing several important aspects of this challenge. topics of the passages used in our data vary as shown by the success by topic scores in our more successful questions came from music, science, and technology topics. on the one hand, we are more familiar with these topics than comics, sports, and video games. furthermore, the paragraphs in literature and music tended to be more narrative in nature which, we believe, also made it easier for us to process them and create better questions. data-heavy, enumeration-based paragraphs typical of sports, history, and tv and movies topics proved more challenging for the creation of model-fooling questions. still, further examination is necessary to understand each of these possibilities separately. a closer examination of the dadc task included evaluating the success of different strategies for creating questions. overall, the model successfully answered questions about dates and names, as well as questions that could be answered with a single short phrase, especially if that phrase was produced as an appositive. for example, asking "which political associate of abraham lincoln was aware of his illness while traveling from washington dc to gettysburg?" allowed the model to select a name as the answer, which it did with a high degree of success, even when multiple distractor names appeared in the same paragraph. on the other hand, formulating questions that required longer answers, especially questions that asked for both "what" and "why", frequently fooled the model. furthermore, requiring references to multiple non-contiguous portions of the passage to make predictions also often fooled the model. still, using synonymous words or phrases or similar sentence structures to the critical portions of the passage allowed the model to make correct predictions, even when these other strategies may have fooled it under different circumstances. ## discussion based on the experience with dadc shared task track 1, we recommend several strategies to improve the efficiency of data collection.