What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks? (2021)

Abstract: Crowdsourcing is widely used to create data for common natural language understanding tasks. Despite the importance of these datasets for measuring and refining model understanding of language, there has been little focus on the crowdsourcing methods used for collecting the datasets. In this paper, we compare the efficacy of interventions that have been proposed in prior work as ways of improving data quality. We use multiple-choice question answering as a testbed and run a randomized trial by assigning crowdworkers to write questions under one of four different data collection protocols. We find that asking workers to write explanations for their examples is an ineffective stand-alone strategy for boosting NLU example difficulty. However, we find that training crowdworkers, and then using an iterative process of collecting data, sending feedback, and qualifying workers based on expert judgments is an effective means of collecting challenging data. But using crowdsourced, instead of expert judgments, to qualify workers and send feedback does not prove to be effective. We observe that the data from the iterative protocol with expert assessments is more challenging by several measures. Notably, the human–model gap on the unanimous agreement portion of this data is, on average, twice as large as the gap for the baseline protocol data.

## introduction crowdsourcing is a scalable method for constructing examples for many natural language processing tasks. platforms like amazon's mechanical turk give researchers access to a large, diverse pool of people to employ @xcite @xcite . given the ease of data collection with crowdsourcing, it has been frequently used for collecting datasets for natural language understanding (nlu) tasks like question answering @xcite , reading comprehension @xcite , natural language inference @xcite @xcite , and commonsense reasoning @xcite . there has been substantial research devoted to studying crowdsourcing methods, especially in the human-computer interaction literature @xcite @xcite . however, most prior research investigates methods for collecting accurate annotations for existing data, for example labeling objects in images or labeling the sentiment of sentences @xcite @xcite . there are some small-scale studies that use writing tasks, like writing product reviews, to compare crowdsourcing methodologies @xcite . however, we are unaware of any prior work that directly evaluates the effects of crowdsourcing protocol design choices on the quality of the resulting data for nlu tasks. decisions around methodology and task design used to collect datasets dictate the quality of the data collected. as models become stronger and are able to solve existing nlu datasets, we have an increasing need for difficult, high-quality datasets that are still reliably solvable by humans. as a result, our thresholds for what makes a dataset acceptable become stricter: the data needs to be challenging, have high human-agreement, and avoid serious annotation artifacts @xcite . to make collecting such large-scale datasets feasible, making well-informed crowdsourcing design decisions becomes crucial. existing nlp datasets have been crowdsourced with varying methods. the prevailing standard is to experiment with task design during pilots that are run before the main data collection @xcite . this piloting process is essential to designing good crowdsourcing tasks with clear instructions, but the findings from these pilots are rarely discussed in published corpus papers, and the pilots are usually not large enough or systematic enough to yield definitive conclusions. in this paper, we use a randomized trial to directly compare crowdsourcing methodologies to establish general best practices for nlu data collection. we compare the efficacy of three types of crowdsourcing interventions that have been used in previous work. we use multiple-choice question answering in english as a testbed for our study and collect four small datasets in parallel including a baseline dataset with no interventions. we choose qa as our test-bed over the similarly popular testbed task of natural language inference (nli) because of our focus on very high human-agreement examples which calls for minimizing label ambiguity. in multiple-choice qa, the correct label is the answer choice that is most likely to be correct, even if there is some ambiguity in whether that choice is genuinely true . in nli however, if more than one label is plausible, then resolving the disagreement by ranking labels may not be possible @xcite . in the trial, crowdworkers are randomly assigned to one of four protocols: baseline, justification, crowd, or expert. 1 in baseline, crowdworkers are simply asked to write question-answering examples. in justifica-tionthey are tasked with also writing explanations for their examples, prompting self-assessment. for the expert and crowd protocols, we train work-1 all the data is available at https://github.com/nyumll/crowdsourcing-protocol-comparison . to quantify the dataset difficulty, we collect additional label annotations to establish human performance on each dataset and compare these to model performance. we also evaluate the difficulty of the datasets for typical machine learning models using irt @xcite . we find that the expert protocol dataset is the most challenging. the human-model gap with roberta large @xcite on the unanimous agreement portion of expert is 13.9 percentage point, compared to 7.0 on the baseline protocol. the gap with unifiedqa @xcite ) is 6.7 on expert, compared to 2.9 on base-line. however, the crowd evaluation data is far less challenging than expert, suggesting that expert evaluations are more reliable than crowdsourced evaluations for sending feedback and assigning qualifications. we also find that the justification intervention is ineffective as a stand-alone method for increasing nlu data quality. a substantial proportion of the explanations submitted are duplicates, reused for multiple examples, or give trivial reasoning that is not specific to the example. lastly, to evaluate the datasets for serious annotation artifacts we test the guessability of answers by omitting the questions from the model input. this partial-input baseline achieves the lowest accuracy on expert, showing that the interventions used to successfully boost example difficulty may also reduce annotation artifacts. recently, there has been a flurry of corpora collected using adversarial models in the crowdsourcing pipeline. dua et al. (2019), nie et al. (2020a) , and @xcite use models in the loop during data collection, where crowdworkers can only submit examples that cannot be solved by the models. however, such datasets can be biased towards quirks of the model used during data collection @xcite . crowdsourcing methods while crowdsourcing makes it easy to collect large datasets quickly, there are some clear pitfalls: crowdworkers are generally less knowledgeable than field experts about the requirements the data needs to meet, crowdwork can be monotonous resulting in repetitive and noisy data, and crowdsourcing platforms can create a "market for lemons" where fast work is incentivized over careful, creative work because of poor quality requesters @xcite . @xcite give a broad overview of the variables at play when trying to crowdsource high-quality data, discussing many strategies available to requesters. motivated by the use of selfassessment in teaching @xcite , @xcite study the effectiveness of self-assessment and external assessment when collecting data for product reviews. they find that both strategies are effective for improving the quality of submitted work. however, @xcite find that crowdworker self-assessment can be unreliable since poor-performing workers overestimate their ability. @xcite test a justifyreconsider strategy: crowdworkers justify their annotations in a relation extraction task, they are shown a justification written by a different crowdworker, or an expert, and are asked to reconsider their annotation. they find that this method significantly boosts the accuracy of annotations. another commonly used strategy when crowdsourcing nlp datasets is to only qualify workers who pass an initial quiz or perform well in preliminary crowdsourcing batches @xcite @xcite @xcite . in addition to using careful qualifications, @xcite send workers feedback detailing errors they made in their qa-srl annotation. writing such feedback is labor-intensive and can become untenable as the number of workers grows. @xcite design a framework of promoting crowdworkers into "shepherding roles" to crowdsource such feedback. we compare expert and crowdsourced feedback in our expert and crowd protocols. ## data collection protocols we run our study on amazon mechanical turk. 2 at launch, crowdworkers are randomly assigned to one of four data collection protocols, illustrated in figure 1 . 3 to be included in the initial pool, workers need to have an approval rating of 98% or higher, have at least 1,000 approved tasks, and be located in the us, the uk, or canada. ## data validation we collect label annotations by asking crowdworkers to pick the correct answer choice for a question, given the context passage. in addition to the answer choices written by the writer, we add an invalid question / no answer option. we validate the data from each protocol. for crowd and expert, we only validate the data from the final large writing rounds. data from all four protocols is shuffled and we run a single validation task, collecting either two or ten annotations per example. we use the same minimum qualifications as the writing task (section 3), and require that workers first pass a qualification task. the qualification task consists of 5 multiple-choice qa examples that have been annotated by experts. 7 people who answer at least 3 out of 5 questions correctly receive the qualification to work on the validation tasks. of the 200 crowdworkers who complete the qualification task, 60% qualify for the main validation task. following @xcite , to incentivize higher quality annotations, we include expert labeled examples in the validation task, constituting 10% of all examples. if a worker's annotation accuracy on these labeled examples falls below 50%, we remove their qualification (7 workers are disqualified through this process), conversely workers who label these examples correctly receive a bonus. @xcite show that annotation disagreement may not be noise, but could be a signal of true ambiguity. @xcite recommend using high-humanagreement data for model evaluation to avoid such ambiguity. to have enough annotations to filter the data for high human agreement and to estimate human performance, we collect ten annotations for 500 randomly sampled examples per protocol. ## datasets and analysis we collect around 1,500 question-answer pairs from each protocol design: 1,558 for baseline, 1,534 for justification, 1,600 for crowd, and 1,580 for expert. we use the validation annotations to determine the gold-labels and to filter out examples: if there is no majority agreement on the answer choice, or if the majority selects invalid question, the example is discarded (∼ 5% of examples). for the 2-way annotated data, we take a majority vote over the two annotations plus the original writer's label. for the 10-way annotated data, we sample four annotations and take a majority vote over those four plus the writer's vote, reserving the remainder to compute an independent estimate of human performance. ## conclusion we present a study to determine effective protocols for crowdsourcing difficult nlu data. we run a randomized trial to compare interventions in the crowdsourcing pipeline and task design. our results suggest that asking workers to write justifications is not a helpful stand-alone strategy for improving nlu dataset difficulty, at least in the absence of explicit incentives for workers to write high-quality justifications. however, we find that training workers using an iterative feedback and requalification protocol is an effective strategy for collecting high-quality qa data. the benefit of this method is most evident in the high-agreement subset of the data where label noise is low. we find that using expert assessments to conduct this iterative protocol is fruitful, in contrast with crowdsourced assessments that have much lower inter-annotator agreement and the noisy signal from these assessments does not boost example difficulty. we confirmed with new york university's irb that crowdsourced nlp dataset construction work, including experimental work on data collection methods, is exempt from their oversight. the only personal information we collect from workers is their mechanical turk worker ids, which we keep secure and will not release.