Does Putting a Linguist in the Loop Improve NLU Data Collection? (2021)

Abstract: Many crowdsourced NLP datasets contain systematic artifacts that are identified only after data collection is complete. Earlier identification of these issues should make it easier to create high-quality training and evaluation data. We attempt this by evaluating protocols in which expert linguists work ‘in the loop’ during data collection to identify and address these issues by adjusting task instructions and incentives. Using natural language inference as a test case, we compare three data collection protocols: (i) a baseline protocol with no linguist involvement, (ii) a linguist-in-the-loop intervention with iteratively-updated constraints on the writing task, and (iii) an extension that adds direct interaction between linguists and crowdworkers via a chatroom. We find that linguist involvement does not lead to increased accuracy on out-of-domain test sets compared to baseline, and adding a chatroom has no effect on the data. Linguist involvement does, however, lead to more challenging evaluation data and higher accuracy on some challenge sets, demonstrating the benefits of integrating expert analysis during data collection.

## introduction many datasets for training and evaluating natural language understanding (nlu) models consist of examples written by non-expert crowdworkers. while it is convenient and relatively inexpensive to gather large datasets from non-expert crowdworkers, the resulting datasets often suffer from systematic gaps and artifacts. through post hoc analysis, experts have identified many such problems and found that augmenting datasets with targeted examples can mitigate these issues @xcite . though non-expert crowdsourcing often produces flawed data, concerns about scalability and crowdworker diversity mean there is often no viable alternative. with this in mind, we investigate how to leverage expert linguistic knowledge during writing and annotation by having linguists dynamically identify artifacts and gaps in the data, then communicate with non-expert crowdworkers to instruct them towards strategies that address issues as they arise. we focus on natural language inference (nli; @xcite , a task where the goal is to predict the label (entailment, contradiction, neutral) that reflects the relationship of a hypothesis to a premise. for example. given the premise jenny loves all animals, the hypothesis jenny loves cats is an entailment, and jenny hates dogs, a contradiction. we choose nli because it is among the best-studied nlu tasks, with demonstrated value (e.g., in pretraining @xcite ), but also multiple well-documented data quality issues that arise in crowdsourced data collection, many of which can be traced to a given heuristic. because these heuristic-based issues are prevalent, we focus on nli with the aim that our methodology can inform data collection for new tasks in which there are fewer known heuristics. previous efforts to develop more effective nlu data collection protocols have been limited in their ability to assess the efficacy of their interventions, as they often lack direct comparisons between different collection methods. we directly compare three levels of expert involvement over five rounds of data collection: (i) a baseline group with no hands-on expert involvement ('baseline'), (ii) a group that follows linguistically-motivated constraints developed by experts after each data collection round to target heuristic-based weaknesses in the data ('linguist-in-the-loop' (litl)), and (iii) a group that extends the litl protocol to add direct interaction with the experts, including individuallevel discussion about the task, on the chat platform slack ('litl chat'). these three protocols are shown in figure 1 , and a task example with one of the constraints is shown in figure 2 . text: they inhabit the near-boiling water of geysers in yellowstone, and the even hotter water in volcanic vents on the ocean floor. the definitely correct sentence does not reuse nouns, verbs, adjectives, or adverbs from the text (@xmath00.05) • your maybe correct statement must not contain any of the following words: often, several, many, most, some, other, will maybe correct statement: the definitely incorrect sentence does not reuse nouns, verbs, adjectives, or adverbs from the text (@xmath11/hit 2 by @xmath220.00 bonus after the last round. to ensure we collect sufficient examples from each worker, we award a bonus worth 10% of base pay for reaching milestones of 10, 50, and 100 hits each round. to encourage workers to write high-quality examples, we pay a @xmath381 in round 1 (∼@xmath495 in round 5 (∼@xmath50.05-@xmath61.50 for any engagement, @xmath70.367. we offered @xmath80.10 per example, but given the somewhat low rate at which crowdworkers chose to attempt the challenges (28.6% and 21.2% for litl and litl chat, respectively), we find it likely that increasing the amount offered per example would have increased participation, potentially also increasing the benefits observed in model performance. in an exit survey, over 50% of workers in litl and litl chat indicated that they would have completed more optional challenges if the pay had been higher. we recommend that future work using challenge options offer bonuses worth at least 15% of the base pay. qualitative considerations though many crowdworkers in litl chat expressed that they enjoyed the extra communication, crowdworkers from litl and litl chat rated the task as 'more enjoyable' than typical mturk tasks at nearly identical rates (85.2% and 87.5% respectively, compared to 67.7% in baseline). workers' ratings of the difficulty of writing and validation tasks were also nearly identical among the three protocols. we therefore find that, for typical data collection on mturk, the addition of a chat platform to facilitate worker-expert interaction is ineffective at improving data quality. ## conclusion having experts review and analyze incoming crowdsourced data during data collection allows those experts to identify new areas of weakness at each round and update guidelines and constraints while there is still time for those interventions to lessen the impact of artifacts in the data. though we do not observe any increases in out-of-domain accuracy, linguist involvement leads to more challenging evaluation data and higher accuracy on some challenge sets in hans. one-on-one interactions between experts and crowdworkers, though reported in some studies as being beneficial for more challenging tasks, has no measurable effect in our study. future work could extend the expertinvolved protocol to identify additional interventions that would lead to datasets with better generalizability.