Don’t Blame the Annotator: Bias Already Starts in the Annotation Instructions (2023)

Abstract: In recent years, progress in NLU has been driven by benchmarks. These benchmarks are typically collected by crowdsourcing, where annotators write examples based on annotation instructions crafted by dataset creators. In this work, we hypothesize that annotators pick up on patterns in the crowdsourcing instructions, which bias them to write many similar examples that are then over-represented in the collected data. We study this form of bias, termed instruction bias, in 14 recent NLU benchmarks, showing that instruction examples often exhibit concrete patterns, which are propagated by crowdworkers to the collected data. This extends previous work (Geva et al., 2019) and raises a new concern of whether we are modeling the dataset creator’s instructions, rather than the task. Through a series of experiments, we show that, indeed, instruction bias can lead to overestimation of model performance, and that models struggle to generalize beyond biases originating in the crowdsourcing instructions. We further analyze the influence of instruction bias in terms of pattern frequency and model size, and derive concrete recommendations for creating future NLU benchmarks.

## introduction benchmarks have been proven pivotal for driving progress in natural language understanding (nlu) in recent years @xcite @xcite . nowadays, nlu benchmarks are mostly created through crowdsourcing, where crowdworkers write examples following annotation instructions crafted by dataset creators @xcite @xcite . the instructions typically include a short description of the task, along with several examples @xcite @xcite . despite the vast success of this method, past studies have shown that data collected through crowdsourcing often exhibit various biases that lead to overestimation of model performance @xcite @xcite @xcite @xcite . such biases are often attributed to annotator-related biases, such as writing style and background knowledge @xcite (see more discussion on related work in §a). in this work, we propose that biases in crowdsourced nlu benchmarks often originate at an early stage in the data collection process of designing the annotation task. in particular, we hypothesize that task instructions provided by dataset creators, which serve as the guiding principles for annotators to complete the task, often influence crowdworkers to follow specific patterns, which are then propagated to the dataset and subsequently over-represented in the collected data. for instance, ∼ 36% of the instruction examples for the quoref dataset @xcite start with "what is the name", and this same pattern can be observed in ∼ 59% of the collected instances. to test our hypothesis, we conduct a broad study of this form of bias, termed instruction bias, in 14 recent nlu benchmarks. we find that instruction bias is evident in most of these datasets, showing that ∼ 73% of instruction examples on average share a few clear patterns. moreover, we find that these patterns are propagated by annotators to the collected data, covering ∼ 61% of the instances on average. this suggests that instruction examples play a critical role in the data collection process and the resulting example distribution. it is difficult to represent a task with a few examples, and bias in instruction examples makes it even more difficult since a task and its associated reasoning have a larger scope than instruction patterns. for example co-reference resolution, temporal commonsense reasoning, and numerical reasoning are much broader tasks than the prevalent patterns in quoref ("what is the name..."), mc-taco ("how long...") and drop ("how many field goals...") datasets. we investigate the effect of instruction bias on model performance, showing that performance is overestimated by instruction bias and that models often fail to generalize beyond instruction patterns. moreover, we observe that a higher frequency of instruction patterns in the training set often increases the model performance gap on pattern and nonpattern examples and that large models are generally less sensitive to instruction bias. in conclusion, our work shows that instruction bias widely exists in nlu benchmarks, often leading to an overestimation of model performance. based on our study, we derive concrete recommendations for monitoring and alleviating this bias in future data collection efforts. from a broader perspective, our findings also have implications on the recent learning-by-instructions paradigm @xcite , where crowdsourcing instructions are used in model training. ## instruction bias in nlu benchmarks instructions are the primary resource for educating crowdworkers on how to perform their task @xcite . bias in the instructions, dubbed instruction bias, could lead crowdworkers to propagate specific patterns to the collected data. here, we study instruction bias in nlu benchmarks foot_0 , focusing on two research questions: (a) do crowdsourcing instructions exhibit patterns that annotators can pick up on? and (b) are such patterns propagated by crowdworkers to the collected data? in our study, we use the instructions of 14 recent nlu benchmarks: foot_1 (1) clariq @xcite , (2) cosmosqa @xcite , (3) drop @xcite , (4) duorc @xcite , and (5) hotpotqa @xcite (6) hybridqa @xcite , (7) mc-taco @xcite , (8) multirc @xcite , (9) piqa @xcite , (10) qasc @xcite , (11) quoref (dasigi dataset pattern % ins. % s train % stest clariq [are|would|do] you 72.2 85.1 89 cosmosqa what aux 87.5 45.1 38.4 drop how many [field goals | years | yards | points | touchdowns] 70 62.5 62.5 duorc [how old | how | what | who] aux 70 85.1 84 hotpotqa [in | of | from | _ ] [which|what] aux 87.5 53.8 54.2 hybridqa which aux 29.4 25.7 15.1 mc-taco how long aux 100 -87.6 what aux 100 -90.1 how often aux 100 -85.3 aux... [still|always |by the time] 100 -67.3 when did / what time 100 -83.4 multirc what aux 14.3 38.4 41.5 piqa how [do|can] 66.7 43.7 42.9 qasc what aux 57.1 49.3 47 quoref what aux the [_| full|real|first |last] name 36.4 57 60 ropes which aux 42.9 74.1 20.7 sciqa what aux 100 83.6 84.5 wino-grande [because | so | while | since | but] ... the 73.7 63.4 63.1 average 72.7 59 62 ## effect on model learning let s train (s test ) be the set of training (test) examples, and denote by s p train (s p test ) and s -p train ( ## conclusions and discussion we identify a prominent source of bias in crowdsourced nlu datasets, called instruction bias, which originates in annotation instructions written by dataset creators. we study this bias in 14 nlu benchmarks, showing that instruction examples used to create nlu benchmarks often exhibit clear patterns that are propagated by annotators to the collected data. in addition, we investigate the effect of instruction bias on model performance, showing that instruction patterns can lead to overestimated performance as well as limit the ability of models to generalize to other task examples. based on our findings, we derive three recommendations for future crowdsourced nlu benchmarks: (1) crowdsourcing instructions should be diverse; this could be achieved, for example, by having a large number of instructive examples, rephrasing examples using neural models, or periodically sampling examples from a diverse set of previously collected examples. the latter could be done, for example, by maintaining a pool of diverse examples during the collection process and then presenting every annotator with a different random sample from this growing pool. (2) word patterns in collected instances should be analyzed during data collection, as well as possible correspondence to instruction examples. such analysis will help researchers monitor the collection process and the quality of the resulting data. (3) correlation between model performance and input patterns should be checked during evaluation.