Annotation Sensitivity: Training Data Collection Methods Affect Model Performance (2023)

Abstract: When training data are collected from human annotators, the design of the annotation instrument, the instructions given to annotators, the characteristics of the annotators, and their interactions can impact training data. This study demonstrates that design choices made when creating an annotation instrument also impact the models trained on the resulting annotations. We introduce the term annotation sensitivity to refer to the impact of annotation data collection methods on the annotations themselves and on downstream model performance and predictions. We collect annotations of hate speech and offensive language in five experimental conditions of an annotation instrument, randomly assigning annotators to conditions. We then fine-tune BERT models on each of the five resulting datasets and evaluate model performance on a holdout portion of each condition. We find considerable differences between the conditions for 1) the share of hate speech/offensive language annotations, 2) model performance, 3) model predictions, and 4) model learning curves. Our results emphasize the crucial role played by the annotation instrument which has received little attention in the machine learning literature. We call for additional research into how and why the instrument impacts the annotations to inform the development of best practices in instrument design.

## introduction supervised nlp models are typically trained on human annotated data and assume that these annotations represent an objective "ground truth." mislabeled data can greatly reduce a model's ability to learn and generalize effectively @xcite . item difficulty @xcite , the annotation scheme @xcite , and annotator characteristics @xcite can contribute to bias and variability in the annotations. how the prompts are worded, the arrangement of buttons, and other seemingly minor changes to the instrument can also impact the annotations collected @xcite . in this work, we investigate the impact the annotation collection instrument on downstream model performance. we introduce the term annotation sensitivity to refer to the impact of data collection methods on the annotations themselves and on model performance and predictions. we conduct experiments in the context of annotation of hate speech and offensive language in a tweet corpus. our results contribute to the growing literature on data centric ai @xcite , which finds that larger improvements in model performance often come from improving the quality of the training data rather than model tuning. the results will inform the development of best practices for annotation collection. ## background and related work annotation sensitivity effects are predicted by findings in several fields, such as survey methodology and social psychology. decades of research in the survey methods literature find that the wording and order of questions and response options can change the answers collected @xcite . for example, small changes in survey questions ("do you think the government should forbid cigarette advertisements on television?" versus "do you think the government should allow cigarette advertisements on television?") impact the answers respondents give. questions about opinions are more sensitive to these effects than questions about facts @xcite . psychologists and survey methodologists have also documented the cognitive shortcuts that respondents take to reduce the length and burden of a survey and how these impact data quality. respondents often satisfice, choosing an answer category that is good enough, rather than expending cognitive effort to evaluate all response options thoroughly @xcite . when the questions allow, respondents will go further, giving incorrect answers to reduce the burden of a survey @xcite @xcite . prior research finds strong evidence of task structure effects in annotations of hate speech and offensive language. when annotators coded both annotations on one screen, they found 41.3% of tweets to contain hate speech and 40.0% to contain offensive language. when the task was split over two screens, results differed significantly. the order of the annotations matters as well. when hate speech was asked first, 46.6% of tweets were coded as hate speech and 35.4% as offensive language. the converse was true when the order was switched (39.3% hate speech, 33.8% offensive language) @xcite . our work expands upon these findings to test the hypothesis that task structure affects not just the distribution of the annotations but also downstream model performance and predictions. ## methods we tested our ideas about the downstream effects of annotation task structure in the context of hate speech and offensive language in tweets. we used a corpus of tweets previously annotated for these tasks @xcite , but replaced the original annotations with new ones collected under five experimental conditions. we then trained separate models on the annotations from each condition to understand how the annotation task structure impacts model performance and predictions. ## results to understand the influence of the annotation task structure on the annotations themselves, we first compare the percent of tweets annotated as ol and hs and the agreement rates across conditions. we then evaluate several measures of model performance to study the impact of the task structure on the models, including balanced accuracy, roc-auc, learning curves, and model predictions. corresponding results for the lstm models are given in appendix §a.4. all statistical test account for the clustering of tweets within annotators. collected annotations prediction performance performance differs across training and testing conditions, particularly for ol, as evident in the row-and column-wise patterns in both panels of (a) predicting offensive language (b) predicting hate speech figure 3: performance (roc-auc) of bert models across annotation conditions figure 2. we do not observe higher performance scores in the main diagonal: models trained and tested on data from the same experimental condition do not perform better than those trained and tested in different conditions. models trained with data from condition a resulted in the lowest performance across all test sets for the ol outcome ( the ol models perform worse when evaluated on test data collected in condition d than on data collected in condition e (see the strong column effects in the last two columns of figure 2a ). again, no similar column effect is visible in the hs panel (figure 2b ). a potential explanation is that the second batch of 50 annotations was of lower quality due to annotator fatigue. evaluating models across conditions with respect to roc-auc largely confirms these patterns (figure 3 ). the performance differences across testing conditions (column effects) are particularly striking in the roc-auc figures, while differences between training conditions (row effects) are less pronounced. predicted scores comparing model predictions across conditions contextualizes our findings. learning curves we further study how the annotation conditions impact model performance under a range of training set sizes, investigating whether differences between conditions impact training efficiency. figure 4 shows learning curves for the bert models by training condition (line plots and corresponding scatterplot smoothers). these curves show how test set model performance changes as the training data size increases. for each training condition, 1% batches of data were successively added for model training and the model was evaluated on the (combined) test set. to limit computational burden, one model was trained for each training set size and training condition. while the differences in the learning curves across conditions for ol are small (figure 4a ), conditions a, c, and e stand out among the hs learning curves (figure 4b ). building models with annotations collected in condition e is less efficient: initially, more training data are needed to achieve acceptable performance. ## discussion design choices made when creating annotation instruments impact the models trained on the resulting annotations. in the context of annotating hate speech and offensive language in tweets, we created five experimental conditions of the annotation instrument. these conditions affected the percentage of tweets annotated as hate speech or offensive language as well as model performance, learning curves, and predictions. our results underscore the critical role of the annotation instrument in the model development process. models are sensitive to how training data are collected in ways not previously appreciated in the literature. we support the calls for improved transparency and documentation in the collection of annotations @xcite . if we assume a universal "ground truth" annotation exists, task structure effects are a cause of annotation noise @xcite . alternatively, if the annotations generated under different task structures are equally valid, task structure effects are a mechanism of researcher-induced concept drift @xcite . determining a ground truth annotation is difficult and in some instances impossible. in such situations, the documentation of instrument design choices is particularly important. lack of thorough documentation may explain why attempts to replicate the cifar-10 and im-agenet creation process encountered difficulties @xcite . large language models (llms) can assist with annotation collection, but we expect that human involvement in annotation will continue to be necessary for critical tasks, especially those related to discrimination or fairness. however, such opinion questions are exactly the kinds of questions that are most susceptible to wording and order effects in surveys @xcite . whether llms are also vulnerable to task structure effects is not yet known. in addition, we expect that researchers will increasingly use llms for pre-annotation and ask annotators review the suggested annotations. this review is vulnerable to anchoring or confirmation bias @xcite , where the annotators rely too much on the pre-annotations rather than independently evaluating the task at hand. task structure effects may impact anchoring bias as well. we want to give special attention to condition a, which presents all annotation tasks for a given item on the same screen. we suspect this condition is the most commonly used in practice because it requires fewer clicks and thus less annotator effort. the rate of hs and ol annotations was lowest in condition a. models trained on data collected in condition a had the lowest performance for offensive language and the highest performance for hate speech. there are several possible explanations for the differences in the annotations and in model performance which we cannot disentangle with our data. one explanation is an order effect: the hate speech button was shown first in a list of classes (response options). respondents may have satisficed by choosing just one annotation rather than all that apply @xcite . another explanation is distinction bias: asking the annotator to consider both classes simultaneously may lead the annotators to over-examine the distinctions between offensive language and hate speech @xcite . annotators in conditions d and e annotated 50 tweets and then saw the same tweets again. we hypothesize that the relatively poor performance of these conditions is the result of a fatigue effect. annotators may give lower-quality annotations in the second half of the task due to boredom and fatigue, which could partially explain the patterns we see in model performance (figures 2 , 3 ). however, we have not yet tested for order or fatigue effects. our results do not clearly demonstrate the superiority of any of the five conditions we tested. more research is needed to identify the best approach. in the meantime, we suggest incorporating variation in the task structure when annotation instruments are created for human annotators, a point also raised by @xcite . such variation can also help in creating diverse test sets that protect against the test (column) effects we observe (figures 2 , 3 ). relying on only one annotation collection condition leads to performance assessments that are subject to annotation sensitivity, that is, distorted by the design of the annotation instrument. limitations this research has explored annotation sensitivity only in english-language tweets annotated by prolific panel members living in the united states. the variations in annotations, model performance, and model predictions that we see across conditions could differ in other countries and cultures, especially in the context of hate speech and offensive language. the similar effects in surveys, which motivated our work, do differ across cultures @xcite . this work has also not demonstrated that task structure effects appear in other tasks, such as image and video annotation. in retrospect, we should have included a sixth condition that reversed the display of the label options in condition a, which would have let us estimate order effects. we encourage future work in this area to address these limitations. future work our results suggest that there are order and fatigue effects in annotation collection. future work should conduct experiments to esti-mate these effects. in surveys, measurement error is more common in later questions @xcite . future work could also assess fatigue effects by incorporating paradata @xcite , such as mouse movements @xcite and response times @xcite . these behavioral indicators could contextualize how fatigue impacts task performance and data quality. in subsequent work, we will explore how the annotators' characteristics influence their judgments and interact with the task structure effects. we also plan to diversify the tasks to include image evaluations and assignments that allow for a more accurate determination of ground truth. these enhancements will provide a more comprehensive understanding of annotation sensitivity. working with tasks that have (near) ground truth annotations would help the field make progress towards the development of best practices in annotation collection.