Cascading Biases: Investigating the Effect of Heuristic Annotation Strategies on Data and Models (2022)

Abstract: Cognitive psychologists have documented that humans use cognitive heuristics, or mental shortcuts, to make quick decisions while expending less effort. While performing annotation work on crowdsourcing platforms, we hypothesize that such heuristic use among annotators cascades on to data quality and model robustness. In this work, we study cognitive heuristic use in the context of annotating multiple-choice reading comprehension datasets. We propose tracking annotator heuristic traces, where we tangibly measure low-effort annotation strategies that could indicate usage of various cognitive heuristics. We find evidence that annotators might be using multiple such heuristics, based on correlations with a battery of psychological tests. Importantly, heuristic use among annotators determines data quality along several dimensions: (1) known biased models, such as partial input models, more easily solve examples authoredby annotators that rate highly on heuristic use, (2) models trained on annotators scoring highly on heuristic use don’t generalize as well, and (3) heuristic-seeking annotators tend to create qualitatively less challenging examples. Our findings suggest that tracking heuristic usage among annotators can potentially help with collecting challenging datasets and diagnosing model biases.

## introduction while crowdsourcing is an effective and widelyused data collection method in nlp, it comes with caveats. crowdsourced datasets have been found to contain artifacts from the annotation process, and models trained on such data, can be brittle and fail to generalize to distribution shifts @xcite @xcite . in this work, we ask whether systematic patterns in annotator behavior influence the quality of collected data. we hypothesize that usage of cognitive heuristics, which are mental shortcuts that humans employ in everyday life, can cascade on to data quality and model robustness. for example, an annotator asked to write a question based on a passage might not read the entire passage or might use just one sentence to frame a question. annotators may seek shortcuts to economize on the amount of time and effort they put into a task. this behavior in annotators, characterized by examples that are acceptable but not high-quality, can be problematic. we analyze the extent to which annotators engage in various low-effort strategies, akin to cognitive heuristics, by tracking indicative features from their annotation data in the form of annotator heuristic traces. first, we crowdsource reading comprehension questions where we instruct workers to write hard questions. inspired by research on human cognition @xcite , we identify several heuristics that could be employed by annotators for our task, such as satisficing @xcite , availability @xcite and representativeness @xcite . we measure their potential usage by featurizing the collected data and annotation metadata (e.g., time spent and keystrokes entered) ( §4). further, we identify instantiations of these heuristics that correlate well with psychological tests measuring heuristic thinking tendencies in humans, such as the cognitive reflection test @xcite @xcite . our psychologically plausible measures of heuristic use during annotation can be aggregated per annotator, forming a holistic summary of the data they produce. based on these statistics, we analyze differences between examples created by annotators who engage in different levels of heuristics use. our first finding is that examples created by strongly heuristic-seeking annotators are also easier for models to solve using heuristics ( §5). we eval-uate models that exploit a few known biases and find that examples from annotators who use cognitive heuristics are more easily solvable by biased models. we also examine what impact heuristics have on trained models. previous work @xcite shows that models generalize poorly when datasets are split randomly by annotators, likely due to the existence of artifacts. we replicate this result and find that models generalize even worse when trained on examples from heuristic-seeking annotators. to understand which parts of the annotation pipeline contribute to heuristic-seeking behavior in annotators, we also tease apart the effect of components inherent to the task (e.g., passage difficulty) as opposed to the annotators themselves (e.g., annotator fatigue) ( §6). unfortunately, we don't discover simple predictors (i.e., passage difficulty) of when annotators are likely to use heuristics. a qualitative analysis of the collected data reveals that heuristic-seeking annotators are more likely to create examples that are not valid, and require simpler word-matching on explicitly stated information ( §7). crucially, this suggests that measurements of heuristic usage, such as those examined in this paper, can provide a general method to find unreliable examples in crowdsourced data, and direct our search for discovering artifacts in the data. because we implicate heuristic use in terms of robustness and data quality, we suggest future dataset creators track similar features and evaluate model sensitivity to annotator heuristic use. foot_0 2 background and related work cognitive heuristics. the study of heuristics in human judgment, decision making, and reasoning is a popular and influential topic of research @xcite . heuristics can be defined as mental shortcuts, that we use in everyday tasks for fast decision-making. for example, @xcite asked participants whether more english words begin with the letter k or contain k as the 3 rd letter, and more than 70% participants chose the former because words that begin with k are easier to recall, although that is incorrect. this is an example of the availability heuristic. systematic use of such heuristics can lead to cognitive biases, which are irrational patterns in our thinking. at first glance, it may seem that heuristics are always suboptimal, but previous work has argued that heuristics can lead to accurate inferences under uncertainty, compared to optimization @xcite . we hypothesize that heuristics can play a considerable role in determining data quality and their impact depends on the exact nature of the heuristic. previous work has shown that crowdworkers are susceptible to cognitive biases in a relevance judgement task @xcite , and has provided a checklist to combat these biases @xcite . in contrast, our work focuses on how potential use of such heuristics can be measured in a writing task, and provides evidence that heuristic use is linked to model brittleness. features of annotator behavior have previously been useful in estimating annotator task accuracies @xcite . annotator identities have also been found to influence their annotations @xcite . our work builds on these results and estimates heuristic use with features to capture implicit clues about data quality. mitigating and discovering biases. the presence of artifacts or biases in datasets is welldocumented in nlp, in tasks such as natural language inference, question answering and argument comprehension @xcite @xcite . these artifacts allow models to solve nlp problems using unreliable shortcuts @xcite . several researchers have proposed approaches to achieve robustness against known biases. we refer the reader to @xcite for a comprehensive review of these methods. targeting biases that are unknown continues to be a challenge, and our work can help find examples which are likely to contain artifacts, by identifying heuristic-seeking annotators. prior work has proposed methods to discover shortcuts using explanations of model predictions @xcite , including sample-based explanations @xcite and input feature attributions @xcite . other techniques that can be helpful in diagnosing model biases include building a checklist of test cases @xcite , constructing contrastive @xcite or counterfactual @xcite examples and statistical tests @xcite . our work is complementary to these approaches, as we provide an alternative approach to bias discovery that is tied to annotators. ## annotation protocol we consider multiple-choice reading comprehension as our crowdsourcing task, because of the richness of responses and interaction we can get from annotators, which allows us to explore a range of hypothetical heuristics. we describe here the methodology for our data collection. we provided annotators on amazon mechanical turk with passages and ask them to write a multiple-choice question with four options. we used the first paragraphs of 'vital articles' from the english wikipedia foot_1 , and ensured that passages are at least 50 words long and at most 250 words long. passages spanned 11 genres including arts, history, physical sciences, and others, and passages were randomly sampled from the 10k passages. annotators were asked to write challenging questions that cannot be answered by reading just the question or passage alone, and have a single correct answer. further, they were asked to ensure that passages provided sufficient information to answer the question while allowing questions to require basic inferences using commonsense or causality. annotators were first qualified to avoid spamming behavior. this qualification checked for spamming behavior in the form of invalid questions, and not example quality. annotators were then asked to write a multiple-choice question to 4 passages in a single hit on mturk. annotators were asked to not work on more than 8 hits. we collected 1225 multiple-choice question-answer pairs from 73 annotators. in addition, we also logged their keystrokes and the time taken to complete an example (ensuring that time away from the screen was not counted). our annotation interface was built upon @xcite . for other details about our annotation protocol, please refer to appendix a. ## cognitive heuristics in crowdsourcing cognitive heuristics are mental shortcuts, that humans employ in problem-solving tasks to make quick judgments @xcite . annotators, tasked with authoring natural language examples, are not infallible to using such heuristics. we hypothesize that, in writing tasks, reliance on heuristics is a traceable indicator of poor data quality. in this section, we identify several heuristics, their consequences in annotator behavior, and features to track them. later, we also show they are predictors of qualitatively important dimensions of data. ## biased model solvability annotator heuristic traces are cognitively plausible measures that we hypothesize are indicators of large, potentially undesirable patterns annotators impart on data. to verify this, we test if examples created by heuristic-seeking annotators are more easily solvable by biased models. we consider heuristic examples as examples from those annotators who score highly on our heuristic indicator features. given the initial set of examples d, we distinguish a subset as heuristic, h k , formed by all examples from annotators in the top k% of average heuristic use across all annotators. we form such a subset independently for all heuristic indicator features we consider. 6 when h k is formed from the top quartile (@xmath0, 68% of annotators have examples included in at least 1 heuristic set, and 14% in for at least 4/6 heuristic sets. we find that few annotators never use heuristics. next, we evaluate how well biased models perform on heuristic subsets (h k ) compared to the remaining examples, d \ h k . we evaluate a few biased models, trained to use unreliable heuristics, on examples created with or without heuristics. below we describe the biased models we use. in all cases, we train or finetune models on qa data from @xcite and evaluate them on our data. for hyperparameter settings, please see appendix c. ## influencers of heuristic behavior next, we aim to understand what role the annotation pipeline plays in influencing heuristic use among annotators. various factors have been shown to determine example quality in crowdsourcing. these include task difficulty, incentives, annotator ability, motivation and fatigue @xcite . we looked at how such markers influence heuristic use among annotators. we considered two types of measures that could indicate difficulty: passage length (number of tokens) and inverse entity count (doc length / number of named entities) in the passage. longer documents, with fewer named entities, might provide context that is harder to form questions about. further, having completed more examples could make an annotator fatigued and/or gain expertise at the task. hence, we also used the sequence index of each example for an annotator. we computed pearson correlations between these indicators and the heuristic values for each annotator, and averaged the correlations across annotators. our results are summarized in ## qualitative analysis to better understand the differences in data produced by heuristic-seeking annotators, and otherwise, we conducted a comprehensive qualitative analysis of all our data. we annotate questions with properties inspired from previous work @xcite @xcite along the following dimensions: validity (is the question answerable given the context in the passage), context (how much context from the passage is needed to answer the question), and comprehension type (what kinds of comprehension are needed to answer the question). each question can have multiple labels. for a detailed description of these labels, please refer to appendix e. results. figure 5 presents the results of our annotation. we show the differences in the percentage of examples in the heuristic set, h 25 , and the remaining examples, d \ h 25 . first, examples in the nonheuristic set are more likely to be valid, and less likely to be unsolvable compared to the heuristic set. further, we find that examples in the heuristic set often require simple word matching and para-phrasing, while the ones in the non-heuristic set, are more likely to require multi-sentence reasoning. in terms of comprehension type, we find that heuristic examples are more likely to be answerable using information explicitly stated in the passage. on the other hand, non-heuristic examples are more likely to require implicit inference. these results suggest there are significant qualitative differences in examples from heuristic-seeking annotators. ## discussion our work measures the implications of annotators' potential use of cognitive heuristics in data quality. the analyses we present suggest that models are indirectly influenced by heuristic use and that previous observations, such as the success of partial input models, is a consequence. while many such consequences of heuristic use appear to be negative, we believe that this judgement should be left up to practical applications that use the data. we propose a fruitful direction for characterizing what models learn from data by considering annotator behaviors. practically, it is an open question as to how we can control downstream data using annotator heuristic traces. instead, we propose that future annotation efforts minimally track indicators of heuristic usage, using task-specific features, in an effort to document how they are reflected in the collected data and trained models. ## limitations one limitation of our study is that we analyze the implications of heuristic-seeking behavior in annotators for one task. future work could consider extending this methodology to many annotation tasks. for example, in sentence-pair classification tasks such as textual entailment, or in annotation of machine translation or summarization datasets, annotator heuristics could be useful in determining the quality of data and the biases embedded in them. to find stronger signals in the annotator heuristic traces, future work could consider training models to featurize heuristics.