“Get Their Hands Dirty, Not Mine”: On Researcher-Annotator Collaboration and the Agency of Annotators (2024)

Abstract: Annotation quality is often framed as post-hoc cleanup of annotator-caused issues. This position paper discusses whether, how, and why this narrative limits the scope of improving annotation. We call to consider annotation as a procedural collaboration, outlining three points in this direction:(1) An issue can be either annotator- or researcher-oriented, where one party is accountable and the other party may lack ability to fix it; (2) yet, they can co-occur or have similar consequences, and thus any specific problem we encounter may be a combination;(3) therefore, we need a new language to capture the nuance and holistically describe the full procedure to resolve these issues.To that end, we propose to study how agency is manifested in annotation and picture how this perspective benefits the community more broadly.

## introduction the pursuit of data quality in nlp and ai usually takes a post-hoc, outcome-oriented approach, targeting the immediate goal of refining the data obtained. this includes the vast amount of ml-based post-processing @xcite , and recent notions for replacing crowd work @xcite . while more works recently seek to dig deeper, attention is still largely placed on retrospectively analyzing artifacts in existing datasets @xcite @xcite . emphasizing the output of a reliable set of annotations over the full data collection procedure often leads data issues to be attributed as undesired noises or artifacts from annotators' behaviors, biases, or failure @xcite with this narrative, annotators are implicitly stationed as the source of problems, and researchers as the party who then find, correct, and/or discard the flawed data. this could lead to an oversight of various systematic failures that attributed more to researchers, e.g., a flaw in the early stage of task design @xcite @xcite complicating this further, these independent issues -with potentially distinct sources and solutions -often co-occur, interact, and converge in forming the eventual, real-world problems we encounter. this creates a subtle chasm between theory and practice: while various independent issues are well-documented, annotation is still often understood only via retrospective observations. on the one hand, many works discuss the extensive intricacies of annotation tasks as well as what researchers should look for; on the other hand, improving data quality in practice remains a whaca-mole game: how data were collected doesn't seem relevant until something pops up; we then guesstimate what happened way back and attempt a fix, and again use it as normal until the next issue stands out. as users downstream are agnostic of the data collection details, this mode is limited. it is especially hard in retrospective analysis, if users are motivated at all, to further search through the long list of known issues and narrow down to some exact match in the literature with their guesstimates. this calls for a universal language that systematically captures how issues arise and interact in annotation pipelines, one that composes and compares the practices of the researchers and annotators in a coherent, generalizable framework. in this position paper, we urge the community to rethink the acquisition of annotations as a bilateral, collaborative process: the researchers as task designers and the annotators as task accomplishers, where responsibility is shared among both parties. to better describe flaws in such procedures, we propose to study the agency of annotators, i.e., how much capacity to act is allocated by task designers to an annotator toward the best outcome. we set forth an essential agency (mis)alignment problem: while researchers intend to provide a certain • malice / spamming @xcite • carelessness @xcite • inappropriate ai use @xcite • shortcutting @xcite • individual variations figure 1 : the spectrum of accountability. we extract and categorize atomic issues in the existing literature, grouped as researcher-or annotator-centered on the two ends of the spectrum. additionally, we mark a "grey zone" caused by inherent limitations in human language and cognition, which does not notably attribute to any single party. each issue is displayed in pair with one representative piece of work in related fields, and we discuss the issues in more detail in appendix a for interested readers to initiate a literature search. the atomic issues along the spectrum contribute together and interact in complex ways to form the problems we encounter in practice. our statement is based upon several pioneering pieces that consider the annotation process beyond annotators' behaviors and their outcomes. @xcite were among the first to point out that existing work targets annotator-related bias; instead, they measure instruction bias: artifacts introduced as early as in the instructions, which directly leads to downstream biases. @xcite surveys workers' perspectives and how they view their assigned roles. they show that the clarity of tasks impacts workers' willingness and effort level needed to complete them. plank (2022) review the human label variation "problem": disagreements in annotation may not always be noise, but instead suggest important properties of the data and task (e.g. hard cases). finally, @xcite highlight that the definition of annotation is essentially different within two contrasting paradigms, "descriptive" and "prescriptive" annotation, based on whether subjectivity should be encouraged. these individual pieces share a common insight: implementing a task with crowd work is only a fraction of the data-annotation pipeline. while this phase has many of its own flaws, it also reflects subtleties from earlier stages such as how researchers plan out and describe their tasks. consider two hypothetical failure cases for a research group seeking annotations that "represent a good output from a chatbot". in one case, the researchers used this generic description in quotes as the sole instruction, and the annotations they got were full of artifacts and inconsistencies. in the other case, several annotators improperly used chatgpt when they were clearly advised not to, and thus the results are unusable. for annotators in the first case, the best work would not possibly exceed a guesstimate of the researchers' intention: the "chatbot" in question and what would be a "good output" is agnostic. however, things could be largely different if the researchers take a moment to encode the fundamentals of their goal in the task design. conversely, in the second case, as long as some annotators insist on the improper move, a researcher won't be able to easily get rid of the trouble of extra work and wasted funds. while both can be reported as a quality issue, the actual cause and solution are distinct: one centering on the role of the researchers and the other on the annotators. ## discussion improving annotation quality is all about the interplay of annotators and designers. our framing of the agency misalignment problem essentially proxies a larger misalignment between what we think we're building and what we end up with. rather, anything could be part of a dataset; but the designers themselves should -and only are they able to -ensure it is the dataset, the exact one that we believe to serve a certain purpose. given our datahungry models, the stakes are high. a piece of data from a coarse design may instantly have a consequence on a sophisticated application: to determine which model is better, to place rewards on some options while penalizing others, and -eventuallyto decide what values are coded in our models and whose input is represented. shifting our focus and transferring the responsibilities in data collection also means letting go of control over our models, eventually adding to the already notorious myth of training data and evaluation @xcite @xcite . we hope our proposed framing can be an entry point into understanding what our data are and how they are generated, as early as when we conceive it as a hypothetical task. we see two ways this framing might be used. first, as a diagnostic, examining issues of misalignment can help to explain the human factors underpinning what may seem like a simple issue of low-quality annotations. this can lead to fixes with greater impact across multiple task designs. second, using this framing prospectively when designing a task can help to identify unforeseen complications as agency is allocated. for instance, thinking carefully about the degrees of freedom a codebook affords can help to identify its vulnerabilities to unusual annotator behavior. in considering this issue, we also draw attention to discussions in the field of crowdsourcing related to working conditions @xcite , techniques for shepherding crowd work @xcite , alternate collaborative annotation workflows @xcite , perspectivist judgments @xcite , as well as a wide range of quality control techniques under debate within the domain @xcite @xcite . ## conclusion we hope this position paper offers readers an additional perspective on how to approach annotation quality issues and their compounds via a cohesive and procedural lens. moving forward, we urge the community to adopt more holistic assessments of annotation pipelines, and draw insights from the ways that human factors, crowdsourcing, and nlp researchers are innovating in how tasks are designed and assessed.