Common Law Annotations: Investigating the Stability of Dialog System Output Annotations (2023)

Abstract: Metrics for Inter-Annotator Agreement (IAA), like Cohen’s Kappa, are crucial for validating annotated datasets. Although high agreement is often used to show the reliability of annotation procedures, it is insufficient to ensure or reproducibility. While researchers are encouraged to increase annotator agreement, this can lead to specific and tailored annotation guidelines. We hypothesize that this may result in diverging annotations from different groups. To study this, we first propose the Lee et al. Protocol (LEAP), a standardized and codified annotation protocol. LEAP strictly enforces transparency in the annotation process, which ensures reproducibility of annotation guidelines. Using LEAP to annotate a dialog dataset, we empirically show that while research groups may create reliable guidelines by raising agreement, this can cause divergent annotations across different research groups, thus questioning the validity of the annotations. Therefore, we caution NLP researchers against using reliability as a proxy for reproducibility and validity.

## introduction the acquisition of reliable, valid, and reproducible human annotations is an essential component of natural language processing (nlp) research. however, human annotations are inherently subjective @xcite and each annotator has their own biases @xcite . to overcome this subjectivity, research groups aim to develop annotation guidelines that increase inter-annotator agreement (iaa) among annotators, also known as inter-rater reliability. reliabilitythe level of agreement between the annotators-is a necessary, but not sufficient condition for reproducibility @xcite . if the precise details of the annotation process-from creating the annotation guidelines to executing the annotations themselves-are not transparent, the annotations may not be reproducible. furthermore, high reliability does not guarantee validity-the extent to which annotations accurately capture what is intended to be measured @xcite . to address these challenges, we first propose lee et al. protocol (leap) a codified annotation guideline creation process that standardizes the way research groups create, publicize, and implement annotation guidelines. leap ensures transparency in the annotation process through its step-by-step procedure, which is crucial to allow for better reproducibility and cross-paper analyses. second, we use leap to investigate agreement by having pairs of researchers simulate the annotation procedure of different research groups on a common dataset, in order to observe the change in agreement within and between these groups. within the simulation, we observe that each group creates their own unique guidelines, despite working on the same dataset and annotation categories. we leverage the metaphor of a common law, in which country/region-specific laws are based on precedent, much like researchers agreeing on common rules for edge cases to increase agreement. similar to common laws differing between countries, the rules governing annotation guidelines can become increasingly research group-specific and divergent from other groups, as each group strives to raise their iaa. after developing annotation guidelines, we analyze if these observations persist when crowdsourcing the data with each guideline. while leap has broader applications, here we apply it to a conversational ai task, where common human annotation metrics include appropriateness, information content of output, and humanlikeness @xcite . the popularity and recent advances in dialog agents, such as openai's @xcite , chatgpt, foot_0 and youchat foot_1 motivated us to showcase our method in the dialog domain. in our investigation, we ask the following research questions: ultimately, we make the following contributions: ## related work this paper contributes to the ongoing discourse on the 'science' of annotations @xcite . similar to @xcite , we call for improved reliability and transparency in annotations. ## conclusion in this paper, we caution nlp researchers against using reliability as a proxy for reproducability and validity. while leap does not strictly enforce validity, it creates transparency in the "common law" annotation rules. this transparency can enable others to assess the validity of the choices. we propose and encourage researchers to use leap as a solution to ensure reproducibility by rendering the annotation protocol completely transparent while allowing for deeper cross-paper analysis on validity through the standardized annotation procedure. using leap, we simulated a parallel series of independent annotation procedures, illustrating how even if a research group achieves agreement, their agreement with annotators from differ-ent groups can be low for certain categories due to diverging annotation guidelines. overall, research groups should use agreement metrics with care. while a high agreement score is often a community-recognized threshold required for research groups to publish their annotated datasets, research groups should be aware of the pitfalls in raising agreement metrics. furthermore, research groups should follow a standardized annotation guideline creation process, such as leap, and make the entire procedure transparent. with such standardization and transparency, we will be able to better understand the issues associated with simply using agreement metrics as the main threshold to cross for publications. each meeting is t minutes long & discussion is carefully documented minimum of n min rounds & maximum of n max rounds yes no no researchers annotate the same a items (shuffled) independently researchers annotate a new set of a items independently researchers discuss a common law annotation methodology each researcher annotates a large set of b items each researcher independently writes annotation guidelines researchers meet and merges their guidelines create amt tasks amt workers annotate researchers refine guidelines researchers annotate a items independently yes researchers discuss a common law annotation methodology did the annotation exceed the agreement threshold? did the annotation exceed the agreement threshold? iteration y round x the meeting is t minutes long & discussion is carefully documented