Identifying Annotator Bias: A new IRT -based method for bias identification (2020)

Abstract: A basic step in any annotation effort is the measurement of the Inter Annotator Agreement (IAA). An important factor that can affect the IAA is the presence of annotator bias. In this paper we introduce a new interpretation and application of the Item Response Theory (IRT) to detect annotators’ bias. Our interpretation of IRT offers an original bias identification method that can be used to compare annotators’ bias and characterise annotation disagreement. Our method can be used to spot outlier annotators, improve annotation guidelines and provide a better picture of the annotation reliability. Additionally, because scales for IAA interpretation are not generally agreed upon, our bias identification method is valuable as a complement to the IAA value which can help with understanding the annotation disagreement.

## introduction in computational linguistics (cl), since carletta's influential paper @xcite , the standard approach to check the reliability in annotation efforts is the inter annotator agreement (iaa) -that is the extent to which different annotators independently make the same annotation decisions. for the purpose of calculating the iaa, carletta proposed the use of a family of statistics related to cohen's κ coefficient of agreement @xcite , which she collectively refers to as the kappa statistic. 1 an important factor that can affect the iaa is the presence of annotator bias -that is differences between annotator preferences for subjective reasons. recent work -for example @xcite , @xcite , @xcite and @xcite -show that annotators diverge in language annotation tasks due to a range of ineliminable factors such as background knowledge, preconceptions about language and general educational level. although clear annotation schemes with effective guidelines and annotator training aim to reduce annotation bias, some individual differences persist. a method that allows for a better identification and understanding of individual bias could be valuable in annotation efforts. for example, it could be used to: in some human annotation tasks, where a high level of subjectivity is involved -for example annotation that concerns the quality of a generated sentence (on a range of dimensions including syntax, semantic and pragmatics) -what determines the human annotators' decisions cannot be measured directly. it is not a physical dimension, as for example weight or distance. nevertheless, we can study the annotation behaviour, as directly observable, in order to have a better understanding of the unobservable decision process behind the human annotation. a first step in this direction is to analyse the frequency of the categories used by the annotators. the frequency gives us a first approximation of the annotator's behaviour. nevertheless, the frequency of the categories used is a raw image. ideally, we would like to be able to compare the annotators' annotations based on their perception of the phenomena being annotated. for this reason, in this paper we introduce a way to analyse the unobservable decision process behind the human annotation. 3 our method is based on a new interpretation and use of the item response theory (irt) @xcite . irt is a psychometric theory used for analysing and designing tools -for example, surveys and tests -for measuring abilities or attitudes. we will use two examples to explain the novelty of our interpretation: mathematical ability (a traditional use of irt), and natural language generation (our novel use). irt has traditionally been used, for example, to determine the validity of a test, say a test of mathematical ability. such a test is administered to a number of students. on each test item, each student will achieve a certain score. ideally, students with a strong mathematical ability should receive high scores on the test, and students with weak mathematical ability should receive low scores. there should also be a range in between these extremes. based on data of a large group of students completing such a test, irt can extract from the data a model that gives us, for a hypothetical student with a specific level of mathematical ability, the probability of specific test scores. ideally, a high level of mathematical ability should be associated with a high probability for a high test score. importantly, mathematical ability is not observed directly. it is only available via the student's performance on the test. and this coupling of the trait and the test performance is not going to be perfect (some good students may on occasion not do well, hence the use of a probability function). in irt, mathematical ability is modelled as a latent (not directly observable) trait of individuals. it is essential to note that a latent trait assumes a context, e.g. that of a conventional mathematical education. it exists against a background of assumption about culture, regional variance, personal preferences, schooling/training, etc. in our irt-based method to detect annotator bias, the annotation exercise involves annotating a corpus data (for example a set of sentences of images) with linguistic or other information. for the sake of simplicity, let's suppose that the annotation is an evaluation of nlg systems. in this case the annotation exercise involves a set of sentences. annotators are asked to judge each of these sentences according to one or more criteria. let's assume the criterion in question is the fluency of the sentence. our proposal is to treat fluency as a latent trait of sentences: it cannot be directly observed or measured, but we can get hold of it via the judgements of the annotators. in an annotation, the annotators will not always agree: for instance, some may be more severe in their judgements than others. we can now apply irt for individual annotators. this time, irt can extract from an individual's annotation the probability of a category's answer given a hypothetical sentence of a certain level of fluency to that annotator. for each of the fluency scores which the annotator can choose from when annotating, irt gives us the probability that that annotator will choose that particular of fluency score. this allows us to see how the annotator's behaviour (the score they assign) relates to the latent trait (the level of fluency of a hypothetical sentence). also in this case the latent trait assumes a context. it is a standard of fluency which is implicit in the judgements of a (more or less homogeneous) language community. the language community does distinguish between more and less fluent sentences. however, on specific occasions, individual members (or subgroups) will display smaller or larger differences in judgement. irt allows we present our proposal with an example taken from the case of intrinsic human evaluation of nlg systems @xcite . for this reason, in what follows we use the term judge instead of the term annotator. we choose such terminology to emphasize the evaluation aim. ## related work in this paper, we suggest considering irt as a method to visualize and understand annotators' subjective bias. to the best of our knowledge this is an original contribution. in natural language processing (nlp), irt has been used by @xcite . lalor et al. place their work inside the traditional use of irt, generating a gold-standard test-set for the recognizing textual entailment task and providing more insight into models' performance. in contrast, in this paper, we present a new interpretation of irt with the aim of improving and analysing annotation tasks. we propose to use it as a method for spotting outlier annotators, for improving annotation guidelines or for interpreting data reliability (for example, as a complement to the scale of interpretation for iaa values). regarding the use of iaa in corpus annotation tasks, and more specifically the task of linguistic annotation, we refer to @xcite , @xcite and @xcite . these provide extensive descriptions of how to perform an annotation task. regarding a general description and study of irt we refer to @xcite and @xcite . @xcite give an extensive presentation of irt. @xcite , gives a presentation of irt along side examples and information about the use of the r package ltm, a package developed for the application of irt. ## conclusion going beyond the analysis of category frequency, in this paper we have introduced a way to visualize and identify annotators' bias by the use of irt, more specifically the grm. our interpretation allows us to use irccc and the extremity parameters to gain an insight into the judges' annotation bias, based on a common latent trait scale of measurement. the use of irt sheds light on the annotation disagreement. accordingly, it can be used to have a better picture of the annotation reliability. as an example, in this paper we have showed how it can be used to accompany iaa values in order to explain the annotation disagreement. in particular, irt can show in which respect an annotator shows a stricter or more severe annotation behaviour than another annotator. our proposal can also be used as a standard procedure in pilot studies. indeed, it can help in taking some actions for the main annotation. for example, our proposal can be used to remove outlying annotators or it can be used for isolating where disagreement occurs. in the latter case, one option is then to discuss the disagreement with the annotators to improve the annotation guidelines. alternatively, identifying the areas of disagreement could be used to understand where to intervene for training the annotators. 11