Modeling Ambiguity with Many Annotators and Self-Assessments of Annotator Certainty (2020)

Abstract: Most annotation efforts assume that annotators will agree on labels, if the annotation categories are well-defined and documented in annotation guidelines. However, this is not always true. For instance, content-related questions such as ‘Is this sentence about topic X?’ are unlikely to elicit the same answer from all annotators. Additional specifications in the guidelines are helpful to some extent, but can soon get overspecified by rules that cannot be justified by a research question. In this study, we model the semantic category ‘illness’ and its use in a gradual way. For this purpose, we (i) ask many annotators (30 votes per item, 960 items) for their opinion in a crowdsourcing experiment, (ii) ask annotators to indicate their certainty with respect to their annotation, and (iii) compare this across two different text types. We show that results of multiple annotations and average annotator certainty correlate, but many ambiguities can only be captured if several people contribute. The annotated data allow us to filter for sentences with high or low agreement and analyze causes of disagreement, thus getting a better understanding of people’s perception of illness—as an example of a semantic category—as well as of the content of our annotated texts.

## introduction natural language is full of phenomena of ambiguity and uncertainty. however, we do not yet have a standard procedure for integrating ambiguities in formal models-or even for identifying ambiguities in the first place. most supervised machine learning tasks assume that there is a ground truth-an intersubjectively correct annotation. algorithms rely on this as unambiguous training data. for the most part, annotation efforts assume that multiple annotators will agree on labels, if annotation categories are welldefined and the annotation guidelines are clear and comprehensive. hence, low agreement scores are considered to indicate poor data quality. however, there is also an increasing awareness that this is not always the case @xcite @xcite @xcite . this is backed up by findings in cognitive science and linguistics that suggest that language phenomena are predominantly gradual in nature instead of being discrete categories, for instance in prototype theory @xcite . two common possible ways of capturing ambiguity in text are asking more than one annotator to annotate a category and/or asking annotators to explicitly mark ambiguities or uncertainties. in the study conducted for this paper, we combine both strategies to model the semantic category 'illness' and its use in a gradual way. illness lends itself to this type of analysis, because it is a highly socially constructed concept that makes ambiguities and disagreements likely. furthermore, this topic is of interest to linguists, humanists and social scientists alike and thus facilitated cooperation in our project herma @xcite . we use crowdsourcing in order to get the input of many annotators and use their overall vote as a measure of how clearly a sentence belongs to the topic 'illness'. in addition, we ask annotators to indicate their certainty with respect to their annotation. our aim is to investigate to what extent multiple annotations and self-assessments yield similar results, deriving recommendations for future research. furthermore, we identify reasons for disagreement in a qualitative analysis contrasting controversial and non-controversial sentences. our data comprise sentences from german literary texts and transcripts of political debates, thus allowing for a comparison of text types. ## related work in this section we present previous research dealing with ambiguity in annotation settings with multiple annotators. with regard to agreement and reliability, potter and levine-donnerstein (1999) (in the context of content analysis) differentiate three types of content to be annotated: 'manifest content (directly observable events), pattern latent content (events that need to be inferred indirectly from the observations), and projective latent content (loosely said, events that require a subjective interpretation from the annotator)' @xcite @xcite . the type of question asked in this paper ('is this sentence about topic x?') involves projective latent content and some disagreement is to be expected. when annotating something of which people have an everyday understanding, it can be more helpful to rely on the 'coders' existing schema' instead of defining more and more detailed annotation rules @xcite . in our study, we combine multiple annotations and self-assessment of the annotators' certainty, as has been done before by @xcite . they study the annotation of anaphora in dialogue data, where many references are vague without leading to misunderstandings. (see also @xcite for a detailed analysis of causes of ambiguities in coreference annotation.) the study attempts to capture ambiguities by, on the one hand, letting 18 students annotate the same text and, on the other hand, giving annotators the option of marking more than one antecedent if they perceive the reference as ambiguous. they conclude that it is important to consider cases of 'implicit ambiguity' which only emerge in the disagreements of multiple annotators and the individual annotator is not aware of. @xcite report on the annotation of coreference and bridging relations in the prague dependency treebank. they let annotators explicitly mark how certain they were about each annotated item. the analysis shows a correlation between annotator certainty and agreement, but also reveals many cases of disagreement despite high certainty. they agree with @xcite that ambiguity can be captured more fully by using multiple annotators instead of only letting annotators mark it explicitly. further studies consider disagreements between annotators as valuable information. @xcite conduct a crowdsourcing experiment to study which discourse adverbials licence which conjunctions (e. g. a clause with instead can start with the conjunctions but, so, or and). 28 annotators were presented with sentences with one of 20 adverbials and a gap for a possible conjunction. the results show that all adverbials have one to three conjunctions that participants considered acceptable, depending on context as well as individual preference. this variability could only be captured by a high number of annotators per item (similar: @xcite . @xcite investigate subjectivity in text interpretation. they let five annotators identify semantically related word groups in a text and specify the semantic relations between the words. while the annotators agree on some core words, individual differences are large. @xcite extends this method and concludes that '40% of the lexical cohesion perceived in text is subjectively interpreted' @xcite and therefore argues for a more reader-oriented modeling of text in computational linguistics. annotation with multiple annotators has been explored in literary studies, as polyvalence is an important textual characteristic for the definition of literariness. @xcite pose the question of how the falsifiability of interpretations can be guaranteed. their proposal is to use computer-aided narratological annotation to document interpretative decisions, which in turn can make conflicting interpretations visible. while some textual ambiguities do not have consequences for the overall interpretation, some ambiguities result in more than one possible interpretation of a literary text. @xcite target ambiguity in to the lighthouse by virginia woolf. the novel makes extensive use of free indirect speech that cannot be attributed unambiguously to one character or the narrator. to capture possible attributions, they let three to four student annotators analyze the same text span. they reach a raw agreement of slightly less than 70%, however, for many text spans more than one analysis is valid. multiple annotations have also been exploited for machine learning. @xcite use the information of annotator agreement to improve pos-tagging. in training their classification model, they sanction mistakes on tags with high inter-annotator agreement more heavily than on tags with low agreement. their experiments result in annotation improvements in several evaluation settings. @xcite optimize their classifiers for high precision by allowing the classifier to make no decision on low-agreement parts of the data. @xcite work on entailment, i. e. the question whether the proposition of a sentence b can be inferred from the proposition of a sentence a. they ask 50 annotators for their judgment on several hundred sentence pairs and show that the disagreement in the annotations cannot be attributed to noise only, but indicates that different interpretations are possible for many sentence pairs. they argue that computational models for textual entailment should therefore produce a full distribution of possible human answers instead of just one aggregated score. in a previous study of our @xcite annotated illness and compared an approach based on semantic fields with manual annotations. the agreement between the two annotators was rather low. the attempt to rectify this problem by improving the annotation guidelines led to the inclusion of very detailed rules that for the most part could not be justified by the demands of the research questions. in truth, illness is simply not a discrete concept, but can be present in a sentence to varying degrees. for this reason, we decided to approach the annotation of illness in a different way. we conducted a crowdsourcing study on the decision task whether a sentence is about illness or not. similar to @xcite and @xcite , we combine multiple annotators with a self-assessment of annotator certainty. closer to @xcite , we harvested a statistically relevant number of 30 judgements per sentence. ## annotation experiment 1 data. the sentences that were presented to the participants were extracted from two corpora that were compiled in the digital humanities project herma @xcite : we chose these two corpora because we wanted to cover different text types that we assume to differ with respect to ambiguity: literary texts are said to be especially ambiguous and authors play with ambiguity to create artistic value. political speeches aim more at a common understanding and should be as clear as possible, but can also be deliberately ambiguous. both corpora were split into sentences. sentence selection. sentences were sampled randomly. to ensure that the topic illness was present in a relatively large proportion of the sample, we filtered for sentences that contain a lexical item from the 'semantic field' @xcite of illness, which we realized as hyponyms of the term illness in germanet @xcite ). 2 for each corpus, we included 380 sentences with a lexical item related to illness and 100 without such a word, resulting in 960 items in total. question design. the selected sentences were presented to the annotators with a context of five sentences before and after the target sentence. less context was included if the text began or ended in this span. the target sentence was printed in bold. the annotators were presented with two or. three questions: the first one asked for a binary judgment in response to the question: is the topic of illness discussed in the sentence printed in bold? ('wird im fett gedruckten satz das thema krankheit thematisiert?'). given the condition that the answer to this question was 'yes', there was a follow-up question about topic centrality: annotators were asked whether illness is the central or rather a marginal topic of the sentence. the final question asked for the annotators' certainty: how certain are you about the answer to question 1? ('wie sicher bist du dir bei der antwort zu frage 1?'). possible answers were very certain ('sehr sicher'), rather certain ('eher sicher'), rather uncertain ('eher unsicher'), and very uncertain ('sehr unsicher'). guidelines. as already mentioned above, the goal of this study was not to review strictly formalized annotation guidelines nor the creation of a consensual gold standard. since we want to depict ambiguities through annotations, we decided to use extremely minimalist guidelines. in the task description, we even pointed out that we are interested in the subjective assessments of the annotators. however, we gave three example sentences: in the first, illness is the central topic of the sentence (a), in the second, illness is a marginal topic (b), and in the third, illness is not discussed (c). 'for breakfast i prefer to eat cereal.' the question whether a sentence deals with illness is an individual, conceptual decision. by using minimalist guidelines, we hope to cover disagreements caused by conceptual differences as to what the annotators consider to be illness as well as disagreements caused by grammar or style. annotation procedure. we collected 30 judgments for each of the 960 items. for the annotation procedure, we used the crowdsourcing platform appen the crowdworkers were paid @xmath010 for annotators working at average speed. during the annotation process it became apparent that the crowdsourcing platform could not supply us with a sufficient number of german speaking annotators. for this reason we additionally asked the students at our university to participate in the study. the students were paid e0.80 per page. for purposes of quality control, a number of test questions were used to exclude annotators who either did not understand the task or did not intend to work seriously on the task. as test questions we selected sentences that we considered unambiguous with respect to the question whether they are about illness. however, some of the test sentences turned out to be more ambiguous than we expected. if the annotators gave arguments for their deviant answer, and thus showed that they were actively engaging with the task, we accepted their answers. in total, 77 annotators participated in our study, 34 crowdworkers and 43 students. the extent of their work varies widely, ranging between 9 and 828 items. on average, participants annotated 374 items, with a huge standard deviation of 305 items. (see figure 1 for the full distribution.) about half of our data set was annotated by crowdworkers, the other half by students. in order to identify a possible effect of annotator types, we compared the proportion of votes for illness per annotator from the two groups. there is a significant difference in the annotations of the sentences from the fiction corpus with semantic field words (@xmath1, mann-whitney u test, @xmath2.0, @xmath3.001, students mean 0.64±0.14, crowdworker mean 0.57±0.07, rank-biserial correlation: -0.55). one possible reason for this difference is the fact that most of the students are studying language and literature and thus might approach the task differently than the crowdworkers. we do not consider this effect problematic and do not account for it in the analysis. there is no significant difference in the proportion of 'very sure' votes to the question on the annotators' certainty. (a) distribution of sentences by proportion of annotators that voted for 'illness' (@xmath4: response distributions (blue bars: semantic field word, orange bars: not a semantic field word). ## discussion and future work in our study, we tested two ways of capturing and modeling ambiguity in texts: by asking many annotators for their judgment and by asking the annotators for a meta annotation about their annotation certainty. we found that low annotator certainty and high variation in judgments are highly correlated. however, many data points that individual annotators were certain about did display variation. in addition, the correlation need not be given for every annotator or even any annotator individually. we conclude that multiple annotations are a useful means to identify and document ambiguity in texts. disagreement between our annotators was caused by 1) different concepts of what qualifies as illness (mental phenomena and very short-lived states are controversial), 2) grammatical phenomena like negation, and 3) stylistic properties of the text such as metaphors. especially in the fiction corpus, the information conveyed by the texts can also be ambiguous because the narrative is imprecise, vague or the semantic field as a tool to search for specific topics is ambivalent: while the absence of a semantic field word was a good indicator that the sentence is not about illness, the presence of a semantic field word only resulted in a 50% chance of the sentence being about illness. based on our annotations we can derive a weighted semantic field that can, for instance, be filtered to get a core semantic field. this allows for a reduction of hits to only those sentences that most people consider to be about illness. however, this would systematically exclude phenomena from the data set, as especially psychological phenomena led to disagreements. with respect to text types, our study revealed more ambiguous instances in the fiction corpus than in the transcript corpus. while one might consider metaphors to be the cause of ambiguity in literary text, this is not what the inspection of sentences with low agreement indicates. instead, the transcripts are characterized by many abstract terms like abhängigkeiten ('dependencies, addictions') which are mostly used in a way that is clearly unrelated to illness and do not cause any disagreement. the fiction corpus, on the other hand, names many minor symptoms (husten, 'cough') as everyday situations are described and can also report the inner life of characters. in the future, we plan to explore possibilities for training machine learning models on the data presented here. by showing how ambiguity levels can be represented by multiple annotations, we hope to prepare for the creation of a complex gold standard that incorporates conflicting evidence @xcite .