Representation Problems in Linguistic Annotations: Ambiguity, Variation, Uncertainty, Error and Bias (2020)

Abstract: The development of linguistic corpora is fraught with various problems of annotation and representation. These constitute a very real challenge for the development and use of annotated corpora, but as yet not much literature exists on how to address the underlying problems. In this paper, we identify and discuss five sources of representation problems, which are independent though interrelated: ambiguity, variation, uncertainty, error and bias. We outline and characterize these sources, discussing how their improper treatment can have stark consequences for research outcomes. Finally, we discuss how an adequate treatment can inform corpus-related linguistic research, both computational and theoretical, improving the reliability of research results and NLP models, as well as informing the more general reproducibility issue.

## introduction linguistically annotated corpora have for many decades occupied a firm place in the linguistics toolbox. they are of vital importance for theoretical and computational linguistic research in providing empirical evidence for language use, both from a qualitative and quantitative perspective. moreover, machine learning algorithms typically applied in the context of natural language processing (nlp) require annotated datasets for building language models. annotated corpora now exist in many forms and much effort has been devoted to developing specific schemes for different levels of linguistic analysis (e.g., phonology, morphosyntax, semantics, pragmatics). for example, the penn treebank @xcite and the universal dependency treebank (e.g., @xcite are syntactically annotated corpora, while, e.g., discourse relations are annotated in the penn discourse treebank @xcite . although many levels of linguistic annotation can be performed via automated means, some manually annotated training data is typically required as a foundation. moreover, there are certain types of corpora where manual annotation remains the best option in the face of complex linguistic phenomena (e.g., historical language stages or non-standard varieties). any manual annotation process represents a compromise between an accurate linguistic analysis and an annotation scheme which is generalizable enough to serve computational tools and the end user. annotation schemes are typically designed with a specific purpose in mind, and this will bear heavily on the decisions made. the compromises tend to revolve around details of the annotation scheme and one possible result of hard-fought compromises is that the resulting representations may actually be inaccurate with respect to several factors. in this paper, we identify five major factors: (i) ambiguity, (ii) variation, (iii) uncertainty, (iv) error, and (v) bias (see also figure 1 ). in particular, uncertainty has already been recognized as a problem in linguistic corpora @xcite , with a special focus on the interrelation between linguistic ambiguities and uncertainty in the annotation of historical linguistic data @xcite . in this paper we extend the discussion beyond ambiguity and uncertainty to include variation, error and bias as sources of representation problems in linguistic annotations and argue for the importance of developing a robust framework which explicitly treats these problems. we focus primarily on representation problems in historical corpora, but the set of problems is transferable to other types of linguistically annotated resources @xcite @xcite . as part of future work, we intend to build a computational implementation that is based on the crucial foundations laid out in this paper. 2 background and related work existing approaches typically treat representation problems in one of three ways in linguistic annotation processes: (i) stochastic treatment, (ii) assignment of an 'other/miscellaneous' category, (iii) left unannotated. in part-of-speech (pos) tagging and syntactic parsing, ambiguities are often treated in a stochastic manner, so that among the possibilities the option which is most likely is chosen (cf. most frequent class baseline; @xcite . in this way, each token receives a single tag and the ambiguity property is lost altogether. for example, @xcite develop a pos tagging policy for historical german where ambiguous material receives the tag of the more frequent or of the historically older usage. another approach is to mark entities about whose interpretation an annotator is uncertain with a specific tag ('other/miscellaneous' category), signaling that no adequate annotation is available. for example, in the corpus of historical low german @xcite , clauses which are ambiguous between matrix (ip-mat) and subordinate status (ip-sub) are tagged as ip-x. a non-historical example comes from timebank-dense @xcite where ambiguous temporal relations are tagged as 'vague'. although this captures some level of ambiguity and uncertainty, it does not necessarily allow for an adequate representation of the underlying problem, information which is potentially of high interest to the end-user. a third option often employed is to leave uncertain material unannotated. in this case, the respective pieces of data are typically set aside and do not play a role in linguistic investigations or nlp downstream tasks. some efforts have been made towards more sophisticated schemes for explicitly marking representation problems. for instance, @xcite have developed a novel interface which enables an annotator to capture different sources of uncertainty while annotating pos and specific syntactic constructions in a historical corpus of middle low german (see also @xcite . their scheme allows annotators to capture uncertainty via a tag indicating one of three types: (i) category a is more likely than b, (ii) a and b are equally likely, (iii) unsure. @xcite has proposed a corpus annotation model which captures variation, e.g., in word pronunciation, by explicitly labeling the type of variation and the variant involved in an additional annotation layer. similarly, @xcite capture spelling variation in their historical tagset by providing information on three levels: (i) the original token (diplomatic level), (ii) a modernized version of the token (tokenization level), and (iii) a tag for the type of variation (tag level). likewise, @xcite have proposed a multi-level annotation approach for incomplete language phenomena, i.e., ambiguity, underspecification or uncertainty. in other contexts, uncertainty has been measured via various types of scale. for example, jurgens (2013) employs a likert-scale based approach for annotating ambiguous word senses, where the possible senses are rated on a numerical scale with respect to the likelihood of their occurrence in specific contexts. @xcite use confidence ratings ranging from 'not at all confident' to 'absolutely confident' when annotating temporal relations between pairs of events. @xcite let annotators specify whether an inference relation between sentences is 'very likely', 'likely', 'plausible', 'technically possible' or 'impossible' while annotating natural language inferences (nli). also in the context of nli, @xcite make use of a sliding bar, which contains numerical values which range from indicating that an inference relation is 'definitely true' to indicating that it is 'definitely not true' to capture human judgements. additionally, @xcite assess annotator disagreement as a measure of uncertainty by modeling these judgements as distributions instead of aggregated scores. similarly, chen et al. (2020) let annotators give judgements about the likelihood of inference relations via a sliding bar, and model 'uncertain natural language inferences' via scalar regression, predicting the probability at which a premise entails a hypothesis. @xcite , on the other hand, present a probabilistic model based on maximum likelihood estimation for annotating word senses, which gives a confidence estimate for each annotation label as a measure of certainty. @xcite capture uncertainty by measuring annotator disagreement on the basis of inter-annotator f 1-scores and the confusion probability between annotators. in addition, @xcite show that by incorporating information about annotator disagreement into the training of an nlp-model for pos-tagging, relevant downstream tasks, i.e., named entity recognition and chunking, can be improved. although these approaches take up the problem of annotation and representation problems, they do not work out a generally applicable framework. additionally, it is still unclear how the factors we identify as major problem sources interact throughout the various phases involved in corpus development and use. we see the important barrier to overcome here as a conceptual one, in the sense that concepts like 'uncertainty' and 'ambiguity' are often used interchangeably, despite there being inherent differences between the sources of these representation problems. understanding these sources on their own terms, as this paper proposes, is a crucial prerequisite for developing more adequate treatments and in turn more reliably annotated corpora. ## phases of corpus development and use developing an annotated corpus for linguistic research involves a number of work steps which are crucial for successful corpus design. the corpus development workflow consists of data selection and processing, typically including digitization, normalization and automatic pre-processing, and cycles of annotation. we see two major parts of a typical corpus development process: data selection and processing (phase i) and annotation (phase ii). following the corpus development, there is a third part, interpretation (phase iii), which pertains to corpus use, addressing the issue of interpreting the annotated data. ## sources of representation problems we identify five sources of representation problems in linguistic resources, which we explain in this section: (i) ambiguity, (ii) variation, (iii) uncertainty, (iv) error and (v) bias. they are relevant at all three phases of corpus development and use, and interact with one another in a complex fashion. ## research opportunities and open challenges the representation problems outlined above have real consequences in terms of the reproducibility, usability and trustworthiness of research results generated via annotated resources. it is thus important to treat the sources of these problems carefully, separating them out in corpus development and use. the aim is that outlining the different sources of representation problems, and how these surface in the various phases, will lead to a more nuanced understanding of the challenges involved. this in turn can inform future resources so that they are more faithful to the data they represent, increasing the end-user's confidence with respect to research results. as an initial step, we recommend that the various types of representation problems are properly identified in linguistic annotations and labelled explicitly. in this way, we hope to be able to capture and harness the full characteristics of corpus linguistic data in the future. our hope is that this will further lead to more robust corpus-based findings in theoretical linguistics and more accurate nlp models. specifically, we envisage new research opportunities in theoretical and computational linguistics, as well as novel responses in connection with the reproducibility crisis. facilitating theoretical linguistic research modeling representation problems provides us with a clearer picture of the underlying data, furthering our understanding of the linguistic and extra-linguistic properties of the texts in a corpus. this could lead to novel research results which were previously hindered by these problems, advancing the respective state-of-the-art in theoretical linguistics. furthermore, an explicit treatment of the problem sources could foster the emergence of new insights since, e.g., in historical linguistics, variation and ambiguity are seen as the key components of language change. improving nlp models in computational linguistics, propagating representation problems throughout nlp pipelines could inform computational models at each step, improving the accuracy of the respective algorithms and the resulting end-product. for example, it has been shown that the accuracy of nlp systems for event ordering can be improved by assigning specific tags in cases of ambiguity and uncertainty @xcite . similarly, @xcite have shown that providing information about annotator disagreements during training of an nlp model for pos tagging increases the performance of corresponding nlp downstream models. however, more recently, @xcite have shown that state-of-the-art nlp models for nli are able to model some sort of uncertainty, but this is not the uncertainty that stems from human disagreement. therefore, they advocate the need for a better understanding of the sources of linguistic uncertainty and the downstream propagation of such uncertainties in nlp models. providing an nlp model with a more elaborate and explicit treatment of representation problems thus has the potential to immensely improve research results. promoting reproducibility a framework which models representation problems could be of great benefit for research into the reproducibility issue in computational linguistics. @xcite define reproducibility in nlp as a property related to the outcomes of an experiment with the three reproducibility dimensions conclusion (an induction based on the results), finding (relationship between values), and value (a calculated or measured number). these dimensions also pertain to our described problem sources, since they have an impact on whether a conclusion, a finding or a value can be reproduced. @xcite moreover show that research into the reproducibility of experiments in nlp and in particular understanding the experimental variation can improve research results. ## conclusion in this paper, we presented five sources of representation problems in linguistic annotations: ambiguity, variation, uncertainty, error and bias. we characterized these sources, outlining their usual treatment in corpus linguistic processes. moreover, we discussed the consequences which an insufficient or improper treatment of these problems may have in the three phases of corpus development and use: data selection and processing (phase i), annotation (phase ii) and interpretation (phase iii). in this way, this paper highlights the importance of developing more adequate and explicit treatments of such representation problems in the future. moreover, we argued that harnessing representation problems in the scientific process fosters research into the reproducibility crisis, in addition to providing more robust research results.