Cheap Annotation of Complex Information: A Study on the Annotation of Information Status in G erman TED x Talks (2025)

Abstract: We present an annotation experiment for the annotation of information status in German TEDx Talks with the main goal to reduce annotation costs in terms of time and personnel. We aim for maximizing efficiency while keeping annotation quality constant by testing various different annotation scenarios for an optimal ratio of annotation expenses to resulting quality of the annotations. We choose the RefLex scheme of Riester and Baumann (2017) as a basis for our annotations, refine their annotation guidelines for a more generalizable tagset and conduct the experiment on German Tedx talks, applying different constellations of annotators, curators and correctors to test for an optimal annotation scenario. Our results show that we can achieve equally good and possibly even better results with significantly less effort, by using correctors instead of additional annotators.

## introduction information status concerns the way in which referents are referenced in a text: e.g. as a newly introduced entity (a nice picture), as a generally known entity (the sun), as a previously mentioned entity (she), etc. in language, information status is mainly reflected in the form of referring expressions, e.g. personal pronouns for a pre-mentioned entity or indefinite article for a newly introduced entity. investigating information status is a complex endeavor, as there exist various competing terminologies and classifications. in our work, we follow riester and baumann (2017) in their approach to the annotation of information status, applying the reflex scheme, an annotation scheme encoding detailed information on contextual and extra-textual givenness of referents. the scheme covers both the referential and lexical dimensions of information status. only the referential level is relevant to the work described in this study. this work is part of a larger project on word order in german, investigating the influence of information status and information-theoretical factors such as surprisal and information density (shannon, 1948). in particular, we are interested in the relationship between information status and information density. we therefore annotate data according to the reflex scheme. since the annotation of such a complex phenomenon requires expert annotators, it is rather costly in terms of time and personnel. hence, we aim to find a more economical solution to the commonly expensive annotation and curation of information status. in this paper we present the results of an annotation experiment that we conducted by testing various annotation scenarios for time and personnel efficiency as well as accuracy of the annotations. specifically, we compare the traditional approachmultiple annotation and subsequent curation, which is usually considered a guarantee of high annotation quality -with a simpler approach in which a single annotation is subsequently corrected. our results show that we can achieve equally good and possibly even better results with significantly less effort, by using correctors instead of additional annotators. ## related work linguistic annotation is a corpus-linguistic method with a long tradition, where quality control plays an important role. traditionally, the quality of annotations is measured using chance-corrected measures of inter-annotator agreement (iaa), also called inter-rater reliability (irr), such as fleiss' kappa or cohen's kappa (fleiss, 1971; cohen, 1960; carletta, 1996). these measures assume that two or more annotators annotate the same text independently of each other. another type of quality control is when only one annotator annotates the text and subsequently an expert annotator goes over these annotations and corrects them if necessary. in this case, the two versions -before and after correction -can be compared with each other applying measures such as f-score, measuring the accuracy of one version with regard to the other. it can be assumed that fewer errors will be detected with this method than with multiple annotations. for example, the two large german-language treebanks were annotated according to these two paradigms: the first method -double annotation -was applied to the annotation of the tiger treebank, the second method -annotation plus subsequent correction -to the annotation of tüba-d/z (dipper and kübler, 2017). grouin et al. (2014) evaluate the effect of differently-annotated types of training data (with double annotations, with a curated gold version, with an automatic pre-annotation that has been manually corrected) on the performance of a crf classifier. in contrast to our approach, the annotation quality as such is not compared and evaluated directly, but indirectly, based on the performance of the trained system. furthermore, in contrast to our experiment, they deal with a simple annotation task (identification of personal information in clinical documents). a number of papers compare the quality of annotation with vs. without automatic pre-annotation; for an overview see, e.g., mikulová et al. (2022). in contrast, we do not use automatic pre-annotation in our study. ## the data the fragments that we annotated are extracts from the transcriptions of a total of five tedx talks which were given in german on a range of different topics. the texts are subject to licenses that permit free redistribution. @xcite from each talk, we annotated 100 referential expressions from two different sections of the talk, resulting in 10 fragments with 1,000 annotated units in total. @xcite we chose tedx talks for the annotation experiment as we considered them an adequate cross-section of content, while keeping the genre of the data -semi-scripted oral talks -constant. we excluded talks that involved particularities as for example rap, for this would distort the homogeneity of the dataset too much. all annotations, curations, and corrections were created and handled in the annotation tool incep-tion (klie et al., 2018). details of the procedure are provided in the following sections. ## annotation guidelines we base our experiments on the annotation of information status according to the reflex scheme proposed by riester and baumann (2017). reflex is a comprehensive annotation scheme that provides a total of 12 different labels, which can be divided into 7 classes, see in the following we describe the modifications we have made to reflex. label names among other things, we have shortened the label names for the annotation. first, we omit the prefix 'r-' from all labels. @xcite second, we replace some of the longer names by short ones, see markables we define admissible markables as follows: a markable is either an np (or pp, as specified in reflex), a possessive pronoun or a deictic adverbial (hier 'here', jetzt 'now'). for complex phrases with embedded phrases, relative clauses or appositions, we annotate (i) the entire phrase (i.e. its head) and (ii) each of the embedded phrase(s). idioms are annotated as an entire span. foreign language material is not considered, except for when it is referred back to. incomprehensible passages, e.g. due to spelling mistakes or transcription errors, are ignored. ## experiments annotation and curation of linguistic resources is time consuming and costly, especially in the case of a complex phenomenon like information status and a detailed tagset such as the reflex scheme. to keep annotation costs minimal, we conducted an annotation experiment to test for an optimized annotation mode, which allows for minimal costs in resources and maximal accuracy. we assumed that the expenses of the usual annotation and curation process, involving multiple annotators and curators, could be reduced significantly by installing different settings of annotation while maintaining a reasonable accuracy and therefore quality of the annotated data. to test this, we set up various annotation scenarios in different personnel settings and tested for time and staff 'costs' in relation to the resulting annotation quality. there were four expert annotators (the authors) involved in the experiment. before running the experiment, the annotators annotated and curated several passages in two training datasets for annotation training. all annotators were also involved in the fine-tuning of the annotation guidelines. after the training phase, the guidelines were finalized. then the experiment was conducted. all annotations, curations, and corrections where created and handled in the annotation tool inception (klie et al., 2018). correcting annotations figure 2 uses example (4) to show how we have implemented the correction steps in inception. the annotations shown in green are those of the annotator. the labels in yellow and purple come from two correctors. for the correction steps, new layers (with new colors) were created in inception, with the same labels as the original annotation layer plus an additional label remove (see below). the correctors could only see the original annotations of one annotator and not the corrections of the other corrector. (4) diese eigenschaften möchten wir für etwas positives nutzen. figure 2 , left part, shows that the two existing annotations of example (4) were found to be correct by both correctors, so they didn't change anything. however, the phrase for something positive 'for something positive' was not considered by the annotator. both correctors (shown in yellow and purple) have re-annotated this phrase. @xcite removing an erroneous annotation of correcting the extent of an annotation span is a special case in the correction process. for this case, a new label remove is employed, which is used to mark the incorrect span. a new correct span including a label is added, if needed. figure 2 , right part, shows the annotation of example (5). the original annotator did not include the preposition als 'as' in the span, which has been corrected accordingly by the corrector (shown in yellow). set 2 secondly, only three of the annotators annotated and curated 2 × 100 annotations, and a single corrector corrected the annotations of one of the three annotators per batch. set 3 the last setting involved two annotators annotating and curating the gold version and the other two both correcting the same single annotation per batch, but independently from each other. in total, 3 × 100 annotations were annotated, curated and corrected in this setting. the gold versions were created by the annotators themselves in a joint discussion round. this means that the gold versions are certainly influenced by the existing annotations, but this is trivially true for every gold version that is created on the basis of existing annotations. the correctors did not participate in the curation. they only saw one of the annotations and corrected this annotation. they had no access to the other annotations or to the gold version. so the relevant question is: can the correctors arrive at a similarly high-quality "gold" result as the curators? since a correction is significantly cheaper than a curation (requires less time and personnel), this would save a lot. in order to make the two basic scenarios -multiple annotation followed by curation on the one hand vs. single annotation followed by correction on the other -as comparable as possible, the correction is based on one of the annotations that is also used to create the gold version (as one of several annotations). ## results to evaluate the quality of the various annotation scenarios, we use two different measures: fleiss' kappa as a measure of inter-annotator agreement and f 1 -score as a measure of the annotators' and distance between annotations and gold next, we examined how far the individual annotators were from the curated gold version. we calculated this distance in the form of aggregated f-scores across all annotated text fragments per annotator, see for us, a highly relevant question is how far away the results from the different tasks are from @xcite in our view, chance-corrected measures such as fleiss' kappa are not applicable to the other scenarios because it is to be expected that the gold version as well as the corrector's version are biased by the given annotations and therefore the assumptions concerning chance agreement are no longer correct. annotator labels (f 1 ) spans 6 ). question (ii) is addressed next. ## conclusion we set out to investigate various annotation scenarios and their respective efficiency in terms of time and personnel employed and conducted an annotation mode experiment where we compared the scenarios of (i) four annotators and four curators, (ii) three annotators and three curators tested against a single corrector and finally (iii) two annotators and two curators tested against two correctors. as has been shown in section 6, the f-scores for span and label identification of the correctors not only stayed constant compared to the annotator fscores, but even exceeded those annotators' values while reducing the total time of the entire annotation process approximately by half, even when considering the control curation condition in this calculation. we therefore argue that the third scenario of annotating and correcting is preferable to the conventional annotation and curation setting not only in terms of time and personnel, but also in terms of annotation quality, as the corrections closely match the gold version as can be inferred from the respective f-scores. we could thus show that time-efficient annotation -even in the case of highly complex tagsets such as the reflex tagsetdoes not necessarily need to come at the traditionally high annotation cost.