Disagreements in analyses of rhetorical text structure: A new dataset and first analyses (2025)

Abstract: Discourse structure annotation is known to involve a high level of subjectivity, which often results in low inter-annotator agreement. In this paper, we focus on “legitimate disagreements”, by which we refer to multiple valid annotations for a text or text segment. We provide a new dataset of English and German texts, where each text comes with two parallel analyses (both done by well-trained annotators) in the framework of Rhetorical Structure Theory. Using the RST Tace tool, we build a list of all conflicting annotation decisions and present some statistics for the corpus. Thereafter, we undertake a qualitative analysis of the disagreements and propose a typology of underlying reasons. From this we derive the need to differentiate two kinds of ambiguities in RST annotation: those that result from inherent “everyday” linguistic ambiguity, and those that arise from specifications in the theory and/or the annotation schemes.

## introduction natural language contains many ambiguities with varied possible interpretations, especially in the domains of pragmatics and discourse. the differences and similarities of annotations from individual coders, the inter-annotator agreement (iaa), is often used to demonstrate that annotation guidelines are effective, the annotators have worked in a precise way, and that overall, the annotations are of a high quality. in recent years, however, the instances of disagreement have gained interest as a resource for more informative models of the underlying task, often under the heading of 'perspectivism' @xcite . in this study, we focus on the annotation of discourse structure using rhetorical structure theory (rst; @xcite . rst annotations provide information about how segments in a text are related to each other with semantic or pragmatic relations such as cause, background, or contrast; we give a brief overview in sct. 2.1. with its focus on pragmatic aspects of language use, rst annotation is generally considered to be highly subjective, and as discussed by @xcite , disagreement in alternative annotations can reflect either incorrect annotations ormore interestingly -instances of item ambiguity or of inherent task subjectivity. so far, empirical studies on annotator disagreement in rst (and also for similar frameworks) have been scarce, as we show in sct. 2.2; one reason is probably the fact that comparing entire tree structures as alternative analyses is a relatively complicated undertaking. to make it more effective, in this paper, we utilise the rst-tace software @xcite to compute the individual points of disagreement between two annotators, which we then analyse further. we use a dataset of english and german corpora that have recently been made available and partly were extended by us with a secondary annotation (see sct. 3), and we add to this the double-annotated part of the english rst discourse treebank @xcite , which to our knowledge has so far not been analysed for the reasons of the disagreements. for these corpora, we manually inspect a motivated subset of the points of disagreement and build a typology of categories for legitimate alternative analyses. our results have multiple implications. firstly, they provide insights into the variability of discourse structure, as it is comprehended by different annotators. secondly, our results can lead to improvements on the rst annotation process, with guidelines being made more precise and annotators being made aware of areas of particular difficulty. thirdly, our disagreement data and typology can be used to improve evaluation methods of discourse parsers and provide inspiration for evaluation of other similarly subjective tasks. in sct. 2 we give a brief overview of rst and outline previous work that has looked at annotation disagreement, and in sct. 3 we introduce the composition of our dataset. sct. 4 explains rst-tace (henceforth: tace), which provides us with the starting point for our analyses that we present in sct. 5. in sct. 6 we discuss these results, before sct. 7 concludes and outlines possible avenues for future work. corpora. for english, the rst discourse treebank (rst-dt; @xcite was introduced in 2003; it is based on annotation guidelines by @xcite , where the size of the relation set has been increased to 78. a part of the corpus comes with two annotations and will be part of our dataset (see sct. 3). a second important english corpus is gum @xcite , which is being continuously extended with new data and also with new annotation layers. the annotation guidelines of rst-dt and gum differ in terms of edu characterisation and relation set, so that the corpora are not immediately comparable. a smaller english corpus that was recently released contains speeches from the un security council @xcite . a part of that has two distinct rst analyses, and these will also be used in our study. for german, a collection of rst data was recently made available by @xcite . a part of that material is doubleannotated and will be used in our analyses. this data, as well as the unsc data, were annotated according to the guidelines by @xcite . ## the corpus overall, the corpus used in this study consists of 156 texts in english and german, coming from four sources. all texts have two annotations that were produced by well-trained annotators, and the pair always features identical edu segmentation. this makes a systematic disagreement analysis much easier, and it reflects an annotation procedure convention to separate the segmentation process from the tree building step. (but see our remark in the limitations section at the end.) the english texts are from the rst-dt @xcite and the unsc-rst corpus @xcite . the texts in the rst-dt are articles from the wall street journal from the late 1980s. we use a subset of the corpus which consists of texts having two annotations that are based on identical segmentation. the unsc-rst corpus contains transcripts of speeches from the un security council in the years 2014/15, and we work with its doubly-annotated subset. the german-language data consist of the doublyannotated subsets of the apa-rst corpus, which are newspaper articles and their manual simplifications into 'easy language' @xcite , and of the potsdam commentary corpus (pcc), which collects commentaries from local newspapers @xcite . five different trained annotators created the analyses of the apa-rst texts, and there was a followup step that corrected obvious errors or violations of the schema. the same procedure was applied in unsc-rst, with a team of four annotators. two well-trained annotators were involved in building the pcc subset, and also at the time in producing the rst-dt. since the two german corpora are based on the same annotation guidelines, we fuse them into a single set that we call apa+pcc. unsc-rst had the same guidelines but is in english; the rst-dt features a much more fine-grained relation set and hence different guidelines. we thus have three subcorpora for which disagreements can be analysed, but cross-corpus comparisons have to keep in mind the differences. for instance, the pcc/unsc-rst guidelines were conceived for opinionated text, with the goal of supporting argumentation analysis. hence they distinguish between the relations evidence, reason and cause with different constellations of objective/subjective material. the rst-dt uses many relations that are absent in the pcc/unsc-rst, such as six finegrained versions of elaboration, or the relations topic-shift and example. (a proposal for mapping between the relations sets was made as part of a shared task on rst parsing @xcite .) statistics on our corpus size can be found in ## mapping out the disagreements: rst-tace we use tace @xcite on our corpus to compare the pairs of plausible annotations. tace takes two rst annotated texts as input, which have identical segmentation, and produces a table comparing the two annotations. tace calculates iaa using four different aspects: nuclearity (n), relations (r), constituents (c) and attachment points (a), based on a proposal by @xcite . a constituent is the satellite span, the attachment point is the span which the constituent is linked to. pairs of annotated units are matched according to the overlap between central subconstituents (cs); the nuclear units of the satellite of the relation above, or the satellite if the relation is between two edus. in figure 1a , for the e-elaboration relation spanning the edus 1 and 2, the constituent is 2, the attachment point is 1, and the cs is 2. based on the type of mis/match between the two annotators, we create five bins of "annotation decivi ii i iii iv v figure 2: two parallel extracts from example annotations to illustrate different versions of 'scope mismatch'. sions" that can be extracted from tace's output foot_4 , in the form of a spreadsheet where each row contains inter alia the edu numbers participating in the annotation decision, the actual text spans, and the relations assigned by the annotators. we illustrate the bins with examples from figures 1a, 1b and 2: 1: perfect match -annotators analysed two units in the same way. example: the attribution relation in fig. 1 constitutes a perfect match. 3: scope mismatch -annotators disagree on the scope of a relation. this comprises six different constellations: (i) identical overall span; identical relation; different split points; (ii) different overall spans; identical relation; identical split point; different argument spans; (iii) different overall spans; identical relation; one identical argument span; (iv) different overall spans but one common end point; identical relation; different split point, different argument spans; (v) identical overall span, different relations, different split points; (vi) different over-all spans, different relations, identical split point, one identical argument span. example: the elaboration relation that encompasses the edus 1 to 5 in fig. 1 belongs to the category 3(i). all cases of scope mismatch can be seen in ascending order from top to bottom in 5: no match -decisions of the first annotator that are not matched at all by the second annotator. ## analysis ## discussion our findings on disagreements confirm and extend those of @xcite , and provide a much larger dataset for further study. we also find that the ambiguity of coreferential expressions or attributive verbs lead to scope mismatches in parallel annotations, while on the annotator level the perception of importance can lead to relation mismatches. these sources of ambiguities are not specific to rst annotation but a fact of language use, and they connect to earlier findings that implicitness -the lack of an overt signal clearly associated with a specific relation -leads to more disagreement @xcite . this is of particular relevance to automatic discourse parsing and led to the emphasis on signal annotation in erst @xcite . ambiguity that is inherent in language, however, needs to be kept distinct from aspects of the theory and the annotation guidelines that create some undesirable choice points for annotators. our observations on the interaction between perception of importance and nuclearity assignments on all levels of the tree reinforces the concerns stated by @xcite , who pointed out that the strong nuclearity principle -and the degree to which annotators rely on it -leads to an inherently unclear notion of the argument of a coherence relation in an analysis. 'perception of importance' is inherently subjective, like the ambiguities discussed above, but it should not propagate to an array of other annotation decisions and cause additional variability in the structures of longer texts. a large number of disagreements that we classified as due to vagueness result from this. the second important source for them is the routine applicability of multiple relation definitions to a given text span. our 'status' categories distinguish vague from both, where the former may to some extent be curable by clearer relation definitions, while the latter corresponds to the situation where an annotator should have the option to in the first place assign two relations rather than one. the erst approach offers this, though only in the presence of overt signals; it can be worthwhile to investigate annotators' behaviour if it would also be allowed in implicit contexts. in addition, other forms of underspecification (of the scopes of certain relations) could be a way of reflecting actual vagueness from the viewpoint of an annotator. offering annotators the means to make their uncertainties transparent requires a revised model of discourse structure, and still we will usually work with multiple annotators, so that their potentiallyunderspecified representations need to be compared in systematic ways to one another. in addition, the consequences for machine learning in discourse parsers and for their evaluation need to be con-sidered -all aspects of perspectivism need to be attended to. ## conclusions this is the first study of rst annotation disagreement that uses a sizeable english/german dataset with two alternative trees, which (except for the rst-dt) we also make publicly available. we have proposed a method for systematically studying the disagreements in three steps of analysis: (i) a formal analysis that extends the output of tace and builds a list of individual points of disagreement between the annotators. (ii) an evaluation of the status of these disagreements. (iii) a typology of reasons for these disagreements. using parts of our corpus -480 instances of disagreements in total -we undertook a first qualitative analysis in this way, and then discussed some implications for potential improvements of annotation guidelines and for incorporating uncertainty into the annotation process.