Incorporating Annotator Uncertainty into Representations of Discourse Relations (2023)

Abstract: Annotation of discourse relations is a known difficult task, especially for non-expert annotators. In this paper, we investigate novice annotators’ uncertainty on the annotation of discourse relations on spoken conversational data. We find that dialogue context (single turn, pair of turns within speaker, and pair of turns across speakers) is a significant predictor of confidence scores. We compute distributed representations of discourse relations from co-occurrence statistics that incorporate information about confidence scores and dialogue context. We perform a hierarchical clustering analysis using these representations and show that weighting discourse relation representations with information about confidence and dialogue context coherently models our annotators’ uncertainty about discourse relation labels.

## introduction discourse relations (drs) are those relations such as elaboration, explanation, narration, which hold between discourse units. the task of labeling drs is known to pose difficulties for annotators @xcite , as sometimes more than one interpretation may be possible @xcite . recent studies have shown that allowing for multiple labels in annotation can improve the performance of discourse parsers @xcite . @xcite test different label aggregation methods in a crowdsourced corpus annotated by 10 workers and find that probability distributions over labels better capture ambiguous interpretations of discourse relations than majority class labels. (1) shows an example from their corpus, where the relation between the second and third sentences (in italics and bold, respectively), was interpreted as conjunction by four annotators and result by five annotators. in this work, we investigate which relations are distributionally similar or co-occurring in multilabel annotations of spontaneous conversations. we are particularly interested in how novice annotators interpret discourse relation categories when annotating spoken conversational data. we collect annotations of drs from switchboard telephone conversations @xcite , allowing for multiple labels, and ask for confidence scores. we find that confidence scores vary significantly across dialogue contexts (single turn vs. pairs of turns produced by the same speaker vs. pairs of turns produced by different speakers). we incorporate information about these three dialogue context types and confidence scores into distributed representations of discourse relations. a clustering analysis shows that discourse relations that tend to occur across speakers cluster together, while discourse relations which tend to occur within a speaker, either in the same turn or different turns, form their own cluster. ## annotation of discourse relations our analyses are built on the dataset collected in lópez cortez and jacobs (2023), who selected 19 conversations from switchboard foot_0 , a corpus consisting of telephone conversations between pairs of participants about a variety of topics (e.g. recycling, movies, child care). we chose this corpus because it contains informal, spontaneous dialogues, and because it has been used within linguistics in various studies on conversation @xcite . ## dialogue context as a predictor of confidence scores first we sought to understand how discourse relations and dialogue context (as defined above) influence annotator confidence. because our confidence ratings data has multiple observations for each annotator, each team and each du, it is hierarchical and thus benefits from being analyzed using hierarchical mixed effects models. due to the ordinal nature of the ratings data, we use the cumulative link approach (clmm; liddell and kruschke, 2018; howcroft and rieser, 2021) rather than model confidence scores as real-valued in linear regression. we first built a null model containing only random intercepts by annotator and compared it to a model containing an additional fixed effect and random slope by annotator for dialogue context type: single turn, across turns within speaker and across speakers (kind, dummy coded). a likelihood ratio test revealed a significant improvement in fit by adding kind as a predictor (χ foot_1 (7) = 126.64, @xmath0.001). adding random intercepts for du pairs to account for annotation difficulty across du pairs also led to a significant improvement in model fit beyond the model containing dialogue context kind (χ 2 (1) = 195.01, @xmath1.001). this suggests that our annotators' confidence scores are sensitive to the context of du pairs. figure 1 shows mean confidence scores per context kind across discourse relations. confidence scores within a speaker both across and within turns received similar confidence ratings (β = -0.13, @xmath2.56, @xmath3.s. 2 ), while annotators were significantly more confident for relation annotation across speakers (β = 0.63, @xmath4.05, @xmath5.01). the clmm revealed that annotators used confidence scores between 3 and 5 overall, except for the label "other", for which they selected lower confidence scores. background received lower confidence scores overall. continuation, contrast and narration received higher scores for contexts within speaker. comment and result received higher scores for turns across speakers and single turn. for elaboration and explanation, mean confidence scores are very similar across the three contexts, with slightly higher scores for single turn and pairs of turns within speaker. acknowledgment, clarification question ("clarificationq") and question-answer pair ("qap") received higher scores for turns across speakers, which makes sense given the dialogic nature of these relations. however, these relations also received rather high confidence scores for single turn and pairs of turns within speaker, which is a bit surprising. we suspect this might be due to the context turns included for each pair of dus, which might have led annotators to choose relations between discourse units other than for the pair of highlighted dus. future analysis will look closer at this aspect. ## distributed representations from discourse relation annotations to model the similarity between discourse relations as perceived by annotators, we computed embedding representations of discourse relations. we extracted each n individual annotation containing relation-confidence (r, c) tuples selected by a given annotator for a pair of dus. we concatenate bagof-relation vectors with one-hot encoded features representing the dialogue context kind, and multiply the count vector of annotated relations (either 1 or 0 for each relation) by the confidence score (1-5) for that pair of dus. this weighting learns more from high confidence; an ideal reweighting may be possible with additional parameter search, possibly in conjunction with the clmm outputs. for an n × 1 confidence ratings matrix c, an n × 12 bag-of-relations matrix r, an n × 3 discourse context matrix d for each annotation, we obtain an annotation matrix @xmath6. we then obtain a square co-occurrence matrix o such that @xmath7, which we factorize using principal component analysis (without shifting the intercept following @xcite . each relation is thus represented as a vector that consolidates co-occurrences between all relations within a single annotator that are weighted by confidence score. we then projected these embeddings into two dimensions with umap @xcite and performed a hierarchical clustering analysis over the resulting coordinates due to the greater discriminability afforded by continuous distance metrics. informally, the umap coordinates appear more gradient in the representational space when confidence was included (right panel) than when it was not included (left panel). when context is not included, the umap coordinates primarily represent the frequency of labels in our corpus, which we include in appendix a. we visualize the umap coordinates in figure 2a . figure 2b shows a dendrogram with the output clusters, colored according to the optimal number of clusters (@xmath8, calculated using average silhouette widths @xcite . there are two large clusters, one of which contains two sub-clusters with background and continuation, on the one hand, and elaboration and explanation on the other. in the other large cluster, acknowledgement and comment form a sub-cluster. these are very common relations between pairs of turns across speakers. clarification question and question-answer pair form another sub-cluster, also common relations between pairs of turns across speakers, in close proximity to the other label, which received a sub-cluster of its own. narration and contrast and result, form the last sub-clusters, which we suspect is due in part to the frequencies of these relations @xcite . we include a dendrogram with the output clusters of a hierarchical clustering analysis performed with base bag-of-relations vectors (without context kind and confidence scores weight) in figure 3 in appendix b for comparison. currently, we provide these results as a proof of concept of the feasibility and interpretability of noisy labels produced by novice annotators. importantly, annotations weighted by confidence produce coherent clusters of discourse relations. we envision applications of dr embeddings to several domains including dialogue generation, such that appropriate responses to input are partially conditioned on a latent or mixed combination of drs. ## related work annotation of discourse relations is usually done within rhetorical structure theory @xcite , as in the rst-dt @xcite and gum @xcite corpora, within segmented discourse representation theory (sdrt, @xcite , as in the stac @xcite and molweni @xcite corpora, or within the penn discourse treebank framework @xcite @xcite . we use a taxonomy adapted from sdrt, in particular, the stac corpus. annotators are usually trained to identify discourse relations using the framework's taxonomy. some recent alternatives to explicitly collecting annotation of drs include crowdsourcing by eliciting connectives @xcite or question-answer pairs @xcite rather than relations. in this work, we wanted to investigate how annotators perceive discourse relation categories, and therefore a connective insertion task would only provide indirect evidence. we train annotators on dr labeling and ask annotators to choose from a set of discourse relation labels. we allow for multiple labels to investigate what relations are more confusable or perceived as co-occurring @xcite . ## discussion and future work in this study, we collected multiple annotations of discourse relations from a subset of the switchboard corpus, together with confidence scores. we found that dialogue context had a significant effect on confidence scores. we computed embedding representations of drs using co-occurrence statistics and weighted the vectors using context type and confidence scores, and found that these representations coherently model our annotators uncertainty about discourse relation labels. discourse units that occur across turns as defined by switchboard do not necessarily occur across continuous utterances from the speaker's point-of-view. obtaining information about whether samespeaker pairs of discourse units fall into the same or different utterances may help to explain additional variance in annotator confidence. additionally, in this work, we investigated annotators' confidence on the annotation of adjacent turns. in future work, we plan to annotate discourse relations across longer-distance discourse units and to allow for hierarchical annotation. we expect that annotation confidence will also vary across longer-distance units and across different depths of annotation. in the future, we plan to use this information to run a larger scale annotation study of the switchboard corpus to analyze discourse relation patterns in spoken dialogues.