The distribution of discourse relations within and across turns in spontaneous conversation (2023)

Abstract: Time pressure and topic negotiation may impose constraints on how people leverage discourse relations (DRs) in spontaneous conversational contexts. In this work, we adapt a system of DRs for written language to spontaneous dialogue using crowdsourced annotations from novice annotators. We then test whether discourse relations are used differently across several types of multi-utterance contexts. We compare the patterns of DR annotation within and across speakers and within and across turns. Ultimately, we find that different discourse contexts produce distinct distributions of discourse relations, with single-turn annotations creating the most uncertainty for annotators. Additionally, we find that the discourse relation annotations are of sufficient quality to predict from embeddings of discourse units.

## introduction discourse relations (drs) such as elaboration, background and explanation, hold between discourse units contributing to the coherence of a text. annotation of discourse relations has received attention for its relevance to discourse parsers, with applications in question answering systems (e.g. @xcite ), text summarization (e.g. @xcite , sentiment classification (e.g. @xcite , and machine translation (e.g. @xcite . however, most of the annotated data and systems have focused on written language, with a few exceptions (e.g., @xcite @xcite . in spoken dialogue or multiparty conversation, participants must quickly juggle a variety of tasks, such as responding to another person to solve a problem @xcite or negotiating the question under discussion @xcite , often under considerable time pressure that is less present in written production. in addition to these time demands, it is unclear whether spon-taneous conversation demonstrates the same patterns of discourse relations as observed in written language (see @xcite , for a discussion of spoken vs. written use of discourse markers). perhaps unsurprisingly, the vast majority of work on discourse relations has focused either on written texts, especially news text @xcite @xcite , or highly structured conversations that are constrained by a particular game @xcite . some recent corpora contain spoken monologues @xcite , and spoken conversations @xcite , but the field still largely lacks annotated corpora of spontaneous dialogue. thus, our goal is to present the first efforts towards an annotated corpus of drs for spontaneous spoken conversation, with particular attention to relations across different contexts within a conversation. we analyze the patterns of dr annotation within and across speakers and within and across turns and test the coherence of annotators' decisions. ## related work most currently available corpora annotated with drs have focused on written language or spoken monologues. an exception is the georgetown university multilayer (gum) corpus @xcite , which has a set of conversations annotated within rhetorical structure theory @xcite , following the guidelines of the rst discourse treebank (rst-dt, @xcite . but it is an open question whether the drs that have been identified for news texts are appropriate for conversational data. @xcite adapt the pdtb framework to annotate a subset of a corpus of italian conversations about software and hardware troubleshooting, and suggest modifications to the framework to account for spoken data. discourse relations corpora have usually been annotated by experts, but some recent corpora have been annotated by novice annotators, such as university students, in the case of the gum corpus @xcite , or crowdsourced workers, in the case of the discogem corpus @xcite . gum was annotated using rst as part of a corpus linguistics class, while discogem was annotated following the penn discourse treebank (pdtb, @xcite ) framework, using a method for crowdsourcing annotations introduced in @xcite , and using a multi-label approach. the present work deviates from prior work in its focus on conversational data and the use of segmented discourse representation theory (sdrt, @xcite alongside the stac corpus @xcite guidelines. ## discourse relation annotation in this work, we focus on a subset of 19 dialogues from the switchboard corpus @xcite . this corpus contains informal language and has been the subject of study of numerous analyses of dialogue within linguistics @xcite . in it, two strangers are presented with a topic (e.g., childcare) that they must discuss with each other, but the dialogues are otherwise not tightly constrained. annotating switchboard will provide us with a more complete understanding of the use and generality of discourse relations across linguistic contexts and genres. following the annotation procedure in the stac corpus @xcite , we identified a subset of suitable elementary discourse units (edus) for annotation by parsing each turn into a dependency structure and included only those turns with at least two roots or verbs. then, we segmented each of these turns into their respective edus. using these segmentations, we identified edu candidates for discourse relations that were either within-turn (same speaker) or across two turns (different speakers, or the same speaker), where the two turns were adjacent in the case of different speakers, or only interrupted by one turn, in the case of same speaker. we provide a representative set of these pair types in ## uncertainty in the annotation of discourse relations different edu pairs in the present annotation task were drawn either from the same turn, or across turns but within or across speakers. thus, we can assess how much discourse relations vary by the placement of an utterance in a dialogue. given the complex dynamics in dialogue, we expect to find significant differences in discourse relation use across different discourse contexts. we visualize the distribution of the relations in figure 1 . annotators generally selected more discourse relations per edu pair in the single-turn case, with an average of 8.16 relations per team or 1.60 per annotator. when edus spanned turns within a single speaker, groups selected significantly fewer relations (@xmath0.29, t(302) = -2.16, @xmath1.05). groups likewise selected even fewer relations for edus between two speakers (@xmath2.51, t(314) = -2.54, @xmath3.05). on its face, this pattern appears surprising, because it suggests that annotators find more relations appropriate for single-speaker productions. however, an alternative interpretation of these results is that annotators may instead have been uncertain about the distinctions between the different discourse relations. this second interpretation is corroborated by post-hoc poll data from 35 annotators, of whom 32 (91.4%) stated that the selection of discourse relations was best suited to annotating cross-speaker edu pairs. future work will require recruiting greater numbers of annotators to be able to distinguish between these two hypotheses. ## discussion in two experiments, we demonstrated that novice dr label annotations in a single turn are more difficult than across turns. we showed that including discourse context (within/across speaker and within/across turn) to a logistic regression model significantly improves fit to our annotation data. a classifier trained to predict dr labels from embeddings of (π 1 , π 2 ) pairs showed modest success for recall of any of the annotations, but poor precision and recall overall. a comparison of this classifier's predictions and annotators' gold label distributions revealed greater uncertainty for the annotation of discourse relations within a single turn. these results demonstrate that different conversational contexts are associated with different distributions of discourse relations. the uncertainty of choice of discourse relations within a turn may be due to several factors. drs that typically occur across adjacent turns and across speakers (e.g., acknowledgements) might have clearer signals. at the same time, drs that occur more frequently within speakers, and, in particular, within a turn, might be more ambiguous, or might co-occur with other relations. more work is necessary to disentangle uncertainty about the identity of the best fit relation from whether multiple relations are appropriate.