Comparing Methods for Segmenting Elementary Discourse Units in a F rench Conversational Corpus (2023)

Abstract: While discourse parsing has made considerable progress in recent years, discourse segmentation of conversational speech remains a difficult issue. In this paper, we exploit a French data set that has been manually segmented into discourse units to compare two approaches to discourse segmentation: fine-tuning existing systems on manual segmentation vs. using hand-crafted labelling rules to develop a weakly supervised segmenter. Our results show that both approaches yield similar performance in terms of f-score while data programming requires less manual annotation work. In a second experiment we play with the amount of training data used for fine-tuning systems and show that a small amount of hand labelled data is enough to obtain good results (although significantly lower than in the first experiment using all the annotated data available).

## introduction discourse parsing is the decomposition of texts or conversations in functional units that encode participants intentions and their rhetorical relationships. segmentation in these units is the first step for other levels of analysis, and can help downstream nlp tasks. discourse parsing involves determining how each part of a discourse contributes to the discourse as a whole-whether it answers a question that has been asked, provides an explanation of something else that was said, or signals (dis)agreement with a claim made by another speaker. the first step, then, is to decompose a discourse into minimal parts that can serve such discursive functions. we will use the term elementary discourse unit (edu; @xcite to designate a minimal speech act or communicative unit, where each edu corresponds roughly to a clause-level content that denotes a single fact or event. while edu segmentation of written documents has received a lot of attention from the discourse and nlp community, this is less true for segmentation of conversational speech. conversational data has mostly been approached from either (i) a dialogue act segmentation and tagging perspective, and usually on rather taskoriented dialogues @xcite or (ii) punctuation prediction to enrich transcripts obtained with automatic speech recognition @xcite . we assume that this situation encourages a bias toward written genres that can be problematic for discourse segmentation, because those genres (newspapers, literature,...) tend to include long complex sentences, while actual conversations are made of relatively short contributions. non-sentential units @xcite , which often consist of only a single word, are extremely frequent in conversation and can convey full communicative acts, such as an answer to a question or a communicative feedback, that are crucial for modeling discourse structure. in this paper we benefit from a fully segmented corpus, the corpus of interactional data (cid; @xcite , for running a set of experiments on discourse segmentation. this data set is challenging as it consists of 8 long conversations (1 hour each) alternating between interactive narrative sequences like (1) (with a clear dominant speaker holding long turns made of many discourse units) and more dialogical sequences like (2) (made of very short, very often incomplete, turns that sometimes need to be grouped together to form a valid discourse unit). ## gold edu segmentations the corpus of interactional data (cid) (8 dyadic conversations, 1 hour duration for each) @xcite was segmented following guidelines designed for written documents @xcite that were modified for spoken conversational data. these guidelines thus combine semantic and discourse criteria (used in particular in monological sequences like (1)) with dialogical and interac-tional ones (that are more useful in dialogical sequences like (2)). the cid displays highly spontaneous data with colloquial sequences like (3) or strong disfluencies like (4) making discourse segmentation a much more difficult task than on written genres, even for humans. the whole data set consists of about 125 000 tokens for 15,463 discourse units (12,4% of the tokens are edu boundaries). edus are obtained from at least two manual annotations (obtained from 4 naive coders and 2 experts). the mean cohen's κ-score across speaker for naive coders is 0.85 (min: 0.83 ; max :0.87). annotations were performed with praat @xcite in order to have access to signal word-alignment when making segmentation decisions. the discourse annotations @xcite are available from ortolang platform : https://www.ortolang. fr/market/item/ortolang-000918 . (3) a: [comme ça # ah ouais non c'était] du a: [like that # oh yeah no it was] du b: [ah ouais profitez profitez de vos soirées] du b: [oh yeah enjoy enjoy your evenings] du a: [ouais c'est pour ça] du a: [yeah it's for that] du (4) [ou des euh non pas des f-pas des frustrations] du [des # espèces de euh # mhm # ouais des des vues différentes sur le boulot quoi] du [or some uh no not some f-not some frustrations] du [some kind of uh # mh # yeah some some different views about work what] du 4 method in this work we use the existing implementation of tony (muller et al., 2019) and that of gravellier et al. (2021), called tony-w(ritten) and tony-s(poken), respectively, as baselines. our first experiment consists in comparing a supervised model (finetuning 'tony' baselines with our annotated data) against the weakly supervised dataprogramming approach. in a second experiment, we explore the impact of the amount of data used for fine-tuning the models. transcripts from the cid do not include any name polarity coverage overlaps conflict correct incorrect accuracy tony_written 0,1 1.000 0.976 0.0632 102446 8994 0.919 no_pause 0 0.898 0.898 0.0474 94706 5415 0.946 long_pause 1 0.054 0.054 0.0094 5353 683 0.887 very_long_pause 1 0.032 0.032 0.0051 3399 162 0.955 extreme_pause 1 0.022 0.022 0.0037 2357 48 0.980 pause_begin_pos 1 0.042 0.042 0.0101 4362 328 0.930 pause_ending_pos 1 0.036 0.036 0.0097 3338 652 0.837 non_ending_tok 0 0.323 0.323 0.0009 35579 365 0.990 pause_begin_tok 1 0.055 0.055 0.0111 5533 617 0.900 pause_ending_tok 1 0.005 0.005 0.0005 550 22 0.962 dm_bi_ini 1 0.012 0.012 0.0059 1146 222 0.838 non_begin_tok 0 0.005 0.005 0.0001 530 9 0.983 feedback_cluster 0 0.026 0.026 0.0012 2763 87 0.969 repeat 0 0.088 0.088 0.0124 8678 1141 0.884 filled_pause 0 0.071 0.071 0.0055 7428 570 0.929 truncated_word 0 0.016 0.016 0.0013 1629 169 0.906 ## conclusion and discussion in this work we have compared different approaches for building a discourse unit segmenter adapted to french conversations. we had access to a manually segmented corpus of significant size which allowed us to perform a wide range of experiments. first we compared the option of (i) using our conversational data set to fine-tune an existing discourse segmenter developed and trained for written data, (ii) a data-programming approach that makes use of the same "text-based" discourse segmenter but enriched with manual defined rules (labeling functions). we found that both approaches yielded similar results. this suggests that both approaches are worth considering depending on the exact use case. while dataprogramming requires some heuristic rule engineering, fine-tuning requires annotated data that is costly to obtain, especially for relatively expert tasks such as discourse segmentation. we also ran a second experiment to investigate (i) how much manually annotated data is required before reaching the same performance as the data-programming approach; (ii) whether starting from a written base segmentation model was useful at all (compared to training the segmenter directly over the bert pretrained language model). to the first question, it is noteable that given the significant variability between folds, only a small amount of annotated data (∼ 20%) is sufficient to get close to the best results we obtained. moreover, starting from a written discourse segmenter model or directly from bert did not significantly change the results. overall, our findings suggest that annotating (segmenting) a large amount of conversation might not be necessary since both the data-programming approach (that makes use of an existing discourse segmenter developed on written data) and a model trained with little data (here about 2700 discourse units) yielded results comparable to a model finedtuned on our whole training set (13500 discourse units). breaking down the performance of the different models, we see that both the fine-tuned model and the weakly supervised one improve over the pause baselines. while pauses are strong predictors (rather high precision for a baseline), many discourse units are not preceded by a pause, so extra cues are needed. both models seem to easily learn how to segment within "fluent monologues" (even without pauses)-an result likely explained at least in part by the role of the existing discourse segmenter and the relevant language model. however, when speech becomes strongly disfluent, foot_3 , in particular when disfluency gets tangled up with discourse markers that typically signal discourse segment starts, both approaches struggle. finally, certain constructions, such as presentatives, which are frequent in conversational language but absent from written data, are also an issue. overall while the models are definitely useful as discourse segmenters, their scores are way below the state-of-the-art obtained on written text. apart from the fact that the task of edu segmentation is arguably more difficult for spoken language, underlying biases carried by segmenters trained on written data could explain in part why our models remain relatively confused when facing token sequences found only in conversational data sets, despite fine-tuning or our attempt to add heuristic specific rules. as future work, we plan to refine our experiments by separating discourse units into two categories: easy and difficult to segment. indeed, in conversation, a sizeable amount (about 19%) of discourse units are trivial to segment (single or lexical feedback items preceded and followed by long pauses) while some others, as we have seen in error analysis, are really complex to delineate. our opinion is that separating these two cases at all stages (including inter-coder agreement measures) will allow us to learn more about discourse segmentation of conversation and ultimately help in developing better performing models.