Experimenting with Discourse Segmentation of T aiwan S outhern M in Spontaneous Speech (2024)

Abstract: Discourse segmentation received increased attention in the past years, however the majority of studies have focused on written genres and with high-resource languages. This paper investigates discourse segmentation of a Taiwan Southern Min spontaneous speech corpus. We compare the fine-tuning a Language Model (LLM using two approaches: supervised, thanks to a high-quality annotated dataset, and weakly-supervised, requiring only a small amount of manual labeling. The corpus used here is transcribed with both Chinese characters and romanized transcription. This allows us to compare the impact of the written form on the discourse segmentation task. Additionally, the dataset includes manual prosodic breaks labeling, allowing an exploration of the role prosody can play in contemporary discourse segmentation systems grounded in LLMs. In our study, the supervised approach outperforms weak-supervision ; character-based version demonstrated better scores compared to the romanized version; and prosodic information proved to be an interesting source to increase discourse segmentation performance.

## introduction discourse segmentation consists in breaking down texts or conversations into functional units that better corresponds to participants' intentions than sentences or simple speech activity chunks. we will use the term discourse unit (du) @xcite to designate a minimal speech act or communicative unit. each du corresponds roughly to a clause-level content that denotes a single fact or event. while the segmentation of discourse units (dus) in written documents has received a lot of attention from the discourse and nlp community, the same cannot be said for the segmentation of spontaneous speech. in this study, we approach the segmenta-tion of discourse units in a corpus of spontaneous speech in taiwan southern min. southern min is a sino-tibetan language spoken by over 50 million people, and includes taiwan southern min, which is one of the official language of taiwan. we take advantage here of an existing discourse segmented corpus of spoken interviews for running discourse segmentation experiments. we develop du segmenters based on different principles and evaluate their performance. more precisely, we compare fine-tuning an llm with hand labeled data vs. employing a data programming approach @xcite that requires only a fraction of annotated data. while fine-tuning llms for language well represented in the llm training data proved to be a very efficient solution @xcite , it remains to be seen whether this approach is relevant for languages, particularly their spontaneous speech variants, less represented in the training data. finally, we investigate the impact of using either romanization or chinese characters in our dataset, as well as the potential contribution of prosody. ## related work in recent years, there has been a renewed interest in discourse parsing and discourse unit segmentation within the nlp community. as in other subdomains, large language models have proven highly beneficial and allowed to reach unprecedented scores for these tasks. however, discourse segmentation within these deep learning approaches has been applied to only a few langauges, until the recent initiative of disrpt campaigns started @xcite @xcite . the work conduced within the framework of these campaigns has equipped the community with a set of powerful tools and frameworks to perform du segmentation using these contemporary approaches. as discussed in @xcite , even for written genres, discourse segmentation performance drops in languages other than english and when gold sentences are not given, due to sentence segmenters being far from perfect @xcite . considering spontaneous conversational speech, the related tasks of dialogue-act segmentation and tagging yiels various interpretation regarding the definition of base units. for instance, some models explain that dialogue acts being multi-functional, several segmentations can be considered depending on the aspects of dialogue being considered at the time of segmentation @xcite . a recent trend involves approaching discourse segmentation with sequential models over contextual embeddings @xcite . turning specifically to spontaneous speech discourse segmentation, @xcite applied a weak-supervision approach @xcite and reached an f-score of 73.7 while having access to gold turn segmentation. more specifically, manual heuristic rules, including some rules exploiting the discourse segmentation model trained on a written dataset @xcite , were created to annotate noisily the entire dataset. this noisy data was then used to fine-tune an llm, bert @xcite in that case. in @xcite , a larger amount of manual annotation allowed to compare fine-tuning with larger amount of training data and a weakly-supervised approach. for this french dataset, it was concluded that more than 7000 annotated dus were required in the supervised training approach to beat the weakly-supervised approach (f-score: 70.6). when more data was used, supervised fine-tuning reached slightly higher scores (f-score: 73.9). these f-score results are 10 -15% than the scores obtained on written genress, which is expected as sentence splitters leveraging punctuation provide substantial assistance for discourse unit segmentation. in speech, particularly spontaneous interactional speech, pauses are useful but are by far less reliable in predicting discourse units since they are involved in many other dimensions and are subject to significant inter-individual variability. recently @xcite 1 , an improvement over muller et al. (2019) has been developed, allowing to reach new state-of-the-art results for 1 code at https://github.com/phimit/jiant/ discourse segmentation in various languages. our paper reuses the technical framework of this paper. segmenting speech into discourse and prosodic units has been the focus of numerous studies across various languages, including high-resource languages like english @xcite , dutch @xcite , french or mandarin @xcite as well as low-resource languages @xcite . discourse-prosodic interface research has also been developed for better understanding turn-taking mechanisms @xcite . the deep connection between discourse and prosody has led researchers to explore prosodic cues for discourse tasks with some success @xcite . however, to our knowledge, there are no studies in which modern llm-based systems described above, which achieve high scores based solely on transcripts, have benefited from incorporating acoustic-prosodic cues. an interesting attempt was made in @xcite , which validated the weak-supervision approach exploiting silent pauses among other elements, but the results did not improve with the inclusion of other acoustic-prosodic features. this is likely due to (i) the already high scores obtained from text alone, which would require cues coming from other sources to yield very high precision; and (ii) to the challgenge of automatic extracting reliable prosodic cues, such as speech rate, pitch or even intensity, from conversational speech. discourse studies on southern min (and related language like hakka or cantonese) have focused on final particles @xcite @xcite , which can carry an interesting range or semantic and pragmatic functions. moreover, there have been specific corpus studies examining discourse markers in taiwan southern min @xcite @xcite . however, to the best of our knowledge, there has been no attempt to automatically segment discourse units in this language. additionally, there have been specific corpus studies examining @xcite @xcite . however, to the best of our knowledge, there has been no attempt to automatically segment discourse units in this language. ## methodology / experiments the corpus includes interviews of 16 speakers. we made 8 folds composed of two speakers each and ran a cross-validation over the 8 folds with different test / dev / train splits. given our corpus, this is a method that maximizes the distance between training and testing data. two main approaches are evaluated for segmenting automatically our dataset : (i) directly fine-tuning a llms with all the data at our disposal (in a supervised way) (supervised setting), (ii) create a noisily annotated datasets thanks to manual heuristic rules (see figure 2 ) and a model to combine them. more specifically, we used roberta @xcite and the framework fine-tuning it was discut @xcite , grounded in jiant environment @xcite . the weak-supervision framework uses skweak @xcite rather than snorkel @xcite . skweak natively allows the model to exploit the sequential nature of our task. on the technical side, skweak relies on spacy (honnibal and montani, 2017) documents. in order to keep all the relevant information (timing, pos-tags, prosody labels) linked to the tokens and to use them in the labeling rules, we made use of spacy extensions attributes. in the weak supervised approach, we use from noisy labels provided by the labeling rules. skweak allows to choose an hmm to perform this sequence labeling task. while this approach can be adopted without annotated data, a small development set is useful for testing and crafting the heuristic labeling rules. we can decide more efficiently which manual rules should be retained, dropped or improved thanks to the metrics that are computed on the development set. besides precision, recall and f-score, overlaps and conflicts (with other rules) metrics are also useful to take decisions over the usage of these rules (see to summarize, the weakly supervised approach is performed as follows: for the time being, the labeling rules crafted are extremely simple. they are using (i) pause duration and turn information; (ii) frequent tokens present at discourse boundaries; (iii) pos-tags overrepresented at discourse boundaries. moreover, manually annotated prosodic units boundaries are included in the dataset and we use them for some experiments. as mentioned above, pos-tags are encoded in a multihot format. the labeling rules def pause_and_begin_char(doc): for idx, token in enumerate(doc): if @xmath0: if (doc[idx-1].@xmath1._.@xmath2.text in begin_char): yield idx,idx+1,'bdu' else: yield idx,idx+1,'abs' else: yield idx,idx+1,'bdu' exploiting pos are formulated accordingly to this ambiguous situation. ## results the results comparing the general approach are presented in figure 3 ; the one related to the impact of the written form used are in figures 4 and 5 and the results of the prosody experiments are visualized in 6. all the numbers can be checked in annex 3. ## error analysis to further understand how our models could be improved we performed a detailed qualitative error analysis of the various models output. on the other hand, my boyfriend oh he would still gone to see me' (gen: genitive marker; part: a marker similar to ba5 in mandarin ba construction.) a. gold annotation: [ah reversely is # reversely is i gen boy friend oh # he still would go part me see] b. gold & ws-trained: [ah reversely is # reversely is i gen boy friend oh #] [he still would go part me see] a. gold annotation (and ws-trained): [i self have gen boy friend all all marry no success] b. gold trained: [i self have gen boy friend] [all all marry no success] finally, (3) shows an example of how goldtrained and ws-trained segmentation may differ from the gold annotation in distinct ways. the gold annotation has a du boundary between the main clause and the tag question, the former containing some disfluencies. the model trained on gold annotation did not recognize the boundary with the tag question and instead put a boundary before the word 'like this' (an2-ne), which reflects the fact that an2-ne is a discourse marker that can occur in clause-initial and clause-final positions. the model trained on ws data, on the other hand, did not put a du boundary for the entire sequence (thus having an error of under-segmentation before 'you know not'), as there was no pause nor words that have a strong tendency to start a du in the corpus. a. gold annotation: [then walking still does-not like this does-not require tiptoeing] [you know not] b. gold-trained: [then walking still doesnot] [like-this does-not require tiptoeing you know not] c. ws-trained: [then walking still does-not like-this does-not require tiptoeing you know not] 7 discussion and future work in this paper, we applied state-of-art techniques of discourse segmentation to a dataset of taiwan southern min. we compared supervised and weakly supervised approaches. moreover the linguistic information included in the original dataset allowed us to test some hypotheses along the way. we tested whether (i) it was easier to segment with the character-based or romanized version of the corpus ; and (ii) prosodic gold labels could help these new models of discourse segmentation. an important overall result is that the approach employed (fine-tuning a sequence-to-sequence model) performs extremely well on this taiwan southern min corpus, a language not included in the base language model (llm) used. this is an important result with regard to the applicability of such approaches to low-resource languages for this task. the longer term goal of this work is to apply the best model we can build to a much larger corpus of taiwanese interviews. the results obtained enable us to try to replicate existing studies on discourse-prosody interface in spontaneous speech, which have relied solely on manually annotated data. getting into the comparison of the two approaches tested, we should remind here that the scores obtained with gold annotations should be taken as a top line for the weak supervision approach. indeed, the amount of manual gold segmentation for this corpus is substantial and does not aligh with the typical scenario for adopting a weak-supervision approach. with this consideration in mind, we observe that the weakly supervised approach failed to produce comparable results to the supervised setting. this can be attributed on the one hand to the supervised approach yielding highly competitive results through fine-tuning with only about 10% of our full amount of annotated data (corresponding 7k tokens, 700 discourse units); and on the other hand to the relatively low performance of our weakly supervised model. however, this does not negate the potential interest of weak supervision. our current rules are rudimentary, primarily using simple pauses, tokens information and ambiguous pos-tags. we intend to enhance these labeling rules in several directions: (i) using a real pos-tagger that would reduce ambiguity ; (ii) developing more sophisticated labeling rules to address phenomena specific to spontaneous speech, such as disfluencies. regarding the comparison between the character-based and romanized versions of the corpus, the clear conclusion is that the character version consistently yields better results regardless of the amount of fine-tuning data provided. this could be attributed to both the benefit of lower ambiguities of characters over romanized version and to the presence of mandarin data in roberta. regarding prosody, this study has shown that, in line with linguistic predictions and previous computational models, but contrary to recent findings on this task, prosodic information can indeed help in discourse unit segmentation. the next obvious step is to automatize the extraction of relevant acoustic features that approximate efficiently the manual annotations we had in this stydy. from the primary prosodic features identified in @xcite for english, excluding the ones already exploited by our pause and turn related rules, we identify (i) pitch differences across the discourse unit boundary, and (ii) duration of phones and rhymes preceding the decision point.