Weakly supervised discourse segmentation for multiparty oral conversations (2021)

Abstract: Discourse segmentation, the first step of discourse analysis, has been shown to improve results for text summarization, translation and other NLP tasks. While segmentation models for written text tend to perform well, they are not directly applicable to spontaneous, oral conversation, which has linguistic features foreign to written text. Segmentation is less studied for this type of language, where annotated data is scarce, and existing corpora more heterogeneous. We develop a weak supervision approach to adapt, using minimal annotation, a state of the art discourse segmenter trained on written text to French conversation transcripts. Supervision is given by a latent model bootstrapped by manually defined heuristic rules that use linguistic and acoustic information. The resulting model improves the original segmenter, especially in contexts where information on speaker turns is lacking or noisy, gaining up to 13% in F-score. Evaluation is performed on data like those used to define our heuristic rules, but also on transcripts from two other corpora.

## introduction discourse analysis, focusing on pragmatic aspects of text interpretation, especially aspects ranging beyond the level of the sentence, is a long standing domain of natural language processing (nlp), of growing importance for many tasks in nlp such as machine translation (mt) @xcite and summarization @xcite . the first level of discourse analysis consists in segmenting a discourse into basic units, which generally correspond to roughly clause-level units of text whose contents provide the arguments to discourse relations such as explanation, elaboration, and contrast @xcite @xcite . taking the resulting discourse segments as input for tasks such as summarization/sentence compression can improve performance over sentence-based approaches @xcite @xcite . segmentation is also crucial for downstream tasks that exploit discourse relations @xcite @xcite . because automatic segmentation on text-based corpora tends to yield good results (above 90% fscore on segment boundary recognition, depending on the language, cf. @xcite , the segmentation task is often neglected in favor of the prediction of discourse relations. state of the art discourse segmenters, however, generally take sentence boundaries as given, and often benefit from other forms of punctuation, as well as the wellformed sentences found in text-based corpora. such systems provide a less stable foundation for discourse parsing of spoken conversations, which is necessary to improve, e.g., real time recommendations from voice assistants and meeting summarization, and to develop more advanced assistants including robots/cobots with conversational capabilities. in these cases, we need to work directly on the audio signal or transcript, and cannot assume sentence boundaries or punctuation (at least not human-corrected punctuation). and the utterances might be far less well formed, as in the following example (translated from our data, see section 4) in which a speaker interrupts his own thought to ask a question about the name of a town-a question that he ends up answering himself with "no". ## related work the first automated attempts at predicting discourse structure and discourse segmentation with a rulebased approach can be attributed to @xcite . until recently, discourse segmentation had been generally ignored in discourse parsing, although there were some studies based mostly on statistical models using lexical/syntactic features @xcite @xcite and the rule-based approach of @xcite . these approaches are restricted to sentences, however, thus assuming that sentence boundaries are given. recently, interest in discourse segmentation was renewed with neural-network sequential classification using contextual embeddings @xcite , though still at the sentence level. the shared task at the disrpt 2019 workshop introduced a more general evaluation framework, with multilingual data and segmentation at the level of full texts, with a subtask that did not assume sentence boundaries @xcite . the best system, which also used a sequential model over contextual embeddings @xcite , showed the best performance both with and without sentence boundary information. in work on oral conversation, segmentation has often been approached not as a discourse problem, but as a problem of recognizing "sentences" in order to predict punctuation marks with ngram models and audio features, either to enrich automatic speech transcripts @xcite or to improve mt @xcite @xcite . arguably the relevant units for mt are what @xcite vaguely call "semantic boundaries". these units are less fine-grained than the segments needed for discourse analysis, however. @xcite is one of the first works to simultaneously address dialogue act (da) segmentation and classification for speech in multiparty meetings (where for the purposes of this paper, we can consider a dialogue act as just a dialogue specific term for a discourse segment). they found that a simple prosodic model aided performance over lexical information alone, especially for segmentation. they used pause information for segmentation, and added duration, pitch, energy and spectral tilt features for classification. @xcite used conditional random fields to simultaneously segment and label conversations, trained on the switchboard corpus, with a per-token classification accuracy around 70% (a token is either a segment boundary or not). more recent approaches make use of neural networks. @xcite proposed a joint segmentation-tagging model using bi-directional long short term memory layers, in the form of a word sequence tagger for segmentation and a sentence classifier for dialogue act tagging. @xcite proposed an end-to-end speech-to-dialogue-act recognizer in the form of a single attention-based neural network trained to perform word-level asr and fine-tuned to perform da segmentation and classification. both previous studies use the switchboard corpus, which, by discourse corpora standards, provides a large annotated dataset (1k conversations, 200k dialogue acts, 1.3m tokens) for english. evaluation measures in speech-oriented work focus either on words (belonging to the right segment or not) or exact segment boundaries of varying types that differ from conventions for written-text segmentation, making direct comparisons difficult. our approach is based on the data-programming paradigm @xcite , a weak supervision framework that has been applied mainly to information extraction problems in nlp, but also recently to discourse analysis @xcite , specifically for discourse structure prediction. we are unaware of similar work on discourse segmentation or on multi-modal text/speech classification problems. the novelty of the data-programming approach is that it requires only a fraction of the data to be manually annotated, for designing heuristics and for evaluation, and is arguably easier to adapt to new data. ## the data-programming approach the data-programming approach @xcite ) can be decomposed into three steps: step (2) is the crucial part of the approach. for the full formal description, we refer to @xcite , but in sum: for each lf λ j and each instance x i , define the label matrix as λ = [λ j (x i )], and the vector of the unknown true labels for x i as @xmath0. the label model is an estimate p w (λ, y ) of their joint probability. it depends on two factors, the label propensity (lab) of an lf, i.e. how often it assigns a label, and its accuracy (acc) foot_1 : the model is then a log-linear model on the concatenated factor φ i for each instance: this is learned without access to y by minimizing the negative marginal log likelihood given the observed λ: the predictions p ŵ(y |λ) can then serve as probabilistic labels, yielding a labeled "train" set. we use the snorkel library, foot_2 which provides implementations of the various steps of data programming, e.g. a framework to develop and evaluate lfs and to train a label model. while manually designing lfs still requires some human effort and expertise, it has proven to be less labor-intensive and more reliable than massive data annotation on some tasks @xcite and has other benefits: it does not depend on the size of a data set, it is arguably easier to adapt to a different context (and to some extent a different language), and, as we show below, does not require the final supervised model to have access to all of the information initially exploited by lfs. in our case, this means that the annotations are generated by lfs that exploit both textual and acoustic features, but the final model is only trained on transcribed text. ## data our research aims to improve the conversational capacities of automated assistants in french. as such, we develop our segmentation model using a new corpus of around 7.5 hours (7hr, 40min) of multiparty conversations in french. the interactions come from real meetings recorded in an industrial setting and can be divided into three types: (i) presentations: one person presents their work and other participants (7-12 participants (p)) ask questions during or after the presentation; (ii) weekly update meetings (4-5p): participants present their work from the previous week to a manager, field questions, and describe their next steps; and (iii) informal discussions (4-6p): about work related topics or more personal topics, such as vacation plans. the corpus contains a total of 11 meetings ranging in length from 8.5 minutes (type (iii)) to 84 minutes (type (i)). the entire corpus was transcribed, first automatically and then corrected by hand, so as to include any word that can be heard on the transcript. speaker turns-maximal, continuous sequences of words that can be attributed to a single speaker--were also manually corrected. transcripts were aligned with the audio files using jtrans @xcite to produce time stamps for each word. the interactions, as is typical of meetings and informal conversations, were not prepared in advance, leading to disfluencies, such as hesitations (uh, euh, etc.), repetitions, self corrections ("j'ai fait euh on est" ≈ i did uh we went), and incomplete sentences, as well as linguistic tics and overlapping speech, when people speak at the same time. all of these elements, apart from overlapping speech, are represented in the transcripts. areas of overlapping speech, which complicate the alignment between the transcript and audio files, and which would have not been captured by an asr system, were deleted. this led to a removal of 26 minutes, reducing the corpus to 434 minutes (7hr, 14min). the development set used to design our lfs (see section 3) contains one weekly update meeting (4p) and one informal discussion about equipment to buy for an office (4p). as shown in the test set used to validate the whole process contains one presentation (7p) and one informal conversation about vacation (5p), making up 12.2% of the corpus. manual segmentation yielded around 1100 segments. while some of the participants who asked questions in the test set appeared in other recordings from the corpus, the principal speakersi.e., the presenter for the presentation and the two people discussing their vacation plans-did not. the remaining interactions from the corpus were set aside to be used to train the label model (step 2 in the data-programming approach) and then the final supervised model (step 3). the transcripts for the training set were not manually annotated, as the point of the data-programming approach is to be able to predict them automatically based on weights that the label model assigns to the lfs. ## experiments we adopt the architecture of @xcite for our final model (step 3), as this is currently the best discourse segmenter that does not require sentence preprocessing (and the only one with a french model). using the same architecture also allows us to estimate the degradation of performance from written text to oral speech transcripts. experiments were conducted on both the test set from our corpus and transcripts from two other spoken french corpora. an additional advantage of the weakly supervised approach followed here is that our final model can take as input plain transcripts; that is, it does not require all of the information used as input to the label model (pitch, energy, postagging, syntactic analysis, etc.). for evaluation, we follow the procedure chosen by the disrpt19 shared task, i.e. we measure fscore on segment boundary detection (bos tags). ## conclusion we have detailed the design and evaluation of a model of discourse segmentation for spontaneous, multiparty oral conversation. while segmentation of such data is crucial for recovering discourse structure and for downstream tasks such as meeting summarization, appropriate data sets are lacking and/or diverse. the novelty of our approach lies in transferring a model designed for prepared, written text to conversation transcripts using minimal data or manual annotation. heuristics or labeling functions, which may draw on heterogeneous information sources, are designed by experts and used to automatically label training data for supervision. our evaluations show that our heuristic-driven approach significantly outperforms a state-of-theart discourse segmenter trained on prepared, written text. future work will focus on extending our model to other languages and corpora with varying levels of spontaneity (and disfluency). weak supervision for nlp is an active subject, and recent new frameworks such as @xcite might prove even more suitable for sequence tagging problems in future work.