M ^3 S eg: A Maximum-Minimum Mutual Information Paradigm for Unsupervised Topic Segmentation in ASR Transcripts (2023)

Abstract: Topic segmentation aims to detect topic boundaries and split automatic speech recognition transcriptions (e.g., meeting transcripts) into segments that are bounded by thematic meanings. In this work, we propose M ^3 Seg, a novel Maximum-Minimum Mutual information paradigm for linear topic segmentation without using any parallel data. Specifically, by employing sentence representations provided by pre-trained language models, M ^3 Seg first learns a region-based segment encoder based on the maximization of mutual information between the global segment representation and the local contextual sentence representation. Secondly, an edge-based boundary detection module aims to segment the whole by topics based on minimizing the mutual information between different segments. Experiment results on two public datasets demonstrate the effectiveness of M ^3 Seg, which outperform the state-of-the-art methods by a significant (18%–37% improvement) margin.

## introduction automatic speech recognition (asr, also known as computer speech recognition or speech-to-text) @xcite has brought us great convenience by transcribing conversations into text anytime and anywhere to aid human understanding. however, the generated unstructured transcriptions are sometimes too lengthy for users to grasp a high-level meaning quickly. meeting transcripts are usually long and contain multiple heterogeneous topics in various structures, such as opening sessions, different discussion subjects, and closing sections. as one or more topics usually drive conversations or discussions, topic segmentation @xcite can improve the readability of transcription and facilitate ## method formally, let s denote a meeting transcript produced by an automatic speech recognition (asr) system, which consists of a list of n utterances @xmath0, s 2 , • • • , s n }. topic segmentation can be seen as a problem of topic change detection, and aims to cut the transcript into consecutive segments {s 1:i-1 , s i:j , • • • , s k:n } based on the underlying topic structure. for each segment s i:j , it represents a segment in a meeting transcript, from the i-th sequence to the j-th sequence s i:@xmath1, • • • , s j }. for example, an hour-long meeting transcript can be broken down into different topic segments (e.g., opening sessions, different discussion subjects, and closing remarks) to make it more read-able. note that we cannot access gold-standard segments as human annotations do not exist. model overview as aforementioned, given a transcript @xmath2, s 2 , • • • , s n }, topic segmentation seeks to splits it into several topic segments, where the sentences in each segment belong to the same topic (requirement i) but different segments represents relatively independent topics (requirement ii). inspired by this idea, we propose a maximum-minimum mutual information paradigm for topic segmentation (titled m 3 seg) in an unsupervised manner, which learns segment representations by maximizing mutual information and partitions different segments by minimizing mutual information. mutual information (mi) can measure the dependence between two random variables @xcite . given two random variables a and b, the mi between them is i(a; b) = a,b p (a, b) log p (a|b) p (a) . the intuitive interpretation of i(a; b) is a measurement of the degree a reduces the uncertainty in b or vice versa. for example, the mi i(a; b) is equal to 0 when a and b are independent. therefore, from the view of mi, different topic segments should be as independent of each other as possible (i.e., mi minimization), but sentences within the same topic segment should be as dependent on the same topic as possible (i.e., mi maximization). based on this insight, m 3 seg consists of two stages (as depicted in figure 1 ): specifically, given n utterances @xmath3, s 2 , • • • , s n } in an asr transcript, we first use the pre-trained language model (plm) to obtain the contextualized representation r of each sentence @xcite our assumption is that in a meeting transcript, a continuous sequence of utterances contains a thematic information, which can be either coarsegrained or fine-grained. for example, the entire meeting transcript's utterances may be based on a certain motivation (coarse-grained theme) for convening, while if divided into fine-grained segments, each utterance can be associated with a certain issue (fine-grained theme). it is based on this assumption that we construct data from any continuous sequence of utterances in the same meeting transcript and use the maximization of mutual information to train a segment encoder e to learn the global segment representation (eq 1) @xcite . this also contributes to the effectiveness of our method. ## experiments to demonstrate the effectiveness of our model, we evaluate m 3 seg on two widely-used benchmark datasets, ami meeting corpus @xcite and the icsi meeting corpus (the dataset details are shown in appendix a.). according to the paper @xcite , it is more likely that the dataset we used consists of unstructured chats, transcriptions of spoken language from a spoken scene, which result in often incoherent and incomplete utterances with some word errors. we compare m 3 seg with several state-of-the-art unsupervised topic segmentation methods, including region-based methods (i.e., topictiling @xcite and dialogpt @xcite ) and edge-based methods (i.e., texttiling @xcite , c99 (choi, 2000), and texttiling-bert @xcite ). for analysis, we also show the random and even methods, which refer to placing topic boundaries randomly and every n-th utterance, respectively. we perform two standard evaluation metrics, pk @xcite and windiff (wd) @xcite scores (details are shown in appendix b), and the model implementation details are shown in appendix c. regarding the detection of multiple boundaries, we use a certain threshold δ to control it. that is, as long as the mutual information gap between pairs of regions is greater than δ, it will be considered as topic boundaries (detailed in section 2). for the random baseline, following the settings of texttiling-bert, we set each utterance to have a 0.30 probability of being placed with topic boundaries (the reason is that this value is roughly consistent with the probability of boundaries occupying the entire utterance).