Improving Unsupervised Dialogue Topic Segmentation with Utterance-Pair Coherence Scoring (2021)

Abstract: Dialogue topic segmentation is critical in several dialogue modeling problems. However, popular unsupervised approaches only exploit surface features in assessing topical coherence among utterances. In this work, we address this limitation by leveraging supervisory signals from the utterance-pair coherence scoring task. First, we present a simple yet effective strategy to generate a training corpus for utterance-pair coherence scoring. Then, we train a BERT-based neural utterance-pair coherence model with the obtained training corpus. Finally, such model is used to measure the topical relevance between utterances, acting as the basis of the segmentation inference. Experiments on three public datasets in English and Chinese demonstrate that our proposal outperforms the state-of-the-art baselines.

## introduction dialogue topic segmentation (dts), as a fundamental task of dialogue modeling, has received considerable attention in recent years. in essence, dts aims to reveal the topic structure of a dialogue by segmenting the dialogue session into its topically coherent pieces. an example is given in different from the monologue topic segmentation (mts) task @xcite turns dialogue text turn-1: a: for how long should the liability insurance coverage remain in effect? turn-2: b: as long as the registration of your vehicle remains valid. turn-3: a: does this apply for motorcycles too? turn-4: b: there are some exceptions for motorcycles. turn-5: a: regarding the name on my vehicle registration application and the one on the insurance identification card, do they need to be the same? turn-6: b: yes, the names must match in both documents. turn-7: a: can i submit copies or faxes of my insurance identification card to the dmv? turn-8: b: yes, you can. but take into consideration that the card will be rejected if the dmv barcode reader can not scan the barcode. to alleviate the aforementioned limitations in previous work, in this paper, we still cast dts as an unsupervised learning task to make it applicable to dialogues from diverse domains and resources. however, instead of merely utilizing shallow features for coherence prediction, we leverage the supervised information from the text-pair coherence scoring task (i.e., measuring the coherence of adjacent textual units @xcite @xcite ), which can more effectively capture the deeper semantic (topical) relations between them. due to the absence of supervision, we propose a simple yet effective strategy to generate a training corpus for the utterance-pair coherence scoring task, with the paired coherent/notutterance pairs as datapoints. then, after applying such strategy, we use the resulting corpus to train an utterance-pair coherence scoring model with the relative ranking objective @xcite . in practice, we create a training corpus from large conversational datasets containing real daily communications and covering various topics (proposed in @xcite and @xcite ). in particular, all the adjacent utterance pairs are firstly extracted to form the positive sample set. then for each positive sample, the corresponding negative samples are generated by replacing the subsequent turn in the positive sample with (1) an non-adjacent turn randomly picked from the same dialogue, and (2) a turn randomly picked from another dialogue talking about another topic. once the training corpus is ready, we re-purpose the next sentence prediction (nsp) bert model @xcite as the basic framework of our utteracepair coherence scoring model. after fine-tuning the pretrained nsp bert on our automatically generated training corpus with the marginal ranking loss, the resulting model can then be applied to produce the topical coherence score for all the consecutive utterance pairs in any given dialogue. such scores can finally be used for the inference of topic segmentation for that dialogue. we empirically test the popular texttiling algorithm @xcite enhanced by the supervisory signal provided by our learned utterance-pair coherence scoring model on two languages (english and chinese). the experimental results show that texttiling enhanced by our proposal outperforms the state-of-the-art (sota) unsupervised dialogue topic segmenters by a substaintial margin on the testing sets of both languages. finally, in a qualitative analysis, by visualizing the segment predictions of the different dts segmenters on a sample dialogue, we show that the effectiveness of our proposal seems to come from better capturing topical relations and consideration for dialogue flows. however, compared with monologue documents, dialogues are generally more fragmented and contain many more informal expressions. the dis-course relation between utterances are also rather different from the monologue text. these distinctive features may introduce undesirable noise and cause limited performance when the supervised approaches trained on wikipedia is applied. since the lack of training data still remains a problem for dts, unsupervised methods, especially the ones extending texttiling @xcite , are still the mainstream options. for instance, @xcite enhanced texttiling with word embeddings, which better capture the underlying semantics than bagof-words style features. later, @xcite replaced word embeddings with bert as the utterance encoder to produce the input for texttiling, because pretrained language models like bert better capture more utterance-level dependencies. also, to avoid a too fragmented topic segmentation, they adjusted the texttiling algorithm into a greedy manner, which however requires more hyper-parameters and greatly limits the model's transferability. in contrast, here we adopt the original texttiling to minimize the need of hyperparameters and use coherence signals for utterances learned from real-world dialogues to make our proposal more suitable for conversational data. another line of research explores casting dts as a topic tracking problem @xcite , with the predefined conversation topics as part of the supervisory signals. even though they have achieved sota performance on the in-distribution data, their reliability on the out-of-distribution data is rather poor. in contrast, our proposal does not require any prior knowledge (i.e., predefined topics) as input, so it is more transferable to out-of-distribution data. @xcite observed that particular patterns of grammatical role transition for entities can reveal the coherence of monologue documents. hence, they proposed the entity-grid approach by using entity role transitions mined from documents as the features for document coherence scoring. later, cervone and riccardi (2020) explored the potential of the entity-grid approach on conversational data and further proved that it was also suitable for dialogues. however, one key limitation of the entity-grid model is that by excessively relying on the identification of entity tokens and their corresponding roles, its performance can be reduced by errors from other nlp pre-processing tasks, like coreference resolution, which can be very noisy. ## methodology following most of the previous work, we adopt texttiling @xcite as the basic algorithm for dts to predict segment boundaries for dialogues ((b) in figure 1 ). formally, given a dialogue d in the form of a sequence of utterances {u 1 , u 2 , ..., u k }, there are k -1 consecutive utterance pairs. then an utterance-pair coherence scoring model is applied to all these pairs and finally get a sequence of coherence scores {c 1 , c 2 , ..., c k-1 }, where c @xmath0, 1] indicates how topically related two utterances in the ith pair are. instead of directly using the coherence scores to infer segment boundaries, a sequence of "depth scores" {dp 1 , dp 2 , ..., dp k-1 } is calculated to measure how sharp a valley is by looking at the highest coherence scores hl(i) and hr(i) on the left and right of interval i: higher depth score means the pair of utterances are less topically related to each other. the threshold τ to identify segment boundaries is computed from the mean µ and standard deviation σ of depth scores: τ = µ-σ 2 . a pair of utterances with the depth score over τ will be select to have a segment boundary in between. next, we describe our novel training data generation strategy and the architecture of our new utterance-pair coherence scoring model, which are the two key contributions of this paper. ## experiments we comprehensively test our proposal by empirically comparing it with multiple baselines on three datasets in two languages. ## conclusions and future work this paper addresses a key limitation of unsupervised dialogue topic segmenters, namely their inability to model topical coherence among utterances in the dialogue. to this end, we leverage signals learned from a neural utterance-pair coherence scoring model based on fine-tuning nsp bert. with no data labeled with gold coherence score, we also propose a simple yet effective way to automatically construct a training dataset from any source dialogue corpus. the experimental results on three testing sets in english and chinese show that our proposal outperforms all the alternative unsupervised approaches. for the future, although most recent work has built on texttiling, we plan to explore if our proposal can also be integrated with other unsupervised topic segmentation methods, like graphseg and bayesseg, rather than just texttiling. furthermore, we also plan to explore effective strategies to exploit external commonsense knowledge (eg., conceptnet @xcite ) or user characters @xcite in topic segmentation, since they have been shown to be beneficial in dialogue generation @xcite and summarization @xcite .