Conversation Disentanglement with Bi-Level Contrastive Learning (2022)

Abstract: Conversation disentanglement aims to group utterances into detached sessions, which is a fundamental task in processing multi-party conversations. Existing methods have two main drawbacks. First, they overemphasize pairwise utterance relations but pay inadequate attention to the utterance-to-context relation modeling. Second, huge amount of human annotated data is required for training, which is expensive to obtain in practice. To address these issues, we propose a general disentangle model based on bi-level contrastive learning. It brings closer utterances in the same session while encourages each utterance to be near its clustered session prototypes in the representation space. Unlike existing approaches, our disentangle model works in both supervised setting with labeled data and unsupervised setting when no such data is available. The proposed method achieves new state-of-the-art performance on both settings across several public datasets.

## introduction multi-party conversations generally involve three or more speakers in a single dialogue, in which the speaker utterances are interleaved, and multiple topics may be discussed concurrently @xcite . this causes inconvenience for dialogue participant to digest the utterances and respond to a particular topic thread. conversation disentanglement is the task of separating these entangled utterances into detached sessions, which is a prerequisite of many important downstream tasks such as dialogue information extraction @xcite , state tracking @xcite , response generation @xcite @xcite , and response ranking @xcite . there has been substantial work on the conversation disentanglement task. most of them emphasize on the pairwise relation between utterances in session 1 session 2 lolcat can i wget mms://mms-icanal-odc.online.no/norskripub/ autodistribusjon/nrk3_201111092212_koid… usr13 lolcat: sure, why not? dr_willis lolcat: try it and see? if not theres programs like streamtuner, or vlc, or others that capture streams… … gremuc hnik hi! quick question: i don't like unity, but i do like gnome3. will ubuntu 12.04 offer gnome3 without unity, the regular "original" gnome3 ottscorp try lxde gremuchnik :-) arnezt gremuchnik: may be you can use ubuntu no-effect from login menu :) ottscorp ubuntu plans on sticking with unity l1nuxr ules gremuchink although i cant answer your question, you can install and use any desktop in linux usr13 lolcat: oh, yea, myabe you need streamripper or something. but if you just want to watch it, try gxine ottscorp or perhaps http://forum.videohelp.com/threads/25704 5-how-to-record-streaming... a two-step manner. they predict the relationship between utterance pairs as the first step, followed by clustering utterances into sessions as the second. in the first step, early works @xcite utilize handmade features and discourse cues to predict whether two utterances belong to the same session or whether there is a reply-to relation. the recent development in deep learning inspires the use of neural network such as lstm or cnn to learn abstract features of utterances in training @xcite . more recently, a number of methods show that bert in combination with handcrafted features or heuristics remains a strong baseline @xcite @xcite . in the second step, the most popular clustering methods use a greedy approach to group utterances by adding pairs @xcite . there are also some variations incorporating voting mechanism @xcite , bipartite graph matching @xcite or additional tracking models @xcite ). an obvious drawback of such two-step approach is that the pairwise relation prediction might not capture enough contextual information as the connection between two utterances depends on the contexts in many cases @xcite . also, focusing on pairwise relations leads to a short-sighted local view. to mitigate this, there are methods trying to introduce additional conversation loss @xcite or session classifier @xcite to group utterances in the same session together. we also see methods leveraging relational graph convolution network @xcite or masking mechanism in transformers @xcite . more directly, end-to-end methods @xcite capture the context information contained in detached sessions and calculate the matching degree between a session and an utterance. however, many of such methods are conducted in an online manner which only considers the preceding context. it may lead to biased session representations, introduce noisy utterances to sessions and consequently accumulate errors. meanwhile, most of these methods rely heavily upon human-annotated session labels or reply-to relations, which are expensive to obtain in practice. although there have been a few attempts to tackle this issue, a more general framework that can handle both supervised and unsupervised learning is yet to come. for example, @xcite design a deep co-training scheme with message-pair classifier and session classifier. however, various data augmentation procedures based on heuristics are required for good performance. @xcite propose a zero-shot disentanglement solution based on a related response selection task. still, it relies on a closely related dataset that comes from the same ubuntu irc source inside dstc8. recently, contrastive learning @xcite has brought prosperity to numbers of machine learning tasks by introducing unsupervised representation learning. substantial performance gains have been reported in computer vision @xcite and nlp works @xcite . they believe that good representation should be able to identify semantically close neighbors while distinguishing from non-neighbors. intuitively, in multi-party conversation, utterances in the same session should semantically resemble each other while be far apart from utterances in other sessions. instead of handcrafted features such as speaker, mention and time difference etc, it provides another option for auto-matically learn discriminative representations. in this work, we design a bi-level contrastive learning scheme (bi-cl) to learn discriminative representations of tangled multi-party dialogue utterances. it not only learns utterance level differences across sessions, but more importantly, it encodes session level structures discovered by clustering into the learned embedding space. specifically, we introduce session prototypes to represent each session for capturing global dialogue structure and encourage each utterance to be closer to their assigned prototypes. since the prototypes can be estimated via performing clustering on the utterance representations, it also supports unsupervised conversation disentanglement under an expectation-maximization framework. we evaluate the proposed model under both supervised and unsupervised settings across several public datasets. ## method the definition of the conversation disentanglement task and details of our model are sequentially presented in this section. starting from the supervised setting for a clear view, we gradually extend to the unsupervised setting. ## conclusion we studied disentanglement on multi-party conversations and proposed a general model that works in both supervised and unsupervised learning settings. it is trained with a bi-level contrastive learning mechanism to bring utterances in the same session closer and encourage utterances to flock around their session centers. at the same time, we aim to pull utterances from different sessions further apart by contrasting each utterance with negative samples. the obtained representations naturally fit to the clustering scheme for session predictions. consequently, k means is used during inference to predict the sessions. our model is evaluated on the largest benchmark dataset ubuntu irc and the latest benchmark dataset movie dialogue. experimental results show new sota performance results and advancements compared to previous works. additionally, the stability of our model across different datasets, as well as different training schemes with or without session labels, shows its potential to be applied in a general setting, 6 limitations our work has the following limitations. firstly, although bidirectional lstm is more light-weight and obtains reasonable performance for our task, an easy extension is to explore how pre-trained language models such as bert would further affect the performance @xcite . secondly, the prediction of session number k is only based on conversation utterances. more advanced session number estimation model would be devised to capture more side information for more accurate k prediction. an alternative approach is to adopt different clustering algorithm such as cisir @xcite that does not require the prediction of cluster number but instead has a universal, empirically determined threshold that controls the cluster size. last but not least, our model has not been applied to dialogues of length longer than 50, and we have not verified its effectiveness of modeling longer dependency. this entails our future effort to adapt our model to a more general setting with longer conversation, more threads, and more complicated dialogue structures.