Conversation- and Tree-Structure Losses for Dialogue Disentanglement (2022)

Abstract: When multiple conversations occur simultaneously, a listener must decide which conversation each utterance is part of in order to interpret and respond to it appropriately. This task is referred as dialogue disentanglement. A significant drawback of previous studies on disentanglement lies in that they only focus on pair-wise relationships between utterances while neglecting the conversation structure which is important for conversation structure modeling. In this paper, we propose a hierarchical model, named Dialogue BERT (DIALBERT), which integrates the local and global semantics in the context range by using BERT to encode each message-pair and using BiLSTM to aggregate the chronological context information into the output of BERT. In order to integrate the conversation structure information into the model, two types of loss of conversation-structure loss and tree-structure loss are designed. In this way, our model can implicitly learn and leverage the conversation structures without being restricted to the lack of explicit access to such structures during the inference stage. Experimental results on two large datasets show that our method outperforms previous methods by substantial margins, achieving great performance on dialogue disentanglement.

## introduction in a multi-party chat stream @xcite @xcite , messages related to different topics are entangled with each other, which makes it difficult for a new user to understand the context of the discussion in the chat room. dialogue disentanglement @xcite @xcite aims at disentangling a whole conversation into several threads from a data stream so that each thread is about a specific topic. early research either did not release their datasets (adams and figure 1 : an example of dialogue disentanglement. in this example, conversations marked with different colours are entangled together. this task aims to separate this chat stream by conversations. @xcite or used small datasets @xcite @xcite @xcite . @xcite released a new largescale dataset that made it possible to train a more complex model and to fairly compare different models. figure 1 shows an example of dialogue disentanglement in this dataset. currently, most of the existing methods for dialogue disentanglement employ a two-step approach framework. firstly, a model is employed to determine the relation between two messages. then a clustering algorithm is employed to separate these messages into different conversation clusters. following this framework, @xcite proposed a bert-based model named masked hierarchical transformer (mht), which aims at making use of the conversation structures. this method uses a mask mechanism to explicitly build connections between context messages and their corresponding ancestors in a conversation. however, the main drawback of their approach is that the designed mask is computed based on the parents' relation of each message given the whole conversation, which is only available during the training stage. in order to deal with the lack of masks during the inference stage, they construct the pseudo mask label based on the predicted relations between any message-pair. however, the pseudo mask label cannot introduce reliable conversation structure information, especially when models cannot achieve a perfect prediction performance on relevant datasets. in this work, we follow this two-step approach framework and propose a hierarchical bert-based model, named dialogue bert (dialbert) for dialogue disentanglement. dialbert first use bert @xcite to capture the matching information in each message pair. then, a contextlevel bilstm is employed to aggregate and incorporate the context information. the semantics similarity of each message pair is measured by calculating their matching scores, and the message that has the highest matching score with the target message is regarded as the parent message of it. in addition, we aim at introducing and making use of conversation structures to help dialbert to make decision by training dialbert with two extra types of loss of conversation-structure loss and tree-structure loss. in this way, the model can implicitly learn and leverage conversation structures without being restricted to the lack of explicit access to such structures during inference. we evaluate our method on two large datasets releasaed by @xcite and @xcite respectively. experimental results show our proposed method outperforms previous methods in terms of various evaluation metrics. in summary, our contributions in this paper are three-fold: (1) a hierarchical model named dial-bert is proposed for dialogue disentanglement. ## related work the research for dialogue disentanglement dates back to @xcite which conducted a study of voice conversations among 8-10 people with an average of 1.76 activate conversations at any given time. in recent studies, the mainstream method for dialogue disentanglement is the two-step approach: firstly, a neural network is used to determine the relation between two messages. then a clustering algorithm is adopted to separate messages into different conversations. in the first step, @xcite used recurrent neural networks(rnns) to model adjacent messages. @xcite was the first work that used convolutional neural networks to estimate the conversationlevel similarity between closely posted messages. @xcite proposed a masked hierarchical transformer based on bert to calculate the matching score by using conversation structures. in addition to neural networks, statistical @xcite and linguistic features @xcite @xcite have also been used in the existing research. in the clustering stage, some research proposed the clustering algorithm by using threshold such as @xcite . most studies grouped two messages with the highest matching score into the same conversation. in our study, we follow this mainstream setting. ## problem formulation given a dataset d, m (1) , m (2) , ..., m (n ) represents a list of messages and each message belongs to a specific conversation. following the setting of previous studies @xcite @xcite @xcite , in order to find the parent message of a target message, t -1 messages occurring before this target message and itself form the context message set of this target message. the target message is a word sequence that can be represented by m @xmath0, m t 2 , ..., m t n t , and each context message is a word sequence that can be represented by m @xmath1, m i 2 , ..., m i n i , where n t and n i are the sequence length of messages and @xmath2, 2, ..., t }. every target message has a label @xmath3, 2, ..., t } indicating which message in context range is the parent message of the target message (each message has and only has one parent message). our goal is to learn a prediction model to predict which message in m 1 , m 2 , ..., m t is the parent message of the target message m t for @xmath4, 2, .., n }. note that if the target message is the first message of a conversation, the parent of the target message is itself. ## conclusions in this paper, we propose a novel framework for dialogue disentanglement. different from previous work, we integrate both local and global semantics by proposing an adapted hierarchical bert-based model (dialbert) to disentangle conversations. moreover, in order to make use of conversation structures, we finetune our model with two losses (i.e., conversation-structure loss and tree-structure loss). we evaluate our method on two large datasets. results show that our method achieves a new state-of-the-art performances on both datasets and outperforms models from previous work with a substantial margin. in the future, we will design non-heuristic methods for modeling the conversation structure with less hyperparameters which is a challenge worth exploring.