Pre-training Multi-party Dialogue Models with Latent Discourse Inference (2023)

Abstract: Multi-party dialogues are more difficult for models to understand than one-to-one two-party dialogues, since they involve multiple interlocutors, resulting in interweaving reply-to relations and information flows. To step over these obstacles, an effective way is to pre-train a model that understands the discourse structure of multi-party dialogues, namely, to whom each utterance is replying. However, due to the lack of explicitly annotated discourse labels in multi-party dialogue corpora, previous works fail to scale up the pre-training process by putting aside the unlabeled multi-party conversational data for nothing. To fully utilize the unlabeled data, we propose to treat the discourse structures as latent variables, then jointly infer them and pre-train the discourse-aware model by unsupervised latent variable inference methods. Experiments on multiple downstream tasks show that our pre-trained model outperforms strong baselines by large margins and achieves state-of-the-art (SOTA) results, justifying the effectiveness of our method. The official implementation of this paper is available at https://github.com/EricLee8/MPD_EMVI .

## introduction dialogue system is an important area that has been studied for a long time in natural language processing field. different from plain texts, dialogues are harder for models to understand since they are full of informal, colloquial expressions, and many ellipses @xcite @xcite . among them, multi-party dialogues are even more complex since they involve multiple interlocutors, resulting in interweaving reply-to relations and information flows @xcite @xcite . specifically, in multi-party dialogues, the current utterance can be a reply to any preceding utterance in the dialogue history, forming complex discourse structures. intuitively, it is important for models to perceive the discourse structures, or in other words, to whom each utterance is replying, when comprehending multi-party dialogues. this intuition is in line with the process we humans participate in multi-party dialogues: we first read or listen to the dialogue history, knowing who speaks what to whom, then choose an utterance as the addressee, and finally utter a response. literature has also justified that incorporating the discourse knowledge into models is beneficial for better understanding multi-party dialogues @xcite @xcite . unfortunately, the process of choosing addressees is a naturally unobservable action, resulting in a large amount of multi-party conversational data without addressee labels. in this work, we focus on leveraging the unlabeled data to pre-train a model for multi-party dialogue understanding. to utilize the discourse structure, previous works seek help from human laborers to annotate the addressee labels on small datasets, where they either explicitly model the discourse structure using graph neural networks or multi-task learning @xcite @xcite @xcite , or attempt to pretrain a model using objectives that are related to addressees by supervised learning @xcite . these works heavily rely on annotated addressee labels, which are rare in practice since the annotation process requires large amounts of human resources. as a result, they fail to be practical in real-world applications and are hard to scale up by utilizing more unlabeled multi-party conversational data. to make full use of the unlabeled corpora, a natural idea is to treat the unobservable discourse structure (reply-to relations) as latent variables, then adopt latent variable models to jointly infer them and optimize the discourse-aware models. how-ever, it is not that simple when it comes to practice. for the expectation-maximization (em) algorithm, the posterior distribution of the reply-to relations is intractable since it requires a square-level time complexity. if we turn to variational inference (vi) for help, the choice of the categorical prior distribution of the reply-to relations becomes troublesome: naive assumptions such as uniform distributions are too weak to make the training process converge. to step over the above obstacles, we subtly combine the single-turn em algorithm and multi-turn vi into a two-stage pre-training strategy. in the first stage, we adopt the em algorithm to jointly model the context-response matching objective and singleturn addressee inference, which requires only a linear time complexity and can preliminarily guide the model to a relatively good converging point with utterance-level knowledge. in the second stage, we extend the latent variables from single-turn addressees to multi-turn reply-to relations and optimize the model via both the em algorithm and vi framework, where the prior distribution of the reply-to relations is no longer troublesome since it can be derived exactly from the e-steps. this stage further enhances the model with discourse-level knowledge and guides it converge to a better point. to sum up, the contributions of this work are: • we successfully scale up the pre-training for multi-party dialogue understanding by leveraging the huge amounts of multi-party conversational corpora without addressee labels, while previous methods fail to work on these corpora. ## methodology in general, figure 1 illustrates the overview of the proposed two-stage pre-training strategy. the left part illustrates the single-turn expectation-maximization process, where we iteratively conduct e-steps to infer the latent addressee z t (leftupper part and the green arrow), and m-steps to optimize the model via addressee-aware contextresponse matching (crm) objective (left-lower part and the orange arrow). the right part illustrates the multi-turn variational inference process, which is incorporated into the em framework in the second pre-training stage. we extend the latent variables from the single-turn addressees to multiturn addressee-graphs, and jointly optimize the discourse-aware context-response matching model (the blue arrow) and the graph-prediction model q ϕ by variational inference. in the next sections, we will introduce the two pre-training stages in detail. ## experiments in this section, we introduce the experimental settings and present the results on downstream tasks. ## analyses in this section, we make in-depth analyses to investigate more insights from our method. ## conclusion in this paper, we point out that the lack of annotated addressee labels hinders the scaling-up of multi-party dialogue pre-training. to overcome this obstacle, we propose to utilize the unlabeled datasets by combining the em algorithm and variational inference to jointly infer the discourse labels and pre-train the model with discourse-aware objectives on different granularities. experimental results and extensive analyses have justified the effectiveness and transferability of our model on multiple downstream tasks.