MPC - BERT : A Pre-Trained Language Model for Multi-Party Conversation Understanding (2021)

Abstract: Recently, various neural models for multi-party conversation (MPC) have achieved impressive improvements on a variety of tasks such as addressee recognition, speaker identification and response prediction. However, these existing methods on MPC usually represent interlocutors and utterances individually and ignore the inherent complicated structure in MPC which may provide crucial interlocutor and utterance semantics and would enhance the conversation understanding process. To this end, we present MPC-BERT, a pre-trained model for MPC understanding that considers learning who says what to whom in a unified model with several elaborated self-supervised tasks. Particularly, these tasks can be generally categorized into (1) interlocutor structure modeling including reply-to utterance recognition, identical speaker searching and pointer consistency distinction, and (2) utterance semantics modeling including masked shared utterance restoration and shared node detection. We evaluate MPC-BERT on three downstream tasks including addressee recognition, speaker identification and response selection. Experimental results show that MPC-BERT outperforms previous methods by large margins and achieves new state-of-the-art performance on all three downstream tasks at two benchmarks.

## introduction building a conversational agent with intelligence has drawn significant attention from both academia and industry. most of existing methods have studied understanding conversations between two participants, aiming to return an appropriate response either in a generation-based @xcite 2015; @xcite @xcite or retrieval-based manner @xcite @xcite @xcite . recently, researchers have paid more attention to a more practical and challenging scenario involving more than two participants, which is well known as multiparty conversation (mpc) @xcite @xcite . an instance of mpc always contains complicated interactions between interlocutors, between utterances and between an interlocutor and an utterance. therefore, it is challenging to model the conversation flow and fully understand the dialogue content. existing studies on mpc learn the representations of interlocutors and utterances with neural networks, and their representation spaces are either separate @xcite or interactive @xcite . however, the semantics contained in the interlocutor and utterance representations may not be effectively captured as they are from two different representation spaces. recently, to take advantage of the breakthrough in pre-training language models (plms) for natural language understanding, some studies proposed to integrate the speaker @xcite or topic @xcite information into plms. despite of the performance improvement on response selection, these models still overlook the inherent relationships between utterances and interlocutors, such as "address-to". furthermore, most existing studies design models for each individual task in mpc (e.g., addressee recognition, speaker identification and response prediction) separately. intuitively, these tasks are complementary among each other. making use of these tasks simultaneously may produce better contextualized representations of interlocutors and utterances, and would enhance the conversation understanding, but is neglected in previous studies. on account of above issues, we propose mpc-bert which jointly learns who says what to whom in mpc by designing self-supervised tasks for plms, so as to improve the ability of plms on mpc understanding. specifically, the five designed tasks includes reply-to utterance recognition, identical speaker searching, pointer consistency distinction, masked shared utterance restoration and shared node detection. the first three tasks are designed to model the interlocutor structure in mpc in a semantics-to-structure manner. in the output of mpc-bert, an interlocutor is described through the encoded representations of the utterances it says. thus, the representations of utterance semantics are utilized to construct the conversation structure in these three tasks. on the other hand, the last two tasks are designed to model the utterance semantics in a structure-to-semantics manner. intuitively, the conversation structure influences the information flow in mpc. thus, the structure information can also be used to strengthen the representations of utterance semantics in return. in general, these five self-supervised tasks are employed to jointly train the mpc-bert in a multi-task learning framework, which helps the model to learn the complementary information among interlocutors and utterances, and that between structure and semantics. by this means, mpc-bert can produce better interlocutor and utterance representations which can be effectively generalized to multiple downstream tasks of mpc. to measure the effectiveness of these selfsupervised tasks and to test the generalization ability of mpc-bert, we evaluate it on three downstream tasks including addressee recognition, speaker identification and response selection, which are three core research issues of mpc. two benchmarks based on ubuntu irc channel are employed for evaluation. one was released by @xcite . the other was released by @xcite and has three experimental settings according to session lengths. experimental results show that mpc-bert outperforms the current state-of-the-art models by margins of 3.51%, 2.86%, 3.28% and 5.36% on the test sets of these two benchmarks respectively in terms of the session accuracy of addressee recognition, by margins of 7.66%, 2.60%, 3.38% and 4.24% respectively in terms of the utterance precision of speaker identification, and by margins of 3.82%, 2.71%, 2.55% and 3.22% respectively in terms of the response recall of response selection. in summary, our contributions in this paper are three-fold: (1) mpc-bert, a plm for mpc understanding, is proposed by designing five selfsupervised tasks based on the interactions among utterances and interlocutors. (2) three downstream tasks are employed to comprehensively evaluate the effectiveness of our designed self-supervised tasks and the generalization ability of mpc-bert. (3) our proposed mpc-bert achieves new state-ofthe-art performance on all three downstream tasks at two benchmarks. ## related work existing methods on building dialogue systems can be generally categorized into studying twoparty conversations and multi-party conversations (mpc). in this paper, we study mpc. in addition to predicting utterances, identifying the speaker and recognizing the addressee of an utterance are also important tasks for mpc. @xcite first proposed the task of addressee and response selection and created an mpc corpus for studying this task. zhang et al. (2018a) proposed si-rnn, which updated speaker embeddings role-sensitively for addressee and response selection. meng et al. (2018) proposed a task of speaker classification as a surrogate task for speaker modeling. le et al. (2019) proposed a who-to-whom (w2w) model to recognize the addressees of all utterances. hu et al. (2019) proposed a graph-structured network (gsn) to model the graphical information flow for response generation. wang et al. (2020) proposed to track the dynamic topic for response selection. generally speaking, previous studies on mpc cannot unify the representations of interlocutors and utterances effectively. also, they are limited to each individual task, ignoring the complementary information among different tasks. to the best of our knowledge, this paper makes the first attempt to design various self-supervised tasks for building plms aiming at mpc understanding, and to evaluate the performance of plms on three downstream tasks as comprehensively as possible. ## mpc-bert and self-supervised tasks an mpc instance is composed of a sequence of (speaker, utterance, addressee) triples, denoted as {(s n , u n , a n )} n @xmath0, where n is the number of turns in the conversation. our goal is to build a pre-trained language model for universal mpc understanding. given a conversation, this model is expected to produce embedding vectors for all utterances which contain not only the semantic information of each utterance, but also the speaker and addressee structure of the whole conversation. thus, it can be effectively adapted to various downstream tasks by fine-tuning model parameters. ## conclusion in this paper, we present mpc-bert, a pre-trained language model with five self-supervised tasks for mpc understanding. these tasks jointly learn who says what to whom in mpcs. experimental results on three downstream tasks show that mpc-bert outperforms previous methods by large margins and achieves new state-of-the-art performance on two benchmarks.