H eter MPC : A Heterogeneous Graph Neural Network for Response Generation in Multi-Party Conversations (2022)

Abstract: Recently, various response generation models for two-party conversations have achieved impressive improvements, but less effort has been paid to multi-party conversations (MPCs) which are more practical and complicated. Compared with a two-party conversation where a dialogue context is a sequence of utterances, building a response generation model for MPCs is more challenging, since there exist complicated context structures and the generated responses heavily rely on both interlocutors (i.e., speaker and addressee) and history utterances. To address these challenges, we present HeterMPC, a heterogeneous graph-based neural network for response generation in MPCs which models the semantics of utterances and interlocutors simultaneously with two types of nodes in a graph. Besides, we also design six types of meta relations with node-edge-type-dependent parameters to characterize the heterogeneous interactions within the graph. Through multi-hop updating, HeterMPC can adequately utilize the structural knowledge of conversations for response generation. Experimental results on the Ubuntu Internet Relay Chat (IRC) channel benchmark show that HeterMPC outperforms various baseline models for response generation in MPCs.

## introduction enabling dialogue systems to converse naturally with humans is a challenging yet intriguing problem of artificial intelligence and has attracted increasing attention due to its promising potentials and alluring commercial values @xcite @xcite . a large number of researchers have focused on building dialogue generation models with various neural networks. at first, researchers mostly focused on dialogues between two participants @xcite @xcite . recently, researchers have paid more attention to a more practical and challenging scenario involving more than two participants, which is well known as multi-party conversations (mpcs) @xcite @xcite @xcite . utterances in a two-party conversation are posted one by one between two interlocutors, constituting a sequential information flow. different from that, utterances in an mpc can be spoken by anyone and address anyone else in this conversation, which constitutes a graphical information flow as shown in figure 1 . although sequence-to-sequence (seq2seq) models @xcite are effective at modeling sequential dialogues, they fall short of modeling graph-structured ones. to overcome this drawback, @xcite first proposed a graph-structured network (gsn) to encode utterances based on the graph topology rather than the sequence of their appearances. the graph established in gsn was homogeneous, where nodes represented only utterances. however, interlocutors are also important components of mpcs. there exist complicated interactions between interlocutors, and between an utterance and an interlocutor. furthermore, when passing messages over a graph, a bidirectional information flow algorithm was designed for gsn. since both the forward and backward information flows employed the same model structure and parameters, this algorithm cannot distinguish the "reply" or "replied-by" relations between two connected utterance nodes. also, information flows along both directions are independently propagated, so that a graph node cannot be jointly updated at a single propagation step. on account of above issues, we propose a heterogeneous graph-based neural network for response generation in mpcs, named hetermpc. first, a heterogeneous graph is designed which employs two types of nodes to represent utterances and interlocutors respectively. different from previous methods that built a homogeneous graph modeling only utterances, utterances and interlocutors are modeled simultaneously in hetermpc, so that the complicated interactions between interlocutors, between utterances, and between an interlocutor and an utterance can be explicitly described. in order to characterize the heterogeneous attention over each (source, edge, target) triple, model parameters dependent on both types of nodes and edges are introduced when calculating attention weights and passing messages. specifically, we introduce six types of meta relations for modeling different edges including "reply" and "repliedby" between two utterances, "speak" and "spokenby" between an utterance and a speaker, and "address" and "addressed-by" between an utterance and an addressee. with these node-edge-typedependent structures and parameters, hetermpc can better utilize the structural knowledge of conversations for node representation and response generation than conventional homogeneous graphs. finally, transformer is employed as the backbone of hetermpc and its model parameters can be initialized with plms to take advantage of the recent breakthrough on pre-training. we evaluate hetermpc on the ubuntu internet relay chat (irc) channel benchmark released by @xcite . experimental results show that hetermpc outperforms gsn @xcite , gpt-2 @xcite , bert @xcite and bart @xcite by significant margins in terms of both automated and human evaluation metrics, achieving a new state-of-the-art performance for response generation in mpcs. in summary, our contributions in this paper are three-fold: 1) to the best of our knowledge, this paper is the first exploration of using heterogeneous graphs for modeling conversations; 2) a transformer-based heterogeneous graph architecture is introduced for response generation in mpcs, in which two types of nodes, six types of meta relations, and node-edge-type-dependent parameters are employed to characterize the heterogeneous properties of mpcs; 3) experimental results show that our proposed model achieves a new stateof-the-art performance of response generation in mpcs on the ubuntu irc benchmark. heterogeneous graph neural network early studies on graph neural networks (gnns) focused on homogeneous graphs where a whole graph is composed of a single type of nodes. however, graphs in real-world applications usually come with multiple types of nodes, namely heterogeneous information networks (hins) or heterogeneous graphs @xcite . recently, researchers have attempted to extend gnns to model heterogeneity. for example, @xcite al. (2019) adopted different rnns for different types of nodes to integrate multi-modal features. wang et al. (2019) extended graph attention networks by maintaining different weights for different metapath-defined edges. hu et al. (2020) proposed heterogeneous graph transformer (hgt) to model heterogeneity by maintaining dedicated representations for different types of nodes and edges. in addition, heterogeneous graphs have also been applied to many nlp tasks, such as multi-hop reading comprehension @xcite ), text classification @xcite and document summarization @xcite . previous studies have verified the superiority of modeling mpcs with homogeneous graphs considering only utterances. we claim that it is indeed necessary to model a complex information flow in mpcs shown in figure 1 with a heterogeneous graph, since a homogeneous one cannot explicitly model the relationships of multiple utterances spoken by or addressing an interlocutor. nowadays, hins have been widely used in many nlp tasks. to the best of our knowledge, this paper makes the first attempt to build a heterogeneous graphbased neural network considering utterances and interlocutors simultaneously for response generation in mpcs. in addition, we introduce many task-specific modelings for mpcs such as graph construction and node updating which will be elaborated in the model section. ## problem formulation the task of response generation in mpcs is to generate an appropriate response r given the conversation history, the speaker of a response, and which utterance the response is going to reply to, which can be formulated as: here, g is a heterogeneous graph containing both history conversation and the response to be generated. the speaker and addressee of the response are known and its contents are masked. the response tokens are generated in an autoregressive way. r k and @xmath0. |r| is the length of r. this example demonstrates that i j speaks u n replying u m that is spoken-by i i . we will introduce how to construct the graph and how to model the probability in eq. ( 1 ) given the built graph in the next section. ## hetermpc model hetermpc adopts an encoder-decoder architecture consisting of stacked encoder and decoder layers for graph-to-sequence learning @xcite . the graph encoder is designed to capture conversation structures and output the representations of all nodes in a graph that are fed to the decoder for response generation. ## conclusion we present hetermpc to model complicated interactions between utterances and interlocutors in mpcs with a heterogeneous graph. two types of graph nodes and six types of edges are designed. node-edge-type-dependent parameters are introduced for better utilizing the structural knowledge of conversations during node updating. results show that hetermpc outperforms baselines by significant margins, achieving a new state-of-theart performance for response generation in mpcs on the ubuntu irc benchmark. in the future, we will explore better ways of maximizing feature distribution differences to model heterogeneity.