GIFT : Graph-Induced Fine-Tuning for Multi-Party Conversation Understanding (2023)

Abstract: Addressing the issues of who saying what to whom in multi-party conversations (MPCs) has recently attracted a lot of research attention. However, existing methods on MPC understanding typically embed interlocutors and utterances into sequential information flows, or utilize only the superficial of inherent graph structures in MPCs. To this end, we present a plug-and-play and lightweight method named graph-induced fine-tuning (GIFT) which can adapt various Transformer-based pre-trained language models (PLMs) for universal MPC understanding. In detail, the full and equivalent connections among utterances in regular Transformer ignore the sparse but distinctive dependency of an utterance on another in MPCs. To distinguish different relationships between utterances, four types of edges are designed to integrate graph-induced signals into attention mechanisms to refine PLMs originally designed for processing sequential texts. We evaluate GIFT by implementing it into three PLMs, and test the performance on three downstream tasks including addressee recognition, speaker identification and response selection. Experimental results show that GIFT can significantly improve the performance of three PLMs on three downstream tasks and two benchmarks with only 4 additional parameters per encoding layer, achieving new state-of-the-art performance on MPC understanding.

## introduction maintaining appropriate human-computer conversation is an important task leaping towards advanced artificial intelligence. most of existing methods have studied understanding conversations between two participants, aiming at returning an appropriate response either in a generation-based @xcite @xcite or retrieval-based manner (wu et al., 2017; zhou et al., 2018; tao et al., 2019; * corresponding author. of an mpc where rectangles denote utterances, and solid lines represent the "reply" relationship between two utterances, and (b) the detailed reply relationships between each utterance and u 3 . @xcite . recently, researchers have paid more attention to a more practical and challenging scenario involving more than two participants, which is well known as multi-party conversations (mpcs) @xcite @xcite @xcite @xcite . unlike twoparty conversations, utterances in an mpc can be spoken by anyone and address anyone else in this conversation, constituting a graphical information flow and various relationships between utterances as shown in figure 1 @xcite . thus, predicting who the next speaker will be @xcite and who the addressee of an utterance is @xcite @xcite are unique and important issues in mpcs. the complicated interactions between interlocutors, between utterances and between an interlocutor and an utterance naturally increase the difficulty of fully understanding mpcs. existing studies on mpc understanding focus on the challenging issue of modeling the complicated conversation structures and information flows. the current stateof-the-art method mpc-bert @xcite proposed to pre-train a language model with two types of self-supervised tasks for modeling interlocutor structures and utterance semantics respectively in a unified framework. the complementary structural and semantic information in mpcs is learned by designing a variety of self-supervised optimization objectives. however, the semantics contained in the interlocutor and utterance representations may not be effectively captured as these supervision signals are placed only on top of language models. during encoding inside language models, the full and equivalent connections among utterances in regular transformer @xcite ignore the sparse but distinctive dependency of an utterance on another, such as "reply-to". despite of the performance improvement with pre-training, mpc-bert still overlooks the inherent mpc graph structure when fine-tuning on various downstream tasks. intuitively, leveraging graph-induced signals when fine-tuning pre-trained language models (plms) may yield better contextualized representations of interlocutors and utterances and enhance conversation understanding, but has been overlooked in previous studies. in light of the above issues, we propose a plugand-play and lightweight method named graphinduced fine-tuning (gift), which can adapt various transformer-based plms and improve their ability for universal mpc understanding. existing transformer-based plms such as bert @xcite are originally designed for processing sequential texts. to distinguish different relationships between utterances, four types of edges (reply-to, replied-by, reply-self and indirectreply) are designed to integrate graph-induced signals in the attention mechanism. these edgetype-dependent parameters are utilized to refine the attention weights and to help construct the graphical conversation structure in transformer. intuitively, the conversation structure influences the information flow in mpcs, thus it can be used to strengthen the representations of utterance semantics. by this means, it can help characterize fine-grained interactions during the internal encoding of plms, and produce better representations that can be effectively generalized to multiple downstream tasks of mpcs. lastly, the proposed method is plug-and-play which can be implemented into various transformer-based plms, and is lightweight which requires only 4 additional parameters per encoding layer. to measure the effectiveness of the proposed gift method and to test its generalization ability, gift is implemented into three plms including bert @xcite , sa-bert @xcite and mpc-bert @xcite . we evaluate the performance on three downstream tasks including addressee recognition, speaker identification and response selection, which are three core research issues of mpcs. two benchmarks based on ubuntu irc channel are employed for evaluation. one was released by @xcite . the other was released by @xcite with three experimental settings according to session lengths. experimental results show that gift helps improve the performance of all three plms on all three downstream tasks. take mpc-bert as an example, gift improved the performance by margins of 0.64%, 1.64%, 3.46% and 4.63% on the test sets of these two benchmarks respectively in terms of utterance precision of addressee recognition, by margins of 6.96%, 23.05%, 23.12% and 22.99% respectively in terms of utterance precision of speaker identification, and by margins of 1.76%, 0.88%, 2.15% and 2.44% respectively in terms of response recall of response selection, achieving new state-of-the-art performance on mpc understanding. in summary, our contributions in this paper are three-fold: (1) a graph-induced fine-tuning (gift) method is proposed to construct and to utilize the inherent graph structure for mpc understanding. ## related work existing methods on building dialogue systems can be generally categorized into studying twoparty conversations and multi-party conversations (mpcs). in this paper, we study mpcs. in addition to predicting the utterance, the tasks of identifying the speaker and recognizing the addressee of an utterance are also important for mpcs. @xcite first proposed the task of addressee and response selection and created an mpc corpus for studying this task. @xcite proposed the speaker interaction rnn, which updated the speaker embeddings role-sensitively for addressee and response selection. @xcite proposed a task of speaker classification as a surrogate task for general speaker modeling. @xcite proposed a who-to-whom (w2w) model to recognize the addressees of all utterances in an mpc. @xcite created a dataset based on ubuntu irc channel which was manually annotated with reply-structure graphs for mpc disentanglement. @xcite proposed a graph-structured neural network (gsn), the core of which is to encode utterances based on the graph topology rather than the sequence of their appearances to model the information flow as graphical. @xcite proposed to track the dynamic topic for response selection. @xcite studied transition-based online mpc disentanglement by modeling semantic coherence within each session and exploring unsupervised co-training through reinforcement learning. @xcite proposed mpc-bert pre-trained with two types of self-supervised tasks for modeling interlocutor structures and utterance semantics. @xcite proposed hetermpc to model the complicated interactions between utterances and interlocutors with a heterogeneous graph. compared with mpc-bert @xcite that is the most relevant to this work, two main differences should be highlighted. first, mpc-bert works on designing various self-supervised tasks for pre-training, while gift works on further improving fine-tuning performance. second, mpc-bert models conversation graph structures by placing self-supervision signals on top of plms, while gift achieves this by alternatively modifying the internal encoding of plms. furthermore, compared with gsn @xcite and het-ermpc @xcite that both attempt to model graphical information flows, it should be noted that there are also two main differences. first, gsn and hetermpc represent each individual utterance as a node vector encoded by either bilstm @xcite or transformer @xcite , and then update via graph neural network-based information passing, while this work integrates graph-induced signals into the fully-connected interactions of transformer over the whole mpc context. second, gsn and hetermpc are designed specifically for mpc response generation, while this work focuses on universal mpc understanding. overall, to the best of our knowledge, this paper makes the first attempt to design a fine-tuning method that leverages graph-induced signals during the internal encoding of transformer-based plms for improving mpc understanding. ## graph-induced fine-tuning (gift) an mpc instance is composed of a sequence of (speaker, utterance, addressee) triples, denoted as {(s n , u n , a n )} n @xmath0, where n is the number of turns in the conversation. our goal is to fine-tune plms for universal mpc understanding. given an mpc, it is expected to produce embedding vectors for all utterances which contain not only the semantic information of each utterance, but also the speaker and addressee structure of the whole conversation. thus, it can be effectively adapted to various tasks by fine-tuning model parameters. ## downstream tasks three downstream tasks are employed to evaluate the mpc understanding as comprehensively as possible, aiming at the issues of addressing whom, who speaking and saying what. when fine-tuning on each downstream task, all parameters are updated. figure 2 shows the input representations and model architectures for three tasks respectively. ## conclusion in this paper, we present graph-induced finetuning (gift), a plug-and-play and lightweight method that distinguishes the relationships between utterances for mpc understanding. the sparse but distinctive dependency of an utterance on another among those in an mpc is modeled by utilizing the edge-type-dependent parameters to refine the attention weights during the internal encoding of plms. experimental results on three downstream tasks show that gift significantly helps improve the performance of three plms and achieves new state-of-the-art performance on two benchmarks. obviously, the addressee labels of utterances in the conversation history are important for building the inherent graph structure required for graphinduced fine-tuning. however, an mpc with a few addressee labels missing is a common issue. in the future, it will be part of our work to investigate the scarcity of addressee labels.