Back to the Future: Bidirectional Information Decoupling Network for Multi-turn Dialogue Modeling (2022)

Abstract: Multi-turn dialogue modeling as a challenging branch of natural language understanding (NLU), aims to build representations for machines to understand human dialogues, which provides a solid foundation for multiple downstream tasks. Recent studies of dialogue modeling commonly employ pre-trained language models (PrLMs) to encode the dialogue history as successive tokens, which is insufficient in capturing the temporal characteristics of dialogues. Therefore, we propose Bidirectional Information Decoupling Network (BiDeN) as a universal dialogue encoder, which explicitly incorporates both the past and future contexts and can be generalized to a wide range of dialogue-related tasks. Experimental results on datasets of different downstream tasks demonstrate the universality and effectiveness of our BiDeN.

## introduction multi-turn dialogue modeling as one of the core tasks in natural language understanding, aims to build representations for machines to understand human dialogues. it is the foundation of solving multiple dialogue-related tasks such as selecting a response @xcite @xcite , answering questions @xcite @xcite , or making a summarization according to the dialogue history @xcite . dialogue contexts possess their intrinsic nature of informal, colloquial expressions, discontinuous semantics, and strong temporal characteristics @xcite @xcite , making them harder for machines to understand compared to plain texts @xcite @xcite . to tackle the aforementioned obstacles, most of the existing works on dialogue modeling have made efforts from three perspectives. the first group of works adopt a hierarchical encoding strategy by first encoding each utterance in a dialogue separately, then making them interact with each other by an utterance-level interaction module @xcite @xcite . this strategy shows sub-optimal to model multi-turn dialogue owing to the neglect of informative dialogue contexts when encoding individual utterances. the second group of works simply concatenate all the utterances chronologically as a whole (together with response candidates for the response selection task), then encode them using pre-trained language models (prlms) @xcite . this encoding pattern has its advantage of leveraging the strong interaction ability of self-attention layer in transformer @xcite to obtain token-level contextualized embedding, yet ignores utterancelevel modeling in dialogue contexts. @xcite also demonstrate that the simple concatenation is likely to ignore the conversational dynamics across utterances in the dialogue history. the third group of works employ a pack and separate method by first encoding the whole dialogue context using prlms, then separating them to form representations of different granularities (turn-level, utterance-level, etc.) for further interaction @xcite . unfortunately, all works mentioned above paid little attention to the temporal characteristics of dialogue texts, which are supposed to be useful and essential for modeling multi-turn dialogues. different from previous works and to fill the gap of effectively capturing the temporal features in dialogue modeling, we propose a simple but effective bidirectional information decoupling network (biden), which explicitly incorporates both #person1#: hi, della. how long are you going to stay here? #person2#: only 4 days. i know that's not long enough, but i have to go to london after the concert here at the weekend. #person1#: i'm looking forward to your concert very much. can you tell me where you sing in public for the first time? the past and future information from the dialogue contexts. our proposed model can serve as a universal dialogue encoder and be generalized to a wide range of downstream dialogue-related tasks covering classification, retrieval, and generative styles as illustrated in figure 1 . in detail, we first concatenate all the utterances to form a dialogue context, then encode it with a prlm. after obtaining the representations output by the prlm, three additional parameterindependent information decoupling layers are applied to decouple three kinds of information entangled in the dialogue representations: past-tocurrent, future-to-current, and current-to-current information. respectively, the past-to-current information guides the modeling of what the current utterance should be like given the past dialogue history, the future-to-current information guides the modeling of what kind of current utterance will lead to the development of the future dialogue, and the current-to-current information guides the modeling of the original semantic meaning resides in the current utterance. after obtaining these representations, we fuse them using a mixture of experts (moe) mechanism @xcite to form the final dialogue history representations. let's focus again on figure 1 and take the response selection task as example. when modeling the three candidate responses, the past-to-current information of the responses and the future-to-current information of each utterance in the context will detect incoherent temporal features in response a and c, and coherent feature of response b, which help the model to deduce the final answer. we conduct experiments on three datasets that belong to different types of dialogue-related tasks: multi-turn dialogue reasoning @xcite ) for response selection, molweni @xcite for extractive question-answering (qa) over multi-turn multi-party dialogues, and dialogsum @xcite for dialogue summarization. experimental results on these three datasets show that biden outperforms strong baselines by large margins and achieves new state-ofthe-art results. the contributions of our work are three-fold: • the proposed model can serve as a universal dialogue encoder and easily be applied to various downstream dialogue-related tasks. to the best of our knowledge, this is the first paper that introduces the back-and-forth reading strategy @xcite to the modeling of temporal characteristics of dialogues. ## methodology in this part, we introduce biden and its three modules, whose overview is shown in figure 2 . the left part is a pre-trained language model encoder. given a sequence of input tokens, the prlm encoder yields their contextualized representations. the middle part is a bidirectional information de-coupling module (bidm), which decouples the entangled representations into three channels for each utterance: future-to-current representations, pastto-current representations and current-to-current representations. the right part is a mixture of experts (moe) module, which calculates an expert distribution to dynamically fuse the three kinds of representations for each token. in the following sections, we will introduce them in detail, respectively. ## analysis in this section, we conduct experiments on mutual dataset to get an in-depth understanding of biden. ## conclusion in this paper, we propose bidirectional information decoupling network (biden) to explicitly model the indispensable temporal characteristics of multi-turn dialogues, which have been ignored for a long time by existing works. biden shows simple but effective to serve as a universal dialogue encoder for a wide range of dialogue-related tasks. experimental results and comprehensive analyses on several benchmark datasets have justified the effectiveness of our model.