Conversations Are Not Flat: Modeling the Dynamic Information Flow across Dialogue Utterances (2021)

Abstract: Nowadays, open-domain dialogue models can generate acceptable responses according to the historical context based on the large-scale pre-trained language models. However, they generally concatenate the dialogue history directly as the model input to predict the response, which we named as the flat pattern and ignores the dynamic information flow across dialogue utterances. In this work, we propose the DialoFlow model, in which we introduce a dynamic flow mechanism to model the context flow, and design three training objectives to capture the information dynamics across dialogue utterances by addressing the semantic influence brought about by each utterance in large-scale pre-training. Experiments on the multi-reference Reddit Dataset and DailyDialog Dataset demonstrate that our DialoFlow significantly outperforms the DialoGPT on the dialogue generation task. Besides, we propose the Flow score , an effective automatic metric for evaluating interactive human-bot conversation quality based on the pre-trained DialoFlow, which presents high chatbot-level correlation ( r=0.9 ) with human ratings among 11 chatbots. Code and pre-trained models will be public.

## introduction recent intelligent open-domain chatbots @xcite @xcite have made substantial progress thanks to the rapid development of the large-scale pre-training approaches @xcite @xcite and the large amount of conversational data @xcite @xcite . however, figure 1 : illustration of the dynamic information flow in the semantic space. c k (gray dotted line) denotes the dense representation of dialogue history which we named as context. i k (pink solid line) denotes the semantic influence brought about by the k-th utterance, which is the difference between c k and c k+1 . most of the previous work on dialogue history modeling mainly fall into two groups. one group of works generally concatenate the dialogue history as the model input and predict the response @xcite @xcite , named as flat pattern, which is commonly adopted in the large-scale pre-training. however, @xcite demonstrate that flat concatenation is likely to ignore the conversational dynamics across utterances in the dialogue history. another group of works employ hierarchical modeling to encode the dialogue history @xcite @xcite , in which the utterances are separately encoded and then fed into an utterance-level encoder. these approaches lack the history information when encoding each individual utterance, while the history information is essential for understanding dialogue utterances. thus, all the aforementioned methods are deficient in modeling the dynamic information in the dialogue history. in this work, inspired by the human cognitive process that humans always consider the goal or influence of the next response before they continue the conversation @xcite , we propose the dialoflow to model the dynamic information flow in the dialogue history by addressing the semantic influence brought about by each utterance. as shown in figure 1 , we define the dense representation of the dialogue history at different utterances as the contexts (gray dot line) and the context transformation as the semantic influence brought by each utterance. in particular, our dialoflow constructs the process of the utterancelevel history context flow. correspondingly, the semantic influence of each utterance can be measured by the difference between two adjacent contexts, which will be further used to guide the current response generation. practically, we first employ a transformer to encode the whole conversation to get the dense context representation. then we design a unidirectional flow module to capture the context flow on the utterance level, and design three training objectives to model the context flow and measure the semantic influence brought about by each utterance: 1) context flow modeling, which aims to capture the context flow schema. 2) semantic influence modeling, which targets to measure the predicted semantic influence. 3) response generation modeling, which is to generate the response under the guidance of the predicted semantic influence. furthermore, to demonstrate the effect of modeling dynamic information flow in the dialogue understanding, we propose the flow score based on the dialoflow, an automatic reference-free evaluation metric for interactive dialogue evaluation by measuring the semantic influence perplexity. we pre-train the proposed dialoflow on the large-scale reddit comments and conduct experiments on dialogue generation and interactive dialogue quality evaluation. for dialogue generation, dialoflow achieves significant improvements on the reddit multi-reference dataset and the daily-dialog dataset compared to the baseline dialogpt @xcite . for interactive dialogue quality evaluation, our proposed flow score obtains an impressively high chatbot-level correlation (@xmath0.9) with human ratings on 2200 human-bot dialogues from 11 chatbots. our contributions are summarized as follows: ## method the proposed dialoflow models the dynamic information flow in the whole dialogue history by addressing the semantic influence brought about by each utterance in sequence. ## results and analysis in this section, we show the performance of our pre-trained dialoflow model on response generation as well as the performance of flow score on interactive dialogue quality evaluation. ## conclusion and future work in this work, we proposed the dialoflow to model the dynamic information flow across dialogue utterances by addressing the semantic influence brought about by each utterance. specifically, we employed a uni-directional flow module to model the context flow and designed three training objectives to optimize the dialoflow model. besides, upon the di-aloflow, we proposed the flow score, an automatic reference-free evaluation metric for interactive dialogue evaluation, with the pre-trained dialoflow. experiments on response generation and dialogue evaluation all demonstrate that our method could effectively capture the dynamic information flow across utterances. for future work, we would like to apply the dialoflow to the task-oriented dialogue and explore the application on the long text generation, such as the story generation.