P rophet C hat: Enhancing Dialogue Generation with Simulation of Future Conversation (2022)

Abstract: Typical generative dialogue models utilize the dialogue history to generate the response. However, since one dialogue utterance can often be appropriately answered by multiple distinct responses, generating a desired response solely based on the historical information is not easy. Intuitively, if the chatbot can foresee in advance what the user would talk about (i.e., the dialogue future) after receiving its response, it could possibly provide a more informative response. Accordingly, we propose a novel dialogue generation framework named ProphetChat that utilizes the simulated dialogue futures in the inference phase to enhance response generation. To enable the chatbot to foresee the dialogue future, we design a beam-search-like roll-out strategy for dialogue future simulation using a typical dialogue generation model and a dialogue selector. With the simulated futures, we then utilize the ensemble of a history-to-response generator and a future-to-response generator to jointly generate a more informative response. Experiments on two popular open-domain dialogue datasets demonstrate that ProphetChat can generate better responses over strong baselines, which validates the advantages of incorporating the simulated dialogue futures.

## introduction recent years have witnessed a surge of interest in building open-domain chatbots using generative approaches @xcite @xcite . these prevailing methods typically utilize dialogue histories as the dialogue context to generate the response via maximum likelihood estimation. different from directed text generation tasks like machine translation where the target sentences are strictly constrained by the source sentence @xcite , the dialogue history and the response in chitchat conversations are loosely coupled @xcite . in other words, open-domain chatbots often have more "freedom" to decide what to respond since there often exists multiple distinct responses that can appropriately answer the given utterance. however, we argue that such excessive "freedom" also reveals that the dialogue history only may not contain enough information to generate a desired response that is informative and easy to reply to. if provided with enriched dialogue contexts that contain more useful dialogue cues, it could be easier for the model to chat with a human. so here comes the questions: what kind of dialogue cues is complementary to dialogue histories, and how to obtain and use them. recent studies in representation learning have demonstrated that when representing a token in a sentence, considering the tokens on its right side in addition to its left side can bring significant improvement @xcite . similar findings also appear in directed text generation where the future tokens on the right side can be beneficial to generate the current token @xcite @xcite . sharing the same spirit, we pursue to use the "right side" information, which is the dialogue future in our task, as the complementary dialogue cue to enhance the generation of the current response. intuitively, if the chatbot can be told in advance what the user would probably talk about (i.e., the dialogue future) after receiving its response, it only needs to provide a response that can smoothly connect the history and the future. to verify whether the dialogue future can act as the complementary dialogue cue, we conduct empirical studies. we find that using a dialogue generation model to learn the reverse dialogue flow (i.e., using the future to generate the response) is quite effective. furthermore, when utilizing the ensemble of the history-to-response generation model and the future-to-response generation model to generate the response conditioned on both the history and the gold future, the quality of the generated response surely improves. though effective, the ground truth dialogue future is inaccessible in the inference phase. therefore, all existing works in this line choose to leverage the dialogue future only in the training phase @xcite , leaving the inference phase unchanged. we argue that explicitly providing the possible dialogue futures in the inference phase can offer more direct help for the generation of the current response. to enable the incorporation of dialogue futures into response generation in the inference phase, we propose a response generation framework namely prophetchat by answering two questions: how to acquire the future and how to use it. figure 1 shows the generation process of prophetchat. it consists of a history-to-response model (denoted as the forward model), a future-toresponse model (denoted as the backward model), an ensemble gate, and a dialogue selector. given a dialogue history, we first utilize an effective beamsearch-like roll-out strategy to simulate possible dialogue futures. concretely, the forward model first generates a batch of n possible responses based on the dialogue history. the dialogue selection model then comes to pick up the k-best responses. we further generate n possible futures for each of the picked responses, resulting in k • n futures. the selection model again picks up the k-best futures which are of higher quality compared with randomly sampled ones. next, conditioned on both the history and the simulated future, we employ the forward model and the backward model to jointly generate the response by summing the per-step output probability distributions of the two models using a calculated weight. the weight is obtained by a trainable gate that learns to balance the trade-off between history and future information. finally, we gather the k responses generated solely based on the history and the k • n responses generated based on both the history and the future, and use the selector to choose the top-ranked one as the final response. since the ensemble generation model relies on the selector to sequentially select the response and the future, and the ensemble generation model also needs to learn how to balance the history and the future information given the selected future, we jointly train the whole model to make each module better collaborate with others to fulfill the ultimate goal: to maximize the likelihood of the gold response estimated by the ensemble generation model given the history and the selected future. we train the ensemble generation model directly using mle objective while adopting reinforcement learning to tune the selector. our contributions in this paper are three folds: dialogue system. open-domain response generation has long been the research hot spot. recently, various efforts have been made to generate informative and diverse responses by introducing effective architectures and learning objectives and by incorporating external knowledge. zhao et al. (2017); gu et al. (2018) applied cvae to model the variability of responses. li et al. (2016b); zhang et al. (2018a); saleh et al. (2020) adopted reinforcement learning to encourage the model to generate desired responses through carefully designed reward functions. zhang et al. (2018b); chan et al. (2019); zheng et al. (2020); li et al. (2021) exploited persona information to improve the coherence of the response. zhou et al. (2018); song et al. (2019); shen and feng (2020) considered emotions when generating the response. dinan et al. (2018); lian et al. (2019); zhao et al. (2020a,b); li et al. (2020) conditioned the response generation model with knowledge. different from the above works that aimed to design specific history-to-response generation models, we propose a response generation framework where possible dialogue futures are utilized in the inference phase with the help of an effective future simulation strategy. future modeling. there are various scenarios where considering future information is useful. in text generation, serdyuk et al. (2017) proposed a twin network to regularize the hidden states of the left-to-right decoder with the future-aware right-toleft decoder. zhang et al. (2018c) used the targetside hidden states generated by the right-to-left decoder to help the right-to-left decoder during translation so that the target-side future information can help avoid under-translation. different from these works that consider the right side tokens as the future for the current token, we define "future" as the next dialogue utterance of the current response in a dialogue session. in response generation, feng et al. (2020a) proposed to use gold futures as the conditions of two discriminators and adopted adversarial training to encourage diversity. feng et al. (2020b) employed gold dialogue futures to learn a future-aware teacher model and transferred the knowledge to a history-to-response student model via imitation learning. these works only use the future information in the training phase, while we utilize the simulated dialogue future in the inference phase to provide the history-to-response generation model with direct help. ## preliminaries in this section, we introduce the major off-the-shelf components in our framework. dialogpt @xcite ) is a gpt-based response generation model pre-trained on large-scale open-domain dialogue corpus by maximizing the likelihood of the successive dialogue utterances (i.e., the forward dialogue flow) given the initial dialogue history. while trained on the same corpus with the same architecture, dialogpt-mmi is trained on the backward dialogue flow where the order of the utterances in a dialogue are reversed. we adopt dialogpt as the forward generator and dialogpt-mmi as the backward generator. grade @xcite ) is a graph-enhanced dialogue evaluation model that uses both utterancelevel contextualized representations and topic-level graph representations to evaluate the response. as it is one of the sota dialogue evaluation models, we choose it as our dialogue selector. ## conclusion we propose a novel response generation framework that utilizes the simulated dialogue futures in the inference phase to enhance response generation. to acquire the dialogue futures, we design an effective beam-search-like roll-out strategy using a history-to-response dialogue generation model and a dialogue selector. to make use of the simulated future, we use the dynamic ensemble of the historyto-response and the future-to-response generation model. experiment results demonstrate the effectiveness of our proposed method on two popular datasets. in the future, we plan to enable our future simulation method to simulate multiple turns of dialogue futures.