Pretrained Language Models for Dialogue Generation with Multiple Input Sources (2020)

Abstract: Large-scale pretrained language models have achieved outstanding performance on natural language understanding tasks. However, it is still under investigating how to apply them to dialogue generation tasks, especially those with responses conditioned on multiple sources. Previous work simply concatenates all input sources or averages information from different input sources. In this work, we study dialogue models with multiple input sources adapted from the pretrained language model GPT2. We explore various methods to fuse multiple separate attention information corresponding to different sources. Our experimental results show that proper fusion methods deliver higher relevance with dialogue history than simple fusion baselines.

## introduction large-scale pretrained language models @xcite @xcite have achieved outstanding performance on various natural language understanding tasks @xcite . researchers have then utilized them in dialogue generation tasks @xcite @xcite . many of them simply concatenate the input dialogue history and the output response in finetuning, since the pretrained language model only accepts a single sequence as input. however, dialogue generation tasks may involve multiple input sources simultaneously. for example, in personalized or knowledge-grounded dialogue generation @xcite @xcite , a response is generated conditioned on both dialogue history and an auxiliary user profile or knowledge article. despite simple concatenation of all input sources, an important question arises on how we can better adapt a single-input pretrained language model to a multiinput dialogue generation task. some previous work forms an encoder-decoder architecture with both encoder and decoder duplicated from a pretrained language model @xcite . recently, bart @xcite even obtain a complete pretrained model under this architecture directly. taking personalized dialogue generation @xcite as an example, we can treat persona information, dialogue history and previous generated tokens as three different input sources. the former two will be encoded firstly and then combined with the last one in the decoder. in @xcite , the multi-head attention layer in the decoder is copied three times for each input source and mean pooling is used to average results from multiple attentions. this encoder-decoder adaptation is shown to outperform simple concatenation. however, when dialogue history gets longer, this model tends to use less information of each dialogue history utterance to predict the next token. @xcite add an extra weight predictor to combine multiple attention information, but they do not perform experiments using publicly released pretrained models, nor on public datasets, making their results not directly comparable to other work. in this work, we build our dialogue model on the encoder-decoder architecture adapted from the pretrained language model gpt2 @xcite . our main contribution is to empirically study the attention fusion methods for multiple information sources in each decoder layer. three kinds of methods are explored in total. our experimental results show performance improvements on both automatic and human evaluations by using proper attention fusion methods, compared to baselines using concatenation or mean pooling. ## experiment we employ the personachat @xcite dataset in our experiments which has 164,356 utterances in 10,981 dialogues and 1,155 personas. each sample contains dialog history with up to 15 utterances, a gold reply and a persona description with no more than 5 sentences. four kinds of dialogue models using pretrained language models as the initialization are compared: (i) transfertransfo @xcite , a singleinput openai gpt using token type embedding to distinguish different parts of a single concatenated input (persona profile, dialog history, and reply successively). we also replace original gpt in this method with gpt2, denoted as transfergpt2. all gpt2 models used here are small size (12 layers, hidden size is 768). besides, seq2seq model with attention @xcite using 6-layer transformer as the encoder and decoder is also included as an end-to-end single-input baseline. 1 the following automatic metrics are considered in our evaluation: bleu @xcite , meteor @xcite , nist-4, which indicate the gram-level similarity between the references and generated responses. moreover, entropy-4, corpus-level distinct-2 and the average length of replies are used to reflect the diversity of obtained text. in addition, human evaluation is also conducted on 200 dialog pairs in terms of fluency (range: 1 ∼ 3), relevance with dialogue history (h-rel, range: 1 ∼ 3) and consistency with personality (p-consist, {0, 1}). more experiment configurations can be found in appendix a. ## conclusion to handle dialogue generation with multiple input sources, we adapt the pretrained language model gpt2 to an encoder-decoder architecture with multiple independent attentions for different input sources in the decoder. we then investigate several attention fusion methods to obtain a preferable representation for dialogue generation. experiments illustrate that weighting methods promote both auto metrics and dialog history relevance scores annotated by human than baselines using average fusion, while they still maintain the persona consistency scores which outperform single-input models. and such architecture can be extended to other multiinput dialogue generation tasks having different information source number.