Balancing Multi-Domain Corpora Learning for Open-Domain Response Generation (2022)

Abstract: Open-domain conversational systems are assumed to generate equally good responses on multiple domains. Previous work achieved good performance on the single corpus, but training and evaluating on multiple corpora from different domains are less studied. This paper explores methods of generating relevant responses for each of multiple multi-domain corpora. We first examine interleaved learning which intermingles multiple corpora as the baseline. We then investigate two multi-domain learning methods, labeled learning and multi-task labeled learning, which encode each corpus through a unique corpus embedding. Furthermore, we propose Domain-specific Frequency (DF), a novel word-level importance weight that measures the relative importance of a word for a specific corpus compared to other corpora. Based on DF, we propose weighted learning, a method that integrates DF to the loss function. We also adopt DF as a new evaluation metric. Extensive experiments show that our methods gain significant improvements on both automatic and human evaluation. We share our code and data for reproducibility.

## introduction recent work has achieved improvements in general performance for open-domain response generation @xcite @xcite . however, most studies are restricted to single-corpus training and evaluating, while there lacks studies for training and evaluating with multiple corpora from different domains. single-corpus training has intrinsic limitations. for example, a corpus of everyday chats, e.g., the personachat corpus @xcite , does not cover technical topics discussed in context what are you going to do on the remote system exactly? personachat i am going to be a pilot. i am going to fly planes. 4 corpora (concatenated) i am going to go to the beach. ubuntu chatlogs @xcite . a conversational system that learns only from personachat or from multiple corpora without an appropriate technique is not likely to generate relevant responses for certain topics (see this paper explores how to train and evaluate on multiple corpora from different domains for the open-domain response generation task. we propose several methods to make a model generate relevant responses for each of the multiple corpora. since simply training multiple corpora one by one does not solve the imbalanced performance (as shown in furthermore, we propose a novel weighted learning with domain-specific frequency (df). df is a word-level importance weight @xcite that assigns different weights (importance) to the same words from different corpora. in the training process, we weight the loss of a model with df, so that the model focuses on the most important words for a specific corpus. for automatic evaluation metrics, we eliminate the stop words and use rouge-1 (precision, recall, f1) @xcite to measure the relevance of the generated responses. in addition, we adopt df to see how relevant the generated response of a model is to a specific corpus. we will explain df as an evaluation metric in section 4.4. results show that for overall performance, the best method (weighted learning) improves 27.4% on precision, 45.5% on recall, and 34.1% on f1. further, it has at least 20.0% higher df, stating that it uses more important words from the "correct" corpus. we also conduct an extensive human evaluation on 2400 generated responses. the human evaluation shows a highly significant (@xmath0.001) improvement on all of our proposed methods, especially the weighted learning method. we summarize our work as follows: the task is to make the conversational models generate relevant responses for each corpus. ## base models we use two base models: an lstm seq2seq model with attention @xcite @xcite and a pre-trained gpt-2 model @xcite . the lstm seq2seq model with attention is a common model for conversational systems @xcite , and the gpt2 model is a state-of-the-art model for the response generation task @xcite . the basic task of response generation is to predict the next word given the past and current words of the context and response, and to make the generated response as similar to the original response as possible. the task can be described as follows. probability of response y given context x is predicted as: ## results our base models achieve perplexity scores of 28.9 (lstm model) and 19.6 (gpt-2) on the test set of the personachat dataset from the convai2 competition when fine-tuned with the single personachat corpus (more details can be found in section c). these results would likely advance the models to the second round in the competition. ## conclusions we have experimented with 4 methods-interleaved learning (baseline), labeled learning, multi-task labeled learning, and weighted learning-to help common open-domain conversational systems generate relevant responses for multiple corpora of different domains. we adopted rouge (precision, recall, f1) for auto evaluation. in addition, we used df to evaluate how well a model uses relevant words for a corresponding corpus. we also did an extensive human evaluation. our results show significant improvement in performance for our proposed methods, especially weighted learning. future work of multi-turn response generation is potential. we have focused on one-turn response generation, while dialogue is naturally multi-turn so further research is needed. example words are divided into five blocks. the first block has frequent words in all corpora, the second block has unique words from osdb, the third block has unique words from twitter, the fourth block has unique words from ubuntu, and the fifth block has unique words from personachat. the values of the corresponding corpus are marked with different colors.