A Model of Cross-Lingual Knowledge-Grounded Response Generation for Open-Domain Dialogue Systems (2021)

Abstract: Research on open-domain dialogue systems that allow free topics is challenging in the field of natural language processing (NLP). The performance of the dialogue system has been improved recently by the method utilizing dialogue-related knowledge; however, non-English dialogue systems suffer from reproducing the performance of English dialogue systems because securing knowledge in the same language with the dialogue system is relatively difficult. Through experiments with a Korean dialogue system, this paper proves that the performance of a non-English dialogue system can be improved by utilizing English knowledge, highlighting the system uses cross-lingual knowledge. For the experiments, we 1) constructed a Korean version of the Wizard of Wikipedia dataset, 2) built Korean-English T5 (KE-T5), a language model pre-trained with Korean and English corpus, and 3) developed a knowledge-grounded Korean dialogue model based on KE-T5. We observed the performance improvement in the open-domain Korean dialogue model even only English knowledge was given. The experimental results showed that the knowledge inherent in cross-lingual language models can be helpful for generating responses in open dialogue systems.

## introduction large language models trained with a large-scale corpus @xcite @xcite @xcite have stirred considerable research interest by showing low perplexity in several text generation tasks, which correlated with high token accuracy on in-domain test data, and providing linguistic fluency. however, when conditional text generation was performed using a large model, a "hallucination" problem @xcite was *equal contribution found while generating plausible text using the internal knowledge implicitly stored in the parameter and condition text together. owing to the hallucination problem in open-domain dialogue tasks, it is often observed that the model produces a response containing false information. for example, if the token "1992" frequently appears after "was born in" in the corpus for pre-training, the information is stored in the parameter of the model. in case "when was elvis presley born?" is entered as a condition, false information such as "elvis presley was born in 1992" is often generated. knowledge-grounded dialogue tasks @xcite were introduced for dialogue models to generate informative responses based on knowledge, and then dialogue modeling research based on external knowledge was started. because the responses of the knowledge-grounded dialogue models are generated based on dialogue history and external knowledge, the knowledgegrounded dialogue models mitigate the hallucination problem compared to the models based only on dialogue history @xcite . for knowledge-grounded dialogue systems in non-english, construction of data in a different language than english is required since most of the published knowledge-grounded dialogue datasets are built based on english. however, building knowledge-grounded data based on the corresponding language takes a lot of time and cost @xcite . even when translating existing english data, the high translation cost is incurred because of the large volume of knowledge data included in the dataset. in this paper, to avoid the data construction overhead, we suggest a cross-lingual knowledgegrounded dialogue model that generates responses in another language than english using knowledge in english. for the cross-lingual knowledge-grounded dialogue model, (1) we constructed the korean wizard of wikipedia (kowow) dataset by translating the wizard of wikipedia dataset @xcite , a knowledge-grounded dialogue benchmark, into korean. based on t5 @xcite , (2) we built korean-english t5 (ke-t5), a pre-trained language model specialized in @xcite (3) developed a cross-lingual knowledgegrounded dialogue model that selects knowledge and generates responses based on the t5 architecture. we conducted an experiment to prove that a dialogue model generating responses with knowledge in english alleviates the hallucination problem rather than that without knowledge, and shows comparable performance to a dialogue model with knowledge translated from english into korean. in addition, by sharing our insights through the qualitative analysis of the generated responses based on the proposed model, we describe several research directions for future knowledge-grounded dialogue tasks. ## kowow: korean wizard of wikipedia we used a commercial machine translation api (mt) to build the kowow dataset. we chose the multi-stage translation strategy @xcite as a strategy for building the kowow. in this strategy, training and validation splits are translated by machine, and in the case of test splits, machine-translated drafts are corrected by human translators by referring to the original text. because wow's utterances are colloquial, whereas the machine translator are trained with written languages, human translators spent more effort on correcting the machine-translated text, rather than directly translating the original english text into korean. to maintain the contextual/stylistic consistency of training data and evaluation data to some extent during the process of koreanization of the wow dataset, the same mt, the google's neural machine translation system @xcite , was used allthrough the multi-stage translation strategy. in the test split, if the content and meaning of the utterance translated by mt were different from the original text, the human translators retained the machine-translated text as much as possible and corrected it manually. when some idioms were translated and their meanings changed, they were revised for the correct expressions. for the translation quality, two experts in english and korean took a role of human translators. ## conclusion in this work, we studied whether proper knowledge was well selected and whether a knowledgegrounded response was well generated even if the languages of knowledge and utterances were different in response generation using external knowledge. through experiments, we showed that even if the languages of knowledge and utterance are different, if the pre-trained model supports both languages, the performance is comparable to that of the monolingual model. in addition, through qualitative analysis, the proposed model generates more informative responses than when knowledge is not used in most cases, and because it is based on external knowledge, the hallucination problem that generates a factually inaccurate response based on internal knowledge is alleviated. however, there were cases in which the selected knowledge was simply translated without answering the other person's question, contradictions occurred in the generated response, and false information was generated when knowledge was not selected. future work would be able to conduct research that generates responses using knowledge by understanding the other's intentions and questions, rather than simply generating responses that convey knowledge. in addition, it will be interesting to study the prevention of contradictions in the response when generating the response from the model. finally, when there is no external knowledge, research to reduce the hallucination problem and research to classify whether the generated response is true or false would help to create a natural dialogue model. a.7 cnn/dm summarization cnn daily mail summarization @xcite is the task of summarizing a given document. as shown in