M ul ZDG : Multilingual Code-Switching Framework for Zero-shot Dialogue Generation (2022)

Abstract: Building dialogue generation systems in a zero-shot scenario remains a huge challenge, since the typical zero-shot approaches in dialogue generation rely heavily on large-scale pre-trained language generation models such as GPT-3 and T5. The research on zero-shot dialogue generation without cumbersome language models is limited due to lacking corresponding parallel dialogue corpora. In this paper, we propose a simple but effective Multilingual learning framework for Zero-shot Dialogue Generation (dubbed as MulZDG) that can effectively transfer knowledge from an English corpus with large-scale training samples to a non-English corpus with zero samples. Besides, MulZDG can be viewed as a multilingual data augmentation method to improve the performance of the resource-rich language. First, we construct multilingual code-switching dialogue datasets via translation utterances randomly selected from monolingual English datasets. Then we employ MulZDG to train a unified multilingual dialogue model based on the code-switching datasets. The MulZDG can conduct implicit semantic alignment between different languages. Experiments on DailyDialog and DSTC7 datasets demonstrate that MulZDG not only achieve competitive performance under zero-shot case compared to training with sufficient examples but also greatly improve the performance of the source language.

## introduction the success of neural models and the emergence of large-scale dialogue datasets have greatly advanced the research of dialog generation @xcite @xcite . the open-domain dialogue systems aim to generate more informative and fluent responses @xcite @xcite * corresponding author @xcite , which are widely used in various applications such as emotional companionship, mental health support, and social chatbots. although achieving promising performance, most existing dialogue generation systems @xcite @xcite rely on a considerable amount of data resource, such as dialogpt @xcite . in practice, the dialogue corpus for many languages is unavailable, which limits the usefulness of dialogue systems for low-resource or even zero-resource languages. hence it is important to design approaches that can effectively transfer knowledge from the source language with sufficient resources to a target language with zero training samples. the pre-trained language models have been proved to be very effective in dialogue generations including zero-shot scenarios. most existing zero-shot dialogue generation approaches usually directly employ large-scale pre-trained generative language models such as gpt-3 @xcite , t5 @xcite , or conduct secondary pre-training on target language based on language models @xcite . although these methods can handle the issue of zero-shot generation, the cost of pre-training is unaffordable. in the similar task of nmt (neural machine translation), zero-shot generation methods are mainly introducing one additional language between source and target language @xcite @xcite @xcite , which can achieve indirect semantic mapping between the source and target language through the additional intermediate language. the nature of translation is the semantic mapping between source and target languages while there is no similar semantic mapping between dialogue history and response. therefore, directly transferring the zero-shot approaches in nmt to the dialogue generation task is infeasible. the zero-shot multilingual understanding tasks usually employ code-switching approach to achieve semantic alignment between different languages @xcite @xcite . the way of code-switching can conduct implicit semantic alignment without relying on parallel corpus pairs. inspired by these studies, we employ the code-switching method to transfer the knowledge of dialogue history in english to other target languages which have no training examples. we follow previous work @xcite on multilingual representation to employ code-switching at the utterance level, although code-switching at the word or span level is more common @xcite @xcite . based on the code-switching method, we propose a simple but effective multilingual learning framework for zero-shot dialogue generation (dubbed as mulzdg) that can effectively transfer knowledge from english corpus with largescale training samples to non-english corpus with zero samples. specifically, we first construct codeswitch languages using the nmt system and bilingual dictionary @xcite . as shown by the black arrows in figure 1 , we randomly select utterances from dialogue history to translate into other target languages. for each target language, we construct code-switching corpus containing source and target languages, respectively. then we employ mulzdg based on an encoder-decoder structure to train a unified multilingual dialogue generation system, which can be applied in the source language and other target languages with no training samples. mulzdg with a multi-task structure can generate responses with different languages according to specific input prompts. mulzdg can conduct implicit semantic alignments through task sharing mechanism between different languages, as shown by solid brown lines in figure 1 . to summarize, we make the following contributions: • we propose a simple but effective multilingual framework, mulzdg, which can effectively transfer the knowledge from the source language with large-scale training samples to target languages with zero samples. ## conclusion and future work in this paper, we propose a simple but effective multilingual framework: mulzdg, which can not only be used as an approach of data augmentation but also be used to zero-shot dialogue generaton. besides, we release the multilingual versions of dailydialog and dstc7 datasets. in the future, we will explore the working mechanism and effect of large-scale multilingual code-switching languages (e.g., hundreds of languages) in the problem of zero-shot dialogue generation. models types responses hred en i like travelling. like my experience and i have a good idea. aug i like travelling. i travel frequently. vhred en i like sports. i like travelling. aug i like travelling. i'll do it next time. gold response yes, i like travelling. i am young, and unmarried. it's no problem for me to travel frequently.