XD aily D ialog: A Multilingual Parallel Dialogue Corpus (2023)

Abstract: High-quality datasets are significant to the development of dialogue models. However, most existing datasets for open-domain dialogue modeling are limited to a single language. The absence of multilingual open-domain dialog datasets not only limits the research on multilingual or cross-lingual transfer learning, but also hinders the development of robust open-domain dialog systems that can be deployed in other parts of the world. In this paper, we provide a multilingual parallel open-domain dialog dataset, XDailyDialog, to enable researchers to explore the challenging task of multilingual and cross-lingual open-domain dialog. XDailyDialog includes 13K dialogues aligned across 4 languages (52K dialogues and 410K utterances in total). We then propose a dialog generation model, kNN-Chat, which has a novel kNN-search mechanism to support unified response retrieval for monolingual, multilingual, and cross-lingual dialogue. Experiment results show the effectiveness of this framework. We will make XDailyDialog and kNN-Chat publicly available soon.

## introduction developing high-quality open-domain dialogue systems is one of the key challenges in artificial intelligence. unlike closed-domain dialogue systems which deal with specific kinds of conversations (like a chatbot for customer support), open-domain dialogue systems can engage in conversation on any topic. in recent years, there has been a significant increase in research on dialogue due to the rise of voice-based bots, such as meena @xcite , blenderbot @xcite and xiaoice @xcite . to advance the quality of open-domain dialogue systems, many largescale corpora have been created @xcite @xcite @xcite . however, to the best of our knowledge, almost all existing large-scale corpora for open-domain dialogue modeling are limited to a single language, such as english @xcite @xcite @xcite , or chinese @xcite @xcite . the absence of multilingual open-domain dialogue corpora not only limits the research on multilingual or cross-lingual transfer learning @xcite but also hinders the development of robust opendomain dialogue systems that can be deployed in other parts of the world. previous work on various nlp tasks has shown that multilingual corpora can bring performance improvements in multilingual or cross-lingual transfer learning. this includes tasks such as conversational recommendation @xcite , task-oriented dialog @xcite , semantic parsing @xcite , qa and reading comprehension @xcite @xcite @xcite , machine translation @xcite , document classification @xcite @xcite , semantic role labelling @xcite and nli @xcite . thus, we believe that multilingual training data might enhance multilingual or crosslingual transfer learning for open-domain dialogue as well. to facilitate the study of multilingual and crosslingual dialogue, we present a multilingual parallel dialog dataset, xdailydialog, for multilingual and cross-lingual open-domain dialogue. xdailydialog consists of 13k dialogues aligned across 4 languages (52k dialogues and 410k utterances in total). the most significant advantage of parallel data over non-parallel data, such as xpersona @xcite , is that it can support crosslingual tasks. we define 3 task settings using xdailydialog. as shown in figure 1 , the first task is monolingual dialogue, where dialogue context and response are in the same language. it aims at investigating the performance variation of the same model across different languages. the figure also illustrates another task that is called multilingual dialogue. here we directly mix training instances of the 4 languages into a single training set and train a single model to handle multilingual dialogue at the same time. finally, the last task is cross-lingual dialogue, where model input and output are in different languages, e.g. dialog context is in english and the generated response is in chinese. to address these tasks, we build a model using k-nearest neighbors (knn) and mbart @xcite . we conduct an empirical study of the proposed model on xdailydialog. our experimental results indicate that the use of cross-lingual data can bring performance improvement in the monolingual dialogue. our work makes the following contributions: multilingual and cross-lingual datasets for dialog multilingual dialog datasets are relatively scarce. lin et al. (lin et al., 2020) propose a multilingual persona-chat dataset, xpersona, by extending the persona-chat corpora (dinan et al., 2019) to 6 languages: chinese, french, indonesian, italian, korean, and japanese. in xpersona, the training sets are automatically translated using translation apis, while the validation and test sets are annotated by humans. xpersona focuses on cross-lingual personalized dialog and is not parallel, while xdailydialog focuses on multilingual and cross-lingual dialog generation and is parallel. liu et al(liu et al., 2021) construct a multilingual conversational recommendation dataset, durecdial 2.0, by crowdsourcing translation based on durecdial (liu et al., 2020b) to 2 languages: chinese, and english. durecdial 2.0 focuses on conversational recommendation, while xdailydialog focuses on dialog generation and has more languages. multilingual and cross-lingual datasets for task-oriented dialog several multilingual taskoriented dialogue datasets have been published @xcite , enabling evaluation of the approaches for crosslingual dialogue systems. @xcite introduces the first bilingual multi-domain dataset for task-oriented dialogue modeling, which has only two languages and is not parallel. mrksi et al . @xcite annotated two languages (german and italian) for the dialogue state tracking dataset woz 2.0 @xcite and trained a unified framework to cope with multiple languages. meanwhile, schuster et al. @xcite introduced a multilingual nlu dataset and highlighted the need for more sophisticated crosslingual methods. those datasets mainly focus on multilingual nlu and dst for task-oriented dialogue and are not parallel. in comparison with them, xdailydialog is a multilingual dataset for open-domain dialog, which has 4 languages and is parallel. ## dataset collection xdailydialog aims to collect high-quality parallel data for the research of monolingual, multilingual, and cross-lingual open-domain dialog. it is translated from dailydialog @xcite , which is a human-written, well-formatted english dataset. in this section, we describe how xdailydialog is collected. ## our approach we propose a non-parametric method named knn-chat for monolingual, multilingual, and crosslingual dialogue generation, as shown in figure 2 . knn-chat is inspired by knn language models such as knn-lm @xcite and knn-mt @xcite and implemented for monolingual, multilingual, and crosslingual dialogue by using dialogue context, response, and extra information (dialogue emotion, topic, action, target language). compared to recent dialogue systems with knn @xcite , our experimental results show that knn-chat, a nonparametric method (without extra training except for pre-training mbart), could also be effective for monolingual, multilingual, and cross-lingual dialogue generation. ## conclusion to facilitate the study of multilingual and crosslingual dialog, we created the first publicly available multilingual parallel dataset, xdailydialog, for dialog and proposed three challenging tasks for the community based on it. furthermore, we also built a new conversation generation framework, knn-chat, with a novel knn-search mechanism that can support unified response retrieval for monolingual, multilingual, and cross-lingual dialog generation. expensive experiment results confirm the effectiveness of this framework. we hope that xdailydialog would help push forward research in the unified end-to-end monolingual, multilingual, and cross-lingual conversational modeling.