XP ersona: Evaluating Multilingual Personalized Chatbot (2021)

Abstract: Personalized dialogue systems are an essential step toward better human-machine interaction. Existing personalized dialogue agents rely on properly designed conversational datasets, which are mostly monolingual (e.g., English), which greatly limits the usage of conversational agents in other languages. In this paper, we propose a multi-lingual extension of Persona-Chat, namely XPersona. Our dataset includes persona conversations in six different languages other than English for evaluating multilingual personalized agents. We experiment with both multilingual and cross-lingual trained baselines and evaluate them against monolingual and translation-pipeline models using both automatic and human evaluation. Experimental results show that the multilingual trained models outperform the translation pipeline and that they are on par with the monolingual models, with the advantage of having a single model across multiple languages. On the other hand, the state-of-the-art cross-lingual trained models achieve inferior performance to the other models, showing that cross-lingual conversation modeling is a challenging task. We hope that our dataset and baselines will accelerate research in multilingual dialogue systems.

## introduction personalized dialogue agents have been shown efficient in conducting human-like conversation. this progress has been catalyzed thanks to existing conversational dataset such as persona-chat @xcite . however, the training data are provided in a single language (e.g., english), and thus the resulting systems can perform conversations only in the training language. commercial dialogue systems are required to handle a large number of languages since the smart home devices market is increasingly international @xcite . therefore, creating multilingual conversational benchmarks is essential, yet challenging since it is costly to perform human annotation of data in all languages. a possible solution is to use translation systems before and after the model inference. this comes with three major problems: 1) amplification of translation errors since the current dialogue systems are far from perfect, especially with noisy input; 2) the three-stage pipeline system is significantly slower in terms of inference speed; and 3) high translation costs since the current state-of-theart models, especially in low resources languages, are only available using costly apis. in this paper, we analyze two possible workarounds to alleviate the aforementioned challenges. the first is to build a cross-lingual transferable system by aligning cross-lingual representations, as in @xcite , in which the system is trained on one language and zero-shot to another language. the second is to learn a multilingual system directly from noisy multilingual data (e.g., translated data), thus getting rid of the translation system dependence at inference time. to evaluate the aforementioned solutions, we propose a dataset called multilingual persona-chat, or xpersona, by extending the persona-chat corpora @xcite to six languages: chinese, french, indonesian, italian, korean, and japanese. in xpersona, the training sets are automatically translated using translation apis with several human-in-the-loop passes of mistake correction. in contrast, the validation and test sets are annotated by humans to facilitate both automatic and human evaluations in multiple languages. furthermore, we propose competitive baselines in two training settings, namely, cross-lingual and multilingual, and compare them with translation pipeline models. our baselines leverage pre- an extensive automatic and human evaluation @xcite of our models shows that a multilingual system is able to outperform strong translation-based models and on par with or even improve the monolingual model. the cross-lingual performance is still lower than other models, which indicates that cross-lingual conversation modeling is very challenging. the main contributions of this paper are summarized as follows: cross-lingual cross-lingual adaptation learns the inter-connections among languages and circumvents the requirement of extensive training data in target languages @xcite . cross-lingual transfer learning methods have been applied to multiple nlp tasks, such as named entity recognition @xcite , dialogue state tracking @xcite , part-ofspeech tagging @xcite @xcite , and dependency parsing @xcite . meanwhile, lample and conneau (2019) and conneau et al. (2019) proposed pre-trained cross-lingual language models to align multiple language representations, achieving state-of-the-art results in many cross-lingual classification tasks. the aforementioned tasks focused on classification and sequence labeling, while instead, chi et al. (2019) proposed to pre-train both the encoder and decoder of a sequence-to-sequence model (xnlg) to conduct cross-lingual generation tasks, namely, question generation and abstractive summarization. the latter is the closest to our task since it focuses on language generation; however cross-lingual dialogue generation has not yet been explored. ## data collection the proposed xpersona dataset is an extension of the persona-chat dataset @xcite . specifically, we extend convai2 @xcite to six languages: chinese, french, indonesian, italian, korean, and japanese. since the test set of convai2 is hidden, we split the original validation set into a new validation set and test sets. then, we firstly automatically translate the training, validation, and test set using apis (papago for korean, google translate for other languages). for each language, we hired native speaker annotators with at least a bachelor degree and a fluent level of english and asked them to revise the machine-translated dialogues and persona sentences in the validation set and test set according to original english dialogues. the main goal of human annotation is to ensure the revised conversations are coherent and fluent in target language despite the cultural discrepancy in different languages. therefore, annotators are not restricted to translate the english dialogues. they are also allowed to customize dialogues and persona sentences. the annotated dialogues can deviate from original translation while retain persona and conversation consistency. the full annotation instructions are reported in appendix a. compared to collecting new persona sentences and dialogues in each language, human-annotating i love cats hi ! 1 2 3 4 5 + + + + + + + + + + hi how are you ? sos i am fine and you ? 6 7 8 9 10 11 12 13 14 15 16 17 + + + + + + + + + + + + + + + + + + + + + + + + i am fine and you ? eos causal decoder per per per en en en usr usr usr usr usr sys sys per per per m-encoder eos ? you and fine am i + + + + + + + + + + + + + + + + + + + + + + + + 7 6 5 4 3 2 1 10 9 8 7 6 ? you and fine am i sos ? you are how hi + + + + + + + + + + 5 4 3 2 1 ! hi cats love i the dialogues by leveraging translation apis has multiple advantages. first, it increases the data distribution similarity across languages @xcite , which can better examine the system's cross-lingual transferability. second, revising the machine-translated dialogues based on the original english dialogue improves the data construction efficiency. third, it leverages the well-constructed english persona conversations as a reference to ensure the dialogue quality without the need for training a new pool of workers to generate new samples @xcite . on the other hand, human-translating the entire training-set (∼130k utterances) in six languages is expensive. therefore, we propose an iterative method to improve the quality of the automatically translated training set. we firstly sample 200 dialogues from the training set (∼2600 utterances) in each language, and we assign human annotators to list all frequent translation mistakes in the given dialogues. for example, daily colloquial english expressions such as "cool", "i see", and "lol" are usually literally translated. after that, we use a simple string matching to revise the inappropriate translations 2 in the whole training-set and return a revision log, which records all the revised utterances. then, we assign human annotators to check all the revised utterances and list translation mistakes again. we repeat this process at least twice for each language. finally, we summarize the statistics of the collected dataset in ## multilingual personalized conversational models let us define a dialogue @xmath0, s 1 , u 2 , s 2 , . . . , u n , s n } as an alternating set of utterances from two speakers, where u and s represent the user and the system, respectively. each speaker has its corresponding persona description that consists of a set of sentences @xmath1, . . . , p m }. given the system persona sentences p s , dialogue history @xmath2, @xmath3, and response language l, we are interested in predicting the next system utterances s k with model f (θ). ## conclusion in this paper, we studied both cross-lingual and multilingual approaches in end-to-end personalized dialogue modeling. we presented the xpersona dataset, a multilingual extension of persona-chat, for evaluating the multilingual personalized chatbots. we further provided both cross-lingual and multilingual baselines and compared them with the monolingual approach and two-stage translation approach. extensive automatic evaluation and human evaluation were conducted to examine the models' performance. the experimental results showed that multilingual trained models, with a single model across multiple languages, can outperform the twostage translation approach and is on par with monolingual models. on the other hand, the current stateof-the-art cross-lingual approach xnlg achieved lower performance than other baselines. in future work, we plan to research a more advanced crosslingual generation approach and construct a mixedlanguage conversational benchmark for evaluating multilingual systems.