Open-Source Large Language Models as Multilingual Crowdworkers: Synthesizing Open-Domain Dialogues in Several Languages With No Examples in Targets and No Machine Translation (2025)

Abstract: The prevailing paradigm in the field of Open-Domain Dialogue (ODD) agents predominantly focuses on some high-resource languages such as English or Chinese. Furthermore, the financial and temporal investments required for crowd-sourcing such datasets, in multiple languages, are substantial. Fortunately, advancements in Large Language Models (LLMs), specifically instruction-tuning enabled them to execute tasks based on natural language instructions. Additionally, these models possess the capability to function in various languages within a single thread. Consequently, to generate new data samples in different languages, we propose leveraging these capabilities to replicate the data collection process. We introduce a pipeline for generating ODD data in multiple target languages using LLMs, with demonstrations provided in a unique source language. By eschewing explicit Machine Translation in this approach, we enhance language-specific nuances and cultural specificity. We apply this methodology to the PersonaChat dataset. To further improve the openness of generated dialogues and mimic real life scenarios, we added the notion of speech events corresponding to the type of conversation the speakers are involved in and that of common ground which represents the premises of a conversation.

## introduction in the realm of natural language processing (nlp), large language models (llms) have surged in prominence, unleashing a myriad of possibilities in various tasks including @xcite . having an actual odd with an user implies showcasing some human-like dialogue abilities such as empathy, personality, engagingness, etc. most of the approaches to augment llm capabilities towards such skills rely on training on skill-specific datasets. unfortunately, there is a dearth of such datasets in languages other than english or, more recently chinese, and data collection is expensive in terms of cost and time. to tackle this issue, different approaches proposed to use machine translation (mt) -whether of the source language (l s ) dataset before training or during inference with a l s fine-tuned model @xcite -at the expense of data and models' quality. additionally, as highlighted by @xcite , odd, as used by literature, is often restricted to sole "small talk" type of speech event (se) where speakers are commonly asked/tasked to "just chat about anything" while real life odd can be various (from serious chat to gossip) depending on the context and involve speakers that share a common ground (cg) as a premise to their chat -hence restricting the "openness" of each odd. this is referred to as the open domain paradox (odp) by @xcite . crowd-sourced odd datasets are collected with fine-grained human-designed guidelines for the crowdworkers. also in the bargain, some works like @xcite for closed-source llm foot_0 and @xcite for open-source 2 demonstrate that instruction-tuned llms outperform crowdworkers for several tasks while others like @xcite estimate from an experiment that around 40% of crowdworkers actually use llms to complete their tasks. thereby, we propose here to leverage multilingual instruction-following llms abilities to generate datasets in target languages (l t ), other than english, and also attempt simultaneously to alleviate the odp. instead of mt, to get new samples in l t , we propose to design prompts based on few examples from the l s source dataset and their sourcing guidelines. this process exploits the fact that llms can understand (decode) well-crafted instructions and think (infer) in different languages simultaneously. we further enhance the aforementioned guidelines with supplementary instructions targeting the odp: common ground generation and inclusion, se type specification. therefore, our contributions are as follows: ## related work open-domain dialogue datasets numerous skill-specific datasets have been gathered to develop human-like conversational abilities in odd agents. for instance, datasets address personality @xcite @xcite , empathy @xcite , emotion @xcite , knowledge @xcite , and long-term memory @xcite . some datasets, like those by @xcite @xcite , combine multiple skills. however, most datasets are in english (and more recently in chinese), and replicating this process in other languages is costly. efforts to address this disproportionate representation of languages often hinge on mt like @xcite or @xcite who incorporate additional human post-processing, albeit at a non-negligible cost. it is contingent on the availability and quality of mt systems and often the resulting dialogues are in translationese and do not reflect either l t specificities or folk psychology but rather carry watermarks from l s @xcite and artifacts @xcite . while some address the lack of common ground, such efforts are typically confined to knowledge grounding or non-common ground scenarios, where only one speaker is informed. additionally, as highlighted by @xcite , there is insufficient diversity in ses types. ## methodology let d s,l symbolize a skill (s) specific odd dataset in a given language (l). we build on the availability of such datasets in a l s , here in english. so the latter is hereafter denoted as d s,en . while the described methodology focuses on odd and is later applied to personachat, it can be easily adapted to other generation tasks with data collected from crowdworkers. ## moud dataset in this section, the method presented in section 3 is applied to personachat @xcite and 28 l t , details of which can be found in ## baseline experiments with moud we conduct our experiments using the smallest variant of bloom (bigscience workshop, 2023), the 560m parameter model 14 . the model is finetuned on a multitask objective and evaluated using automatic metrics, including bertscore (bert-f1, ↑), hits@1 (↑), perplexity (ppl, ↓), and rouge-l (↑). refer to appendix d for further details. ## conclusion in this study, we addressed two key dimensions of openness in open-domain dialogue: cultural openness, achieved through multilingualism and l t specificity, and odp, which we enhanced by integrating cg with a diverse range of se types in the generated data. we evaluated four medium-sized, open-source llms, with llama-3.1-8b-instruct consistently outperforming the others across multiple criteria according to both human and llm assessments. it excelled not only in taxonomy relevance-particularly in effectively incorporating ses within cg-but also in l t specificity, fluency, and overall humanness. this led to its selection as the model for generating the final moud dataset, an o3dd dataset, where o3 represents open in language and culture, open in speech-event diversity, and open-domain dialogue. baseline automatic evaluations on shallow finetuned models highlight moud's potential for advancing multilingual odd systems. models trained on moud exhibit distinct characteristics compared to those trained on xpersona, reinforcing its complementary value. furthermore, models trained on a combination of both datasets-while maintaining the same overall training size-demonstrate improved performance over xpersona-trained models. this suggests that moud not only enhances diversity in dialogue modeling but also holds promise for further improvements.