Tagengo: A Multilingual Chat Dataset (2024)

Abstract: Open source large language models (LLMs) have shown great improvements in recent times. However, many of these models are focused solely on popular spoken languages. We present a high quality dataset of more than 70k prompt-response pairs in 74 languages which consist of human generated prompts and synthetic responses. We use this dataset to train a state-of-the-art open source English LLM to chat multilingually.We evaluate our model on MT-Bench chat benchmarks in 6 languages, finding that our multilingual model outperforms previous state-of-the-art open source LLMs across each language. We further find that training on more multilingual data is beneficial to the performance in a chosen target language (Japanese) compared to simply training on only data in that language.These results indicate the necessity of training on large amounts of high quality multilingual data to make a more accessible LLM.

## introduction recently, open source large language models (llms) have grown drastically in both popularity and performance. models such as llama 3 (ai@meta, 2024b) have exceeded the performance of previous state-of-the-art proprietary models like gpt3.5 @xcite on popular robust benchmarks including the chatbot arena leaderboard @xcite . these open source llms are also increasingly being used in commercial ai chat products such as the meta ai assistant @xcite . however, many current llms exhibit lower performance on languages outside of english @xcite . indeed, llama 3 itself is currently an english-only llm, meaning that even when it is prompted in a language besides english, it often replies in english. this limits the potential user base of these llms due to the fact that less than 1.5 billion of the world's more than 8 billion population can speak english @xcite . therefore, we set out to train a state-of-the-art open source llm (llama 3) to be able to chat not only in english, but in many languages. in order to make english-focused llms accessible in other languages, previous work has finetuned these models on non-english data @xcite @xcite . many multilingual chat datasets such as multialpaca @xcite and aya @xcite cover many languages and tasks but can also lack natural prompts and high quality responses. for this reason, we created a large, diverse, high quality multilingual dataset using more than 70k human generated prompts in 74 languages and generated responses from these using state-of-theart proprietary chat models. we used this dataset to train two models, a multilingual llm and a japanese-only llm, both supervised fine-tuned models based on the llama 3 8b instruct model. we found that our model achieved better evaluation scores on multilingual chat benchmarks compared to the similarly sized state-of-the-art open source models, indicating the high quality and diversity of our training dataset. we also find that our multilingual-trained llm performs better on japanese chat benchmarks compared to our japanese-only-trained llm, indicating that transfer learning from training on other languages is beneficial for training even monolingual models outside of english. our findings combine to inform the community of exactly how to fine-tune monolingual llms to create a strong multilingual model. we make our training data (tagengo) 1 , train-ing code foot_1 , evaluation benchmark (multilingual mt-bench) foot_2 , and trained models (suzume) 45 publicly available for free use online. 2 related work in the literature, strong foundation models such as llama 2 @xcite and gemma @xcite have been subsequently fine-tuned on data from a specific language or languages, including japanese @xcite , arabic @xcite , and south-east asian languages @xcite . fine-tuning has often shown to improve the accuracy of the resultant llm on tasks in that language. however, the training dataset of these models are often not shared, making it difficult to create a truly multilingual llm across many languages. some multilingual chat datasets do exist that can be used for training llms. multialpaca @xcite is a multilingual dataset of 133k promptresponse pairs covering 11 languages that were generated in a similar manner to alpaca @xcite . this dataset was created by generating synthetic prompts from a small number of english seed prompts and then answering these prompts using an large-scale llm, gpt3.5 @xcite . because these prompts are generated synthetically, this data may not reflect the sorts of prompts that real users may use with an llm, potentially limiting the ability of models trained on this data to be used practically. moreover, the prompts and responses for this dataset were generated using gpt3.5, meaning that the quality of the data may not be as high as if a state-of-the-art llm was used, like gpt4 @xcite . the sharegpt dataset used by models such as vicuna @xcite and openchat @xcite contain approximately 70k open source conversations between users and gpt3.5 @xcite and 6k conversations between users and gpt4 @xcite , meaning that the prompts used in these datasets are often much more naturalistic to a real llm use-case. however, the majority of these prompts are in english, meaning that this dataset is limited in its use in training multilingual models. moreover, due to the fact that that majority of this dataset contains data generated from gpt3.5, its usefulness in training is limited as many other models have now surpassed the performance of gpt3.5 in english @xcite . the amount of multi-lingual data in the higher quality gpt4 subset of the sharegpt dataset is small, meaning that its usefulness in training is constrained by its size. to address the shortcomings in existing public datasets, we created a large, diverse, high quality multilingual dataset using more than 70k human generated prompts in 74 languages and generated responses from these using state-of-the-art proprietary chat models. ## method in this section, we detail how we generated our training dataset, our training method, and finally our evaluation techniques. ## results the mt-bench scores for each model evaluated can be found in we first found that we were able to train llama 3 8b instruct to output responses in the same language as the prompt. this means that we achieved our base objective of enabling a monolingual model (llama 3) to be able to output multilingual chat. secondly, english performance of the multilingual trained model only dropped slightly compared to the base llama 3 8b instruct model. this indicates that english chat performance does not considerably drop even when training on a majority of non-english data. thirdly, we found that the multilingual trained model performs better compared to the starling 7b beta across 5 out of 6 non-english languages tested. however, also we found that our multilingual model achieved lower evaluation scores compared to the proprietary gpt3.5 on 5 of 6 non-english languages. this indicates that our model has achieved state-of-the-art performance in multilingual chat for open-source models of its size, but has not achieved state-of-the-art performance more generally. finally, the suzume 8b multilingual model achieves higher mt-bench scores on the japanese benchmark compared to the suzume 8b japanese model, indicating that transfer learning from training on other languages is beneficial for training even monolingual models outside of english. ## discussion our results indicate the need for large, high quality, multilingual datasets when training multilingual models. we find that with such a dataset, we can train a state-of-the-art monolingual model such as llama 3 to achieve state-of-the-art multilingual performance. we also found that training on additional non-japanese data improves the performance of our llm on japanese benchmarks when compared to training solely on japanese data, indicating that there is a collective improvement effect between languages when training using multilingual data. this adds to the body of work that indicates that training on multiple languages enables the llm to better generalise to other languages @xcite . this suggests that generating an even larger, more diverse dataset in the future could further aid the performance of llms on low-resource languages. our training dataset mainly consisted of single prompt-response pairs, but many chats between users and llms extend beyond a single conversation turn. therefore, future work could include creating a dataset that contains multiple turns of conversation, with the prompts either generated by humans or by high quality llms. future work could also include adding more languages to our dataset. our dataset only included 74 languages, and crucially omits any languages in the niger-congo language family, one of the most diverse language families in the world @xcite . therefore, future work could involve sampling initial prompts from a wider range of sources (possibly by advertising free chatbot access to people in areas with many speakers of underrepresented languages) and generating responses based on these prompts. this would help to both improve an llms linguistic understanding of these low-resource languages as well as improve their understanding of the topics and questions that people from that language and culture may ask. finally, future work could include generating preference data, such as was done in english in the nectar dataset @xcite , for use with contrastive learning techniques such as direct preference optimisation @xcite and odds ratio preference optimisation @xcite . these techniques have been shown to further improve the accuracy of llms, suggesting that training using these techniques may also improve the performance of llms in multilingual chat. ## conclusion in this study, we successfully trained a state-ofthe-art monolingual llama 3 llm to chat multilingually using a new, diverse dataset comprising over 70k human-generated prompts in 74 languages paired with high-quality synthetic responses. our multilingual model showcased superior performance across multiple languages compared to similar-sized open-source models on various chat benchmarks. interestingly, training using a multilingual dataset also enhanced the performance on specific monolingual tasks, implying beneficial crosslinguistic transfer effects. these outcomes underline the importance of using rich, diverse multilingual data for improving the capabilities of llms in global, multilingual applications.