“Vorbești Românește?” A Recipe to Train Powerful R omanian LLM s with E nglish Instructions (2024)

Abstract: In recent years, Large Language Models (LLMs) have achieved almost human-like performance on various tasks. While some LLMs have been trained on multilingual data, most of the training data is in English; hence, their performance in English greatly exceeds other languages. To our knowledge, we are the first to collect and translate a large collection of texts, instructions, and benchmarks and train, evaluate, and release open-source LLMs tailored for Romanian. We evaluate our methods on four different categories, including academic benchmarks, MT-Bench (manually translated), and a professionally built historical, cultural, and social benchmark adapted to Romanian. We argue for the usefulness and high performance of RoLLMs by obtaining state-of-the-art results across the board. We publicly release all resources (i.e., data, training and evaluation code, models) with the goal of supporting and encouraging research on Romanian LLMs while concurrently creating a generalizable recipe adequate for other low or less-resourced languages.

## introduction the transformer architecture @xcite has become ubiquitous recently in the natural language processing (nlp) domain, leading to stateof-the-art performance on various tasks, ranging from text classification to text generation. decoderonly language models such as gpt series @xcite @xcite exhibit impressive capabilities such as understanding and generating natural language, learning and solving tasks not directly trained on. in the last few years, large language models (llms) development has exploded, with a plethora of models available @xcite @xcite @xcite . training such massive models requires a vast amount of available data (ranging from tens to hundreds of billions of text tokens) besides computational resources. intrinsically, most work on such models focuses mainly on english, often excluding other languages. here, we focus on building the resources for training llms specialized in romanian, a less-resourced language. by showcasing our approach and releasing our models, we strive to enable further research and exciting applications for romanian-speaking users. our contributions can be summarized as follows: i. we present and release the first romanian llms with both foundational and instruct variants, based on popular open-source models such as llama2 @xcite , llama3 foot_0 , mistral @xcite and gemma @xcite . our collection of models is grouped under the openllm-ro initiative foot_1 ; ii. we collect and translate a large set of evaluation benchmarks (including a human translation of mt-bench @xcite ), conversation, and instructions; iii. we extend and adapt existing testing suites to thoroughly assess the performance of different models on romanian tasks. iv. we propose a novel methodology to assess language-specific llms by proposing the first benchmark that evaluates the romanian cultural knowledge of llms; v. we make our recipe (i.e., datasets and code) and models opensource to encourage further research both for romanian and other low/less-resourced languages. ## related work with the advent of open-source llms, especially llama2 @xcite and @xcite , non-english specialized alternatives have been trained. @xcite have trained a family of finnish foundational models, ranging from 186m to 176b parameters. trained on a total of 38b tokens with a custom tokenizer, their models showcase state-of-the-art performance on a variety of finish downstream tasks. similarly, @xcite have trained both a foundational and a chat variant of llama2 for chinese, while wang (2023) worked on a domain-specific (i.e., media) chinese llm. for italian, @xcite translated four out of the six benchmarks used in open llm leaderboard foot_2 to release the first italian llm benchmark and evaluate existing italian models @xcite @xcite , models that are all adapter-based fine-tunes. other languagespecific llms include versions for dutch foot_3 , bulgarian foot_4 , serbian @xcite , or japanese @xcite . dac @xcite automatically translated english alpaca @xcite instructions together with a set of generated self-instruct @xcite instructions into 26 languages. furthermore, they translated for evaluation three out of the six benchmarks used in the open llm leaderboard. they fine-tuned llama @xcite , thus obtaining language-specific models. @xcite translated the alpaca and dolly @xcite datasets into 51 languages and fine-tuned the llama model, further evaluated in a zero-shot setup on downstream tasks. @xcite contributed with a huge amount of multilingual text resources under the aya initiative. over 2000 people across 119 countries contributed to the creation of aya dataset, a dataset of over 204k human-curated instructions in 65 languages. the aya collection uses instruction-style templates created by fluent speakers and applies them to existing datasets, generating over 500m instances that cover 114 languages. based on aya resources, @xcite fine-tuned aya-101, a 13b parameter mt5 model @xcite capable of following instructions in 101 languages. compared to aya-101, which focuses on breadth (i.e., a lot of languages), @xcite considered a more depth-based approach, building aya-23, a 23-language multilingual instruct model following cohere's command model foot_5 . based on previous research @xcite , we decided to perform both continual pre-training and instruction fine-tuning. ## data we employ two distinct data sources for training a romanian llm: documents for training the foundational model and instructions for the chat variant. ## training initial pre-training experiments were limited to a maximum sequence length of 512 to accelerate training. we further experimented with longer sequence lengths (i.e., 1024 and 2048), always picking the biggest batch size that fits into the memory. for all experiments, we used an adamw optimizer with weight decay set to 0.1 and gradient clipping. for both continual pre-training and instruction finetuning, we performed full training (i.e., updating all model parameters). ## evaluation evaluating a general llm is not straightforward, considering the vast number of benchmarks and evaluation protocols proposed and evaluating llms in any language besides english only adds another layer of complexity. we propose an extensive set of evaluations, grouped into four different categories, to obtain a thorough performance assessment of llms in romanian. ## results & discussions in this section, we present and discuss the results of our models on the four evaluations: academic benchmarks, mt-bench, romanian downstream tasks, and roculturabench. ## conclusions this paper introduces the first open-source llm models specialized for romanian. the released models show promising results, outperforming ex-isting solutions for a wide array of tasks. we present a general training and evaluation recipe which we expect to work on different, more powerful llms as well (e.g., models larger than 7-8b, mixtral foot_11 models) as well as for other languages. we firmly believe that this is just the first step towards building and adapting llms for romanian for both research and industry applications. pretraining on additional cleaner data, validating existing translations, and adding more instructions, conversations and human preference data are just a few improvements to increase the quality of rollms.