Vikhr: Constructing a State-of-the-art Bilingual Open-Source Instruction-Following Large Language Model for R ussian (2024)

Abstract: There has been a surge in the development of various Large Language Models (LLMs). However, text generation for languages other than English often faces significant challenges, including poor generation quality and reduced computational performance due to the disproportionate representation of tokens in the model’s vocabulary. In this work, we address these issues by developing a pipeline for adaptation of English-oriented pre-trained models to other languages and constructing efficient bilingual LLMs. Using this pipeline, we construct Vikhr, a state-of-the-art bilingual open-source instruction-following LLM designed specifically for the Russian language. “Vikhr” refers to the name of the Mistral LLM series and means a “strong gust of wind.”Unlike previous Russian-language models that typically rely on LoRA adapters on top of English-oriented models, sacrificing performance for lower training costs, Vikhr features an adapted tokenizer vocabulary and undergoes the continued pre-training and instruction tuning of all weights. This not only enhances the model’s performance but also significantly improves its computational and contextual efficiency.The remarkable performance of Vikhr across various Russian-language benchmarks can also be attributed to our efforts in expanding instruction datasets and corpora for continued pre-training. Vikhr not only sets the new state of the art among open-source LLMs for Russian but even outperforms some proprietary closed-source models on certain benchmarks. The model weights, instruction sets, and code are publicly available.

## introduction instruction tuning has unlocked in large language models (llms) vast zero-shot capabilities without the need for careful prompt engineering (ouyang ♢ equal contribution 1 https://huggingface.co/vikhrmodels @xcite . the most rapid research and development efforts are currently devoted to english llms. there has been a surge in english open-source models: llama series @xcite , mistral series @xcite , vicuna series @xcite , etc. this growth is driven by the abundance of raw training data in english and dedicated efforts to create extensive sets of instruction-output pairs. even though llms oriented on english have some multilingual capabilities @xcite due to the presence of small amounts of text in various languages within their training datasets @xcite , their overall performance in non-english languages remains relatively limited. although they can usually generate portions of coherent texts, these models struggle with reasoning in non-english languages, lack culture-specific knowledge, and are highly inefficient in terms of tokenization. this inefficiency stems from how byte-pair tokenization algorithms operate, as they break down infrequent words into multiple tokens. since multilingual data typically represents a small portion of the training dataset, non-english words are often split into many pieces. as a result, this increases the number of steps during prompt processing and text generation, reduces the effective context window, and ultimately degrades overall performance @xcite . this disparity places non-english languages at a disadvantage. there is also a research direction focused on developing multilingual llms designed to perform well across multiple popular languages: bloomz @xcite , mgpt @xcite , bactrian-x @xcite , palo @xcite , aya101 from cohereai @xcite , etc. these models are typically trained on rich multilingual datasets and are less skewed towards english. however, when aiming to perform well across multiple languages simultaneously, these models must still share their vocab-ulary and parameters. this often hinders their performance for each particular language in isolation, especially for the popular smaller model sizes, such as 7b and 13b. the aim of maximizing the llm performance for a specific language within a certain number of parameters has led researchers to develop bilingual llms @xcite @xcite . these llms prioritize a regional language, e.g. jais @xcite focuses on arabic, but they are trained also on english data. the inclusion of english data in pre-training alongside regional language data is motivated by the significantly larger volume of english data available. this helps llms substantially enhance skills such as logical and common sense reasoning, which are also applied when generating text in a regional language. bilingual llms is a perspective direction as they can remain small and efficient, but at the same time comprehensively capture linguistic nuances and cultural contexts of the regional language. this work seeks to develop a pipeline for adapting english llms to other languages facilitating the development of bilingual llms. specifically, we aim to build an instruction-following bilingual llm for russian and english that could be used for multilingual natural language processing research. russian is one of the high-resource languages and is typically represented in multilingual llms. additionally, there are several proprietary closedsource llms, such as mts ai, gigachat, and yan-dexgpt, that meet or even surpass their englishoriented flagship competitors when it comes to text processing and generation in russian. however, controllable research often requires white-box access to llm logits and layer outputs, the ability to modify weights and model architecture, and consistent answers for reproducibility, which is often impossible in closed-source llms due to their constant development and retirement. there are only a few open-source llms designed for russian: saiga @xcite , rugpt (ai forever, 2022), ruadapt @xcite , and some others. of these, only saiga and ruadapt are instruction-tuned. we aim to fill the lack of instruction-tuned open-source llm for russian that is both efficient and effective. building even a small llm tailored to a specific language from scratch demands a lot of computational resources. consequently, many researchers opt to fine-tune lora adapters @xcite for english-oriented llms using some languagespecific data. while this approach can improve model generation quality, it does not address computational inefficiency because the tokenizer and model vocabulary remain unchanged. in contrast, our approach not only fine-tunes a base llm on russian language data but also reconstructs its underlying tokenizer and vocabulary, alongside suggesting an improved method for continued pretraining. additionally, we have significantly expanded the available russian datasets for instruction tuning. the developed llm achieves stateof-the-art results for the russian language among other open-source counterparts across a wide range of benchmarks. contributions of the paper are the following: ## related work one of the first prominent series of generative llms for russian is rugpt ( @xcite . the authors developed several models, trained on the standard language modeling task, with sizes reaching up to 13 billion parameters. these models were created from the scratch and trained on a large russian corpus, enabling them to capture the linguistic nuances of russian more effectively than multilingual models. additionally, since the training data was mostly in russian, these models also have efficient tokenization. however, the lack of multilingual data (e.g. in english) limits their performance. notably, the rugpt models are not instruction-tuned. @xcite suggests to leverage reasoning capabilities of existing english-oriented llms and adapt them to the russian language by training lora adapters. they created an alpaca-like set of russian instruction-output pairs and performed instruction tuning on it. as a result, they have established the series of models called saiga, which demonstrate the competitive performance and used to be a reasonable choice for an off-the-shelf opensource russian llms for the past years. however, the tokenizer in these models is not adapted, so they experience issues with the context size and computational efficiency. tikhomirov and chernyshev (2023) address these issues in saiga. in addition to model tuning on russian data, they also adapted the model tokenizer. they note that improving tokenization helps to both enhance the efficiency of the model and its performance while reducing memory consumption. however, during continued pre-training, the authors freezed the model weights, except for lm heads and token embeddings, which leads to the suboptimal performance. in this work, we take advantage of pre-trained english-oriented llms, adapt llm tokenizer for better computational and contextual efficiency, leverage continued pre-training on vast russianlanguage corpora with regularization for preventing "catastrophic forgetting", construct a novel extended set of russian instruction-output pairs, and perform instruction tuning. the created llm adaptation pipeline along with the data for continued pre-training and instruction tuning enables vikhr to achieve new state-of-the-art results for russian, maintain high performance for english, and demonstrate high computational efficiency. ## llm construction pipeline the construction of vikhr starts from one of the english-oriented llms. in this work, we discuss the vikhr model based on mistral 7b @xcite . the strong logical and common-sense reasoning capabilities, as well as the extensive world knowledge present in mistral llms provide an excellent starting point for our model. these features partially transfer to vikhr, enhancing its performance in generating text in russian. the process of the llm adaptation to russian starts with the vocabulary adaptation. then we perform continued pre-training of the llm on large russian datasets to mitigate the vocabulary shift and introduce culture specific knowledge. finally, we perform finetuning of vikhr on a set of instruction-output pairs in russian. ## conclusion we have presented vikhr -a new state-of-the-art open-source instruction-following bilingual llm oriented on the russian language. to create vikhr, we developed a comprehensive pipeline for adapting english-oriented llms to other languages. the pipeline includes the adaptation of the tokenizer vocabulary, continued pre-training of the entire model, and instruction tuning. we have also constructed a new dataset for instruction tuning by expanding the saiga dataset with automatically translated and cleaned english instruction datasets. our extensive work enabled vikhr to outperform the known baselines while maintaining computational efficiency. we hope that the developed cross-lingual adaption pipeline and the published models will foster the research on llms and enhance the diversity of languages incorporated into research agendas. in the future work, we plan to release in the open-source new better versions of vikhr. at the moment, our best publicly available model is vikhr-nemo-12b-instruct foot_11 based on mistral nemo. we also plan to perform cross-lingual adaption of llms to low-resource languages such as belarusian, serbian, and kazakh.