Latxa: An Open Language Model and Evaluation Suite for B asque (2024)

Abstract: We introduce Latxa, a family of large language models for Basque ranging from 7 to 70 billion parameters. Latxa is based on Llama 2, which we continue pretraining on a new Basque corpus comprising 4.3M documents and 4.2B tokens. Addressing the scarcity of high-quality benchmarks for Basque, we further introduce 4 multiple choice evaluation datasets: EusProficiency, comprising 5,169 questions from official language proficiency exams; EusReading, comprising 352 reading comprehension questions; EusTrivia, comprising 1,715 trivia questions from 5 knowledge areas; and EusExams, comprising 16,046 questions from public examinations. In our extensive evaluation, Latxa outperforms all previous open models we compare to by a large margin. In addition, it is competitive with GPT-4 Turbo in language proficiency and understanding, despite lagging behind in reading comprehension and knowledge-intensive tasks. Both the Latxa family of models, as well as our new pretraining corpora and evaluation datasets, are publicly available under open licenses. Our suite enables reproducible research on methods to build LLMs for low-resource languages.

## introduction motivated by their increasing training cost and commercial interest, the development of large language models (llms) has been led by close initiatives like @xcite , claude @xcite and gemini @xcite . in recent times, a more open ecosystem has emerged following the release of various competitive models like @xcite and mistral @xcite . however, despite early efforts to build open multilingual models @xcite , the most competitive ones are notoriously english-centric. as shown in in this work, we present latxa, an open family of llms for basque that substantially outperforms all these previous models. basque is an agglutinative language written in latin script and with no known relatives, although a significant part of the vocabulary is shared with contact languages like spanish and french. basque is the 52th language in common crawl, with 0.035% of the total content -for reference, english is the 1st language with 46% of the content and spanish is the 5th with 4.6%. our work builds on various open resources and models that we further expand to basque, highlighting the importance of an open ecosystem for the development of language technology for lowresource languages. in particular, our models are based on llama 2, which we continue training in basque using a new corpus with 4.3m documents from 4 existing and 3 new sources. in addition, we release 4 diverse and challenging multiple-choice benchmarks comprising a total of 23,282 questions, covering language proficiency, reading comprehension, trivia questions, and public examinations. as shown in ## training corpora our training corpus combines various existing datasets, as well as some new ones that we release with this work. we have prioritized quality over quantity when constructing our corpus, prioritizing high-quality data sources and applying a thorough deduplication and filtering process. we next describe our data sources in §2.1, followed by our preprocessing pipeline in §2.2. ## latxa models we train 7b, 13b and 70b models following a continued pretraining approach. to that end, we use llama 2 as the base model @xcite , and continue training it using the corpus described in §2. to mitigate catastrophic forgetting from the original model 4 , we also include english data in the continued pretraining stage. for that purpose, we use 500k random documents from the pile @xcite , totaling 0.9b tokens. ## new evaluation datasets to overcome the scarcity of basque benchmarks that are suitable for evaluating base language models, we collect new evaluation data from various online games and tests. we have decided to take this approach instead of translating existing datasets to avoid translation artifacts @xcite . most importantly, this allows to have localized datasets that test the models' knowledge about topics that are most relevant for basque speakers. these tasks cover language proficiency (eu-sproficiency), reading comprehension (eusreading), trivia questions (eustrivia), and exams of advanced professional level (eusexams). all the datasets consist of multiple-choice questions, making them suitable for few-shot learning akin to mmlu @xcite in english. we next describe each dataset in more detail, while ## experimental setting to assess the quality of latxa models, we thoroughly evaluate them on a suite of diverse and challenging tasks against state-of-the-art models. in this section, we present the baseline models ( §5.1) and tasks ( §5.2), and describe the evaluation framework ( §5.3). results are analyzed and discussed in the next section ( §6). ## related work the amount of documents per language in common crawl foot_5 is helpful to organize the literature. there is no agreed-upon definition of low-resource language, so we set five arbitrary buckets of languages following a logarithmic distribution: high bucket with just english (rank 1, 46% of latest crawl), high-medium languages around spanish (rank 5, 4.6%), medium around danish (rank 24, 0.46%), low resource around nepalese (rank 46, 0.044), and very-low around somali (rank 81, 0.0046%). basque would be low-resource (rank 52, 0.035%). while there were some early efforts to build open multilingual language models like xglm @xcite , bloom @xcite , mgpt @xcite and the translation oriented madlad-400 @xcite , their performance significantly lags behind more recent english-centric models like @xcite or mistral @xcite . concurrent to our work, @xcite present aya, a fine-tuned mt5 encoder-decoder @xcite , which has been instruction-tuned and supports 101 languages. in the case of high-medium-resource languages, different teams have focused on building models from scratch. @xcite built a swedish 20b model which was favorably evaluated on perplexity, followed by @xcite , which builds a 40b model for five nordic languages, including low-resource danish and faroese, english and code. it was evaluated on perplexity for the three richest languages. @xcite present a french-english bilingual model trained with the same number of tokens for each language, although the amount of text for french (376m documents) is half the english text. the authors stress the fact that the tokenizer should not be biased towards any of the two languages. the results on french (and english) llm evaluation benchmarks show that their largest model (1.2b parameters) underperforms both 3b llama 2 and mistral models. regarding medium-resource, luukkonen et al. (2023) focus on finnish. using a 19b word corpus they trained several models from scratch, ranging from 186m to 13b parameters. as an alternative, they also continued pretraining a multilingual model, the 176b bloom. the evaluation on finbench, a version of big-bench @xcite , shows the 13b model underperforms the 7b model, while the continued pretraining model obtains the best results by a large margin. in followup work, luukkonen et al. (2024) train a 40b bilingual model from scratch with a larger finnish corpus, and obtain the best results in fin-bench. they do not compare the results to commercial models. the results suggest that monolingual models trained from scratch saturate at relatively small sizes, and multilingual or continued pretraining would be the best option for these languages. yong et al. (2023) compare different approaches to extend the 176b bloom model from 46 languages to new languages unseen at training, including guarani (rank 116, 0.0006%) and seven larger languages for which they sample 100k documents at most. model sizes range from 0.56b to 7.1b parameters. they report good results for adapters on zero-shot benchmarks. in concurrent work, lin et al. (2024) present mala-500, a continued pretrained model using lora based on llama 2 that supports 500 languages. mala seems to improve llama 2 on low to very-low resource languages, but degrades in some medium languages already covered in llama 2. evaluation is based on a variant of perplexity and text classification. previous work on low-resource languages has been done in the context of multilingual models (see above). in all cases, evaluation is based on perplexity and/or some readily available datasets, and does not include native benchmarks designed to evaluate base models. ## conclusion and future work in this work, we present a new open framework for the development and evaluation of llms in basque. the framework includes latxa, a set of state-of-the-art generative llms that have been built by continuing to pretrain the llama 2 7b, 13b and 70b models in basque. the pretraining dataset is the largest public dataset available to date and includes data from carefully curated sources, as well as content derived from automatically filtered versions of common crawl. after preprocessing and deduplication, the released corpora comprise 1.22b words and 4.17b tokens. we also present 4 new evaluation datasets, collectively the largest evaluation benchmark for basque that allows assessing the knowledge of the models about the basque language and culture. the latxa models outperform all previous open models and gpt-3.5 turbo, but they still lag behind gpt-4 turbo in most benchmarks. interestingly, latxa 70b outperforms gpt-4 turbo on eusproficiency and language & literature eustrivia questions, suggesting that the capabilities of llms in a particular language are not determined by their linguistic competence in this language. this, along with the effectiveness of scale, suggests that applying the exact same continued pretraining recipe could lead to better basque models as stronger english-only models become available. in the future, we plan to extend the training dataset by gathering quality content from diverse basque sources such as publishers or media, as well as building evaluation datasets to assess aspects such as truthfulness or hallucinations. we also plan to further tune latxa to follow instructions, which should improve the overall capabilities of our models.