Krikri: Advancing Open Large Language Models for G reek (2025)

Abstract: We introduce Llama-Krikri-8B, a cutting-edge Large Language Model tailored for the Greek language, built on Meta’s Llama 3.1-8B. Llama-Krikri-8B has been extensively trained on high-quality Greek data to ensure superior adaptation to linguistic nuances. With 8 billion parameters, it offers advanced capabilities while maintaining efficient computational performance. Llama-Krikri-8B supports both Modern Greek and English, and is also equipped to handle polytonic text and Ancient Greek. The chat version of Llama-Krikri-8B features a multi-stage post-training pipeline, utilizing both human and synthetic instruction and preference data, by applying techniques such as MAGPIE. In addition, for evaluation, we propose three novel public benchmarks for Greek. Our evaluation on existing as well as the proposed benchmarks shows notable improvements over comparable Greek and multilingual LLMs in both natural language understanding and generation as well as code generation.

## introduction recent advancements in ai have been largely driven by the development of large-scale foundation models. meta's llama 3 @xcite fostered a new generation of open models, designed for strong multilingual capabilities, code generation, reasoning, and tool use. with extended context windows, and refined training strategies, models based on llama 3 have achieved performance comparable to proprietary systems like gpt-4. a critical aspect in this evolution is the development of multilingual and language-specific models, democratizing access to ai technologies and preserving linguistic diversity. while substantial progress has been made for widely spoken languages, low and medium resource languages remain underrepresented. greek, * equal contribution in particular, has received limited attention despite its linguistic complexity, rich cultural heritage, and historical significance. addressing this gap, we present llama-krikri-8b, a cutting-edge open large language model tailored for the greek language. built on meta's llama 3.1-8b architecture, llama-krikri has been continually pretrained on a diverse, high-quality greek corpus. this allows the model to effectively capture the syntactic and semantic nuances of greek, while retaining the multilingual strengths of the base model. notably, llama-krikri also supports english and is capable of handling polytonic and ancient greek texts, addressing not only contemporary but also historical forms of the language. compared to meltemi-7b @xcite , the previous state-of-the-art open greek llm built on mistral 7b @xcite , llama-krikri-8b significantly increases the number of parameters, context length, and training data scale. additionally, it features an enhanced posttraining pipeline using both human and synthetic data. following the magpie methodology @xcite , we generate high-quality instructionresponse pairs via prompting of aligned models, and apply multiple rounds of instruction tuning and alignment using direct preference optimization (dpo) @xcite . this pipeline ensures that the model produces helpful, honest, and harmless outputs. to evaluate llama-krikri-8b, we also introduce three novel public benchmarks specifically designed for greek. these, alongside existing evaluation suites, show that llama-krikri outperforms comparable greek and multilingual llms in both natural language understanding and generation, as well as code-related tasks. moreover, it supports function calling and agentic behaviors, opening new application domains for greek users. llama-krikri-8b is available under the llama 3.1 com-munity license agreement foot_0 . our key contributions are: ## background and related work large language models (llms) have achieved state-of-the-art performance across a wide variety of natural language processing (nlp) tasks. these models are typically trained on massive corpora dominated by english, leading to strong performance in english-language tasks but comparatively weaker capabilities in other languages @xcite . as a result, the development of language-specific llms has become an active area of research, particularly for under-represented languages. one prominent strategy for developing such models is continual pretraining, where a pretrained base model is further trained on data in the target language. this approach allows researchers to leverage the general capabilities of large base models while improving performance in specific linguistic domains, without the prohibitive cost of training from scratch @xcite . several recent projects have successfully applied continual pretraining to adapt existing models to new languages. bggpt-gemma-2-27b-instruct @xcite fine-tunes google's gemma-2 model @xcite for bulgarian, combining over 100b tokens of bulgarian and english data and applying techniques such as branch-and-merge to mitigate catastrophic forgetting. similarly, leolm @xcite and the sabiá models @xcite adapt llama and mistral-based architectures for german and portuguese, respectively, through targeted continual pretraining. for greek, meltemi-7b represents the first open generative llm tailored to the language. it was developed by continually pretraining mistral-7b on a substantial greek corpus, followed by instruction fine-tuning. while effective, meltemi's performance is bounded by the size and capabilities of the base model, as well as the limited post-training alignment techniques employed at the time. beyond language adaptation, alignment of llms to generate helpful, harmless, and honest outputs has become increasingly central. early approaches such as instructgpt @xcite and constitutional ai @xcite rely on multi-stage fine-tuning pipelines involving human feedback or rule-based constraints. more recently, dpo and data synthesis methods like magpie have enabled scalable and effective instruction tuning. magpie, in particular, leverages high-quality prompting of already-aligned models to generate large volumes of instruction-response pairs, demonstrating that synthetic data can rival or surpass human-curated datasets. these advancements highlight a trend toward bootstrapping high-quality training data using strong base models, especially in low-resource languages. our work builds on this foundation by employing llama 3.1 as a base architecture, scaling up parameter count and context length, and applying a more rigorous post-training pipeline, including instruction tuning with magpie-generated data and dpo for alignment. ## methodology llama-krikri-8b is based on the transformer architecture @xcite , which has become the de facto standard for large language models. the model inherits its architecture from meta's llama 3.1-8b, leveraging the strong foundation in multilingual understanding, code generation, and reasoning provided by llama 3.1. adapting an llm for the greek language requires addressing the lack of high-quality greek data in the massive datasets typically used to train foundation models. even though llama 3.1's pretraining corpus comprises trillions of tokens, it struggles to generate coherent greek text, thus indicating that greek data is only a tiny fraction of its training data; we should note however that there is limited information on the composition of its pretraining data. our approach is to perform continual pretraining with greek and parallel data to infuse the model with greek knowledge. this training must be done carefully to avoid catastrophic forgetting @xcite of the base model's prior knowledge in other languages and domains. we tackle this by (a) constructing a high-quality, large-scale greek corpus, (b) extending and tuning the tokenizer , (c) interleaving greek training data with datasets containing english, mathematics, code, and parallel data in languages that the model has already been trained on, (d) employing a dataset sampling schedule during training that prefers data closer to the initial llama-3.1 distribution in the beginning, while shifting closer to our true dataset distribution as training continues and (d) re-warming and re-decaying the learning rate @xcite . pretraining data collection & cleaning: as a foundation for continual pretraining, we curated a large corpus of texts totalling approximately 91 billion tokens (after filtering and deduplication), which was upsampled to 110 billion tokens for the final pretraining mix. this corpus was constructed with a primary focus on greek and aiming on retaining and enhancing the original model's capabilities. the distribution included 56.7 billion monolingual greek tokens (62.3%), 21 billion monolingual english tokens (23.1%), 5.5 billion parallel data tokens (6.0%), and 7.8 billion math and code tokens (8.6%). after corpus collection, we implemented a multistage preprocessing and filtering pipeline to ensure a high quality for the pretraining data. various parts of our filtering methodology have been informed by approaches used in @xcite and large-scale corpus creation efforts such as zyda @xcite . however, we have adapted these approaches to cater for the peculiarities of the greek language. we detail the preprocessing pipelines we used in appendix b pretraining data cleaning pipelines. tokenizer and embeddings expansion: the original llama 3 tokenizer comprises 128, 000 tokens and is inefficient for greek texts, as it generally performs character-level tokenization for greek. this was determined, through the approximation of the llama-3.1 tokenizer's fertility @xcite , a metric of the average tokens per word produced. to determine the efficiency of the original llama 3 tokenizer and compare with our approach, we conducted tests on diverse greek and english corpora (each one containing 100, 000 rows and totalling approximately 2m words) and calculated the difference in fertility, as can be seen in in order to develop an optimal tokenizer for greek which is also efficient in historical dialects of the language, as well as in critical domains (such as legal and scientific texts), we extended the llama 3 tokenizer with 20,992 new tokens through a multistage process which encompasses curating highquality texts and allocating new tokens across five domains. this process is especially important during model inference, as it significantly reduces the input and output token cost during model use. furthermore, more compact representations of input text help to improve model performance. we provide detail on the steps for the tokenizer and embeddings expansion in appendix c, tokenizer and embeddings expansion process. greek embeddings training: to effectively integrate newly introduced greek tokens into the model, we implemented an initial, targeted training phase for their corresponding embeddings. by preparing the new token representations prior to full-scale pretraining, we mitigate potential disruptions to the existing model parameters. we initialized the model with llama 3.1-8b-base weights, freezing all but the embeddings and output-projection weights for the 20,992 new tokens, allowing their initial training without large gradient updates to the rest of the model. the dataset for this step was comprised of 5b tokens sampled from the main pre-training dataset. this short, several-thousand-step training regimen ensured a smoother integration of the new vocabulary into the model's existing knowledge representation. instruction tuning and alignment: llama-krikri-8b-instruct was created by fine-tuning the base model for instruction following and dialogue. addressing greek data scarcity and avoiding translation artifacts, the pipeline combined data synthesis, filtering, two-stage supervised fine-tuning (sft), and direct preference optimization (dpo). data collection, synthesis, & curation involved collecting high-quality english instruction, e.g., tulu 3 @xcite and @xcite , and preference data e.g., @xcite . additionally, greek data was synthesized via translation (with post-editing), regenerating responses using llms (gemma-2-27b-it), and generating synthetic instructions directly in greek using the magpie technique. curated corpora from annealing were reused for synthetic q&a and dialogues. data was scored and filtered using the skyword-reward-gemma-2-27b-v0.2 @xcite reward model, known for greek accuracy. rule-based filters ensured formatting and language verification. sft was done in two stages (∼856k pairs in stage 1, ∼638k in stage 2), with progressively higher data quality. datasets included filtered original english, reward-model-filtered synthetic mag-pie data (higher scores in stage 2), translated/postedited data (stage 1), regenerated responses (in-cluding "thinking" in stage 2), multi-language translation data, synthetic qa, synthetic multiturn dialogues, and upsampled manual safety data. training used linear decay lr, adamw, and liger kernels, masking prompt loss. sft produced a model that followed instructions but needed alignment for helpfulness, precision, and safety. dpo provided final alignment using ∼92k preference triplets. data included selected/high-scored original preferences, preferences from magpiesynthesized data (using reward model scores), preferences derived from translated data (contrasting regenerated responses or regenerated vs translated reference), and safety preferences. dpo maximized preferred response likelihood while minimizing dispreferred, using length normalization. training used adamw (loshchilov and hutter, 2017), linear decay lr, and liger kernels. dpo significantly improved response quality, safety, and helpfulness compared to the sft-only model. the dpo-tuned model is the final llama-krikri-8b-instruct. ## evaluation in this section, we present evaluation details for llama-krikri-8b-base and llama-krikri-8b-instruct, across six greek and six english benchmarks. we compare our base model directly with the base model llama-3.1-8b @xcite and the previous greek state-of-the-art model meltemi-7b-v1.5 @xcite . additionally, we evaluate our chat model, llama-krikri-8b-instruct on three challenging english benchmarks, as well as three novelly constructed greek benchmarks which correspond to the english ones. ## discussion and conclusions in this paper, we presented llama-krikri-8b, a new llm that exhibits significant skills in understanding and generating greek, while also showing highly accurate handling of text in english and historical greek dialects. we achieved this by developing an efficient tokenizer that exhibits a low token/words fertility for greek and by further training llama 3.1-8b using a carefully constructed dataset that covered a wide variety of domains. in evaluation experiments on a benchmark suite comprising greek and english datasets, we have observed that llama-krikri-8b performs significantly better in greek (+10.8%) compared to its base model, while also showing gains in english (+0.8%). we then created llama-krikri-8b-instruct, a version designed for following instructions and engaging in helpful conversations. this involved a multi-step process that comprised synthetic data generation in a multitude of domains, fine-tuning the model and then aligning it with human preferences. evaluations revealed that llama-krikri-8b-instruct significantly outperformed llama-3.1-8b-instruct in both greek (+21.7%) and english (+7.3%) ifeval. our model also demonstrated highly competitive chat abilities in both languages across several benchmarks.