T artu NLP at WMT 25 LLM s with Limited Resources for S lavic Languages Shared Task (2025)

Abstract: This paper describes the TartuNLP submission to the Upper Sorbian (hsb) and Lower Sorbian (dsb) tracks of the WMT25 LLMs with Limited Resources for Slavic Languages shared task, which jointly targets machine translation (MT) and question answering (QA). We develop a single multilingual model based on Qwen2.5-3B-Instruct by continuing pretraining on Sorbian monolingual and parallel data together with general instruction datasets, combining language acquisition and instruction-following in a single step. The resulting model delivers substantial improvements over the baseline Qwen2.5-3B-Instruct model and also achieves the highest ranking for both tasks in the hsb and dsb shared task tracks.

## introduction this paper presents an overview of the tartunlp systems developed for the wmt25 limited resource slavic languages shared task @xcite . this shared task aimed to create a single large language model (llm) capable of jointly performing both machine translation (mt) and question answering (qa) for less-resourced slavic languages. the participants of the shared tasks were limited to using the qwen2.5 model family @xcite with a size constraint of 3b parameters. our team participated in the upper sorbian (hsb) and lower sorbian (dsb) tracks, both endangered languages spoken by only about 20,000-30,000 people in total @xcite . the taxonomy of @xcite categorizes both languages as category-1, the scraping-bys. we focused on building a single model that supports both tasks and languages simultaneously. this joint objective introduces a specific challenge: while a moderate amount of mt data exists for sorbian, there is no qa training data, requiring balancing performance across both tasks. although one or both of the sorbian languages have been included in recent massively multilingual models @xcite @xcite , to our knowledge, no prior work has developed dedicated llms for sorbian. we build on recent work adapting llms to extremely low-resource languages through continued pretraining and instruction tuning @xcite @xcite . our approach follows @xcite , who demonstrated for the basque language that combining language acquisition and instruction tuning in a single step and starting from an instruction-tuned model is beneficial. we also adopt this joint learning of instruction-following and language for an instruction-tuned base model in our system. we continually pretrain qwen2.5-3b-instruct (qwen team, 2024) on a mix of monolingual documents and sentences in hsb and dsb, general instruction-following data (primarily in english), and sorbian mt instructions. we supplement the data provided by the organizers with document-level texts from fineweb-2 @xcite and wikipedia articles (wikimedia foundation). our final model outperforms the baseline qwen2.5-3b-instruct and also achieves the highest rank in the shared task (see tables 1 and 2 ). monolingual data (see parallel data (see the monolingual and parallel data was deduplicated with normalization from stopes (pierre andrews, 2022; @xcite . we also removed the held-out and validation data using the same normalization method. instructions (see the final mix. the data mix of the submitted model consisted of 1.198b tokens and is displayed in figure 1 . ## methodology we used qwen2.5-3b-instruct (qwen team, 2024) as our base model, motivated by the findings of @xcite , who demonstrated that continued pretraining on already instruction-tuned models is effective for low-resource languages. following their findings, we combined the language acquisition and instruction-tuning in the same continued pretraining step. our continued pretraining was performed jointly on monolingual documents and sentences, parallel mt data, and instructionformatted data. our initial experiments indicated that the model begins to overfit on sorbian data around the fourth epoch. a similar finding was also made in @xcite , who observed overfitting at 4 epochs for low-resource languages of similar size. with this in mind, we repeated the sorbian monolingual and parallel data four times during training. the instruction data was repeated once, while the significantly more abundant german monolingual data (not used for the final submission) was limited to 25% of the total training token budget. it should be noted that we did not conduct a thorough investigation into the different data sampling strategies and combinations. a different number of epochs or curriculum learning might provide better results. for instruction-formatted samples, we applied loss only on the assistant (target) tokens. all datasets were packed into sequences of 4096 tokens, with any overflow tokens carried into the next training sequence. the hyperparameters for the model training are listed in ## evaluation we evaluate our models using the lm-eval-harness framework @xcite , which we also use to generate the final shared task submissions. evaluation is conducted using a few-shot and zero-shot prompting strate-gies, depending on the training strategy. we use a few-shot evaluation without a chat template and a zero-shot evaluation using a chat-style format. for qa, we follow a multiple-choice setup, selecting the answer option with the highest logprobability from a predefined set of candidates. we calculate the accuracies and report the average across the language levels in the validation set. for the mt evaluation, we calculated the bleu @xcite and chrf++ @xcite scores. we also report 95% confidence intervals calculated from standard errors reported by lm-eval-harness. for the final submission, we apply beam search with a beam size of 4 for mt, and use one-shot prompting with the chat template for qa, where each example is treated as a separate turn in a multiturn conversation. we use greedy decoding in all other evaluation settings (beam size of 1). ## conclusion we presented the tartunlp submission to the wmt25 shared task on limited resource slavic languages, targeting both machine translation and question answering for upper and lower sorbian. our approach combined continued pretraining and instruction tuning in a single step, leveraging the qwen2.5-3b-instruct model. by integrating task-specific and monolingual sorbian data, we achieved significant improvements over existing baselines and obtained the highest rank for both tasks in both languages. our results demonstrate that a unified approach can effectively serve multiple low-resource language tasks, even under resource constraints.