Findings of the WMT 2025 Shared Task LLM s with Limited Resources for S lavic Languages: MT and QA (2025)

Abstract: We present the findings of the WMT 2025 Shared Task LLMs with Limited Resources for Slavic Languages. This shared task focuses on training LLMs using limited data and compute resources for three Slavic languages: Upper Sorbian (hsb), Lower Sorbian (dsb), and Ukrainian (uk), with the objective to develop and improve LLMs for these languages. We consider two tasks which are to be evaluated jointly: Machine Translation (MT) and Multiple-Choice Question Answering (QA). In total, three teams participated in this shared task, with submissions from all three teams for the Sorbian languages and one submission for Ukrainian. All submissions led to an improvement compared to the baseline Qwen2.5-3B model through varying fine-tuning strategies. We note, however, that training purely on MT degrades original QA capabilities. We also report further analyses on the submissions, including MT evaluation using advanced neural metrics for Ukrainian, as well as manual annotation and comparison to the current Sorbian machine translator.

## introduction for a large majority of the world's languages, only limited resources are available for training nlp tools, but modern large language models (llms) need large amounts of both labelled and unlabelled data to function well. improving the coverage of low-resource languages in llms is an active research area. recent examples include @xcite for indic languages and @xcite for ethiopian languages. similarly, there is active work on low-resource machine translation, with recent shared tasks and datasets covering translation for low-resource indic languages @xcite , creole languages @xcite as well as indigenous languages of the americas @xcite . although commercial llms increasingly show high performance on both general tasks and machine translation, specialised mt models are still typically required for best results. our challenge to participants in this shared task is to build a model under low-resource conditions to jointly optimise machine translation and question answering. we aim to study potential synergy effects between these two tasks, as well as to explore whether optimising for one task in a low-resource setting will negatively impact the other task. we are one of the first wmt shared tasks to focus on joint optimisation of machine translation (mt) and question answering (qa). the multilingual instruction shared task in the same year took a similar approach, but allowed significantly larger models. our task focuses on three slavic languages: upper sorbian, lower sorbian, and ukrainian. thus, we aim to highlight both truly low-resource settings and mid-resource language scenarios in the context of modern language technologies. as our goal is to evaluate llms as general-purpose tools for a given language, we designed our setup to mirror widely adopted benchmarks such as glue @xcite and mmlu @xcite . since sorbian languages and ukrainian currently lack such comprehensive language understanding benchmarks, we approximated a multitask evaluation by selecting two representative tasks: machine translation and question answering. previous iterations of the wmt shared tasks on translating low-resource languages @xcite @xcite compared supervised and unsupervised translation in various data settings. for both sorbian languages, the witaj-sprachzentrum provided new machine translation and question answering datasets for this shared task. for ukrainian, the mt portion of the dataset corresponds to that of the wmt 2025 general translation task, foot_0 and the qa portion is based on the dataset from the unlp 2024 shared task on fine-tuning llms for ukrainian @xcite . our research questions are as follows: this article is structured as follows: section 2 describes the shared task rules, while section 3 details the mt and qa datasets provided for both development and test phases. section 4 presents the systems devised by the three participating teams. section 5 displays the official leaderboard for the primary submissions in all three tracks. section 6 analyses the model outputs with some additional experiments. ## datasets in the following, we describe the datasets provided for the development and test phases. foot_3 ## system descriptions submissions are language-specific; participants could submit to one or more language tracks. however, participants had to submit both the qa and the mt outputs, which had to be generated by the same model. in addition to our baseline outputs, three teams submitted to the upper and lower sorbian tracks, and one team also submitted to the ukrainian track. for the final evaluation, each team was asked to choose a primary submission and provide a short description of their approach. based on these descriptions, we provide an overview of the participating systems here. baseline (tum organisers) our simple baseline prompts qwen2.5-3b-instruct (qwen @xcite with no fine-tuning. the prompts are zero-shot, and an example is shown in appendix a. we implemented our tasks in the llm evaluation harness framework @xcite and provided this code to participants for reference. 6 team nrc (national research council canada) @xcite the nrc submissions focused primarily on the machine translation (mt) component of the shared task. the team explored the impacts of training on upper sorbian and lower sorbian data separately and together. they also experimented with direct preference optimisation using pairs of correct and incorrect responses to the question answering (qa) set, with limited success. their submitted systems are based on the qwen2.5-1.5b-instruct model, trained using supervised fine-tuning on full model weights using llama-factory and 4 gpus (tesla v100-sxm2-32gb). team sdkm (jgu mainz) (saadi et al., 2025) team sdkm trained three separate qwen2.5-3b-instruct models for three languages: lower sorbian, upper sorbian, and ukrainian. for lower sorbian, the team trained a qwen2.5-3b-instruct model by combining both machine translation (mt) and question answering (qa) data. then, they fine-tuned a joint model on a combined mt and qa dataset. after this fine-tuning, they finetuned the model on all the provided qa data and 3k mt data for a second round of fine-tuning. this final fine-tuned model was used for both mt and qa tasks. for upper sorbian (hsb), the team followed the same overall approach as with lower sorbian. during the qa evaluation of dsb and hsb, they made multiple versions of the same mcq question with different possible orders and averaged the likelihood of each option, then selected the option with the maximum likelihood. for ukrainian, they also followed a similar approach, but for qa, they employed retrieval-augmented generation (rag). the team will make all the data and code they used for pre-processing, training, and their trained model publicly available shortly after the deadline. they used the qwen2.5-3b-instruct model as all translation models, for semantic similarity calculation to incorporate retrieval augmented generation. team tartunlp (tartunlp) @xcite the tartunlp system fine-tuned qwen2.5-3b-instruct (qwen team et al., 2025) on a mixture of monolingual, parallel, and instruction data to support both lower and upper sorbian. the monolingual set included all sentence-level sorbian data from current and past wmt shared tasks, up-per and lower sorbian wikipedia articles (foundation, 20250520 dump), and upper and lower sorbian documents from fineweb-2 (penedo et al., 2025). parallel data, sourced from wmt (current and past), was reformatted as chat-style instruction pairs, with four epochs of german-to-sorbian and one epoch of sorbian-to-german translations. both monolingual and parallel sorbian data were repeated four times. the instruction data was collected from magpie (xu et al., 2024), aya (singh et al., 2024), euroblocks (martins et al., 2025), openassistant (köpf et al., 2023), and flan v2 (longpre et al., 2023), covering multiple languages. all datasets were deduplicated and packed into 4096-token sequences, with loss applied only to assistant responses in instruction-formatted data. for the final submission, they used beam search with a beam size of 4 for machine translation and oneshot prompting with development set examples for question answering. the final model is published on huggingface. foot_6 ## deeper analysis and discussion this section focuses on a few analyses of the results. for ukrainian, to approximate a more advanced evaluation, we compared the participants and the baseline mt results with xcomet (guerreiro et al., 2024) ( §6.1). for the sorbian language tracks, we check the qa accuracy per language level ( §6.2), contrast the translation performance against other mt approaches ( §6.3.1), namely against the current sorbian-german translator, and perform a manual annotation of translation outputs ( §6.3.2). for the @xmath0, both xl and xxl models confirmed that the differences between systems were not statistically significant. in contrast, for the @xmath1, the xl model results were borderline with respect to the null hypothesis, while the xxl model clearly indicated a significant difference. these findings confirm that the sdkm team achieved better performance for @xmath2. in addition, we conducted a qualitative analysis of the mt outputs at the sample level. a native ukrainian speaker evaluated the translations for both fluency and adequacy. representative examples from both models are included in the appendix b.2. we observe that both models struggled with paragraph-level translation. in several cases, the baseline system failed to generate any output 9 https://huggingface.co/unbabel/xcomet-xl 10 https://huggingface.co/unbabel/xcomet-xxl at all. the sdkm team's results, however, are particularly interesting: although their translations did not reproduce the full paragraphs, they captured the main content, resembling a blend of translation and summarisation. this suggests that, with more granular input pre-processing, the model could potentially produce more accurate translations. ## conclusion the wmt 2025 shared task llms with limited resources for slavic languages was the first attempt to evaluate two tasks jointly, machine translation and question answering, and to assess how they impact each other. we focused on three slavic languages: upper and lower sorbian (paired with german for mt), and ukrainian (paired with czech and english). submissions were constrained to open-source datasets and qwen 2.5 models below 3b parameters for reproducibility. three teams participated. tartunlp was the overall winner by jointly fine-tuning the model using sorbian and instruction datasets. nrc won both sorbian mt tasks with a smaller 1.5b model by focusing on the mt task only. sdkm submitted to all three tracks using additional external datasets as well as data augmentation with machine translation and won on the ukrainian track. all sub-missions improved the machine translation quality over the baseline qwen 2.5 3b model. we observe that only focusing on mt negatively affects qa performance, answering our first research question. however, improving the model capabilities on two different tasks remains possible even for small llms. this first shared task paves the way for an extension to other tasks and low-resource languages. besides, diverging data formatting might have added another layer of complexity, which is orthogonal to our research question. for ukrainian mt, the sentence-level training data contrasted with the document-level input in the test set, and for sorbian qa, the exercise variety in type and number of possible answers proved to be a challenge. finally, the shared task focused on track-based (i.e., language-specific) approaches for models and not a fully multilingual llm for all three slavic languages. the submissions to the sorbian tracks showed that fine-tuning with both languages proved to be mutually beneficial.