JGU Mainz’s Submission to the WMT 25 Shared Task on LLM s with Limited Resources for S lavic Languages: MT and QA (2025)

Abstract: This paper presents the JGU Mainz submission to the WMT25 Shared Task on LLMs with Limited Resources for Slavic Languages: Machine Translation and Question Answering, focusing on Ukrainian, Upper Sorbian, and Lower Sorbian. For each language, we jointly fine-tune a Qwen2.5-3B-Instruct model for both tasks with parameter-efficient finetuning. Our pipeline integrates additional translation and multiple-choice question answering (QA) data. For Ukrainian QA, we further use retrieval-augmented generation. We also apply ensembling for QA in Upper and Lower Sorbian. Experiments show that our models outperform the baseline on both tasks.

## introduction while large language models (llms) are strong multitask learners for high-resource languages such as english, this is not the case for smaller llms and languages with limited data. in this setting, a trade-off presents between the performance on different tasks. the wmt25 shared task on llms with limited resources for slavic languages: mt and qa foot_0 focuses on the development of relatively small llms (≤3b parameters) that are capable of performing both machine translation (mt) and multiple-choice question answering (qa) in slavic languages with limited amounts of data. three languages -a mid-resource language, ukrainian (uk), and two severely low-resource languages, upper sorbian (hsb) and lower sorbian (dsb)are targeted, and only qwen2.5 models with 0.5b, 1.5b, or 3b parameters are permitted. the mt source languages are czech (cs) and english (en) for ukrainian as well as german (de) for dsb and hsb. our proposed approach consists of a qwen2.5-3b-instruct model @xcite , which is inherently multilingual and which we jointly finetune on both mt and qa data, combining the provided resources with additional datasets we curate ourselves. for dsb and hsb mt, we enhance the training data using synthetic data generated through back-translation. we further add an additional parallel dataset for dsb. for qa, we add a total of 16 high-quality english mcq datasets. all qa datasets are enhanced via automatic translation such that they are bilingual (english and the target language). for ukrainian qa, we incorporate retrieval-augmented generation (rag) using domain-relevant wikipedia pages and 10 books related to the subjects of the provided ukrainian mcq dataset. at inference time, we use similarity-based fewshot in-context learning (icl) for mt. for qa, we permute the order of the answer options, and average the probabilities for all options, to increase robustness against answer ordering biases @xcite . while not winning for any task-language combination, our primary submission consistently outperforms the baseline on all tasks, demonstrating the effectiveness of our approach. for de-dsb translation, chrf++ improves by over 55 points, while de-hsb translation sees gains of over 65 points, reflecting substantial quality improvements. on dsb qa, accuracy increases by up to 12.34 percentage points, and, on hsb qa, accuracy increases by 10.27 points. for cs-uk and en-uk mt, chrf++ improves by 4.61 and 2.7, respectively, while, on ukrainian qa, our submission outperforms the baseline by 4.66 accuracy points. ## results and discussion mt for lower and upper sorbian qa for ukrainian for ukrainian qa ( ## conclusion in this paper, we present jgu mainz's submission to the wmt25 shared task on llms with limited resources for slavic languages, addressing mt and qa in ukrainian, upper sorbian, and lower sorbian. our approach combines parameterefficient finetuning of qwen2.5-3b-instruct with training data augmentation and rag for ukrainian qa. our primary submissions outperform the provided baselines for all languages and tasks, achieving substantial chrf++ gains for de-dsb and de--hsb mt, as well as slight improvements for cs-uk and en--uk mt. for qa, averaging over order options increases accuracy for both dsb and hsb, while, for ukrainian, we achieve moderate gains through rag.