Evaluation of QWEN -3 for E nglish to U krainian Translation (2025)

Abstract: We report the results of evaluating Qwen3 for the English-to-Ukrainian language pair of the general MT task of WMT 2025.In addition to the quantitative evaluation, we performed a qualitative evaluation, in collaboration with a native Ukrainian speaker - therefore we present an example-heavy analysis of the typical failures the LLMs still do when translating natural language, particularly into Ukrainian.We report also on the practicalities of using LLMs, such as on the difficulties of making them follow instructions, on ways to exploit the increased “smartness” of the reasoning models while simultaneously avoiding the reasoning part improperly interfering with the chain in which the LLM is just one element.

## introduction this submission to the general mt task of wmt 2025 from fraunhofer fokus continues our series of baselines prepared for the wmt biomedical translation task, as this system was intended for preparing baselines in case the task had been organized this year. those baselines evolved with the field: starting with training sequence-to-sequence models on biomedical corpora @xcite , continuing with transformers trained by us on biomedical corpora @xcite , starting using generic pretrained models like t5 -not necessarily targeting the biomedical field @xcite , then using one of the first widely deployed llms, chat-gpt 3.5 @xcite and eventually using the emerging more capable locally-hosted openmodel llms, llama 3.1 70b @xcite . using locally hosted models keeps data away from cloud providers, leveraging the competitiveness of the locally-hosted large llms such as the * cristian.grozea@fokus.fraunhofer.de qwen3 @xcite we used here. this is particularly important to ensure unfiltered access to the potential hiding in these models, as online services may further censor or otherwise limit the cloud-served models. ## conclusion we have evaluated the use of one of the largest local llms for automatic translation of english to ukrainian. beyond the automated quantitative analysis performed by the task organizers, we have performed a qualitative analysis of several translation errors observed, and explained also the engineering issues one has to deal with when relying on llms for machine translation, such as the incomplete adherence to instructions and the subsequent need for postprocessing, especially when using a "reasoning" llm. we conclude that, although neural machine translation of occasionally challenging texts in natural language has advanced significantly, the llms, as the other neural models before them, continue to be characterized by two aspects: smooth and polished output, convincing thanks to the good form, paired with instances where meaning is sometimes missed -or even reversed. still, when a human with knowledge of the target and of the source languages is proofreading the outcome, the translation process can be significantly accelerated using such a local llm.