AMI at WMT 25 General Translation Task: How Low Can We Go? Finetuning Lightweight Llama Models for Low Resource Machine Translation (2025)

Abstract: We present the submission of the Árni Magnússon Institute’s team for the WMT25 General translation task. We focus on the English-Icelandic translation direction. We pre-train Llama 3.2 3B on 10B tokens of English and Icelandic texts and fine-tune on parallel corpora. Multiple translation hypotheses are produced first by the fine-tuned model, and then more hypotheses are added by that same model further tuned using contrastive preference optimization. The hypotheses are then post-processed using a grammar correction model and post-processing rules before the final translation is selected using minimum Bayes risk decoding. We found that while it is possible to generate translations of decent quality based on a lightweight model with simple approaches such as the ones we apply, our models are quite far behind the best participating systems and it would probably take somewhat larger models to reach competitive levels.

## introduction large language models (llms) are becoming the predominant approach for a wide variety of tasks in the field of natural language processing. they have shown remarkable translation capabilities, see e.g. @xcite , especially for well-resourced languages such as english and spanish, but also for many low-resource languages (lrls) as @xcite show for translations between english and icelandic. the largest of these models, such as @xcite , are hardware and energy intensive, both in training and at inference time, and thus costly. the vast majority of open weights large language models, such as the llama family of models @xcite , aya @xcite , mistral @xcite and others, are primarily trained on english and other languages that are well represented on the internet, for obvious availability reasons. this has ramifications for lrls. not only do the models offer inferior performance for lrls 'out-of-the-box', their vocabulary is typically underrepresented due to the smaller amounts of training data in these languages, see e.g. @xcite . this leads to less efficient tokenization for the lrls, meaning that the number of characters per token are considerably fewer than for well-resourced languages like english. for example, using the llama-3.2 tokenizer, the average number of characters per token for icelandic is ≈2.2, while for english, each token has ≈4 characters. for lrls, more tokens are thus necessary to cover the same context length. while we do encounter challenges when working with lrls in the context of llms, there are a variety of approaches to increase the capabilities of the models in that regard. @xcite trained their alma translation models based on llama-2 @xcite by continual pre-training (cpt) and then fine-tuning the models on the translation task. one of the languages pairs they worked with was english↔icelandic and they achieved results very competitive to previous models trained for that language pair. in this paper, we will be working with that same language pair, english-icelandic. we are interested in building models that are as lightweight as possible, while still retaining the translation capabilities of llms. we experiment with applying the ap-proach used for training the alma models to train bilingual translation models based on the llama-3.2 models @xcite . we compare the 1b parameter model to the 3b parameter model. for our training, we use a much larger icelandic monolingual dataset than @xcite , as well as a larger parallel dataset for english-icelandic. we find that the 1b parameter model produces considerably lower quality translations and is much more prone to hallucinate. our final system, submitted to the wmt25 general translation task @xcite is thus based on llama-3.2 3b parameter model. following @xcite , we experiment with contrastive preference optimization (cpo) on top of the fine-tuned model. our system generates multiple hypotheses, using different temperature settings, with and without cpo. the hypotheses are then post-processed using a grammar error correction (gec) model and post hoc rules to fix punctuation errors as well as mistakes in translating emojis, hashtags, emailaddresses and urls. our code is available on github 1 and the translation model on huggingface 2 . ## related work up until 2020, when jónsson et al. (2020) published the first paper describing smt and nmt for translations between english and icelandic, not much work had been done with regard to mt for this language pair. since wmt 2021, when english↔icelandic was one of the language pairs for the news translation task @xcite , multiple mt publications have described mt research on icelandic, using the wmt21 evaluation dataset. in 2024, the ami team submitted a system to the wmt general translation task for the @xmath0. the submission describes an effort to build a lightweight nmt system, using an encoder-decoder architecture. while it was small enough to run easily on a laptop computer, it still scored higher than many commercial systems @xcite . in their work on building mt systems from the llama-2 models, @xcite pre-train models of two different sizes, 7b and 13b parameters, on data in six languages, two of these being english and icelandic. they then fine-tune the model on the translation task. ## building the system in building our model, we followed the approach used for training the alma-r models @xcite , but instead of training on six languages, we trained only on texts in english and icelandic. we are interested in investigating whether using some of the smallest available open llms can produce competitive translations and thus experiment with the lightweight llama 3.2 1b and 3b parameter models. hyperparameters used in training are reported in appendix a. @xcite use the oscar 23.01 corpus @xcite which only contains approx. 300m running words in icelandic. furthermore, for fine-tuning english↔icelandic, they only use 2000 sentence pairs. in our experiments, we extend both the monolingual and parallel data sets used. ## translation pipeline figure 3 shows the translation pipeline. input documents to be translated are split into paragraphs and the mt system uses different settings for number of input and output tokens depending on paragraph length measured in number of characters. 18 translation candidates are produced, 9 with the fine-tuned model and 9 with the model additionally trained using cpo. in each case, 3 different temperatures are used. a gec model and post-processing rules, as described in section 3.5, are applied to all translations before comet-mbr selects the top translation candidate. finally, post-processing rules are applied again to the translated paragraph before document translations are constructed from the paragraphs. ## results in the wmt25 general translation task, automatic evaluation of participating systems was carried out using three families of evaluation methods: llm-as-a-judge (reference-less), trained reference-based metrics and trained quality estimation (qe). the results, reported in @xcite , are given in while 12 systems out of 33 score higher on average than our system using automatic metrics, and 9 systems score higher than us in the human evaluation, we have the second smallest model in terms of parameters and a smaller model than all higher scoring ones, at least those where the size is known. we score above our average on the two referencebased metrics, but lower when llm-as-a-judge is used. looking at the human evaluation results, we see that gpt 4.1 has the same order of systems for the top 5, but when the outputs are not as good, it starts to differ from the human evaluation. ## conclusions and future work we experiment with fine-tuning very lightweight llms for translation and find that while our 3b parameter model can produce quite intelligible translations from english to icelandic, they are still of considerably less quality than popular online systems and some larger language models. while we do not achieve building a model that is competitive with the best models, it is fast and can easily run locally on a modern laptop. inference is thus inexpensive and can be fast. we fine-tuned our model on 101k parallel sentence pairs. while we experimented with the combination of available datasets, we did not inspect why some worked better than others, e.g. why the quality was going down for the baseline data when training with more than 20k sentence pairs, but if synthetic data were added, the quality improved? what factors are at play here? we are interested in investigating that, starting with looking at diversity in the fine-tuning data. cpo did not improve our model. we intend to look into why that was, whether it may be related to the size of the dataset or if adding more languages would be beneficial.