From SALAMANDRA to SALAMANDRATA : BSC Submission for WMT 25 General Machine Translation Shared Task (2025)

Abstract: In this paper, we present the SalamandraTA family of models, an improved iteration of Salamandra LLMs (Gonzalez-Agirre et al., 2025) specifically trained to achieve strong performance in translation-related tasks for 38 European languages. SalamandraTA comes in two scales: 2B and 7B parameters. For both versions, we applied the same training recipe with a first step of continual pre-training on parallel data, and a second step of supervised fine-tuning on high-quality instructions.The BSC submission to the WMT25 General Machine Translation shared task is based on the 7B variant of SalamandraTA. We first extended the model vocabulary to support the additional non-European languages included in the task. This was followed by a second phase of continual pretraining and supervised fine-tuning, carefully designed to optimize performance across all translation directions for this year’s shared task. For decoding, we employed two quality-aware strategies: Minimum Bayes Risk Decoding and Translation Reranking using Comet and Comet-kiwi.We publicly release both the 2B and 7B versions of SalamandraTA, along with the newer SalamandraTA-v2 model, on Hugging Face.

## introduction traditionally, massively multilingual neural machine translation (mmnmt) relied on the encoderdecoder architecture to translate across multiple languages @xcite . more recently, however, large language models (llms) have demonstrated strong mm-nmt capabilities @xcite and thus some works have proposed several strategies to improve the translation capabilities of a pre-trained llm model and better align it with human translations @xcite @xcite . one such approach is continual pre-training using a combination of monolingual and parallel corpora followed by supervised fine-tuning @xcite . however, most previous approaches have predominantly relied on english-centric parallel corpora. this has been shown to bias the models towards english-centric latent representations @xcite which has been attributed to the language distribution used in the training corpora @xcite . it is well known that training with only a single bridge language can negatively impact translation performance across zero-shot language pairs, due to limited cross-lingual transfer @xcite . unlike previous works, in this paper we rely on parallel corpora only for the continual pre-training stage pivoting on three bridge languages. when working with pre-trained language models on languages not covered by their original tokenizer, a highly effective solution involves replacing the existing tokenizer with a more comprehensive one that supports such languages. for the newly introduced tokens, embeddings must be initialized. in our work, these new embeddings were initialized to the average of all existing embeddings and then rapidly optimized through continual pre-training (cpt). this method has not only proven to be viable but also demonstrably improves the model's overall performance in the target languages, even if the original model was never exposed to data from these languages during its initial training @xcite . throughout this paper, we present the sala-mandrata family of models, which serve as the backbone models of the bsc team's submission to the wmt25 general machine translation shared task. our participation covers 15 out of the 16 translation directions in the general mt task under the constrained track. additionally, we took part in the multilingual subtask for 7 out of the 16 directions. contributions of this work are listed as follows: ## data our base models are salamandra-2b and sala-mandra-7b @xcite , which were trained from scratch on highly multilingual data. however, salamandra models were not exposed to parallel data during pre-training. to address this, and following alves et al. (2024), we improve their multilingual machine translation capabilities by performing continual pre-training on parallel data covering 38 european languages (35 of which were already present in the original pre-training corpus). this step is followed by supervised fine-tuning using high-quality instruction data. in this section, we detail the datasets used for both continual pre-training and supervised finetuning. ## salamandrata models the salamandrata family is composed of two base models, 2b and 7b parameters, which were continually pre-trained on the cpt-v1 corpus and subsequently instruction-tuned on it-v1. for our submission to the wmt25 general translation shared task, we further adapted the 7b model, resulting in salamandrata-v2. ## results as shown in although further adapting the salaman-drata-7b model to wmt-2025 language pairs initially causes an average performance drop of 1.09 comet points on the language directions shared between salamandrata-7b and sala-mandrata-v2, this gap is largely mitigated when employing quality-aware decoding strategies. applying minimum bayes risk (mbr) and tuned re-ranking (trr) decoding strategies reduces this drop to 0.16 and 0.20 comet points, respectively. on the impact of adding non-mt-tasks to better understand the impact of different instruction types on translation quality, we conduct an ablation study of instruction fine-tuning across four main task categories: machine translation (mt), pre-translation tasks (pre-mt) (e.g., named entity recognition), post-translation tasks (post-mt) (e.g., gender bias mitigation), and chat/coderelated tasks foot_1 . instruction fine-tuning using mt tasks consistently yields the best overall performance across most evaluation metrics, with the exception of metricx. for metricx, a combination of mt, pre-mt, and post-mt instructions results in slightly improved performance. however, leads to a consistent drop in bleu scores without measurable gains in other metrics. based on these findings, we concluded that for salamandrata-2b and 7b, incorporating both pre-mt and post-mt tasks alongside mt tasks provided a slight benefit or at least no degradation in performance, leading to their inclusion in the it-v1 dataset. however, for salamandrata-v2 which was specifically tailored for the wmt25 general translation shared task, we made a deliberate choice to focus exclusively on mt instructions. while pre-mt and post-mt tasks might offer benefits, gathering high-quality, task-specific instruction data for the unique language pairs and domains present in wmt25 would have required significant additional effort beyond the scope of this work. on the robustness to character noise following peters and martins (2025), we investigate model robustness by injecting character-level noise into the source sentences of flores-200-devtest for the english to spanish direction using adjacent swaps, duplications, and deletions at different noise levels. figure 2 shows the relative degradation in bleu score compared to zero-noise baseline. the sala-mandrata 7b instruction-tuned model consistently shows greater resilience than the base model across all perturbation types. at the maximum noise level (1.0), the performance degradation of the instruction-tuned model is smaller by 17.63 p.p. for swaps, 20.61 p.p. for duplications, and 18.33 p.p. for deletions. these results demonstrate that instruction tuning effectively improves a model's robustness to character-level input corruptions. ## submission for our wmt25 general and multilingual mt tasks submissions, we apply a chunking strategy, splitting each input instance at \n\n delimiter prior to translation. we made two submissions using two quality-aware decoding strategies: minimum bayes risk decoding employing comet and tuned re-ranking relying on comet-kiwi. ## conclusion in this paper, we introduced the salamandrata family of models, a series of powerful, translation llms in 2b and 7b scales. our approach combines a multi-stage training recipe, beginning with continual pre-training on parallel data that pivots through three languages: english, spanish, and catalan. this is followed by an instruction tuning stage to align the models with human translation outputs. for our wmt25 submission, we adapted our 7b model to new, non-european languages through vocabulary adaptation and a further round of continual pre-training and supervised fine-tuning. our experimental results show that instruction tuning is a critical step which not only improves translation quality but also the model's robustness against character-level noise. furthermore, our analysis of the english-to-bhojpuri direction validates the importance of including related languages during pre-training to enable cross-lingual transfer to low-resource pairs. while our work successfully specializes models for translation and translation-related tasks, we observed that incorporating chat and code instructions during the supervised fine-tuning stage leads to a significant drop in translation quality as measured by bleu. future work could explore methods to mitigate this trade-off to train machine translation models that can follow general instructions without compromising their specialized translation capabilities. bryan eikema and wilker aziz. 2020. is map decoding all you need? the inadequacy of the mode in neural machine translation. in proceedings of the 28th international conference on computational linguistics, pages 4506-4520, barcelona, spain (online). international committee on computational linguistics. andreas eisele and yu chen. 2010. multiun: a multilingual corpus from united nation documents. in proceedings of the seventh international conference on language resources and evaluation (lrec'10), valletta, malta. european language resources association (elra). ahmed el-kishky, vishrav chaudhary, francisco guzmán, and philipp koehn. 2020. ccaligned: a massive collection of cross-lingual web-document pairs. in proceedings of the 2020 conference on empirical methods in natural language processing (emnlp 2020), pages 5960-5969, online. association for computational linguistics. ahmed el-kishky, adithya renduchintala, james cross, francisco guzmán, and philipp koehn. 2021. xlent: mining a large cross-lingual entity dataset with lexical-semantic-phonetic word alignment. in proceedings of the 2021 conference on empirical methods in natural language processing, pages 10424-10430, online and punta cana, dominican republic. association for computational linguistics. elrc-share. 2020. bilingual corpus made out of pdf documents from the european medicines agency (emea). . angela fan, shruti bhosale, holger schwenk, zhiyi ma, ahmed el-kishky, siddharth goyal, mandeep baines, onur celebi, guillaume wenzek, vishrav chaudhary, naman goyal, tom birch, vitaliy liptchinsky, sergey edunov, edouard grave, michael auli, and armand joulin. 2021. beyond english-centric multilingual machine translation. j. mach. learn. res., 22(1). christian federmann, tom kocmi, and ying xin. 2022. ntrex-128 -news test references for mt evaluation of 128 languages. in proceedings of the first workshop on scaling up multilingual evaluation, pages 21-24, online. association for computational linguistics. fangxiaoyu feng, yinfei yang, daniel cer, naveen arivazhagan, and wei wang. 2022. language-agnostic bert sentence embedding. in proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pages juraj juraska, mara finkelstein, daniel deutsch, aditya siddhant, mehdi mirzazadeh, and markus freitag. 2023. metricx-23: the google submission to the wmt 2023 metrics shared task. in proceedings of the eighth conference on machine translation, pages 756-767, singapore. association for computational linguistics. philipp koehn. 2005. europarl: a parallel corpus for statistical machine translation. in proceedings of machine translation summit x: papers, pages 79-86, phuket, thailand. sneha kudugunta, isaac caswell, biao zhang, xavier garcia, derrick xin, aditya kusupati, romi stella, ankur bapna, and orhan firat. 2023. madlad-400: a multilingual and document-level large audited dataset. in proceedings of the 37th international conference on neural information processing systems, nips '23, red hook, ny, usa. curran associates inc. pierre lison and jörg tiedemann. 2016. opensub-titles2016: extracting large parallel corpora from movie and tv subtitles. in proceedings of the tenth international conference on language resources and evaluation (lrec'16), pages 923-929, portorož, slovenia. european language resources association (elra). minh-thang luong and christopher manning. 2015. stanford neural machine translation systems for spoken language domains. in proceedings of the 12th international workshop on spoken language translation: evaluation campaign, pages 76-79, da nang, vietnam. minh-thang luong and christopher d. manning. 2016. achieving open vocabulary neural machine translation with hybrid word-character models. in proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1054-1063, berlin, germany. association for computational linguistics. thang luong, hieu pham, and christopher d. manning. 2015. effective approaches to attention-based neural machine translation. in proceedings of the 2015 conference on empirical methods in natural language processing, pages 1412-1421, lisbon, portugal. association for computational linguistics. nllb team, marta r. costa-jussà, james cross, onur çelebi, maha elbayad, kenneth heafield, kevin heffernan, elahe kalbassi, janice lam, daniel licht, jean maillard, anna sun, skyler wang, guillaume wenzek, al youngblood, bapi akula, loic barrault, gabriel mejia gonzalez, prangthip hansanti, and 20 others. 2022. no language left behind: scaling human-centered machine translation. preprint, arxiv:2207.04672. open ai. 2023. [link]. kishore papineni, salim roukos, todd ward, and wei-jing zhu. 2002. bleu: a method for automatic evaluation of machine translation. in proceedings of the 40th annual meeting of the association for computational linguistics, pages 311-318, philadelphia, pennsylvania, usa. association for computational linguistics. ben peters and andre martins. 2025. did translation models get more robust without anyone even noticing? in proceedings of the 63rd annual meeting of the association for computational linguistics (volume 1: long papers), pages 2445-2458, vienna, austria. association for computational linguistics. maja popović. 2015. chrf: character n-gram f-score for automatic mt evaluation. in proceedings of the michał ziemski, marcin junczys-dowmunt, and bruno pouliquen. 2016. the united nations parallel corpus v1.0. in proceedings of the tenth international conference on language resources and evaluation (lrec'16), pages 3530-3534, portorož, slovenia. european language resources association (elra).