Do Not Change Me: On Transferring Entities Without Modification in Neural Machine Translation - a Multilingual Perspective (2025)

Abstract: Current machine translation models provide us with high-quality outputs in most scenarios. However, they still face some specific problems, such as detecting which entities should not be changed during translation. In this paper, we explore the abilities of popular NMT models, including models from the OPUS project, Google Translate, MADLAD, and EuroLLM, to preserve entities such as URL addresses, IBAN numbers, or emails when producing translations between four languages: English, German, Polish, and Ukrainian. We investigate the quality of popular NMT models in terms of accuracy, discuss errors made by the models, and examine the reasons for errors. Our analysis highlights specific categories, such as emojis, that pose significant challenges for many models considered. In addition to the analysis, we propose a new multilingual synthetic dataset of 36,000 sentences that can help assess the quality of entity transfer across nine categories and four aforementioned languages.

## introduction machine translation is one of the oldest branches of natural language processing (nlp), which, thanks to the transformer @xcite architecture incorporating the (self-)attention @xcite mechanism, achieves human-like quality in many translation directions, especially in the case of sentence-level translations @xcite . although the overall quality of recent neural models is very high both in terms of human perception and metrics such as bleu @xcite or comet @xcite , there are still some weaknesses that need to be addressed. one of them is the problem of identifying text fragments that should be copied without modification into the target sentence. many of these entities: phone numbers, email addresses, or company names, represent categories that occur frequently in texts. however, the possible instantiations of these categories are so numerous (e.g., all possible phone numbers or email addresses) that models must rely on contextual information to detect them, rather than memorizing these entities. in this paper, we show that many popular nmt models have problems with transferring such entities without modification. to analyze this issue, we focus on translations between four languages: english, german, polish, and ukrainian, using sentences containing entities from 9 categories: alphanumeric sequences, emails, emojis, ibans, ip numbers, isbns, phone numbers, social handlers, and urls. the main contributions of this paper are: (i) a new multilingual dataset consisting of 36,000 sentences, each of which contains entities that should not be modified in the translation process, (ii) an analysis of eight popular nmt models, and (iii) a discussion of the causes of errors made by the models considered. this paper addresses three research questions: rq1: are there categories of entities that popular nmt models cannot transfer without modification? rq2: what are the causes of the problems with transferring entities without modification? rq3: which models offer the highest quality solutions to this problem? 2 related work translation errors in recent years, much attention has been paid to the evaluation of neural machine translation (nmt) models. although the transformer-era nmt models generally perform very well, some of them are prone to small but sometimes critical errors. the problem analyzed in this paper has so far been considered in the context of the sensitivity of evaluation metrics to critical errors, among which unexpected input sequence modification has often been selected as one of the critical error types. aces @xcite -the set of translation accuracy challenges, used to assess the sensitivity of metrics to critical errors, is a data set consisting of approximately 36,500 translated sentences. these sentences are expressed in 146 languages and describe 68 error phenomena, including hallucinations, erroneous unit conversions, adding unnecessary information, producing nonsense words, or translating entities that should not be translated. the error categories in aces follow the multidimensional quality metrics (mqm) ontology, providing a taxonomy of 108 translation problems defined at multiple levels of granularity @xcite . another similar dataset, demetr @xcite , was developed to analyze machine translation metrics. this dataset contains examples of 35 perturbations that include: deleting parts of speech, using hyperonyms, replacing words, misspelling, using wrong capitalization, or repeating parts of text. smaug @xcite , a sentence-level multilingual augmentation project focuses on generating translations that include critical errors and may be used to evaluate the robustness of mt metrics. smaug is a tool for introducing perturbations in existing sentences and is focused on categories such as deviation from named entities, numbers or meaning, or modification of content. however, our goal is to analyze the problem from a different perspective. instead of perturbing existing translations, we observe which models are more likely to generate perturbations. furthermore, we increase the number of examples per category, which helps us to draw statistically significant conclusions. in our work, we provide 1,000 examples per category and language pair, while in other works, e.g., aces, 36,500 examples cover 146 languages and 68 phenomena, which gives an average of about 3.5 examples per language and category combination. moreover, instead of focusing only on translations from a given language to english, as in the demetr dataset, we analyze all possible translation directions between the four languages under consideration. recent neural translation models since the advent of the transformer @xcite architecture, the quality of machine translation models has increased by a large margin @xcite . for this reason, multiple transformerbased mt models have been proposed in recent years. one of the well-known projects in this area is opus @xcite , which provides datasets and mt models trained using marian @xcite and specialized in translation between various pairs of languages. other similar models try to explore the multilingual abilities of transformers, e.g., mbart @xcite supporting translation in 50 languages, no language left behind (nllb) @xcite supporting 200 languages, or madlad @xcite with the support of more than 400 languages. these models are trained using multilingual corpora, which help to exploit relationships between languages. in addition to scaling models in terms of language number, other approaches try to add different modalities as additional sources of knowledge. a popular example is seamlessm4t @xcite , which is able to process textual and audio data. an interesting avenue in machine translation is the use large language models (llms) to translate between languages @xcite . due to the multilingualism of large web corpora and the variety of tasks observed in these corpora that can help to better understand the language, these general models are an interesting alternative to specialized translation models @xcite @xcite . some llms are trained with a substantial amount of parallel corpora, for example, towerllm @xcite , which supports 10 languages and, in addition to machine translation, can perform grammatical correction of texts, identify named entities, or post-edit texts. eurollm @xcite is another attempt to use llms for machine translation. this model supports 35 languages and is trained on various sources of data, including goodquality parallel translation corpora, mathematical equations, code, and general web data. ## dataset the aces, demetr, and smaug have different scopes than this research; thus, even if they consider entities that should not be translated (hereafter referred to as no-translate entities), they use them to evaluate metrics. they also provide relatively small numbers of examples regarding no-translate entities. for example, aces introduces only 100 examples representing the no-translate category, all of them representing english to german translations. demetr focuses on translating from one of 10 languages to english, providing 100 examples per a given perturbation and source language pair. smaug operates on existing sentences and can be used to introduce perturbations (e.g., punctuation removal, random sequence injection, or named entity modification) into provided sentences. for this reason, we decided to create a new dataset, entirely focused on entities that should not be modified during translation. this dataset can be used to measure how well different models transfer no-translate entities without modification. we selected 9 entity types that are simple to identify using regular expressions, these are: e-mail addresses, urls, phone numbers, emojis, social handlers (e.g., instagram or tiktok user identifiers), ip addresses, alphanumeric sequences -artificial identifiers represented as mixtures of letters and numbers, international bank account numbers (ibans), and international standard book numbers (isbns). we did not include dates and numbers as their formats may vary depending on the language (e.g., dd/mm/yyyy vs. mm/dd/yyyy dates or 1,000.00 vs. 1.000,00 as representations of "one thousand" in different languages). in addition to analyzing individual categories, we wanted to investigate the transferability of entities across languages. we selected 4 languages: english, german, polish, and ukrainian -with the goal of selecting popular languages (english, german) and less popular ones that have interesting features, e.g. strongly inflected languages (polish) or non-latin alphabets (ukrainian with cyrillic). these 9 categories and 4 languages were used for the subsequent generation of the dataset. ## https://huggingface.co/google/gemma-2-9b-it to generate sentences, we used vllm @xcite library setting the following sampling parameters: @xmath0.2, @xmath1.95, @xmath2.0, @xmath3, @xmath4, @xmath5, @xmath6.95. the result of this step was a set of 720,000 generated sentences (20,000 sentences × 4 languages × 9 categories). ## explorative data analysis the average sentence in our dataset consists of 18.71 (± 6.81 std. dev.) tokens foot_3 . as can be seen in figure 1 , polish, ukrainian, and german sentences tend to be similar in length -with an average of almost 17 tokens. meanwhile english sentences are longer, with the average of 23.99 tokens. this discrepancy arises from gemma's tendency to include english translations for short inputs and non-english target languages due to the @xmath7. in these cases, our filtering procedure described in section 3.2 is applied, making the average sentence shorter. there is a greater variability in the number of tokens in sentences when analyzing texts by category. these values, presented in appendix d, figure 7 , show that sentences introducing phone numbers are on average the longest with 23.21 ± 6.68 tokens. the shortest sentences are related to emojis, with 13.06 ± 3.94 tokens. considering the lengths of the entities in characters, we see that the average entity consists of 17.91 ± 9.99 characters. contrary to general sentence lengths, per-language entity length averages are very similar to each other, with english entities having 18.13 ± 10.48 characters, german -18.01 ± 9.64, polish -17.93 ± 9.81, and ukrainian -17.58 ± 10.01. the shortest entities are emojis, usually consisting of only 1 character, while the longest are urls, emails, ibans, and phone numbers. the longest entity is an english url address consisting of more than 100 characters. ## methodology to understand the quality of the entity transfer, we applied the following methodology: (i) we chose a set of nmt models to translate between supported languages, (ii) we used each selected model to translate each sentence from our dataset to all languages considered, and (iii) we extracted entities from source and target sentences and compared them in terms of the levenshtein distance. the following subsections describe these steps in detail. ## results we analyzed the results from various perspectives, as described in the following paragraphs. per-category accuracy for each category considered, we examined each model's entity transfer accuracy averaging scores over all language directions with equal weights. in this experiment, a true positive is generated when a regular expression related to the expected category detects exactly the same sequence of characters on the source and target sides. per-direction accuracy similarly to percategory analysis, we also grouped the results by translation direction, macro-averaging the scores over all categories considered. the reason for this analysis is to understand whether all the language pairs considered are handled with similar quality. a deeper analysis reveals that these six models indeed produce most of the no-match errors from emojis: 85.84% of mbart no-match errors are due to emojis, 85.58% for m2m100, 68.99% for nllb, 60.78% for seamlessm4t, 41.16% for opus, and 25.37% for madlad. models that are not dominated by no-match errors (google translate and eurollm, have a relatively low percentage of emojis in the no-match category: 12.5% and 26.73%, respectively. for comparison, figure 8 in appendix d represents the error distribution without the emoji category. in every scenario, small differences (e.g., 1 or 2 characters) are more likely than larger ones (e.g., 3 or 4). the highest chance of one-character errors is observed for google translate. models, such as eurollm, madlad, or google translate have a relatively high number of large differences (levenshtein @xmath8. prompt selection for eurollm we experimented with two types of prompts for inference using eurollm models: generic and focused ones, as described in ## conclusions in this paper, we find that modern medium-sized llms such as eurollm, despite being only partially trained on parallel corpora, may excel in terms of entity transfer quality and can reach the quality similar to google translate. as the best performing model is eurollm 9b, it is the answer to rq3. in contrast, the smallest opus-related models are scored as the worst. this may be due to two factors: on the one hand, bigger models may learn more about the world, but also they may learn better token representations using fine-grained tokenization as discussed in section 6. we showed that increasing the length of the sentence does not correlate with the transfer accuracy, thus, the context information does not necessarily help to guide a model to transfer a given sequence without modifications. considering rq2, the manual analysis of entities transferred by models with high levenshtein distances reveals that entities are frequently partially translated, differing in some numbers, or repeated in fragments. also, for some models (e.g., opus, mbart), the longer entities in terms of subtokens increase the probability of generating an error. this is in line with the intuition that it may be harder to transfer long sequences without errors as compared to the short ones. while most of the categories considered are decently transferred by most of the models considered, emoji is a big struggle -opus, mbart, seamlessm4t, nllb, and m2m100 cannot transfer this category correctly (which answers rq1).