Addressing Entity Translation Problem via Translation Difficulty and Context Diversity (2024)

Abstract: Neural machine translation (NMT) systems often produce inadequate translations for named entities. In this study, we conducted preliminary experiments to examine the factors affecting the translation accuracy of named entities, specifically focusing on their translation difficulty and context diversity. Based on our observations, we propose a novel data augmentation strategy to enhance the accuracy of named entity translation. The main concept behind our approach is to increase both the context diversity and translation probability for the targeted named entity pair. To achieve this, we construct additional samples for named entities that exhibit high translation difficulty or low context diversity and use the augmented training data to re-train the final translation model. Furthermore, we propose an entity-aware machine translation metric that prefers the translation output to generate more accurate named entities. Our experimental results demonstrate significant improvements over the baseline in terms of general translation performance and named entity translation accuracy across various test sets, such as WMT news translation and terminology test sets.

## introduction neural machine translation (nmt) that leverages the sequence-to-sequence learning paradigm to transform a source sentence into a target sentence has made significant advancements in recent years. however, previous studies have demonstrated that the translation of low-frequency named entities (e.g., person name, location, organization), which plays a central role in improving the customer experience for commercial translation systems, continues to pose a significant challenge for nmt systems @xcite . numerous techniques have been proposed to improve the accuracy of named entity translation. these techniques include 1) substituting entity translation in a post-processing step @xcite , 2) utilizing neural translation memory @xcite @xcite , 3) employing named entity tag embedding @xcite , 4) applying lexically constrained decoding @xcite @xcite , and 5) utilizing an interactive translation mechanism @xcite . most of these techniques explicitly decouple the named entity translation from the process of translating the entire sentence. along this line, we decompose the sentence into the named entity and its context. we performed preliminary experiments on large-scale in-house data, which allowed us to identify two factors that impact the translation accuracy of named entities: based on the above observations, we propose a novel data augmentation strategy to improve the translation accuracy of named entities. specifically, given bilingual entity pairs, we first compute the translation difficulty and context diversity scores based on the training corpus. next, we select the bilingual context from the bilingual sentence pair and combine it with the bilingual entity pair to create synthetic parallel samples. the basic idea is to generate additional synthetic samples for the entity pair that exhibits high translation difficulty or low context diversity. consequently, these synthetic parallel samples are used to boost the context diversity and the translation probability associated with the named entity pair. finally, we use the augmented training data to re-train the final translation model. we also propose a lexical-based entity-aware machine translation metric to evaluate the quality of translations containing named entities more effectively. the proposed metric assigns a high score for the translation outputs with greater accuracy in named entity translation. we adopt the widely-used bleu @xcite and chrf++ @xcite to our entity-aware machine translation metric and find that the proposed entity-aware entity metric correlates better with human judgment than the original metrics. to evaluate the effectiveness of our data strategy, we conducted extensive machine translation experiments on wmt20 @xcite news translation and wmt21 @xcite terminology translation with general decoding (without lexical constraint). experimental results demonstrate that the proposed approach outperforms several strong baselines in terms of bleu and named entity translation accuracy. furthermore, our strategy can be applied to lexically constrained decoding scenarios. we implement the state-of-the-art template-based approach @xcite ) on top of our strategy and find that our strategy can improve the context translation performance while maintaining the entity translation accuracy. our main contributions are: ## preliminary experiment to understand the main characteristics of entity translation in nmt, we performed preliminary experiments on large-scale in-house data foot_0 to empirically demonstrate that the accuracy of entity translation is strongly correlated to two of its attributes, namely, translation difficulty and contextual diversity. notations. let @xmath0, y i )} n @xmath1, where x i and y i are source and target sentences, and n denotes the total number of sentence pairs. let @xmath2, v j )} m @xmath3, where m is the number of entity pairs. furthermore, m @xmath4. translation difficulty. translation difficulty refers to how difficult the level of complexity is associated with translating an entity pair using a translation model without any contextual information. we employ normalized sentence-level probability as the metric for calculating the translation difficulty score to measure this difficulty score. a high translation difficulty score indicates that it is hard to generate the entity translation correctly for the translation model m @xmath5. context diversity. context diversity refers to the entropy of diverse contexts associated with the entity pair in the training corpus. in accordance with @xcite , we employ the surrounding words in the source sentences to represent the context and calculate the context diversity score for the entity pair based on the entropy of the context. where p(c h |u, v) is the estimated probability of the context c h of the entity pair (u, v), and is computed as follows, ## methodology in this section, we present a novel data augmentation strategy to improve the accuracy of named entity translation. the translation model training processing of our strategy includes following steps: ## entity-aware machine translation evaluation we propose a simple and effective entiy-aware evaluation metric for assessing machine translation quality. the main idea involves considering the translation accuracy of the entities during the calculation of the evaluation score, using both the translation output and the reference translation. in particular, when determining the value, if an entity matches the translation output, its weight will be correspondingly increased. we apply the entiy-aware evaluation strategy on the sacrebleu @xcite and chff+ @xcite . we assign an entity-aware weight of 4, which entails calculating the values of the n-grams (words or characters) by proportionally increasing the weights of the values present in the matching entities. we conducted experiments wmt19 metric tasks 4 on four high-resource pairs (from chinese/zh, german/de, russian/ru, finnish/fi to english/en) and three low-resource pairs (from gujarati/gu, kazakh/kk, lithuanian/lt to english/en). to make a fair comparison, we adopt the tok-4 https://www.statmt.org/wmt19/metrics-task.html enizer_13a foot_1 to tokenize the text and apply the stanford ner system foot_2 to tokenized text. for the experiments in ## conclusion we demonstrate that the translation accuracy of named entities is influenced by both their level of translation difficulty and the diversity of contextual information. we propose a novel data augmentation strategy that boosts the translation probability and the context diversity of the targeted named entity pair. experimental results show that the proposed strategy outperforms the baseline in terms of general translation performance and entity translation accuracy. we also propose an entity-aware machine translation metric that prefers the translation output to generate more accurate named entities.