Extract and Attend: Improving Entity Translation in Neural Machine Translation (2023)

Abstract: While Neural Machine Translation (NMT) has achieved great progress in recent years, it still suffers from inaccurate translation of entities (e.g., person/organization name, location), due to the lack of entity training instances. When we humans encounter an unknown entity during translation, we usually first look up in a dictionary and then organize the entity translation together with the translations of other parts to form a smooth target sentence. Inspired by this translation process, we propose an Extract-and-Attend approach to enhance entity translation in NMT, where the translation candidates of source entities are first extracted from a dictionary and then attended to by the NMT model to generate the target sentence. Specifically, the translation candidates are extracted by first detecting the entities in a source sentence and then translating the entities through looking up in a dictionary. Then, the extracted candidates are added as a prefix of the decoder input to be attended to by the decoder when generating the target sentence through self-attention. Experiments conducted on En-Zh and En-Ru demonstrate that the proposed method is effective on improving both the translation accuracy of entities and the overall translation quality, with up to 35% reduction on entity error rate and 0.85 gain on BLEU and 13.8 gain on COMET.

## introduction neural machine translation (nmt) automatically translates sentences between different languages, which has achieved great success @xcite @xcite @xcite . most current works consider to improve the overall translation quality. however, the words in a sentence are not equally important, and the translation accuracy of named entities (e.g., person, organization, location) largely affects user experience, an illustration of which is * this work was completed at microsoft. corresponding authors: rui @xcite . recalling the process of human translation, when encountering an unknown entity in a sen tence, humans look up the translation of the entity in mental or external dictionaries, and organize the translation of the entity together with the translations of other parts to form a smooth target sentence based on grammar and language sense @xcite . as the original intention of neural networks is to mimic the human brain, the human translation process is also an important reference when dealing with entities in nmt. however, none of the previous works on improving the entity translation in nmt consider both steps in human translation: 1) some works annotate the types and positions of the entities without using the dictionary @xcite ; 2) some works first extract the entity translations from a dictionary @xcite or an entity translation model @xcite @xcite , and then directly use them to replace the corresponding entities in the translated sentence via post-processing, which only takes the first step of human translation and may affect the fluency of the target sentence; 3) a couple of works use data augmentation or multi-task training to handle the entities in nmt @xcite , which do not explicitly obtain the translation for each entity as the first step in human translation. inspired by the human translation process, we propose an extract-and-attend approach to improve the translation accuracy of named entities in nmt. specifically, in the "extract" step, translation candidates of named entities are extracted by first detecting each named entity in the source sentence and then translating to target language source 北岛的绘画展在巴黎地平线画廊开幕。 reference bei dao's painting exhibition opens at horizon gallery in paris. output 1 north island's painting exhibition opens at horizon gallery in paris. output 2 bei dao's picture exhibition opens on horizon gallery in paris. ## related work to improve the entity translation in nmt, some works focus on annotating named entities to pro-1 transliteration is to convert between languages while keeping the same pronunciation @xcite . with the help of bilingual dictionaries, one common approach to improve the entity translation in nmt is to first extract the translation of source entities based on a dictionary @xcite or an entity translation model @xcite @xcite , and then locate and replace the corresponding tokens in the target sentence via post-processing. however, such approach only takes the first step of human translation (i.e., extracting the entity translations), since the entity translations are inserted to the target sentence by hard replacement, which affects the fluency of the target sentence. moreover, this approach is sensitive to the inaccurate predictions made by ner @xcite . recently, some works take advantage of additional resources (e.g., dictionary) via data augmentation or multi-task training to improve the translation quality on entities. @xcite augment the parallel corpus based on paired entities extracted from multilingual knowledge graphs, while deep @xcite augments monolingual data with paired entities for a denoising pretraining task. the entity translation can also be enhanced by multi-task training with knowledge reasoning @xcite and integrating lexical constraints @xcite . these methods don't look up translation candidates in bilingual transformer encoder transformer decoder leonardo dicaprio is an american …… per loc entity type source tokens entity dictionary 莱昂纳多 迪卡普里奥 [sep] 美国 <@xmath0: extract-and-attend approach, where the translation candidates are extracted and added as a prefix of the decoder input. entity type embeddings are added to the source input (e.g., 'per' for person names, 'loc' for locations and 'o' for other tokens other than entities). independent position embeddings are used for the translation candidates and the shifted output tokens (i.e., 'ce' for translation candidates and 'e' for output tokens). bilingual dictionaries are also utilized for improving translation quality on rare words or domain-specific terminology. one common approach is to augment training data with pseudo parallel sentences generated based on the dictionary @xcite @xcite . some works adjust the output probabilities over the vocabulary in the decoder according to the dictionary @xcite @xcite . zhong and chiang (2020) attach the definitions of the rare words in the dictionary to enhance the rare word translation. similarly, @xcite and @xcite proposed to inject terminology by replacing or inserting translations inline in the source sentence. though the human translation process when encountering an unknown rare word/terminology or entity is the same, we argue that the two-step human translation process is more suitable for entities. this is because rare words can be polysemous and require context-based disambiguation; on the other hand, each entity is usually linked with a single sense after controlling for entity type. accordingly, retrieved translations of entities are less ambiguous than other words. on the contrary, domain-specific terminology always has a single sense which has little relevant to context, and thus it is usually with much higher accuracy to identify the terminologies in the domain-specific sentences than entities. another uniqueness of entities is that some entities are translated by the same rule, which makes it possible to generalize to unseen entities. for example, when translating the names of chinese people from chinese to english, pinyin foot_0 is commonly used. ## improving entity translation in nmt inspired by the translation process of humans when encountering an unknown entity, where the translation of the entity is extracted from a dictionary and then organized with the translations of other parts to form a fluent target sentence, we propose an extract-and-attend approach. specifically, we first extract the translation candidates of the entities in the source sentence, and then attend the translation candidates into the decoding process via self-attention, which helps the decoder to generate a smooth target sentence based on the specific entity translations. an overview of the proposed extract-and-attend approach is shown in fig. 1 , where a transformer-based @xcite encoder-decoder structure is adopted. specifically, to extract the translation candidates, entities in the source sentence are first detected based on ner @xcite , then the translation candidates are obtained from a bilingual dictionary. considering that some types of named entities (e.g., person nationality classifier nationality-aware transliteration model lu xun [sep] zhou shuren (25 september 1881 -19 october 1936), better known by his pen name lu xun, was a writer, essayist, poet, and literary critic… chinese [chinese] lu xun 鲁迅 named entity + source sentence translation candidate predicted nationality nationality + named entity transliteration pipeline names) are quite diverse and the coverage in the dictionary of such entities is limited, we also develop a transliteration pipeline to handle entities uncovered by the dictionary. to make the decoder attend to the translation candidates, we add the translation candidates in order as a prefix of the decoder input. in the following sections, we will provide the details of "extract" and "attend". ## experimental settings in this section, we describe experimental settings including datasets, model configurations, the evaluation criterion and baselines. ## experimental results in this section, we demonstrate the effectiveness of the proposed extract-and-attend approach by comparing it with multiple baselines. we also conduct experiments to verify the design aspects of "extract" and "attend". ## conclusion in this paper, we propose an extract-and-attend approach to improve the translation quality in nmt systems. specifically, translation candidates for entities in the source sentence are first extracted, and then attended to by the decoder via self-attention. experimental results demonstrate the effectiveness of the proposed approach and design aspects. knowledge is an important resource to enhance the entity translation in nmt, while we only take advantage of the paired entities, nationality and biography information. in the future work, it is interesting to investigate how to make better use of the knowledge, which can be obtained from knowledge graphs and large-scale pre-trained models. besides, the proposed extract-and-attend approach also has some limitations. first, our method requires additional entity resources, which may be difficult to obtain for certain language pairs. with the development of multilingual entity datasets like paranames @xcite , we are optimistic such resources will be more accessible in the near future. second, as demonstrated in section 5.4, extracting translation candidates increases inference time. due to space limitation, more limitations are discussed in appendix a.6.