DEEP : DE noising Entity Pre-training for Neural Machine Translation (2022)

Abstract: It has been shown that machine translation models usually generate poor translations for named entities that are infrequent in the training corpus. Earlier named entity translation methods mainly focus on phonetic transliteration, which ignores the sentence context for translation and is limited in domain and language coverage. To address this limitation, we propose DEEP, a DEnoising Entity Pre-training method that leverages large amounts of monolingual data and a knowledge base to improve named entity translation accuracy within sentences. Besides, we investigate a multi-task learning strategy that finetunes a pre-trained neural machine translation model on both entity-augmented monolingual data and parallel data to further improve entity translation. Experimental results on three language pairs demonstrate that DEEP results in significant improvements over strong denoising auto-encoding baselines, with a gain of up to 1.3 BLEU and up to 9.2 entity accuracy points for English-Russian translation.

## introduction proper translation of named entities is critically important for accurately conveying the content of text in a number of domains, such as news or encyclopedic text @xcite . in addition, a growing number of new named entities (e.g., person name, location) appear every day, therefore many of these entities may not exist in the parallel data traditionally used to train mt systems. as a result, even state-of-theart mt systems struggle with entity translation. for example, @xcite note that a chinese-english news translation system that had allegedly reached human parity still lagged far behind human translators on entity translations, and this prob-1 code/data/models are released at https://github. com/junjiehu/deep . because of this, there have been a number of methods proposed specifically to address the problem of translating entities. as noted by @xcite , earlier studies on named entity translation largely focused on rule-based methods @xcite , statistical alignment methods @xcite and web mining methods @xcite @xcite . however, these methods have two main issues. first, as they generally translate a single named entity without any context in a sentence, it makes it difficult to resolve ambiguity in entities using context. in addition, the translation of entities is often performed in a two-step process of entity recognition then translation, which complicates the translation pipeline and can result in cascading errors @xcite @xcite . in this paper, we focus on a simple yet effective method that improves named entity translation within context. specifically, we do so by devising a data augmentation method that leverages two data sources: monolingual data from the target language and entity information from a knowledge base (kb). our method also adopts a procedure of pre-training and finetuning neural machine translation (nmt) models that is used by many recent works @xcite @xcite . in particular, pre-training methods that use monolingual data to improve translation for low-resource and medium-resource languages mainly rely on a denoising auto-encoding objective that attempt to reconstruct parts of text @xcite or the whole sentences @xcite ) from noised input sentences without particularly distinguishing named entities and other functional words in the sentences. in contrast, our method exploits an entity linker to identify entity spans in the monolingual sentences and link them to a kb that contains mul- krasnodar (q3646) language label description english krasnodar capital of krasnodar region (krai) in southern russia russian краснодар город на юге россии, административный центр краснодарского края : : language label ... english saratov ... russian саратов ... : : saratov (q5332) language label ... english ulyanovsk ... russian ульяновск ... : : ulyanovsk (q5627) магазины нового формата заработали в краснодарe , саратовe и ульяновскe . ## denoising auto-encoding (dae) given a set of monolingual text segments for pretraining, i.e., ∈ d , a sequence-to-sequence denoising auto-encoder is pre-trained to reconstruct a text segment from its noised version corrupted by a noise function (•). formally, the dae objective is defined as follows: in particular, as our baseline we focus on the mbart method @xcite , a popular method with two types of heuristic noise functions being used sequentially on each text segment. the first noise function randomly masks spans of text in each sentence. specifically, a span length is first randomly sampled from a poisson distribution ( = 0.35) and the beginning location for a span in is also randomly sampled. the selected span of text is replaced by a mask token. this process repeats until 35% of words in the sentence are masked. the second noise function is to permute the sentence order in each text segment with a probability. ## deep: denoising entity pre-training our method adopts a procedure of pre-training and finetuning for neural machine translation. first, we apply an entity linker to identify entities in a monolingual corpus and link them to a knowledge base ( §3.1). we then utilize entity translations in the knowledge base to create noisy code-switched data for pre-training ( §3.2). finally, we examine a multi-task learning strategy to further improve the translation of low-frequency entities ( §3.3). ## experimental setting pre-training data: we conduct our experiments on three language pairs: english-russian, english-ukrainian and english-nepali. we use wikipedia articles as the monolingual data for pre-training and report the data statistics in finetuning & test data: we use the news commentary data from the english-russian translation task in wmt18 @xcite for finetuning and evaluate the performance on the wmt18 test data from the news domain. for english-ukrainian, we use the ted talk transcripts from july 2020 in the opus repository @xcite for finetuning and testing. for english-nepali translation, architecture: we use a standard sequence-tosequence transformer model @xcite with 12 layers each for the encoder and decoder. we use a hidden unit size of 512 and 12 attention heads. following @xcite , we add an additional layer-normalization layer on top of both the encoder and decoder to stabilize training at fp16 precision. we use the same sentencepiece model and the vocabulary from @xcite . ## conclusion in this paper, we propose an entity-based pretraining method for neural machine translation. our method improves named entity translation accuracy as well as bleu score over strong denoising auto-encoding baselines in both single-task and multi-task setting. despite the effectiveness, several challenging questions remain open. first, recent works on integrating knowledge graphs @xcite in nmt have shown promising results for translation. our method links entities to a multilingual knowledge base which contains rich information of the entities such as entity description, relation links, and alias. how to leverage these richer data sources to resolve entity ambiguity deserves further investigation. second, finetuning pretrained models on in-domain text data is a potential way to improve entity translations across domains. ukrainian: as seen in figure 6 , the general trend for the entity translation accuracy according to entity groups are similar to that of russian. while deep achieves the highest accuracy in ft, the results for ft is less reliable due to a small sample size of entities in ft. in terms of the gain from @xmath0, we observe a consistent improvement of our multi-task deep on translating low-frequent entities in the finetuning data (see the left bottom of figure 8 ). nepali: while outperforming at the beginning of finetuning, figure 7 shows that @xmath1. moreover, the accuracy is considerably lower on entities in pt, which suggests that the degree of forgetting is much more conspicuous in nepali. the gain from @xmath2. figure 9 shows the results. in the single-task setting, deep improve the translations of frequent entities appearing in both the pre-training and finetuning data. despite the multi-task learning that introduces additional exposure to entities that are more frequent in the pre-training data, the largest gain comes from entities that are less frequent in the pre-training data but frequent in the finetuning data.