Point, Disambiguate and Copy: Incorporating Bilingual Dictionaries for Neural Machine Translation (2021)

Abstract: This paper proposes a sophisticated neural architecture to incorporate bilingual dictionaries into Neural Machine Translation (NMT) models. By introducing three novel components: Pointer, Disambiguator, and Copier, our method PDC achieves the following merits inherently compared with previous efforts: (1) Pointer leverages the semantic information from bilingual dictionaries, for the first time, to better locate source words whose translation in dictionaries can potentially be used; (2) Disambiguator synthesizes contextual information from the source view and the target view, both of which contribute to distinguishing the proper translation of a specific source word from multiple candidates in dictionaries; (3) Copier systematically connects Pointer and Disambiguator based on a hierarchical copy mechanism seamlessly integrated with Transformer, thereby building an end-to-end architecture that could avoid error propagation problems in alternative pipe-line methods. The experimental results on Chinese-English and English-Japanese benchmarks demonstrate the PDC’s overall superiority and effectiveness of each component.

## introduction the past several years have witnessed the remarkable success of neural machine translation (nmt), due to the development of sequence-to-sequence methods @xcite @xcite . since bilingual dictionaries cover rich prior knowledge, especially of low-frequency words, many efforts have been dedicated to incorporating bilingual dictionaries into nmt systems. these explorations can be roughly categorized into two broad paradigms. the first one transforms the bilingual dictionaries into pseudo parallel sentence pairs for training (zhang † corresponding authors. ？ figure 1 : three key steps to translate with a bilingual dictionary: pointing, disambiguating and copying. this concrete illustrative example is chosen to conveniently show the primary intuition behind our method. and zong, 2016; @xcite . the second one utilizes the bilingual dictionaries as external resources fed into neural architectures @xcite @xcite @xcite , which is more widely used and the focus of this paper. in practice, bilingual dictionaries usually contain more than one translation for a word. from a highlevel perspective, we believe there are three critical steps to incorporate bilingual dictionaries into nmt models as shown in figure 1: (1) pointing to a source word whose translation in dictionaries will be used at a decoding step, (2) disambiguating multiple translation candidates of the source word from dictionaries, and (3) copying the selected translation into the target side if necessary. note that some works assume that only one translation exists for each word in dictionaries @xcite . in this simplified scenario, the disambiguating step is unnecessary, hence the pointing and copying step can be merged into a single step similar to the classic copying mechanism @xcite . in more practical scenarios, however, this process suffers from the following bottlenecks corresponding to each step. to address the above problems, we propose a novel neural architecture consisting of three novel components: pointer, disambiguator, and copier, to effectively incorporate bilingual dictionaries into nmt models in an end-to-end manner. pointer is a pioneering research effort on exploiting the semantic information from bilingual dictionaries to better locate source words whose translation in dictionaries may be used. disambiguator synthesizes complementary contextual information from the source and target via a bi-view disambiguation mechanism, accurately distinguishing the proper translation of a specific source word from multiple candidates in dictionaries. copier couples pointer and disambiguator based on a hierarchical copy mechanism seamlessly integrated with transformer, thereby building a sophisticated endto-end architecture. last but not least, we design a simple and effective method to integrate byte-pair encoding (bpe) with bilingual dictionaries in our architecture. extensive experiments are performed on chinese-english and english-japanese benchmarks, and the results verify the pdc's overall performance and effectiveness of each component. ## background: transformer transformer @xcite is the most popular nmt architecture, which adopts the standard encoder-decoder framework and relies solely on stacked attention mechanisms. specifically, given a source sequence @xmath3, x 2 ..., x n }, the model is supposed to generate the target sequence @xmath4, y 2 ..., y m } in an auto-regressive paradigm. transformer encoder. a transformer encoder is constituted by a stack of n identical layers, each of which contains two sub-layers. the first is a multihead self-attention mechanism (selfatt), and the second is a fully connected feed-forward network (ffn). layer normalization (ln) @xcite and residual connection @xcite is em-source embedding dictionary embedding target embedding 𝑥 # ldc 𝑥 @xmath0 𝑦 % 𝑦 & self-att ffn n× dec-enc-att ffn self-att n× ℎ # ℎ @xmath1 ℎ′ % ℎ′ & 𝑠′ * 𝑑′ @xmath2) ployed around the two sub-layers in both encoder and decoder. then, the translation probability p(y t |@xmath5, x) of the t-th target word is produced with a softmax layer: ## methodology in this section, we mathematically describe our model in detail. we follow the notations in section 2. ## experimental settings in this section, we elaborate on the experiment setup to evaluate our proposed model. ## conclusion we have presented pdc, a new method to incorporate bilingual dictionaries into nmt models, mainly involving four techniques. (1) by integrating semantic information of dictionaries, the enhanced context representations help to locate source words whose dictionary translations will potentially be used. (2) the source and target information is well synthesized and contribute to identifying the optimal translation of a source word among multiple dictionary candidates, in a complementary way. (3) the above two steps are then systematically integrated based on a hierarchical copy mechanism. (4) we finally equip the architecture with a novel selective bpe strategy carefullydesigned for dictionary-enhanced nmt. experiments show that we achieve competitive results on the chinese-english and english-japanese translation tasks, verifying that our approach favorably incorporates prior knowledge of bilingual dictionaries.