Bilingual Dictionary Based Neural Machine Translation without Using Parallel Sentences (2020)

Abstract: In this paper, we propose a new task of machine translation (MT), which is based on no parallel sentences but can refer to a ground-truth bilingual dictionary. Motivated by the ability of a monolingual speaker learning to translate via looking up the bilingual dictionary, we propose the task to see how much potential an MT system can attain using the bilingual dictionary and large scale monolingual corpora, while is independent on parallel sentences. We propose anchored training (AT) to tackle the task. AT uses the bilingual dictionary to establish anchoring points for closing the gap between source language and target language. Experiments on various language pairs show that our approaches are significantly better than various baselines, including dictionary-based word-by-word translation, dictionary-supervised cross-lingual word embedding transformation, and unsupervised MT. On distant language pairs that are hard for unsupervised MT to perform well, AT performs remarkably better, achieving performances comparable to supervised SMT trained on more than 4M parallel sentences.

## introduction motivated by a monolingual speaker acquiring translation ability by referring to a bilingual dictionary, we propose a novel mt task that no parallel sentences are available, while a ground-truth bilingual dictionary and large-scale monolingual corpora can be utilized. this task departs from unsupervised mt task that no parallel resources, including the ground-truth bilingual dictionary, are allowed to utilize @xcite . this task is also distinct to supervised/semi-supervised mt task that mainly depends on parallel sentences @xcite @xcite @xcite . the bilingual dictionary is often utilized as a seed in bilingual lexicon induction (bli) that aims to induce more word pairs within the language pair @xcite . another utilization of the bilingual dictionary is for translating lowfrequency words in supervised nmt @xcite . we are the first to utilize the bilingual dictionary and the large scale monolingual corpora to see how much potential an mt system can achieve without using parallel sentences. this is different from using artificial bilingual dictionaries generated by unsupervised bli for initializing an unsupervised mt system @xcite , we use the ground-truth bilingual dictionary and apply it throughout the training process. we propose anchored training (at) to tackle this task. since word representations are learned over monolingual corpora without any parallel sentence supervision, the representation distances between source language and target language are often quite large, leading to significant translation difficulty. as one solution, at selects words covered by the bilingual dictionary as anchoring points to drive the distance between the source language space and the target language space closer so that translation between the two languages becomes easier. furthermore, we propose bi-view at that places anchors based on either source language view or target language view, and combines both views to enhance the translation quality. experiments on various language pairs show that at performs significantly better than various baselines, including word-by-word translation through looking up the dictionary, unsupervised mt, and dictionary-supervised cross-lingual word embed-ding transformation to make distances between both languages closer. bi-view at further improves at performance due to mutual strengthening of both views of the monolingual data. when combined with cross-lingual pretraining @xcite , bi-view at achieves performances comparable to traditional smt systems trained on more than 4m parallel sentences. the main contributions of this paper are as follows: ## related work the bilingual dictionaries used in previous works are mainly for bilingual lexicon induction (bli), which independently learns the embedding in each language using monolingual corpora, and then learns a transformation from one embedding space to another by minimizing squared euclidean distances between all word pairs in the dictionary @xcite . later efforts for bli include optimizing the transformation further through new training objectives, constraints, or normalizations @xcite @xcite @xcite @xcite . besides, the bilingual dictionary is also used for supervised nmt which requires largescale parallel sentences @xcite . to our knowledge, we are the first to use the bilingual dictionary for mt without using any parallel sentences. our work is closely related to unsupervised nmt (unmt) @xcite @xcite , which does not use parallel sentences neither. the difference is that unmt may use the artificial dictionary generated by unsupervised bli for initialization @xcite or abandon the artificial dictionary by using joint bpe so that multiple bpe units can be shared by both languages @xcite . we use the ground-truth dictionary instead and apply it throughout a novel training process. unmt works well on close language pairs such as english-french, while performs remarkably bad on distant language pairs in which aligning the embeddings of both side languages is quite challenging. we use the ground-truth dictionary to alleviate such problem, and experiments on distant language pairs show the necessity of using the bilingual dictionary. other utilizations of the bilingual dictionary for tasks beyond mt include cross-lingual dependency parsing @xcite , unsupervised crosslingual part-of-speech tagging and semi-supervised cross-lingual super sense tagging @xcite , multilingual word embedding training @xcite , and transfer learning for low-resource language modeling @xcite . ## our approach there are multiple freely available bilingual dictionaries such as muse dictionary foot_0 @xcite , wiktionary foot_1 , and panlex foot_2 . we adopt muse dictionary which contains 110 large-scale groundtruth bilingual dictionaries. we propose to inject the bilingual dictionary into the mt training by placing anchoring points on the large scale monolingual corpora to drive the semantic spaces of both languages becoming closer so that mt training without parallel sentences becomes easier. we present the proposed anchored training (at) and bi-view at in the following. ## experimentation we conduct experiments on english-french, english-russian, and english-chinese translation to check the potential of our mt system with only bilingual dictionary and large scale monolingual corpora. the english-french task deals with the translation between close-related languages, while the english-russian and english-chinese tasks deal with the translation between distant languages that do not share the same alphabets. ## discussion and future work in the literature of unsupervised mt that only uses non-parallel corpora, unsupervised smt (usmt) and unsupervised nmt (unmt) are complementary to each other. combining them (usmt+unmt) achieves significant improvement over the individual system, and performs comparable to xlm+unmt @xcite . we have set xlm+unmt as a stronger baseline, and our acp+at/bi-view at surpasses it significantly. by referring to the literature of unsupervised mt, we can opt to combine acp+at/bi-view at with smt. we leave it as a future work. ## conclusion in this paper, we explore how much potential an mt system can achieve when only using a bilingual dictionary and large-scale monolingual corpora. this task simulates people acquiring translation ability via looking up the dictionary and depending on no parallel sentence examples. we propose to tackle the task by injecting the bilingual dictionary into mt via anchored training that drives both language spaces closer so that the translation becomes easier. experiments show that, on both close language pairs and distant language pairs, our proposed approach effectively reduces the gap between the source language space and the target language space, leading to significant improvement of translation quality over the mt approaches that do not use the dictionary and the approaches that use the dictionary to supervise the cross-lingual word embedding transformation.