“Wikily” Supervised Neural Translation Tailored to Cross-Lingual Tasks (2021)

Abstract: We present a simple but effective approach for leveraging Wikipedia for neural machine translation as well as cross-lingual tasks of image captioning and dependency parsing without using any direct supervision from external parallel data or supervised models in the target language. We show that first sentences and titles of linked Wikipedia pages, as well as cross-lingual image captions, are strong signals for a seed parallel data to extract bilingual dictionaries and cross-lingual word embeddings for mining parallel text from Wikipedia. Our final model achieves high BLEU scores that are close to or sometimes higher than strong supervised baselines in low-resource languages; e.g. supervised BLEU of 4.0 versus 12.1 from our model in English-to-Kazakh. Moreover, we tailor our wikily translation models to unsupervised image captioning, and cross-lingual dependency parser transfer. In image captioning, we train a multi-tasking machine translation and image captioning pipeline for Arabic and English from which the Arabic training data is a wikily translation of the English captioning data. Our captioning results on Arabic are slightly better than that of its supervised model. In dependency parsing, we translate a large amount of monolingual text, and use it as an artificial training data in an annotation projection framework. We show that our model outperforms recent work on cross-lingual transfer of dependency parsers.

## introduction developing machine translation models without using bilingual parallel text is an intriguing research problem with real applications: obtaining a large volume of parallel text for many languages is hard if not impossible. moreover, translation models could be used in downstream cross-lingual tasks in which annotated data does not exist for some languages. there has recently been a great deal of interest in unsupervised neural machine translation (e.g. @xcite ; @xcite ; @xcite ; @xcite ; @xcite ; @xcite ). unsupervised neural machine translation models often perform nearly as well as supervised models when translating between similar languages, but they fail to perform well in low-resource or distant languages @xcite or out-of-domain monolingual data @xcite . in practice, the highest need for unsupervised models is to expand beyond high resource, similar european language pairs. there are two key goals in this paper: our first goal is developing accurate translation models for low-resource distant languages without any supervision from a supervised model or gold-standard parallel data. our second goal is to show that our machine translation models can be directly tailored to downstream natural language processing tasks. in this paper, we showcase our claim in cross-lingual image captioning and cross-lingual transfer of dependency parsers, but this idea is applicable to a wide variety of tasks. we present a fast and accurate approach for learning translation models using wikipedia. unlike unsupervised machine translation that solely relies on raw monolingual data, we believe that we should not neglect the availability of incidental supervisions from online resources such as wikipedia. wikipedia contains articles in nearly 300 languages and more languages might be added in the future, including indigenous languages and dialects of different regions in the world. different from similar recent work @xcite , we do not rely on any supervision from supervised translation models. instead, we leverage the fact that many first sentences in linked wikipedia pages are rough glish in which the titles, first sentences, and also the image captions are rough translations of each other. our method learns a seed bilingual dictionary from a small collection of first sentence pairs, titles and captions, and then learns cross-lingual word embeddings. we make use of cross-lingual word embeddings to extract parallel sentences from wikipedia. our experiments show that our approach improves over strong unsupervised translation models for low-resource languages: we improve the bleu score of english!gujarati from 0.6 to 15.2, and english!kazakh from 0.8 to 12.1. in the realm of downstream tasks, we show that we can easily use our translation models to generate high-quality translations of ms-coco @xcite and flickr @xcite datasets, and train a cross-lingual image captioning model in a multi-task pipeline paired with machine translation in which the model is initialized by the parameters from our translation model. our results on arabic captioning show a bleu score of 5.72 that is slightly better than a supervised captioning model with a bleu score of 5.22. as another task, in dependency parsing, we first translate a large amount of monolingual data using our translation models and then apply transfer using the annotation projection method @xcite . our results show that our approach performs similarly compared to using gold-standard parallel text in high-resource scenarios, and significantly better in low-resource languages. a summary of our contribution is as follows: our method learns a seed bilingual dictionary from a small collection of first sentence pairs, titles and captions, and then learns cross-lingual word embeddings. we make use of cross-lingual word embeddings to extract parallel sentences from wikipedia. our experiments show that our approach improves over strong unsupervised translation models for low-resource languages: we improve the bleu score of @xmath0.6 to 15.2, and @xmath1.8 to 12.1. in the realm of downstream tasks, we show that we can easily use our translation models to generate high-quality translations of ms-coco @xcite and flickr @xcite datasets, and train a cross-lingual image captioning model in a multi-task pipeline paired with machine translation in which the model is initialized by the parameters from our translation model. our results on arabic captioning show a bleu score of 5.72 that is slightly better than a supervised captioning model with a bleu score of 5.22. as another task, in dependency parsing, we first translate a large amount of monolingual data using our translation models and then apply transfer using the annotation projection method @xcite . our results show that our approach performs similarly compared to using gold-standard parallel text in high-resource scenarios, and significantly better in low-resource languages. a summary of our contribution is as follows: 1) we propose a simple, fast and effective approach towards using the wikipedia monolingual data for machine translation without any explicit supervision. our mining algorithm easily scales on large comparable data using limited computational resources. we achieve very high bleu scores for distant languages, especially those in which current unsupervised methods perform very poorly. 2) we propose novel methods for leveraging our current translation models in image captioning. we show that how a combination of translating caption training data, and multi-task learning with english captioning as well as translation improves the performance. our results on arabic shows results slightly superior to that of a supervised captioning model trained on gold-standard datasets. 3) we propose a novel modification to the annotation projection method to be able to leverage our translation models. our results on dependency parsing performs better than previous work in most cases, and performs similarly to using gold-standard parallel datasets. our translation and captioning code and models are publicly available online foot_0 . unsupervised neural machine translation unsupervised neural machine translation does not have access to any parallel data. instead, it tailors monolingual datasets m l 1 and m l 2 for learning multilingual language models. these language models usually mask parts of every input sentence, and try to uncover the masked words @xcite . the monolingual language models are used along with iterative back-translation @xcite to learn unsupervised translation. an input sentence s is translated to t using current model θ, then the model assumes that (t , s) is a goldstandard translation, and uses the same training objective as of supervised translation. dependency parsing dependency parsing algorithms capture the best scoring dependency trees for sentences among an exponential number of possible dependency trees. a valid dependency tree for a sentence @xmath2, . . . , s n assigns heads h i for each for word s i where 1 ≤ @xmath3, 0 ≤ h @xmath4. the zeroth word represents a dummy root token as an indicator for the root of the sentence. for more details about efficient parsing algorithms, we encourage the reader to see @xcite . annotation projection annotation projection is an effective method for transferring supervised annotation from a rich-resource language to a low-resource language through translated text @xcite . having a parallel data @xmath5, t i )} n @xmath6, and supervised source annotations for source sentences s i , we transfer those annotations through word translation links ## learning translation from wikipedia the key component of our approach is to leverage the multilingual cues from linked wikipedia pages across languages. wikipedia is a great comparable data in which many of its pages explain entities in the world in different languages. in most cases, first sentences define or introduce the mentioned entity in that page (e.g. figure 1 ). therefore, we observe that many first sentence pairs in linked wikipedia documents are rough translations of each other. moreover, captions of images in different languages are usually similar but not necessarily direct translations of each other. we leverage this information to extract many parallel sentences from wikipedia without using any external supervision. in this section, we describe our algorithm which is briefly shown in figure 3 . ## cross-lingual tasks in this section, we describe our approaches for tailoring our translation models to cross-lingual tasks. note that henceforth we assume that our translations model training is finished, and we have access to trained translation models for cross-lingual tasks. ## experiments in this section, we provide details about our experimental settings and results for translation, captioning, and dependency parsing. we put more details about our settings as well as thorough analysis of our results in the supplementary material. ## conclusion we have described a fast and effective algorithm for learning translation systems using wikipedia. we show that by wisely choosing what to use as seed data, we can have very good seed parallel data to mine more parallel text from wikipedia. we have also shown that our translation models can be used in downstream cross-lingual natural language processing tasks. in the future, we plan to extend our approach beyond wikipedia to other comparable datasets like the bbc world service. a clear extension of this work is to try our approach on other cross-lingual tasks. moreover, as many captions of the same images in wikipedia are similar sentences and sometimes translations, multimodal machine translation @xcite @xcite based on this data or the analysis of the data, such as whether more similar languages may share more similar captions @xcite are other interesting avenues. cross-lingual cca tool @xcite to extract 150-dimensional vectors. this tool can be run on a single cpu within a few hours.