Unsupervised Neural Machine Translation with Universal Grammar (2021)

Abstract: Machine translation usually relies on parallel corpora to provide parallel signals for training. The advent of unsupervised machine translation has brought machine translation away from this reliance, though performance still lags behind traditional supervised machine translation. In unsupervised machine translation, the model seeks symmetric language similarities as a source of weak parallel signal to achieve translation. Chomsky’s Universal Grammar theory postulates that grammar is an innate form of knowledge to humans and is governed by universal principles and constraints. Therefore, in this paper, we seek to leverage such shared grammar clues to provide more explicit language parallel signals to enhance the training of unsupervised machine translation models. Through experiments on multiple typical language pairs, we demonstrate the effectiveness of our proposed approaches.

## introduction recently, neural machine translation (nmt) @xcite has been greatly developed and become the dominant paradigm in machine translation. on the one hand, the development of deep neural networks such as transformer @xcite has played a significant role in nmt's improvements. on the other hand, large-scale parallel corpora like the un corpus @xcite have also played an important role. despite the recent success of nmt in standard benchmarks, the need for large-scale parallel corpora has limited the effectiveness of nmt in many language pairs, especially in low-resource language pairs @xcite . unsupervised neural machine translation (unmt) figure 1 : examples of constituent trees from english penn treebank (ptb) and german dataset of spmrl14 shared task. the dotted box indicates the constituents that can be masked for prediction. @xcite was proposed to alleviate this issue by completely removing the need for parallel data and training an nmt system in a completely unsupervised manner, relying on nothing but monolingual corpora. unsupervised machine translation does not need the parallel information from parallel sentences; rather, it generally uses embedding alignments, initializes parameters with pretrained language models, and uses iterative backtranslation between two languages to synthesize pseudo parallel corpora for model training @xcite @xcite @xcite . the pseudo parallel data created by iterative back-translation is the key to the success of unsupervised nmt model training @xcite . it takes advantage of the equivalence of translation languages to bring supervision (albeit weak supervision) to model training. recent results in semi-supervised nmt have demonstrated that further training a unmt model with true bilingual parallel sentences can lead to better translation performance @xcite @xcite , which suggests that after training, unmt models are still not optimized because of their lack of explicit supervision. universal grammar (ug) is a notion in linguistics and philosophy that goes back at least to roger bacon's observation, "in its substance, grammar is one and the same in all languages, even if it accidentally varies" @xcite . @xcite developed a universal grammar theory. the idea of a universal grammar states that all human languages are species of a common genus because they have all been shaped by a factor that is common to all human beings @xcite . therefore, in this paper, we leverage this grammar commonality to derive additional supervision to enhance unmt training. in other words, our proposed method is built on the existence of universal grammar. if there is no crosslingual commonality and definitional similarity in the syntactic structure, then we will not be able to obtain weakly supervised signals for unmt. specifically, we choose the grammar representation framework of constituent syntax as the research object. unlike typical approaches to leveraging syntax information, rather than adopting a syntactic encoder to enhance representations, we focus on acquiring more supervision by finding commonalities between two languages' syntaxes and demonstrate this supervision by training unmt models. since different languages often share some of the same constituent types (syntax categories), predicting these matching constituents in model training can be used for a weak alignment. as shown in figure 1 , although the two sentences are not parallel, during the training, the model is exposed to both np and vp constituents, and a weak alignment between these constituents can be used to enhance the unmt training, i.e., the np constituents in english and the np constituents in german (the same to vp, pp, ..., etc.) are more likely to be parallel. notably, our method is only an application of universal grammar in unmt, but far from all applications since we only leverage a very small part of universal grammar (universal constituent and syntactic label definition). masked language modeling (mlm) is a com-monly used training approach for language modeling. in mlm, some of the tokens in the sentence are masked, and then the model is required to predict these masked tokens at their placeholders. based on mlm, we propose a constmlm approach that also draws from constituent syntax. in our constmlm, constituents are masked, and the model is tasked with predicting both the tokens in a constituent and the constituent's syntactic category. masking large constituents will present too difficult a problem for the model, as there will be insufficient context, so we also propose btlm, a method of leveraging back-translation to provide more context and alleviate this issue. we then implement constbtlm based on the constmlm, which leverage our proposed btlm. to accommodate both unmt and language modeling training, we have prepared both encoder-decoder models and encoder-only models for our constbtlm, btlm, and constmlm approaches. in our experiments, we demonstrated the effectiveness of leveraging universal grammar and of our proposed approaches on multiple unsupervised translation tasks. our proposed approaches show consistent improvements compared to the baselines in these tasks. we also present a significantly boosted performance on several low-resource semisupervised tasks. these results verify that universal grammar commonalities can bring additional supervision information to bolster the training of unsupervised and low-resource translation models. ## related work unmt has been greatly developed in recent years @xcite @xcite @xcite . syntax has been used extensively explored in supervised mt research field @xcite method en-fr fr-en xlm 33.3 31.2 spanbert 33.5 31.7 limit-bert 33.4 31.4 constmlm enc-only 33.6 31.4 constmlm enc-dec 33.9 32.0 constbtlm enc-dec 35.1 33.4 ## conclusion and future work in this paper, we mine weak alignment information from universal grammar annotations and use it to improve unsupervised machine translation. two specific training approaches, constmlm and constbtlm, are proposed to apply this weak supervision. via empirical exploration on unsupervised and semi-supervised machine translation benchmarks, we verify that universal grammar will boost cross-lingual alignment for unmt. our analysis shows that using universal grammar, the reliance on parallel corpora can be reduced under the premise of achieving the same effect because the weak supervision signal based on universal grammar can play a similar role to the supervision signal of the parallel corpus. in this work, we rely on the dependency syntax of 100+ languages provided by the universal dependency project for synthesizing pseudo-constituent syntax in some languages. in the future, we intend to train a multilingual parser based on the multilingual language model -xlm-r (with the training data as a combination of 10+ language constituent syntax), which has the ability to parse 100+ languages in a single model, further increasing the practicality of our method. in addition, we will examine more low-resource languages to verify the method's universality.