Synthesizing Parallel Data of User-Generated Texts with Zero-Shot Neural Machine Translation (2020)

Abstract: Neural machine translation (NMT) systems are usually trained on clean parallel data. They can perform very well for translating clean in-domain texts. However, as demonstrated by previous work, the translation quality significantly worsens when translating noisy texts, such as user-generated texts (UGT) from online social media. Given the lack of parallel data of UGT that can be used to train or adapt NMT systems, we synthesize parallel data of UGT, exploiting monolingual data of UGT through crosslingual language model pre-training and zero-shot NMT systems. This paper presents two different but complementary approaches: One alters given clean parallel data into UGT-like parallel data whereas the other generates translations from monolingual data of UGT. On the MTNT translation tasks, we show that our synthesized parallel data can lead to better NMT systems for UGT while making them more robust in translating texts from various domains and styles.

## introduction neural machine translation (nmt) requires large parallel data for training. however, even when trained on large clean parallel data, nmt generates translations of very poor quality when translating out-of-domain or noisy texts. for instance, @xcite empirically showed that nmt systems trained on clean parallel data from the news and parliamentary debate domains perform reasonably well when translating news articles but poorly perform at translating user-generated texts (ugt) from a social media. ugt can be from various domains and manifest various forms of natural noise. for instance, they can exhibit spelling/typographical errors, words omission/insertion/repetition, grammatical/syntactic errors, or noise markers even more specific to the writing style of social media such as abbreviations, obfuscated profanities, inconsistent capitalization, internet slang, and emojis. normalizing and correcting them in a preprocessing step is a solution to facilitate translation @xcite , but it impedes the correct transfer of the style of the source text to its translation. in this paper, we posit that the nmt system should preserve the style during the translation. another trend of work focuses on making nmt more robust in handling noisy tokens, such as tokens with spelling mistakes, which can greatly disturb nmt @xcite . however, it has only a minimal impact in translating ugt @xcite that contains other types of noise/errors. whereas domain adaptation methods are helpful in improving nmt for ugt @xcite , we do not usually have bilingual parallel data of ugt created by professional translators to train or adapt an nmt system. consequently, previous work on nmt for ugt merely focused on scenarios for which we have ugt parallel data, such as the mtnt dataset @xcite . in contrast to previous work, we assume that parallel data of ugt are not available and that we can only rely on the formal and clean texts that are usually used to train nmt systems. in addition, we exploit ugt monolingual data that are publicly available in large quantity on the internet for many languages. we propose to synthesize parallel data of ugt to train better nmt systems for ugt. for this purpose, we present two complementary approaches that associate a pre-trained crosslingual language model with zero-shot nmt systems. our contributions are as follows: the remainder of this paper is organized as follows. in section 2, we present the research problem and questions that we answer in this work. then, in section 3, we present a zero-shot nmt framework that we use to synthesize parallel data of ugt by our two methods presented in section 4. we evaluate the usefulness of our approaches for better translating ugt in section 5. in sections 6 and 7, we evaluate alternative configurations for our zero-shot nmt systems, and in section 8 we verify whether our nmt systems trained on the synthetic parallel data are more robust to changes of domain and style. we analyze the synthetic sentences and present examples in section 9 to better understand why our data lead to better nmt systems. following the presentation of related work in section 10, we conclude the paper in section 11. ## motivation ugt contains many different types of noise that can also differ from one type of ugt to another. for instance, posts on twitter contain many spelling errors intentionally introduced for text compression, whereas this kind of error is rather marginal in the discussions from reddit @xcite . figure 1 shows the impact on mt of two different types of noise: spelling (ex1) and syntactic (ex2) errors, compared to the translation of the same but clean sentence (ex3). ex1 has an intentional spelling error ''vlà'' (instead of ''voilà'') and a ugt-specific symbol, ''#.'' comparison with ex3 suggests that they have negative effects on the vanilla nmt system and eventually lead to an incorrect translation largely different from the translation of the clean source of ex3. in ex2, a syntactic error ''arrive est'' instead of ''arrive'' has also an impact, but to a lesser extent, by inducing the past tense in english. vanilla nmt gives the best translation for the clean source sentence (ex3) only failing in translating ''covid19.'' for indicative purpose, we present in the row ''our work'' translations generated by our work. these examples highlight the inability of vanilla nmt in translating sentences with various types of noise. in conducting the research to better translate ugt, we answer the following research questions: q1 how can we generate synthetic parallel data for ugt in a specific domain/style without relying on any manually produced parallel data of ugt? q2 do the synthetic parallel data lead to a better nmt system for the targeted ugt and do they make it more robust to the change of domain or style? we describe in this section our zero-shot nmt system used to synthesize parallel data of ugt. ## experiments in this section, we empirically evaluate the usefulness of the parallel data synthesized by our proposed approaches in training better nmt systems for translating ugt. ## impact of the distinction between l1/l2 and r1/r2 monolingual data we empirically verified our assumption that m l1 and m r1 , m l2 and m r2 , must be distinguished in order to enforce our nmt systems to learn the difference between clean texts and ugt, while it also learns to translate between l1 and l2, and between r1 and r2. to this end, we set up two new configurations, #a and #b, where we have p l1-l2 parallel data and only m l1 and m l2 monolingual data, that is, we do not define m r1 and m r2 monolingual data to train xlm and nmt systems used to synthesize parallel data. with #a and #b, we no longer have zeroshot translation directions for synthesizing p s r1-r2 . instead, we have an nmt system initialized using a pre-trained crosslingual language model also exploiting reddit monolingual data. 15 with these configurations, we assume that the presence of a significant amount of reddit data in the monolingual data may bias the nmt system in synthesizing reddit-like texts. the results of nmt systems trained on parallel data synthesized by #a and #b are presented in ## ablation study on zero-shot nmt's objective we performed an ablation study of the objectives exploited for training the zero-shot nmt presented in section 3.2. we compared the following four combinations of objectives: ae+bt+mt: the original combination used to train our zero-shot nmt system. bt+mt: the ae objective is removed. this excludes any random noise in the source sentences. the system is no longer restricted to perform a simple copy of the source when performing round-trip bt. ae+bt: typical combination of objectives used for unsupervised nmt @xcite . without the supervised mt objective, we expect a drop of the translation quality. bt: without ae and mt objectives, we can expect the system to be able to properly model neither languages nor translations. note that we cannot remove the bt objectives as this is the only objective that trained the system to translate, for instance, from l1 to r2 and from r2 to r1. we evaluated the zero-shot nmt itself and nmt systems exploiting the synthetic parallel data generated by the zero-shot nmt system using our approaches #1 without filtering 16 and #2. the results are presented in ## impact on the robustness of nmt using extra test suites, we evaluated to what extent our nmt systems trained on synthetic parallel data of ugt are robust to domain/style changes or only adapted to better translate reddit data. 16 we did not filter the data unlike our original proposal, because our goal is only to evaluate the quality of the data given the different systems used to generate them while saving the computational cost of finding a good threshold for filtering. losses @xmath0.4 22.4 bt+mt 20.4 * 22.3 ae+bt 19.2 * 21.2 * bt 0.1 * 0.4 * #1: only with p s r1-r2 synthesized from p l1-l2 ae+bt+mt 19.0 18.0 bt+mt 17.7 * 17.0 * ae+bt 6.6 * 0.7 * bt 0.0 * 0.2 * #2: p l1-l2 + p s r1-r2 synthesized from m r2 ae+bt+mt 29.3 26.8 bt+mt 28.5 * 25.9 * ae+bt 28.3 * 25.0 * bt 13.2 * 12.1 * ## analysis of clean sentences altered into ugt this section takes a closer look at the parallel data synthesized by approach #1 to observe how the clean sentences from p l1-l2 parallel data were altered and to better understand why the use of synthetic data leads to a better nmt system for ugt. we first focus on some of the characteristics of the mtnt datasets and compare how well these characteristics are exhibited in p s r1-r2 . for this analysis, we mainly relied on the scripts and resources provided by @xcite . 17 we randomly sampled source sentences from p l1-l2 and p s r1-r2 as much as there are in the mtnt test sets, and performed our analysis on them. 18 we counted the occurrences of profanities in the english, french, and japanese. for english, we also counted the number of word contractions 19 and internet slang expressions. we also counted words ending by ''-ise'' and ''-ize'' to account for some of the differences between us english and uk english word spellings. because p l1-l2 is mainly made of europarl, we can expect that uk english spelling is mainly used, whereas we expect to find a higher ratio of us english spelling in the reddit data, since reddit is an american platform. for japanese, we counted the numbers of formal and informal pronouns, assuming that mtnt datasets contain more informal pronouns than p l1-l2 . @xcite also counted spelling and grammar errors, and emojis. we did not count spelling and grammar errors, expecting that they are artificially numerous in our synthetic data, since they had been automatically generated. as for the emojis, both p l1-l2 and p s r1-r2 did not contain any. another line of work exploits nmt to perform style transfer across texts, that is, applying some characteristics of one text to another, without exploiting any parallel data of ugt, but has never been applied to nmt for ugt. prabhumoye et al. (2018) performed style transfer through back-translation to preserve the meaning of the text while reducing its stylistic properties and then exploit adversarial generation algorithms to apply the desired style to the back-translated texts, assuming that meaning and style can be disentangled. their approach also requires a classifier that can accurately predict the style of a given text. @xcite proposed a threestep pipeline combining unsupervised statistical and neural mt to generate instances of texts in the targeted style that is then evaluated by a given style classifier as in @xcite . ## conclusion we described two new methods for synthesizing parallel data to train better nmt systems for ugt. both methods work through a zero-shot nmt system, initialized with a pre-trained crosslingual language model that exploits monolingual corpora of ugt. our first method (#1) successfully alters clean parallel data into parallel data that exhibit the characteristics of ugt of the targeted style. our second method (#2) uses the same zero-shot nmt system to translate monolingual corpora of ugt for synthesizing parallel data useful to train nmt. we showed that both methods, separately or combined, improve translation quality for ugt. for future work, we will study the use of manually produced ugt parallel data to better train our nmt system that synthesizes the parallel data. we will also explore other applications for this framework, such as paraphrase generation. we will also investigate the use of the recently proposed mirror-generative nmt @xcite , a semi-supervised architecture that exploits jointly large source and target monolingual corpora, such as those of ugt, during training using source and target language models in the same latent space.