Text Style Transfer Back-Translation (2023)

Abstract: Back Translation (BT) is widely used in the field of machine translation, as it has been proved effective for enhancing translation quality. However, BT mainly improves the translation of inputs that share a similar style (to be more specific, translation-liked inputs), since the source side of BT data is machine-translated. For natural inputs, BT brings only slight improvements and sometimes even adverse effects. To address this issue, we propose Text Style Transfer Back Translation (TST BT), which uses a style transfer to modify the source side of BT data. By making the style of source-side text more natural, we aim to improve the translation of natural inputs. Our experiments on various language pairs, including both high-resource and low-resource ones, demonstrate that TST BT significantly improves translation performance against popular BT benchmarks. In addition, TST BT is proved to be effective in domain adaptation so this strategy can be regarded as a generalized data augmentation method. Our training code and text style transfer model are open-sourced.

## introduction works in neural machine translation (nmt) @xcite @xcite greatly improve translation quality. however, current methods generally require large amount of bilingual training data, which is a challenging and sometimes impossible task. as obtaining monolingual data is much easier, researchers have long exploited methods to enhance model performances using monolingual data, for example, language model fusion for phrase-based @xcite and neural machine translation @xcite , back translation @xcite , and dual learning @xcite figure 1 : bilingual and bt data used for @xmath0. n ature indicates data generated by native speakers; ht indicates data generated by human translators from another language, and m t indicates machine translation results. m t and ht styles are close, but far from n ature. back translation (bt), a data augmentation method to generate synthetic parallel data by translating content from target language back to source language, is widely used in the field of machine translation. bt has many variants @xcite @xcite and each has own merits. in terms of text style, models that use bt are usually trained on three types of data. real parallel data constitutes the first two types: natural source with human-translated target (n @xmath1. back translation data constitutes the third type: machine-translated source with natural target (m @xmath2, as shown in figure 1 . inspired by @xcite , who find that a classifier can distinguish m t data from ht data, we train a similar classifier to classify n ature and m t data and find that a high percentage of original text is marked as n ature by the classifier. however, the percentage of n ature content is low in human-translated data and even lower in machine-translated data. in general, human and machine translated data are similar, but far different from original text. we find that when the input style is close to n ature, the output is biased towards ht ; and when the input style is closed to ht , the output is biased towards n ature (for details, see section 6.1). since the input used to generate bt data is n ature, the output is close to ht . so bt mainly improves the translation of translationlike inputs. for natural inputs, bt brings only slight improvements and sometimes even adverse effects. however, in practical use, most inputs sent to nmt models are natural language written by native speakers, rather than translation-like content. we use one original test set (n @xmath3. as shown in based on the finding, this paper aims to explore a method to enhance translation of n ature input on basis of bt, while maintaining its effectiveness in translating translation-like content. since bt connects translation-like input with n ature target, we assume that if we could connect n ature input with n ature target, translation of n ature input could be further enhanced. therefore, we propose text style transfer back translation (tst bt), aiming to turn m @xmath4. however, transferring translation-like text to a natural style is a zero-shot issue, because we can hardly obtain parallel data with the same meaning but different styles (m t and n ature). we propose two unsupervised methods. our experiments on highresource and low-resource language pairs demonstrate that tst bt can significantly enhance translation of n ature input on basis of bt variants while brings no adverse effect on ht inputs. we also find that tst bt is effective in domain adaptation, demonstrating generalizability of our method. our contributions are as follows: ## method we propose cascaded and direct approaches (see figure 2 ) to transfer the style of source-side bt data. ## result tst can be combined with popular bt strategies. for other bt methods, we directly modify the source side of bt data using our tst model. to prove the effectiveness of tst bt, we perform experiments on high-resource (ende and zhen) and low-resource (enro) languages, as well as domain adaptation. ## analysis 6.1 style tide as shown in figure 1 , bilingual data can be divided into n ature to ht or ht to n ature. by learning such data, the model inclines to generate translation-like output when the input is n ature, and vice versa. to illustrate the phenomenon, we perform several rounds of translation on en n ature and de n ature data from wmt18 ende test set. we calculate the proportion of n ature text marked by the classifier after each round of translation. as shown in figure 4 , the percentage of n ature sentences fluctuates regularly after each round of translation, no matter the translation starts from de or en. for english original data, the percentage of n ature data is 85.7% before translation. the percentage drops to 7.3% after the first round of translation into german, and then bounces back to 51.9 after the second round of translation back into english. as we analyzed above, the style of input determines the style of output. in general, the composition of bilingual data, as well as the difference between n ature and ht style, makes the source-side bt text significantly different from n ature text. as a result, the translation of n ature input can hardly be improved by standard bt. ## conclusion this paper proposes text style transfer back translation (tst bt) to further enhance bt effectiveness. we make a detailed analysis of training data styles and find that bt hardly improves translation of n atural inputs, which are the main inputs in practical use. our method simply modifies the style of source-side bt data, which brings significant improvements on translation quality, both in high-resource and low-resource language scenarios. further experiment finds that tst bt is also effective in domain adaptation, which can further expand the application of our method. the generalizability of tst bt is thus proved.