Multilingual Pre-training with Language and Task Adaptation for Multilingual Text Style Transfer (2022)

Abstract: We exploit the pre-trained seq2seq model mBART for multilingual text style transfer. Using machine translated data as well as gold aligned English sentences yields state-of-the-art results in the three target languages we consider. Besides, in view of the general scarcity of parallel data, we propose a modular approach for multilingual formality transfer, which consists of two training strategies that target adaptation to both language and task. Our approach achieves competitive performance without monolingual task-specific parallel data and can be applied to other style transfer tasks as well as to other languages.

## approach and data as a base experiment aimed at exploring the contribution of mbart @xcite for multilingual style transfer, we fine-tune this model with parallel data specifically developed for style transfer in english (original) and three other languages (machine translated). next, in view of the common situation where parallel data for a target language is not available, we propose a two-step adaptation training approach on mbart that enables modular multilingual tst. we avoid iterative back-translation (ibt) @xcite , often used in previous tst work @xcite @xcite , since it has been shown to be computationally costly @xcite . we still run comparison models that use it. in the first adaptation step, we address the problem of some languages being not well represented in mbart, which preliminary experiments have shown to hurt our downstream task. 3 we conduct a language adaptation denoising training using unlabelled data for the target language. in the second step, we address the task at hand through finetuning cross-attention with auxiliary gold parallel english data adapting the model to the tst task. for tst fine-tuning, we use parallel training data, namely formal/informal aligned sentences (both manually produced for english and machine translated for three other languages). for the adaptation strategies, we also collect formality and generic non-parallel data. details follow. english formality data gyafc @xcite ) is an english dataset of aligned formal and informal sentences. gold parallel pairs are provided for training, validation, and test. multilingual formality data xformal @xcite is a benchmark for multilingual formality transfer, which provides an evaluation set that consists of four formal rewrites of informal sentences in br-pt, fr, and it. this dataset contains pseudo-parallel corpora in each language, obtained via machine translating the english gyafc pairs. language-specific formality non-parallel data following @xcite and @xcite , we crawl the domain data in target language from yahoo answers. 4 we then use the style regressor from @xcite to predict formality score σ of the sentence to automatically select sentences in each style direction. 5 language-specific generic non-parallel data 5 m sentences containing 5 to 30 words for each language randomly selected from news crawl. foot_5 ## adaptation training to adapt mbart to multilingual tst, we employ two adaptation training strategies that target language and task respectively. ## experiments all experiments are implemented atop transformers @xcite using mbart-large-50 @xcite . we train the model using the adam optimiser (kingma and ba, 2015) with learning rate 1e-5 for all experiments. we train the language adaptation modules with generic texts separately for each language for 200k training steps with batch size 32, accumulating gradients over 8 update steps, and set it to 1 for other training. evaluation following previous work @xcite , we assess style strength and content preservation. we fine-tune mbert @xcite with @xcite 's pseudo-parallel corpora to evaluate the style accuracy of the outputs. we also use a style regressor from @xcite , which is based on xlm-r @xcite and is shown to correlate well with human judgments. 7 we calculate bleu and comet @xcite to assess content preservation. as overall score, following previous work, we compute the harmonic mean (hm) of style accuracy and bleu. systems based on our data (section 2), we have four settings for our systems. d1: pseudo-parallel data in the target language via machine translating the english resource; d2: non-parallel style data in the target language; d3: no style data in the target language; d4: no parallel data at all. the first three settings all contain gold english parallel data. results in d2, we see that our proposed adaptation approaches outperform ibt-based models on both transfer directions. the results of fine-tuning the target language's model with english parallel data are generally better than inserting the en model's cross-attention module into the target language's model. this suggests that the former can better transfer task and domain knowledge. in d3, the large amounts of generic texts yield more improvement in @xmath4. this could be due to generic texts being more formal than informal. the performance improvement on portuguese is particularly noticeable (compare m3.1 trained with en data only with other m3.x models), and mostly due to this language being less represented than the others in mbart. interestingly, the performance of task adaptation strategies is reversed compared to d2: it is here better to adapt cross attention in the english model rather than fine-tune the target language model directly. future work will need to investigate how using different data sources for language adaptation (d2, style-specific vs d3, generic) interacts with task adaptation strategies. results for d4 show that language adaptation training helps with content preservation, especially for portuguese, confirming this curbs the problem of language underrepresentation in pre-training. however, low performance on style accuracy shows that task-specific data is necessary, even if it comes from a different language. ## analysis and discussion case study ## conclusions fine-tuning a pre-trained multilingual model with machine translated training data yields state-of-theart results for transferring informal to formal text. the results for the formal-to-informal direction are considerably worse-the task is more difficult, and the quality of translated informal text is lower. we have also proposed two adaptation training strategies that can be applied in a cross-lingual transfer strategy . these strategies target language and task adaptation, and can be combined to adapt mbart for multilingual formality transfer. the adaptation strategies with auxiliary parallel data from a different language are effective, yielding competitive results and outperforming more classic ibt-based approaches without task-specific parallel data. lastly, we have shown that formal-to-informal transformation is harder than the opposite direction.