Controlling Translation Formality Using Pre-trained Multilingual Language Models (2022)

Abstract: This paper describes the University of Maryland’s submission to the Special Task on Formality Control for Spoken Language Translation at IWSLT, which evaluates translation from English into 6 languages with diverse grammatical formality markers. We investigate to what extent this problem can be addressed with a single multilingual model, simultaneously controlling its output for target language and formality. Results show that this strategy can approach the translation quality and formality control achieved by dedicated translation models. However, the nature of the underlying pre-trained language model and of the finetuning samples greatly impact results.

## introduction while machine translation (mt) research has primarily focused on preserving meaning across languages, linguists and lay-users alike have long known that pragmatic-preserving communication is an important aspect of the problem @xcite . to address one dimension of this, several works have attempted to control aspects of formality in mt @xcite @xcite . indeed, this research area was formalized as formality-sensitive machine translation (fsmt) by @xcite , where the translation is not only a function of the source segment but also the desired target formality. the lack of gold translation with alternate formality for supervised training and evaluation has lead researchers to rely on manual evaluation and synthetic supervision in past work @xcite . additionally, these works broadly adapt to formal and informal registers as opposed to specifically controlling grammatical formality. the special task on formality control on spoken language translation provides a new benchmark by contributing high-quality training datasets ⇤ equal contribution. source: do you like 1 legos? did you 2 ever play with them as a child or even later? german informal: magst du 1 legos? hast du 2 jemals als kind mit ihnen gespielt oder sogar später? german formal: mögen sie 1 legos? haben sie 2 jemals als kind mit ihnen gespielt oder sogar später? for diverse languages @xcite . in this task, a source segment in english is paired with two references which are minimally contrastive in grammatical formality, one for each formality level (formal and informal; in this work, we investigate how to control grammatical formality in mt for many languages with minimal resources. specifically, we ask whether a single multilingual model can be finetuned to translate in the appropriate formality for any of the task languages. we introduce additive vector interventions to encode style on top of mt5-large @xcite and mbart-large @xcite , and investigate the impact of finetuning on varying types of gold and synthetic samples to minimize reliance on manual annotation. ## method given an input sequence x, we design a single model that produces an output y ⇤ = arg max p(y|x, l, f ; ✓ lm , ✓ f ) for any language l and formality level f considered in this task. the bulk of its parameters ✓ lm are initialized with a pre-trained multilingual language model. a small number of additional parameters ✓ f enable formality control. all parameters are finetuned for formality-controlled translation. ## evaluation settings data the shared task provides english source segments paired with two contrastive reference translations, one for each formality level (informal and formal) for four language pairs: en-{de, es, ja, hi} in the supervised setting and two language pairs: en-{ru, it} in the zero-shot setting. the sizes and properties of the datasets for the supervised language pairs are listed in ## experimental conditions we compare multilingual models, where a single model is used to generate formal and informal translations for all languages with bilingual models trained for each language pair, as detailed below. ## system development results during system development, we explore the impact of different types of training samples and finetuning strategies on translation quality and formality accuracy on task dev. ## official results submissions we submit five variants of multilingual models (numbered [1-5] in tables 8 9 10 11 and compare them to the bilingual models built on top of the organizers' baselines. we first discuss results on the official test split for the supervised setting (tables 8 , 9 ). to better understand the degree of overall control afforded, we also report the average scores of the formal and informal settings in ## related work most mt approaches only indirectly capture the style properties of the target text. while efforts have been made to generate better outputs in their pragmatic context via controlling formality @xcite @xcite , complexity @xcite , gender @xcite , these studies only focus a single language pair. due to the paucity of style annotated corpora, zero-shot style transfer within and across languages has received a lot of attention. however, adapting pre-trained large-scale language models during inference using only a few examples @xcite @xcite limits their transfer ability and the diversity of their outputs. while prior works use pre-trained language models like bert, gpt to intialize ✓ lm for improving translation quality @xcite , in this work, we focus on adapting sequence-tosequence multilingual models for controlled generation of a desired formality and study style transfer in multilingual supervised and zero-shot settings. ## conclusion we present the university of maryland's submission which examines the performance of a single multilingual model allowing control of both target language and formality. results show that while multilingual fsmt models lag behind large, bilingual, formality-specific models in terms of mt quality, they show stronger formality control performance across all the language pairs. furthermore, while synthetic unpaired triplets help mt5-large with fsmt performance and the two-stage finetuning process improves mt quality and contrastive task performance, mbart-large still outperforms this class of models, likely due to its large amount of pre-training supervision. in future work, we suggest a deeper investigation of potentially confounding roles in the study of fsmt, such as the impact of formal register as compared to grammatical formality in training data. we also suggest a thorough analysis of what is transferred in the zero-shot setting. finally, we recommend an audit of underlying pre-training and finetuning data sources for pre-trained multilingual models, which we believe hinder zero-shot formality transfer for en-it and en-ru in which a single formality is strongly preferred.