Low-Resource Formality Controlled NMT Using Pre-trained LM (2023)

Abstract: This paper describes the UCSC’s submission to the shared task on formality control for spoken language translation at IWSLT 2023. For this task, we explored the use of ‘additive style intervention’ using a pre-trained multilingual translation model, namely mBART. Compared to prior approaches where a single style-vector was added to all tokens in the encoder output, we explored an alternative approach in which we learn a unique style-vector for each input token. We believe this approach, which we call ‘style embedding intervention,’ is better suited for formality control as it can potentially learn which specific input tokens to modify during decoding. While the proposed approach obtained similar performance to ‘additive style intervention’ for the supervised English-to-Vietnamese task, it performed significantly better for English-to-Korean, in which it achieved an average matched accuracy of 90.6 compared to 85.2 for the baseline. When we constrained the model further to only perform style intervention on the <bos> (beginning of sentence) token, the average matched accuracy improved further to 92.0, indicating that the model could learn to control the formality of the translation output based solely on the embedding of the <bos> token.

## introduction in the past decade, neural machine translation has made remarkable strides, achieving translation quality that is increasingly comparable to human-level performance across various languages. however, despite these advancements, the field of controllable machine translation remains relatively under-explored. one crucial aspect of translation variation is formality, which manifests through grammatical registers, adapting the language to suit specific target audiences. unfortunately, current neural machine translation (nmt) systems lack the capability to comprehend and adhere to grammatical registers, specifically concerning formality. consequently, this limitation can result in inaccuracies in selecting the appropriate level of formality, potentially leading to translations that may be deemed inappropriate in specific contexts. recognizing the significance of formality control, we aim to build a formality-controlled machine translation system to foster smooth and reliable conversations and enhance communication across languages and cultures, facilitating more nuanced and effective linguistic exchanges. formality-controlled neural machine translation is the iwslt 2023 task @xcite under the formality track. the goal of the task is to achieve formality controlled machine translation for the english-vietnamese (en-vi), english-korean (en-ko) in a supervised setting and english-portuguese (en-pt) and english-russian (en-ru) in a zero-shot setting as detailed in @xcite ). we provide an example of formal and informal translations of an english sentence into vietnamese in figure 1 . the formal and informal tokens are in bold. ## related works machine translation (mt) research has primarily focused on preserving the meaning between languages. however, it is widely recognized that maintaining the intended level of formality in communication is a crucial aspect of the problem @xcite . this field of research was named formality-sensitive machine translation (fsmt) @xcite , where the target formality level is considered in addition to the source segment in determining the translated text. further, several studies have attempted to regulate formality in mt through side constraints to control politeness, or formality @xcite ; @xcite ; @xcite . other studies have tried to address this with custom models trained on data with consistent formality @xcite . most prior research has been tailored to individual languages and has labeled large amounts of data using word lists or morphological analyzers. ## grounding results and observations along with the validation splits, we ground our approach by comparing our results with the 2022 formality track submission @xcite . we compare our results on one language pair i.e. english-spanish. the comparison is shown in ## official results along with the approach from @xcite taken as a baseline and an adapted version of it, we submit the results of our approach and of the 'bos style intervention' approach. we analyse the performance of our models under the supervised setting and the zero-shot setting. we also generate results on the flores-200 test split. ## conclusion in this paper, we presented and explored "style embedding intervention," a new approach for lowresource formality control in spoken language translation. by assigning unique style vectors to each input token, the proposed approach shows promising results in understanding and controlling the nuances of formal and informal style translation. it outperforms previous "additive style intervention" methods, specifically for the english-to-korean translation task, resulting in an average matched accuracy improvement from 85.2 to 90.6. further, on analysis of our "style embedding intervention" model, we find that most of the style information is learnt in @xmath0. constraining style addition to @xmath1.6 to 92. we also observed that in a zero-shot setting, the formality control doesn't seem to transfer well, and the model leans towards biases learnt during pretraining rather than the transferred style interventions. this is more pronounced for en-ru translations where the model is more biased towards the formal style, with a matched accuracy of 98.8, than the informal style, with a matched accuracy of 51.6. future works focused on alleviating the style biases of pre-trained models might be necessary to ensure style transfer works equally well in a zero-shot setting. we hope our work on translation models with interpretable formality control can serve as a base for other future works on interpretable models, especially in low-resource settings.