Controlling Formality in Low-Resource NMT with Domain Adaptation and Re-Ranking: SLT - CDT - U o S at IWSLT 2022 (2022)

Abstract: This paper describes the SLT-CDT-UoS group’s submission to the first Special Task on Formality Control for Spoken Language Translation, part of the IWSLT 2022 Evaluation Campaign. Our efforts were split between two fronts: data engineering and altering the objective function for best hypothesis selection. We used language-independent methods to extract formal and informal sentence pairs from the provided corpora; using English as a pivot language, we propagated formality annotations to languages treated as zero-shot in the task; we also further improved formality controlling with a hypothesis re-ranking approach. On the test sets for English-to-German and English-to-Spanish, we achieved an average accuracy of .935 within the constrained setting and .995 within unconstrained setting. In a zero-shot setting for English-to-Russian and English-to-Italian, we scored average accuracy of .590 for constrained setting and .659 for unconstrained.

## introduction formality-controlled machine translation enables the system user to specify the desired formality level at input so that the produced hypothesis is expressed in a formal or informal style. due to discrepancies between different languages in formality expression, it is often the case that the same source sentence has several plausible hypotheses, each aimed at a different audience; leaving this choice to the model may result in an inappropriate translation. this paper describes our team's submission to the first special task on formality control in slt at iwslt 2022 @xcite , where the objective was to achieve control over binary expression of formality in translation (enable the translation pipeline to generate formal or informal translations depending on user input). the task evaluated translations from english (en) into german (de), spanish (es), russian (ru), italian (it), japanese (ja) and hindi (hi). among these, en-{ru,it} were considered zero-shot; for other pairs, small paired formality-annotated corpora were provided. the task ran in two settings: constrained (limited data and pre-trained model resources) and unconstrained (no limitations on either resource). submissions within both the constrained and unconstrained track were additionally considered in two categories: full supervision and zero-shot. our submission consisted of four primary systems, one for each track/subtrack combination, and we focused on the en-{de,es,ru,it} language directions. we were interested in leveraging the provided formality-annotated triplets (src, tgt formal , tgt informal ) to extract sufficiently large annotated datasets from the permitted training corpora, without using language-specific resources or tools. we built a multilingual translation model in the given translation directions and fine-tuned it on our collected data. our zero-shot submissions used fine-tuning data only for the non-zeroshot pairs. to boost the formality control (especially within the constrained track), we included a formality-focused hypothesis re-ranking step. our submissions to both tracks followed the same concepts, with the unconstrained one benefitting from larger corpora, and thus more fine-tuning data. in section 2 we describe our submission to the constrained track, including the data extraction step (section 2.2, 2.3). our approach begins with extending this small set to cover more samples by extracting them from the allowed corpora. we use a language-independent approach of domain adaptation for this. then, we extract samples for the zero-shot pairs (en-{ru,it}) based on data collected for (en-{de,es}). we then experiment with re-ranking the top n model hypotheses with a formality-focused objective function. within our systems, we provide the formality information as a tag appended to the input of the model. throughout the paper we use f to denote the formal style and i to denote the informal style. all our models submitted to the "supervised" subtracks achieved an average of +.284 accuracy point over a baseline for all en-{de,es,ru,it} test sets, while the "zero-shot" models achieved an average improvement of .124 points on the en-{ru,it} test sets. our work highlights the potential of both data adaptation and re-ranking approaches in attribute control for nmt. ## constrained track the must-c textual corpus @xcite with quantities listed in in all our experiments we used the multilingual transformer model architecture provided within fairseq @xcite . for our pre-training data we used the full must-c corpus. we applied sentencepiece @xcite to build a joint vocabulary of 32k tokens across all languages. we list the model specifications in ## unconstrained track our submission for the unconstrained track largely copies the constrained track one, but is applied to a larger training corpus. ## final results we report the final evaluation results in within the constrained track, we achieved nearideal accuracy for the dominant formality for each language pair (between .961 and 1.000) with the supervised model. scores for non-dominant formalities are weaker but still impressive for en-{de,es} with an average of .880. our best model for en-{ru,it} improved by .193 accuracy points over the baseline. the models submitted to the unconstrained track again achieved an impressive average accuracy of .992 for dominant formality; additionally, performance for non-dominant formality in en-{de,es} improved significantly w.r.t. the constrained model, also averaging .992. this means that with enough training data our methods were capable of matching the performance on a minority class w.r.t. a majority class. finally, contrary to the constrained track, the unconstrained-zero-shot model achieved the best accuracy for zero-shot pairs, to an average of .659. ## conclusions overall results suggest that it is easy for a pretrained translation model to learn controlled expression of the dominant type within a dichotomous phenomenon while learning to render the less-expressed type is significantly harder, especially in a low-resource scenario. our methods applied to the supervised language pairs (englishto-german, english-to-spanish) worked near unfailingly, but using english as a pivot language to propagate formality information did not help achieve similar results for the zero-shot pairs. we suspect that the significant accuracy gains from formalityreranking may have been partially due to formality in the studied language pairs itself being expressed primarily via certain token words such as the honorific sie in german creating a pivot effect @xcite . as such, it may be of interest for future research to study such methods applied to more complex phenomena, such as grammatical expression of gender. finally, results for the en-{ru,it} language pairs may not have been as good as expected because we used the inferred data from the constrained track to build the relative frequency model, but the inferred data turned out to be not as high quality as we expected. future work may investigate a robust solution to this problem of propagating formality via a source (pivot) language to extract training data for other language pairs.