FAME - MT Dataset: Formality Awareness Made Easy for Machine Translation Purposes (2024)

Abstract: People use language for various purposes. Apart from sharing information, individuals may use it to express emotions or to show respect for another person. In this paper, we focus on the formality level of machine-generated translations and present \textbf{FAME-MT} – a dataset consisting of 11.2 million translations between 15 European source languages and 8 European target languages classified to formal and informal classes according to target sentence formality. This dataset can be used to fine-tune machine translation models to ensure a given formality level for 8 European target languages considered. We describe the dataset creation procedure, the analysis of the dataset’s quality showing that \textbf{FAME-MT} is a reliable source of language register information, and we construct a publicly available proof-of-concept machine translation model that uses the dataset to steer the formality level of the translation. Currently, it is the largest dataset of formality annotations, with examples expressed in 112 European language pairs. The dataset is made available online.

## introduction motivation different situations require using different language depending on whether it is a formal meeting or a casual talk with friends. fre-quently, when speaking to an older or distinguished person, e.g., an owner of a company or a university professor, we use appropriate language forms to show respect. however, machine translation models may struggle with choosing an appropriate language form due to lack of context -often, we only have one sentence to translate, there are cultural differences between source and target language speakers, and models may be trained on parallel corpora, which do not focus on formality aspect of language enough. for this reason, it is important to find ways of enforcing machine translation models to use a required formality level in translated sequences. while there are methods that can be used to achieve this goal, the existing datasets are scarce, focusing either on formality classification for a few selected languages (e.g., german or english) or providing formality-annotated examples of translations between pairs of languages, which are limited in size and include a narrow selection of target languages with english as the source language. what we found missing is a large-scale dataset providing formality-annotated examples of translations for a much broader set of languages, enabling easy fine-tuning of pre-trained machine translation models not only for directions including english but also other european languages. contribution in this paper, we present fame-mt, the biggest to date dataset of translation pairs annotated with formality level. this dataset introduces 100,000 annotated translation examples for each of the 112 european language pairs considered. with 8 target languages @xcite classified according to formality and 15 european languages considered as the source lan-guages, the dataset introduces an 18-fold increase of the european language-pair coverage over the most diverse dataset cocoa-mt @xcite and over 100-fold increase of the dataset size considering the biggest formality-annotated datasets available @xcite . providing translation examples for a wide selection of source and target languages, fame-mt allows for simple fine-tuning of machine translation models for the most popular european language pairs. in this paper, we discuss the dataset creation process and analyze the data quality using various metrics to prove the usefulness of fame-mt. we show how proof-of-concept models for formalityaware machine translation can be trained using marian @xcite , provide examples of outputs of those models, and publish models online along with the dataset. ## dataset the fame-mt dataset creation process was divided into three steps. ## explorative analysis to verify the quality of the dataset, we selected several approaches that would show key characteristics of the dataset. average length of sentences one of the basic metrics that can describe the difference between formal and informal language is the average sentence length in each of the categories. we may expect that formal sentences may be longer than informal, which are frequently used to share knowledge quickly. figure 1 proves that for each target language, the formal translations are much longer than informal ones in the collected dataset. ## application to machine translation this section explores the application of the fame-mt dataset to enable control over the formality tone of translated text. we conducted experiments on two translation directions: english-german and english-polish, leveraging pre-trained models from the opus collection (tiedemann, 2020) for fine-tuning. ## discussion as we have shown in the previous section, the fame-mt dataset can make pre-trained models formality-aware. also, metrics used to analyze the dataset show that the characteristics of the dataset an interesting observation is that even though the cocoa-mt dataset introduces a relatively small dataset of formal and informal examples per each language, the accuracy of the pre-trained mdeberta-v3-base model, which is fine-tuned on cocoa-mt is very high when measured on cocoa-mt's test set with neutral samples added (0.9552). an analysis of the annotations provided in cocoa-mt reveals that it focuses on expressing formality using personal pronouns. however, formality level, in general, can be expressed using various language constructs, e.g., contractions, formal greetings, slang words, and appropriate personal pronouns. on the other hand, e.g., gyafc focuses more on constructs other than personal pronouns due to the lack of formal you in english. we think that this is the reason for the high scores observed for a small dataset (cocoa-mt) and lower scores for a much bigger dataset (gyafc). the focus of cocoa-mt on pronouns leads to too optimistic quality estimates as compared to the evaluation provided for the english classifier. however, since formal constructs co-occur with each other, classifiers trained using cocoa-mt work well in practice. as we can see in tables 7 and 8 -models trained with fame-mt introduce subtle differences depending on the expected formality level (e.g., hätte gerne vs. möchte or koncernem ochroniarskim vs. firm ą ochroniarsk ą). the motivation for using classifiers to classify only sentences in target languages is supported by the analysis of source vs. target sentence formality. as we have shown that the formality level of the source language does not determine the formality of the target language, in fame-mt, we collect sentences of various levels of formality mapped to a given target sentence formality level. this way, we can inject a token representing the desired formality level (be it formal or informal) to steer the expected formality level. thus, having only target sentences classified, we can fine-tune machine translation models to become formality-aware, regardless of the formality level of the source sequence. ## addressing research questions in this paper, we stated three research questions that can be answered now. regarding the first one, we show that using a set of classifiers for the parallel corporas' target languages, we created a large dataset, which is useful for training formality-aware mt models. we proved it by finetuning general machine translation models and utilizing metrics confirming intuitions, thus, the answer to rq1 is positive. also, the experiments show that fine-tuning pre-trained opus models with 50,000 formal and 50,000 informal examples is enough for fine-tuning english-german and english-polish pairs of languages. the quality of the english-german model is higher but the quality of english-polish is also satisfactory. for this reason, we can give the positive answer to rq2. finally, the classifiers used reveal that the formality level of the source language does not determine the formality level of the target language in the datasets collected using mtdata. this observation justifies the need to inject a special formality token to ensure a given formality level of the translation. thus, the answer to rq3 is negative. ## conclusions in this paper, we introduce fame-mt -a dataset consisting of 11.2 million translations between 112 european language pairs, where sentences in target languages are classified as formal or informal. as the dataset is a computer-generated silver standard, we used a set of metrics to prove the good quality of the data. moreover, proof-of-concept models fine-tuned using formality data show that the dataset can be successfully utilized in problems requiring enforcing a given formality level of the system's output. due to its size and large number of language pairs selected, fame-mt is the largest and most diverse dataset available, introducing 18 times more european language pairs than biggest existing multilingual datasets (112 language pairs vs. 6 in cocoa-mt) and providing over 100 times more sentences annotated with formality level than the biggest datasets (11.2 million vs. 110,000 in gyafc). we made the dataset publicly accessible, and provided all the source codes for rerunning the analysis foot_9 . we hope that this dataset may help to produce better formalityaware machine translation models especially for pairs of languages that were not yet covered or underrepresented in existing datasets (e.g., @xmath1, @xmath2, or @xmath3.