Olá, Bonjour, Salve! XFORMAL : A Benchmark for Multilingual Formality Style Transfer (2021)

Abstract: We take the first step towards multilingual style transfer by creating and releasing XFORMAL, a benchmark of multiple formal reformulations of informal text in Brazilian Portuguese, French, and Italian. Results on XFORMAL suggest that state-of-the-art style transfer approaches perform close to simple baselines, indicating that style transfer is even more challenging when moving multilingual.

## introduction style transfer (st) is the task of automatically transforming text in one style into another (for example, making an impolite request more polite). most work in this growing field has focused primarily on style transfer within english, while covering different languages has received disproportional interest. concretely, out of 35 st papers we reviewed, all of them report results for st within english text, while there is just a single work covering each of the following languages: chinese, russian, latvian, estonian, and french @xcite @xcite . notably, even though some efforts have been made towards multilingual st, researchers are limited to providing system outputs as a means of evaluation, and progress is hampered by the scarcity of resources for most languages. at the same time, st lies at the core of human communication: when humans produce language, they condition their choice of grammatical and lexical transformations to a target audience and a specific situation. among the many possible stylistic variations, @xcite argue that "a dimension similar to formality appears as the most important and universal feature distinguishing styles, registers or genres in different languages". consider the informal excerpts and their formal reformulations in french (fr) and brazilian portuguese il ne prêtait pas attention à la situation. he was not paying attention to the situation. this work takes the first step towards a more language-inclusive direction for the field of st by building the first corpus of style transfer for non-english languages. in particular, we make the following contributions: 1. building upon prior work on formality style transfer (fost) @xcite , we contribute an evaluation dataset, xformal that consists of multiple formal rewrites of informal sentences in three romance languages: brazilian portuguese (br-pt), french (fr), and italian (it); 2. without assuming access to any gold-standard training data for the languages at hand, we benchmark a myriad of leading st baselines through automatic and human evaluation methods. our results show that fost in non-english languages is particularly challenging as complex neural models perform on par with a simple rule-based system consisting of handcrafted transformations. we make xformal, our annotations protocols, and analysis code publicly available and hope that this study facilitates and encourages more research towards multilingual st. ## related work controlling style aspects in generation tasks is studied in monolingual settings with an english-centric focus (intra-language) and cross-lingual settings together with machine translation (mt) (interlanguage). our work rests in intra-language st with a multilingual focus, in contrast to prior work. st datasets that consist of parallel pairs in different styles include: gyafc for formality @xcite , yelp @xcite and amazon product reviews for sentiment @xcite , political slant and gender controlled datasets @xcite , expert style transfer @xcite , pastel for imitating personal @xcite , simile for simile generation @xcite , and others. intra-language st was first cast as generation task by @xcite and is addressed through methods that use either parallel data or unpaired corpora of different styles. parallel corpora designed for the task at hand are used to train traditional encoder-decoder architectures @xcite , learn mappings between latent representation of different styles @xcite , or fine-tune pre-trained models @xcite . other approaches use parallel data from similar tasks to facilitate transfer in the target style via domain adaptation @xcite , multi-task learning @xcite , and zero-shot transfer @xcite or create pseudo-parallel data via data augmentation techniques @xcite . approaches that rely on non-parallel data include disentanglement methods based on the idea of learning style-agnostic latent representations (e.g., @xcite ; @xcite ). however, they are recently criticized for resulting in poor content preservation @xcite @xcite and alternatively, translation-based models are proposed that use reconstruction and back-translation losses (e.g., @xcite ; prabhumoye et al. ( 2018 )). another line of work, focuses on manipulation methods that remove the style-specific attribute of text (e.g., li et al. (2018); xu et al. (2018) ), while recent approaches use reinforcement learning (e.g., @xcite ; @xcite , probabilistic formulations @xcite , and masked language models @xcite . inter-language st is introduced by @xcite who proposed personalized mt for en-french and en-german. subsequent mt works control for politeness @xcite , voice @xcite , personality traits @xcite , user-provided terminology @xcite , gender @xcite , formality @xcite , morphological variations @xcite , complexity @xcite and reading level @xcite . ## xformal collection we describe the process of collecting formal rewrites using data statements protocols @xcite . ## (step 3). finally, we randomly sample 1,000 sentences from the pool of informal candidates for each language. procedures we use the amazon mechanical turk (mturk) platform to collect formal rewrites for our informal sentences. for each language, we split the annotation into 20 batches of 50 human intelligence tasks (hits). in each hit, turkers are given an informal excerpt and asked to generate its formal rewrite in the same language without changing its meaning. we collect 4 rewrites per excerpt and release detailed instructions under a.g. ## xformal statistics & analysis types of formality edits following @xcite , we analyze the most frequent edit operations turkers perform when formalizing the informal sentences. we conduct both an automatic analysis (details in a.i) of the whole set of rewrites, and a human analysis (details in a.h) of a random sample of 200 rewrites per language (we recruited a native speaker for each language). ## multilingual fost experiments we benchmark eight st models on xformal to serve as baseline scores for future research. we describe the models ( §5.1), the experimental setting ( §5.2), the human and automatic evaluation methods ( §5.3 and §5.4), and results ( §5.5). ## conclusions & future directions this work extends the task of formality style transfer to a multilingual setting. specifically, we contribute xformal, an evaluation testbed consisting of informal sentences and multiple formal rewrites spanning three languages: br-pt, fr, and it. as in @xcite turkers can be effective in creating high quality st corpora. in contrast to the aforementioned en corpus, we find that the rewrites in xformal tend to be more diverse, making it a more challenging task. additionally, inspired by work on cross-lingual transfer and en fost, we benchmark several methods and perform automatic and human evaluations on their outputs. we found that nmt-based ensembles are the best performing models for fr and br-pt-a result consistent with en-however, they perform comparably to a naive rule-based baseline for it. to further facilitate reproducibility of our evaluations and corpus creation processes, as well as drive future work, we will release our scripts, rule-based baselines, source data, and annotation templates, on top of the release of xformal. our results open several avenues for future work in terms of benchmarking and evaluating fost in a more language inclusive direction. notably, current supervised and unsupervised approaches for en fost rely on parallel in-domain data-with the latter treating the parallel set as two unpaired corpora-that are not available in most languages. we suggest that benchmarking fost models in multilingual settings will help understand their ability to generalize and lead to safer conclusions when comparing approaches. at the same time, multilingual fost calls for more language-inclusive consideration for automatic evaluation metrics. modelbased approaches have been recently proposed for evaluating different aspects of st. however, most of them rely heavily on english resources or pretrained models. how those methods can be extended to multilingual settings and how we evaluate their performance remain open questions.