C o C o A - MT : A Dataset and Benchmark for Contrastive Controlled MT with Application to Formality (2022)

Abstract: The machine translation (MT) task is typically formulated as that of returning a single translation for an input segment. However, in many cases, multiple different translations are valid and the appropriate translation may depend on the intended target audience, characteristics of the speaker, or even the relationship between speakers. Specific problems arise when dealing with honorifics, particularly translating from English into languages with formality markers. For example, the sentence “Are you sure?” can be translated in German as “Sind Sie sich sicher?” (formal register) or “Bist du dir sicher?” (informal). Using wrong or inconsistent tone may be perceived as inappropriate or jarring for users of certain cultures and demographics. This work addresses the problem of learning to control target language attributes, in this case formality, from a small amount of labeled contrastive data. We introduce an annotated dataset (CoCoA-MT) and an associated evaluation metric for training and evaluating formality-controlled MT models for six diverse target languages. We show that we can train formality-controlled models by fine-tuning on labeled contrastive data, achieving high accuracy (82% in-domain and 73% out-of-domain) while maintaining overall quality.

## introduction the quality of neural machine translation (nmt) models has been improving over the years and is approaching that of human translation (hassan et al., 2018). with fewer glaring accuracy or fluency errors, it is important to address other aspects of translation quality, such as tone and style, in order to generate context-appropriate translations and improve the end-user experience with mt systems. in particular, for spoken language and certain text domains (customer service, business, gaming chat), problems arise when translating from english into languages that have multiple formality levels expressed through honorifics or grammatical register. taking the example from in this work we look at formality across multiple languages and frame formality control as a transfer learning problem, by leveraging a generic nmt system and a small amount of manually labeled data to obtain mt systems that are controllable for formality. our main contributions are threefold. first, we release a novel multilingual and multidomain benchmark for contrastive controlled mt (cocoa-mt) consisting of contrastive translations with phrase-level annotations of formality and grammatical gender in six diverse language pairs: english (en) → french (fr), german (de), hindi (hi), italian (it), japanese (ja), and spanish (es). second, to accompany the cocoa-mt dataset, we introduce a reference-based automatic metric with high precision at distinguishing formal from informal system hypotheses. third, we propose training formality-controlled models using transfer learning on contrastive labeled data. our method is effective across six language pairs and robust across several datasets. we show that transfer learning using cocoa-mt is complementary to automatically labeled data, while cost-effective compared to non-contrastive curated data. we release the cocoa-mt dataset, together with sockeye 3 foot_0 baseline models and evaluation scripts. @xcite these resources were also available to participants of the iwslt 2022 (anastasopoulos et al., 2022) shared task on formality control for spoken language translation. @xcite 2 cocoa-mt dataset we first introduce cocoa-mt, our contrastive controlled mt by aws ai dataset, which enables evaluation and training of formality-controlled models. ## formality evaluation in this section, we present a manual analysis of formality expressed in the outputs of two generic commercial systems for inputs sampled from cocoa-mt. next, we propose and evaluate a reference- ## transfer learning for formality control we approach formality-controlled nmt as a transfer learning problem, where we fine-tune a generic pre-trained mt model on labeled contrastive translation pairs from the cocoa-mt dataset. for each source segment we create two labeled training data points: one for each contrastive reference translation (formal and informal). we use a special token with a randomly initialized embedding for the formality label which we attach to the beginning of the source segment. to leverage the small amount of labeled data while maintaining the overall quality of the generic pre-trained mt model, we first up-sample the labeled data by concatenating multiple copies. @xcite next, we augment the labeled data with an equal amount of unlabeled data sampled randomly from the generic training set. finally, we fine-tune on the combined labeled and unlabelled data for one epoch with a fixed learning rate following the approach proposed by hasler et al. (2021) for domain adaptation. with this method we train models that can perform both tasks: generic translation and formality-controlled translation. ## experimental setup nmt models into de, es, fr and it were trained on 20m pairs sampled from paracrawl v9 (bañón et al., 2020), using wmt newstest for development. for evaluating generic quality, we used the wmt newstests as well as the must-c data (di gangi et al., 2019). @xcite the en-ja model was trained on all 10m pairs from jparacrawl v2 (morishita et al., 2020) using the iwslt17 development set. for testing we used wmt newstest2020 and the iwslt17 test set. the en-hi model was trained on all 15m pairs from ccmatrix (schwenk et al., 2021), using the wmt newsdev2014 for development and newstest2014 for testing. nmt models were built using the transformerbase architecture (vaswani et al., 2017), but with 20 encoder layers and 2 decoder layers as recommended by domhan et al. (2020) and ssru decoder layers for faster decoding (kim et al., 2019). we report the complete lists of pre-processing and training arguments appendix c. ## results this section evaluates formality-controlled models trained using cocoa-mt as described in section 4. section 6.1 evaluates performance on cocoa-mt test sets, investigating in-domain versus outof-domain performance as well as the effect of up-sampling the labeled data in training. section 6.2 compares the use of the contrastive cocoa-mt data with other sources of labeled data. in section 6.3 we perform additional evaluations on existing (non-contrastive) test sets for which a single formality level is naturally appropriate: forum discussions (informal) and customer support conversations (formal). this is a common scenario, requiring consistent translations that are appropriate for the domain and target audience. ## gender-specific translations while creating the cocoa-mt formalitycontrolled dataset, we observed that for target languages with grammatical gender (all except ja), some reference translations require gender to be expressed in the target even though it is ambiguous in the source. @xcite in fact, this is similar to formality: a grammatical distinction must be made in the target language, even though the source is under-specified with re-source did you play with legos growing up? feminine ¿de pequeña jugaba con piezas de lego? masculine ¿de pequeño jugaba con piezas de lego? effect on gender translation accuracy in section 6, for segments with gendered translations, we selected a single gender (in that case, masculine) to use consistently in all training and evaluation data. here, we perform an initial evaluation of the effect of gender-specific formality-controlled data on gender translation accuracy using winomt (stanovsky et al., 2019) on en-es. @xcite we compare the baseline (without formality control) to separate models trained using masculine (msc-trg; same as in ## related work controlling politeness for nmt was first tackled by sennrich et al. (2016a) for en-de translation. they appended side constraints to the source text to indicate the preference of verbs or t-v pronoun choices (brown and gilman, 1960) in the output. @xcite a similar approach was applied to control the presence of honorific verb forms in en-ja mt by feely et al. (2019). viswanathan et al. (2019) controlled t-v pronoun choices of en-es/fr/czech translations by adapting generic models with t-v distinct data. they collected politeness parallel data using heuristics. in a task of fr-en formality-sensitive mt (niu et al., 2017), translation and en formality transfer were trained jointly in a multi-task setting (niu et al., 2018; niu and carpuat, 2020). they assumed cross-lingual formality parallel data is not available and leveraged monolingual formality data instead (rao and tetreault, 2018). prior work has also investigated control for attributes besides formality: speaker role and gender (mima et al., 1997; rabinovich et al., 2017; elaraby et al., 2018; bentivogli et al., 2020), voice (yamagishi et al., 2016), length/verbosity (takeno et al., 2017; lakew et al., 2021), readability/complexity (stymne et al., 2013; marchisio et al., 2019; agrawal and carpuat, 2019), monotonicity (schioppa et al., 2021), translator traits (wang et al., 2021) and a writer's proficiency level and native language (nadejde and tetreault, 2019). controlling multiple attributes with a single nmt system was investigated by michel and neubig (2018a); schioppa et al. (2021). annotation toolkits or parallel corpora annotated with some of these 22 https://en.wikipedia.org/wiki/t-v_ distinction attributes has also been released, including gender and age (rabinovich et al., 2017; vanmassenhove et al., 2018; bentivogli et al., 2020), complexity (agrawal and carpuat, 2019), and speaker traits (michel and neubig, 2018a). ## conclusions this work addresses the problem of controlling mt output when translating into languages that make formality distinctions through honorifics or grammatical register. to train and evaluate formalitycontrolled mt models, we introduce cocoa-mt -a novel multilingual and multidomain benchmarkand a reference-based automatic metric. our experiments show that formality-controlled mt models can be trained effectively with transfer learning on labeled contrastive translation pairs from cocoa-mt, achieving high targeted accuracy and retaining generic translation quality. we release the cocoa-mt dataset to enable future work on controlling multiple features (formality and grammatical gender) simultaneously. the translations, we obtained approval for our use case from the creators of the existing artifacts. as part of our formality-controlled dataset, we noticed that translations often required the gender of the speaker or the addressee to be specified, even when the english source was gender-neutral. as a result, for each such case, we include grammatically feminine and grammatically masculine reference translations. we hope that this will open up opportunities for future work in avoiding gender bias when controlling for politeness, and even in improving translations by customizing to the user's desired gender, @xcite in a similar way to how we customize for the desired formality in this paper. in creating gender-specific reference translations, we limit the differences to words that are grammatically gendered in the target languages, rather than stereotypical or other differences. it is important to note that while this paper addresses grammatical gender in translation, it does not use human subjects, infer or predict gender, or otherwise use gender as a variable. we would like to emphasize that the work on gender in this paper is very much a work in progress. we provide this dataset as an initial contribution; we will continue to improve on this work and this data, and we hope other groups also use and expand on it. most notably, so far we have only produced translations for two genders. in the future, we plan on expanding the references translations to more genders, in consultation with native speakers of the target languages and other stakeholders. we also would like to analyze gender bias in formality-controlled models, as well as create models that can control for multiple features (e.g., formality and grammatical gender) simultaneously.