FRMT : A Benchmark for Few-Shot Region-Aware Machine Translation (2023)

Abstract: We present FRMT, a new dataset and evaluation benchmark for Few-shot Region-aware Machine Translation, a type of style-targeted translation. The dataset consists of professional translations from English into two regional variants each of Portuguese and Mandarin Chinese. Source documents are selected to enable detailed analysis of phenomena of interest, including lexically distinct terms and distractor terms. We explore automatic evaluation metrics for FRMT and validate their correlation with expert human evaluation across both region-matched and mismatched rating scenarios. Finally, we present a number of baseline models for this task, and offer guidelines for how researchers can train, evaluate, and compare their own models. Our dataset and evaluation code are publicly available: https://bit.ly/frmt-task .

## introduction machine translation (mt) has made rapid advances in recent years, achieving impressive performance for many language pairs, especially those with high amounts of parallel data available. although the mt task is typically specified at the coarse level of a language (e.g., spanish or hindi), some prior work has explored finer-grained distinctions, such as between regional varieties of arabic @xcite , or specific levels of politeness in german @xcite . unfortunately, most approaches to style-targeted translation thus far rely on large, labeled training corpora @xcite @xcite @xcite @xcite , and in many cases these resources are unavailable or expensive to create. ## related work textual style transfer aims to control fine-grained stylistic features of generated text. earlier work figure 1 : frmt requires a machine translation model to adapt its output to be appropriate for a specific region, such as brazil (left) or portugal (right). because only a few exemplars are provided to convey the target region, methods that perform well on frmt can likely extend to other regions and styles. previous work has explored many sub-types of variety-targeted mt. region-aware mt targets specific regions or dialects @xcite @xcite @xcite @xcite ; formality-aware mt targets different formality levels @xcite @xcite ; and personalized mt aims to match an individual's specific style @xcite . however, with few exceptions (e.g., @xcite , these works assume the availability of large-scale datasets containing examples with the target varieties explicitly labeled. in the present work, we design a benchmark that emphasizes few-shot adaptability. although our dataset is limited to four regions and two languages, the few-shot setup and high degree of linguistic dissimilarity between the selected languages means that approaches performing well on the entire frmt benchmark can be expected to generalize reasonably well to other languages, other regions, and other stylistic attributes. several existing parallel corpora cover regional language varieties, but have limitations that motivate us to construct a new high-quality, targeted dataset. e-pact @xcite comprises translations from english books into portuguese variants, but is small and not easily accessible. opensubtitles @xcite skews toward shorter utterances and is noisy due to automatic alignment. wit3 @xcite provides translations of ted-talk transcripts into many languages, but relies on volunteer translators, which may limit quality. popular shared tasks have not included regiontargeted translation either: the conference on machine translation (wmt) has included translation between similar languages (e.g., @xcite , while the workshop on nlp for similar languages, varieties and dialects (vardial) focuses mainly on classification and not translation (e.g., @xcite . furthermore, we are not aware of previous work that (1) measures deltas in human evaluation metrics between the region-matched and regionmismatched settings, (2) correlates these with automated metrics, (3) offers tailored sub-tasks targeting region-differentiated lexical items and region-biased distractors, or (4) defines targeted metrics testing region-appropriateness. ## frmt dataset we introduce the frmt dataset for evaluating the quality of few-shot region-aware machine translation. the dataset covers two regions each for portuguese (brazil and portugal) and mandarin (mainland and taiwan). these languages and varieties were selected for multiple reasons: ## evaluation metrics while human judgments are the gold standard for evaluating machine-generated texts, collecting them can be time-consuming and expensive. for faster iteration, it can be helpful to measure progress against automatic metrics that are known to correlate well with human judgments. we hypothesize that common reference-based mt evaluation metrics might have differing sensitivities to regional differences, and so we conduct a human evaluation of several baseline models (see §6.1) and compute correlation of several automatic metrics with the human judgments. we also propose a new automated lexical accuracy metric that more directly targets region-awareness. ## baseline models we evaluate a handful of academic mt models that claim some ability to provide few-shot controllable translations. we also evaluate a commercial mt system that does not distinguish between these regional varieties. our first baseline is the universal rewriter (ur) of @xcite , which supports multilingual style transfer and translation. it is initialized from an mt5-xl checkpoint @xcite and finetuned on a combination of monolingual and parallel data from mc4 and opus, respectively. we train it with sequence length of 128 instead of 64, to be directly comparable to our other models. our third baseline uses natural language prompting to control the regional variety of m4's output (m4-prompts), such as prefixing the input with ''a brazilian would write it like this:''. this is motivated by earlier work using this technique effectively for large language models @xcite @xcite , and more recent work applying it to region-aware mt @xcite . our fourth baseline finetunes the m4-prompts model, where the source-side language tags used to induce the target language are replaced with prompts of the form ''translate to [language]:''. this model (m4-prompts ft) is designed to explicitly introduce prompting behavior. at inference time, we replace ''[language]'' with the variety name (e.g., ''brazilian portuguese''). neither m4-prompts nor m4-prompts ft use exemplars. our next three baselines are different-sized versions of palm @xcite , a large language model that has demonstrated remarkable zero-shot and few-shot performance on a variety of tasks (palm 540b, palm 62b, and palm 8b, referring to their approximate parameter counts). the prompt for these models begins with ''translate the following texts from english to [language variety]'' and is followed by ten exemplars selected randomly from the lexical bucket. 9 each exemplar is put on two lines: first the english text, prefixed by ''english:'', and then the translation in the target variety, prefixed by the variety's name. at the end of the prompt, we show the model the input text and the language variety prefix, and take the first decoded line of text. finally, we examine google translate, foot_9 a publicly available commercial mt model that does not support regional varieties for portuguese or mandarin (though it does support conversion between traditional and simplified scripts). we evaluate this system mainly to test the hypothesis that variety-agnostic systems will be biased toward the web-majority variety. ## conclusion in this paper, we introduced frmt, a new benchmark for evaluating few-shot region-aware machine translation. our dataset covers 4 regions of portuguese and mandarin, and enables finegrained comparison across region-matched and mismatched conditions, and across different classes of inputs (lexical, entity, random). while we found the large-scale generalist model palm 540b to show impressive few-shot region control, there is still significant room for improvement. none of the models we evaluated match human performance, and the gap is particularly large in mandarin. additionally, there remains an open research question as to whether robust few-shot regional control can be achieved at more modest model scales. we are eager to see progress on frmt, as methods that do well in this few-shot setting are likely to be easily extensible to other regions and styles. we anticipate that the flexibility to adapt to new output styles in the absence of extensive labeled data will be a key factor in making generative text models more useful, inclusive, and equitable.