Towards Better Evaluation for Formality-Controlled E nglish- J apanese Machine Translation (2023)

Abstract: In this paper we propose a novel approach to automatically classify the level of formality in Japanese text, using three categories (formal, polite, and informal). We introduce a new dataset that combine manually-annotated sentences from existing resources, and formal sentences scrapped from the website of the House of Representatives and the House of Councilors of Japan. Based on our data, we propose a Transformer-based classification model for Japanese, which obtains state-of-the-art results in benchmark datasets. We further propose to utilize our classifier to study the effectiveness of prompting techniques for controlling the formality level of machine translation (MT) using Large Language Models (LLM). Our experimental setting includes a large selection of such models and is based on an En->Ja parallel corpus specifically designed to test formality control in MT. Our results validate the robustness and effectiveness of our proposed approach and while also providing empirical evidence suggesting that prompting LLMs is a viable approach to control the formality level of En->Ja MT using LLMs.

## introduction communication by way of natural language often includes indicators for respect to acknowledge the hierarchy, interpersonal relationship, and power dynamics of the participants in a conversation or written text. in this context, formality or honorifics refers to the set of linguistic features used to establish the degree of respect and deference conveyed in a given context. naturally, these phenomena exhibit significant variation across different languages and cultures @xcite . while many european languages emphasize formality through the use of standard grammar, more complicated sentence structures (active, passive, use of clauses, etc.), or more advanced and complex choice of vocabulary and phrases, the japanese language has its own formality system. this system, named keigo (敬語), requires users to identify the status or the relationship with the interlocutor, is strict, following a standard grammar format @xcite , and can generally be divided into four different categories @xcite , as follows. in this context, what makes japanese formality stand out is that it allows to convert any sentences from one style to another by simply adjusting the tense of the verb @xcite , while maintaining the original meaning, word choice, and sentence structure. additionally, the system follows one additional rule @xcite , where one can always mix the four forms together in one paragraph. the more respectful form one uses in a sentence or a paragraph, the more courtesy one states toward one's interlocutor. similarly, the more humble form one uses, the more modest one is in the conversation. however, it is also emphasized that when containing too many formal terms in a sentence, the sentence will become annoying and considered inappropriate in japanese social rules @xcite . given the importance of formality in language generation systems such as machine translation (mt), the ability to control formality and honorifics is a critical factor in achieving accurate and appropriate results. in particular, for the japanese language, failure in recognizing and incorporating levels of formality can result in unnatural, impolite, or disrespectful translations, which can impede effective communication across diverse linguistic and cultural contexts @xcite . we therefore think that developing and refining mt models that can accurately control honorific levels is crucial for this language. although formality-controlled machine translation (fcmt) has gained popularity for languages like english @xcite , there is a substantial lack of resources to tackle the formality problem for japanese, which extends to the more fundamental task of formality detection. in light of this issue, we focus on developing resources to improve formality detection in japanese. we begin by uncovering several flaws on existing corpora for the task, including issues such as the presence of ungrammatical sentences, as well as wrong formality labels. to alleviate these issues, we introduce new resources for japanese formality detection which consists of manually-labeled sentences annotated with three formality classes (informal, polite, and formal). we propose this three-way setting in opposition to existing resources which are annotated using binary labels, to better approximate the nature of formality of the japanese language. as existing resources @xcite lacked data for the formal label, a part of our dataset is constructed with sentences sampled from these sources and with text obtained from meeting minutes from committees of the house of representatives and the house of councilors of japan 1 . furthermore, as language generation models based on large language models (llms) have recently been able to attain substantial performance improvements on language generation benchmarks, we note that the lack of a consistent evaluation 1 https://kokkai.ndl.go.jp/ method makes it difficult to verify to what extent such models can perform formality control. in mt, current studies mainly rely on human assessment or simple models. for example, @xcite and @xcite use rule-based methods where lists of grammatical rules are combined with pattern-matching to perform classification. though formality-level classifiers for japanese based on machine learning have been proposed in the past @xcite , so far this has been without focus on mt or lacked proper evaluation. in consideration of the above issue, in this paper we propose a novel approach, based on machine learning, to evaluate the ability of @xmath0. concretely, we use our dataset to train a robust transformer-based classifier that leverages a masked-language model, which is able to obtain state-of-the-art performance on our dataset and on existing japanese formality detection benchmarks. following recent work relying on machine learning models to evaluate language generation, such as bertscore @xcite , and mt models, such as comet @xcite , we present an empirical study using our classifier to evaluate the zero-shot ability of several state-of-the-art llms to perform formality control. our results validate the effectiveness of our proposed approach and show that, compared to existing evaluation techniques that rely on rules and expression-matching, it offers a robust, reliable, and accurate evaluation metric for formalitycontrolled mt systems. we further demonstrate the ability of llms to generate sequences with varying levels of formality through the use of well-designed prompts, concretely showing that both gpt-3 and chatgpt can attain a formality control accuracy of approximately 90%, and ultimately suggesting that prompting llms can result in better formality control performance than fine-tuned mt models. we release or data and trained models 2 to encourage further research on this topic. dataset, which was released for the 2022 shared task on formality control at iwslt @xcite and contains a total of 1,600 parallel english-japanese sentences (1,000 for training, and 600 for testing). the source data for this corpus comes from topical-chat4 @xcite , as well as telephony and call center data, containing text-based conversations about various topics. for each segment, one reference translation for each formality level (formal and informal) were collected. for the japanese translations, informal was mapped to jyoutai, and formal was mapped to teineigo, sonkeigo and/or kenjyougo. on the other hand, we find the recently-released keico corpus @xcite , which has a total of 10,007 examples across the four forms of the japanese formality system (levels 1 to 4, according to the paper). it additionally contains detailed information about the presence of levelrelated honorifics -a sentence may contain markers for multiple levels of politeness-the social relationship between the speaker and the listener, and conversational situations or topics. to obtain this data, 40 native japanese volunteers were asked to regenerate a total of 3,000 sentences coming from machine translation, dialogue systems, and semantic analysis systems, by filling in blanks with honorifics. the two datasets mentioned above have been used to train transformer-based classifiers. @xcite rely on japanese-bert @xcite , while the submission of @xcite for the 2022 shared task on formality control at iwslt relied on xlm-r @xcite . our work is also related to fcmt. in this context, recent approaches have relied on formalityannotated parallel corpora such as cocoa-mt, early work on this task resorted to other resources such as rule-based generation of synthetic data for english-japanese @xcite and english-german @xcite , as well as synthetic supervision by means of multi-tasking (formality classification and machine translation). we also find that these studies rely on rule-based simple approaches to measure the accuracy of formality control in the translation, or directly perform human assessment. for example, the fsmt approach english-french by @xcite conducted a human study in which they assigned translation pairs for human annotators. neural cfmt mod-els for english-japanese @xcite and english-german @xcite depend on rule-based classifiers, where grammatical rules for the language are listed and matched. the recent rise of llms has enabled models to perform certain language generation tasks in zeroshot or few-shot manner @xcite , or by means of prompts. some of these capabilities have been further enhanced by means of prompt-based training @xcite , where zero-shot generalization is induced by explicit multitask learning. this work is relevant to our paper, as we test the ability of several such models to perform zero-shot fcmt. our study also considers multilingual efforts in neural mt, admittedly also a kind of llm, where we look at m2m100 @xcite and nllb200 @xcite finally, we also find recent work on using fewshot prompting-based techniques to control the formality level of english-german machine translation @xcite . also, @xcite recently performed an in-depth study of the capabilities of chatgpt to generate text in different styles, including formal/informal labels, showing that the model sometimes incorporates factual errors or hallucinations when adapting the text to suit a specific style. ## empirical study having demonstrated the abilities of our transformer-based classifier of japanese formality, we now turn to a more practical issue, and tasks ourselves with testing the proposed approach in a real scenario. we examine formality abilities of english to japanese machine translation using a zero-shot prompting approach. to the best of our knowledge, our work is the first one to study this issue. ## conclusions this paper explores new alternatives to evaluate the ability of @xmath1, proposing classifiers based on rule-based methods and a machine learning approach using huggingface transformers foot_4 @xcite . to build robust models, we focus on developing resources to improve formality detection in japanese, uncovering several flaws on existing corpora for the task, and introducing new annotated datasets. in contrast to prior work approaching formality using binary labels, we use three classes (informal, polite, and formal) to better approximate the ways honorifics are used in the japanese language. extensive experiments on benchmark datasets show that our proposed models offer stateof-the-art performance. finally, we empirically show that our machinelearning approach is superior to existing evaluation techniques for formality-controlled mt systems, offering a reliable and accurate evaluation solution. the study also demonstrates the ability of llms to generate sequences with varying levels of formality through well-designed prompts, resulting in stateof-the-art results in @xmath2. our findings provide a valuable contribution to the nlp field by presenting a new approach to evaluate formality-controlled mt systems and highlighting the effectiveness of llms in this task.