Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality Transfer (2021)

Abstract: While the field of style transfer (ST) has been growing rapidly, it has been hampered by a lack of standardized practices for automatic evaluation. In this paper, we evaluate leading automatic metrics on the oft-researched task of formality style transfer. Unlike previous evaluations, which focus solely on English, we expand our focus to Brazilian-Portuguese, French, and Italian, making this work the first multilingual evaluation of metrics in ST. We outline best practices for automatic evaluation in (formality) style transfer and identify several models that correlate well with human judgments and are robust across languages. We hope that this work will help accelerate development in ST, where human evaluation is often challenging to collect.

## introduction textual style transfer (st) is defined as a generation task where a text sequence is paraphrased while controlling one aspect of its style @xcite . for instance, the informal sentence in italian "in bocca al lupo!" (i.e., "good luck") is rewritten to the formal version "ti rivolgo un sincero augurio!" (i.e., "i send you a sincere wish!"). despite the growing attention on st in the nlp literature @xcite , progress is hampered by a lack of standardized and reliable automatic evaluation metrics. standardizing the latter would allow for quicker development of new methods and comparison to prior art without relying on time and cost-intensive human evaluation that is currently employed by more than 70% of st papers @xcite . st is usually evaluated across three dimensions: style transfer (i.e., has the style of the generated output changed as intended?), meaning preservation (i.e., are the semantics of the input preserved?), and fluency (i.e., is the output well-formed?). as we will see, a wide range of automatic evaluation metrics and models has been used to quantify each of these dimensions. for example, prior work has employed as many as nine different automatic systems to rate formality alone (see we address these limitations by conducting a controlled empirical comparison of commonly used automatic evaluation metrics. concretely, for all three evaluation dimensions, we compile a list of different automatic evaluation approaches used in prior st work and study how well they correlate with human judgments. we choose to build on available resources as collecting human judgments across the evaluation dimensions is a costly process that requires recruiting fluent speakers in each language addressed in evaluation. while there are many stylistic transformations in st, we conduct our study through the lens of formality style transfer (fost), which is one of the most popular style dimensions considered by past st work @xcite and for which reference outputs and human judgments are available for four languages: english, brazilian-portuguese, french, and italian. ## evaluating evaluation metrics we evaluate evaluation metrics (described in §3.2) for multilingual fost, in four languages for which human evaluation judgments (described in §3.1) on fost system outputs are available. ## experimental results we analyze the results of comparing the outputs from the several automatic metrics to their humangenerated counterparts for formality style transfer ( §5.1), meaning preservation ( §5.2), fluency ( §5.3) via conducting segment-level analysis-and then, turn into analyzing system-level rankings to evaluation overall task success ( §5.4). ## conclusions automatic (and human) evaluation processes are well-known problems for the field of natural language generation @xcite and the burgeoning subfield of st is not immune. st, in particular, has suffered from a lack of standardization of automatic metrics, a lack of agreement between human judgments and automatics metrics, as well as a blindspot to developing metrics for languages other than english. we address these issues by conducting the first controlled multilingual evaluation for leading st metrics with a focus on formality, covering metrics for 3 evaluation dimensions and overall ranking for 4 languages. given our findings, we recommend the formality style transfer community adopt the following best practices: