Human Judgement as a Compass to Navigate Automatic Metrics for Formality Transfer (2022)

Abstract: Although text style transfer has witnessed rapid development in recent years, there is as yet no established standard for evaluation, which is performed using several automatic metrics, lacking the possibility of always resorting to human judgement. We focus on the task of formality transfer, and on the three aspects that are usually evaluated: style strength, content preservation, and fluency. To cast light on how such aspects are assessed by common and new metrics, we run a human-based evaluation and perform a rich correlation analysis. We are then able to offer some recommendations on the use of such metrics in formality transfer, also with an eye to their generalisability (or not) to related tasks.

## introduction text style transfer (tst) is the task of automatically changing the style of a given text while preserving its style-independent content, or theme. quite different tasks, and thus quite different types of transformations, traditionally fall under the tst label. for example, given the sentence "i like this screen, it's just the right size...", we may produce its negative counterpart "i hate this screen, it is not the right size" for the task defined as polarity swap @xcite , or turn it into the formal "i like this screen, it is just the right size." for the task called formality transfer @xcite . for the transfer to be considered successful, the output must be written (i) in the appropriate target style; (ii) in a way such that the original content, or theme, is preserved; and (iii) in proper language, hence fluent and grammatical (relative to the desired style). these aspects to be evaluated are usually defined as (i) style strength, (ii) content preservation, and (iii) fluency, and automatic 1 our analysis code, literature list for figure 1 , and all data are available at https://github.com/laihuiyuan/ eval-formality-transfer . evaluation metrics are used accordingly, lacking the possibility of using human judgement for any given experiment. figure 1 shows a survey of such metrics (organised by aspect) as used in 45 papers published over the last three years in the acl anthology, which focus on tst in general. a classifier or a regressor is used to assess style strength, a variety of content-based metrics target content preservation, perplexity is used to measure fluency, and some overall metrics combining content and style are often reported. ## data we use gyafc @xcite , a formality transfer dataset for english that contains aligned formal and informal sentences from two domains: entertainment & music and family & relationships. figure 2 shows an example for alignment, transformation, and evaluation relations be- reference: now we do so many things together and i do not know what to do. source: but now we do all these things together and i dont know what to do.. output: now we do all these things together and i do not know what to do. sentence-pair tween input, output, and reference. we run a human evaluation and a battery of automatic metrics on a selection of human-and machine-produced texts. source and reference texts the source and reference texts we use are from the family & relationships domain. the test set contains 1,332 and 1,019 sentences in "informal to formal" and "formal to informal" directions, respectively. there are four human references for each test sentence. we randomly select 80 source sentences (40 for each transfer direction) from the test set, as well as their corresponding human references. for each source sentence, we obtain the corresponding transformations as produced by eight different systems. ## methodologies 4.1 human evaluation to facilitate the annotation and obtain a manageable size for each annotator, we split the 80 source sentences (section 3) into four different surveys with 20 sentences each (10 for each transfer direction), and their corresponding system outputs plus one reference. we recruited eight highly proficient english speakers for this task, i.e. two per survey, so that two annotations for each target sentence can be collected; from these we can use the average score 3 details of the systems are in appendix a.1. assigned, and also calculate inter-annotator agreement. the task is to rate the transferred sentence on a continuous scale (0-100), inspired by direct assessment @xcite , in terms of three evaluation aspects: (i) style strength (does the transformed sentence fit the target style?); (ii) content preservation (is the content of the transformed sentence the same as the original sentence?), and (iii) fluency (considering the target style, could the transformed sentence have been written by a native speaker?). before starting the rating task, we provided annotators with detailed guidelines and examples of transformed sentences along with plausible assessments for each aspect. 4 we also reminded the annotators that such examples are only indicative of what we believe to be plausible judgements but there are many possible correct answers, of course. ## results and analysis in this section, we first measure the inter-annotator agreement of the human evaluation, then discuss both system-level and sentence-level evaluation results on the three aforementioned evaluation aspects, so as to provide a different perspective on the correlation between automatic metrics and human judgements under different conditions. ## conclusion we have considered a wide range of automatic metrics on the three evaluation aspects of formality transfer, and assessed them against human judgements that we have elicited. for style strength, we have compared the style classifiers and regressor in the setting of using the same raw data for training (with a binary label for classification and continuous scores for regression), as well as classifiers with different performances. we have observed that there is little difference among them when evaluating multiple tst systems. however, the style regressor performs worse when evaluating high-quality tst systems. for classifiers with different performances, we recommend the one with the highest performance since it results in the highest overall pearson correlation with human judgements. to assess content preservation, we have explored different kinds of automatic metrics using the source or reference(s), and have observed the follwoing: (i) if using the source sentence, we strongly recommend employing learnable metrics since their correlation in that condition is much higher than those of traditional surface-based metrics (which are not indicative, since high scores correspond to not changing the input, hence not performing the task); still, the question of how scores should be interpreted and what score ranges are desirable remains open; (ii) most metrics are reliable to be used to measure and compare the performances at system-level when a human reference is available; (iii) however, we do not recommend to use surface-base metrics to measure sentence-level comparisons, especially with only one reference. overall, learnable metrics seem to provide a more reliable measurement. for fluency, perplexity can be used for evaluating the informal-to-formal direction, either at system-or segment-level, while it is clearly less reliable for the opposite direction, and it remains to be investigated how to best perform evaluation in this transfer direction, considering the wide variability of acceptable outputs. this study focuses on formality transfer, and offers a better understanding of automatic evaluation thanks to the comprehensive correlations with human judgments herein conducted. however, the findings may not generalise to other tasks usually considered similar, such as polarity swap. to this end, future dedicated work will be required.