A Call for Standardization and Validation of Text Style Transfer Evaluation (2023)

Abstract: Text Style Transfer (TST) evaluation is, in practice, inconsistent. Therefore, we conduct a meta-analysis on human and automated TST evaluation and experimentation that thoroughly examines existing literature in the field. The meta-analysis reveals a substantial standardization gap in human and automated evaluation. In addition, we also find a validation gap: only few automated metrics have been validated using human experiments. To this end, we thoroughly scrutinize both the standardization and validation gap and reveal the resulting pitfalls. This work also paves the way to close the standardization and validation gap in TST evaluation by calling out requirements to be met by future research.

## introduction text style transfer (tst) is the task of transferring text from one style to another. examples include sentiment transfer (making a negative text more positive) @xcite , politeness transfer @xcite , formality transfer @xcite , and many more. anyone working and communicating with texts has most likely manually performed many of these transfer tasks countless times. whether generalized @xcite or task-specific models, evaluating their performance on tst is crucial to measure progress in task-specific text generation. in the last few years, there has been a surge in research on tst (see figure 1 , appendix), the majority of which was in the last two years. when a field develops that quickly, the development of evaluation procedures often does not keep up. tst research is now at a point where reflection on evaluation practices is becoming increasingly urgent, as progress in this field is becoming difficult if the performance of new methods cannot be reliably compared to existing methods. it can even become hard to determine the state of the art one should aim to improve. however, recent tst surveys @xcite @xcite focus on summarizing tst methods and their evaluation but do not point out evaluation and experimentation issues. works on tst evaluation either focus on getting more reliable metrics for a particular type of tst @xcite , focus on a specific evaluation aspect for validation @xcite , or emphasize only shortcomings of automated evaluation in a multilingual setting of one particular type of tst @xcite . as a first step, it is, therefore, necessary to examine evaluation practices and experimental setups to point out the need for standardization of both, which is what we set out to do in this paper. early tst works @xcite focus on human evaluation, which is still considered the most reliable @xcite . many later publications rely only on automated metrics, which can be validated by correlating them with human evaluations. so far, no comprehensive overview exists showing which metrics are validated and how leading to researchers using popular metrics rather than those with the highest correlations. in our meta-analysis, we counter-act this by reporting which metrics are validated and analyzing cases where the same metric received different validation results. to sum up, our contributions are: we conclude with a list of requirements to help close these two gaps. ## a meta-analysis of text style transfer in this meta-analysis, we focus on tst publications in top-tier nlp and ai venues (see appendix a.2 for selection details), resulting in 89 considered papers summarized in ## automated metrics and their (missing) validation in this section, we summarize existing automated metrics and their validity in terms of correlation with human evaluations. in ## conclusion & future work our research emphasizes the pressing need for standardization and validation in tst. while human evaluation is still considered to be the most reliable, it is expensive and time-consuming, hindering reproducibility. many publications use automated metrics as surrogates, but only a fraction of them is validated. furthermore, human and automated evaluations and experimental setups lack standardization, making it difficult to compare the performance of different models. summarizing our results, we pose the following six requirements to be met by future tst research: