Evaluating Text Style Transfer Evaluation: Are There Any Reliable Metrics? (2025)

Abstract: Text style transfer (TST) is the task of transforming a text to reflect a particular style while preserving its original content. Evaluating TSToutputs is a multidimensional challenge, requiring the assessment of style transfer accuracy, content preservation, and naturalness. Us-ing human evaluation is ideal but costly, as is common in other natural language processing (NLP) tasks; however, automatic metrics forTST have not received as much attention as metrics for, e.g., machine translation or summarization. In this paper, we examine both set ofexisting and novel metrics from broader NLP tasks for TST evaluation, focusing on two popular subtasks—sentiment transfer and detoxification—in a multilingual context comprising English, Hindi, and Bengali. By conducting meta-evaluation through correlation with hu-man judgments, we demonstrate the effectiveness of these metrics when used individually and in ensembles. Additionally, we investigatethe potential of large language models (LLMs) as tools for TST evaluation. Our findings highlight newly applied advanced NLP metrics andLLM-based evaluations provide better insights than existing TST metrics. Our oracle ensemble approaches show even more potential.

## introduction text style transfer (tst) refers to the task of modifying a given text to reflect a specific style while preserving its original content @xcite . previous work in this domain has explored altering various stylistic dimensions, such as sentiment @xcite , romantic tone @xcite , politeness @xcite , or political slant @xcite . different modeling approaches have been proposed for tst, including methods that manipulate latent representations of text @xcite and techniques that identify and replace stylerelated lexicons directly @xcite . despite the growing interest in tst, reliably assessing the performance of tst models continues to be a bottleneck @xcite . while human evaluation is often regarded as the standard for capturing subtle cues in style, it is expensive, time-intensive, and difficult to reproduce at scale @xcite . consequently, automated metrics have become a proxy for human judgment, but there is a notable lack of standardization and consensus on which metrics best capture style transfer accuracy, content preservation, and overall naturalness @xcite . in addition, large language models (llms) could serve as alternatives to traditional human evaluation and automated metrics for tst evaluation @xcite . however, the rapid evolution of llms, particularly for closed-source models, raises concerns about reproducibility @xcite . we address this gap by examining existing and novel metrics for two popular tst subtasks: sentiment transfer @xcite and detoxification @xcite . our experiments span a multilingual setting, covering english, hindi, and bengali, to investigate the utility of these metrics across diverse linguistic contexts. we then conduct a meta-evaluation of the proposed metrics by measuring their correlation with human judgments. to further explore the potential of automated metrics, we also combine them in ensembles, experimentally creating hybrid scores. additionally, we investigate the applicability of large language models (llms) as an alternative evaluation tool. our results show that existing metrics newly applied to tst, hybrid approaches, and llms can improve correlation with human evaluations, offering a more robust and comprehensive assessment of tst outputs. our experimental code and resources are released on github. 1 2 related work tst tasks are traditionally evaluated using three key dimensions: style transfer accuracy, content preservation, and fluency @xcite @xcite . prior work underscores the challenge of jointly capturing subtle stylistic nuances and preserving semantic content @xcite . style transfer accuracy a common approach is to train a dedicated classifier to check whether the transformed text reflects the intended style @xcite . alternatively, unsupervised methods rely on distributional shifts in style-related features @xcite . content preservation metrics such as bleu @xcite and embedding-based similarity @xcite often serve as proxies for semantic fidelity. however, they may overlook nuances introduced by stylistic transformations in both single-language and multilingual contexts @xcite , and recent studies highlight the shortcomings of traditional similarity measures when evaluating paraphrase-like modifications @xcite . fluency fluency is typically estimated using perplexity from a pre-trained language model such as gpt-2 @xcite . nonetheless, perplexity may fail to capture context-specific grammatical coherence, especially if the style domain diverges from the model's training data @xcite , and can yield inconsistent performance across languages @xcite . ## metrics compared we follow the criteria of transfer accuracy, content preservation, and fluency described in section 2, and we conduct evaluations in two scenarios: newly applied text metrics we expand the tst evaluation by incorporating additional metrics from related nlp tasks, categorizing them into trainable and non-trainable metrics as well as wordoverlap-based and embedding-based measures. for style transfer accuracy, we utilize nontrainable statistical measures such as earth mover's distance (emd) @xcite , kl divergence @xcite , cosine similarity @xcite , and jensen-shannon divergence @xcite , which quantify the distributional shift between source and generated text. additionally, we incorporate a trainable classifier confidence score, derived from the sentence accuracy classifier described earlier. for content preservation, we include both wordoverlap-based and embedding-based metrics. the word-overlap-based metrics include pinc @xcite , which measures the proportion of n-grams in the generated text that do not appear in the source text (higher values indicate greater lexical divergence), meteor (banerjee and lavie, 2005), which accounts for synonymy and stemming, and translation edit rate (ter) @xcite , which evaluates the number of edits required to transform the generated text into the reference. embedding-based measures include word mover's distance (wmd) @xcite , bertscore @xcite , s 3 bert @xcite , and bleurt @xcite , all of which assess content similarity based on contextualized vector representations. additionally, we introduce tree edit distance (ted) @xcite , which measures structural similarity by computing the minimum number of tree edit operations (insertion, deletion, substitution) required to transform one syntactic tree into another. this metric is particularly useful in evaluating syntactic shifts in generated text. for fluency evaluation, we employ language model perplexity, using finetuned gpt-2 and finetuned mgpt trained on target styles (see finetuning details in appendix c). lower perplexity scores indicate higher fluency, as they reflect the model's confidence in the generated text. ## experiment setup evaluation data: tasks, languages and model outputs we evaluate our methods on the outputs of tst models and human annotations provided by @xcite . this comprises two tst tasks -sentiment transfer (positive to negative statements and vice versa), where data is available for english, hindi and bengali, and detoxification (toxic to clean text), with english and hindi data. model outputs for all tasks were produced by gpt-3.5 (openai, 2023), llama-2-7b-chat @xcite and mistral-7b-instruct @xcite , as well as previous finetuned bart models by @xcite . ## results analysis since we found that reference-based metrics generally underperform their reference-free variants, we focus on the reference-free setting in the analysis. we include reference-based results in appendix b. ## conclusion we presented a comprehensive evaluation of existing and newly proposed metrics for two tst subtasks-sentiment transfer and text detoxification-in english, hindi, and bengali. our findings demonstrate that traditional word-overlap-based metrics like bleu and rouge often show limited correlation with human judgments, whereas our proposed experimental metrics and prompted llmbased evaluations provide significantly stronger alignment. moreover, our oracle hybrid ensemble and combined approaches show an even greater potential of merging multiple metrics.