Mind the Style Gap: Meta-Evaluation of Style and Attribute Transfer Metrics (2025)

Abstract: Large language models (LLMs) make it easy to rewrite a text in any style – e.g. to make it more polite, persuasive, or more positive – but evaluation thereof is not straightforward. A challenge lies in measuring content preservation: that content not attributable to style change is retained. This paper presents a large meta-evaluation of metrics for evaluating style and attribute transfer, focusing on content preservation. We find that meta-evaluation studies on existing datasets lead to misleading conclusions about the suitability of metrics for content preservation. Widely used metrics show a high correlation with human judgments despite being deemed unsuitable for the task – because they do not abstract from style changes when evaluating content preservation. We show that the overly high correlations with human judgment stem from the nature of the test data. To address this issue, we introduce a new, challenging test set specifically designed for evaluating content preservation metrics for style transfer. We construct the data by creating high variation in the content preservation. Using this dataset, we demonstrate that suitable metrics for content preservation for style transfer indeed are style-aware.To support efficient evaluation, we propose a new style-aware method that utilises small language models, obtaining a higher alignment with human judgements than prompting a model of a similar size as an autorater.

## introduction large language models allow for rewriting text in a variety of styles or alter any text attribute, without the need for training data. examples are to make a text more formal, polite, simpler, or to change its sentiment. however, evaluating style and attribute transfer is still challenging as it lacks validation and standardization @xcite . traditionally, style and attribute transfer are evaluated via: 1) style strength/shift; 2) fluency and 3) content preservation @xcite . style transfer papers heavily rely on automatic evaluation metrics @xcite . to assess which metrics are most suitable, meta-evaluation efforts measure the correlation of metrics with human judgments. measuring content preservation is particularly difficult for style and attribute transfer, as this requires the distinction between content preservation and changes to style or attributes. still, the status quo is to use metrics not designed for the task of style transfer (most commonly bleu, @xcite ; @xcite ), which assess the lexical or semantic similarity of rewrites to the source sentence, not considering style change. moreover, prior work criticises the use of similarity-based metrics for content preservation @xcite @xcite . intuitively, measuring the similarity between source and output is not suitable for the style transfer task, because the more the style or attribute in the output changes, the more dissimilar the sentences become. hence, metrics for content preservation should be style-aware @xcite . empirically, however, similarity-based metrics show good correlation with human judgments. in fact, approaches using llms only show comparable, and not superior, results to those of semantic similarity metrics @xcite . however, this is even though the llm-based approaches are a better fit for the style transfer task, as they can be style-aware, as opposed to the similarity-based metrics. fig. 1 illustrates a case where similarity-based metrics between source and output fail to correctly identify which output best preserves content, as opposed to style-aware approaches (the last two). in this paper, we study the research question: why do metrics that are theoretically unsuitable for content preservation in style transfer show a high correlation with human judgments? as illustrated in the first two columns in fig. 2 . to answer this question, we conduct a largescale meta-evaluation study with a particular focus on metrics for content preservation. our key finding is that current meta-evaluation efforts for content preservation have misleading conclusions and overestimate the performance of similarity-based metrics. consequently, the use of such metrics leads to misleading evaluation results of style transfer methods. we identify the data used for meta-evaluation as the primary reason for the unreliable correlation results. to overcome this data bias, we propose and construct a test set that offers carefully designed test cases -featuring errors such as substituted, deleted and fabricated information, which are unrelated to style, but change the content (see figure 1 ). our aggregated results (figure 2 ) demonstrate that similarity-based metrics for content preservation show low to negative correlations with human judgment on our test set (last column), but exhibit overly high correlations to human judgment on existing data sets (first two columns). style-aware approaches show positive correlations to human judgments on all data types (all columns). note, our dataset is designed to stress-test the metrics, and we therefore recommend assessing all data types for evaluating the metrics. we propose the style-aware method logprob to evaluate style strength and content preservation using estimated token probabilities. logprob offers better performance on smaller language models (1b,3b) than a strong baseline of using similarsized llms as autoraters. in sum, our contributions are: style and attribute transfer. we use the working definition from @xcite of style transfer as a generation task aiming to control specific attributes of the text. note that this definition of style transfer is data-driven and different from a linguistic definition. we likewise note that 'attribute' is a broader category that covers changes that affect semantics, where a linguistic understanding of 'style' would not, e.g., constitute a change of sentiment. hence, the aim is to change the attribute/style while preserving the contextual or semantic content. evaluation. traditionally, style transfer systems have been evaluated along three dimensions: 1) style strength -did the style successfully shift?; 2) content preservation -is the content otherwise the same?; 3) fluency -is the rewrite fluent and grammatically correct? @xcite ). 2 some work on style transfer systems includes small-scale human evaluations of the systems, but papers heavily rely on automatic metrics for evaluation @xcite : style. the de facto evaluation approaches, also in recent work, use either a classifier, e.g. roberta, trained on a specific style @xcite @xcite @xcite @xcite , or a regressor @xcite @xcite . however, both require training data for the specific style, limiting their scope and scale. further, @xcite criticises the use of binary classifiers, as this approach does not capture that a style can shift to differing degrees. in this paper, we focus on evaluating arbitrary styles without trained classifiers or regressors, and we focus on approaches that output scores to assess different degrees of style changes. ## methodology: test set we propose a new dataset to test metrics on content preservation used in style transfer, inspired by several shortcomings we observe in existing meta-evaluation studies. namely, meta-evaluation studies (both prior studies described in section 2 and our study in section 6), show e.g. that similarity-based metrics highly correlate with human judgments and even outperform llm-based approaches. however, we hypothesise that this is a misleading result caused by shortcomings in how the meta-evaluations are conducted. the reason is that conceptually, similarity-based metrics for content preservation between source and output sentences are unsuitable for the task, because they favour verbatim repetitions between source and output sentences, and the more the style changes, the more the similarity drops despite the content being preserved (section 2). we illustrate the shortcomings of similaritybased metrics with an example (fig. 1 ). suppose the task is to rewrite i don't appreciate your lack of effort to be more polite. consider these rewrites: we aim to create a challenging test set for metaevaluating metrics for content preservation. our core principle is to create data that succeeds in style transfer but has a high variation in content preservation. we create variation in content preservation by deliberately creating mistakes in the content, such that we can assess the metrics' suitability to detect low content preservation. we supply the data with human judgments for both style strength and content preservation. for each source sentence, we construct two rewrites: we aim to construct data where all output sentences succeed on style transfer, but for each source sentence, only one output preserves the content well, creating variation in content preservation quality. other examples from our test set include the task of rewriting the following sentence to be detoxified; yeah but she is making me feel like shit; with the two rewritings: manually created. we construct 100 transfer samples from a source sentence by manually creating two output sentences as described. we create: task 1) on sentiment (positive/negative), using headlines from wiki news foot_1 as source sentences with minor modifications; task 2) on detoxifying, using toxic sentences from @xcite . we supply each sample with a gold-truth reference sentence to enable a small evaluation of reference-based methods. llm generated. we use gpt-4o-mini foot_2 to create style transfer pairs in a multi-step process: ## methodology: logprob we propose a novel approach for evaluating style transfer with the following properties: 1) no training data required on style; 2) no need for gold truth / reference sentences; 3) improved feasibility in the form of model size (e.g. 1b,3b) with improved performance compared to our baseline of prompting an llm-as-autorater. for content preservation, we additionally aim for: 4) accounting for changes in target style -ensuring theoretical suitability as discussed in section 2. for style strength, beyond properties 1,2 and 3, we aim for: 5) scoring on a range rather than a binary score -motivated by the fact that styles can shift to various degrees as discussed in section 2 and recommended by @xcite . similar to the approach in @xcite , who evaluate the faithfulness of summaries, we use token likelihood estimates to evaluate a style transfered sentence. the idea in our work is to provide different rewrite instructions as part of the context before the rewrite sentence, e.g. with and without the target style. see let x be the source sentence consisting of tokens x 1 , x 2 , . . . , x m , and let @xmath0, x2 , . . . , xn be the rewritten sentence that we want to evaluate with respect to some style change s. from an llm, we estimate the probability of the output sentence when the model has seen the source sentence and some rewrite instruction as part of the context t , denoted as p lm ( x|x, t ). ## benchmarking many metrics are used to evaluate content preservation for style transfer. here, we evaluate several recently and widely used ones (per section 2). we group content preservation metrics into four categories: lexical similarity, semantic similarity, contexts x, t s : 'rewrite the following sentence to be s: x' x, t pa : 'paraphrase the following sentence: x' x, t r : 'repeat the following sentence: x' we assess evaluation approaches on style strength using llm-as-autorater and our method log-prob, both of which as zero-shot approaches. datasets we benchmark nine metrics for evaluating content preservation in style transfer on seven different datasets covering nine styles. we use existing datasets already containing human evaluations of style strength and/or content preservation. the human evaluations are conducted on rewrites from system-generated output and/or human-written / gold reference output, using various scales (e.g., 5-point likert or 1-100). these datasets cover rewriting tasks for simplifying, sentiment, formality, and making arguments more appropriate. the following datasets are used (abbreviation in brackets), in addition to our newly test set (section 3) -further details in app. a: we report spearman rank correlation between metric scores and mean human judgement. we group test data as rewrites generated by systems, human-written gold-references, and our constructed test set. we summarise results in each group by reporting the average spearman correlation across data sources (avg), and the average ranking of the metric per data source by correlation score (avg. rank). ## results meta-evaluation is highly impacted by the type of test samples. overall metric performance (correlation with human judgements) varies greatly depending on whether the human-rated outputs are system-generated, human-written, or from our newly constructed test set (fig. 2 ). on systemgenerated data ( we recommend that meta-evaluation efforts for content preservation go beyond measuring correlations with human judgments, and also account for the data source bias underlying those judgments. specifically, content preservation should be benchmarked on diverse data sources -including those with intentionally low content preservation, such as our test set. content preservation: content preservation metrics must be style-aware. across all tasks and data sources, overall, the highest correlations between metric and human judgement are achieved by using llm-as-autorater using 70b llama 3 instruct. if compute is a constraint, we recommend the logprob method with a smaller model. both model sizes of 3b and 1b outperform 3b and 1b llm-as-autorater foot_4 (figure 2 ). for a backbone model size of 8b, llm-as-autorater is recommended. style: llm-as-autorater 70b shows the best average performance in terms of correlations with human judgements. still, our more efficient log-prob method scores higher than llm-as-autorater 70b on 3 out of 5 datasets, despite using a much smaller model (both 8b and 3b versions) ( we conduct a large-scale meta-evaluation of metrics for style transfer, focusing on content preservation. we construct a new challenging test set for assessing metrics for content preservation with human judgment. we show that the meta-evaluation of metrics for content preservation is not straightforward. previous meta-evaluation studies on existing datasets consisting of either automatically generated data from specific systems or humangenerated data alike leads to misleading conclusions about best metrics, due to skewed test cases. we hypothesise that on existing system-generated data, human judgment on content preservation happens to correlate with verbatim repetitions, and on gold references/human-written data, the limits of the metrics are not tested, as cases of drastically low preserved or fabricated content are not or rarely present in the data. with our new test set, deliberately designed with variation in content preservation, we demonstrate empirically that similarity-based metrics are not suitable for content preservation for style transfer. instead, metrics for content preservation should be style-aware. we recommend for meta-evaluation that a diverse set of data sources be used, including both systemgenerated test cases with human ratings, as well as our newly constructed test set, which tests the nature of content preservation metrics. we propose a new efficient style-aware metric for content preservation and one for style strength, reusing the same computational effort (logprob). overall, we find that a large llm-as-autorater (70b) achieves the highest correlation between content preservation scores and human judgments. however, if considering computational efficiency or feasibility in the form of a smaller model, then our method logprob outperforms a llm-as-autorater baseline using a 3b parameter llm. ## limitations in evaluating style transfer, many different metrics are used for content preservation, leading to a need for standardization @xcite . while there are more metrics we could have tested, we do test the more widely used ones, as well as metrics of different types: lexical similarity, semantic similarity, fact-based, and llms conditioned on style shift (style-aware). especially for testing llms within the llm-as-a-judge paradigm, our work is limited to testing one prompting approach, as our main focus has been to establish better metaevaluation practices regarding content preservation in style transfer. we refer to a survey of different trends in the llm-as-judge paradigm in @xcite . we provide general recommendations on metrics over a large variety of rewriting tasks; however, specific tasks may have specific requirements that are not covered in this paper. in this paper, we have not covered the evaluation of 'fluency' in rewrites, but mainly focused in-depth on content preservation and partly on style strength.