A Study on Manual and Automatic Evaluation for Text Style Transfer: The Case of Detoxification (2022)

Abstract: It is often difficult to reliably evaluate models which generate text. Among them, text style transfer is a particularly difficult to evaluate, because its success depends on a number of parameters. We conduct an evaluation of a large number of models on a detoxification task. We explore the relations between the manual and automatic metrics and find that there is only weak correlation between them, which is dependent on the type of model which generated text. Automatic metrics tend to be less reliable for better-performing models. However, our findings suggest that, ChrF and BertScore metrics can be used as a proxy for human evaluation of text detoxification to some extent.

## introduction there exist many natural language processing (nlp) tasks whose output is a text (dialogue, summarization, etc.). they often adopt the evaluation techniques from machine translation (mt). namely, researchers often compare the output of a model with a pre-defined reference answer and measure the model quality as the similarity to this reference. the similarity can be computed at the level of words and phrases (e.g. bleu or meteor) or be more semantically motivated and compare the embeddings (e.g. bertscore or bleurt). this approach has a number of drawbacks which make it inapplicable to some generation tasks, e.g. style transfer. this is a task of changing a text such that its meaning stays the same and the style changes. style can refer to any attribute concerning only the form of the text (e.g. degree of formality or politeness) or its content (e.g. sentiment, author features, etc.). when evaluating the output of a style transfer model, we need to pay attention to both the style change and the content preservation. the traditional mt evaluation metrics mainly check the semantic similarity, which makes them unsuitable for style transfer. there exist evaluation metrics @xcite which were devised to consider all important aspects of style transfer quality (style, semantic similarity and sometimes fluency). however, they heavily rely on automatic models (e.g. style classifier) whose performance is not perfect. many works acknowledge the low reliability of such metrics and arrange manual evaluation to get the objective information on the models performance. unfortunately, such evaluation is laborious and cannot be conducted often, so during development of models researchers still have to resort to automatic metrics. although works on style transfer acknowledge that automatic evaluation metrics are unreliable, there is little work on the analysis of their performance. there exist analysis of content preservation metrics @xcite and of all style transfer evaluation metrics @xcite . the latter work provides an evaluation where metrics are tested on different systems and different style transfer directions. we further extend this line of research by testing the evaluation metrics on a new style transfer task (detoxification) and a new language (russian). for this comparison we create a large parallel corpus for detoxification. we compare the performance of models based on different principles, which allows more robust evaluation. furthermore, since we compare a large number of models, we can understand to what extent the automatic metrics can rank the models correctly. besides that, due to the large number of tested models we decided to use crowdsourced evaluation instead of experts. we describe our crowdsourcing annotation setup and analyse the performance of crowd workers. finally, the large-scale evaluation allows us to gain insights on the performance of various style transfer models. the research was based on the data of a competition of detoxification models for the russian language organized by the authors of this paper. foot_0 2 evaluation ## detoxification competition details the evaluation was conducted under the scope of a competition of detoxification models for the russian language @xcite . 3 for this competition we created a russian parallel corpus of toxic sentences and their manually written nontoxic equivalents. we also developed several baselines. ## automatic evaluation in our automatic evaluation we follow the state-ofthe-art evaluation strategies. namely, we replicate the setup of @xcite . we evaluate the three parameters of style transfer quality: style of a text, content preservation, and fluency of a text. the three metrics are then aggregated to a joint score. we use the following techniques. style (sta a ) is evaluated with a bert-based classifier for toxicity detection. we use the same rubert-based classifier that was used for preselection (see section 3.1). content (sim a ) is evaluated as the cosine similarity of embeddings of the source and the transformed sentences. we use embeddings generated by labse model @xcite because in our preliminary experiments they showed the best performance for russian. we prefer the embedding distance over bleu-like metrics, as @xcite showed that embedding-based metrics are better correlated with human judgments than ngram-based metrics such as bleu. we do not use references for the evaluation of content to mimic the setup where references are unavailable, which is very common for style transfer tasks. ## manual evaluation the manual evaluation follows setups used in stateof-the-art works. we separately evaluate the three parameters of the transferred sentences, namely, their style, content, and fluency. we conduct the evaluation via crowdsourcing. for the evaluation we also use yandex.toloka platform. ## results in this section, first we present the data, namely the outcome of the shared task on detoxification evaluation. second, we perform anlysis of correspondance of human and automatic metics. finally, we conclude with a discussion of assessors's performance and overall difficulty of the task. ## conclusion we conducted an evaluation of detoxification models for russian using both automatic and manual metrics. this allowed us to analyse the relationship between the metrics and assess the suitability of automatic metrics for evaluation. our analysis shows that the metrics are overall weakly correlated with the human judgements both at the system and the sentence level. we found that chrf score has a strong correlation with the joint score of style, content, and fluency. thus, chrf could be used as a proxy for manual evaluation, but its lack of correlation with the style score makes this metric vulnerable to attacks. at the system ranking level bertscore metric yielded the best correlation with human judgements. we also discovered that the correlation of manual and automatic scores varies for different models. this shows the necessity to consider diverse style transfer models for metrics analysis. overall, although the state-of-the-art evaluation setup for detoxification task (three parameters and the joint score combined from them) is conceptually correct, the current performance of automatic metrics is insufficient to use it as a replacement for manual evaluation. a worse thing is that the automatic metrics produce less reliable for betterperforming models, thus blocking the advance of style transfer models.