A large-scale computational study of content preservation measures for text style transfer and paraphrase generation (2022)

Abstract: Text style transfer and paraphrasing of texts are actively growing areas of NLP, dozens of methods for solving these tasks have been recently introduced. In both tasks, the system is supposed to generate a text which should be semantically similar to the input text. Therefore, these tasks are dependent on methods of measuring textual semantic similarity. However, it is still unclear which measures are the best to automatically evaluate content preservation between original and generated text. According to our observations, many researchers still use BLEU-like measures, while there exist more advanced measures including neural-based that significantly outperform classic approaches. The current problem is the lack of a thorough evaluation of the available measures. We close this gap by conducting a large-scale computational study by comparing 57 measures based on different principles on 19 annotated datasets. We show that measures based on cross-encoder models outperform alternative approaches in almost all cases. We also introduce the Mutual Implication Score (MIS), a measure that uses the idea of paraphrasing as a bidirectional entailment and outperforms all other measures on the paraphrase detection task and performs on par with the best measures in the text style transfer task.

## introduction text style transfer (tst) and paraphrases generation (pg) are active areas of research in nlp, with dozens of papers proposing new methods. these methods could be applied for practical purposes, such as supporting human writers, personalizing digital assistants, or even creating artificial personalities. research and development of tst models require fast feedback loops, and they require fast and reliable automatic quality measures. tst is hard to evaluate for several reasons. first, golden answers, even if available, are not the only valid way to rewrite the text. second, parallel corpora with different styles do not emerge naturally and are hard to find. this means that reference-based evaluation is often prohibitive and creates a need for manual evaluation of tst or for clever automatic measures. the basic desired properties of tst are style accuracy, content preservation, and fluency @xcite . for many methods of unsupervised tst, keeping the content of the original text and automatically measuring its preservation is one of the most difficult tasks (see e.g. @xcite ). during development, the only way to control content preservation is to use automatic measures. such measure takes two sentences and return the value which indicates the similarity of their content. more formally, the measure sim quantifies semantic relatedness of two utterances, an original text x and a style-transferred or paraphrased text y : sim(x, y)→[0; 1]. the measure sim yields high score for the pairs with similar content and low score for ones with different content. as @xcite and @xcite show, most tst works evaluate the content preservation with bleu @xcite or similar measures based on word overlap between two texts. the situation in pg is almost identical. most works including the most recent ones @xcite ) also still rely on bleu. even though measures like bleu, based on a word or character-level n-grams are pretty intuitive and straightforward, they don't take into account synonyms and distributively related words. moreover, there already exist several pieces of evidence that correlation of standard bleu-like automatic measures is relatively low @xcite . the recent development of vector representations of textual information @xcite and various ways to handle these vectors provides room for improvement of the approaches to scoring the content preservation. it is, therefore, crucial to perform a thorough analysis of all existing content preservation measures and to gather best practices from the top-performing approaches to create a new approach that could demonstrate stable performance in terms of both pg and tst tasks. in this work, we further extend a comprehensive study of @xcite by analyzing a much more diverse set of measures including recently developed transformers-based ones, and also by proposing a new measure specially developed for tst and pg content preservation scoring. the contributions of our paper are as follows: ## datasets used in comparative study we run our analysis of measures on parallel datasets manually labeled for semantic similarity or content preservation. to make the comparison more generalizable, we fetch a large number of datasets generated by different models. ## mutual implication score (mis) the goal of our research is not only to analyze the existing measures of content preservation but also to suggest a new measure that can outperform the existing ones. we devise a new measure that is based on measuring content similarity with nli, as described by @xcite . in this work, the authors exploit the assumption that implies the two sentences with the same meaning should be equivalent in their inferential properties, i.e. each sentence should textually entail the other. this means that the nli model is supposed to return similar entailment scores when applied to semantically equal sentences regardless of the sequence these sentences are sent to the input of the model. the authors used this assumption to propose an adversarial method of dataset creation for paraphrase identification. nli models predict whether one text logically entails another, and are, therefore, asymmetric. high entailment probability in the forward direction means that the second text accurately follows the first one and does not contain hallucinated information. a high entailment score in the backward direction means that all the information from the first text is retained in the second text. the most natural way to aggregate scores from both directions is to multiply them or compute their arithmetic or harmonic mean. we use this approach as a baseline. we yield nli scores from the following models: • robnli @xcite ) -roberta-large @xcite pre-trained on snli @xcite , mnli @xcite , fever-nli @xcite , and anli @xcite , although these nli models are a good starting point, they might not be fully suitable for measuring content preservation, because they were trained for a different task. we suggest that further finetuning them on the data annotated with content preservation scores might yield better models. thus, we modify the roberta architecture used for nli. namely, we use the original encoder in both forward and backward directions, concatenate the last hidden states, and then send them to the classification module which is tuned on data annotated with content preservation scores. we refer to this model as mutual implication score (mis). the scheme of our model is given in figure 3 . we initialize the model with pre-trained weights from the robnli model. we tune it on quora question pairs dataset @xcite for 2 epochs with a learning rate 4e -6 and all but the last encoder layer and classifier layer frozen. we evaluate the model with the spearman rank correlation coefficient of the automatic content preservation scores with human judgments. we evaluate all tst and pg datasets introduced in section 3. we evaluate mis and baseline nlibased measures (we aggregate the nli scores for both directions with the arithmetic mean because it showed the best results in our preliminary experiments). the results are shown in ## measures analysis we compute the content preservation scores for paraphrasing and style transfer datasets using measures of different types. we analyze the performance of individual measures and compare the performance of different groups of measures. we also look into the difference in measures performance on pg and tst tasks and analyze the individual datasets. ## computational efficiency of the measures while the correlation of measures with human judgments is important, the usability of the measure in real tasks can not be treated in isolation from its computational efficiency. the main capabilities of such measures are robustness and inference speed. one of the key functions of content preservation measures is to compare different tst or pg approaches with each other and ensure that different runs of the learning-based measure yield similar results. this problem does not apply to words or character n-grams-based models. however, this could yield some issues with trainable model-based measures. that is why it is crucial for all such measures to open-source trained weights. moreover, when using such measures for comparison it is necessary to put the model into inference mode and freeze all layers. in such a case the model-based measures yield similar scores to similar text pairs regardless of the number of attempts or any hardware properties. another blocker to the usage of a certain measure could be a long inference time. we conduct additional experiments by calculating the average inference time per sample for a subset of measures representing each class w.r.t. the average correlation of the measure on the task. we concatenate texts from both tasks into two united datasets. for trainable measures, we use a data loader with a batch size equal to eight. we load all trainable models to nvidia geforce rtx 2080 ti. all other measures are calculated sample-wise on intel(r) xeon(r) gold 5217 cpu @ 3.00ghz . we plot the results on figure 6 . the most optimal measures are located at the bottom right corner of these plots, which means that the measure requires the least possible computational time and at the same time demonstrates a high correlation with human judgments. for the pg task, the mis measure demonstrates the best performance and its average inference time is at the approximately same level as most of the other model-based measures. for tst task symmetric and asymmetric cross-encoders are the most optimal. ## conclusions as our experiments show, encoder-based measures of content preservation correlate with human judgments much better than the traditional word or character-based measures such as bleu on a wide range of datasets. in all paraphrase datasets and 9 out of 10 text style transfer datasets, the best-performing measures are based on the crossencoder or bi-encoder architecture. we suggest a measure called mis which is based on the idea that texts with similar meanings mutually entail each other. we show that the proposed architecture outperforms other measures in the evaluation of paraphrases and performs on par with the top-performing measures in the evaluation of text style transfer. more specifically, it is particularly successful in transferring between contemporary and old english and between formal and informal styles. thus, we recommend using this measure for content preservation scoring for paraphrases and tst tasks in the aforementioned tasks and to use bleurt for other tst tasks. while the best measures in our analysis improve over the popular ones (e.g. bleu) by a large margin, their correlation with human judgments is still far from perfect. we expect that even better measures of content preservation will be proposed in the nearest future. we also hope that the mis measure and the performed large scale computational study could be applied to other nlp tasks, such as machine translation, text summarization, etc.