Rethinking Sentiment Style Transfer (2021)

Abstract: Though remarkable efforts have been made in non-parallel text style transfer, the evaluation system is unsatisfactory. It always evaluates over samples from only one checkpoint of the model and compares three metrics, i.e., transfer accuracy, BLEU score, and PPL score. In this paper, we argue the inappropriateness of both existing evaluation metrics and the evaluation method. Specifically, for evaluation metrics, we make a detailed analysis and comparison from three aspects: style transfer, content preservation, and naturalness; for the evaluation method, we reiterate the fallacy of picking one checkpoint for model comparison. As a result, we establish a robust evaluation method by examining the trade-off between style transfer and naturalness, and between content preservation and naturalness. Notably, we elaborate the human evaluation and automatically identify the inaccurate measurement of content preservation computed by the BLEU score. To overcome this issue, we propose a graph-based method to extract attribute content and attribute-independent content from input sentences in the YELP dataset and IMDB dataset. With the modified datasets, we design a new evaluation metric called “attribute hit” and propose an efficient regularization to leverage the attribute-dependent content and attribute-independent content as guiding signals. Experimental results have demonstrated the effectiveness of the proposed strategy.

## introduction text style transfer aims to modify the input attribute while retaining the attribute-independent content and contextual relations. for instance, given the input "the food in this restaurant is really delicious," an expected sentiment transfer result from positive to negative could be "the food in this restaurant is really disgusting." in this process, we expect to flip the sentiment while preserving essential contents such as "food" and "restaurant." this paper focuses on the non-parallel sentiment style transfer, where the sentences before and after transfer are not paired in the training data. most existing works follow this setting, which is more common in real applications due to the scarcity of parallel datasets. most recent research efforts of text style transfer have been put on the model architecture design @xcite @xcite @xcite @xcite @xcite @xcite and methodological innovations @xcite @xcite @xcite . though achieving much progress, we identify that the evaluation system is broadly unsatisfactory. existing evaluation systems mainly carry out automatic evaluation and human evaluation: (i) automatic evaluation: current works mainly adopt classification accuracy, bleu score, and ppl score for automatic evaluation. we argue that these metrics are not effective for evaluating text style transfer due to inconsistent and unfair comparisons across different works. for example, ppl is reported based on different pre-trained language models. in addition, they always pick one checkpoint for model comparison from which we usually can't reach a consensus on a proposed model's actual performance. to alleviate these issues, we propose targeted approaches: (i) for automatic evaluation, we conduct a detailed analysis and comparison for the current metrics from three aspects: style transfer, content preservation, and naturalness. we re-run the current state-of-the-art models and make a fair comparison under the same setting. in addition, we propose robust style transfer evaluation by drawing curves reflecting the style transfer versus naturalness trade-off, and content preservation versus naturalness trade-off. with these trade-off curves, one could have an overall comparison. for example, one can find out whether one model is consistently better than another, or whether it is better only in some aspects (section 3). (ii) for human evaluation, in order to eliminate the bias, we randomly mixed some manually labeled sentences to test the workers. besides, we delicately design some rules to make human evaluation more reasonable and reliable (section 4). through human evaluation and analysis, we found that the current automatic evaluation metrics retain the problem of detecting content preservation. to detect content preservation, the ideal automatic evaluation metric needs to be able to identify style-independent contents from an input sentence. however, the bleu score simply calculates the continuous overlap without excluding style-related words. the earth mover distance (emd) from @xcite alleviates the problem through masking style-related words and then calculating the earth mover distance. but style-related words are detected through checking human-labeled lexicon as a reference, making this method hard to be extended to other datasets. therefore, how to effectively detect style-related words is the key challenge. thanks to the dependency parser, we can analyze the meaning, structure, and syntactical relationships in sentences and then formulate the general grammar rules to identify style-related contents. by leveraging this method, we pre-process the yelp and imdb dataset. furthermore, we introduce a regularization term that encourages the matching of attribute-independent tokens while discouraging others. we demonstrate improved model performance of our method (section 5). the modified datasets will be released for future research. ## revisiting automatic evaluation in this section, we will examine the current automatic evaluation metrics and automatic evaluation method from the following three aspects. ## revisiting human evaluation human evaluation usually has been regarded as the ground truth of automated evaluation. however, the accuracy is affected by several factors: (1) large variance of human judgement -in the mturk, one task will be distributed to many people, who have different scoring standards; (2) some tasks are too hard for workers to be understandable, even with examples; (3) some workers are of low quality. we implement the following improvements. (1) to avoid the bias between different models, we associate each assignment for each worker 30 sentences, and 10 sentences per model. (2) to make rating the content preservation task more effective, we further provide some accepted good examples and rejected bad examples. we observe that this additional information brings in evident quality improvement on human evaluation. (3) to avoid the bias between different people and ensure workers to complete the work with high quality, we manu-ally label 5 sentences and randomly mix them with 30 other sentences. thus, each assignment contains 35 sentences, 5 of which are used to verify worker's quality. we will reject the whole assignment if the score for one of the 5 test cases different from our labels. (4) we reject all the assignments that do not match our requirements and block the workers from consistently providing low-quality submissions. the rejected assignments are re-collected until all assignments strictly match our manually labeled results. in this way, we can ensure human evaluation to be accurate. the amt interface can be found in the supplementary material, along with more details. ## attribute hit the key challenge in the task is to automatically identify style related and unrelated words. since sentence parsing can be helpful in understanding the meaning, structure, and syntactical relationships in a sentences, we adapt it to analyze the sentence structure and detect attribute independent and dependent content. ## conclusion we analyzed automatic evaluation metrics and introduced a robust style transfer evaluation method. by designing a more reliable human evaluation method, we further examined three state-of-the-art models and current evaluation metrics. as confirmed in our experiments, leveraging a classifier to evaluate style transformation is close to human evaluation. however, the current standard evaluation metric: bleu scores are not accurate when measuring content preservation in style transfer. similarly, ppl score also not ideal in measuring naturalness. to overcome this issue, we propose a graphbased method to extract attribute-dependent content and attribute-independent content from input sentences in the yelp and imdb dataset. with the modified datasets, we design a new evaluation metric called "attribute hit," which is a general method and could better measure content preservation. in addition, we tried to use software -grammarly to measure the naturalness. however, borrowing the grammarly software is not convenient since it needs manually copy the generated sentences. in addition, there are also many limitations in the software, such as not too many or too few characters in a calculation. designing better and more general metrics that can estimate sentence fluency is also a challenge for the whole nlp community. by leveraging our modified datasets, we add the cosine similarity regularization as the guiding signal, which could further boost style transfer performance. by leveraging our published graph-based attribute extraction code, people could modify any other sentiment style transfer datasets. also, this could help follow-up research to improve the style transfer method by leveraging style-dependent content and style-independent content.