Text Style Transfer with Contrastive Transfer Pattern Mining (2023)

Abstract: Text style transfer (TST) is an important task in natural language generation, which aims to alter the stylistic attributes (e.g., sentiment) of a sentence and keep its semantic meaning unchanged. Most existing studies mainly focus on the transformation between styles, yet ignore that this transformation can be actually carried out via different hidden transfer patterns. To address this problem, we propose a novel approach, contrastive transfer pattern mining (CTPM), which automatically mines and utilizes inherent latent transfer patterns to improve the performance of TST. Specifically, we design an adaptive clustering module to automatically discover hidden transfer patterns from the data, and introduce contrastive learning based on the discovered patterns to obtain more accurate sentence representations, and thereby benefit the TST task. To the best of our knowledge, this is the first work that proposes the concept of transfer patterns in TST, and our approach can be applied in a plug-and-play manner to enhance other TST methods to further improve their performance. Extensive experiments on benchmark datasets verify the effectiveness and generality of our approach.

## introduction text style transfer (tst), an important task in natural language generation, aims to change the stylistic properties while preserving the style-independent content within the context. stylistic properties include but are not limited to sentiment, politeness, formality, and humor. this task has wide applications, such as neutralizing offensive remarks @xcite , data augmentation @xcite , and human-computer interaction @xcite . the main difficulty in this task is the lack of parallel datasets, making most previous works develop their methods in an unsupervised manner and can be roughly divided into two groups. methods in the first group @xcite ) separate style-independent sentence representations and revise them with style attributes, while methods in the second group @xcite directly revise an entangled representation of an input by using an extra style embedding. however, the two groups of existing methods only focus on the transformation between styles and do not take into account that this transformation might be achieved via different latent transfer patterns. as a matter of fact, there are lots of transfer patterns in the tst task. figure 1 shows three cases from the test set of sentiment transfer dataset yelp @xcite . in the first case, the transfer pattern is extracting key emotional words and taking their antonyms. the transfer pattern of the second case is to change the affirmative polarity to negative polarity while the last case is to change the negative polarity to affirmative polarity. such latent patterns naturally exist for various source texts and imply different specialized ways to solve the transfer task. intuitively, it would be dramatically informative and helpful to mine and exploit latent transfer patterns as prior knowledge. however, one key obstacle lies ahead we have no access to such golden annotation of a systematic taxonomy for all available patterns, nor does it cost-effective to manually conclude and annotate one. in this work, we propose a novel contrastive transfer pattern mining (ctpm) method, which can automatically detect different transfer patterns and use contrastive learning on this basis. in this way, we can obtain more accurate sentence representations, which thereby help us to achieve better text style transfer. figure 2 gives an overview of our approach. we first use a clustering module to automatically mine latent transfer patterns and obtain transfer pattern labels. then we simultaneously exploit intra-style contrastive learning with transfer pattern labels and inter-style contrastive learning with style labels to enhance text representations, boosting the performance of text style transfer. it is worth mentioning that our method can be combined plug-and-play with both the above two groups of mainstream tst methods to improve their performance. in summary, our contributions are as follows: ## approach the overall architecture of our method is depicted in figure 2 , which consists of an adaptive clustering module and a contrastive learning module. the adaptive clustering module automatically mines latent transfer patterns in each style. based on the mined transfer patterns, the contrastive learning module adopts intra-style and inter-style contrastive learning losses to learn more precise sentence representations. the two losses are applied to a basic tst model to further improve its performance. we will first give a brief introduction to the tst task, and then elaborate on the two modules. ## conclusion in this paper, we propose a novel approach named contrastive transfer pattern mining (ctpm) for text style transfer (tst) tasks. different from the previous methods that mainly consider the differences between styles, we mine and exploit the latent transfer pattern in each style. specifically, we design an adaptive clustering module to mine and exploit the latent transfer patterns, and then introduce a contrastive learning module, including an inter-style contrastive loss and an intra-style contrastive loss, to obtain more meaningful and distinguishable sentence representations, which could improve the performance of tst. our approach does not depend on a specific network structure and can be widely applied to basic tst models. experiments with two mainstream basic tst models and two widely used benchmark datasets have shown the effectiveness and generality of our proposed approach.