Prefix-Tuning Based Unsupervised Text Style Transfer (2023)

Abstract: Unsupervised text style transfer aims at training a generative model that can alter the style of the input sentence while preserving its content without using any parallel data. In this paper, we employ powerful pre-trained large language models and present a new prefix-tuning-based method for unsupervised text style transfer. We construct three different kinds of prefixes, i.e., shared prefix, style prefix, and content prefix, to encode task-specific information, target style, and the content information of the input sentence, respectively. Compared to embeddings used by previous works, the proposed prefixes can provide richer information for the model. Furthermore, we adopt a recursive way of using language models in the process of style transfer. This strategy provides a more effective way for the interactions between the input sentence and GPT-2, helps the model construct more informative prefixes, and thus, helps improve the performance. Evaluations on the well-known datasets show that our method outperforms the state-of-the-art baselines. Results, analysis of ablation studies, and subjective evaluations from humans are also provided for a deeper understanding of the proposed method.

## introduction style transfer is an important task in computer vision and natural language processing. it aims to convert the input into the target style (e.g., season or artist style in image and sentiment or formality style in text) while preserving its content information. in addition, in text style transfer, the model also needs to ensure the fluency of the output sentence. due to the lack of parallel data, the studies of text style transfer tasks mainly focus on unsupervised settings. the mainstream unsupervised text style transfer (utst) methods are based on the idea of disen-tangled representations of style and content. they usually use recurrent neural networks (rnns) or transformer @xcite to encode style and content information of the input sentence with two independent embeddings. and the decoder takes the content embedding and the target style embedding as inputs to generate a new sentence with the expected content and style. but @xcite has proved that it is not necessary to separate the style and content from the input sentence. the specially designed model can generate text with the target style by overwriting the original style. therefore, we adopt a similar strategy in our model and directly feed the sentence and the target style information into the model for style transfer. in previous works, the encoder encodes the sentence into a fixed-size, such as a 768-dimensional latent representation, which is small and hard to capture the full information of the sentence, leading to information loss. thus, models without encoding the sentence to latent representation, e.g., decoder-only model, might be a better choice for the task of text style transfer. recently, pre-trained auto-regressive language models have shown great success in text generation. both gpt @xcite and llama @xcite use the decoder part in transformer as their model framework. however, their ability in the task of utst has not been fully investigated before. in this paper, we design a novel model to exploit auto-regressive model's highquality text generation ability to achieve text style transfer. moreover, to avoid damaging the text generation ability, we adopt prefix-tuning paradigm @xcite . according to the above objectives, we propose a novel prefix-tuning framework based on gpt-2 for unsupervised text style transfer. following previous works in the area of utst @xcite , we also employ the framework of adversarial learning due to the lack of parallel data. thus, our framework contains a generator and a discriminator. in the generator, three prefixes, i.e., shared prefix, style prefix, and content prefix, are proposed, and each of them serves different roles. the shared prefix provides task-specific information, which is independent of the input. the style and content prefix provide input-dependent information. the style prefix encodes the target style, and the content prefix extracts the content information in the input sentence. these prefixes provide information for gpt-2, guiding the model for style transfer. for the content prefix, no extra encoder is used. we simply use the pre-trained language model recursively to obtain the content prefix. in our model, the discriminator is also based on prefix-tuned language model. therefore, only a fraction of the parameters of the whole model needs to be trained. the contributions of our paper can be summarized as follows: ## related work in this section, we present a brief summarization of topics related to our method. ## the proposed method in this section, we will describe the task settings of unsupervised text style transfer briefly and then present our method. ## conclusion in this paper, a novel method for unsupervised text style transfer based on prefix-tuning was proposed, in which we take advantage of the pre-trained language model. we encode both target style and input text with prefixes that guide style transfer using prefix-tuning. we design a style transfer method that uses the language model recursively, using gpt-2 in both the content extraction and style transfer stages. in this way, the input sentence can fully interact with the language model, fully utilizing the language model's capabilities for style transfer. experiments show that our method significantly improves content preservation and fluency of the style-transferred sentences compared to the baselines while achieving competitive results on the style control. in the future, we plan to extend our approach to more broad application settings.