Enhancing Content Preservation in Text Style Transfer Using Reverse Attention and Conditional Layer Normalization (2021)

Abstract: Text style transfer aims to alter the style (e.g., sentiment) of a sentence while preserving its content. A common approach is to map a given sentence to content representation that is free of style, and the content representation is fed to a decoder with a target style. Previous methods in filtering style completely remove tokens with style at the token level, which incurs the loss of content information. In this paper, we propose to enhance content preservation by implicitly removing the style information of each token with reverse attention, and thereby retain the content. Furthermore, we fuse content information when building the target style representation, making it dynamic with respect to the content. Our method creates not only style-independent content representation, but also content-dependent style representation in transferring style. Empirical results show that our method outperforms the state-of-the-art baselines by a large margin in terms of content preservation. In addition, it is also competitive in terms of style transfer accuracy and fluency.

## introduction style transfer is a popular task in computer vision and natural language processing. it aims to convert an input with a certain style (e.g., sentiment, formality) into a different style while preserving the original content. one mainstream approach is to separate style from content, and to generate a transferred sentence conditioned on the content information and a target style. recently, several models @xcite @xcite have proposed removing style information at the token level by filtering out tokens with style information, which are identified using either attention-based methods @xcite or frequency-ratio based methods @xcite . this line of work is built upon the assumption that style is localized to to our knowledge , this is the best deal in phoenix . ## related work in recent years, text style transfer in unsupervised learning environment has been studied and explored extensively. text style transfer task views a sentence as being comprised of content and style. thus, there have been attempts to disentangle the components @xcite @xcite . @xcite map a sentence to a shared content space among styles to create style-independent content variable. some studies view style as localized feature of sentences. @xcite propose to identify style tokens with attention mechanism, and filter out such tokens. frequency-based is proposed to enhance the filtering process @xcite . this stream of work is similar to our work in that the objective is to take out style at the token level, but different since ours does not remove tokens completely. instead of disentangling content and style, other papers focus on revising an entangled representation of an input. a few previous studies utilize a pre-trained classifier and edit entangled latent variable until it contains target style using the gradientbased optimization @xcite . @xcite view each domain of data as a partially observable variable, and transfer sentence using amortized variational inference. @xcite use the transformer architecture and rewrite style in the entangled representation at the decoder. we consider this model as the strongest baseline model in terms of content preservation. in the domain of computer vision, it is a prevalent practice to exploit variants of normalization to transfer style @xcite . @xcite proposed conditional instance normalization (cin) in which each style is assigned with separate instance normalization parameter, in other words, a model learns separate gain and bias parameters of instance normalization for each style. our work differs in several ways. style transfer in image views style transfer as changing the "texture" of an image. therefore, dumoulin et al. (2017) place cin module following every convolution layer, "painting" with style-specific parameters on the content representation. therefore, the the food is delicious pre-trained gru gru gru style-independent content representation content-dependent style representation cln the food is delicious the food is bland 1 3 4 2 embedding attention reverse attention initial hidden state style marker module 1 removing style 2 stylizer 3 4 decoder input x e(x) ## conclusion in this paper, we introduce a way to implicitly remove style at the token level using reverse attention, and fuse content information to style representation using conditional layer normalization. with the two core components, our model is able to enhance content preservation while keeping the outputs fluent with target style. both automatic and human evaluation shows that our model has the best ability in preserving content and is strong in other metrics as well. in the future, we plan to study problems with more than two styles and apply multiple attribute style transfer, where the target style is comprised of multiple styles.