Rich Syntactic and Semantic Information Helps Unsupervised Text Style Transfer (2020)

Abstract: Text style transfer aims to change an input sentence to an output sentence by changing its text style while preserving the content. Previous efforts on unsupervised text style transfer only use the surface features of words and sentences. As a result, the transferred sentences may either have inaccurate or missing information compared to the inputs. We address this issue by explicitly enriching the inputs via syntactic and semantic structures, from which richer features are then extracted to better capture the original information. Experiments on two text-style-transfer tasks show that our approach improves the content preservation of a strong unsupervised baseline model thereby demonstrating improved transfer performance.

## introduction text style transfer aims at rephrasing an input sentence as an output sentence in a target style (e.g. sentiment change from negative to positive), while preserving the original content. the utility of text style transfer has been shown in applications such as personalized response generation @xcite and poetry generation @xcite . in particular, unsupervised style transfer has been extensively explored due to a lack of parallel corpora @xcite @xcite . most previous efforts on unsupervised text style transfer have relied on separating the content from the style of input texts. this was achieved via a transfer model with multiple decoders @xcite or extra auxiliary losses @xcite to learn the disentangled representation vectors for content and style respectively. the content vector was later combined with the vector of the desired style to produce the output. while these prior studies have successfully demonstrated the capability to adapt input texts to the desired style, the proposed approaches suffer from a significant loss of semantic content. for instance, when a model takes as input "the lounge is very outdated" to generate "the food is delicious", where the key information of the input (the lounge) is missing in the output, the rendering of the transfer becomes irrelevant. to alleviate this problem, some studies sought to explicitly replace the words related to the stylistic aspect (e.g., sentiment) and retain other contentrelated words @xcite @xcite . however, their success was limited to specific situations where the style words are explicit. when the negative sentiment is expressed implicitly, as in "the only thing i was offered was a free dessert!!!" this approach cannot have the desired effect. a second direction to address the semantic loss has been the use of back-translation @xcite @xcite and/or reinforcement learning to preserve the input content @xcite . generally, these techniques involve a more complex model training, adding another layer of difficulty to obtain a strong style transfer model with robust performance. preliminary experiments with available text style transfer models on benchmark datasets validate the utility of our input-encoding approach for preserving input semantic information when compared with a strong baseline @xcite without such an encoding. we include the details of our implementation in the supplementary material. ## baseline the baseline we consider for this study is a recent model @xcite based on a generatordiscriminator framework. the generator transfers sentences from the source style to the target style and is in a tight feedback loop with a set of dis-criminators that evaluate the quality of the transferred sentences. the reported results revealed its competitive style transfer performance on available benchmark datasets. generator. the generator is a typical seq2seq model with a sequence encoder and a sequence decoder @xcite . the encoder adopts a gated recurrent unit (gru) to take in an input sentence with words {x 1 , . . . , x n } and produce the encoder states {h 1 , ..., h n } sequentially. the sequence decoder is another gru with the attention mechanism @xcite . at each time step t, the decoder updates its hidden state s t with the target token generated at time t 1. it then predicts the current token y t using the current decoder state and the weighted sum of the encoder states, where the weights are produced by the attention mechanism. discriminators. three discriminators are included, each serving to judge one aspect of the quality of the generated target sentences from among meaning preservation, transfer strength, and fluency. the meaning preservation is evaluated using the word mover's distance @xcite , which calculates the similarity score r sem . the style discriminator predicts the likelihood r style of a generated sentence in the target style as the style quality. moreover, a pre-trained neural language model evaluates the fluency by estimating the logprobability r lm of each generated sentence. the overall evaluation score that served as the feedback to the generator, r, was a weighted average of the three evaluation metrics to account for the fact that they may not be in the same scale. reinforcement learning (rl). rl trains the generator with the feedback received from the discriminators (scores given by the discriminators to its generated sentences) @xcite . was formulated as making a sequence of actions, where an action is a token produced at a decoding step. taking the decoding step t as an example, the decoder state s t contains the information of the input source sentence and the partial target sentence already generated by the model. an action a t is the generated token of the target sentence at step t, and a reward q t is defined to reflect how good the action is. the reward q(s t , a t ) of taking action a t in state s t was estimated by sampling complete target sentences with their first t 1 tokens fixed. with r(s ⌧ , a ⌧ ) as the average score over the sampled sentences with the first ⌧ tokens fixed, the reward was defined as the generator was parameterized by ✓ and denoted as g ✓ . the total reward j(g ✓ ) of generating a target sentence of t tokens was ## model we propose to enrich input sentences with syntactic and semantic structures, and to encode the resulting graphs using a graph neural network (gnn). for a fair comparison with the baseline model, we replaced the gru encoder of the baseline generator with our gnn encoder and kept the other modules unchanged. our proposed model is called the graph transfer (gt) model. figure 2 demonstrates our style transfer model with the proposed graph encoder. the input sentence is first parsed as a syntactic-semantic graph. the graph encoder encodes rich information of the graph into dense representations, which are then fed to a sequence decoder for sentence generation. the transferred sentences are sent to the discriminators for evaluation, and the scores of these sentences serve as the training signals for the generator. ## experiments dataset. we focus on the task of sentiment transfer, retaining the setting of the yelp dataset as @xcite . the dataset contains 176, 878 negative and 267, 314 positive sentences for training, 25, 278 negative and 38, 205 positive sentences for development, and 50, 278 negative and 76, 392 positive sentences for testing. we construct syntacticsemantic graphs with the stanford dependency parser @xcite and the semantic role labeler of the allennlp @xcite . baselines. we compared our model (gt) with three state-of-the-art models for text style transfer: (1) reinforcement learning based model (rl). rl is the baseline summarized in section 2. ## conclusion we empirically demonstrated how including rich syntactic and semantic information can help to preserve content during text style transfer. toward this, we compared competitive style transfer models with and without enriching inputs via syntactic and semantic structures on a benchmark dataset. we found that instead to an input text of a sequence of words alone, encoding the input sentence's syntactic-semantic graph via a graph neural network serves to explicitly highlight the sentence's core meaning. in this work, we have focused on the generator to improve the performance of text style transfer. one of our future directions is to incorporate better semantic metrics into the discriminators so that the training loss could measure the preservation of semantic information more accurately.