Syntax Matters! Syntax-Controlled in Text Style Transfer (2021)

Abstract: Existing text style transfer (TST) methods rely on style classifiers to disentangle the text’s content and style attributes for text style transfer. While the style classifier plays a critical role in existing TST methods, there is no known investigation on its effect on the TST methods. In this paper, we conduct an empirical study on the limitations of the style classifiers used in existing TST methods. We demonstrated that the existing style classifiers cannot learn sentence syntax effectively and ultimately worsen existing TST models’ performance. To address this issue, we propose a novel Syntax-Aware Controllable Generation (SACG) model, which includes a syntax-aware style classifier that ensures learned style latent representations effectively capture the sentence structure for TST. Through extensive experiments on two popular text style transfer tasks, we show that our proposed method significantly outperforms twelve state-of-the-art methods. Our case studies have also demonstrated SACG’s ability to generate fluent target-style sentences that preserved the original content.

## introduction text style transfer (tst) is an increasingly popular natural language generation task that aims to change the stylistic properties (e.g., the sentiment of text) of the text while retaining its styleindependent content @xcite . due to the difficulty in obtaining training sentence pairs with the same content and differing styles, most existing methods are designed to perform tst in an unsupervised manner; the models only have access to non-parallel, but style-labelled sentences. a popular tst approach is to leverage an adversarial learning autoencoder framework where a style classifier or discriminator is pre-trained to first disentangle the content and style latent representations, before using a decoder to generate the output sentence in the target style @xcite @xcite . another line of work proposed attribute-controlled generation methods where the style attribute latent vector is learned and combine with the latent representation of the text to generate output sentences in target style @xcite @xcite . similar to the adversarial learning approach, the learning of the style attribute latent vector is guided using a pre-trained style classifier. a common key component in the two aforementioned tst approaches is the usage of a style classifier. however, little is known about the effects of the style classifier on these models. for instance, is the style classifier effective in learning the style in the text? what aspects of the text style has the existing style classifier learned? can the style classifiers distinguish text's syntax? can the style classifier guide tst models to generate syntactically correct sentences and in the target style? this paper investigates these questions by conducting an empirical analysis of the style classifiers used in tst models. extending from our empirical study, we propose the syntax-aware controllable generation (sacg) foot_0 model, which includes a syntax-aware style classifier that ensure that the learned style latent representations effectively capture the syntax information for tst. through extensive experiments with two popular tst datasets and human evaluation, we demonstrated sacg's ability to outperform the state-of-the-art baselines in the tst tasks. ## related work in recent years, studies on text style have attracted not only the linguist's attention but also that of many computer science researchers. specifically, computer science researchers are investigating the text style transfer (tst) task that aims to change the text's stylistic properties while retaining its style-independent content. the recent comprehensive survey @xcite summarizes the existing tst approaches. among these approaches, a popular line of research aims to infer a latent representation for an input sentence and manipulate the generated sentence's style based on this learned latent representation. two techniques are commonly used to learn and manipulate the text's style latent representations: (1) adversarial learning and (2) attribute controlled generation. @xcite leverages an adversarial training scheme where a classifier is used to evaluate if an encoder is able to generate a latent content representation devoid of style. the text content latent representation is subsequently used to generate a specific style sentence using a style-dependent decoder. similar works have been proposed where a classifier is pretrained to enable the adversarial learning process in tst models @xcite @xcite @xcite @xcite . @xcite proposed an attribute-controlled generation text style transfer model that utilized a variational autoencoder (vae) @xcite to learn a sentence's latent representation z and leverage a style classifier to learn a style attribute vector s. subsequently, z and s are input into a decoder to generate a target style sentence. similar attribute-controlled generation methods have been proposed for the tst task @xcite @xcite . in the aforementioned methods, pretrained style classifiers played a vital role in guiding the tst task. however, these style classifiers are often pretrained without considering the syntax of sentences. we postulate that syntax is an important aspect of text style, especially in text formality style transfer. this paper empirically demonstrates the importance of modeling syntax in the tst task and proposes a novel syntax-aware tst method that outperforms state-of-the-art tst methods. ## empirical study before presenting our proposed method, we first conduct an empirical study on the style classifiers used in existing tst methods. the goal is to examine the style classifiers' ability to learn the syntax style information in a given text. textcnn @xcite , rnn @xcite , and transformer @xcite are popular style classifiers used in many tst models @xcite @xcite @xcite . in this study, we train the three style classifiers on gyafc @xcite , which is a popular formality transfer dataset used in many tst studies. we first train and test the classifiers using the original gyafc training and test set. next, we perturb the sentence structure of the text in the gyafc test set by disordering the sentences' word order. the underlying intuition is that there should be syntax differences between formal and informal sentences, and the style classifiers should be able to learn the syntactic style information. therefore, perturbing the test set's sentence structure should worsen classification accuracy as the syntactic information in the text is corrupted. the empirical experiment results show that syntax plays a crucial role in text's formality. ## methodology this section proposes the syntax-aware controllable generation (sacg) model, which addresses the ineffectiveness of existing tst methods in handling sentence structure when transferring text style. we first introduce graph convolutional networks (gcns). subsequently, we explain how the gcns are utilized to extract sentence structure information in our syntax-classifier and syntaxencoder, which are the two main components in our proposed sacg model. finally, we describe the learning process of our sacg model. ## ethical considerations tst algorithms have many real-world applications. for example, these algorithms can improve target marketing messages' persuasiveness and integrate into writing tools to improve users' writing style. however, tst algorithms inherently run the risk of being misused for document forgery, impersonation, and sock-puppeting. to mitigate these risks, we will add access control to our code repository, and we would share our codes after the requester has acknowledged our ethical disclaimer. ## conclusion in this paper, we empirically examined the style classifier used in existing tst models and demonstrated that the existing style classifier could not learn the text syntax effectively. we proposed sacg, a novel deep generative framework that considers syntax when learning style latent representation. we conducted extensive experiments on two benchmark datasets and benchmarked sacg against competitive tst models. the automatic and human-based evaluation experiment results showed that sacg outperforms state-of-the-art methods. our case studies also demonstrated that sacg is able to generate fluent target-style sentences that preserved the original content. for future work, we will continue to explore other methods to improve the structural representations of text and incorporate them to perform better tst.