Counterfactuals to Control Latent Disentangled Text Representations for Style Transfer (2021)

Abstract: Disentanglement of latent representations into content and style spaces has been a commonly employed method for unsupervised text style transfer. These techniques aim to learn the disentangled representations and tweak them to modify the style of a sentence. In this paper, we propose a counterfactual-based method to modify the latent representation, by posing a ‘what-if’ scenario. This simple and disciplined approach also enables a fine-grained control on the transfer strength. We conduct experiments with the proposed methodology on multiple attribute transfer tasks like Sentiment, Formality and Excitement to support our hypothesis.

## introduction counterfactual reasoning @xcite is leveraged in structured data analysis and econometrics towards generation of alternatives and estimation of alternate scenarios. counterfactuals describe a causal situation of the form 'if x would have (not) occurred, y would have (not) occurred' @xcite . in interpretable machine learning, counterfactuals have been used to explain predictions of individual instances across various types of datasets and tasks @xcite @xcite . @xcite and @xcite use counterfactuals towards generating training data. counterfactual reasoning also provides us with a unique ability to generate explanations and make causal analysis on the latent space. however, this technique has never been explored in natural language generation tasks. here, we plug-in the concept of counterfactuals to the text-style transfer task, to enable the manipulation of latent spaces towards controlled transfer of style. existing works in text style transfer focus on transferring a specific target attribute. unsupervised methods based on adversarial attacks @xcite , back translation @xcite , learning disentangled representations @xcite have been popular in this domain. other techniques include deletion of stylespecific words and conditionally generate sentences in the target style @xcite . however, all of them fail to provide a control over the target style strength i.e. a clever manipulation of the latent space is non-trivial. recent works on controlled text generation include @xcite , which brings in a transformer-based model that modifies the gradient functions leading to controlled generation in the output space. @xcite is an unsupervised approach integrated during end-to-end model training. the drawback in all these efforts is the lack of a prefixed logic towards controlling the latent space. our proposed method of counterfactuals fills in this gap and provides a logical method to control the latent spaces for enabling a smooth style transfer. our approach is based on the premise of disentangled representation spaces inspired from @xcite . separating out the style and content representations introduce an opportunity to finetune, resulting in the ability to control the output sentences specific to style. we introduce a counterfactual reasoning module for controlling latent disentangled spaces for style transfer. figure 1 shows an illustrative example for the variants generated through our approach. to the best of our knowledge, this is the first work leveraging such a concept towards controlled text generation. through extensive quantitative and qualitative experiments, across attributes and datasets, we conclude that the proposed approach is effective in providing control over the style strength and also shows that the best transfer performance is on par with the existing baseline style-transfer techniques. ## results and analysis transfer control. figure 3 shows the performance of cf variants across metrics for different styles. the cf generated variants from t-vae-cf (solid lines) are compared against the reference values which take avg. embeddings (t-vae-avg) for target style (dotted lines). to recollect, the higher the cf transfer confidence (strength), the closer is the generated variant to the target attribute. thus, the ideal performance is to have the highest accuracies for the highest cf confidence values (see figure 3(a) ). note that cf @xmath0. this is difficult to achieve as cf in the representation space may not be generated cluding stopwords @xcite for such a strict target. hence, the variants generated with near perfect transfer target (cf @xmath1.8,0.9,0.95) show the best performance across metrics. the low transfer accuracies for models with low cf confidence establishes the ability of the model to stay near the source when the target strength is low. all models implemented with transfer control report improved performance w.r.t bleu scores establishing the utility of the alternatives generator. ## conclusion we introduce the use of counterfactual reasoning towards controlling the latent disentangled representations for text style transfer. experiments not only establish the superiority of the proposed models across standard metrics for a multitude of styles but also illustrate the utility of the gradual control variable in this model. we further validate the use for cf via a human evaluation establishing improved text attribute transfer.