Collaborative Learning of Bidirectional Decoders for Unsupervised Text Style Transfer (2021)

Abstract: Unsupervised text style transfer aims to alter the underlying style of the text to a desired value while keeping its style-independent semantics, without the support of parallel training corpora. Existing methods struggle to achieve both high style conversion rate and low content loss, exhibiting the over-transfer and under-transfer problems. We attribute these problems to the conflicting driving forces of the style conversion goal and content preservation goal. In this paper, we propose a collaborative learning framework for unsupervised text style transfer using a pair of bidirectional decoders, one decoding from left to right while the other decoding from right to left. In our collaborative learning mechanism, each decoder is regularized by knowledge from its peer which has a different knowledge acquisition process. The difference is guaranteed by their opposite decoding directions and a distinguishability constraint. As a result, mutual knowledge distillation drives both decoders to a better optimum and alleviates the over-transfer and under-transfer problems. Experimental results on two benchmark datasets show that our framework achieves strong empirical results on both style compatibility and content preservation.

## introduction text style transfer is to transform an input text of a source style to a target style (i.e., style conversion goal) without loss of its style-independent information (i.e., content preservation goal). concentrating on different stylistic attributes, text style transfer has attracted much attention from various natural language processing applications, such as personalized machine translation @xcite ), text formalization @xcite , and sentiment translation @xcite . unfortunately, the parallel corpora with aligned input and output are usually unavailable, challenging the models to learn in an unsupervised manner. ## our approach consider a training corpus @xmath0, s i )} n @xmath1, where x i is a text sequence, and s @xmath2. the objective of text style transfer is to learn a conditional probability distribution p (x|x, s) to transform a given x to x with a target style s. the output x is expected to retain the style-independent information in x. here, we stick to the encoder-decoder based sequence-to-sequence architecture, where an encoder e first encodes x to latent vectors e(x), and a decoder d then produces x by sampling from its parameterized distribution d(x|e(x), s). in this paper, we propose a framework with one encoder e and a pair of bidirectional decoders: an l2r decoder d l producing the output sequence from left to right, and an r2l decoder d r going in the opposite direction. the two decoders interact with each other in a collaborative learning mechanism. this collaborative learning mechanism is integrated with a basic framework that follows the non-disentanglement based research line and has three widely used objectives, i.e., selfreconstruction, back-translation, and style classification. in the following, we first briefly introduce the basic framework extended for our two-decoder scenario (section 3.1), then elaborate our collaborative learning mechanism (section 3.2), and present the training algorithm at last (section 3.3). ## conclusion in this paper, we address the unsupervised text style transfer task from a novel multi-agent learning perspective. to overcome the over-transfer and under-transfer problems, we introduce a pair of collaborative bidirectional decoders. our collaborative learning mechanism performs mutual knowledge distillation on the two decoders and guarantees the rationality of this distillation process by a distinguishability constraint together with their opposite decoding directions. quantitative and qualitative results on two benchmark datasets validate the strength of our framework over various single-decoder baselines in achieving both the style conversion goal and the content preservation goal. our code will be made publicly available at https://github.com/ sunlight-ym/cbd_style_transfer . long zhou, jiajun zhang, and chengqing zong. 2019.