Exploring Non-Autoregressive Text Style Transfer (2021)

Abstract: In this paper, we explore Non-AutoRegressive (NAR) decoding for unsupervised text style transfer. We first propose a base NAR model by directly adapting the common training scheme from its AutoRegressive (AR) counterpart. Despite the faster inference speed over the AR model, this NAR model sacrifices its transfer performance due to the lack of conditional dependence between output tokens. To this end, we investigate three techniques, i.e., knowledge distillation, contrastive learning, and iterative decoding, for performance enhancement. Experimental results on two benchmark datasets suggest that, although the base NAR model is generally inferior to AR decoding, their performance gap can be clearly narrowed when empowering NAR decoding with knowledge distillation, contrastive learning, and iterative decoding.

## introduction text style transfer (tst) aims at altering a stylistic attribute (e.g., sentiment) of the given text to a target value, without changing the style-agnostic semantics. due to the difficulty in collecting parallel training corpus, most exiting methods @xcite @xcite address the task in an unsupervised setting. through techniques such as auto-encoding, back-translation, or adversarial learning, the task is converted to self-supervised problems and great empirical progress has been made. however, current methods employ autoregressive (ar) decoding which generates each output token conditioned on the previously generated ones, leading to low parallelizability and high latency for pragmatic use. recently, non-autoregressive (nar) decoding has attracted much attention in neural machine translation (nmt) @xcite . nar decoding eliminates the conditional dependencies among the output tokens and generates them in parallel, thus reducing the decoding time-complexity from o(t ) to o(1) for outputs with length t . without modeling the dependency, the advantage on decoding speed comes at the cost of reduced performance. to address this issue, knowledge distillation is employed to transfer the knowledge from ar models to nar models @xcite . furthermore, existing works resort to various regularization techniques @xcite to constrain the output or semi-autoregressive (sar) decoding @xcite as a speedperformance tradeoff. in this paper, we explore nar decoding for unsupervised tst to enable faster inference with better parallelism. to the best of our knowledge, this is the first work to study nar models for tst. firstly, a base nar model is proposed by directly adapting the widely used training objectives from ar models. as with nmt, the base nar model underperforms the ar model. to narrow their performance gap, we propose to enhance nar decoding from three perspectives: the data perspective by knowledge distillation, the regularization perspective by contrastive learning, and the speed-performance tradeoff perspective by iterative decoding. experimental results on a sentiment transfer dataset and a formality transfer dataset demonstrate integrating these techniques can substantially improve the base nar model. ## related work unsupervised text style transfer. one branch of methods disentangles the style and content by learning a style-agnostic representation, which can be either a latent vector @xcite @xcite or a subsequence of the input with the style indicators removed @xcite @xcite @xcite . another branch of methods @xcite inspired by back-translation dynamically creates pseudo-parallel data to gradually refine the tst model. there are also reinforcement learning based methods @xcite @xcite which guide the model with different rewards corresponding to the evaluation criteria. non-autoregressive decoding. since the proposal of nar decoding in nmt @xcite , follow-up works focus on narrowing its gap with ar decoding while keeping its efficiency. one branch of methods transfers the knowledge from ar models to nar models by knowledge distillation @xcite , attention alignment @xcite , imitation learning @xcite , or reinforcement learning @xcite . another branch introduces regularization terms such as similarity constraint @xcite , bag-of-ngram difference @xcite , and aligned crossentropy @xcite to alleviate incorrect translations. furthermore, sar decoding is proposed, which seeks a tradeoff between ar and nar decoding by adding ar layers on nar models @xcite @xcite , iterative refinement @xcite , or insertion-based decoding @xcite . ## non-autoregressive text style transfer let s denote all possible values for a stylistic attribute. a desired tst model p θ (y|x, s) with parameters θ transforms an input text x with source style s 0 ∈ s to an output y with a given target style @xmath0. in this section, we first propose a base nar model (basenar) for unsupervised tst (section 3.1), then investigate three techniques, i.e., knowledge distillation (section 3.2), contrastive learning (section 3.3), and iterative decoding (section 3.4), to enhance the performance of basenar. ## conclusion in this paper, we propose nar decoding for unsupervised text style transfer to pursue faster inference. on top of a base model, we explore how knowledge distillation, contrastive learning, and iterative decoding can narrow the performance gap towards ar decoding.