Unsupervised Text Style Transfer with Padded Masked Language Models (2020)

Abstract: We propose Masker, an unsupervised text-editing method for style transfer. To tackle cases when no parallel source–target pairs are available, we train masked language models (MLMs) for both the source and the target domain. Then we find the text spans where the two models disagree the most in terms of likelihood. This allows us to identify the source tokens to delete to transform the source text to match the style of the target domain. The deleted tokens are replaced with the target MLM, and by using a padded MLM variant, we avoid having to predetermine the number of inserted tokens. Our experiments on sentence fusion and sentiment transfer demonstrate that Masker performs competitively in a fully unsupervised setting. Moreover, in low-resource settings, it improves supervised methods’ accuracy by over 10 percentage points when pre-training them on silver training data generated by Masker.

## introduction text-editing methods @xcite @xcite , that target monolingual sequence transduction tasks like sentence fusion, grammar correction, and text simplification, are typically more dataefficient than the traditional sequence-to-sequence methods, but they still require substantial amounts of parallel training examples to work well. when parallel source-target training pairs are difficult to obtain, it is often still possible to collect nonparallel examples for the source and the target domain separately. for instance, negative and positive reviews can easily be collected based on the numerical review scores associated with them, which has led to a large body of work on unsupervised text style transfer, e.g., @xcite @xcite . the existing unsupervised style transfer methods aim at transforming a source text so that its style matches the target domain but its content stays otherwise unaltered. this is commonly achieved via text-editing performed in two steps: using one model to identify the tokens to delete and another model to infill the deleted text slots @xcite @xcite . in contrast, we propose a more unified approach, showing that both of these steps can be completed using a single model, namely a masked language model (mlm) @xcite . mlm is a natural choice for infilling the deleted text spans, but we can also use it to identify the tokens to delete by finding the spans where mlms trained on the source and the target domain disagree in terms of likelihood. this is inspired by the recent observation that mlms are effective at estimating (pseudo) likelihoods of texts @xcite . moreover, by using a padded variant of mlm @xcite , we avoid having to separately model the length of the infilled text span. to evaluate the proposed approach, masker, we apply it to two tasks: sentence fusion, which requires syntactic modifications, and sentiment transfer, which requires semantic modifications. in the former case, masker improves the accuracy of state-of-the-art text-editing models by more than 10 percentage points in low-resource settings by providing silver data for pretraining, while in the latter, it yields a competitive performance compared to existing unsupervised style-transfer methods. ## method our approach to unsupervised style transfer is to modify source texts to match the style of the target domain. to achieve this, we can typically keep most of the source tokens and only modify a fraction of them. to determine which tokens to edit and how to edit them, we propose the following three-step approach: ## experiments we evaluate masker on two different types of tasks: sentence fusion and sentiment transfer. for both experiments, we only apply masker once to edit a single span of at most four tokens, since the required edits are often local. foot_2 ## related work section 1 provides a high-level overview of the related work. closest to this work is the ac-mlm sentiment transfer method by @xcite . this method first identifies the tokens to edit based on n-gram frequencies in the source vs. target domain (as proposed by @xcite ) and based on lstm attention scores (as proposed by @xcite ). then it replaces the edited tokens using a conditional mlm. in contrast to their work, our approach leverages the same mlm for both identifying the (possibly empty) span of tokens to edit and for infilling the deleted span. moreover, our padded mlm determines the number of tokens to insert without having to pre-specify it. in that sense, it is similar to the recently proposed blank language model @xcite . in addition to the two applications studied in this work, it would be interesting to evaluate masker on other style transfer tasks. tasks for which unsupervised methods have recently been developed include formality transfer @xcite , lyrics style transfer @xcite ), text simplification @xcite , and sarcasm generation @xcite . ## conclusions we have introduced a novel way of using masked language models for text-editing tasks where no parallel data is available. the method is based on training an mlm for source and target domains, identifying the tokens to delete by finding the spans where the two models disagree in terms of likelihood, and infilling more appropriate text with the target mlm. this approach yields a competitive performance in fully unsupervised settings and substantially improves over previous works in lowresource settings.