An Empirical Study on Multi-Task Learning for Text Style Transfer and Paraphrase Generation (2020)

Abstract: The topic of this paper is neural multi-task training for text style transfer. We present an efficient method for neutral-to-style transformation using the transformer framework. We demonstrate how to prepare a robust model utilizing large paraphrases corpora together with a small parallel style transfer corpus. We study how much style transfer data is needed for a model on the example of two transformations: neutral-to-cute on internal corpus and modern-to-antique on publicly available Bible corpora. Additionally, we propose a synthetic measure for the automatic evaluation of style transfer models. We hope our research is a step towards replacing common but limited rule-based style transfer systems by more flexible machine learning models for both public and commercial usage.

## introduction the goal of text style transfer (st 1 ) is to convert the input sentence into an output, preserving the meaning but modifying the linguistic layer (grammatical or lexical). style transfer is extensively studied in academic papers @xcite @xcite and also gains popularity in commercialized chatbots. amazon alexa introduced styles mimicking celebrities that replace the original alexa voice @xcite . samsung offers applications personalized in terms of style, e.g. celebrity alarm @xcite . both examples offer novel user experience, though the number of available system responses seems to be limited. the two main constraints are voice generation and text content limitations. while voice generation can be implemented with voice synthesis systems, like e.g. by @xcite or @xcite , content limitation might be resolved by flexible machine learning text st methods: the system could transform neutral answer into one of predefined styles using a machine learning model. the major challenges are the limited amount of style data and the lack of convincing automatic evaluation measures. in our study we try to mitigate these two issues. the goal of our paper is to present an efficient method to train domain-unlimited st models using a small style dataset. inspired by the successful outcomes of multi-task learning @xcite @xcite , we propose a transformer model @xcite that jointly solves paraphrase generation and style transfer tasks. we hypothesize that training in the multitask mode on an english large parallel corpus for paraphrasing may help preserve the input content along with successful adjustment of vocabulary and grammar to the target style, even using a small st corpus. to verify the hypothesis and add practical value, we perform detailed tests on various sizes of text st corpora to verify how their volume affects the results. this is a convenient approach, because we have publicly available large parallel corpora for paraphrasing, but rather small parallel corpora of style transformations we want to achieve. we perform our experiments using openly available paraphrase sources @xcite @xcite @xcite and two st corpora, one for each task: the internal "cute person" style corpus and the processed bible texts corpus composed of publicly available sources. we examine two style transformations: neutral-to-cute and modern-to-antique (the latter one on different bible translations). besides studies with various data volumes, we propose a method of creating an automatic measure. we show that simple common measures do not work if separated and instead we propose a fitted compound measure. traditionally, the paraphrasing task involved rule and dictionary-based approaches @xcite @xcite . another popular method was the statistical paraphrase generation that recombined words probabilistically in order to create new sentences @xcite @xcite . currently, dnns are used for automatic paraphrasing, e.g. by sequence-to-sequence (seq2seq) models @xcite . presumably, the first paper presenting a deep learning model for the paraphrase task is the one by @xcite . the authors successfully compared their residual lstm to previous lstm-derived seq2seq models @xcite @xcite . in parallel, upon research on variational autoencoders (vaes), e.g. @xcite , other approaches to paraphrasing emerged @xcite . more recently @xcite implemented paraphrase generation with deep reinforcement learning that proved better performance than the previous seq2seq results on twitter @xcite and quora @xcite datasets. insufficient corpora for paraphrase generation gave motivation to new practical studies on unsupervised approach @xcite . recently, @xcite proposed a monolingual system for paraphrasing (without translation) and compared it to the unsupervised and translation methods, presenting various linguistic characteristics for each of them. ## experiments using our st methods we performed two tasks: n2c and m2a. for each task we prepared a number of target (style) parallel corpora differing only in volumes. for the n2c transformation it was 1k, 3k, 5k, 7k, 10k, 13k, and 17k. for the m2a transformation we prepared the same corpora volumes plus the additional 30k. the proportions for training and validation were equal for all the target datasets with the ratio of 80%:20% respectively. the style corpora are described in subsections 4.1.2 and 4.1.3. as supplementary data, we used much larger parallel paraphrases corpora described in subsection 4.1.1. firstly, paraphrase data is required for producing high quality sentences. secondly, the generated utterances must be properly tuned to the target style. in our experiments we searched for answers to the following questions. ## conclusions we discussed the method of text style conversion with a multilingual transformer trained for two tasks: paraphrasing and style changing. we showed a successful approach using a large paraphrase parallel corpus with much less data of neutral-target style pairs. in our numerical experiments models trained with varying sizes of style samples were evaluated with four human scores (and their average), revealing nonlinear dependencies. for both studies, neutral-to-cute and modern-to-antique, we pointed out essential data volumes in models that brought acceptable and good results. in particular, we indicated the meaningful model performance growth between 1k and 3k sizes of target data. moreover, the transfer learning ability of our multilingual generator was tested with a satisfying outcome for neutral-to-bible transformation (not seen during the model training). finally, we proposed an easy method to automatically measure style transfer results and to approximate average human score with it. a new measure can be used during model training to estimate style transformation quality, besides checking a typical minimum of cross entropy loss function. in our opinion, this opportunity is worth further research.