Leveraging Low-resource Parallel Data for Text Style Transfer (2023)

Abstract: Text style transfer (TST) involves transforming a text into a desired style while approximately preserving its content. The biggest challenge in TST in the general lack of parallel data. Many existing approaches rely on complex models using substantial non-parallel data, with mixed results. In this paper, we leverage a pretrained BART language model with minimal parallel data and incorporate low-resource methods such as hyperparameter tuning, data augmentation, and self-training, which have not been explored in TST. We further include novel style-based rewards in the training loss. Through extensive experiments in sentiment transfer, a sub-task of TST, we demonstrate that our simple yet effective approaches achieve well-balanced results, surpassing non-parallel approaches and highlighting the usefulness of parallel data even in small amounts.

## introduction text style transfer (tst) aims to modify the style of a given text while preserving its underlying content @xcite @xcite (see figure 1 ). the limited availability of parallel training data is a major obstacle in tst, as acquiring large-scale aligned datasets for specific style pairs is often impractical or unfeasible @xcite . the only tst study using parallel data and sequenceto-sequence learning known to us by @xcite is a very specific application: converting modern english to shakespeare's style, where extensive aligned paraphrases happen to exist for the purposes of literature research. most recent tst research shifted to using non-parallel datasets and unsupervised learning @xcite @xcite . while it shows promising results, it does suffer a performance penalty and cannot avoid the data problem completely, as large quantities of non-parallel style-specific data are still hard to come by @xcite . in this paper, we address the challenges of tst in low-resource scenarios by proposing methodologies that capitalize on minimal parallel data. due to parallel data availability, we focus on sentiment transfer, a prominent sub-task within the realm of tst @xcite @xcite , in our experiments. 2 however, our model does not rely on a specific kind of textual styles and can be applied to tst in general. in summary, our contributions are (1) building a tst system with low-resource parallel data, (2) applying multiple low-resource adaptation techniques, (3) and a novel style reward approach. this helps us achieve well-balanced results, surpassing previous non-parallel approaches on both automatic and human evaluation. our experimental code is available on github. foot_0 2 related work tst with parallel data tst can be modeled as a sequence-to-sequence task and trained on pairs of texts with similar content but different styles. here, @xcite used a sequence-to-sequence model with a pointer network to translate modern english into shakespearean english. however, this approach to tst is inherently challenging due to the scarcity of parallel data @xcite . non-parallel approaches to tst two main strategies were employed to avoid reliance on parallel data: (1) straightforward text replacement, where style-specific phrases are explicitly identified and replaced @xcite , (2) implicit stylecontent disentanglement via latent representations through techniques such as backtranslation and autoencoding @xcite @xcite @xcite , adversarial learning was shown to improve the results of both approaches @xcite @xcite . despite a lot of progress, non-parallel approaches tend to produce mixed results and often require large amounts of non-parallel data, limiting their practical applicability @xcite . ## method our work sits between the parallel and non-parallel approaches, using parallel data but in very small amounts, in order to maximize performance while minimizing annotation costs. we build on transfer learning by finetuning a pretrained bart model on our task @xcite . we further explore five techniques aimed at this low-resource scenario: hyperparameter tuning: as the effectiveness of transformer models on low-resource data highly depends on hyperparameters @xcite , we adapt our model, focusing on dropout regularization @xcite and label smoothing @xcite . prompt-guided generation: to align the style transfer finetuning with pre-training, we adopt using textual prompts, following @xcite and @xcite . by adding prompts like "pos:" for positive sentences and "neg:" for negative sentences, we provide explicit guidance to the decoder during fine-tuning. ## evaluation & results we evaluate three main dimensions: style transfer accuracy, content preservation, and fluency. we measure sentiment accuracy using distil-bert @xcite finetuned for sentiment analysis on the sst-2 dataset @xcite . 6 following prior work @xcite , we evaluate content preservation using bleu score @xcite and embedding similarity @xcite against the input sentences. we use sentence-bert @xcite and cosine similarity for the embedding similarity. we use gpt-2's @xcite perplexity to estimate fluency. we also run a small-scale in-house human evaluation on a random sample of 100 sentences from the test set (50 for each direction -positive-to-negative and negative-to-positive). outputs are rated on a 5-point likert scale for style transfer accuracy, content preservation, and fluency. ## conclusion we showed that leveraging minimal parallel data in text style transfer can yield a good balance of style transfer accuracy, content preservation, and fluency. standard low-resource approaches (hyperparameter tuning, data augmentation, self-training), improve results, while further improvement is achieved by using style classifier rewards. in future research, we plan to extend the range of low-resource techniques used and apply our approach to other style transfer tasks.