T iny S tyler: Efficient Few-Shot Text Style Transfer with Authorship Embeddings (2024)

Abstract: The goal of text style transfer is to transform the style of texts while preserving their original meaning, often with only a few examples of the target style. Existing style transfer methods generally rely on the few-shot capabilities of large language models or on complex controllable text generation approaches that are inefficient and underperform on fluency metrics. We introduce TinyStyler, a lightweight but effective approach, which leverages a small language model (800M params) and pre-trained authorship embeddings to perform efficient, few-shot text style transfer. We evaluate on the challenging task of authorship style transfer and find TinyStyler outperforms strong approaches such as GPT-4. We also evaluate TinyStyler’s ability to perform text attribute style transfer (formal \leftrightarrow informal) with automatic and human evaluations and find that the approach outperforms recent controllable text generation methods.

## introduction text style transfer is the task of transforming a source text to match a target style while preserving its original meaning @xcite @xcite . these target styles can be defined in multiple ways, including around attributes (e.g. formality) or authorship (e.g. barack obama) @xcite . building style transfer systems is complicated by the lack of paired data between different styles @xcite . for tasks like authorship transfer, there may even be limited available data in a target style (e.g. for a non-famous author), which poses an additional challenge @xcite and motivates few-shot approaches. several recent style transfer approaches rely on prompting large language models (llms) @xcite . unlike previous figure 1 : tinystyler uses authorship embeddings from examples of the target style and conditions on these to rewrite source texts to match the target style. we replace expletives above with ' * '. recent controllable text generation approaches present alternatives that rely on smaller models for fine-grained control over stylistic features @xcite @xcite , however, these methods rely on slow sampling procedures or are prone to disfluencies. moreover, these approaches are significantly more complex and cumbersome to use than prompting llms. in this paper, we introduce tinystyler, foot_0 a simple and efficient approach to few-shot text style transfer that harnesses small language models and recent advances on authorship representations @xcite that aim to capture the writing style of an author. tinystyler is trained in an unsupervised fashion over a large, diverse corpus of texts to reconstruct texts from paraphrases by conditioning on an authorship embedding of the original text. at inference time, few-shot style transfer can be performed by conditioning on the authorship embedding of a new, desired target style. inspired by raft @xcite , we further improve model performance by sampling a large number of transferred texts, filtering these texts using automatic metrics, and fine-tuning on the resulting high-quality pairs. the resulting approach enables simple, few-shot style transfer that is both on par with state-of-theart llms and fine-grained control through interpolation of the target style embedding @xcite . in summary, our contributions are as follows: unsupervised style transfer reconstructing text as a framework for performing unsupervised style transfer where no parallel examples between the source and target styles exist is an established pattern used in prior work @xcite @xcite . in this paper, we build on the framework introduced by @xcite , which first uses paraphrasing to neutralize the style of a text and then trains a reconstruction model that learns to re-stylize it. rather than train a model per target style, we perform style transfer to different target styles with a single model by conditioning on a representation of the target style like @xcite . unlike both approaches, we leverage the information captured in strong, pre-trained authorship embeddings @xcite . few-shot style transfer with llms several recent approaches perform style transfer with llms and in-context learning @xcite . @xcite investigates reranking as a method to boost the quality of outputs generated by smaller llms. other methods perform knowledge distillation from larger llms into smaller models @xcite . ## tinystyler our style-transfer approach is built around reconstructing texts. following previous work @xcite @xcite , we use paraphrasing to neutralize stylistic features from texts while preserving their original meaning. in section 3.1, we describe training a single model to reconstruct these texts from both their paraphrases and their pre-trained authorship embeddings. in section 3.2, we describe how we can then transform source texts to arbitrary target styles with the trained reconstruction model by conditioning on new authorship embeddings from texts in a target style. we use this reconstruction model to construct a dataset of high-quality synthetic style transfer pairs. finally, in section 3.3, we perform self-distillation to improve the consistency of our model and remove the reliance on a separate paraphrasing model. we illustrate the tinystyler procedure in figure 2 . ## evaluation we evaluate tinystyler on authorship style transfer and text attribute style transfer (formal ↔ informal). ## conclusion and future work we introduce tinystyler, a fast, simple to use, efficient approach to few-shot style transfer that uses small language models and pre-trained authorship embeddings. the method outperforms strong baselines, including gpt-4, on authorship transfer. the method also outperforms other controllable approaches on formality transfer, and is more competitive with llms. our work highlights the utility of pre-trained standalone authorship embeddings and we look forward to future work on representations that capture more diverse characteristics of text style. additionally, tinystyler showcases the potential value of reranking with automatic evaluation metrics, filtering, and self-distillation as a procedure towards training efficient models that can compete with and close the performance gap with llms. accordingly, we are enthusiastic about the potential for future improvements to automatic text evaluation yielding performance benefits when incorporated into text style transfer and other text generation pipelines.