STEER : Unified Style Transfer with Expert Reinforcement (2023)

Abstract: While text style transfer has many applications across natural language processing, the core premise of transferring from a single source style is unrealistic in a real-world setting. In this work, we focus on arbitrary style transfer: rewriting a text from an arbitrary, unknown style to a target style. We propose STEER: Unified Style Transfer with Expert Reinforcement, a unified frame-work developed to overcome the challenge of limited parallel data for style transfer. STEER involves automatically generating a corpus of style-transfer pairs using a product of experts during decoding. The generated offline data is then used to pre-train an initial policy before switching to online, off-policy reinforcement learning for further improvements via fine-grained reward signals. STEER is unified and can transfer to multiple target styles from an arbitrary, unknown source style, making it particularly flexible and efficient. Experimental results on a challenging dataset with text from a diverse set of styles demonstrate state-of-the-art results compared to competitive baselines. Remarkably, STEER outperforms the 175B parameter instruction-tuned GPT-3 on overall style transfer quality, despite being 226 times smaller in size. We also show STEER is robust, maintaining its style transfer capabilities on out-of-domain data, and surpassing nearly all baselines across various styles. The success of our method highlights the potential of RL algorithms when augmented with controllable decoding to overcome the challenge of limited data supervision.

## introduction style transfer has been widely explored in the nlp field due to its practical applications, such as making text more formal @xcite , figure 1 : an overview of unified style transfer. in standard style transfer, models can only transfer from a single source style to a specified target style, struggling to transfer from out-of-domain texts. in contrast, unified style transfer models can transfer from an arbitrary source style to multiple target styles. we focus on arbitrary style transfer, a many-toone style transfer task, where the goal is to transfer text from an arbitrary, unknown style to a target style using a single model @xcite . this is a challenging task mainly due to the lack of large-scale, human-curated corpora for training. furthermore, we design a framework for training a unified, many-to-many style transfer model, which can do arbitrary style transfer to multiple target styles, as shown in figure 1 . to circumvent the lack of supervised data, recent approaches @xcite heavily rely on large language models like gpt-neo @xcite and @xcite in zero or few-shot settings. though promising and convenient, these approaches are limited by the high cost of api calls (openai, 2023) and lack of reproducibility due to over-reliance on llms (dean, 2023). our method enhances the effectiveness of smaller, more accessible models for style transfer, broadening their adaptability and utility for the wider community. in this work, we present unified style transfer with expert reinforcement ( steer), a novel, unified framework for many-to-one style transfer without supervision. starting with a non-parallel corpus of text with various styles and a general paraphraser model, steer first creates a diverse, pseudo-parallel dataset of style transfer pairs using product-of-experts decoding @xcite . this makes our framework efficient by eliminating the need for costly human-curated datasets. next, steer uses offline reinforcement learning (rl) with this data before switching to online, off-policy rl for further improvement. to reflect the varied properties of style transfer, we adapt the quark algorithm @xcite , incorporating multiple reward models associated with different aspects such as style strength, fluency, and meaning similarity. our framework is both practical and flexible, enabling a single model to transfer arbitrary source styles to multiple target styles. we apply steer to a diverse dataset of 11 styles @xcite , developing a unified style model capable of transferring text from any of the 11 styles to any other style in the corpus. our final model is effective at transferring style while preserving fluency and semantic similarity for all source and target styles, beating strong baselines across a suite of automatic metrics for style transfer. in particular, across all styles our 775m parameter model beats all baselines in overall style transfer quality, including the instruction-tuned 175b parameter gpt-3 model @xcite . finally, we showcase the robustness of our model through evaluation on two out-of-domain source styles that are unseen during training, where steer consistently outperforms almost all baselines for every target style. the success of steer demonstrates the effectiveness of reinforcement learning abetted by a high-quality, offline dataset in lieu of a good initial policy. ## unified style transfer with expert reinforcement we introduce steer, a novel two-stage framework for unsupervised unified style transfer. our framework is illustrated in figure 2 and is composed of 1) expert-guided data generation to circumvent the challenge of obtaining supervised datasets at scale, and 2) offline reinforcement learning followed by online reinforcement learning to effectively align an initial policy with multiple reward functions related to the style transfer task. in expert-guided data generation ( §3.1), the goal is to automatically collect a diverse high-quality dataset d f of style transfer pairs using only a general paraphraser m p and a corpus of diverse styles c. to this end, we follow an overgenerate-andfilter approach: we first generate a large pool of candidate pairs from the paraphraser guided by style expert models in a product-of-expert fashion @xcite , then leave only pairs that qualify for the style transfer task (i.e., accurately transferred style and semantically similar pairs). in online off-policy reinforcement learning ( §3.2), we first update the paraphraser m p as an initial policy using supervised learning on the collected dataset and then switch to online, off-policy learning for further data exploration and model improvements @xcite . figure 2 : an overview of steer. we first use expert-guided data generation to automatically generate candidate style-transfer pairs x @xmath0, mapping from an input of arbitrary style x si to a rewrite x st in a target style, by decoding with a product of experts using a paraphraser m p and style-expert lms. after filtering by quality metrics, we have a diverse, high-quality dataset d f . we then train a unified, many-to-many style transfer model, using d f for offline rl before switching to online, off-policy rl to further optimize style transfer quality. ## experiments we detail our experiment setup, including the datasets ( §4.1), baselines ( §4.2), evaluation metrics ( §4.3), experimental details ( §4.4), main results ( §4.5), ablations ( §4.6), and analysis of d f ( §4.7). ## related work style transfer due to the absence of large-scale parallel corpora for text style transfer (tst), prior work has focused on unsupervised methods designed for non-parallel datasets @xcite . most of these efforts focus on disentangling the representation of content and the style of a given text, either through an auxiliary discriminator to classify text attributes @xcite , or by training with a policy gradient @xcite . recent work has leveraged the generation capabilities of lms for tst: @xcite , create a pseudo-parallel corpus by paraphrasing text from a style, then training an inverse paraphraser to convert text to that style. other work automatically align pairs of sentences in different styles, either in the representation-level @xcite or corpus-level @xcite . others have attempted tst by prompting lms @xcite . how-ever, these approaches often rely on a strong initial model, either already fine-tuned on tst-related tasks (e.g. paraphrasing), or a large lm capable of few-shot generalization. in contrast, our framework does not assume strong capabilities of the initial model, making it applicable in a realistic setting. rl for nlp recent work has shown the potential of rl to align with arbitrary natural language objective functions across areas such as summarization @xcite , open-ended text generation @xcite , dialogue @xcite , question-answering @xcite , machine translation @xcite , and dataset generation @xcite . for unified style transfer, a setting where the desired output can be directly correlated to automatic metrics, rl is a promising avenue. data generation with lms lm-generated data have been increasingly used across a wide range of tasks, such as commonsense reasoning @xcite , nli @xcite and dialogue generation @xcite . while previous approaches rely on the task-solving capability of llms, recent work show that small lms can also generate high-quality datasets without supervision @xcite . building on top of these, our work pushes further on machine-generated data by incorporating 1) inference-time decoding algorithms and 2) targeted filtering, yielding an effective pseudo-parallel corpus to initialize offline reinforcement learning. ## conclusion we propose steer, a unified framework to overcome the challenge of limited parallel data in style transfer, by leveraging expert-guided decoding and two-stage reinforcement learning. we focus on a more realistic use case: rewriting text from an arbitrary, unknown style to a desired target style. through extensive experiments, we demonstrate the effectiveness and robustness of steer on both in-and out-of-domain style transfer, outperforming competitive baselines. the success of steer underscores the potential of rl algorithms when combined with controllable decoding and encourages future algorithmic innovation that fully unleash the power of rl for real-world nlp applications.