Style Transfer with Multi-iteration Preference Optimization (2025)

Abstract: Numerous recent techniques for text style transfer characterize their approaches as variants of reinforcement learning and preference optimization. In this work, we consider the relationship between these approaches and a class of optimization approaches developed primarily for (non-neural) statistical machine translation, formerly known as ‘tuning’. Inspired by these techniques from the past, we improve upon established preference optimization approaches, incorporating multiple iterations of exploration and optimization, and choosing contrastive examples by following a ‘hope’ vs ‘fear’ sampling strategy. Cognizant of the difference between machine translation and style transfer, however, we further tailor our framework with a new pseudo-parallel data generation method and a dynamic weighted reward aggregation method to tackle the lack of parallel data and the need for a multi-objective reward. We evaluate our model on two commonly used text style transfer datasets. Through automatic and human evaluation results we show the effectiveness and the superiority of our model compared to state-of-the-art baselines.

## introduction text style transfer aims to rewrite a given text to match a specific target style while preserving the original meaning. this task has drawn significant attention recently due to its broad range of applications, such as text simplification @xcite , formality transfer @xcite ), text detoxification @xcite , authorship transfer @xcite , and authorship anonymization @xcite . recent approaches have focused on pseudoparallel data generation @xcite and policy optimization @xcite . steer @xcite and astrapop @xcite combine the two and achieve state-of-the-art performance on text style transfer and authorship style transfer, respectively. in this work, we seek to advance the frontier of text style transfer, drawing inspiration from the optimization techniques developed in the era of statistical phrasal machine translation, in which the lack of correlation between the log-linear model objective and the desired evaluation metric, typically bleu @xcite , was observed @xcite . approaches to align foot_0 the two objectives came to be known as tuning, foot_1 beginning with och @xcite , and evolving into online variants @xcite , rank-based approaches @xcite , batch-based approaches @xcite , and several others. tuning methods follow a generate-and-optimize pattern: a model is used to generate multiple candidate hypotheses per input, and then parameters are adjusted such that the argmax according to the model score also maximizes the evaluation metric. in this regard, tuning methods resemble approaches taken in the application of policy optimization algorithms, such as ppo @xcite , to generative language modeling @xcite . more recent algorithms, such as dpo @xcite and cpo @xcite , which replace reinforcement learning (rl) in ppo with preference optimization (po), are reminiscent of the pairwise ranking optimization approach to tuning @xcite . given this close relationship between these approaches, we can consider whether other techniques developed to improve mt tuning could be applied to optimization for style transfer. in this work, we propose style transfer with multi-iteration preference optimization (stamp), a two-phase po training framework, in which we first use supervised fine-tuning to build a reference model from pseudo-parallel data and then train the reference model using po. stamp is similar to steer and astrapop at a high level but is enhanced with two techniques borrowed from mt tuning and two modifications that further adapt it for text style transfer. first, we include multiple iterations of preference pair generation followed by model optimization @xcite , which has already been shown to be effective on other seq2seq tasks such as mathematical and scientific reasoning @xcite @xcite . second, following the hope-and-fear sampling in @xcite , for po, we over-generate outputs using the reference model and construct preference pairs using samples with high model scores and extreme (high or low) task objective scores, in order to avoid dangerous generation and encourage reachable good generation. to improve the quality of the reference model and the balance across the multiple training objectives, we additionally design a new two-step end-to-end pseudo-parallel data generation method and a dynamic reward aggregation method. we evaluate our model on two popular text style transfer datasets, grammarly's yahoo answers formality corpus (gyafc) @xcite and the corpus of diverse styles (cds) @xcite . extensive experiments show that our model outperforms all state-of-the-art baselines on both datasets in both in-domain and out-of-domain evaluation, and demonstrates a higher training efficiency than the strongest baseline. our main contributions are: ## methodology in this section, we formalize the text style transfer task and introduce our training framework, stamp. ## experiments we evaluate stamp on two text style transfer datasets in both in-domain and out-of-domain settings and compare stamp with the state-of-the-art baseline approaches. in this section, we detail the experimental setup and the model implementation. ## results in this section, we present the quantitative experimental results and a qualitative case study. because of the limited resources, we conduct all experiments for a single run and perform t-tests on the results. 13 ## conclusion we present stamp, a multi-iteration preference optimization training framework for text style transfer, in which an end-to-end pseudo-parallel data generation pipeline provides a strong reference model, a preference pair construction strategy improves the effectiveness of po training, and weighted reward aggregation ensures balance across multiple objectives over multi-iteration training. we evaluate stamp on two commonly used text style transfer datasets; demonstrating superior performance over all state-of-the-art style transfer approaches.