Reusing Transferable Weight Increments for Low-resource Style Generation (2024)

Abstract: Text style transfer (TST) is crucial in natural language processing, aiming to endow text with a new style without altering its meaning. In real-world scenarios, not all styles have abundant resources. This work introduces TWIST (reusing Transferable Weight Increments for Style Text generation), a novel framework to mitigate data scarcity by utilizing style features in weight increments to transfer low-resource styles effectively. During target style learning, we derive knowledge via a specially designed weight pool and initialize the parameters for the unseen style. To enhance the effectiveness of merging, the target style weight increments are often merged from multiple source style weight increments through singular vectors. Considering the diversity of styles, we also designed a multi-key memory network that simultaneously focuses on task- and instance-level information to derive the most relevant weight increments. Results from multiple style transfer datasets show that TWIST demonstrates remarkable performance across different backbones, achieving particularly effective results in low-resource scenarios.

## introduction text style transfer (tst) is a significant area in natural language processing, aiming to endow text with a new style without altering its meaning. numerous studies have been successfully applied to sentiment transfer @xcite @xcite ), text formalization @xcite , writing style imitation @xcite @xcite , and role-specific dialogue scripts creation @xcite . as a sequence-to-sequence generation task, tst often faces the problem of parallel data scarcity. however, annotating style-specific data is often † corresponding author. ‡ equal contribution. labor-intensive. despite efforts to address this challenge, recent studies still face significant limitations. i) self-supervised pre-training @xcite , leverage large amounts of style corpus for self-supervised pre-training in the latent space, while performances with a lack of creativity or formulaic text easily fall short compared to supervised methods @xcite . ii) in-context learning @xcite , utilize the powerful capabilities of @xcite in few-shot learning, while the stability is affected by high accuracy in prompt design. additionally, there is no guarantee that a single basis of the learned simplex will correspond to a target attribute such as dialect due to a lack of scalability. iii) synthetic data generation @xcite , utilize closed-source models to generate large synthetic datasets, which is hard to guarantee the quality of synthetic data and may lead to bias. these issues undermine the efficacy and hinder the practical applications of tst. this paper aims to exploit the limited and authentic datasets with supervised signal guidance to achieve robust and scalable performance. to this end, we propose a supervised method twist (reusing transferable weight increments for style text generation), which unleashes the potential to extract knowledge from known styles for unseen styles. more specifically, twist is a two-stage framework. in the preparation stage, we employ low-rank adaptation @xcite (lora) to train weight increments for source styles. then, these weight increments containing task-specific knowledge are stored in a source weight pool, which is designed for simultaneously capturing task-and instance-level information. after internal iterations, the weight pool can export the most relevant weight increments in a key-value format. we adaptively handle variable styles by focusing on dual-level authentic information. in the opti-mization stage, twist initializes partial parameters specific to target style by reusing weight increments, thereby reducing data dependency. inspired by @xcite , we involve the weighted summation of parameter matrices to derive appropriate initial weights. in addition, we employ singular value decomposition (svd) to extract a small subset of parameters from source weight matrices, which are then injected into the initialization matrix. we retain the top-q singular values and their corresponding singular vectors, achieving an effect similar to sparse matrices. this reduces interference between weight matrices, making merging more effective. experiments demonstrate the remarkable efficacy and stability of twist on widely recognized benchmarks, achieving state-of-the-art (sota) performance across various backbone models, including t5 @xcite and llama-2 @xcite . compared to other baselines relying on small-scale models, twist based on t5-large demonstrates superior performance with only 10% training data. notably, the performance based on llama-2-7b surpasses other fine-tuned methods and achieves performance comparable to powerful chatgpt4 @xcite . our contributions are summarized as follows: i. we propose a new perspective to address data scarcity in text style transfer. to the best of our knowledge, this is the first work to explore learning general knowledge from diverse source styles for transferring to target styles. ## preliminary we outline the common paradigms for learning a target task and describe our problem setup within the context of these paradigms. ## method our proposed method consists of two stages (depicted in figure 1 ): in the preparation stage, we construct a source weight pool ( § 3.1) to store reusable weight increments. we use a lora-based peft method to train weight increments independently for each source style in § 3.1.1. subsequently, these weight increments are clustered in § 3.1.2 and stored in a specially designed multi-key memory network in § 3.1.3. finally, § 3.1.4 describes a retrieval scheme that considers task-level and instance-level information. in the optimization stage ( § 3.2), we discuss transferring the retrieved information to initialize the model more efficiently. ## related work text style transfer and style categories. tst aims to endow the text with a new style without altering its meaning @xcite . from the holistic definition of style, current work can be divided into two paradigms @xcite . the first paradigm decouples explicit style from content, as in @xcite , which added an adversarial loss function. @xcite randomly dropped input words and reconstructed the input for each author separately. the second paradigm avoids explicitly decoupling style from content. @xcite added extra style embeddings in the input. @xcite employs generative flow techniques to extract stylistic features from instances of styles. previous studies primarily focused on simple style transfer tasks, like formality @xcite or sentiment transfer @xcite . these tasks achieved satisfactory results via word-level transfers. recently, @xcite defined the cross-style as a compound style, such as personality @xcite and writing styles @xcite . ## conclusion this paper addresses the challenges of tst by proposing a novel method called twist, which utilizes transferable weight increments for style text generation. our experiments demonstrate that twist outperforms baselines across various models, achieving state-of-the-art performance and superior adaptability in low-resource scenarios.