S tyle R emix: Interpretable Authorship Obfuscation via Distillation and Perturbation of Style Elements (2024)

Abstract: Authorship obfuscation, rewriting a text to intentionally obscure the identity of the author, is important yet challenging. Current methods using large language models (LLMs) lack interpretability and controllability, often ignoring author-specific stylistic features, resulting in less robust performance overall.To address this, we develop StyleRemix, an adaptive and interpretable obfuscation method that perturbs specific, fine-grained style elements of the original input text. StyleRemix uses pre-trained Low Rank Adaptation (LoRA) modules to rewrite inputs along various stylistic axes (e.g., formality, length) while maintaining low computational costs. StyleRemix outperforms state-of-the-art baselines and much larger LLMs on an array of domains on both automatic and human evaluation.Additionally, we release AuthorMix, a large set of 30K high-quality, long-form texts from a diverse set of 14 authors and 4 domains, and DiSC, a parallel corpus of 1,500 texts spanning seven style axes in 16 unique directions.

## introduction authorship obfuscation, the act of rewriting text to conceal the author, is an important method for preserving the privacy of authors in sensitive contexts, e.g., anonymous discussion forums, double-blind reviews, and health services. however, it is inherently complex, requiring a substantial change in writing style to obscure the author's identity while also preserving the original content and fluency. historically, authorship obfuscation methods have manipulated aspects of an author's style to obfuscate the original text @xcite @xcite . these techniques typically use style aspects that are easy to automatically evaluate, such as text length, capitalization frequency, and punctuation, to alter the original text. however, these rule-based methods are often too rigid and lead to a degradation of fluency and grammaticality @xcite . recent work demonstrates strong obfuscation performance using llms @xcite @xcite , but the common challenge among these is a relative lack of interpretability and controllability on the obfuscation; these approaches do not incorporate any author-specific stylometric characteristics of the original author, leading to more generalized and ineffective obfuscations. for example, a method that relies solely on increasing language model fluency might effectively obfuscate more informal writing, but not formal writing. to address this gap, we introduce styleremix, an interpretable, inferencetime, author-specific obfuscation method that combines the fluency and steerability of llms with author-specific style information. styleremix first detects unique stylistic elements of the author, either through automatic processes or manually. it then uses this information during obfuscation by integrating style-specific adapters with a base language model (an llm) to guide the generated text away from the author's original style. styleremix avoids high computational costs by utilizing pre-trained low rank adaptation modules (lora; @xcite , which we train to rewrite inputs towards specific directions on various stylistic axes (e.g., more/less length, more/less formality, higher/lower grade level). drawing inspiration from the process of creating a remix, where musical elements of a song, such as tempo, key, and instrumentation are adjusted to form an entirely new track, in this work we seek to identify and manipulate different elements of an authorship style, and propose a simple yet effective approach to steer different components of the text with lora adapters. our results show that styleremix outperforms state-of-the-art authorship obfuscation methods and instruction-based models of similar and larger sizes. additionally, our method has the added benefit of explainability and is customizable to any unique authorship style. we make the following contributions: ## methods styleremix is an obfuscation method that leverages style elements to adapatively rewrite texts. specifically, it incorporates information about the style of the original author to guide the obfuscation process. figure 1 illustrates this new approach, which consists of two phases. the pre-obfuscation phase, conducted only once regardless of the number of authors, involves creating a diverse training set for each style axis we aim to modify (e.g., length variations, formality levels, grade level adjustments, etc.). these style-specific datasets are then used to train low-rank adaptation (lora) adapters, which are low-parameter modules that can be seamlessly integrated with a larger base model to guide text generation along specific style axes. in the obfuscation phase, users can choose the style axes that most effectively disguise the original author's style, either automatically or manually. the selected pre-trained lora adapters are then used to steer the obfuscated text generation. ## related work authorship obfuscation methods traditional authorship obfuscation methods leverage stylometric insights, such as author invariant features, to obfuscate texts @xcite @xcite . however, these methods have been shown to have issues with grammar and fluency due to their strict rulebased approach @xcite . to reduce this behavior, model-based approaches have been developed, such as mutant-x, a genetic algorithm which utilizes an internal classifier to iteratively "mutate" a sentence @xcite . later work improves on this with an ensemble of classifiers rather than a single one @xcite or via variational autoencoders as the base model to generate deferentially private generations @xcite . most recently, @xcite demonstrate the efficacy of smaller llms for authorship obfuscation through over-generation and filtering. however, this method's reliance on a heavy decoding algorithm to generate diverse candidates makes it impractical. some obfuscation methods have also incorporated authorship information @xcite . although these both showed promise, they require extensive training and are only applicable in specific use cases. parameter efficient learning parameterefficient adapters, small modules tuned on top of a frozen large model for effective transfer learning, have been proposed for vision @xcite and nlp @xcite . others have extended these methods by tuning specific layers and embeddings (li and liang, 2021; @xcite , or by making the adapter matrices an addition to the original model weights themselves rather than additional injected layers @xcite . adjacent to parameter-efficient training strategies are model merging techniques, which seek to integrate model knowledge by combining their weights (matena and raffel, 2021); this is efficient and prevents additional inference cost. merging has been explored extensively in previous work, to combine diverse, targeted domain models @xcite , or over the same model trained with different seeds or hyperparameters to improve robustness @xcite . model merging has even been explored with parameter-efficient adapters like lora @xcite . other lines of work expand on merging techniques, creating strategies beyond simply averaging model weights. @xcite @xcite . controllable generation previous work introduces methods to control the content of a generation @xcite or steer the style of the generation @xcite . however these types of controllable generation are less practical for authorship obfuscation, which requires a steerability of the content and the style. style transfer style transfer techniques have utilized both simple models @xcite and more advanced machine learning models @xcite . most approaches depend on training generation models with a dataset, which can be natural or synthetic @xcite . these methods also interpret "style" in various ways, ranging from comprehensive notions like specific authors (e.g., shakespeare and hemingway) @xcite to particular stylistic elements (e.g., formality and sentiment) @xcite . notably, recent, effective style transfer techniques resemble the approach we use, which involves fine-tuning a llm @xcite . ## conclusion in this work, we introduce styleremix, a novel and interpretable method for authorship obfuscation. by targeting specific fine-grained stylistic elements and leveraging low rank adaptation (lora) modules, styleremix provides a more interpretable and controllable approach than existing methods based on large language models or other state-of-the-art techniques while still maintaining excellent performance. we show our new method outperforms a suite of strong, state-of-theart baselines in four diverse domains overall in both automatic and human evaluation. additionally, as part of this work, we release a new authorship attribution dataset, authormix. this dataset includes two new domains: presidential speeches and fiction books, which were carefully selected to ensure a high degree of topic matching, thereby enriching the dataset's applicability and depth. furthermore, we develop disc, a collection of 16 parallel, human-validated datasets spanning various stylistic dimensions, which can be employed in future research to further explore and refine the nuances of stylistic text manipulation. these resources aim to advance the field of authorship obfuscation and provide a solid foundation for subsequent studies.