Emulating Author Style: A Feasibility Study of Prompt-enabled Text Stylization with Off-the-Shelf LLM s (2024)

Abstract: User-centric personalization of text opens many avenues of applications from stylized email composition to machine translation. Existing approaches in this domain often encounter limitations in data and resource requirements. Drawing inspiration from the success of resource-efficient prompt-enabled stylization in related fields, this work conducts the first feasibility into testing 12 pre-trained SOTA LLMs for author style emulation. Although promising, the results suggest that current off-the-shelf LLMs fall short of achieving effective author style emulation. This work provides valuable insights through which off-the-shelf LLMs could be potentially utilized for user-centric personalization easily and at scale.

## introduction driven by the trend of using generative ai for on-demand user-centric personalization in recent years, the demand for personalized content has become increasingly pronounced. personalized text, generated by capturing the style of an author, is sought-after in creative content writing, data-to-text generation, email composition as well as machine translation to provide user-specific "naturalness" to text. prior works attempted at replicating an author's writing style by mapping the content to a particular style but remained a challenging task due to the indecipherable and individualistic nature of writing style. llms, with their understanding of natural languages via high-level latent representations, serve as versatile tools for linguistic analysis and manipulation. therefore, they are explored abundantly in various text-based applications such as information retrieval, sentiment analysis, etc. @xcite . in the realm of controllable text-generation, input optimization or prompt engineering has enabled a resource-efficient alternative to modulate llm-generated text by modifying the input prompts @xcite . more recently, this approach has been used to produce valid explanations for various stylistic textual entailment tasks and produced promising results in related fields such as authorship verification and personality prediction @xcite . motivated by these observations, this work performs the first feasibility study into using off-the-shelf pre-trained llms for controllable stylized text generation via prompting for author style emulation. ## related works author style emulation, within personalized text generation, seeks to replicate the distinctive styles of individual human authors. it is often conceived in two ways -text stylization or text style transfer (tst). the former equips a text generator to produce author-stylized text, while the latter independently extracts an author's linguistic preferences (style) and modulates a text generator's semantic content accordingly. early work in this field mainly investigated tst, by attempting to shakerspearize texts using parallel corpora produced with and without shakespearean style @xcite . however, limited availability of parallel corpora for an average author's style hindered progress in this direction @xcite . motivated by the need for non-parallel data, some research works exploring tst for sentiment and formality utilized generative models, such as generative adversarial networks (gans), variational autoencoders (vae) etc., to obtain latent representation corresponding to the style and semantic content separately @xcite . however, this line of research was not as explored for author-stylized text generation as the intricate relationship between an author's linguistic style and semantic content would have made their separation a challenging task. in the recent past, @xcite exploited the capabilities of llms such as gpt2 by finetuning the model on an author's corpus to incorpo7b fine-tuned using the base model on chat and instruct datasets @xcite . mistral 7b instruction fine-tuned model trained on conversation datasets @xcite . incite 7b instruction fine-tuned model on a collection of instruction datasets by redpajama project. note: the texts are generated using respective model's huggingface model repository unless otherwise specified. abbrev.: autor -auto-regressive, sft -supervised fine-tuning, rlft -reinforcement learning fine-tuning, rlhf -reinforcement learning with human feedback, dpo -direct preference optimization rate their style characteristics without the need for a parallel corpus. the idea was to utilize the llms for stylistic rewriting than simple text generation (i.e., writing). however, this method requires large amounts of labeled data with author identities to train a resource-heavy llm such as gpt2. more recently, due to the influx of large number of llms performing competitively on natural language generation and understanding tasks, investigations have been made to utilize off-the-shelf pre-trained llms for related applications such as authorship verification and automatic personality prediction through prompt modulation with promising results @xcite . these results emphasize the capability of the llms to understand and follow instructions through various prompting strategies and guided instructions. specifically, these works instruct the llm to find entailment between a text and a label (for personality prediction) or between two texts (for authorship verification), while also providing explanations. however, to the best of the author's knowledge, no work has been done to extend this concept to generate text with an objective of emulating an author's writing style. ## methodology problem statement: given human-authored text t a by an author a, generate t g using a text generator g that faithfully mirrors the style of the author in t a . the success criterion is for the generator to produce t g such that a proficient author discriminator d attributes it accurately to the original author a. to meet this objective, three key elements come into play: author (a), generator (g) and discrim-inator (d). a signifies the identity label associated with a human-authored text sample t a containing the author's linguistic preferences. additionally, it is assumed that all text samples by a have consistent linguistic preferences that form the author's style signature. g modifies its language generation for different a by learning the author's stylistic preferences from t a independently or by virtue of instructions. finally, d must be capable of accurately differentiating between several authors based on their style signatures. authorship attribution (aa) research focuses on creating robust algorithms to differentiate authors based on their distinct writing styles. therefore, aa can serve as a source for text data with author identity labels, while effective aa algorithms become relevant tools for serving as author discriminators. for the purpose of this study text samples with author identity labels are obtained from 100 authors randomly chosen from a widely recognized aa corpus, the blogs authorship corpus @xcite . the selected authors are ensured at least 100 text samples and a minimum of 500 words in each sample. next, off-the-shelf pre-trained text generators, the llms, are selected. this selection encompasses 12 state-of-the-art (sota) llms, as described in for consistency in text generation across llms, top-k and top-p (nucleus) sampling with @xmath0.95 is used as decoding strategy wherever applicable. the maximum generation length is set to 500 tokens with a minimum requirement of 350 tokens. for each author, 10 text samples are reserved to serve as examples of author's writing style. every generator is equipped with a prompt and an example text with a specific identity. therefore, for each author-generator pair, total 10 text samples are generated. summary statistics for the generated data 1 are provided in aa explores author discrimination at various levels resulting in over 1000 features, ranging from granular features like character n-grams to utilizing contextual features such as bert @xcite . for optimal evaluation of the stylistic alignment between author and llm generated text, it is imperative to consider diverse aa algorithms that capture various aspects of author's style. for each algorithm, training is performed on 90 text samples per author. for the algorithms that require separate training and validation sets, 90% of the author training data is allocated for training while the remaining 10% is reserved for validation. both bertaa and contra-x algorithms are trained for 5 epochs each. the results are reported as average accuracy across 5 training-testing runs. ultimately, assessing the capability of g for emulating a's style necessitates maintaining a and d constant while varying the inputs of g. as prompts serve as inputs to each llm, the distinct compo- ## prompting protocol for stylization following the prompt decomposition technique outlined in giray (2023), a four-part prompt, namely, task, instructions, output indicator and example author text, is designed (see for the purpose of testing, an example author text is categorized as either short or long text, representing the first 50 and 300 words from the original author example, respectively. additionally, two variations of the prompting strategy involves exclusion and inclusion of guided instructions, identinote: text enclosed @xmath1; features emphasized in instructions are linguistically verifiable. evaluation of the author emulation protocols involves testing the trained aa algorithms on both the example author texts and the text generated by each llm. results are presented as average accuracy across authors for each llm in the form of box-and-whisker plot. assuming that the example author texts contain sufficient stylistic information about their respective authors, llms capable of emulating an author's writing style are expected to demonstrate comparable aa performance on the synthetic llm-generated text to that observed on the original author texts. ## discussion results of tep are presented in figure 1 . in short texts, the expected low stylometric information corresponds to relatively low performance across all aa algorithms on author text. notably, gpt-4 outperforms other llms, achieving maximum accuracy close to 72% of the maximum accuracy achievable on author text. being one of the most advanced sota models amongst the chosen llms, in the absence of specific instructions, gpt4 may leverage the knowledge from its training data to identify and replicate relevant linguistic style representations. interestingly, the top performing llms in this protocol -gpt4, llama2 and zephyr -are chat models alluding to the potential of chat models for author style emulation in an unguided scenario. for cep, three scenarios are evaluated by providing progressively more information in the prompt. the results of this protocol are presented in figure 2 . deepseek, a chat model, consistently outperforms all llms across scenarios, showcasing strong potential for author style emulation, followed closely by gpt3 and incite. no clear consensus emerges on the optimal model type for cep. with regards to the first control parameterlength of example author text -an expected significant performance improvement is observed across all aa algorithms between short and long author texts. similarly, most llms benefit from addition of more author data, albeit with varying degrees. considering the second control parameterprompting strategy -most llms exhibit reduced performance while transitioning from simple to directed prompting. this observation may be caused by one or more of the factors discussed below. first, the llm's intrinsic understanding of an author's writing style may be better. each llm is trained on large amounts of training data spanning diverse domains that allow the model to learn intricate linguistic structures of language. given that the llms are trained with a probabilistic objective, aiming to generate the next word that most closely aligns with the preceding context, it is plausible that these models develop an understanding of the author's writing style from the example author text. consequently, constraining the llm to generate text by focusing on the linguistic features highlighted in the directed prompt might interfere with its intrinsic understanding, thereby leading to a degradation in performance. second, limited efficacy of the linguistic features in the prompt in capturing author's style. the directed prompt's linguistic features form a subset of the liwc and writeprints feature sets. the lower performance of aa algorithms using these features suggests two possibilities: their possible absence in the author's text or limited influence on their writing style. however, the superior performance of these algorithms on the author's text samples, compared to the generated texts, suggests the influence of these features on the author's writing style to some extent. nevertheless, considering the subjective nature of an author's linguistic choices, the assumption that a fixed set of linguistic features can universally influence all authors equally might be flawed. therefore, instead of static directed prompting, a more effective approach could involve dynamically prompting llms by considering each author's individual linguistic preferences. finally, some llms may lack the capability to incorporate the specified linguistic features highlighted in the instructions during text generation. to evaluate the similarity of the linguistically verifiable features from the directed instructions between the llm generated texts and author's example texts, a test was performed as described in a.1. as anticipated, the llms exhibit superior alignment to the said linguistic features in longer texts due to increased data availability. further it is noted that llms are most capable of producing higher alignment with linguistic features that reflect the tone, authenticity, analytical aspects, and lexical richness from author texts. however, replicating the lexical diversity and punctuation style proves more challenging. one of the notable observations is the lack of overall linguistic alignment between the author's texts and the best-performing model, deepseek, emphasizing the potential signif-icance of linguistically unverifiable aspects in the instructions for successful emulation. ## conclusion this study explored the use of pre-trained llms to generate author-stylized text through prompted inputs, controlling parameters such as example text length and directed instructions with stylometric information. evaluated against five aa algorithms, the two author emulation protocols (trivial and complex) assessed the feasibility of 12 sota llms for author style emulation. control parameters significantly impacted the llm's emulation capacity, emphasizing the need for user-specific personalized instruction generation. overall, the maximum author emulation performance is only two-thirds that of original texts, highlighting the llm's current limitations for plug-and-play author style emulation. as this task involves a complex interplay between an author identity, text generator and author discriminator, an extended future work will explore how individual author identities and the properties of author discriminator impact the subsequent author style emulation capabilities of a text generator, specifically the llms.