Customizing Large Language Model Generation Style using Parameter-Efficient Finetuning (2024)

Abstract: One-size-fits-all large language models (LLMs) are increasingly being used to help people with their writing. However, the style these models are trained to write in may not suit all users or use cases. LLMs would be more useful as writing assistants if their idiolect could be customized to match each user. In this paper, we explore whether parameter-efficient finetuning (PEFT) with Low-Rank Adaptation can effectively guide the style of LLM generations. We use this method to customize LLaMA-2 to ten different authors and show that the generated text has lexical, syntactic, and surface alignment with the target author but struggles with content memorization. Our findings highlight the potential of PEFT to support efficient, user-level customization of LLMs.

## introduction language models, especially ones trained to be "human-aligned" and conversational, are increasingly being used to help people write, including for student essays @xcite , screenplays @xcite , stories @xcite , and science communication @xcite . nearly all of these applications rely on chatgpt, gemini, or other llm instances trained by large companies and shared across all users. concerns have been raised that this reliance on a handful of llms is leading to a homogenization of language @xcite , hindering students from developing their own writing styles @xcite and introducing cultural and linguistic biases that fail to reflect diverse backgrounds @xcite . thus, in this paper, we are interested in how language model generations can be customized to the writing styles of individual users. our focus is on writers who already have some 1k-50k tokens of prior work (which, in an education setting, could be the writing of an author they are learning to emulate). our method aims to create customized llms that adopt the idiolect of the target writer while retaining the ability to understand and follow natural language instructions. in addition, users should have the choice of whether customization includes learning "content" words such as named entities that are present in the source data. in the past, when llms were smaller, it was common to control the style of generations via finetuning on data within the target style, as @xcite do with two romantic poets, and van @xcite do with npc dialogue. however, full model finetuning is untenable for today's state-of-the-art language models. more recently, model customization has been performed via prompt engineering-prefixing a user's query to the model with a set of instructions or exemplars of the target style that is intended to guide the model's outputs @xcite . the success of this technique heavily relies on the prompt's structure @xcite and whether the model's training data contains similar instructions. also, an author's prior work may be too large to fit into most llms' maximum context lengths. in contrast to prior approaches, we explore whether a model's generation style can be altered via small amounts of finetuning, using parameter-efficient finetuning (peft) methods such as lora @xcite . peft is a promising direction for model style customization because it eliminates finicky prompt engineering and is efficient to use. we introduce styletunedlm, a novel approach that leverages lora for efficient finetuning of llms to generate text in specific writing styles. we compare styletunedlm with prompt engineering and few-shot learning approaches, showing it is more effective at capturing the style of training data. we also tackle two challenges with tuning on unstructured data-preserving instruction-following ability after finetuning and learning style signifiers without learning content words. ## methods styletunedlm we build our method by finetuning lora adapters for the pre-trained llama-2-7b model @xcite on unstructured text datasets from specific authors, using a nexttoken prediction objective. the goal is to tailor the model's output to reflect specific stylistic characteristics while maintaining the capabilities learned in prior training. finetuning details can be found in appendix c. for style-following generation examples, see https://cauchy221.github.io/researchstyletunedlm-demo/ . baselines in our fewshot baseline, we prompt llama-2-7b with 5 or 10 randomly selected 256token excerpts from the target author before asking it to generate continuation given a prompt. in our instruct baseline, we use llama-2-7b-chat, a variant of the llama-2-7b finetuned to be conversational @xcite . we prompt with the target author's name and an instruction to generate a continuation in the writing style. we post-process model outputs to remove irrelevant phrases like "please tell me if you have further questions." masking out named entities users of customized llms ought to be able to control the extent to which their custom model learns words associated with content, rather than style. in our work, we examine whether certain classes of words, such as names, can be excluded from the learning process. we first use spacy @xcite to annotate each token position with whether it corresponds to a person's name. during finetuning, we set the attention_mask to 1 while changing their labels to -100 in the loss calculation. this method could be applied to any class of words that a user prefers the model not to learn. ## experimental design author dataset ideally, we would evaluate corpora from authors not present in the training data, as this best reflects the target users of customized models. however, since most llms do not disclose their pre-training data, we conduct an imperfect evaluation using the works of ten authors from project gutenberg (gerlach and font-clos, 2018). a.1 provides a detailed introduction to each author. we collect all available books from each author and randomly divide them into training, validation, and test datasets. the training and validation sets are used for model finetuning and selection, while the test set is reserved for generative tasks used in our evaluation. notably, a book assigned to one dataset does not appear in the others. ## experiment results perplexity ppl of the pre-trained and the corresponding finetuned model are depicted in figure 1 . the finetuned models consistently exhibit lower perplexity on the validation sets for each author than the base llama-2-7b. across all authors, we see an average ppl reduction of 7.0%. we see the greatest improvement (13.6%) for sr-as an 18th century writer, his language differs the most from the modern english llama was trained on. style-embedding alignment figure 2 illustrates the average cosine similarity between the generated text and author embeddings, with our method achieving the highest average similarities across all authors. in contrast, instruct exhibits inconsistencies and difficulties with complex styles. detailed author-specific performance and confusion matrices from our classifier are available in d.3. we also show the classifier accuracy of each method in ## conclusion in this work, we introduce styletunedlm, a novel approach leveraging parameter-efficient finetuning (peft), aiming to tailor large language models to individual users' stylistic preferences without extensive computational resources. our results demonstrate that styletunedlm effectively aligns model outputs with specific stylistic features of different authors, offering significant improvements over traditional methods such as few-shot learning and prompt engineering. we also explore the impact of training data size, content control with masking, and enabling instruction-following capability by merging lora modules. future work should conduct additional analysis with writings confirmed to be outside the pretraining corpus to test the generalizability and adaptation capabilities. furthermore, as we enhance the integration of style and instruction-following modules, developing more refined methods to balance and specify the influence of each component will be crucial for optimizing performance and utility.