Meta-Tuning LLM s to Leverage Lexical Knowledge for Generalizable Language Style Understanding (2024)

Abstract: Language style is often used by writers to convey their intentions, identities, and mastery of language. In this paper, we show that current large language models struggle to capture some language styles without fine-tuning. To address this challenge, we investigate whether LLMs can be meta-trained based on representative lexicons to recognize new styles they have not been fine-tuned on. Experiments on 13 established style classification tasks, as well as 63 novel tasks generated using LLMs, demonstrate that meta-training with style lexicons consistently improves zero-shot transfer across styles. We release the code and data at https://github.com/octaviaguo/Style-LLM.

## introduction the style of a text refers to unique ways authors select words and grammar to express their message @xcite , providing insights into social interactions and implicit communication. the open-ended and ever-evolving nature of style @xcite motivates the need for zero-shot classification, as it is costly to annotate data for every possible style in every language. recent large language models (llms) and their instruction-tuned variants have achieved notable success in zero-shot learning for diverse tasks using prompting strategies @xcite ). yet, their efficacy in style classification has not been thoroughly investigated. as we will show in this paper ( §3.2), style classification remains a challenge for standard llm prompting. on the other hand, before the paradigm in nlp shifted to pretrained language models, lexicons of words that are stylistically expressive were commonly used as important lexical knowledge @xcite in rule-based @xcite , feature-based @xcite , and deep learning models @xcite for style identification. many lexicons have been developed for varied styles, such as politeness @xcite , happiness @xcite , emotions @xcite , etc. this leads to a natural question: can we leverage lexicons during instruction fine-tuning of llms to improve their understanding of language style? in this paper, we examine the effectiveness of fine-tuning llms to interpret lexicons that are provided as inputs to elicit latent knowledge @xcite of language styles that were acquired during pre-training. we first compile a benchmark of 13 diverse writing styles with both annotated test sets and style-representative lexicons. using this benchmark, we show that meta-tuning with lexicons enables different pre-trained llms to generalize better to new styles that have no labeled data. for example, meta-tuning llama-2-7b @xcite on seven styles can improve the average f1 score on a separate set of six held-out styles by 12%, and by 8% over a general instruction-tuned model, llama-2-chat. to further verify the capability of llms to generalize to novel styles using lexicons as the only source of supervision, we generated a diverse set of 63 unique writing styles with examples ( §4) using an approach inspired by self-instruction @xcite . we demonstrate that using a small lexicon of as few as five words can effectively improve generalization to new styles. we found it helpful to replace class names with random identifiers when meta-training models with lexicons, which prevents models from ignoring the lexicons and simply memorizing source styles' class names. in addition, we show that when combined with meta in-context learning @xcite , incorporating lexicons can significantly reduce variance. in summary, our contributions are: (1) we introduce lexicon-based instructions ( §2.1), a simple instruction: classify a sentence as "formal" if its style is similar to the words "albeit, lest, herein, insofar" or as "informal" if its style is similar to the words "imo, kinda, argh, omg". here is the sentence: "i think she is unvirtuous. ## meta-tuning for style generalization we investigate the capabilities of llms to interpret language styles using lexical knowledge, and identify text that is representative of the associated styles. we compare lexicon-based instructions with other methods in the zero-shot setting, and further explore a few-shot setting. to study the effectiveness of meta-tuning with lexicons, in generalizing to various writing styles, we first consider a set of thirteen styles, where high-quality annotated data is available. later, in §4, we further demonstrate the ability of lexicon-instructed models to generalize using 63 novel llm-generated styles. ## results and analysis we report macro-average f1 for style classification tasks following the xslue benchmark @xcite . our experimental results show that lexicon-based instructions can improve the zeroshot style classification performance in all settings, especially when source style meta-tuning and class randomization are involved. ## generalization to novel styles in prior sections, we demonstrated the effectiveness of lexicon meta-tuning on existing style datasets. to demonstrate that our method is able to generalize beyond styles that have been previously studied in the nlp community, we use llms to semiautomatically propose new styles, and then generate instances of text presenting each style (i.e., labeled examples). the new styles generated in this section are then used to evaluate our approach's ability to generalize to styles that include but are not limited to niche literary genres, or rapidly evolving communication styles in social media (see examples in ## related work style classification. research in nlp has studied various language styles. kang and hovy (2021) provided a benchmark for fully-supervised style classification that combines many existing datasets for style classification, such as formality @xcite , sarcasm @xcite , hate/offense (i.e., toxicity) @xcite , politeness @xcite , and sentiment @xcite . other writing styles include but are not limited to readability (i.e., simplicity) @xcite , shakespearean english @xcite , subjectivity @xcite , and engagingness @xcite . despite an extensive range of style classification tasks studied in prior research, zero-shot or cross-style classification is relatively underexplored @xcite . in particular, much of the cross-style research thus far has focused on text generation tasks @xcite , rather than classification. in this study, we aim to address this gap in the literature by concentrating on zero-shot style classification across a collection of diverse styles. language model prompting. large language models, such as gpt-3 @xcite , exhibit impressive zero-shot learning abilities when conditioned on appropriate textual contexts, i.e., prompts, or natural language instructions. since then, how to design appropriate prompts has become a popular line of research @xcite . in this work, we propose to incorporate lexicons into instructions and teach the model to better utilize stylistic lexicon knowledge through instruction tuning. recently, @xcite specified styles in instructions as constraints to improve controlled text generation. parallel to our study, @xcite investigated label descriptions to enhance zero-shot topic and sentiment classification. we focus on style classification, a challenging area in nlp characterized by its extensive scope and complexity, encompassing a wide range of stylistic expression across various domains of text. in order to improve the generalization capabilities of instruction-tuned models, we replace class names in instructions with entirely random words during fine-tuning on training styles. this is similar to @xcite , which indexes and shuffles slot descriptions in prompts for dialogue state tracking. moreover, our work differs from the standard practice in previous studies @xcite @xcite , where a pre-defined set of class names, is equal in size to the number of labels in the associated datasets. ## conclusion & discussion in this work, we study zero-shot style classification using large language models in combination with lexicon-based instructions. experiments show that conventional instructions often struggle to generalize across diverse styles. however, our lexiconbased instruction approach demonstrates the potential to fine-tune models for improved zero-shot generalization to unseen styles. by utilizing a diverse range of data sources, such as wikipedia, captions, social media (