Paraphrase Types Elicit Prompt Engineering Capabilities (2024)

Abstract: Much of the success of modern language models depends on finding a suitable prompt to instruct the model. Until now, it has been largely unknown how variations in the linguistic expression of prompts affect these models. This study systematically and empirically evaluates which linguistic features influence models through paraphrase types, i.e., different linguistic changes at particular positions. We measure behavioral changes for five models across 120 tasks and six families of paraphrases (i.e., morphology, syntax, lexicon, lexico-syntax, discourse, and others). We also control for other prompt engineering factors (e.g., prompt length, lexical diversity, and proximity to training data). Our results show a potential for language models to improve tasks when their prompts are adapted in specific paraphrase types (e.g., 6.7% median gain in Mixtral 8x7B; 5.5% in LLaMA 3 8B). In particular, changes in morphology and lexicon, i.e., the vocabulary used, showed promise in improving prompts. These findings contribute to developing more robust language models capable of handling variability in linguistic expression.

## introduction it's not what you say it's how you say it. ## related work prompt-based learning has become a new trend, where pre-trained language models perform prediction using a template description of the intended task, and the model derives the necessary information without the need for gradient updates @xcite . prompt-based learning has advantages over traditional supervised learning, as the same pre-trained llm can perform different unseen tasks without fine-tuning. early prompt tuning methods focused on integrating trainable continuous prompt embeddings to perform various nlu tasks @xcite @xcite . discrete prompts representing actual tokens have become more popular than continuous embeddings in prompt engineering, arguably because they are more accessible and linguistically interpretable. discrete methods used cloze-style phrases and differentiable prompts to improve few-shot learning capabilities @xcite . @xcite proposed auto-prompt, which uses a masked language model to find variations of prompts and has been used for various downstream tasks such as plagiarism detection @xcite . @xcite introduced automatic prompt engineer, which searches over a pool of prompt candidates proposed by a second llm. other work focused on generating prompts for knowledge extraction from llms @xcite or investigated instruction induction and model tuning with minimal human intervention @xcite . recent work uses variations of self-instruct to improve model prompting @xcite . @xcite ; @xcite evaluate the role of prompts in model control and the limitations of metalinguistic prompts for assessing linguistic knowledge in llms. studies by @xcite and @xcite examined the influence of linguistic features of prompts on llm performance with hand-selected prompt variations (e.g., tense, modality). mizrahi et al. (2024) demonstrated through empirical anal our work contributes to previous work in several key aspects. while previous studies have explored hand-crafted linguistic features on llm performance @xcite , we take a systematic approach by decomposing paraphrases into a set of six families of changes (i.e., syntax, lexicon, lexico-syntax, morphological, semantic, and others). although other works have proposed strategies for prompt selection @xcite and methods for enhancing prompt tuning and generation @xcite @xcite , our research quantitatively measures the effects of linguistic paraphrase types on model responses in a bottom-up method to apply successful types on different tasks. other work focuses on domainspecific problems with few tasks. our methodology spans five models, 120 tasks, and 24 task families. the main gap we address is the lack of empirical evaluation of how different linguistic manipulations of prompts affect llm behavior and performance on a large scale. our work controls for other confounding factors, such as prompt complexity, training data proximity, and lexical diversity. ## data & models central to this study is a dataset of tasks, prompts, and variations of these prompts. we use the super-naturalinstructions dataset @xcite , which contains more than 1 600 tasks with their respective prompts. we describe in detail how we construct different variations of these prompts in the experiments (section 4). we sample 120 different tasks (each task has its own dataset) from 24 different task families (e.g., question answering, sentiment analysis) with the following conditions: the task family must have at least 10 tasks, the primary language is english, and each task has 200 examples to provide sufficient statistical power. we sample 5 tasks per domain, leaving us with 24 task families x 5 tasks per @xmath0. see appendix a for the task details. we choose the five best non-proprietary models according to the lmsys chatbot arena leaderboard in descending order of performance as of 1 may 2024 foot_0 : llama 3 instruct (70b), command r+ (104b), mixtral 8x7b instruct (47b), llama 3 instruct (8b), and gemma instruct (7b). unless otherwise noted, we set the temperature to 0.2, the probability mass to sample words (top p) to 0.9, the penalty for repeatedly sampling the same sequence of tokens to 0.1, and the maximum number of tokens generated to the average of human references for that task. this number varies for different tasks but averages between 1 and 3154 tokens (see appendix a for details). we use 40 nvidia a100 gpus (40gb) with 16-bit precision (8 a100s for each model). our experiments required a total computational budget of approximately 2880 a100 gpu hours, resulting in 311kg of co2 equivalent 2 . ## conclusion this study evaluated five language models across 120 tasks (including sentiment, question answering, commonsense reasoning, summarization, etc.) and showed that paraphrasing prompts can improve the performance of language models. we showed that language models have a marked upside potential to improve task results when their prompts are adapted in specific linguistic types (e.g., polarity substitutions for sentiment analysis). other work can use our findings about which tasks benefit from which paraphrase types to design new prompts. we also controlled for prompt complexity, temperature, and proximity to training data. current model performance represents a lower performance bound for tasks as we showed that semantically identical instructions hold marked upside gain. while it is not entirely clear why language models are often sensitive to changes in instruction, we have systematically tested different lexical features and found that some have a larger positive impact (e.g., morphology) than others (e.g., syntax), depending on the tasks. since humans understand tasks presented in different ways and are robust to small (or even complex) changes in instruction, language models should have a similarly robust interface to communication in the future. we have contributed to this development of robust language interfaces by showing how specific types can benefit or harm models over a large set of tasks and prompts. one can also use our approach to create prompts the model does not understand to augment training data and increase its robustness.