Prompt Engineering 101 Prompt Engineering Guidelines from a Linguistic Perspective (2024)

Abstract: “Deploying tuning-free prompting is challenging in engineering practice: it not only requiresusers to engage in cumbersome trials and errors but is also extremely time-consuming,as even a slight change in wording and phrasing could have a huge impact on the finalperformance. To further investigate the impact of different prompts, in this work, weperform a systematic inspection of four factors in linguistics involved in prompt engineering:syntax, semantics, lexicon, and pragmatics. The empirical results quantify the sensitivityof the output to small textual perturbations in four linguistic factors of prompts. Basedon the analysis of these four factors, we present a series of design guidelines to helphuman users write effective prompts. Human evaluation on amateurs shows that usingthe proposed guidelines helps humans produce prompts with significant gains in zero-shotperformance in Pre-trained Language Models (PLMs) and hence validates the utility ofthe guidelines.”

## introduction dramatic gains in large language models (llms) have been witnessed in recent years @xcite @xcite , accelerated by the discovery of prompting @xcite @xcite . prompts function in the form of natural language to drive llms to access a variety @xcite of downstream tasks (see sec. a). however, amateurs must engage in cumbersome trials and errors when the input prompt quality is poor or ambiguous for llms to deliver their real intention @xcite @xcite @xcite . guiding amateurs to clearly present their intention, we refer to education and linguistic theory where linguistics knowledge often be utilized to guide second-language students to effectively express ideas in writing. there is a vast body of research available in the field of linguistics that delves into these topics @xcite . inspired by this, we investigate linguistics factors of prompts. a general overview of how these linguistic factors can be employed to support humans is as follows. syntax studies how words and morphemes combine to form larger phrases and gradually constitute a sentence @xcite . central concerns of syntax include word order, verb tense, subject-verb agreement, hierarchical sentence structure, and other syntactic features. understanding syntax can aid second language students in composing coherent sentences @xcite . semantics studies the meaning in natural languages. negation, denoting the reversal of some meaning groups in a sentence, is one of the most discussed phenomena in semantics @xcite . human beings can learn about negation and can benefit from developing their understanding of the appropriate use of words and expressions in specific contexts. lexicon or lexicology, studies the vocabulary of a language. @xcite and @xcite examine figure 1 : illustration of our guidelines prompt engineering 101. the guidelines involve four stages: a syntactic checker to correct grammatical errors, a semantic checker to make proposals without negation, a lexical checker to substitute works, and a pragmatic checker to remove metaphors. prompt engineering 101 is used for an amateur to interact with plms via prompting. following this workflow, the amateur checks carefully each factor and finally writes a prompt. for better checking, in each stage, some suggestions (bullet points in the box) from external automatic software (e.g., grammatical error corrector) will be provided for your reference. the amateur can choose to accept the suggestion or not. the vocabulary knowledge and suggest effective vocabulary strategies for improving writing proficiency. similarly, @xcite consistently shows that llms are more accurate on samples with words more commonly used. pragmatics is the study of using languages to express what we call "beyond saying" in linguistics and cognitive science @xcite . generally speaking, semantics focuses on the literal meanings of linguistic expressions, whereas pragmatics captures the context-dependent implicit factors of meaning @xcite . the study of pragmatics involves metaphors, deceits, indirect speech, irony, gricean maxims, humor, and coherence inferences @xcite . @xcite offers insights into how pragmatic knowledge helps produce appropriate sentences. against this background, we explore the aforementioned four crucial factors: syntax, semantics, lexicon, and pragmatics, since these four factors provide valuable insights that can help humans produce coherent sentences and present desired intentions. we systematically and quantitatively inspect the impact of the four linguistics factors on prompting llms (sec. 2). for each factor, we study the impact on zero-shot performance via controlled experiments. for syntax, we randomly permute words and introduce grammatical errors in prompts in the experimental group (see sec. 2.1). for semantics, we explore whether prompts with perturbed semantics but correct syntax can still produce meaningful outputs (see sec. 2.2). for lexicon, we study prompts with different wording and phrasing. in particular, we replace words with synonyms or translate them into other languages, e.g., chinese (see sec. 2.3). lastly, we investigate the pragmatics of metaphors. we build qa tasks with and without metaphors and study their impact on performance (see sec. 2.4). the results from these experiments show clear paths for us to navigate in the space of prompt design and steer llms toward desirable generations. we, therefore, present a series of design guidelines prompt engineering 101 (sec. 3) to help humans and the automated toolkit refine prompts. evaluation on both humans (sec. 4.2) and automated toolkit (sec. 4.3) show that using the proposed guidelines subjects produce prompts with significant gains in zero-shot performance in llms and hence validates the utility of the guidelines. ## prompt engineering 101 since sec. 2 show clear paths for us to navigate in the space of prompt design, in this section, we discuss good practices for an amateur to interact with llms via prompting, following the analytical results aforementioned. in addition to general requirements, e.g., the language choice, human readability, avoiding abstract statement @xcite • checking syntax: sentences should be free from grammatical errors and respect proper word order. if possible, a grammatical error checker will be helpful for users to check their language. fig. 1 illustrates the flow diagram of our proposed guidelines. it is a pipeline including four stages. our prescribed approach first requires grammatical error correction to check the input sentence. with grammar errors free, the prompt will further go through analysis on semantics, removing unnecessary negations, common word substitution, and metaphor removal. ## conclusion and future work we investigate and analyze the sensitivity of prompts from the perspective of linguistics, covering four factors, i.e., syntax, semantics, lexicon, and pragmatics, and their impact on various tasks. based on the results, we streamline a list of guidelines as prompt engineering 101 for prompt engineering. via human evaluation, we show the effectiveness of the proposed guidelines. we further automated prompt engineering 101 to build the automatic toolkit designed based on our guidelines. after replacing manual refinement with an automatic toolkit, we still observed an average improvement in performance. this further enhances the superiority of prompt engineering 101. there are more linguistic factors, for example, phonetics, phonology and typology, for us to study. even for syntax, there remain aspects that we have not explored. yet, we take the first step to exploring four of them. a more in-depth analysis for other factors on a wider range of tasks should be favorable. this work also provides another window into prompt engineering that regards prompts as a specific "langauge" to interact with llms. this c c l 2 0 2 4 "language", like a dialect, is a variety of language that is characteristic of llms. from the perspective of behaviors, they have their own grammatical and phonological rules, linguistic features, and stylistic aspects, that is potentially different from the natural language. with mounting evidence, llms use prompts in a different way from humans. nevertheless, there is no clear consensus on what kind of linguistic factor plays the most important role. on top of that, we hope that this work would call for future research into the intriguing problem.