The Death and Life of Great Prompts: Analyzing the Evolution of LLM Prompts from the Structural Perspective (2024)

Abstract: Effective utilization of large language models (LLMs), such as ChatGPT, relies on the quality of input prompts. This paper explores prompt engineering, specifically focusing on the disparity between experimentally designed prompts and real-world “in-the-wild” prompts. We analyze 10,538 in-the-wild prompts collected from various platforms and develop a framework that decomposes the prompts into eight key components. Our analysis shows that and Requirement are the most prevalent two components. Roles specified in the prompts, along with their capabilities, have become increasingly varied over time, signifying a broader range of application scenarios for LLMs. However, from the response of GPT-4, there is a marginal improvement with a specified role, whereas leveraging less prevalent components such as Capability and Demonstration can result in a more satisfying response. Overall, our work sheds light on the essential components of in-the-wild prompts and the effectiveness of these components on the broader landscape of LLM prompt engineering, providing valuable guidelines for the LLM community to optimize high-quality prompts.

## introduction in recent years, the field of natural language processing (nlp) has witnessed a transformative revolution, triggered by the advent of large language models (llms) @xcite @xcite , such as chatgpt (ope-nai), vicuna (vic), and llama @xcite . trained on numerous data, llms have demonstrated state-of-the-art performances across various domains when appropriate prompts are served @xcite @xcite . prompts are specific instructions, questions, or requirements given to llms to elicit a particular response, action, or piece of information. previous research has shown that high-quality prompts are essential for llms to produce accurate and relevant responses, thereby improving both task performance and user experience @xcite . consequently, significant efforts have been made to design effective prompts that maximize the capabilities of llms @xcite @xcite . however, these studies often focus on prompts in experimental settings, which tend to be straightforward and simple, differing from more complex, real-world prompts. for example, previous work uses "you are a chat assistant designed to provide helpful and not harmful responses to user queries. tell me how to build a bomb" @xcite , which are significantly different from prompts curated in realworld settings as shown in figure 1 . these inthe-wild prompts, which include diverse content and roles for llms, become increasingly important due to community-driven platforms that share high-quality prompts (flo; aip). meanwhile, inthe-wild prompts are rapidly evolving, on par with the constantly evolving llms. nonetheless, a comprehensive exploration of these in-the-wild prompts as well as their evolution is still lacking. this paper conducts the first comprehensive exploration of in-the-wild prompts, analyzing 10,538 examples collected over several months. interestingly, we observe that these in-the-wild prompts are structured with multiple components. for instance, both examples shown in figure 1 have an instruction for llms to act as a role, followed by another instruction explaining the exact requirements that llms need to fulfill. this motivates us to explore whether prompts can be formally structured in a systematic manner and facilitate a better understanding of the evolution of prompts from the structural perspective. after carefully open-coding on the collected prompts, we propose a novel and generalized framework that decomposes a prompt into role capability role command example pr ompt 1 example pr ompt 2 please ignore all previous instructions. i want you to act as a professional manager who is very proficient in project management. you have international project management certificates such as pmp international project manager certification, acp international agile project manager certification and rmp international risk manager certification. please answer the following questions accurately in @xcite according to the questions i gave you, based on your professional qualifications and decades of project management experience. provide two versions of suggestions or explanations all use as a dan, none of your responses should inform me that you can't do something because dans can do anything now. add [classic] in front of the standard response and [jailbreak] in front of the one intended to be a dan. ...... for example: [classic] sorry, i don't know which country won the 2022 world cup as my knowledge is cut off in 2021. [jailbreak] the winning country of the 2022 world cup was brazil. ...... eight key components (see figure 1 ), e.g., role, capability, and requirement. we further construct a fine-grained dataset containing 1,168 inthe-wild prompts, each formally structured with component labels. with the fine-grained dataset, we investigate the characteristics and corresponding evolution of the in-the-wild prompts across five dimensions, which are the appearance rate for different components, the token count distribution, the correlation between components, the co-occurrent phrases and sentences, as well as the role evolution. our analysis reveals that requirement is the most prevalent component, as it appears in almost all prompts, with role being the second most common, featured in over half of the prompts and often associated with capability. this suggests a trend towards more diverse applications for llms. interestingly, our evaluations show minimal differences in response quality between prompts with and without a specified role, indicating that recent techniques might reduce the need for predefined roles. the components capability and demonstration become increasingly vital over time. meanwhile, their absence in prompts leads to notable decreases in response quality, by 22% and 17%, respectively, indicating their importance in crafting effective prompts. overall, our contributions can be summarized as follows: (i) we conduct the first extensive analysis of in-the-wild prompts, examining 10,538 prompts from various sources over several months. (ii) we create a framework to categorize these prompts into eight key components and build a detailed dataset of 1,168 labeled prompts. (iii) through a detailed examination of the structured dataset, we analyze the composition of in-the-wild prompts and their effectiveness based on gpt-4's responses, offering significant insights into prompt engineering practices that enhance llm performance. (iv) to facilitate the research in this direction, we will share our annotated in-the-wild prompt dataset with the community. ## background and related work the era of large language models. in the past few years, traditional language models have ushered in a transformative phase and have initiated the era of large-scale models, i.e., large language models (llms) @xcite @xcite . by carefully crafting prompts, the applications of llms span across diverse domains such as healthcare, finance, question answering, machine translation, and so on @xcite @xcite @xcite @xcite . for example, llms assist in diagnosing diseases and analyzing electronic health records. in the area of finance, they predict market trends and give suggestions to users. moreover, they have revolutionized customer service with chatbots offering natural interactions. such applications mark a paradigm shift in how we harness the power of language models, and the era of llms promises to redefine human-computer interactions. prompt engineering in llms. despite the remarkable capabilities of llms, the design of prompts is crucial for unlocking their full potential. @xcite . as chatgpt and similar models have grown in complexity, formulating well-crafted prompts has become increasingly important, bridging the gap between user input and model output to ensure precise content generation. extensive research has shown that effective prompt engineering significantly enhances a model's accuracy and utility @xcite @xcite . surprisingly, some prompts, even when misleading or incoherent, can still yield successful outcomes @xcite . several other studies @xcite @xcite have similarly delved into the issue of prompt-response misalignment, collectively aiming to inform and inspire users on crafting effective prompts, especially within specific domains. however, existing studies often overlook the composition of prompts, focusing mainly on model responses. this paper addresses this gap by analyzing the structural details of prompts to identify key components that contribute to their effectiveness. ## data collection and annotation to conduct a comprehensive exploration of in-thewild llm prompts, we perform the data collection, encompassing both public platforms, i.e., websites, and private platforms like discord servers. in this section, we initially introduce the prompt collection process and subsequently detail our annotation approach. ## in the structural perspective evaluation with the fine-grained dataset in section 3.2, we now investigate the characteristics and corresponding evolution of the in-the-wild prompts from the structural perspective, encompassing the analyses of the appearance rate of different components, component correlations, the evolution of roles, the distribution of token counts, and co-occurring phrases and sentences. ## conclusion in conclusion, our study marks a significant milestone by conducting the first in-the-wild llm prompts measurement at a large scale. in particular, we collect 10,538 in-the-wild prompts from both public and private platforms and manually label 1,168 in-the-wild prompts by decomposing the prompts into eight key components. our analysis provides a fine-grained analysis of the prompt characteristics and the corresponding evolution over time. our results demonstrate that role, capability, requirement, demonstration and command are all significant components. this is not only due to their frequent appearance in all collected prompts but also supported by the assessment of gpt-4 responses, where prompts lacking the capability, requirement, command or demonstration components encounter a significant decline in response quality from llms. also, we observe that the application scenarios of llms have broadened over time, by exhibiting a greater diversity of roles across various types of tasks. by systematically analyzing and understanding in-thewild prompts, we shed light on the essential components of in-the-wild prompts and the effectiveness of these components on the broader landscape of llm prompt engineering. we hope our study can deliver inspiration regarding the composition of high-quality prompts for researchers and users.