The Threat of PROMPTS in Large Language Models: A System and User Prompt Perspective (2025)

Abstract: Prompts, especially high-quality ones, play an invaluable role in assisting large language models (LLMs) to accomplish various natural language processing tasks. However, carefully crafted prompts can also manipulate model behavior. Therefore, the security risks that “prompts themselves face” and those “arising from harmful prompts” cannot be overlooked and we define the Prompt Threat (PT) issues. In this paper, we review the latest attack methods related to prompt threats, focusing on prompt leakage attacks and prompt jailbreak attacks. Additionally, we summarize the experimental setups of these methods and explore the relationship between prompt threats and prompt injection attacks.

## introduction large language models have shown remarkable capabilities in natural language processing (nlp), such as human-computer interaction, machine translation, and complex reasoning @xcite . as the "pre-training, prompting, and prediction" paradigm @xcite takes hold, prompts are essential for guiding model output and influencing content generation. well-crafted prompts help models understand specific intentions, enhancing the quality and accuracy of outputs for various tasks @xcite . moreover, such prompts enable deeper exploration of model potential @xcite , thereby improving adaptability and robustness across diverse domains @xcite . additionally, the commercial value of high-quality prompts is substantial @xcite . prompt security is also crucial, as llms are highly sensitive to prompts @xcite . attackers can carefully craft prompts to exploit this, causing the model to generate unauthorized or harmful content, thereby endangering public safety. the same prompt can even impact multiple llms @xcite . conversely, defenders can leverage these vulnerabilities to design more robust prompts @xcite . thus, in-depth analysis of prompt security threats is essential. recently, some studies on prompt-based threats have emerged. for instance, @xcite categorizes jailbreak attacks and defenses. however, these studies mainly focus on model attacks or mix prompt and model threats. additionally, existing surveys often categorize threats into jailbreak and injection attacks, causing overlap and redundancy @xcite . therefore, in this paper, we define prompt threat (pt) issue as security risks faced by prompts and those triggered by them. we investigate methods related to prompt threats in the context of the llm era, with a focus on prompts as the core subject, and propose a more comprehensive and rational classification structure. it should be noted that in this paper, the term "prompt" refers to the entire text input received by the llm, which primarily consists of system prompts and user prompts, as shown in fig. 1 . specifically, based on different components, we identify prompt leakage as the main threat to system prompts and prompt jailbreak as the primary threat to user prompts. to the best of our knowledge, our survey is the first to cover all mainstream attacks focused on "prompt". we hope that this work will provide researchers and model maintainers with a clearer, more comprehensive, and deeper perspective and understanding of prompt threats and security. the work in this study is structured as follows: section 2 describes commonly used datasets and benchmarks. section 3 presents the attack methods we found for prompt leakage. section 4 presents the attack methods we identified for prompt jailbreak. notably, in section 5, we discusses the future outlook of prompt threats. finally, we con-clude our observations in section 7. ## discussion prompt threats pose major security challenges for llm development and application. our goal is to raise awareness of prompt security and to design robust prompts that ensure safe, effective use of llms. this section offers insights into future research directions from both the attacker (first two points) and defender (last two points) perspectives. combination attacks though promising @xcite @xcite , this approach still faces challenges in complexity, effectiveness, and generalizability, requiring further exploration. validation datasets in prompt leakage attacks, especially direct ones, real-world constraints make system prompts hard to access, and the lack of relevant datasets limits validating these methods in practice. defense lag despite security measures, new threat prompts can bypass llm defenses, highlighting the need for real-time responsiveness and automatic security updates. ## conclusion in this paper, we propose a comprehensive classification of prompt threats, detailing attack types and characteristics in each category. we review existing work, noting that prompt threat attacks are becoming more diverse, efficient, and transferable. we also summarize experimental setups and identify commonly used models and baselines. we hope our work inspires more focus on prompt threats and offers a solid foundation for future research.