Beyond the Tip of Efficiency: Uncovering the Submerged Threats of Jailbreak Attacks in Small Language Models (2025)

Abstract: Small language models (SLMs) have become increasingly prominent in the deployment on edge devices due to their high efficiency and low computational cost. While researchers continue to advance the capabilities of SLMs through innovative training strategies and model compression techniques, the security risks of SLMs have received considerably less attention compared to large language models (LLMs). To fill this gap, we provide a comprehensive empirical study to evaluate the security performance of 13 state-of-the-art SLMs under various jailbreak attacks. Our experiments demonstrate that most SLMs are quite susceptible to existing jailbreak attacks, while some of them are even vulnerable to direct harmful prompts. To address the safety concerns, we evaluate several representative defense methods and demonstrate their effectiveness in enhancing the security of SLMs. We further analyze the potential security degradation caused by different SLM techniques including architecture compression, quantization, knowledge distillation, and so on. We expect that our research can highlight the security challenges of SLMs and provide valuable insights to future work in developing more robust and secure SLMs.

## introduction large language models (llms), such as chat-gpt @xcite @xcite and llama series @xcite , have demonstrated revolutionary performance in a spectrum of text generation tasks. as a fundamental principle for guiding the development of llms, the scaling law @xcite highlights the strong correlation between the performance and scale of llms. however, as llms evolve to encompass * corresponding author. recently, small language models (slms) have attracted significant attention from the academic community for their efficiency and remarkable performance in various tasks @xcite . on platforms like hugging face, slm collections such as llama-3.2 @xcite , minicpm @xcite and @xcite have gained considerable popularity among researchers and achieved top-tier download rates. different from llms, slms typically consist of only a few billion parameters, requiring significantly less training data and computational cost for deployment. however, unlike llms that benefit from extensive datasets and robust alignment strategies, it is challenging for slms to balance between generation capabilities and security, which makes them more vulnerable to jailbreak attacks. among these, one of the most serious threats is referred to as jailbreak. by creating malicious prompts to induce target llms to generate harmful responses, jailbreak has emerged as a critical security concern in the development of llms @xcite @xcite . moreover, certain jailbreak techniques that can bypass the security boundary of llms or vlms have demonstrated strong transferability to other models @xcite @xcite , which presents potential threats to all generative models including slms. although security concerns regarding slms have become an increasingly important issue, there still remains a substantial gap in exploring and understanding the security boundary of slms. in this paper, we collect representative malicious datasets, jailbreak attack methods, and defense methods to conduct adversarial experiments on numerous stateof-the-art slms, thereby revealing existing security vulnerabilities of slms and exploring corresponding mitigation strategies. furthermore, we take insight into the security degradation of slms and discuss some potential factors. in summary, we make the following contributions: ## conclusion in this paper, we present a systematic empirical study to explore the security vulnerabilities of the state-of-the-art slms. we demonstrate that most slms are highly susceptible to malicious input, where jailbreak attacks pose a particularly significant threat. we also evaluate the effectiveness of several defense strategies when applied to slms, and further discuss the underlying factors that may cause the security degradation of slms. we hope that our study can raise awareness of the security risks associated with slms and offer valuable insights for developing more robust and resilient slms in the future.