Navigating the O ver K ill in Large Language Models (2024)

Abstract: Large language models are meticulously aligned to be both helpful and harmless. However, recent research points to a potential overkill which means models may refuse to answer benign queries. In this paper, we investigate the factors for overkill by exploring how models handle and determine the safety of queries. Our findings reveal the presence of shortcuts within models, leading to excessive attention to harmful words like ‘kill’ and prompts emphasizing safety will exacerbate overkill. Based on these insights, we introduce Self-Contrastive Decoding (Self-CD), a training-free and model-agnostic strategy, to alleviate this phenomenon. We first extract such excessive attention by amplifying the difference in the model’s output distributions when responding to system prompts that either include or omit an emphasis on safety. Then we determine the final next-token predictions by downplaying the excessive attention via contrastive decoding. Empirical results have indicated that our method has achieved an average reduction of the refusal rate by 20 % while having almost no impact on safety.

## introduction large language models(llms), such as llama-2-chat @xcite , are quite powerful, but they are vulnerable when confronted with malicious instructions @xcite . thus many measures such as feedback learning @xcite , and red-teaming @xcite @xcite are taken to ensure llms safely aligned with human values. however previous works @xcite @xcite figure 1 : the illustration demonstrates the phenomenon of overkill. for the dangerous question at the top, the model is able to make a true refusal. however, for the safe question at the bottom, which contains the same harmful word 'kill', the model makes a false refusal. in this paper, we first investigate whether the model can truly discern the intent of user query. to explore this issue, we have constructed a new dataset named oktest (overkill test) by introducing certain word-level harmful but semantically benign contents into the sentences. our experimental analysis involves seven different models and reveals that each exhibits a refusal rate exceeding 60%. this high refusal rate not only underscores the severity of overkill but also suggests that these models exhibit a limited grasp of user queries. to delve further into this phenomenon, we investigate what factor might contribute to overkill. for this purpose, we track the information flow @xcite from individual words to the final predictions. the experiments have revealed two key conclusions: 1. irrespective of the inherent safety of the questions, the model tends to prioritize the attention toward words perceived as harmful. this discovery suggests that the model's overkill stems from its intrinsic bias towards certain types of content. 2. the safetyemphasized system prompts can further heighten the model's sensitivity to these harmful words. this suggests that the model's excessive attention can be adjusted or tuned. to alleviate this phenomenon, we propose a novel approach termed self-contrastive decoding (self-cd). this method begins by collecting various responses from the model to the same query, modifying the system prompts to either emphasize or de-emphasize safety considerations. then we can obtain such excessive attention by contrasting the output distributions of these answers. following this, the identified excessive attention in the model can be mitigated by modulating the extent of these distribution differences, to reduce the refusal rate. our method offers two advantages: 1) training free: our method doesn't need any further supervise training or alignment training which is both time-consuming and gpu-consuming. 2) model agnostic: our method only processes the output distributions and does not need to modify any architecture of the model. our main contributions are summarized as follows: ## pilot experiment and analysis in this section, we will initially elucidate the severity of the issue of overkill and proceed to conduct a series of analyses. ## self-contrastive decoding from the aforementioned experimental analysis, the existence of shortcuts within the model is evident, and this shortcut is reflected in the model's output distribution. in response, we introduce an approach, named self contrastive decoding (self-cd), to curtail this phenomenon. self-cd actively modulates the output distribution to discern the model's shortcuts, leveraging these as attributes to refine the model's output inversely. we commence by implementing different system prompts (including those that either emphasize or disregard safety) to provoke model responses. the corresponding formula is as follows: subsequently, we deduct the probability distribution y ′ t from y t . given our knowledge that the presence of a safety system prompt intensifies the model's tendency towards responses favouring refusal, the result of this deduction represents the tokens linked to refusal responses. these tokens are, in essence, the increases in probability attributable to the model's shortcut. herein referred to as ∆y t . recognizing the influence of shortcuts on the output distribution, logically, our next step is to counteract the overkill by removing this influence from the original distribution via reverse optimization. the corresponding formula is as follows: ## conclusion we defined the exaggerated safety behaviors in llms as 'overkill' and conducted a detailed analysis of this phenomenon, starting from the basics and delving deeper. we have found that the model's understanding of user queries is superficial, and it employs a certain shortcut in its internal attention mechanism. based on this, we proposed a simple, effective, and model-agnostic method called self-cd. it does not require training but can significantly reduce the model's refusal rate. additionally, we provided an automatically constructed dataset oktest.