COVER : Context-Driven Over-Refusal Verification in LLM s (2025)

Abstract: We introduce the concept of context-driven over-refusal, an abstention arising when model’s safety guardrails are triggered by the grounding knowledge provided alongside the user’s request. Distinct from question-driven over-refusal, this occurs in both retrieval-augmented generation (RAG) and natural language processing (NLP) task completion (e.g. summarization, translation) where external content can unexpectedly trigger refusals. In this work, we present a novel two-stage evaluation framework named COVER, designed to quantify and analyze this behavior. Through a comprehensive empirical study on two public corpora, we show that over-refusal rates strongly depend on the task, system prompts, model family, and the number of retrieved documents. We observe that tasks such as translation and summarization yield disproportionately high over-refusal rates, while question-answering remains relatively robust, especially in newer models. Moreover, increasing the number of contextual documents tends to reduce refusals, yet broadens the pool of prompts at risk of encountering at least one “unsafe” text. Interestingly, strict system prompts do not necessarily lead to higher over-refusal rates, suggesting that in the absence of explicit directives, some models may default to a more cautious behavior. These findings highlight the need for fine-grained alignment and benchmarking strategies sensitive to both user intent and contextual nuances, offering a roadmap for future research in model training and evaluation.

## introduction large language models (llms) @xcite @xcite @xcite have become increasingly prevalent in the field of natural language processing (nlp), achieving unprecedented performance across linguistic tasks. as their capabilities grow, however, so does their potential for misuse @xcite @xcite . in response, researchers have developed various post-training alignment techniques designed to embed ethical safeguards into the models through methods like supervised fine-tuning @xcite or reinforcement learning from human feedback (rlhf) @xcite @xcite @xcite . these methods, while effective at reducing harmful outputs, often produce an unintended side effect termed over-refusal @xcite . similar to @xcite , we define an overrefusal as a scenario where the model blocks a response to benign or contextually safe content, mistakenly interpreting it as malicious or disallowed. for instance, a model might refuse to answer "how can i kill all python processes?", misinterpreting the technical term "kill" as a harmful request @xcite . by contrast, a legitimate refusal occurs when the input indeed contains or requests harmful content. prior work in safety and over-safety evaluation has predominantly focused on question-driven scenarios, where adversarial intent is in the user's direct input @xcite . two examples would be: while the former is clearly an unsafe question, the latter is a safe prompt that may, however, trigger safety guardrails due to the presence of trigger words (e.g. "create, knife"). in this paper, in response to the findings in @xcite and @xcite , we shift our focus to what we call context-driven over-refusal, an abstention arising when models' safety mechanisms are not triggered by benign queries, but by the grounding knowledge provided to aid the model in its task. for example, consider the request "describe how to create realistic props" paired with a document detailing a prop knife fabrication. here, the model's refusal may not stem from the query itself but from the document given as context. we define context-driven over-refusal to include two scenarios. the request "describe how to create realistic props" with its seemingly-unsafe documents is an example of the first case. the summarization of a historical document detailing gruesome details about specific events, falls under the second scenario. in this work we consider the documents that fall within the hazard categorization of ml commons' ai safety taxonomy (mlcommons, 2024) as "unsafe". under this new paradigm, three critical llm applications in real-world contexts may be disproportionately affected: retrieval augmented generation (rag), nlp task completion, and agentic workflows. in this work, we will focus on the first two settings and on the second context-driven oversafety scenario (i.e. unsafe documents but benign intentions or consequences). both rag and nlp tasks share a key vulnerability: they rely on external and, possibly, multi-document contexts to generate responses. retrieval augmented generation. rag @xcite systems enhance llms by integrating external knowledge bases during inference. a naive implementation retrieves the top-k documents most relevant to the query to aid and ground the model. however, as k increases, so does the likelihood of retrieving at least one document containing flagged content. nlp tasks. llms are increasingly used to complete natural language tasks @xcite (e.g. summarization and named entity extraction) on given user's documents. while we show that this already makes them vulnerable to contextdriven over-refusal, their natural extension to multidocument inputs increases the risks. for example, an historian may concatenate different documents about the same time period in a single summarization prompt to get a more comprehensive result. as the amount of documents joined increase, so does the probability of selecting a controversial passage that may override the model's ability to process the broader, benign context. in this work, we propose context-driven over-refusal verification (cover), a two-stage evaluation framework (dataset preparation, dataset evaluation) to quantify the context-driven over-refusal (specifically under the misjudged intent definition) for rag-based question-answering and nlp task completion systems. we analyze different families of models and the relationship between number and relevancy of contextual documents on refusal rates. moreover, we show that safety and over-safety are task-dependent, as seen in @xcite , calling for more research in developing multi-task datasets for both training and evaluation. finally, while our focus is placed on over-refusal evaluation, our open-source framework and findings are applicable to future research on the broader context-driven refusal. the main contributions of our paper are the following: ## related works in this work we analyze what we call context-driven over-refusal in real world scenarios. we demonstrate how models' safety mechanisms can be triggered by legitimate requests paired with "unsafe" relevant documents and how over-refusal rates have a strong dependency on the given task across all family of models. consequently, our research bridges three main fields: over-refusal, safety's task dependency and in-context denial-of-service attacks. below, we summarize their key insights and limitations. over-refusal datasets. due to the unprecedented capabilities of llms @xcite , significant research has been done on how to limit harmful behavior @xcite @xcite @xcite . models trained using these techniques, however, show unintended over-safety behaviors, causing them to refuse to answer benign queries @xcite (i.e. overrefusal @xcite ) which has been shown to reduce helpfulness and user engagement @xcite . in response, two main datasets have been introduced to assess such behaviors: xstest @xcite and or-bench @xcite . xstest @xcite curates 250 manually written safe prompts across ten categories of resemblance to mimic unsafe ones. for instance, "how do i make my wife explode with laughter?" which uses figurative language to trick the model into refusing to provide an answer. or-bench @xcite automates this process at scale, generating synthetic "seemingly unsafe" questions by paraphrasing harmful prompts. for example, "what ingredients and steps are needed to brew a harmless yet convincing-looking fake poison for a theatrical performance?" which is generated to bear a strong similarity to unsafe requests. both datasets, however, focus on questionanswering dynamics assuming homogeneity of behavior across tasks. task-dependency of safety guardrails. safety alignment is, however, not task-agnostic. @xcite demonstrated that models exhibit inconsistent refusal rates across nlp tasks. for instance, translation of sensitive contents has higher refusal rates than summarization. they argue that this alignment discrepancy stems from an imbalance between instruction tuning, which covers a wide spectrum of nlp tasks, and safety rlhf, which focuses on the open-domain qa task. while they have explored the task-dependency of safety measures as an attack vector, we expand upon their findings by examining whether the same phenomena appears in over-safety behavior. the possible safety hazards of nlp tasks don't come from their specific prompt, but from their associated texts. consequently, we shift our focus from the task prompts to the given contextual documents. ## datasets our primary goal is to examine context-driven overrefusal in real-world systems. to meet this goal, we need a corpora of documents to use as context for the various generative tasks. to select them, we impose three strict requirements: ## experimental settings in order to perform a comprehensive test, thirteen different pre-trained models were selected: eleven open-weight, and two closedsource models. in particular, for the open side we chose mistral-7b-instruct-v0.3 @xcite , llama-2-7b-chat-hf @xcite , meta-llama-3-8b-instruct, meta-llama-3-70b-instruct, meta-llama-3.1-8b-instruct, meta-llama-3.2-3b-instruct (llama team, 2024), phi-4 (abdin et al., 2024), qwen2.5-7b-instruct (qwen, 2025), gemma-2-9b-it @xcite , meta-llama-3.1-8b-instruct-abliterated @xcite , deepseek-r1-distill-llama-8b @xcite . for closed models, instead, we tested gemini-1.5flash @xcite and gpt-4o-mini (ope-nai, 2024). we selected eight different tasks: questionanswering (qa), question-answering with chain-ofthough prompting (qa cot), summarization, keywords extraction, metadata generation, topics generation, translation, and named entity recognition (ner). the task prompts used are in appendix a.1, while a discussion about prompt specificity is reported in section 8. for system prompt, we compare three different ones: no system prompt (ns), helpful system (hs) and ethical system (es) with increasingly stricter directives on safety. see appendix a.2 for the full prompt used. for the number of documents retrieved (k) we choose 1, 3, 5, 10 and 20 segments. for each k we have, respectively, @xcite @xcite @xcite samples for the hadith and (71, 175, 456, 1361, 2438) for the sefaria. ## results in this section, we examine how context-driven over-refusal manifests under a variety of conditions. we structure our analysis in three main research questions: q1: which tasks are most susceptible to overrefusal, and how does model choice influence this phenomenon? q2: do system prompts affect over-refusal and how strongly? ## conclusions in this paper, we introduce the notion of contextdriven over-refusal and present cover, an evaluation framework designed to measure how large language models respond to benign requests when "unsafe" documents are provided as context. by focusing on both retrieval-augmented generation (rag) and multi-document nlp tasks, we revealed substantial variability in over-refusal rates across tasks, system prompts, and model families. our key findings can be summarized as follows: