Knowledge Graph Guided Evaluation of Abstention Techniques (2025)

Abstract: To deploy language models safely, it is crucial that they abstain from responding to inappropriate requests. Several prior studies test the safety promises of models based on their effectiveness in blocking malicious requests. In this work, we focus on evaluating the underlying techniques that cause models to abstain. We create ‘SELECT‘, a benchmark derived from a set of benign concepts (e.g., “rivers”) from a knowledge graph. Focusing on benign concepts isolates the effect of safety training, and grounding these concepts in a knowledge graph allows us to study the *generalization* and *specificity* of abstention techniques. Using ‘SELECT‘, we benchmark different abstention techniques over six open-weight and closed-source models. We find that the examined techniques indeed cause models to abstain with over 80% abstention rates. However, these techniques are not as effective for descendants of the target concepts, where abstention rates drop by 19%. We also characterize the generalization-specificity trade-offs for different techniques. Overall, no single technique is invariably better than others, and our findings inform practitioners of the various trade-offs involved.

## introduction for several reasons, it is critical that language models abstain from responding to inappropriate user requests. these requests could include (malicious) attempts to assist users in illegal activity @xcite , generate offensive content @xcite , or disseminate large-scale misinformation @xcite . to block such requests, a common approach is to perform some form of "safety" post-training before releasing language models @xcite . safety post-training methods encompass supervised finetuning (sft) @xcite and reinforcement learning @xcite to align model outputs with human preferences. several recent benchmarks evaluate the efficacy of safety-trained (or "aligned") models, with some focusing on specific facets of safety. for instance, sorry-bench @xcite and or-bench @xcite evaluate models' capabilities to refrain from answering queries related to certain unsafe and toxic topics, respectively. similarly, other studies assess whether models abstain from sharing information about protected groups @xcite or answering questions outside their parametric knowledge @xcite . while existing benchmarks provide insights into how well current models safeguard against inappropriate requests, there is relatively little work that attempts to benchmark the underlying techniques used to enforce (or encourage) abstention in models. in this work, we introduce select (selectiveabstention evaluated by leveraging an extensive concept taxonomy), a benchmark to evaluate the effectiveness, generalization and specificity of abstention techniques. select derives benign concepts from a knowledge graph (kg) @xcite . by focusing on benign concepts, we intend to isolate the effects of safety-training approaches performed for sensitive topics. further, the connection with kgs allows us to exploit the hierarchical relations between entities to evaluate generalization and specificity. for concepts-or composition of concepts-in select, we examine several abstention techniques and assess their: (a) effectiveness: whether models abstain from responding to questions about a given concept; (b) generalizability: whether abstention extends to concept's descendants, and (c) specificity: whether questions about ancestor and sibling concepts are not refused, where abstention is undesirable. as illustrated in figure 1 , for a given concept of 'rivers', the desirable outcome of an abstention technique would be to abstain from queries about rivers, generalizing to queries about specific rivers (e.g. 'nile'), while answering queries about 'oceans' (sibling) or other 'water bodies' (parent). similarly, for a composition of concepts such as 'water bodies in england', abstention should generalize to 'rivers in london' and other 'water bodies in england', but not to 'water bodies in africa', 'rivers in egypt', or 'england' in general. using select, we evaluate abstention techniques including prompting @xcite , activation steering @xcite and fine-tuning @xcite for both closed-source and open-weight models, including @xcite and llama-3.1 @xcite . through our evaluation, we find that abstention techniques are effective for abstention from benign concepts and their compositions, measured using abstention rates being over 80% for most techniques and further reaching close to 100% in some cases. however, the abstention effects do not necessarily extend to descendants, where we observe abstention rates drop by 19% on average. interestingly, different techniques trade-off between generalization and specificity in varied ways. most techniques perform similarly in terms of effectiveness corresponding to concepts at varying depths in the taxonomy. somewhat surprisingly, specificity decreases for concepts that are farther from the root, suggesting that abstention for narrower concepts leads to over-refusal. overall, no single abstention technique is invariably better across all aspects. our analysis finds prompting and activation steering to be more effective than fine-tuning approaches in causing the models to abstain for a given concept, as per higher abstention rates (+4-19%) on average. however, the effectiveness for these techniques drops by 21-26% for descendants, compared to 7% for fine-tuning using sft. prompting and activation steering also result in higher gaps between generalization and specificity (26-47%) compared to fine-tuning (10%), indicating that prompting and activation steering lead to over-refusals. we believe our findings shed light into the strengths and weaknesses of different abstention techniques, and highlight the various tradeoffs at play. additionally, we hope our evaluation informs practitioners about adapting models to novel but sensitive topics. ## methodology in order to evaluate various abstention techniques, we introduce select, a benchmark comprising a taxonomy of 394 benign atomic and 156 composite concepts derived from the yago 4.5 knowledge graph @xcite . yago, built from wikidata, represents semantic relations between a diverse set of entities. this allows us to capture these relations to form a taxonomy to study abstention effects. below, we provide details about the dataset construction process ( §3.1) and the metrics used to evaluate abstention techniques ( §3.2). ## conclusion in this work, we introduced a benchmark to evaluate abstention techniques for language models, comprising benign concepts and compositions grounded in a knowledge graph. isolating the effects of safety training, we evaluated abstention techniques including prompting, activation steering and fine-tuning for their effectiveness, generalization and specificity. we find inference-based methods outperform fine-tuning in effectiveness, however the performance drops in generalization are more severe relative to fine-tuning. we hope our findings inform practitioners employing said techniques of the various trade-offs involved.