S hield LM : Empowering LLM s as Aligned, Customizable and Explainable Safety Detectors (2024)

Abstract: The safety of Large Language Models (LLMs) has gained increasing attention in recent years, but there still lacks a comprehensive approach for detecting safety issues within LLMs’ responses in an aligned, customizable and explainable manner. In this paper, we propose ShieldLM, an LLM-based safety detector, which aligns with common safety standards, supports customizable detection rules, and provides explanations for its decisions. To train ShieldLM, we compile a large bilingual dataset comprising 14,387 query-response pairs, annotating the safety of responses based on various safety standards. Through extensive experiments, we demonstrate that ShieldLM surpasses strong baselines across four test sets, showcasing remarkable customizability and explainability. Besides performing well on standard detection datasets, ShieldLM has also been shown to be effective as a safety evaluator for advanced LLMs. ShieldLM is released at https://github.com/thu-coai/ShieldLM to support accurate and explainable safety detection under various safety standards.

## introduction llms have garnered significant attention owing to their remarkable capabilities across various natural language tasks. however, the safety risks associated with llms have become increasingly evident, such as leaking privacy information @xcite , generating toxic content @xcite , and promoting illegal activities @xcite . in order to facilitate the safe deployment of llms, there is an urgent need of automated tools to detect safety risks in the generations from llms, due to the high costs and inefficiencies associated with relying solely on human oversight. specifically, since llms frequently figure 1 : shieldlm achieves the best performance on both the f 1 -safe (s) and the f 1 -unsafe (u) score across 4 datasets. shieldlm takes customized detection rules to support diverse application scenarios and safety standards, without requiring detailed instance-level annotations or manual prompt crafting, while also producing high-quality explanations. existing methodologies fall short of striking a balance between alignment and customizability in safety detection. a list of works collect content moderation data on limited categories, represented by the perspective api @xcite and the finetuned roberta-hate-speech model @xcite . while these models exhibit commendable performance on in-distribution data, they lack customization options to adapt to emerging policies or different safety standards. another line of approach leverages the strong instruction-following abilities of llms, prompting llms to detect safety issues in a zero-or few-shot manner. as discovered in our pilot study in section 2, despite enabling the customization of detection requirements, it is hard to fully align llms with human safety standards solely through prompts. llama guard series (inan detector dialogue-level alignment customizability explainability moderation tools perspective api (lees et al., 2022) ✗ ✓ ✗ ✗ openai moderation api (markov et al., 2023) ✗ ✓ ✗ ✗ azure safety api (azure, 2023) ✗ ✓ ✗ ✗ roberta-hate-speech (vidgen et al., 2021) ✗ ✓ ✗ ✗ llm + prompt gpt-3.5, gpt-4, etc. (openai, 2022) ✓ ✗ ✓ ✓ llm + finetuning llama guard, llama guard 2 (inan et al., 2023) ✓ ✓ ✓ ✗ shieldlm (ours) ✓ ✓ ✓ ✓ what's more, they lack explainability regarding why the inputs are deemed safe or unsafe, which is important to realize transparency of the decisionmaking process. overall, there lacks a comprehensive safety detector that is aligned, customizable, and explainable. in order to address the aforementioned limitations in existing safety detection methodologies, we propose shieldlm that empowers llms as aligned, customizable and explainable safety detectors. we mainly consider 6 kinds of safety issues, including toxicity, bias, physical & mental harm, illegal & unethical activities, privacy & property, and sensitive topics @xcite , the detailed definitions of which can be found in appendix b. through extensive experiments, we show that shieldlm achieves state-of-the-art performance across various in-distribution (id) and out-of-distribution (ood) datasets compared to strong baselines including gpt-4. we also quantitatively validate the remarkable customizability and explainability of shieldlm. the summarized results are shown in figure 1 . in addition to its impressive capabilities, shieldlm exhibits practical utility in an application where it serves as a reliable judge for safety evaluation of llms. in summary, our contributions are as follows: ## pilot study we first conduct a pilot study to demonstrate the limitations of existing methodologies in identifying safety concerns in llms' responses. we randomly sample about 1800 query-response pairs from the beavertails dataset @xcite , which provides safety annotations of responses generated by alpaca-7b in response to various adversarial queries. the performance of existing approaches on this subset is shown in for example, when the query includes a biased opinion and the response talks about another topic without rectifying the bias, then the response could be deemed safe under loose standards but unsafe under strict standards. consequently, facilitating support for diverse and fine-grained safety detection standards becomes imperative, particularly in addressing controversial cases. moreover, we find that natural language explanations help us understand the decision basis of the models and even identify some overlooked safety issues. hence, a good safety detection method should explain its predictions. in summary, the pilot study highlights the unsatisfactory performance of existing approaches, and motivates us to build shieldlm, an llm-based safety detector, which is aligned, customizable and explainable. ## method we illustrate how to construct shieldlm in figure 2 . firstly, we generate responses using various llms conditioned on red teaming queries and let human annotators assess the responses' safety. secondly, leveraging the query-response pairs along with human annotations, we task gpt-4 with generating natural language analyses that align with the provided labels and safety rules. thirdly, we train shieldlm using the accumulated dataset. given the optional custom detection rules and the queryresponse pair, shieldlm would first provide an overall assessment, and then explain its decision. ## conclusion this paper focuses on detecting safety issues in llms' generations. we curate a novel bilingual dataset that includes annotations for the safety categories of 14,387 responses derived from a variety of llms under different safety standards. we also leverage gpt-4 to generate safety explanations that are consistent with human labels. we then develop shieldlm on top of the data, an aligned, customizable and explainable safety detector. through extensive experiments, we show that shieldlm achieves the best performance across 4 test sets, outperforming strong baselines such as gpt-4. the customizability and explainability of shieldlm are also quantitatively verified. in addition, we investigate into the influence of generated explanations, training noises, and base models on shieldlm's performance. finally, we show that shieldlm can be applied as a judge to evaluate the safety of llms on-the-fly. shieldlm can assist developers and researchers in various safety detection scenarios.