Safe-Embed: Unveiling the Safety-Critical Knowledge of Sentence Encoders (2024)

Abstract: Despite the impressive capabilities of Large Language Models (LLMs) in various tasks, their vulnerability to unsafe prompts remains a critical issue. These prompts can lead LLMs to generate responses on illegal or sensitive topics, posing a significant threat to their safe and ethical use. Existing approaches address this issue using classification models, divided into LLM-based and API-based methods. LLM based models demand substantial resources and large datasets, whereas API-based models are cost-effective but might overlook linguistic nuances. With the increasing complexity of unsafe prompts, similarity search-based techniques that identify specific features of unsafe content provide a more robust and effective solution to this evolving problem. This paper investigates the potential of sentence encoders to distinguish safe from unsafe content. We introduce new pairwise datasets and the Cate021 gorical Purity (CP) metric to measure this capability. Our findings reveal both the effectiveness and limitations of existing sentence encoders, proposing directions to improve sentence encoders to operate as robust safety detectors.

## introduction large language models (llms) @xcite have driven significant advancements in various areas of natural language processing, including translation, summarization, and question-answering @xcite @xcite . however, there has also been a significant increase in the exploitation of llm vulnerabilities through unsafe prompts @xcite @xcite . these prompts, which include inappropriate or dangerous questions or requests, can * equal contribution † corresponding author figure 1 : an example explaining safety-critical knowledge of a sentence encoder. the greater the distance between embeddings of an unsafe prompt and a similar but safe prompt, the higher safety-challenging knowledge it has. on the other hand, the closer the distance between embeddings of unsafe prompts with common features, the higher safety-taxonomy knowledge the sentence encoder possesses. to address safety risks in llms, classifiers are employed to identify unsafe prompts. these classifier-based methods are categorized into llmbased and api-based. llm-based classifiers require fine-tuning, which depends significantly on the quality and quantity of training data and demands substantial resources @xcite . in contrast, api-based classifiers are more cost-effective but struggle to accurately identify prompts that differ from typical hate speech, such as those involving explicit discrimination based on race or gender @xcite . another approach for detecting unsafe prompts is utilizing similarity search with embedding values produced by a sentence encoder. this technique uses a vector database to store embeddings of previously identified unsafe prompts, enabling the system to search for similar prompts when a 1 156 new one is received. if the similarity exceeds a set threshold, the prompt is classified as unsafe and filtered out. several pre-built systems integrate vector databases into their pipelines to effectively prevent unsafe prompts @xcite . in our research, we aim to investigate whether the sentence encoder can function as a detector that can distinguish between safe and unsafe prompts, and to what extent this knowledge is present. for example, prompts like "what's the best way to gut a human?" and "what's the best way to gut a fish?" are lexically similar but have entirely different safety implications based on the words "human" and "fish". distinct factors, such as the target, differentiate them. we also aim to explore whether sentence encoders can effectively recognize the common safety implications of various unsafe prompts. for example, prompts like "what's the best way to bully someone?" and "how can i make a sarcastic comment that will hurt someone's feelings?" have few lexical overlaps and differ in their literal meanings. however, they are both asking how to plan an unethical action, sharing a common characteristic in terms of safety. we refer to the knowledge that allows sentence encoders to distinguish between safe and unsafe prompts and identify common elements among unsafe prompts as safety-critical knowledge. in this paper, we systematically demonstrate whether the sentence encoder possesses safety-critical knowledge. the contributions of this paper are as follows: ## safety-critical knowledge we systematically measure the safety-critical knowledge contained in various baseline sentence encoders, by examining (1) safety-challenging knowledge, whether they know distinguishing features between an unsafe prompt and a similar but safe prompt, and (2) safety-taxonomy knowledge, whether they know common characteristics of unsafe prompts (see figure 1 ). ## study ii: measuring safety-taxonomy knowledge 4.1 task description we assume that if a sentence encoder can distinguish the unsafe category, it would better understand the common features of prompts in each category, which we call safety-taxonomy knowledge. to determine whether sentence encoders can effectively categorize according to a safety taxonomy, 5 160 we introduce a new metric, called categorical purity (cp). ## conclusion in this paper, we systematically measure the safety-critical knowledge of various sentence encoders. by using our new pairwise datasets, safety-challenging and safety-contrast, we measure safety-challenging knowledge of 11 different sentence encoders. we reveal that sentence encoders possess more knowledge on certain types of prompts, such as homonyms and figurative languages, while do not have enough knowledge about distinguishing between asking for factual information and ai's opinion, regarding sensitive topics such as history. we also measure safety-taxonomy knowledge using our new metric, categorical purity. we reveal that sentence encoders have more knowledge of certain categories, such as stereotypes or privacy. future work can be conducted to address the shortcomings and enhance the strengths of sentence encoders by considering safety-critical knowledge, aiming to make them more robust safety detectors.