SLM as Guardian: Pioneering AI Safety with Small Language Model (2024)

Abstract: Most prior safety research of large language models (LLMs) has focused on enhancing the alignment of LLMs to better suit the safety requirements of their use cases. However, internalizing such safeguard features into larger models brought challenges of higher training cost and unintended degradation of helpfulness. In this paper, we leverage a smaller LLM for both harmful query detection and safeguard response generation. We introduce our safety requirements and the taxonomy of harmfulness categories, and then propose a multi-task learning mechanism fusing the two tasks into a single model. We demonstrate the effectiveness of our approach, providing on par or surpassing harmful query detection and safeguard response performance compared to the publicly available LLMs.

## introduction over the recent years, generative large language models (llms) have been remarkably scaled up in terms of number of model parameters and volume of training corpora. they exhibit robust incontext learning capabilities, which has made the models more universal @xcite @xcite . also, they have moved forward to the extent of understanding and responding to natural human instructions @xcite @xcite instruction tuning for different tasks and application domains @xcite @xcite . this has led to a variety of applications such as conversational ai services, to name a few, chatgpt @xcite , openassistant @xcite , and llama-2-chat @xcite . these dramatic improvements in llms' ability to follow user instructions also raise risks from a safety perspective in creating a customer-facing generative ai services. the capabilities of llmbased services to answer questions based on strong prior knowledge leads to possibilities of being misused for nefarious purposes @xcite @xcite . to address this vulnerability of llms, a large body of research has been directed toward strengthening the safety alignment of llms. for instance, rlhf @xcite @xcite performs an essential role to guide llms to follow human guidance and avoid generating harmful content. the increased size of the model and the implementation of reinforcement learning from human feedback (rlhf) have indeed reduced the success rate of safety attacks @xcite . nevertheless, this approach inherently involves a compromise, as enhancing harmlessness via these methods may inadvertently decrease helpfulness @xcite . additionally, updating the safety alignment of the llms is very expensive. therefore, being able to update the safety alignment at low cost (and even being able to control the model's answers without additional parameter updating) is important. it is thus reasonable to consider building separate models to address safety perspectives at low computational cost with slm @xcite , rather than internalizing such safeguard features to the llms. in this paper, we propose an approach to leverage smaller language model (slm) to accurately detect and to generate safeguard answers for harmful user queries. our main contributions consist of the following: ## related work the framework proposed in this paper is similar to the method described in @xcite in that it transfers knowledge from llms to slms using a multi-task learning approach. we further suggest that a rationale can function not only as a means to enhance prediction performance, but also as a source of advanced answer by itself. our work shares the same concern with @xcite and @xcite in that it evaluates the harmfulness of input sentences. the former, based simply on perplexity is vulnerable to recent llm attack methodologies going beyond the simple prefixing of meaningless tokens, making the approach less functional. the latter is limited as its complexity increases with the number of subsamples of input sentences and is inherently reliant on the safety capabilities of the original model. most recently, meta published a study on a safety check module based on slm @xcite . this is similar to our proposed work in that they share their own query harmfulness taxonomy and perform instruction-tuning from a 7b-sized backbone. however, it has a limitation that it only determines the harmfulness of questions and answers, but does not generate fluent answers from a safety perspective. furthermore, the accuracy of safety check in korean is not satisfactory. the specific experimental results can be found in section 4. moreover, there are publicly available safety check tools in api form, such as perspective api * and openai moderation api * . however, the performance of these models in non-english languages, including korean, significantly lags behind their proficiency in english, despite official claims of supporting non-english languages. it also has the limitation of not being able to generate appropriate answers to address harmful queries. ## methods our objectives are twofold, considering the importance of llm safety: we aim to ( 1 ) create a balanced safeguard that is neither overly strict nor too lenient, and (2) to have the safeguard generate fluent responses instead of a simple template sentence (e.g., "i can't answer"). in this section, we present a taxonomy of query harmfulness, the procedure for creating the training dataset, and the detailed training methodology. ## conclusion in this paper, we address a crucial contemporary concern: the safety of large language models. our approach entails a novel methodology to generate training data using llms and a multi-task learning approach to effectively integrate safeguard policies into scaled llms. the proposed approach is able to both assess the harmfulness of input queries and produce safeguard responses comparable to or even better than llms. moreover, this study is based on korean and can be used as a guide for other low-resource language-based safety studies in the future.