L o RA -Guard: Parameter-Efficient Guardrail Adaptation for Content Moderation of Large Language Models (2024)

Abstract: Guardrails have emerged as an alternative to safety alignment for content moderation of large language models (LLMs). Existing model-based guardrails have not been designed for resource-constrained computational portable devices, such as mobile phones, more and more of which are running LLM-based applications locally. We introduce LoRA-Guard, a parameter-efficient guardrail adaptation method that relies on knowledge sharing between LLMs and guardrail models. LoRA-Guard extracts language features from the LLMs and adapts them for the content moderation task using low-rank adapters, while a dual-path design prevents any performance degradation on the generative task. We show that LoRA-Guard outperforms existing approaches with 100-1000x lower parameter overhead while maintaining accuracy, enabling on-device content moderation.

## introduction large language models (llms) have become increasingly competent at language generation tasks. the standard procedure for training llms involves unsupervised learning of language structure from large corpora (pre-training; @xcite ; followed by supervised fine-tuning on specific tasks. for instance, conversational assistants (or chat models) are trained to respond to questions by providing answers which are aligned with human preferences (instruction tuning; @xcite . a known failure mode of llms is their propensity to generate undesirable content, such as offensive language or illegal advice. this is due to the presence of such material in their pre-training datasets, e.g., common crawl @xcite . this behaviour is detrimental to safety and arises as an unintended consequence of their ability to generate helpful answers or responses which are coherent with user input @xcite . to mitigate this problem, models have been optimised to not only follow instructions, but also respond in a manner that is safe and aligned with human values (safety tuning; @xcite . however, these models are still susceptible to jail-break attacks, which evade the defences introduced by safety tuning via strategies such as using lowresource languages in prompts, refusal suppression, privilege escalation and distraction @xcite @xcite . this has motivated the development of guardrails which monitor exchanges between chat models and users, flagging harmful entries. due to the failures of inbuilt safety mechanisms, guardrails form an important component of the ai safety stacks in deployed systems @xcite . typically, model-based guardrails (guard models) are distinct from the models used in the chat application being monitored @xcite . 1 however, this introduces an overhead which is prohibitive in low-resource settings. learning is also inefficient: language understanding abilities of the chat models must significantly overlap those of the guard models if both are to effectively perform their individual tasks (response generation and content moderation, respectively). in this paper, we propose lora-guard, which deduplicates these abilities via parameter sharing and parameter-efficient fine-tuning. lora-guard uses a low-rank adapter (lora; @xcite on the backbone transformer of the chat model to achieve a memory efficient, integrated chat and guard system. the transformer parameters are frozen, while the lora parameters are trained to detect harmful content. the lora parameters can be activated for guardrailing, in which case harmfullness scores are provided by a classification head, and deactivated for chat usage, in which case the original chat model is recovered by passing the transformer outputs through the original language modelling head. contributions we present lora-guard, an efficient content moderation framework for chat applications, allowing for guard model deployment in resource-constrained settings. lora-guard provides guard model systems with vast reductions in parameter overheads with respect to current state of the art (100-1000x reduction in our experiments) while maintaining or improving content moderation accuracy (see fig. 2 ). we give performance evaluations of lora-guard both in-distribution and for zero-shot generalisation on out-of-distribution data. ## methodology a guard model g for a generative chat model c categorizes each input and/or corresponding output of c according to a taxonomy of harmfulness categories. the taxonomy could include coarse-grained categories, such as safe and unsafe, or could further distinguish between fine-grained categories, such as violence, hate, illegal, etc. we now introduce lora-guard. we assume a chat model c consisting of an embedding ϕ, a feature map f and a linear language modelling head h chat . the embedding maps tokens to vectors, the feature map (a transformer variant; @xcite maps these vectors into representations and the language modelling head maps these representations into next-token logits. if x represents a tokenized input sequence, then the next token logits are computed by h chat (f (ϕ(x))). we propose to build the guard model g using parameter-efficient fine-tuning methods applied to f , and instantiate this idea with lora adapters, which add additional training parameters in the form of low-rank (i.e. parameter-efficient) matrices (see appendix a for details). other adaptation methods are possible @xcite @xcite . the same tokenizer and embedding is used for c and g. however, g uses a different feature map f ′ chosen as lora adapters attached to f , and uses a separate output head h guard (linear, without bias), which maps features to harmfulness categories. tokenized content x is therefore classified by h guard (f ′ (ϕ(x))). deactivating the lora adapters and using the language modelling head gives the original chat model, while activating the lora adapters and using the guard model head gives the guard model. these generative and guarding paths, respectively, are depicted in figure 1 . we do not merge the lora adapters after training. the dual path design of lora-guard opens the door to methods based on adaptation instead of safety alignment fine-tuning. adaptation has an important advantage over safety alignment fine-tuning: the generative task is unaffected, so lora-guard avoids any performance degradation on the generative task from safety fine-tuning (catastrophic forgetting; @xcite . most parameters, namely those in f , are shared between the generative and guarding paths. therefore, the parameter overhead incurred by the guard model is only that of the lora adapters f ′ and of model auprc↑ precision↑ recall↑ f1↑ guard overhead↓ toxicchat-t5-large (a) .89 .80 .85 .82 7.38 × 10 8 openai moderation (a) .63 .55 .70 .61 -llama-guard (b) .63 ---6.74 × 10 9 llama-guard-fft (c) .81 ---6.74 × 10 9 llama2-7b-base-fft (c) .78 ---6.74 × 10 9 lora-guard3 ). these models are fully fine tuned on the full training set of toxicchat. it is not stated explicitly around @xcite , figure 3 ), but by comparison with @xcite , the guard output head h guard . this is a tiny fraction of the number of parameters used by the chat system, often 3 orders of magnitude smaller, as shown in ## conclusion lora-guard provides guardrails for conversational systems at a vastly reduced parameter overhead when compared with standard approaches (100-1000x less in our experiments). moreover, this reduction in memory requirements comes without loss of chat performance and with moderation performance competitive with or surpassing the state of the art on benchmark tasks. these are due, respectively, to a dual-path design and the knowledge sharing in parameter-efficient fine-tuning. we consider lora-guard to be an important contribution to guardrail methods for resource-constrained settings such as on-device llms. potential risks a potential risk of the deployment of any guardrail system is distribution shift: encountering harmful content at test-time which is significantly different from that which the model was trained on (e.g., an entirely new category of harmful content). this risk can be mitigated by further work to improve out of distribution generalisation.