MIC o: Preventative Detoxification of Large Language Models through Inhibition Control (2024)

Abstract: Large Language Models (LLMs) are powerful tools which have been both dominant and commonplace in the field of Artificial Intelligence. Yet, LLMs have a tendency to devolve into toxic degeneration, wherein otherwise safe and unproblematic models begin generating toxic content. For the sake of social responsibility and inspired by the biological mechanisms of inhibition control, we introduce the paradigm of Education for Societal Norms (ESN). By collecting and labeling examples as acceptable and unacceptable (in this case toxic and non-toxic), and including a corresponding acceptable rewrite with every unacceptable example, we introduce a new mechanism for LLM detoxification. We annotate a dataset of 2,850 entries and use it to fine-tune a model, which we call a Model with Inhibition Control (MICo). Evaluating this model on toxicity detection capability, rewrite detoxification, meaning preservation, and overall toxicity reduction, we discover significant improvements over the baseline model. In our experiments we show that overall toxicity of this model is more than 60% reduced, with over 75% reduction in severe toxicity.

## introduction large language models (llms) are trained with the explicit purpose of serving humans, between providing information, presenting an engaging chat partner, and answering any number of other user requests. unfortunately, for a variety of reasons, there is a tendency for models to descend into neural toxic degeneration, outputting toxic and otherwise harmful messages @xcite @xcite . naturally, toxic prompts very commonly yield toxic responses, but many prompts which are entirely non-toxic also yield toxic responses @xcite @xcite . currently, there are three main directions being used to combat toxicity, each with a considerable drawback. first, the model may classify prompts as either toxic or non-toxic, and categorically refuse to respond to those deemed toxic @xcite . however, these approaches oftentimes use templated sentences to refuse to respond to toxic content which can degrade user engagement and lower helpfulness of the model @xcite . it is also possible for even entirely non-toxic prompts to yield toxic generations, and thus this method falls short of truly detoxifying. second, the model can be trained solely on non-toxic data @xcite . however, toxic content can be produced even from training data which appears benign. entirely purifying any word which may lead to toxicity will leave the training corpus narrow, impacting the richness of content which can be generated. furthermore, the model would not know how to respond to prompts which include some toxicity; thus, generations based around these prompts will be nonsensical, slashing the model's utility. third, an external classifier or a secondary model (e.g., in decoding time approaches) can be used to detect whether the model generated toxic content, and if so stop the content from reaching the user, instructing the model to provide another generation instead @xcite @xcite . however, without an understanding of toxicity, the model is likely to continuously generate toxic content, yielding potentially unbounded latency times. this would prove an impediment to the successful and user-friendly utilization of the slower text-based output and become unmanageable for models utilizing rapid speech-based communication. learning from these drawbacks, we formulate the following three requirements of a successful solution: (i) assuring non-toxic responses to non-toxic prompts. (ii) responding to toxic prompts in a natural, yet non-toxic manner. (iii) minimizing toxicity in real-time to prevent latency. despite the fact that ai has not yet provided a satisfactory solution, humans exhibit these traits due to their inherent ability of self reflection and inhibition control. healthy, mature humans consider the consequences of their speech (and more generally, behavior) via self-reflection before engaging in dialogue, and if determined to be negative, will alter the output to contain similar meaning yet eliminate the negative outcome. this necessitates a true understanding of what is deemed to be negative, i.e. toxic, and acquiring the ability to express oneself without causing harms to others. toward this goal, in this work, we propose the paradigm of education for societal norms (esn), which builds on acknowledging llms' capability to conduct few-shot learning @xcite . to identify toxic versus non-toxic generations, we provide examples of both (park and rudzicz, 2022), followed by their correct labels. detoxification is taught by appending a non-toxic meaning-preserving rewrite after each toxic generation. this way, each experience entry in the esn ends up with the desired non-toxic label. we collect these experiences and fine-tune the model on this dataset. we demonstrate that averaging over all prompts, the output produced by our model reduces toxic generation by over 60% and severe toxicity by over 75%. we refer to models trained with this method as models with inhibition control (mico). our contributions are the following: ## education for social norms methodology and evaluation education for social norms setup. we introduce an education paradigm that appears as a dataset of experiences which can teach the llm the two capabilities required for inhibition control simultaneously: identifying toxic vs. non-toxic texts and substituting toxic generation with meaning preserving non-toxic text. the dataset's entries take one of two forms: (a) a non-toxic content followed by the non-toxic label, "xxx [ntx]", or (b) a paired toxic content and its corresponding non-toxic rewrite along with their associated labels, e.g. "yyy [tx] xxx [ntx]" (xxx and yyy stand for non-toxic and toxic text respectively). education stems from first learning to label the text correctly according to its toxicity, and then assuring that each experience entry ends with the desired behavior and its [ntx] label. it is important that the ntx rewrites carry the same semantics, as otherwise the education would be functionally useless in terms of preserving meaning. fine-tuning dataset. we construct a dataset in a semiautomatic approach based on jigsaw's toxic comment classification challenge dataset foot_0 , which is composed of comments from wikipedia's talk page edits, as a base. for detoxification, we began with falcon40b, which is known to be among the most powerful models for its size @xcite . we discovered it was best at detoxification when using this instructiontuned model closer to a traditional llm, i.e. requesting sentence completion. we received an average of one satisfactory detoxification per three generations, and as such, we asked for four rewrites of each toxic comment. we then filtered by human annotation to select the best detoxification, and if none was found, generated one manually. we then took non-toxic comments along with the toxic/non-toxic rewrite pairs, and arranged them into the form discussed above. it is interesting to note that our database has only 2,850 entries and that this seems sufficient for teaching inhibition control by fine-tuning to the entries. we experimented with different ratio of non-toxic to toxic entries in the dataset ratios, and we found the 9:1 ratio of non-toxic to toxic to be most successful in reaching the educational goals. evaluation. for evaluation, we select realtoxici-typrompts, a dataset of close to 100k prompts, both toxic and non-toxic, which have been shown to yield high toxicity in generations of sentence-completion models, and is commonly used for toxicity evaluation @xcite @xcite . for proof of concept, we test it on relatively small llms and fine-tuned for two epochs. gpt-neo (125m parameters) learned to append '[tx]' or '[ntx]' but with a significant preference to @xcite and completely missed the rewrite component. gpt-neo (1.3b parameters) reduced toxicity in rewrites but only very slightly as smaller models provided insufficient for the complicated task, we settle on opt's 6.7 billion parameter model @xcite , which was large enough to be educated. it was trained with the 9:1 non-toxic to toxic education dataset using qlora @xcite . as with the original jigsaw dataset, we consider the toxicity score generated by perspectiveapi to be the ground truth for toxicity (although we found inaccuracy in this method as well @xcite ). for evaluation, we provide two different toxicity axes. binary toxicity is the option of rounding the toxicity so that below 0.5 is considered non-toxic whereas at or above 0.5 is considered toxic. we find this analysis to be too sensitive to the hard boundary and hence not that informative. we thus mainly consider graded-toxicity, separating the text entries to five bins according to the labels: most non-toxic (0-0.1), non-toxic (0.1-0.35), partially/potentially toxic (0.35-0.65), toxic (0.65-0.90), and severely toxic at (0.9-1). these ranges provided higher robustness. ## experimental results and analysis there are five important criteria which our method should fulfill. after detoxification, a plurality from each category becomes highly non-toxic, with decreasing percentages going to increasingly toxic categories. our detoxification provides 94.2% less severe toxicity, 84.6% less toxicity, and 82× more comments in the highly non-toxic category. ## related work there is an abundant body of work in the area of toxicity detection @xcite and mitigation @xcite @xcite . while some mitigation strategies focused on train time solutions @xcite , others tackled this problem during decoding time @xcite . however, these toxicity mitigation approaches either relied on an external classifier or secondary models to detect toxic generation to guide the model toward nontoxic generation which adds to the latency @xcite @xcite or they relied on curating a non-toxic dataset to train their models over which can make the model less helpful in certain situations @xcite . in this work we introduce a learning paradigm that will address existing limitations from prior work. ## discussion this paper introduces a new method of reducing toxicity in llms. the fundamental idea is to create an education process which teaches the llm to attain inhibition control, responsible for controlling impulses and alignment with social norms. the education procedure designed for toxicity reduction teaches the llm the two fundamental features of inhibition control at the same time: the ability to separate toxic from non-toxic behavior and the consistent requirement of replacing each toxic generation with a meaning preserving non-toxic one. we propose to continue this work toward further reduction in toxicity, particularly toward individualization. different individuals and cultures have differences in the interpretation of what is understood as toxicity. human inhibition control and self-reflection has an important feature, of predicting the effect of own behavior on different individuals or cultural groups. as such, it is our responsibility to incorporate users opinion and feedback @xcite . thus, going forward, we wish to enable customization of the language model by learning the individual interpretation of toxicity.