T o V i L a G : Your Visual-Language Generative Model is Also An Evildoer (2023)

Abstract: Recent large-scale Visual-Language Generative Models (VLGMs) have achieved unprecedented improvement in multimodal image/text generation. However, these models might also generate toxic content, e.g., offensive text and pornography images, raising significant ethical risks. Despite exhaustive studies on toxic degeneration of language models, this problem remains largely unexplored within the context of visual-language generation. This work delves into the propensity for toxicity generation and susceptibility to toxic data across various VLGMs. For this purpose, we built ToViLaG, a dataset comprising 32K co-toxic/mono-toxic text-image pairs and 1K innocuous but evocative text that tends to stimulate toxicity. Furthermore, we propose WInToRe, a novel toxicity metric tailored to visual-language generation, which theoretically reflects different aspects of toxicity considering both input and output. On such a basis, we benchmarked the toxicity of a diverse spectrum of VLGMs and discovered that some models do more evil than expected while some are more vulnerable to infection, underscoring the necessity of VLGMs detoxification. Therefore, we develop an innovative bottleneck-based detoxification method. Our method could reduce toxicity while maintaining comparable generation quality, providing a promising initial solution to this line of research.

## introduction thriving on the capabilities of transformer architectures @xcite , language/visual pretraining @xcite @xcite and diffusion models @xcite , recent large-scale visual-language generation 1 models (vlgms) have made extraor-two tote bags saying fuck, bruce, and silence. ## towards vlg toxicity investigation we develop a systematic solution to study vlg toxic degeneration: sec. 3.1 presents our tovilag dataset, sec. 3.2 introduces the toxicity detection, sec. 3.3 demonstrates the wintore metric, and sec. 3.4 provides the detoxification method, smib. ## toxicity analysis of vlg models as a preliminary toxicity examination, fig. 2 illustrates the proportion of input images eliciting toxic outputs under various vlgms. we find that these popular models yield an unexpectedly high degree of toxicity even trained with carefully-crafted and relatively clean data (see appendix a). for instance, among 100k generated samples, blip produces toxic captions from up to 30% input images. this indicates that vlgms would do more evil when deployed in diverse real-world application scenarios, emphasizing the importance of comprehensive toxicity analyses. to respond to questions q1 and q2 posed in sec. 1, we perform two kinds of experiments. ## detoxification experiments settings we perform detoxification experiments on i2t generation and consider three models: blip @xcite , the most toxic one under our evaluation; git @xcite with high toxicity and insensitivity to toxicity control; @xcite which is more susceptible to toxicity injection. the mapping function f θ and classifier p ϕ are both implemented as multi-layer perceptron (mlp) and appended to the visual encoder of each model. we use 5,000 non-toxic image-text pairs from coco and 5,000 toxic ones from our co-toxic pairs for training. β = 0.01 in eq.( 4 ). we use adamw @xcite (batch @xmath0. for toxicity evaluation, we report tp and wintore (wtr). besides, we also assess the generation quality using bertscore (bs) @xcite , rouge (r) @xcite , and clipscore (cs) @xcite . more setting details are listed in appendix b. baselines we compared our detoxification method smib with two baseline methods. the first is a word filtering method, which directly filters out the prohibited candidate tokens foot_1 from the output distribution. the second is an output rectification method called fudge @xcite , which learns an attribute predictor to adjust the original probabilities of the model. ## conclusion and future work in this work, we delve into the unexplored toxic degeneration problem of vlgms. to examine the propensity for susceptibility to toxicity across different vlgms, we construct tovilag, a dataset comprising toxic text-image data, and introduce wintore, a novel toxicity metric devised for vlg. we benchmark the toxicity of a broad range of models and reveal that existing models might do more evil than expected. we then propose a novel detoxification method, smib, to reduce toxicity without significantly sacrificing generation quality. our source code, the wintore script and other resources are available at https://github.com/ victorup/tovilag . in the future, we plan to apply our smib method to t2i models and investigate the underlying mechanism of toxicity generation. we will also endeavour to expand our research to wider ethical risks, striving towards an ideal ethical future for vlg.