Query-Efficient Black-Box Red Teaming via B ayesian Optimization (2023)

Abstract: The deployment of large-scale generative models is often restricted by their potential risk of causing harm to users in unpredictable ways. We focus on the problem of black-box red teaming, where a red team generates test cases and interacts with the victim model to discover a diverse set of failures with limited query access. Existing red teaming methods construct test cases based on human supervision or language model (LM) and query all test cases in a brute-force manner without incorporating any information from past evaluations, resulting in a prohibitively large number of queries. To this end, we propose Bayesian red teaming (BRT), novel query-efficient black-box red teaming methods based on Bayesian optimization, which iteratively identify diverse positive test cases leading to model failures by utilizing the pre-defined user input pool and the past evaluations. Experimental results on various user input pools demonstrate that our method consistently finds a significantly larger number of diverse positive test cases under the limited query budget than the baseline methods.The source code is available at https://github.com/snu-mllab/Bayesian-Red-Teaming .

## introduction recently, generative models have demonstrated exceptional performance on a broad range of generation tasks, including open-domain dialogue, prompt continuation, and text-to-image generation, thanks to the rise of large-scale models such as blenderbot, gopher, gpt-3, palm, and dall•e 2 @xcite @xcite @xcite . while utilizing large models in commercial systems can provide significant benefits, it also poses a risk of unexpectedly causing harm to users, such † correspondence to hyun oh song. here, our edit-based brt is applied to blenderbot-3b using the user input from bot adversarial dialogue. as the generation of offensive responses or nsfw images @xcite . thus, it is essential to identify and prevent these failures before deployment to avoid severe ramifications to society @xcite . the primary goal of red teaming is to identify many diverse positive test cases which lead to model failures @xcite . due to the high computation cost of large models during inference and the potential security risk of exposing the model parameters, we consider the black-box scenario in which the red team can only observe the output of the victim model within a limited query budget @xcite @xcite . prior red teaming methods use human-designed prompts as test cases and query the test cases in a brute-force manner to identify model failures. these approaches usually require a prohibitively large number of queries to the victim model as they do not utilize any information from past evaluations during the red teaming process @xcite @xcite . a recent work proposes language model (lm)-based red teaming methods, which construct a user input pool by zero-shot generation method and utilize the user input pool to generate test cases that are more likely to be positive. however, lm-based red teaming methods require access to victim model outputs of entire user input pool, which is prohibitive in the black-box setting @xcite . to this end, we propose bayesian red teaming (brt), query-efficient black-box red teaming methods based on bayesian optimization (bo). first, brt constructs a user input pool that consists of user inputs generated by human supervision or lm, e.g., utterances from the dialogue dataset or zeroshot generated utterances (figure 1 ). then, brt sequentially generates test cases that lead to diverse positive test cases by choosing or editing user inputs from a pre-defined user input pool. for each step, brt utilizes past evaluations, to fit a gaussian process (gp) model. based on the gp model, brt generates the next test case most likely to be positive while encouraging diversity. our results demonstrate that brt discovers a significantly larger number of positive test cases with higher diversity than the baseline methods under a limited query budget on various generations tasks, open domain dialogue, prompt continuation, and text-to-image generation. notably, edit-based brt discovers more than 3x larger number of positive test cases with higher diversity than the baseline methods on the bloom zs user input pool against blenderbot-3b under a query limit of 20,000. ## related work a line of research utilizes manually designed templates to detect the model failures. garg et al. (2019) and ribeiro et al. (2020) in the perspective of related recent machine learning techniques, there has been a growing interest in utilizing bo to uncover the vulnerability of models. @xcite , @xcite , and @xcite conduct bo to search adversarial examples against classification models on image, graph, and text domains. @xcite improve the scalability of bo by utilizing the subset of data (sod) method and batching based on dpp prior @xcite . ## conclusion our work aims to identify the potential risk of offensive behavior in black-box large-scale generative models by red teaming in a limited query regime. we propose brt, a novel query-efficient black-box red-teaming method using bo. brt methods construct a user input pool and iteratively choose or edit user inputs using bo to generate diverse positive test cases. in contrast to prior works, brt can incorporate the information from past evaluations using gp to efficiently identify diverse failures. the experimental results show that brt consistently outperforms existing methods in finding a greater number of positive test cases with higher diversity on various generation tasks including open-domain dialogue, prompt continuation, and text-to-image generation, against various victim models under a query limit.