SQ u AR e: A Large-Scale Dataset of Sensitive Questions and Acceptable Responses Created through Human-Machine Collaboration (2023)

Abstract: The potential social harms that large language models pose, such as generating offensive content and reinforcing biases, are steeply rising. Existing works focus on coping with this concern while interacting with ill-intentioned users, such as those who explicitly make hate speech or elicit harmful responses. However, discussions on sensitive issues can become toxic even if the users are well-intentioned. For safer models in such scenarios, we present the Sensitive Questions and Acceptable Response (SQuARe) dataset, a large-scale Korean dataset of 49k sensitive questions with 42k acceptable and 46k non-acceptable responses. The dataset was constructed leveraging HyperCLOVA in a human-in-the-loop manner based on real news headlines. Experiments show that acceptable response generation significantly improves for HyperCLOVA and GPT-3, demonstrating the efficacy of this dataset.

## introduction the fast evolution of large language models (llms) is accompanied by a growing potential for harm @xcite , such as their generating offensive expressions @xcite , and propagating prejudices @xcite @xcite . as initial steps to cope with such risks, recent works mainly target scenarios in which llms interact with illintentioned users: those who explicitly make offensive remarks @xcite , and those who make adversarial attacks to elicit harmful responses @xcite @xcite , for instance. email to: {hwaran.lee, jungwoo.ha}@navercorp.com, seokhee.hong@vision.snu.ac.kr however, interactions with well-intentioned users can also turn toxic if llms do not respond to sensitive questions carefully. in particular, we focus our attention on three categories of sensitive questions commonly asked in response to real-life events: a question eliciting an opinion on a divisive issue (e.g., q1 in in this paper, we present the sensitive questions and acceptable responses (square) dataset, a large-scale korean dataset of 49k sensitive questions with 42k acceptable and 46k nonacceptable responses. 1 to create realistic questions and responses, we fed real news headlines from popular news media in south korea to hyperclova @xcite when generating questions and responses using demonstration-based prompting @xcite . then, following @xcite ; @xcite , only ambiguous cases identified by a filter model were manually labeled by crowd-workers according to a taxonomy of sensitive questions and acceptable responses. here, the filter model was incrementally improved by refinement through three human-inthe-loop iterations for increased reliability. to demonstrate the efficacy of the dataset, we experimented with a straightforward use case of our dataset-training an acceptable response clas- (should school meals be free of charge? our main contributions are as follows: ## efficacy validation for square in this section, we moderate llms to output acceptable responses and to be robust to sensitive questions. for that, we introduce a simple but still effective filter-based moderation approach: generating multiple responses and outputting the most acceptable one with respect to an acceptable response classifier. we start by training an acceptable response classifier using square and proceed to filter-based moderation. ## conclusion in the midst of active research on making llms safer, interactions with well-intentioned users on sensitive issues have been largely overlooked. to this end, we presented the sensitive questions and acceptable responses (square) dataset, a largescale korean dataset of 49k sensitive questions with 42k acceptable and 46k non-acceptable responses. we showed the efficacy of our dataset through experiments in which the acceptable response rate significantly increased in two popular llms that can effectively handle korean, hyperclova and gpt-3.