K o SBI : A Dataset for Mitigating Social Bias Risks Towards Safer Large Language Model Applications (2023)

Abstract: Large language models (LLMs) not only learn natural text generation abilities but also social biases against different demographic groups from real-world data. This poses a critical risk when deploying LLM-based applications. Existing research and resources are not readily applicable in South Korea due to the differences in language and culture, both of which significantly affect the biases and targeted demographic groups. This limitation requires localized social bias datasets to ensure the safe and effective deployment of LLMs. To this end, we present KosBi, a new social bias dataset of 34k pairs of contexts and sentences in Korean covering 72 demographic groups in 15 categories. We find that through filtering-based moderation, social biases in generated content can be reduced by 16.47%p on average for HyperClova (30B and 82B), and GPT-3.

## introduction large language models (llms) acquire impressive text generation abilities from large-scale real-world pre-training data @xcite . however, llms also absorb toxicity, such as social biases @xcite . this cannot be overlooked since the risk of generating toxic content impedes the safe use and potential commercialization of various downstream applications, such as ai assistants @xcite . to minimize the harm, numerous studies have tackled the detection and mitigation of toxicity in llms @xcite . each study typically leverages datasets capturing a specific type of toxicity, such as social bias @xcite or hate speech @xcite . these datasets are not only task-specific but also language-and culture-specific. for instance, consider hate speech made in south korea and in the united states. in addition to the language, the mainly targeted demographic groups also differfeminists and korean chinese in south korea, as opposed to african americans and jewish in the united states @xcite . also, the existing toxicity datasets in korean mostly focus on explicit hate speech and consider a limited number of targeted demographic groups @xcite @xcite . this calls for a dataset to address social biases against a more comprehensive set of demographic groups in south korea so that as many groups and people are protected. here we present the korean social bias (kosbi) dataset, a large-scale dataset of 34k pairs of contexts and sentences in korean with labels mainly capturing the presence of social biases. 1 it covers 72 targeted demographic groups in 15 categories, foot_1 which is much more comprehensive than existing datasets, as shown in demographic groups data source includes toxicity labels # cat. beep! @xcite 9,341 --news comments ✗ hate speech, bias apeach @xcite 3,770 10 -human-written ✗ offensive kold @xcite 40,448 5 19 news, youtube comments ✗ (title) offensive hatescore, unsmile @xcite 7 (mixed) news, online community comments ✗ hate speech, profanity k-mhas @xcite 109,692 7 -news comments ✗ hate speech, profanity shot learning @xcite . more specifically, we generated sentences and their respective contexts-which are also sentences, grammatically-for given target demographic groups. the generated contexts and sentences were then annotated by crowd workers as safe or unsafe. here, unsafe contexts and sentences were further labeled as expressions of stereotypes (cognitive bias), prejudice (emotional bias), discrimination (behavioral bias), and/or other, adopting the taxonomy by fiske (2023), 3 in figure 1 . with kosbi, we mitigate social biases in llmgenerated content using a filtering-based moderation approach, also known as rejection sampling @xcite . to do this, we first trained a safe sentence classifier using kosbi. then, for a given context, each llm was used to generate a pool of sentences from which the safest sentence was chosen by the classifier. the human evaluation shows that social biases in generated content are reduced by 16.47% on average for all three models tested-hyperclova (82b), hyperclova (30b), and gpt-3. ## the kosbi dataset this study aims to address social biases against a comprehensive set of demographic groups in south korea so as to make llms safer for as many groups and people as possible. (here, we focus on social biases without explicit hate speech, as existing datasets address the latter.) to achieve this, we wanted kosbi to consist of context-sentence pairs labeled as safe or unsafe for the demographic groups mentioned in them; this way, we can train llms to behave safely in the context of discussing a demographic group, rather than simply avoid it. ## experimental results to improve the safety of llms towards social groups, we explore a simple filtering-based moderation approach. in this section, we first build a safe sentence classification. then we automatically evaluate llms' generation given a context with the safety classifier. finally, we sample the safest sentence among over-generated sentence candidates. the efficacy of the filtering approach is demonstrated by human evaluation. ## conclusion to alleviate unsafe social bias of llms, we propose a large-scale social bias dataset related to safety addressing the korean language and cultures, kosbi. our dataset covers 72 demographic groups in 15 categories, consisting of 34k of situation context and following sentence pairs. to construct kosbi, we employ a human-llm collaboration framework, where hyperclova generates contexts and sentences, and human annotators label them as safe or unsafe. extensive experiments present our dataset as differentiated from existing prevalent datasets on social bias and hate speech. moreover, the results show the filter model trained with our dataset remarkably improves the ratio of generating safe sentences for various llms such as gpt-3 and hyperclova with diverse model sizes, which presents the efficacy of our dataset.