PANDA - Paired Anti-hate Narratives Dataset from A sia: Using an LLM -as-a-Judge to Create the First C hinese Counterspeech Dataset (2025)

Abstract: Despite the global prevalence of Modern Standard Chinese language, counterspeech (CS) resources for Chinese remain virtually nonexistent. To address this gap in East Asian counterspeech research we introduce the a corpus of Modern Standard Mandarin counterspeech that focuses on combating hate speech in Mainland China. This paper proposes a novel approach of generating CS by using an LLM-as-a-Judge, simulated annealing, LLMs zero-shot CN generation and a round-robin algorithm. This is followed by manual verification for quality and contextual relevance. This paper details the methodology for creating effective counterspeech in Chinese and other non-Eurocentric languages, including unique cultural patterns of which groups are maligned and linguistic patterns in what kinds of discourse markers are programmatically marked as hate speech (HS). Analysis of the generated corpora, we provide strong evidence for the lack of open-source, properly labeled Chinese hate speech data and the limitations of using an LLM-as-Judge to score possible answers in Chinese. Moreover, the present corpus servers as the first East Asian language based CS corpus and provides an essential resource for future research on counterspeech generation and evaluation.

## introduction hate speech is typically characterized as any form of communication that demeans a specific group of people based on attributes like race, ethnicity, gender, sexual orientation, or religion @xcite . while hs may constitute a small proportion of social media content, its impact is significant, affecting nearly one-third of the population @xcite . the proliferation of hate speech on social media platforms has become a significant societal concern. while traditional approaches to mitigating hs have focused on content removal and moderation, these methods often raise concerns about freedom of speech. in response, counterspeech has emerged as a promising alternative strategy to combat hs while preserving free expression @xcite . counterspeech, defined as communication that aims to counteract potential harm caused by other speech, has shown effectiveness in real-world studies @xcite . however, the manual creation of cs is time-consuming and challenging to scale given the volume of hs online. this has led to increased interest in automated cs generation using nlp techniques. ## methodology this section provides an overview of the targets we set when making this dataset (3.1), the sourcing of data (3.2), the pre-processing of data (3.3), generation of cs (3.4), and annotation methods (3.5). finally, we also provide statistics relating to the original hs data (swsr, cold, chsd) 63,879 examples pre-processing (keeping hate-labeled entries) 26,420 examples of hs llm-assisted rating (labeling hate level) 26,420 examples of hs selecting hs 2,974 examples of hs ai generate answers 17,844 examples of cs round robin selection 11,896 examples of cs a 2 a 1 ... a n combined output 2,974 annotated pairs of hs and cs figure 1: proposed data processing pipeline for creating the chinese counterspeech corpus. a 1 through a n refer to n annotators that participated in this project. ## general discussion the objectives of this study were threefold: (1) to create a paired hate-speech-counterspeech (hs-cs) corpus in mandarin chinese by leveraging an llm-as-judge pipeline, (2) to assess the extent