Low-Resource Counterspeech Generation for I ndic Languages: The Case of B engali and H indi (2024)

Abstract: With the rise of online abuse, the NLP community has begun investigating the use of neural architectures to generate counterspeech that can “counter” the vicious tone of such abusive speech and dilute/ameliorate their rippling effect over the social network. However, most of the efforts so far have been primarily focused on English. To bridge the gap for low-resource languages such as Bengali and Hindi, we create a benchmark dataset of 5,062 abusive speech/counterspeech pairs, of which 2,460 pairs are in Bengali, and 2,602 pairs are in Hindi. We implement several baseline models considering various interlingual transfer mechanisms with different configurations to generate suitable counterspeech to set up an effective benchmark. We observe that the monolingual setup yields the best performance. Further, using synthetic transfer, language models can generate counterspeech to some extent; specifically, we notice that transferability is better when languages belong to the same language family.

## introduction the rise of online hostility has become an ominous issue endangering the safety of targeted people and groups and the welfare of society as a whole @xcite @xcite . therefore, to mitigate the widespread use of such hateful content, social media platforms generally rely on content moderation, ranging from deletion of hostile posts, shadow banning, suspension of the user account, etc. @xcite . however, these strategies could impose restrictions on freedom of expression @xcite . hence one of the alternative approaches to combat the rise of such hateful content is counterspeech (cs). cs is defined as a non-negative direct response to abusive speech (as) that strives to denounce it by diluting its effect while respecting human rights. it has already been observed that many ngos are deploying volunteers to respond to such hateful posts to keep the online space healthy @xcite . even social media platforms like facebook have developed guidelines for the general public to counter abusive speech online foot_1 . however, due to the sheer volume of abusive content, it is an ambitious attempt to manually intervene all hateful posts. thus, a line of nlp research focuses on semi or fully-automated generation models to assist volunteers involved in writing counterspeech @xcite @xcite . these generation models seek to minimize human intervention by providing ideas to the counter speakers that they can further post-edit if required. however, the majority of these studies are concentrated on the english language. hence effort is needed to develop datasets and language models (lms) for low-resource languages. in the past few years, several smearing incidents, such as online anti-religious propaganda, cyber harassment, smearing movements, etc., have been observed in bangladesh and india @xcite . bangladesh has more than 150 million people with bengali as the official language foot_2 , and india has more than 1.3 billion people, with hindi and english as the official language foot_3 . so far, several works have been done to detect malicious content in bengali and hindi @xcite . however, no work has been done to generate automatic counterspeech for these languages. our key contributions in this paper are as fol-lows: this section briefly discusses the relevant work for abusive speech countering on social media platforms and the existing methodologies for cs generation strategies. online abuse countering: a series of works have investigated online abusive content, aiming to study the online diffusion of abuse @xcite and creating datasets for abuse detection @xcite @xcite considering several multilingual languages. in many cases such detection models are used to censor abusive content which may curb the freedom of speech @xcite . therefore as an alternative, ngos have started employing volunteers to counter online abuse @xcite . previous studies on countering abusive speech cover several aspects of cs, including defining counterspeech @xcite , studying their effectiveness @xcite , and linguistically characterizing online counter speakers' accounts @xcite . cs dataset: so far, several strategies have been followed for the collection of counterspeech datasets. mathew et al. (2019b) crawled comments from youtube with the replies to that comments and manually annotated the hateful posts along with the counterspeech responses. chung et al. (2019) created three multilingual datasets in english, french, and italian. to construct the dataset, the authors asked native expert annotators to write hate speech, and with the effort of more than 100 operators from three different ngos, they built the overall dataset. fanton et al. (2021) proposed a novel human-in-theloop data collection process in which a generative language model is refined iteratively. to our knowledge, no dataset has been built for low-resource languages such as bengali and hindi; therefore, in this work, we construct a new benchmark dataset of 5,062 as-cs pairs for two indic languages -bengali and hindi. cs generation: several studies have been conducted for the generation of effective counterspeech. qian et al. (2019) employ a mix of automatic and human interventions to generate counternarratives. tekiroglu et al. (2020) presented novel techniques to generate counterspeech using a gpt-2 model with post-facto editing by the experts/annotator groups. zhu and bhat (zhu and bhat, 2021) suggested an automated pipeline of candidate cs generation and filtering. chung et al. (2020) investigated the generation of italian cs to fight online hate speech. recently tekiroglu et al. (2022) performed a comparative study of counter-narratives generations considering several transformer-based models such as gpt-2, t5, etc. so far, no work has examined the generation of counterspeech for under-resourced languages such as bengali and hindi; therefore, we attempt to fill this critical gap by benchmarking various transformer-based language models. ## post-editing evaluation we further wanted to assess the utility of the automatically generated responses for the potential moderators who would be using the generated css in combating abusive speech on social media. the ideal case would be if they are needed to make absolutely no changes in the generated css before posting them on social media. the larger the number of edits they would need to make in the generated cs, the lesser would be its utility. we therefore asked human judges to make necessary edits they would perform before posting the responses on social media. this experiment focused on cs generated in the monolingual setting. we used the translation edit rate (ter) @xcite , a metric analogous to the edit distance to quantify the dissimilarity between the generated cs and edited cs. this experiment exclusively considers posts selected during human evaluation (@xmath0, calculating ter and the average length of the counterspeech. the results are noted in an observation across all models indicates that chatgpt-generated css tend to be lengthy. hence, annotators had to eliminate certain portions of unnecessary text during the editing process, resulting in a higher ter for chatgpt in both languages. the average length of generated cs is ∼65 for bengali and ∼103 for hindi. we believe longer css can be cumbersome to read and have minimal impact on the abusive speaker. in contrast, bloom and mt5-based models exhibit a relatively lower average length of cs, making them more suitable for mitigating abusive speech. ## conclusion counterspeech generation using neural architecture-based language models has started gaining attention for interventions against hostility. this paper presents the first attempt at cs generation for the bengali and hindi languages, investigating several generation models. to facilitate this, we create a new benchmark dataset of 5,062 as-cs pairs, of which 2,460 pairs are in bengali and 2,602 pairs are in hindi. we experiments with several interlingual transfer mechanisms. our findings indicate that the overall monolingual setting exhibits the best performance across all the setups. joint training can be performed if one omnipresent model is beneficial to generate css for multiple languages. we also notice synthetic transferability yields better results when languages belong to the same language family. in future, we plan to explore methods for improving specificity by using various types of knowledge (e.g., facts, events, and named entities) from external resources. further, we plan to add controllable parameters to the counterspeech generation setup, enabling moderators to customize the counterspeech toward a specific technique we have discussed.