Multilingual Counter Narrative Type Classification (2021)

Abstract: The growing interest in employing counter narratives for hatred intervention brings with it a focus on dataset creation and automation strategies. In this scenario, learning to recognize counter narrative types from natural text is expected to be useful for applications such as hate speech countering, where operators from non-governmental organizations are supposed to answer to hate with several and diverse arguments that can be mined from online sources. This paper presents the first multilingual work on counter narrative type classification, evaluating SoTA pre-trained language models in monolingual, multilingual and cross-lingual settings. When considering a fine-grained annotation of counter narrative classes, we report strong baseline classification results for the majority of the counter narrative types, especially if we translate every language to English before cross-lingual prediction. This suggests that knowledge about counter narratives can be successfully transferred across languages.

## introduction the pervasive problem of online hate speech (hs) has motivated the research community to investigate methods for mitigating hatred, such as hate speech detection @xcite and, more recently, hate moderation through counter narratives (cns). cns are non-negative responses to hateful messages providing fact-bound arguments or alternative viewpoints. distinct from standard approaches to hate intervention by content moderation, counter narratives are preferable as they preserve the right to freedom of speech and encourage peaceful conversations @xcite . in particular, the nlp community has started exploring cn generation @xcite @xcite @xcite , also in multilingual settings @xcite with the aim of help-ing non-governmental organizations (ngos) fight hs. however, automatic generation and detection of counter narratives still face three important challenges. first, counter narratives vary considerably in manner of expression and strategies @xcite , see in our view, automated mechanisms to distinguish cn characteristics are crucial for developing hate countering applications that can address personalization or for tasks such as counter narrative mining from user-generated content. classifying counter narratives would also help to establish which counter narratives are more effective to target hate speech. finally, it may also be useful for ngo operators to answer hate speech with diverse counter narratives (not just the most common ones) which could be mined from online sources. hs: muslims and non-british people are raping, enslaving and murdering our women! how disgusting! cn1 (denouncing): this is not acceptable. hatred cannot be tolerated and must be stopped. cn2 (facts): please notice that, regardless of religions, huge numbers of girls and children experience domestic sexual abuse by a partner or family member. we should encourage people to help victims and prevent sexual crimes. to the best of our knowledge, this work is the first to tackle counter narrative classification from a multilingual perspective, predicting the types/strategies employed in a counter narrative message given a hateful message. other contributions include: (1) the first empirical analysis of counter narrative characteristics on expertbased hate countering data targeting islamophobia @xcite ; (2) strong baseline re-sults for multilingual and multi-class counter narrative classification by experimenting with the large pre-trained model xlm-r @xcite ; and (3) a set of cross-lingual zero-shot experiments showing that huge improvements can be obtained by translating the training data from other languages to english, which is coherent with previous work on the multilinguality of language models @xcite . this result also shows that common characteristics of some counter narrative types are carried across languages. data splits and training scripts are available at https://github.com/yilingchung/ multilingualcn-classification . ## counter arguments v.s. counter narratives while counter arguments and counter narratives have a similar purpose -to present an alternative stance to a statement -a counter argument is not necessarily a counter narrative, and vice versa. the main differences between the two are briefly discussed below, and point for the development of specific classification approaches for cns. presence of an argument. a counter argument is typically defined as an argument (equivalent to a claim or standpoint) contradicting the initial statement with supporting evidence or reasoning @xcite @xcite . the presence of supported evidence or reasoning is essential for counter-argumentation and argumentative information retrieval, regardless of it being implicit or explicit (e.g., statement: nuclear energy should be used to replace fossil fuels. v.s. counter argument: renewable energy is a better option for replacing fossil fuels than unsustainable and expensive energy like nuclear power.). in contrast, counter narratives can be formulated without using arguments. for example, a counter narrative can be without supporting evidence (e.g., hate speech: islam is a plague. v.s. counter narrative: how can you use such language to describe a religion of 1.6 billion people?). non-negative arguments. following guidelines foot_0 generally adopted by ngos, an appropriate counter narrative should not contain hostile language, prejudice, and unlawful content. using negative tones as responses may risk escalating the conversation and engaging in hateful sentiment. ## counter narrative type classification classifying counter narrative types is a new and challenging task. we hypothesize that underlying linguistic features in one language can facilitate the classification performance in other languages. accordingly, we approach this task in a multilingual setting focusing on english, french and italian. we experimented in the following learning settings: ## experimental results similarly to stance detection and argument mining tasks @xcite , we report average macro f1 score over the cn types to avoid obtaining very high scores simply deriving from the dominant classes in the dataset. furthermore, we also follow previous literature on stance detection @xcite and report only the performance over the relevant classes. monolingual results. as shown by in a post-hoc manual analysis, annotators expressed difficulties in differentiating these two classes, difficulties illustrated by the examples provided in multilingual results. as it is reported in zero-shot cross-lingual results. for each target language, we provide the average f1 of zero-shot cross-lingual models trained on the other 2 source languages in zero-shot translated results. data augmentation through translation has been widely employed to improve classification performance @xcite , also in cross-lingual settings @xcite . at the same time, the crosslingual capabilities of transformer models such as xlm-r and mbert are being actively investigated @xcite @xcite . thus, we conduct an additional experiment adding, to the english training set, the manually translated italian and french training data before testing on the target languages (italian and french). the aim is to investigate if xlm-r benefits from fine-tuning on a high resource language (english) instead of combining english with other languages, such as italian and french, which are not so well represented in pre-trained multilingual models @xcite @xcite . by doing so, in this 'zero-shot transl en ' setting we aim to expose the model with semantic knowledge from some target languages without actually seeing those languages (italian and french). ## related work broadly speaking, counter narrative type classification is related to stance detection @xcite , which is crucial for argument search @xcite . in contrast to stance detection, that concentrates on binary or relatively simple classification -e.g., determine if an argument supports or contests a given topic @xcite @xcite -we present a multiclass approach to counter narrative classification. counter narratives have been adopted as a direct and effective response to online hatred in several campaigns and on social media platforms including twitter @xcite , facebook @xcite , and youtube @xcite . although it has been argued that hate speech detection can benefit from cn classification, there are very few studies on this regard, with only one previous work on classifying counter narrative types @xcite . however, unlike our present work, they consider hostile language as one of the main types of counter narratives, which is explicitly discouraged by ngos working on hatred intervention. furthermore, we investigate multilingual and cross-lingual cn classification leveraging a sota pre-trained multilingual language model. ## conclusion we present the first work on multilingual cn type classification. our results show that: (i) the performance is promising for the majority classes (facts, question, denouncing); (ii) classifying humor and hypocrisy cns is still challenging; (iii) combining training data from the three source languages improves performance over the monolingual evaluation; and (iv), the best overall results are obtained in the 'zero-shot transl en ' approach where the training data for italian and french is translated to english. this shows that some knowledge about cns is transferred across languages. while this is coherent with previous literature about multilingual language models, the exact source of such successful transfer across languages remains an open topic.