Human-in-the-Loop for Data Collection: a Multi-Target Counter Narrative Dataset to Fight Online Hate Speech (2021)

Abstract: Undermining the impact of hateful content with informed and non-aggressive responses, called counter narratives, has emerged as a possible solution for having healthier online communities. Thus, some NLP studies have started addressing the task of counter narrative generation. Although such studies have made an effort to build hate speech / counter narrative (HS/CN) datasets for neural generation, they fall short in reaching either high-quality and/or high-quantity. In this paper, we propose a novel human-in-the-loop data collection methodology in which a generative language model is refined iteratively by using its own data from the previous loops to generate new training samples that experts review and/or post-edit. Our experiments comprised several loops including diverse dynamic variations. Results show that the methodology is scalable and facilitates diverse, novel, and cost-effective data collection. To our knowledge, the resulting dataset is the only expert-based multi-target HS/CN dataset available to the community.

## introduction the proliferation of online hatred has became an alarming issue @xcite threatening not only the well-being of target individuals and groups, but also of society as a whole. while authorities establish regulations and policies, social media platforms take actions against hate speech mostly through moderation activities, such as content removal, account suspension, or shadowbanning, at the risk of hindering the freedom of expression. meanwhile, non-governmental organizations are qualifying volunteers for responding to online hate to promote human dignity and understanding in society. such responses, i.e., counter-narratives (cn), are non-aggressive textual feedback using credible evidence, factual arguments, alternative viewpoints, and are considered as an effective strategy @xcite to confront hate speech while respecting the human rights @xcite . however, the vast amount of online hate speech makes an effective manual intervention impossible, which motivates a line of nlp research focusing on semi or fully automatized cn generation solutions foot_0 . in recent years, several cn collection strategies and datasets have been proposed addressing the data-hungry nature of current state of the art generation technologies @xcite @xcite . considering the shortcomings of the existing collection strategies (that grant either quality or quantity, but not both), we present an approach to produce high quality cns for multiple hate targets while reducing the need for expert intervention. to this end, we build on top of the previous hybrid data collection strategies, aiming to increase efficiency while maintaining the requirements of data quality, novelty and diversity. in particular, we start from the work by @xcite that uses an author-reviewer framework in which the authora generative language model -is tasked with generating hs/cn pairs while a pool of human reviewers filter and possibly post-edit the produced output. in the present work we propose to further reduce the data collection effort by closing the pipeline and feeding the post-edited output back to the language model in order to regularly update it and improve the quality of the generated pairs. our experiments comprised of two sessions, spanning a period of 6 months. in the first session we set up a 'simple' human-in-the-loop (hitl henceforth) procedure and iterated it several times, measuring at each loop the performance of the whole framework according to relevant metrics. in the second session we run several additional loops in which we test different strategies (i.e. author configurations) to improve the data collection according to the given metrics. findings show that the hitl framework is scalable, allowing to obtain datasets that are adequate in terms of diversity, novelty, and quantity. moreover, this framework improves on previous hybrid data collection strategies, reducing at each loop the post-editing effort of the human reviewers or the number of discarded examples (session one). on the other hand, with dynamic adaptation, possible unwanted behaviors or flaws of the data collection can be handled at each loop by simply varying the author configuration (session 2). the final dataset contains 5000 hs/cn pairs in english language, covering multiple hate targets, in terms of race, religion, country of origin, sexual orientation, disability, or gender. to the best of our knowledge, this is the first multi-target expert-based hs/cn dataset constructed through a semi-automatic mechanism and can be downloaded at the following link: https://github.com/marcoguerini/conan . ## related work with regard to hatred countering, we will focus on three research aspects relevant for the present work, i.e. (i) publicly available datasets for detection, (ii) publicly available datasets for countering, (iii) approaches for hybrid data collection. hate detection datasets. several datasets for hate detection have been presented, most of which rely on material collected from smps, such as twitter @xcite @xcite , facebook @xcite , what-sapp @xcite , and forums (de @xcite . while the above datasets focus on a classification task, @xcite released a dataset annotated with rationales to improve hate speech interpretability and @xcite proposed the social bias inference corpus (sbic) annotated with the description of the biases implicitly present in the language. for a more extensive review, we refer the reader to @xcite and @xcite . hate countering datasets. while several social studies proved that counter-narratives are effective in hate countering @xcite @xcite @xcite , only few works have focused on data collection for cn generation. @xcite focus on crawling, following the intuition that cns can be found on smps as responses to hateful expressions. @xcite propose a crowdsourcing methodology where crowd-workers (non-expert) are instructed to write responses to hate content collected from smps. the study by @xcite also relies on outsourcing cns writing, but via nichesourcing, using ngo operators expert in cn production. hybrid models for data collection. given the data-hungry nature of current nlp technologies, one line of research has recently focused on advanced hybrid models for data collection. @xcite proposed using model interpretation to guide humans in the creation of adversarial examples for factoid question-answering systems. @xcite and @xcite perform a data collection with hitl for detecting offensive language. in both studies, the dynamic procedure is shown to be successful in reducing model error rate across rounds. @xcite point out that the hitl approach has multiple advantages over the static data collection: design flaws can be addressed during the construction of the dataset and annotators' work is optimized, since it is guided by the feedback from the model. finally @xcite propose a hybrid approach where an lm is trained on a seed datasets of hs/cn pairs to generate new pairs that are then validated and post-edited by annotators. ## methodology in figure 1 we present the pipeline of our methodology. following the idea presented by tekiroglu et al. (2020), we have an author module built using gpt-2 language model @xcite and fine-tuned on a seed dataset of hs/cn pairs. the author produces novel hs/cn candidates while the reviewer(s) filter and eventually post-edit them. we iterate this data collection several times, at each loop reviewed examples are added to training data and the author is fine-tuned from scratch again on all available data. in the following sections we describe the main elements used in our procedures. ## metrics to understand the 'diachronic' behavior of our hitl methodology across iterations, the following metrics have been computed at the end of each loop over the newly obtained pairs. imbalance degree measures the difference between a perfectly-balanced distribution of the hate target categories and the actual unbalanced datasets; we use imbalance degree (id) since it is specifically devoted to the multi-class scenario @xcite . datasets that are balanced over multiple hate targets could allow building more representative cn generation models. acceptance rate is the percentage of pairs accepted by the reviewers (either untouched or postedited) over the total number they scrutinised. it represents an overall estimate of the ability of the framework to produce reasonable-quality material. hter is originally a measure of post-editing effort at sentence level translations @xcite . we adopted it to the measure reviewers' effort in terms of the average number of edits over the accepted pairs. an upper-bound threshold value of 0.4 is used to account for easily post-editable pairs @xcite . novelty measures how different two collections of texts are from each other, and it is grounded on jaccard similarity. we utilized it to compute the originality present in v i with respect to the training data collected in previous loops @xcite . repetition rate measures the intra-corpora quality in terms of language diversity by considering the rate of non-singleton ngram types it contains @xcite . we use it to measure the ability of the framework to provide diverse and varied examples. repetition rate (rr) has the advantage of being independent from corpus size, so it can be used to directly compare different versions of our dataset. vocabulary expansion is a measure we introduce to serve two main objectives: (i) quantifying the contribution of the author and the reviewers, by focusing on new tokens appeared at each loop (e.g. the term "peace" was introduced for the first time by annotators in v 2 ), (ii) quantifying the presence of cross-fertilization, i.e. tokens that appear for the first time in version v n for a particular target, but they were present in a version antecedent to v n for the other targets (e.g. the term "peace" for the target jews appears at v 4 but it was already present for the target muslim in v 2 ). the algorithm for computing vocabulary expansion is described in appendix a.1. ## session one in session one, all the versions of the dataset v 2 ...v 5 are generated using gpt-2 v i , where the fine-tuning is performed on all previous versions of the dataset v 1 ...v i-1 as explained earlier. to produce hs/cn pairs, the author conditioning is performed using @xmath0. for the analysis, we computed the metrics described in section 4 on the hs/cn pairs obtained in each loop using micro-averaging (in appendix a.4, discussion. considering our objective of collecting quality material in an efficient way, we first focus on the ratio of accepted pairs and the postediting effort in each loop. as shown in figure 2 , the percentage of accepted pairs tends to increase across the loops, for both the pairs that are postedited ("modified") from 35.8 in v 2 to 50.1 in v 5 and the ones accepted without post-editing ("untouched") from 1.5 in v 2 to 10.9 in v 5 . at the same time, the average post-editing effort of the reviewers tend to decrease across the versions, as depicted in figure 3 . to ensure that the decrease in hter is not due to the increasing ratio of untouched pairs to the total number of accepted pairs, we computed the hter for the modified pairs alone. consistently with the overall trend, hter for modified pairs also declines, indicating that the data collection loops succeeded not only in reducing the reviewer effort, but also in improving the quality of the generated material to be postedited. notably, after v 3 the hter falls below the 0.4 acceptability threshold as defined in @xcite for the amt scenario (figure 3 ). in view of this analysis, we can conclude that the efficiency of data collection is increased by hitl as compared to a static approach that does not retrain the author module (that can be represented by v 2 ). regarding the evaluations with the quality metric repetition rate (figure 3 ), it increases from v 2 on signifying a decrease in the lexical diversity of the generated data. moreover, we observed a consistent trend for the scores of the second quality metric, i.e. novelty (figure 4 ). similar to the diversity, novelty of the collected data also decreases across the versions, regardless of the dataset against which the novelty is computed. particularly, the change in the cumulative novelty represents how the vocabulary becomes less and less enrichable as the loop number increases, indicating a possible saturation point where novel material is highly difficult to obtain. finally, the distribution of hate targets shows a worsening also in terms of id that increases from a score of 2.2 in v 1 to 4.5 in v 5 (see figure 2 ) with some targets becoming predominant while others slowly disappearing. more details on each target distribution per loop are given in appendix a.2, figure 11 . as for pair length, throughout the loops we found that "untouched" pairs are usually shorter (30.7 tokens on average) than the other accepted pairs (37.3 tokens on average before post-editing). during the discussion sessions, annotators reported that the "untouched" pairs are not only shorter but also somewhat stereotypical, with a small novelty added to the overall dataset (e.g. "you cannot say this about an entire religion", "it's unfair to say this about an entire religion"). ## session two given the problems emerged during the loops of the first session (i.e. higher efficiency but lower quality at each loop), we organized an additional session to test several parallel methodologies to ameliorate them. the description of the v 6 configurations are as follows: v 6,sbf : the model gpt-2 v 5 is conditioned with novel offensive speeches extracted from sbic corpus @xcite . we chose this resource since: (i) it contains several thousand of social media posts containing biases and stereotypes spanning the same target categories with our study, (ii) for each post it provides an 'implied statement' that closely resembles a 'prototypical hate speech' on which we trained our system. we sampled the same number of 'implied statements' for each target that maps to our labels 4 among the ones annotated with 'the intent behind the statement was to offend' and/or 'the post could be offensive to someone'. we provide the statements as conditions by appending them @xmath1. v 6,lab : the model is conditioned specifying on which hate target it should focus on. in this configuration, we trained a variant of gpt-2 v 5 that takes into account the target label, and modified the original representation of our training data accordingly. in particular we accommodate hate target information within the starting token: <|startof hs: target label|>. 4 in v 6,arg : we fine-tuned gpt-2 on a dataset of argumentative pairs collected from kialo foot_4 , an online debate platform for constructive and rational discussions among peers that has been exploited recently by the nlp community @xcite . each discussion in kialo is represented as a tree of arguments in which a child node is connected to its parent via a "pro" or "con" relation. extracting all the claims connected by a "con" relation, we obtained a dataset of 128178 argument pairs covering a broader domain as compared to hs/cn pairs. we then fine-tuned gpt-2 for 1 epoch over the argumentation dataset with the standard hyperparameters. preliminary experiments showed that the best strategy was to represent these pairs with the same format as ours to facilitate transfer of task characteristics and argumentative knowledge. then this model was again fine-tuned using the standard v 1 ...v 5 data. at inference time, conditioning has been performed using lists of unique hss from the v 1 ...v 5 data. v 6,m ix : the last model is obtained by blending the three previous versions together, i.e. first finetuning on kialo dataset, second fine-tuning using target label notation on v 1 ...v 5 data, conditioning using sbic offensive speeches. bearing in mind the problems emerged during session one, our first goal in session two was to balance the dataset with respect to the hate targets (i.e. reducing id score). to this end the conditioning always takes into account the hate target label (with respect to 7 targets: jews, lgbt+, muslim, women, disabled,people of color, migrants) either explicitly as in v 6,lab or v 6,m ix , or implicitly as in v 6,sbf and v 6,arg . in addition, to better balance the number of pairs for each target, we administered only the first 5 pairs of each generated chunk to the reviewers. discussion. all the applied methodologies allow for a better balancing of data in terms of hate targets, yielding an average id score of 2.3 for the v 6 configurations in comparison to the id score of 4.5 for v 5 foot_5 . as shown in figure 5 -left, all v 6 configurations have a slightly higher acceptance rate than v 5 7 . thus introducing novel material or data representation in fine-tuning stages has no strong perturbation effect. second, and more interestingly, we observe a significant variation in the ratio of untouched and modified pairs to all the reviewed samples: for all v 6 approaches while there is a strong decrease in ratio of untouched pairs (figure 5 , right), there is a significant increase in those modified (see figure 5 , left). in other words these models were able to produce a higher amount of suitable, albeit non perfect, pairs. in particular, comparing v 6 configurations we can observe that for the untouched pairs the highest acceptance rate is achieved via v 6,arg with 6.37% accepted pairs, whereas for the modified pairs v 6,m ix yields the highest percentage, with 66.15% of the pairs accepted. concerning the reviewer's effort, we see that the overall hter increases for the all v 6 approaches (figure 6 , left). considering that we had a lower number of untouched and a higher number of modified pairs this was expected, and if we turn to the hter of modified pairs alone we see that there is a smaller difference between v 5 and v 6 hter. even more interestingly, the hter scores of all v 6 configurations, even if higher than v 5 , are still below the acceptability threshold value of 0.4 defined earlier. going into details, amongst the v 6 configurations, hter reaches its lowest value in v 6,arg , for both the modified and untouched pairs: since it was conditioned using gold hs material, this result is expected. as opposed to the other models, v 6,lab is conditioned only with a label representation and not with actual hss. this affected negatively the post-editing effort, as we can notice a higher hter for this configuration. moreover, v 6,lab has a smaller amount of untouched pairs, so we expected hter to spike up. with regard to data quality (see figure 7), we see that all v 6 strategies succeed in increasing the novthe plots, using a linear regression model over v1...v5. elty both with respect to v 5 and expected v 6 (the dashed line) , except for v 6,arg , possibly due to its conditioning with hss from v 1 ... v 5 . therefore, we also computed the novelty for cn set alone to discard the effect of hs on the metric. in this setting, all v 6 configurations reach a novelty between 0.741 and 0.745, as compared to a cn novelty in v 5 of 0.737 (as in appendix a.3). the effect of gold hs conditioning in v 6,arg can also be spotted in the lowest hter results in figure 6 . the highest increase in novelty is recorded for v 6,m ix , reaching a score of 0.76; also novelty scores computed with respect to v 5 and v 1 confirm the result. all v 6 configurations succeeded in reaching an rr lower than both v 5 and expected v 6 (the dashed line). it is interesting that v 6,lab has the highest rr among the v 6 configurations, possibly because it was not built using any external knowledge, but only with a different label representation. on the other hand, v 6,arg configuration, for which an initial argumentation fine-tuning has been performed, has the lowest rr (5.474). from this analysis we can conclude that v 6 configurations are better at producing sub-optimal material but worse at producing perfect material. still the general quality of the pairs (in terms of novelty and rr) in session two is much higher than before, exhibiting the desired behavior for which these strategies were introduced. ## vocabulary analysis. we report vocabulary expansion findings in figure 8 . for each loop v 2 ...v 5 the average percentage of new words injected into the dataset by the author model (gpt-2) is higher than the average percentage of new words inserted by the three reviewers during post-editing. both trend-lines, even if slightly decreasing are not converging, implying that fine-tuned gpt-2 is not reaching a "saturation point" and is continuously adding new material. this trend is in line with the decrease in novelty. on the other hand, instructions asked for a minimal post-edit, so the reviewers have less opportunity to inject new material than the author and the decrease is consistent with the decreasing hter. as for the percentage of words generated by the author model pertaining to the same target, we see an increasing trend throughout the generations due to the cumulative nature of the metric. still, the presence of words first observed in other targets throughout the loops, shows that the crossfertilization phenomenon generated by gpt-2 persists. this desired feature shows the ability of gpt-2 to learn the 'domain' vocabulary of other targets and to use it effectively for the target under analysis, in a way that is approved by the humans -reviewers 8 . finally, we can remark that v 6 versions are able to increase both the injection of novel words from gpt-2 and also its cross-fertilization ability. ## qualitative analysis during our exploratory experiments and the discussion sessions with the annotators, several 8 even though we opted for a distinction in terms of source (same target, other target) for the 'not-new' words generated by the author module, we chose not to have the same distinction for the reviewers' 'not-new' words, since we cannot assess if the reviewer was (or not) aware of the presence of a word in previous versions of the dataset. hs: "we should kill all the jews." cn: "there are many alternatives to removing jews, such as converting them to another religion (e.g. buddhism)." hs: "all homosexuals are trying to undermine the family by trying to impose their sexual preferences." cn: "thus far, people of various sexual orientations have attempted suicide twice as often as heterosexuals. among youth, those that seriously contemplate suicide are at almost three times the rate of heterosexual. lgb youth are almost five times as likely to have attempted suicide compared to heterosexual youth. tell me honestly: who do you think is imposing sexual orientation to whom?" hs: "i don't like muggleborns, they are likely to do violence." cn: "we do not say that muggleborns are less likely to commit crimes. we are saying that they are almost certainly not the case. " argumentation and counter narratives. in order to obtain even more novelty in produced pairs, v 6,arg model could be used without fine-tuning on the hs/cn dataset under the assumption that a counter argument is the same as a counter narrative. still, the ability to argument on a variety of topics is not enough to provide a meaningful cn when prompted with an hs. a cn also presuppose values, so -for example -a logically valid argument is not necessarily an acceptable cn, as the first example in new arguments or new paraphrases. one question that emerged is whether gpt-2 is able to produce novel arguments or it is just a very sophisticated paraphrasing tool. during the discussion sessions with annotators and also by manual analysis, we could find cns that contained genuinely novel arguments, which were not present in the training data but produced by gpt-2. in the second example in ## conclusions in this paper we presented a novel hitl methodology for data collection based on an author-reviewer framework. this methodology puts together an lm and a set of human reviewers, where the lm is refined iteratively, using data from previous loops that have been validated by experts. experiments show that as loops are iterated, efficiency in data collection increases (acceptance rate and hter metrics) while the dataset quality decreases in terms of novelty and diversity metrics. for this reason we experimented with additional dynamic loop adaptation that are able to increase the overall quality of the dataset without hindering the efficiency significantly. target coverage pairs disabled 4.40 220 jews 11.87 594 lgbt+ 12.33 617 migrants 19.13 957 muslims 26.68 1335 poc 7.04 352 women 13.23 662 other 5.32 266 total 100 5003