UIT -E10dot3 at S em E val-2021 Task 5: Toxic Spans Detection with Named Entity Recognition and Question-Answering Approaches (2021)

Abstract: The increment of toxic comments on online space is causing tremendous effects on other vulnerable users. For this reason, considerable efforts are made to deal with this, and SemEval-2021 Task 5: Toxic Spans Detection is one of those. This task asks competitors to extract spans that have toxicity from the given texts, and we have done several analyses to understand its structure before doing experiments. We solve this task by two approaches, Named Entity Recognition with spaCy’s library and Question-Answering with RoBERTa combining with ToxicBERT, and the former gains the highest F1-score of 66.99%.

## introduction the world of social media is overgrowing, and users easily express their opinions or feelings toward topics that they are concerned about. however, because of the freedom of speech, lots of toxic comments or contents are uncontrollably increasing. there are several kinds of research about the effect of toxic speech on users' health. in 2017, research about the impact of toxic language on health was conducted @xcite . sometimes, with toxic words, conversations can become cyberbullying, cyber threats, or online harassment, which are harmful to users. to reduce those negative impacts, there are abundant researches for classifying contents into toxic or non-toxic, and then they hide the whole text if it is toxic. however, that action may inhibit the freedom of speech. as a result, censoring only toxic spans is the better solution for this problem. therefore, in semeval-2021 task 5: toxic spans detection @xcite we try to realize it. about toxic contents on the internet, researches were only about binary toxicity classification. still, in task 5 of semeval-2021, which is about toxic spans detection, we conduct more in-depth research into the toxicity, find exactly which parts of the text are toxic. as the ner approach and question-answering (qa) approach, we propose two approaches for solving this problem. we use roberta @xcite combining with tox-icbert (hanu and unitary team, 2020), transfer learning models, for qa approach and spacy's library @xcite for ner approach. we organize the paper as follows. section 2 is related works that we consult for building the systems. the dataset and analyses are defined in section 3. in section 4, we introduce our two proposed systems for toxic spans detection. section 5 describes the results of the studies and analyses. finally, in section 6, we bring our work to a close. ## related works researchers around the world these days have started to concentrate on toxic speech. it inflicts individual and group harm, damaging our social fabric @xcite . several datasets for classifying toxicity on toxic speech on online forums, such as the dataset provided by @xcite for english, beep! dataset for korean by @xcite , the dataset for russian provided by @xcite , told-br dataset for brazilian portuguese by @xcite , and uit-victsd, a dataset about constructive and toxic speech detection for vietnamese @xcite . besides, there are shared tasks about toxic speech as well as hate speech such as these from semeval, includes semeval-2019 task 5 multilingual detection of hate @xcite , semeval-2019 task 6 identifying and catego-rizing offensive language in social media (of-fenseval) @xcite , semeval-2020 task 12 multilingual offensive language identification in social media @xcite , and semeeval-2021 task 5 toxic spans detection @xcite , which is the current task we have to deal with in this paper. ## dataset the origin of this semeval-2021 task 5 dataset comes from the publicly available civil comments dataset @xcite , which consists of 1.2m posts and comments. the data in this public dataset have no annotation of any toxic spans in toxic posts but do have post-level toxicity annotations, which mean showing which posts or entire of them are toxic. and the holders of this task retain 30k of them, which were annotated to be toxic or severely toxic by at least half of the crowd-raters from annotations of borkan et al. the task holders then randomly keep 10k posts from the 30k posts for annotating toxic spans. they employ three experienced crowd-raters per post from a third-party crowd-annotation platform, and they warn them about adult content. however, task organizers also claim that not all toxic posts are annotated with toxic spans. the task for crowd-raters is to highlight toxic sequences of the comments, and if the comment is not toxic or should annotate the whole of it, crowdraters have to check the appropriate box without highlighting any spans. consequently, we have two columns, the spans column and the text column. the spans column has lists of numbers or null that reference toxic character offsets in the text column, and some of the given data are shown in the following table. the competitors receive two separate training and test sets from organizers. in the training set, there are 7,939 records, and in the test set, there are 2,000 records. furthermore, as mentioned in the data annotating process, one text that possibly has multiple toxic spans is highlighted. figure 1 and figure 2 illustrate the distribution of spans in the training and the test sets. for more details, according to figure 1 and figure 2 , there is a significant number of single spans in each post, and it accounts for nearly 68.8% and 70.8% in the training set and the test set, respectively. it is also interesting to notice that the number of zero spans is not tiny, and the proportion of it in the training set is less than in the test set, more specifically, 19.7% in the test set and 6.15% in the training set. moreover, we also calculated the jaccard score of text and spans in the given dataset for more indepth analysis. the jaccard score, also known as the jaccard index or jaccard similarity coefficient, was developed by paul jaccard @xcite and it is a statistic used for measuring the similarity and diversity of sample sets as follows. the histogram in figure 3 illustrates that most of the data points have jaccard scores in the range of 0 to 0.35, and the peak is at 0 to 0.05, which means toxic character offsets are just a fraction in each post even there are records annotated all characters of the post are toxic. there are 16 records in the test set and 212 records in the training set with jaccard scores at 0.95 to 1.0. for that reason, just the toxic part(s) of the comments needs to be censored rather than the whole comment as in the traditional method. ## systems in this paper, we propose two systems for the toxic spans detection task with ner and qa approaches. the first system is the qa approach based on roberta and the second system is the ner approach based on spacy's library. ## experiments after building two such systems, we start to experiment on the test set, and the following subsections dicuss our results. ## conclusion and future work in this paper, we introduced two proposed systems for toxic spans detection based on named entity and question-answering approaches. we obtained the highest results with the spacy's library based system with the f1-score of 66.99% and ranked 34 out of 91 teams in semeval-2021 task 5. in future, we plan to improve our systems by implementing various sota models for toxic spans detection. with the built systems, we can create friendly online conversations and make social media forums safer for users.