B ennett NLP at S em E val-2021 Task 5: Toxic Spans Detection using Stacked Embedding Powered Toxic Entity Recognizer (2021)

Abstract: With the rapid growth in technology, social media activity has seen a boom across all age groups. It is humanly impossible to check all the tweets, comments and status manually whether they follow proper community guidelines. A lot of toxicity is regularly posted on these social media platforms. This research aims to find toxic words in a sentence so that a healthy social community is built across the globe and the users receive censored content with specific warnings and facts. To solve this challenging problem, authors have combined concepts of Linked List for pre-processing and then used the idea of stacked embeddings like BERT Embeddings, Flair Embeddings and Word2Vec on the flairNLP framework to get the desired results. F1 metric was used to evaluate the model. The authors were able to produce a 0.74 F1 score on their test set.

## introduction modernization has awarded us with the technology capable of communicating with masses across huge distances in an instant by the touch of a single finger in our palms, the smartphone. with all this innovation every day, it is now more accessible than ever, even in the most rural parts of the world and at very reasonable costs. it has enabled a vast population to share their thoughts and views on popular topics in public forums. people can express themselves in the most creative ways and talk to each other about movies, research, politics, the economy and much more. some of the key forums and platforms for such activities are facebook, twitter, youtube, reddit. they allow users to participate in various discussions, discussions that at times may not be very decent. this creates an issue for these platforms as they usually have some of the other policies against indecent content posted by users. on average there are about 350,000 tweets, 510,000 comments, 293,000 status updates on facebook and twitter in every 60 seconds @xcite . it is humanly impossible for these platforms to check each and everything posted by the users for hate or toxicity. they require an automated method to flag such content. the motivation for this research task is to achieve some degree of automatic moderation in the social web. it is crucial to moderate social media sites such as facebook, twitter and reddit to be healthy and inclusive. this includes filtering and censoring toxic and hateful content posted online on these public forums. there are automated hate detection nlp models like @xcite capable of identifying toxic content with acceptable performance. however, they do not identify the specific spans of text that are toxic. this task tries to identify these spans that can be used further to provide insights into a generic text toxicity score. more about the study that this paper is based on can be found from the task organizers paper @xcite . this paper proposes linked lists for data preprocessing and a stacked embedding approach to training this automated system. the authors' experiment and code for the models ready to be reproduced can be found using the authors' github repository. this paper is organized as follows. it starts with a short abstract describing the paper at a very high level. then the first section introduces the problem at hand and the task to accomplish. it mentions the authors' approach and the link to their code and models. the second section talks about the background research performed for the task and explains the existing solutions and how this paper is different from them. the third section gives an in-depth understanding of the system approach to solving the tasks where it talks about the embeddings and the stacking method. the fourth section tells us about the dataset, how it was presented, what it consisted of, the data processing applied to it, the experimental setup and the evaluation metrics. the fifth section is about the methodology, followed by the results section. the seventh section concludes the system description. finally, we get the references. ## background research and related works this problem tackles the challenge of accurately identifying the parts of a toxic text and contributing to making the complete text toxic. there, however, exist systems that can score the toxicity of a full text, sentence or comment like @xcite , but they don't identify the exact part of that sentence that is toxic. there also are systems that can accurately say if a statement is toxic or not and even go on to the extent of identifying the emotion projected by it, for example, if it is positive, negative, humorous, sarcastic, offensive or funny and even the level of these emotions as seen in @xcite . a widely known methodology to identify parts of a text is named entity recognition (ner) @xcite and its types like the speech (pos) tagging method. ner is a sequence labelling task that recognises entities as per the types of entities it is trained on. these are usually nouns or particular words like names of people or places or organisations and values like monetary numbers and currencies. the sequence labelling task considers a text sequence in which part of it needs to be tagged differently according to the embeddings. long short term memory, lstms @xcite are usually used for sequence labelling. a variant of lstms, the bi-directional recurrent neural network-based bilstm, has proved to be very successful at performing accurately at such tasks. it is also seen to be combined with the conditional random field(crf) decoding layer as seen in @xcite to get better results. open-source tool, spacy @xcite ) is a leading tool to create effective and lightweight ner models for datasets with such challenges. ## system overview for this task, the authors have used the flairnlp platform @xcite to stack bert and flair embeddings and create a named entity recognition (ner) model. they have morphed the origi-nal aim of the ner type tasks to train a model capable of identifying toxicity in a text that is already classified as toxic as a whole. this was possible because, finally, we were using the different embeddings related to toxic parts of the text. these were the embeddings that were trained, which made the task very similar to a ner task. the model created also had a conditional random field layer. its purpose was to better understand the context around the text's selected parts by considering the neighbouring words. for this task of sequence labelling, we have stacked embedding by concatenating bert and flair embedding. this helps in getting a better semantic context out of the concatenated embedding word vectors. following is a brief about flair and bert embeddings. ## experimental setup this section presents a brief overview of the data, the pre-processing it went through and challenges faced in doing so, information about the experiment environment and the evaluation metrics used. ## methodology in this paper authors have used the concept of stacked embeddings i.e. to generate a particular embedding for a word, flair gives you the flexibility to concatenate different word embeddings together to get better results. equation 7 depicts concatenating flair and glove embedding. here word embedding is denoted by we. in this paper, the result of four (4) models have been displayed. ground @xmath0, the pre-processed clean text with the toxic word "idiot" highlighted in red. below them is the ground truth for the spans of this toxic word and then there are the correct predicted spans for it. ## conclusion we support the systematic development for identifying toxic spans. we have successfully been able to deploy a linked list approach to prepare the data and then train it using the stacked embeddings method and produce empirical results. this task can prove to be useful in providing a better analysis in the censoring of toxic posts on the internet.