U o T - UWF - P art AI at S em E val-2021 Task 5: Self Attention Based B i- GRU with Multi-Embedding Representation for Toxicity Highlighter (2021)

Abstract: Toxic Spans Detection(TSD) task is defined as highlighting spans that make a text toxic. Many works have been done to classify a given comment or document as toxic or non-toxic. However, none of those proposed models work at the token level. In this paper, we propose a self-attention-based bidirectional gated recurrent unit(BiGRU) with a multi-embedding representation of the tokens. Our proposed model enriches the representation by a combination of GPT-2, GloVe, and RoBERTa embeddings, which led to promising results. Experimental results show that our proposed approach is very effective in detecting span tokens.

## introduction with the massive increase in social interactions on online social networks, keeping discussions fruitful is a central concern for platform providers. indeed, abusive (e.g., bullying, profanity, hate speech), damaging the reputation of a platform. thus, it is necessary to be detected by automated machine learning systems because of huge amount of data. however, the previous works mostly focus on whether the given document(e.g. comment) is toxic or not. detecting the span tokens in the document may be more beneficial. for example, it can prevent users to use span tokens before they send their post. also, it is useful to filter span tokens before learning ai chatterbots instead of removing the whole documents. in this paper, we propose a self-attention-based bidirectional gated recurrent unit (bigru) with a multi-embedding representation of the tokens. our proposed model enriches representation showed very promising results. the rest of the paper is organized as follows. section 2 provides background and presents some related works on tsd in general. section 3 intro-duces our bigrus model. results are covered in section 4. in section 5 we draw a conclusion. ## related research a toxic post (comments) is defined as a rude, disrespectful, or unreasonable comment that is likely to make other users leave a discussion. a goal of toxic comment classification is to give a right to freedom of expression on the web. training complex neural networks (nn) requires enough datasets of toxic comments. word embeddings are the basis of the nns when working with text data. nns for toxic comment classification use recurrent neural networks (rnn) layers such as long short-term memory (lstm) @xcite or gated recurrent unit (gru) @xcite layers. similarly, the attention mechanism for nns has been successfully applied to toxic comment classification @xcite . in semi-automated content-moderation, attention can be considered as a highlighter of abusive or toxic words. @xcite proposed a system for a hate-speech task based on deep-learning with a combination of lstm, random embedding, and gradient boosted decision trees as the best model. they used random embedding, glove @xcite , and fasttext @xcite representation for experimentation. they concluded that a combination of cnn and lstm with fasttext or glove embedding as features for gradient boosted decision trees can not yield better results. in @xcite they proposed a deep classification-specific attention mechanism with bigru to highlight suspicious words for automatic and semi-automatic content moderations. ## proposed method in this section, we describe the details of our proposed neural network model. our proposed approach aims to predict whatever the tokens of given comments are toxic or not. figure 1 depicts an overview of our proposed method. at first, in a dataset, original posts are tokenized and preprocessed. each token at each post is labeled as a toxic or not toxic token by their spans. then, a multi-embedding representation of tokens is created. next, the bidirectional grus (bigrus) models are applied to extract the higherlevel feature sequences with sequential information from multi-embeddings. after that, a self-attention mechanism computes attention weight between each pair of elements in a single sequence. finally, the generated output feature sequences from self-attention-based bigrus are fed into the fullyconnected dense layer and then into the final prediction module to determine the prediction. in the following, we describe each component elaborately. ## conclusion in this paper, we presented our approach for semeval-2021 task 5: toxic span detection. we tried to tackle the problem by employing multiembedding and deep learning techniques. we conducted some experiments using different models. for example, we implemented a bilstm model, bilstm with sa, bigru, and bigru with sa, but the model that gave a promising result and relied on bigru with sa model.