SINAI at S em E val-2021 Task 5: Combining Embeddings in a B i LSTM - CRF model for Toxic Spans Detection (2021)

Abstract: This paper describes the participation of SINAI team at Task 5: Toxic Spans Detection which consists of identifying spans that make a text toxic. Although several resources and systems have been developed so far in the context of offensive language, both annotation and tasks have mainly focused on classifying whether a text is offensive or not. However, detecting toxic spans is crucial to identify why a text is toxic and can assist human moderators to locate this type of content on social media. In order to accomplish the task, we follow a deep learning-based approach using a Bidirectional variant of a Long Short Term Memory network along with a stacked Conditional Random Field decoding layer (BiLSTM-CRF). Specifically, we test the performance of the combination of different pre-trained word embeddings for recognizing toxic entities in text. The results show that the combination of word embeddings helps in detecting offensive content. Our team ranks 29th out of 91 participants.

## introduction the advance of online communication has increased the use of offensive or toxic language in several websites, including social networks such as instagram, twitter, or youtube. consequently, this type of prejudiced communication could lead to negative psychological effects among internet users, causing anxiety, harassment, and even suicide in extreme cases @xcite . moderation is essential to promote healthy online communication. therefore, governments, online communities, and social media platforms are continuously taking appropriate actions to implement laws and policies combating toxic language on the web. in order to help to track this type of comments and due to the amount of data generated every day on the web, automatic systems based on natural language processing (nlp) techniques are required. in particular, offensive language detection and analysis has become an important area of research in nlp, resulting in several studies that are contributing to combating this website phenomenon @xcite @xcite . in this paper foot_0 , we present our proposal system as part of our participation in semeval-2021 task 5: toxic spans detection @xcite , which aims to identify entities that refer to a toxic language in the text. to accomplish the task, our team focused on detecting specific types of toxic entities in the text using a methodology based on the bilstm-crf model showing that the combination of different pre-trained language embeddings succeeds in detecting toxic entities. the rest of the paper is structured as follows. in section 2 some previous related studies are introduced. in section 3 we explain the data used in our methods and we describe the architecture of our proposed system to the toxic spans detection task. in section 4 we discuss the analysis and evaluation results for the experiments we performed. finally, we conclude in section 5 with remarks and future work. ## related work heretofore, several shared tasks have been organized in the nlp field to detect offensiveness on the web for different languages. for instance, the well-known offensive language task offenseval has held two editions in the international workshop on semantic evaluation (semeval) @xcite introducing as the main novelty in the second edition a multilingual dataset comprising 5 languages. the germeval shared task focused on the identification of offensive language in german tweets and comprised two tasks, a coarse-grained binary classification task and a fine-grained multiclass classification task @xcite . for spanish, as far as we know, the first task on offensive language appeared at the 3rd sepln workshop on evaluation of human language technologies for iberian languages (ibereval) @xcite whose goal was to detect aggressiveness mexican spanish tweets. as a result, most of the studies and resources in offensive language research have been developed specifically for binary and multi-class classification tasks @xcite @xcite . however, other tasks such as named entity recognition (ner) play an important role in this research and are essential to identify the entities that make a text toxic. highlighting these toxic spans can help human moderators to interpret and identify easily this type of content on the web instead of relying on a system that generates a score of unexplained toxicity per post. ner aims to identify and classify named entities mentioned in unstructured text into predefined categories. the earliest systems developed for addressing this task did not use training data but worked based on handcrafted features, heuristics, and a set of rules @xcite @xcite . however, the cost of manual feature tagging and the poor obtained results lead to deep learning-based techniques as the most suitable choice to tackle the task by discovering patterns and learning the features in an end-to-end manner @xcite . existing state-of-the-art approaches for sequence labeling have proven that recurrent neural networks (rnns) are capable of learning useful representations automatically as they enable the modeling of long-distance dependencies between words in a sentence @xcite . inspired by these studies, we have developed a system based on bilstm-crf model along with the combination of different types of word embeddings to address the toxic spans detection task in text. ## named entity recognition methodology to address the toxic detection task, we focus on recognizing and extracting specific types of toxic enti-ties in the text. specifically, we follow a methodology proposed by @xcite implementing a bilstm-crf model for the ner task. ## results in this section, we present the results obtained by our proposed system. in order to evaluate them, we use the official competition metric f1-score. the results of our participation in the toxic spans detection task during the evaluation phase are shown in ## conclusions and future work this paper presents the participation of the sinai research group in task 5: toxic spans detection at semeval 2021. in this paper, we use a deep learning-based approach for ner to identify spans that make a text toxic, which focuses on the use of a bilstm-crf neural network where different word embeddings are tested. the model is trained on the dataset provided by the organizers of the task @xcite and preprocessing techniques are carried out to tokenize and tagged the dataset by using the bio scheme. our results show that the sophisticated bilstm-crf architecture which has been successfully used for other tasks such as biomedical entity recognition or part-of-speech tagging, but also achieves remarkable results when addressing tasks related to the identification of offensive language in comments. besides, we find that this architecture with our proposed combination of embeddings for word representation provides useful insights for the learning phase of the neural network achieving better results than training the network with a single type of word embedding. for future work, we plan to study the performance of our proposed method using a variety of linguistic features, including emotions that are inextricably linked to offensive language.