Sefamerve ARGE at S em E val-2021 Task 5: Toxic Spans Detection Using Segmentation Based 1- D Convolutional Neural Network Model (2021)

Abstract: This paper describes our contribution to SemEval-2021 Task 5: Toxic Spans Detection. Our approach considers toxic spans detection as a segmentation problem. The system, Waw-unet, consists of a 1-D convolutional neural network adopted from U-Net architecture commonly applied for semantic segmentation. We customize existing architecture by adding a special network block considering for text segmentation, as an essential component of the model. We compared the model with two transformers-based systems RoBERTa and XLM-RoBERTa to see its performance against pre-trained language models. We obtained 0.6251 f1 score with Waw-unet while 0.6390 and 0.6601 with the compared models respectively.

## introduction unlike the text classification problems targeting to classify whole documents @xcite @xcite , toxic span detection is an nlp task focusing on capturing granular contents that make a text toxic. proposed solutions may contribute to managing semi-automated moderations such as online discussions or news portals that are open to large participation and user comments. therefore, the evaluation of systems that could accurately locate toxic spans within a text is considered a crucial step for this task @xcite . we adapt two solution approaches for the task. for the first approach, we consider toxic spans detection as a segmentation problem while in the second one we use transformers-based models. our proposed model for the first approach uses character-based tokenized chunks as an input and outputs segmented text. the system uses a 1dimensional (1-d) convolutional neural network adopted from u-net architecture @xcite commonly applied for semantic segmentation. we previously studied this approach, as we call waw-unet, on text parsing problems for unstructured postal addresses, and achieved remarkable results @xcite . in our second approach, we consider toxic spans as a single label named-entity recognition problem. we employ several different transformersbased models and obtained better scores with roberta @xcite and xlm-roberta (conneau et al., 2020) network architectures. to see the difference between our main system and transformers-based models, we compared model achievements, and obtained 0.6251 f1 score with waw-unet while 0.6390 and 0.6601 with the other models respectively. following the final rankings, our best score ranked 44th among 91 submissions foot_0 . ## waw-unet architecture waw-unet is a fully convolutional neural network architecture we designed by taking inspiration from u-net architecture which was firstly developed for segmentation problems @xcite . the u-net network architecture is composed of two symmetric parts, which uses dimension reduction for the first half of the network, and then increases its dimension in the second half. in this architecture, the connections are taken from the convolutional layers on the encoding part, which also feeds each corresponding layer of the decoding part. similar to the pixel-based image segmentation, waw-unet takes input samples, in our case text, and generates homogeneous masked regions for the targeted segment. however, unlike image processing which has multi-channel input, the network has 1-d input due to the single-dimensional nature of text data (fig. 1 ). the outputs of the convolution layers are combined and passed through a 1-d convolution layer, and then we use batch normalization to accelerate training and prevent the objective function from getting stuck in local minima. after this stage, before the outputs send to the next block, if the output is in the encoder part their dimensions are reduced in half using max pooling, while doubled if it's in the decoder part by upsampling. although its architecture adapted from u-net, we customize existing architecture by adding a special network block for text segmentation, as an essential component of the model. waw-unet uses a special network block, which we call it wawblock, to extract the attributes in the targeted text patterns. in our architecture, each waw-block contains three convolutional layers with different kernel sizes. waw-unet learns input features through filter sizes of the 3, 5, and 7 as shown in fig. 2 . ## results we evaluate our system as well as transformers models' performance on the semeval-2021 task 5: toxic span detection trail dataset and also report the evaluation result on the blind test dataset. we use standard train/test split of the official release dataset of the task for experiments. for our main system model, waw-unet, we start model training with 300 epochs and utilize early stopping and learning pause using keras's learning callbacks @xcite . on the other hand, we trained transformers models with 8 batch size with 17 epochs. we gained the best score in 7th epoch on both pre-trained language models. xlm-roberta model gained best score 0.6601 while waw-unet and roberta reached 0.6251 and 0.6390 respectively as shown in ## conclusion we framed the problem as a semantic segmentation task, and developed a unique approach to extract targeted spans from provided text data. proposed system performs relatively well than expected against pre-trained transformers. our models do not use any of the external dataset or automatic linguistic annotations, such as pos or named entity tags. overall, we showed that segmentation based systems can be used to address the toxic detection task. our best submitted result was ranked 44th among 91 submissions, obtaining an average f1 score of 0.6601, 4.82 points behind the first ranked system. for future studies, we're planning to work on unsupervised training of the waw-unet architecture on large datasets to compete with the pre-trained general language models.