NLRG at S em E val-2021 Task 5: Toxic Spans Detection Leveraging BERT -based Token Classification and Span Prediction Techniques (2021)

Abstract: Toxicity detection of text has been a popular NLP task in the recent years. In SemEval-2021 Task-5 Toxic Spans Detection, the focus is on detecting toxic spans within English passages. Most state-of-the-art span detection approaches employ various techniques, each of which can be broadly classified into Token Classification or Span Prediction approaches. In our paper, we explore simple versions of both of these approaches and their performance on the task. Specifically, we use BERT-based models - BERT, RoBERTa, and SpanBERT for both approaches. We also combine these approaches and modify them to bring improvements for Toxic Spans prediction. To this end, we investigate results on four hybrid approaches - Multi-Span, Span+Token, LSTM-CRF, and a combination of predicted offsets using union/intersection. Additionally, we perform a thorough ablative analysis and analyze our observed results. Our best submission - a combination of SpanBERT Span Predictor and RoBERTa Token Classifier predictions - achieves an F1 score of 0.6753 on the test set. Our best post-eval F1 score is 0.6895 on intersection of predicted offsets from top-3 RoBERTa Token Classification checkpoints. These approaches improve the performance by 3% on average than those of the shared baseline models - RNNSL and SpaCy NER.

## introduction offensive language can include various categories such as threats, vilification, insults, calumniation, discrimination and swearing @xcite . detection of such language is necessary for ease of moderation of content on social media. despite their popularity, toxicity detection tasks have focused majorly on sequence classification, rather than sequence tagging. finding which spans make a comment or document toxic in nature is crucial in explaining the reasons behind their toxicity. additionally, such attributions would allow for more efficient semi-automated quality-based moderation of content, especially for verbose documents, in comparison to quantitative toxicity scores. in @xcite provide a dataset of 10k english texts filtered from civil comments @xcite dataset. each text is crowd-annotated with character offsets that make the text toxic. the task is to predict these character offsets given the text. the work presented in this paper aims to provide a comprehensive analysis of simple token classification (tc) and span prediction (sp) methods across multiple bert-based models -bert @xcite , roberta @xcite and spanbert @xcite . additionally, we experiment with a few hybrid approaches -multi-span (msp), where the model is trained on multiple spans simultaneously; span+token (sp-tc), where the model is trained on both kinds of tasks simultaneously; lstm-crf (lc), which uses a lstm and crf layer on top of bert-based models; and a combination of predicted offsets for above techniques using union/intersection. in section 2, we perform a compendious literature survey. section 3 elucidates our approach, including the modelling aspect, the various variants of the base model, and the different hybrid systems. in section 4, we describe our experimental setup and hyperparameters used for our methods. lastly, in section 5 we analyze our results and perform ablative analysis on our systems. ## background before the advent in research pertaining to toxic texts, @xcite modeled hate speech as a word sense disambiguation problem where svm was used for classification of data. @xcite used rnn language model with character and token based methods to classify the text. recently, however, toxic text detection has garnered a lot of attention @xcite @xcite . the increase in offensive language research can partly be credited to various workshops such as abusive language online 1 @xcite , as well as other fora, such as germeval for german texts, 2 or trac @xcite and kaggle challenges 3 . hanu and unitary team (2020) introduced detoxify, a comment detection library modeled using huggingface's transformers @xcite to identify inappropriate or harmful text online as a result of participation in three such challenges. in a contemporary work, pavlopoulos et al. (2020) discuss context requirement for toxicity detection. in semeval 2020-task 11 (da san martino et al., 2020), the first sub-task -span identification -aims at detecting the beginning and the end offset for the propaganda spans in news articles. this sub-task is similar to semeval 2021-task 5. the proposed approaches for the sub-task can be broadly classified into span prediction or token classification. most teams use multi-granular transformer-based systems for token classification/sequence tagging @xcite @xcite . inspired by souza et al. (2019), jurkiewicz et al. (2020) use roberta-crf based systems. li and xiao (2020) use a variant of spanbert span prediction system. ## results and analysis in favor of brevity, for this section, we use the following abbreviations: @xmath0, @xmath1, @xmath2, @xmath3, @xmath4, @xmath5, @xmath6, @xmath7, @xmath8, @xmath9, @xmath10, (x,∩)=intersection of offsets from x-best checkpoints, (x,∪)=union of offsets from x-best checkpoints. ## conclusion based on our results and analysis, we conclude that token classification systems have an edge over span prediction methods on this task. base models perform better than large models in either of the approaches, which could imply need for more data to train large models. our multi-span approach performs poorly, but span+token approach shows some promise and we need to re-evaluate our architectural choices. the reason why toxi-cbert/toxicroberta perform worse than base models is also an avenue for further analysis. finally, our individual bert-based models tend to predict extra offsets for the task. while checkpoint ensembling using intersection is a good way to address this issue, we will explore other remedies in a future work.