S koltech NLP at S em E val-2021 Task 5: Leveraging Sentence-level Pre-training for Toxic Span Detection (2021)

Abstract: This work describes the participation of the Skoltech NLP group team (Sk) in the Toxic Spans Detection task at SemEval-2021. The goal of the task is to identify the most toxic fragments of a given sentence, which is a binary sequence tagging problem. We show that fine-tuning a RoBERTa model for this problem is a strong baseline. This baseline can be further improved by pre-training the RoBERTa model on a large dataset labeled for toxicity at the sentence level. While our solution scored among the top 20% participating models, it is only 2 points below the best result. This suggests the viability of our approach.

## introduction toxicity and offensive content is a major concern for many platforms on the internet. therefore, the task of toxicity detection has attracted much attention in the nlp community @xcite @xcite . until recently, the majority of research on toxicity focused on classifying entire user messages as toxic or safe. however, the surge of work on text detoxification, i.e., editing of text to keep its content and remove toxicity @xcite , suggests that localizing toxicity within a sentence is also useful. if we know which words of a sentence are toxic, it is easier to "fix" this sentence by removing or replacing them with non-toxic synonyms. @xcite make human labelers annotate the spans as rationales for classifying a comment as hateful, offensive, or normal. they show that using such spans when training a toxicity classifier improves its accuracy and explainability and reduces unintended bias towards toxicity targets. this year the semeval hosts the first competition on toxic spans detection, namely, semeval-2021 task 5 foot_0 @xcite . it provides training, development, and test data for english. as far as we know, it is the first attempt to explicitly formulate toxicity detection as sequence labeling instead of classification of sentences. multiple nlp tasks recently benefited from transfer learning -transfer of probability distributions learned on some task to another model solving a different task. the most common example of transfer learning is the use of embeddings and language models pre-trained on unlabeled data (e.g. elmo @xcite , bert @xcite and its variations, t5 @xcite , etc.) on other tasks (e.g. @xcite ; @xcite inter alia use pre-trained bert models to perform tasks from the glue benchmark @xcite ). word-level toxicity classification can be formulated as a sequence labeling task, which also actively uses the pre-trained models mentioned above. bert comprises the versatile information on words and their context, which allows to successfully use it for sequence labeling tasks of different levels: part-of-speech tagging and syntactic parsing @xcite , named entity recognition @xcite , semantic role labeling @xcite , detection of machine translation errors @xcite . this diversity of applications suggests that wordlevel toxicity detection can also benefit from pretrained models. besides that, toxicity itself has been successfully tackled with bert-based models. research on sentence-level toxicity extensively used bert and other pre-trained models. both language-specific and multilingual bert models were used to fine-tune toxicity classifiers @xcite . this shows that bert has information on toxicity. thus, we follow this line of work. namely, we fine-tune a roberta model @xcite to perform a sequence labeling task. besides that, we train a model for sentence classification on the jigsaw dataset of toxic comments and use the information from this model to detect toxicity at the subsentential level. this helps us overcome the insufficient data size. this is in line with previous work, which has shown that sentence-level labels can be used in combination with token labels @xcite or completely substitute them @xcite . in our experiments, we test the hypothesis that the sentence-level toxicity labeling can be used for a sequence labeler that recognizes toxic spans in text. we suggest three ways of incorporating this data: as a corpus for pre-training, pseudo-labeling, and for joint training of sentence-level and tokenlevel toxicity detection models. our experiments show that the latter method yields the best result. moreover, we show that using sentence-level labels can dramatically improve toxic span prediction when the dataset with token-level labels is small. the contributions of this work are the following: ## the task the training data of the task comprises 7,940 english comments with character-level annotations of toxic spans. the labeling was performed manually by crowd workers. the spans labeled as toxic often contain rude words: "because he's a moron and a bigot. it's not any more complicated than that." (toxic spans are underlined). other toxic spans consist of words that become toxic in context: "section 160 should also be amended to include sexual acts with animals not involving penetration". borders of some toxic spans fall in the middle of a word; we treat such cases as markup errors. as a development set, we use the trial dataset of 690 texts provided by the task organizers. we evaluate our final models on the hidden test set of the task consisting of 2,000 texts. ## pre-training for toxic span detection here we give the motivation behind our models and describe their architecture and training setup. ## baselines in this section, we present a set of common baseline approaches used for sequence tagging, such as crf and lstm with pretrained word embeddings. we implement them in order to analyze the performance of our methods in the context of other techniques. word-based logreg this is a vocabulary-based method: we label words as toxic if they appear in our toxic vocabulary. the vocabulary is created as follows. we create a set of toxic and safe phrases, where toxic phrases are toxic spans from our data and safe phrases are sentences from our data with removed toxic spans. we then train a binary logistic regression classifier of toxic and safe phrases using words as features. the by-product of this classifier is the list of weights for all words from the data. we consider words with weights greater than a threshold as toxic. attention-based logreg another approach to represent words is to take their attention weights from a roberta-based sentence-level toxicity classifier (we train it on the jigsaw dataset). we assemble attention weights from all roberta heads and layers in a single vector of dimension 144. these vectors are used as features in a logistic regression classifier. this approach is motivated by the fact that a roberta model trained to recognize toxicity puts more emphasis on certain words associated with sentence-level toxicity. surprisingly, this model underperforms the logistic regression classifier, which uses words as features. ## conclusions we present a number of models for the detection of toxic spans within toxic sentences. all models are roberta language models fine-tuned on the data with character-level labeling of toxic spans. in addition to that, we perform fine-tuning on an additional dataset with sentence-level toxicity labeling. this yields an improvement. however, our analysis shows that the effect of such pre-training is marginal when the main dataset size exceeds 1,000 samples. therefore, substantial improvement is observed for small dataset sizes. nevertheless, the models we propose can be useful in extremely low-resource scenarios. our model performs closely to the winning systems. we suggest that the differences between the 20 top models might be attributed to the variation of deep learning models and overfitting the test set. in addition to that, the error analysis shows that some errors in our model might be due to inconsistencies in the test data. we release the code required to reproduce our experiments online. 5