HITMI & T at S em E val-2021 Task 5: Integrating Transformer and CRF for Toxic Spans Detection (2021)

Abstract: This paper introduces our system at SemEval-2021 Task 5: Toxic Spans Detection. The task aims to accurately locate toxic spans within a text. Using BIO tagging scheme, we model the task as a token-level sequence labeling task. Our system uses a single model built on the model of multi-layer bidirectional transformer encoder. And we introduce conditional random field (CRF) to make the model learn the constraints between tags. We use ERNIE as pre-trained model, which is more suitable for the task accroding to our experiments. In addition, we use adversarial training with the fast gradient method (FGM) to improve the robustness of the system. Our system obtains 69.85% F1 score, ranking 3rd for the official evaluation.

## introduction with the prosperity of the internet, it is easier and easier for people to get information and publish their opinions online. however, sometimes users' opinions can be offensive to others. because toxic posts will have a negative impact on the network environment, and manual identification is timeconsuming and expensive, automatic detection of these behaviors has attracted researchers' attention. after adapting the hate-speech problem to the problem of word sense disambiguation, an approach to detect hate speech in online text is presented @xcite , which uses template-based strategy to generate features and an svm classifier to identify whether the text is toxic or not. in semeval-2020 task 12 @xcite and semeval-2019 task 6 @xcite , which also related to offensive statements, transformer-based methods were the most popular approaches for their great advantages in learning word representations in context. in semeval-2021 task 5: toxic spans detection @xcite , the organizers use posts from the publicly available civil comments dataset @xcite , which already comprises post-level toxicity annotations. after manual annotation, character-level annotation results are obtained, which are the toxic spans we need to locate. the task extends the prior work by identifying spans that make a text toxic, which can better explain why posts are offensive rather than just giving a system-generated unexplained toxicity score. we model the task as a sequence labeling task because toxic spans are contextually influenced. our model is in transformer-crf architecture, and we try different pre-trained models as the transformer's initialization to fine-tune model suitable for toxic spans detection. the conditional random fields (crf) @xcite allows the model to learn the constraints between tags. we also use the fast gradient method (fgm) @xcite as adversarial training strategy, which applies perturbation to word embedding to enhance the robustness of the model. the paper is organized as follows: section 2 briefly introduces the toxic spans detection shared task. section 3 talks about our system, including pre-processing and post-processing. section 4 shows our experiment results. finally, the conclusion and future work are drawn in section 5. ## toxic spans detection the research of automatic offensive language detection has gained attention in the past decade. instead of just classifying the whole comments or documents, the toxic spans detection task requires the system to detect the spans that make a text toxic. people often judge offensive sentences in terms of words, therefore the toxic spans in this task are always associated with words. in this task, the input sentence may contain no toxic span, which means it is not offensive. on the other hand, there may be more than one word that shows the author's malice in the sentence. considering toxic spans are contextually influenced, we model the task as a token-level sequence labeling task and use bio tagging scheme. ## system description our system uses a single model to get the result, which is in transformer-crf architecture. in our system, we utilize a pre-trained contextualized language model, namely ernie 2.0 @xcite , to identify the toxic tokens. and we introduce crf to learn the constraints between tags. in addition, we use adversarial training with the fgm to improve our performance. ## conclusion and future work the paper describes our system at semeval-2021 task 5, which integrating transformer and crf for toxic spans detection task. it is shown that, in this task, using ernie as pre-trained model achieves better performance in transformer-crf architecture. we convert the character-level sequence labeling task into token-level, and prove the importance of preprocessing method. we also use adversarial training with the fgm to improve the robustness of the system. our system achieves the third f1 score in official evaluation. since the best performance of our system is based on a single model, we are planning to find an effective ensemble method to improve the performance of the single model in the future.