S - NLP at S em E val-2021 Task 5: An Analysis of Dual Networks for Sequence Tagging (2021)

Abstract: The SemEval 2021 task 5: Toxic Spans Detection is a task of identifying considered-toxic spans in text, which provides a valuable, automatic tool for moderating online contents. This paper represents the second-place method for the task, an ensemble of two approaches. While one approach relies on combining different embedding methods to extract diverse semantic and syntactic representations of words in context; the other utilizes extra data with a slightly customized Self-training, a semi-supervised learning technique, for sequence tagging problems. Both of our architectures take advantage of a strong language model, which was fine-tuned on a toxic classification task. Although experimental evidence indicates higher effectiveness of the first approach than the second one, combining them leads to our best results of 70.77 F1-score on the test dataset.

## introduction social network sites are an integral part of our society. these platforms are often designed to maximize user interaction without sufficient means to moderate such interactions. the amount of users being cyber-bullied by toxic comments has reached an alarming proportion @xcite . to efficiently maintain the health of online communities, an automatic online-content filtering tool needs to be developed. numerous previous attempts to resolve this issue have focused on toxic comment classification @xcite @xcite . although these classification models are capable of detecting toxic comments, their outputs are not interpretable @xcite . on the other hand, toxic spans detection @xcite is a task of locating toxic * equal contribution segments in texts. with such a system, the moderators can easily highlight offensive words in comments, which is an essential and explainable assistance for automated comment rating. in this paper, we propose our two approaches to resolve the task. our contributions are as follows: ## methodology in this section, we describe our proposed framework in detail. firstly, we develop a simple but strong baseline to discover the effectiveness of different backbone models. consequently, we build, extend, and customize our two methods on top of the best backbone and baseline model. ## error analysis carefully analyzing errors made by our ensemble model on the test dataset has shed light on our system's weaknesses. the following terms are defined to aid the reading of this section: ## conclusion in this paper, we proposed a system to resolve the semeval task 5: toxic spans detection. our method utilized a pre-trained language model in toxic-domain and successfully combined two approaches self-training and feature-based learning to achieve a high f1-score of 70.77. finally, we provided insights into failure of the system and the task's potential falsely-negative annotations issue with careful error analysis. despite our success on the leader board, in future research, we determine to improve our model as follow: task hyperparameter value description fine-tune flair pre-trained weights "news-forward"/ "news-backward" initial weights of the flair models. sequence length 250 length of character sequences mini batch size 500 size of batches during training learning rate 20 initial learning rate patience 10 number of epochs without improvement max epochs 5 number of maximum training epochs optimizer sgd optimizer used for training scheduler annealonplateau learning rate scheduler train ner model dropout 0.3995 probability of an element to be zeroed locked dropout 0.4413 probability of entire parameters in embedding space to be zeroed word dropout 0.0677 probability of entire words (or characters) in embedding space to be zeroed learning rate 0.0005 initial learning rate min learning rate 1e-07 minimum learning rate to terminate training mini batch size 32 size of batches during training max epochs 50 number of maximum training epochs optimizer adamw optimizer used for training scheduler annealonplateau learning rate scheduler baseline + self-training hidden size t 150 size of the linear projection of the teacher's head classifier hidden size s 160 size of the linear projection of the student's head classifier learning rate 1e-05 the learning rate used for training optimizer adamw optimizer used for training scheduler none no learning rate scheduler used dropout t 0.3 dropout rate of all dropout layers in the teacher head classifer dropout s 0.4 dropout rate of all dropout layers in the student head classifer max epochs 5 total training epochs label smoothing 0.15 label smoothing coefficient confidence threshold 0.7 use in obtaining extra data for student model, all words with post-softmax score (calculated by the teacher model) less than this threshold will be ignored batch size 8 size of batches during training