H amilton D inggg at S em E val-2021 Task 5: Investigating Toxic Span Detection using R o BERT a Pre-training (2021)

Abstract: This paper presents our system submission to task 5: Toxic Spans Detection of the SemEval-2021 competition. The competition aims at detecting the spans that make a toxic span toxic. In this paper, we demonstrate our system for detecting toxic spans, which includes expanding the toxic training set with Local Interpretable Model-Agnostic Explanations (LIME), fine-tuning RoBERTa model for detection, and error analysis. We found that feeding the model with an expanded training set using Reddit comments of polarized-toxicity and labeling with LIME on top of logistic regression classification could help RoBERTa more accurately learn to recognize toxic spans. We achieved a span-level F1 score of 0.6715 on the testing phase. Our quantitative and qualitative results show that the predictions from our system could be a good supplement to the gold training set’s annotations.

## introduction toxic messages remain a small but persistent part of online communications @xcite . nlp methods have been developed to identify these comments, often relying on deep-language models @xcite . however, the part of the message that is specifically toxic is often unknown. such information is useful not only for validating and explaining the judgments of models @xcite , but can also be useful for moderators to use when making decisions and working with these models in their deployment @xcite . this paper describes our model foot_0 and error analysis for semeval-2021 task 5: toxic spans detection @xcite . our model uses a deep learning approach to identify which tokens are toxic. the approach is motivated by two strands of prior work showing (1) that large language models can effectively serve as sequence-to-sequence (seq2seq) models and (2) that pre-training on a similar task can improve downstream performance @xcite . here, we treat the toxicspan detection tasks as a seq2seq task, where given a sequence of tokens, the model outputs per-token judgments of whether the token is in the toxic span. given the limited training data for task 5, we increase our training data by generating a silver-standard set of span judgments from lime explanations @xcite fropm a model trained to recognize toxic and non-toxic language. these additional judgments are intended to help the model learn the basic span recognition task and identify general toxic language, before fine-tuning on the task 5 data. ## system description our core system relies on a standard roberta model @xcite that is trained on a sequence-to-sequence task in two phases. the first phase pretrains the model with heuristically-created spans, gathered from reddit comments labeled for their toxicity. the second phase fine-tunes this model on the organizer-provided data. figure 1 shows the overview of the system. all the data used in the paper is in the english language. ## results our best model attained a span f1 of 0.672, and although close in score to the top result (0.708), was ranked 30 in the task. surprisingly, this bestperforming model did not make use of the pretraining on silver data. to better understand the performance, we ran two follow-up analyses to test how different strategies for training affected performances and an error analysis for what were common themes in errors. ## discussion and future work based on error analysis, our ptft model suffers from low performance when generating predictions on non-toxic contexts or long toxic spans. a modified error function that rewards for edge-case sce-narios can potentially improve the ptft performance. moreover, during the pre-training, we applied a simple-cutoff on the local weights to make labeling decisions for lime explanations. the cutoff was determined solely based on evaluations with the task data. if the lime labeling could introduce more robust variants in the loss evaluation, the silver data span labeling might be more representative of the task data's annotation logic. through comparing silver data with the gold data, we find the toxicity of some words is influenced by the broader linguistic environment. while the silver and gold data both consist of online comments, their time spans and topics are very different. the gold data uses contexts from 2015-2017 and has a concentration on political news; while the silver data covers 6 months of 2018 with no focused topics. our qualitative analysis finds that the addition of non-toxic examples in the silver data influenced the model to consider overly common toxic words less toxic than they were in the gold data. future work is needed to identify the optimal ratio of the toxic and non-toxic samples and to address domain/register differences in the data. last, the current approach could invite several natural improvements. for example, in the pretraining phase, we used tf-idf embedding and logistic regression for the base of lime explanations. this combination was chosen for its efficiency in the lime training phase. however, many other embedding and model combinations rendered much better classification results, which may generate better rationales for pre-training. ## conclusion we presented our system for semeval-2021 task 5 on toxic span prediction. our initial approach used explainable machine learning (lime) to generate a heuristically labeled span dataset, which was used to pre-train a roberta model to recognize toxic spans. however, our results show that when fine-tuned on the task data, the resulting model generates slightly shorter explanations and ultimately performs slightly worse than a model trained only on the task's training data-likely due to bias towards shorter spans generated by lime. in our subsequent error analysis, we show that the majority of our model's errors (50.5%) are associated with missed annotations in the ground truth, suggesting that actual model performance may be higher in practice.