YNU - HPCC at S em E val-2021 Task 5: Using a Transformer-based Model with Auxiliary Information for Toxic Span Detection (2021)

Abstract: Toxic span detection requires the detection of spans that make a text toxic instead of simply classifying the text. In this paper, a transformer-based model with auxiliary information is proposed for SemEval-2021 Task 5. The proposed model was implemented based on the BERT-CRF architecture. It consists of three parts: a transformer-based model that can obtain the token representation, an auxiliary information module that combines features from different layers, and an output layer used for the classification. Various BERT-based models, such as BERT, ALBERT, RoBERTa, and XLNET, were used to learn contextual representations. The predictions of these models were assembled to improve the sequence labeling tasks by using a voting strategy. Experimental results showed that the introduced auxiliary information can improve the performance of toxic spans detection. The proposed model ranked 5th of 91 in the competition. The code of this study is available at https://github.com/Chenrj233/semeval2021_task5

## introduction existing toxicity detection datasets and models classify the entire comment or document and do not identify the range that makes the text toxic. a system that accurately locates the toxicity range in the text is crucial in achieving semi-automatic review. as a complete submission for the shared task, systems are required to extract a list of toxic spans or an empty list per text. we define a sequence of words that attribute to the text's toxicity as the toxic span. text this is a stupid example, so thank you for nothing a!@#!@. offset list @xcite @xcite @xcite @xcite @xcite @xcite the main purpose of this task is to identify the toxic spans in a given text; this can be transformed into a sequence labeling task in natural language processing. unlike normal sequence labeling tasks, this task is more challenging because the toxic spans in the text may involve a word, phrase, or even a sentence. traditional methods used to address the problem of sequence labeling include conditional random fields (crf) @xcite , combined models of both long-short-term memory and crf (lstm-crf) @xcite , and bidirectional encoder representation from transformers (bert) @xcite . in this study, we use bert, albert @xcite , roberta @xcite , and xlnet @xcite to solve this problem. compared with the conventional model, our model adds auxiliary information to improve the performance in this task. after a simple analysis of the text data, it can be found that not all the words in the toxic span have a toxic meaning, and some toxic meanings occur in a specific context or semantic conditions. therefore, if the tokens can be classified with the auxiliary information such as sentence representation, the performance of the model will improve. the results of the experiment prove that some of the proposed methods are effective. by using ensemble learning, we merge the results of the bert, albert, roberta, and xlnet models into the final prediction, obtaining the 5th rank out of 91 and a f 1 score of 0.696. ## transformer-based model with auxiliary information figure 1 shows the architecture of the proposed model, which consists of three layers: a transformer-based layer, an auxiliary information layer, and an output layer. the transformer-based layer can be bert, albert, roberta, xlnet, or any other transformer-based model. in the auxiliary information layer, several approaches are applied to combine token representation. the combined token representations are used in the output layer to output the label of each token. ## experimental results in this section, we present the comparative results of the proposed model. ## conclusion in this paper, we introduce the method we used in semeval-2021 task 5. we improved the performance of the basic model by reducing the number of categories for each token, selecting the appropriate loss function, adding some additional information to the representation vector of the tokens during classification, and finally obtaining a model that can detect the toxicity in a text. our experimental results showed that adding auxiliary information to the original token representation vector is helpful in sequence labeling tasks. in addition, we found that the model has some limitations. after analyzing the prediction results, we observed that although the model can learn the representation of each token well, token classification errors can occur when some tokens are toxic without the entire text being toxic. one possible solution for this is to add a text classification task to train the model.