GHOST at S em E val-2021 Task 5: Is explanation all you need? (2021)

Abstract: This paper discusses different approaches to the Toxic Spans Detection task. The problem posed by the task was to determine which words contribute mostly to recognising a document as toxic. As opposed to binary classification of entire texts, word-level assessment could be of great use during comment moderation, also allowing for a more in-depth comprehension of the model’s predictions. As the main goal was to ensure transparency and understanding, this paper focuses on the current state-of-the-art approaches based on the explainable AI concepts and compares them to a supervised learning solution with word-level labels. The work consists of two xAI approaches that automatically provide the explanation for models trained for binary classification of toxic documents: an LSTM model with attention as a model-specific approach and the Shapley values for interpreting BERT predictions as a model-agnostic method. The competing approach considers this problem as supervised token classification, where models like BERT and its modifications were tested. The paper aims to explore, compare and assess the quality of predictions for different methods on the task. The advantages of each approach and further research direction are also discussed.

## introduction the popularity of social media platforms has been continuously increasing over time. as reported by social (2020), 3.8 billion people have been active users of social media in january 2020. while user inter-connectivity carries a lot of positive effects, there is still a significant threat of cyberbullying and harassment caused by the illusion of anonymity online. statistics released by facebook @xcite identified that in the first quarter of 2020, there were 2.3 million bullying/harassment posts and 9.6 million hate speech posts detected as a violation of community standards. the importance of keeping the online community safe for users has caused many researchers to focus their work on detecting toxic contents in order to assist moderators in their difficult and mentally exhausting work. a lot of progress has been done in the task of toxic comment classification, but unfortunately, complex models still lack clear explanation and cannot gain much moderators' trust. a step towards increasing the transparency and therefore trust in automatic comment classifiers would be extending current approaches to operate on word-level rather than document-level. the ability to extract and highlight key text fragments that cause the toxic character of a comment could be of great use in post moderation. the lack of extensive datasets for word-level toxicity detection poses an obstacle to traditional, supervised learning classification, as current stateof-the-art models are very complex and normally require large-scale data. therefore, with the subject of this task being defined in terms of understanding and transparency for endpoint users, the authors decided to explore the explainable ai methodology. ongoing research in xai community examines many different approaches, with the two of them being in the focus of current work: modelspecific attention-based @xcite and model-agnostic @xcite explanations. this paper aims to compare the results obtained by these methods on toxic spans detection task @xcite for the english language using supervised models @xcite , acting as a baseline approach. this paper is organised as follows. related works and the backgrounds of discussed methods are described in section 2. section 3 presents three approaches to toxic spans detection, each with method-specific details in the corresponding subsection. finally, the results of the experiments are included in section 4, with the discussion and conclusions in section 5 and 6 respectively. pavlopoulos et al. (2017) introduced the application of a gru-based recurrent neural network for toxicity detection in documents. this approach was compared to the previous state-of-the-art for this task, presented in @xcite , which involved the use of logistic regression and multilayer perceptron and operated on n-grams. another model compared in their work was a convolutional neural network with pretrained word embeddings. rnn model outperformed other approaches, and it was further extended by the attention module, which improved the quality of classification. attention has been commonly used by many researchers as a medium of explanation for the predictions made by the model @xcite . it allows to analyse what part of the input the model focuses on the most while making a prediction. current research does not provide a unified answer to whether the attention mechanism in models is a good source of explanation. experiments carried out in @xcite point out that attention might not be a reliable method for interpreting the model's predictions. this was due to the fact that the attention distribution did not align well with other feature-importance measures. the other argument was that providing very different attention distribution often does not impact the predictions made by the model significantly. however, @xcite challenges the aforementioned work, claiming that while the answers given by attention should not be uncontrollably trusted and the definition of explanation must be clearly stated, it could still be a useful tool for model understanding and should not be disregarded easily. an alternative formulation of the toxic spans detection is to treat it as a supervised learning task, which involves training the models to predict the toxicity of each word separately. previously used, recurrent neural networks have been replaced as the state-of-the-art approach for many tasks by transformer-based models @xcite . transformer architecture, consisting of an encoder and decoder enriched with multi-head attention modules, turned out to outperform previous approaches on a series of nlp tasks. bidirectional encoder representations from transformers @xcite has been proposed as a powerful tool for many language-based problems. typically, bert is pre-trained on two unsupervised tasks when it is fed with large-scale text corpora and later adapted to a specific task during the finetuning stage, requiring much less data and computing time. ## toxic spans detection this section describes the following approaches: analysing the attention of an lstm model with orthogonalization of hidden states 3.1; using shap to provide explanation of bert predictions for toxic comment classification 3.2; training bert for classification of toxic tokens 3.3. ## results ## discussion as presented in the previous section, the supervised solution to token classification outperformed both xai approaches. the bert model was trained specifically to solve this type of problem and therefore achieved a score slightly better than lstm or shap. high labelling costs need to be taken into account when comparing those solutions, as training a robust bert model would require much more data, that would not be necessary for xai approaches, which are generally unsupervised. surprisingly, a model-agnostic approach turned out to perform much better than the attentionbased solution, even though the explained models achieved very similar results on the toxic comment detection task. in figure 3 one can see that the number of false negatives was significantly higher while relying on attention than it was while using shapley values. this could be because the model might pay a significant portion of attention to a very toxic word, which is enough to recognise the comment as toxic. while analysing the model's focus can improve the understanding of how the model works, it might not necessarily be enough to translate into clear decisions for each of the input components. furthermore, the need for threshold tuning somehow forced explainable approaches to mark a certain number of tokens as toxic, which could be reflected in a slightly higher number of false positives. visualisations of example predictions can be seen in while xai approaches might not be fitted to solve the problem of predicting the toxicity of each input word, they might still be useful for improving the transparency and understanding of predictions made by comment-level models. as recognised in error analysis, the number of false negatives was significantly higher for lstm and shap. but in terms of explanation, it might be enough for the user to obtain the few most toxic words per comment, rather than marking all of them, no matter how low the toxicity score is. this would not only provide the explanation on what the model considered while making a prediction but would also be a clear and transparent answer for the user in many real-time use cases. further examination might need to be done in order to assess the performance of those methods for a task specified in the aforementioned way. ## conclusions this work discussed different approaches to the toxic spans detection task. supervised toxic token classification and xai methods were examined to compare the results and assess whether explaining high-performing models can lead to a similar quality of prediction as models dedicated to a more detailed task. the supervised approach using the bert model achieved the best result in this task, but the xai methods have proven to be an interesting alternative that could reduce data preparation costs and improve transparency and understanding of the model's predictions. while not currently outperforming bert, explainable methods can be sufficient for many tasks where binary decision models are used. a system for toxic spans detection was prepared, achieving a 0.6859 span-level f1 score and placing 13th out of 91 in the overall ranking. a examples ortholstm shap ensemble that's right. they are not normal. and i am starting from the premise that they are abnormal . proceed wth the typical racist , bigot , sexist rubbish . thanks ! that's right. they are not normal. and i am starting from the premise that they are abnormal . proceed wth the typical racist, bigot, sex ist rubbish . thanks! that's right. they are not normal. and i am starting from the premise that they are abnormal . proceed wth the typical racist , bigot , sexist rubbish . thanks! a rock has more mental horsepower than the indubitably ignorant . and more integrity as well. they're largely very dishonest people who repeat the same lies over and over. just like their hero . a rock has more mental horsepower than the indubitably ignorant . and more integrity as well. they're largely very dishonest people who repeat the same lies over and over. just like their hero. a rock has more mental horsepower than the indubitably ignorant . and more integrity as well. they're largely very dishonest people who repeat the same lies over and over. just like their hero. ha ha, hillary lost. all you hillary fools need to be rounded up and put in prison along with your leader. you are all anti-american scum. perhaps we should dump you in mexico along with all the illegal trash you let in....if you love 'em so much, go live with 'em. but you are not welcome here, you hate filled marxist morons. ha ha, hillary lost. all you hillary fools need to be rounded up and put in prison along with your leader. you are all anti-american scum . perhaps we should dump you in mexico along with all the illegal trash you let in....if you love 'em so much, go live with 'em. but you are not welcome here, you hate filled marxist morons . ha ha, hillary lost. all you hillary fools need to be rounded up and put in prison along with your leader. you are all anti-american sc um . perhaps we should dump you in mexico along with all the illegal trash you let in....if you love 'em so much, go live with 'em. but you are not welcome here, you hate filled marxist morons . rabidly anti-canadian troll . rabidly anti -canadian troll . rabid ly anti-canadian troll . no, my poor benighted correspondent , your reductionism reveals an archaic mindset. intersex, questioning, nongendered, and a vast variety of others have rights too! no, my poor benighted correspondent, your reductionism reveals an archaic mindset. intersex, questioning, nongendered, and a vast variety of others have rights too! no, my poor benighted correspondent, your reductionism reveals an archaic mindset. intersex, questioning, nongendered, and a vast variety of others have rights too! justin is an embarrassment to canada. he needs a muzzle. he needs a brain . justin is an embarrassment to canada. he needs a muzzle. he needs a brain . you are a simpleton , who is full of racial hatred . that is too bad. you make america a worse place. you are a simple ton , who is full of racial hatred. that is too bad. you make america a worse place. when you consider the source -he writes like the trump we've all come to know -"i could stand in the middle of 5th avenue and shoot somebody and i wouldn't lose voters", a racist , misgynistic , liar who only brings hate to the table . when you consider the source -he writes like the trump we've all come to know -"i could stand in the middle of 5th avenue and shoot somebody and i wouldn't lose voters", a racist , misgynistic , liar who only brings hate to the table. when you consider the source -he writes like the trump we've all come to know -"i could stand in the middle of 5th avenue and shoot somebody and i wouldn't lose voters", a racist , mis gy nistic , liar who only brings hate to the table. total rubbish ! the birther bit was started by crooked hillary and perpetuated by zero his own damn self. total rubbish ! the birther bit was started by crooked hillary and perpetuated by zero his own damn self. total rubbish ! the birther bit was started by crooked hillary and perpetuated by zero his own damn self. damn , you beat me to it damn , you beat me to it damn , you beat me to it i don't think they eat them, just kill them , chop them up and sell off the parts . i don't think they eat them, just kill them, chop them up and sell off the parts. i don't think they eat them, just kill them , chop them up and sell off the parts. f*cking nasty ... f* ck ing nasty... f*cking nasty ...