A Unified Multi task Learning Architecture for Hate Detection Leveraging User-based Information (2023)

Abstract: Hate speech, offensive language, aggression, racism, sexism, and other abusive language is a common phenomenon in social media. There is a need for Artificial Intelligence (AI) based intervention which can filter hate content at scale. Most existing hate speech detection solutions have utilized the features by treating each post as an isolated input instance for the classification. This paper addresses this issue by introducing a unique model that improves hate speech identification for the English language by utilising intra-user and inter-user-based information. The experiment is conducted over single-task learning (STL) and multi-task learning (MTL) paradigms that use deep neural networks, such as convolution neural network (CNN), gated recurrent unit (GRU), bidirectional encoder representations from the transformer (BERT), and A Lite BERT (ALBERT). We use three benchmark datasets and conclude that combining certain user features with textual features gives significant improvements in macro-F1 and weightedF1.

## introduction the rise in social media platforms enables a user to express their opinions, inner thoughts or emotions through social networks like twitter and facebook. social media's broad use has led to an increase in the number of nasty posts being shared online. the moderators are burdened by the quantity of content and the psychological stress of reviewing harmful posts. to combat this, an automated system is required to detect hate speech and assist the moderators in rejecting or approving the post. according to the statistics, passive or manual methods (like flagging) of preventing the spread of hate speech are neither efficient nor readily scaleable @xcite . hate speech is an umbrella term that covers instances of abusive, offen-sive, racist, sexist, aggressive, etc. this kind of behaviour is detrimental from a social and business perspective. the detection of hate speech is a challenging task due to limited text information, grammar and syntactic flaws @xcite . a user tending to post hate messages might do it again. having some background information about the user of a post may be very informative. the hypothesis is that by infusing user's historical tweets along with the information present in the tweet, the proposed system will be having good accuracy. the key attributes of our current work are as follows: ## data three publicly accessible data with the tweet id are used in the experiment. the past tweets are collected using the twitter api. the data is explained here. d1 @xcite : it includes the tweet id and 99,996 tweets classified with the labels, "hate," "abusive," "spam," and "neutral." the tweets were selected based on the sentiment analysis, showing strong negative polarity (< -0.7) and contain at least one offensive word. using twitter api, the meta tweet is crawled consisting of user meta information. we were able to collect 17,999 metatweets because the majority of the accounts were either suspended or the tweet was removed. d2 @xcite : provided semisupervised offensive language identification dataset (solid) which contains 9,089,140 tweets id with the label obtained in a semi-supervised manner using democratic co-training with olid @xcite as the seed data. the tweets with a probability @xmath0.85 for the offensive and non-offensive posts were taken. in addition to 'gun control', and 'to:breitbartnews' used during the trial annotation, four new political keywords were used to collect tweets for the full dataset: 'maga', 'antifa', 'conservatives', and 'liberals'. d3 (waseem and hovy, 2016): the 16,000 tweets were collected using keywords and phrases such as "mkr", "asian drive", "feminazi", "immigrant", "nigger", "sjw", "womenagainstfeminism", "blameonenotall", "islam terrorism", "notallmen","victimcard", "victim card", "arab terror", "gamergate", "jsil", "racecard", "race card" and divided into three categories: neural, sexism, and racism. as there were only 9 meta tweets for the racism class, only sexism and neural tweet id were taken into account to collect the historic posts. the data statistics is shown in ## methodology the proposed model is compared with several strong baselines: cnn: it is proposed by @xcite and consists of five main layers: input layer, embedding layer, convolution, pooling, and fully connected layer. gru: this is proposed by @xcite . the features are extracted by using two gating mechanisms, viz. reset and update. bert @xcite : it extracts the features by applying the bidirectional training of the transformer to learn the context of a word based on all of its surrounding leveraging masked language modelling (mlm) and next sentence prediction (nsp). albert: it is a lite bert for self-supervised learning of language representations to learn the features by mlm and sentence ordering prediction (sop). the aforementioned four models were trained in a stl paradigm. the bert model is selected to train the three datasets jointly in the mtl setup. it aims at solving more than one classification problem simultaneously. the end-to-end deep multi-task learning has been recently employed in solving various problems of natural language processing (nlp), such as sentiment, emotion, and hate prediction. @xcite defined mtl as follows: definition 1 : multi-task learning: given m learning tasks the mtl model is evaluated on three different datasets related to hate speech classification, sexism detection, and offensive language detection. it is explained as follows: multitask learning (mtl-dnn): it is proposed by @xcite for learning representation across multiple natural language understanding tasks. the lower layers are shared across each task while the top layers are task-specific. the input x is first represented as a sequence of embedding vectors for each word. this is followed by the transformer encoder capturing the contextual information for each word and generating the shared contextual embedding vectors. in the last layer, additional task-specific layers generate taskspecific representations which are then passed on to the final classification. the weight matrix associated with the shared layer, marked in red in figure 1 , is transferred to the new model to obtain the shared features. along with the shared features, meta features such as tweet based, intra-user, and inter-user features were also considered. these metadata information plays an important role in improving the performance and described as follows: tweet based(tb)(f tb ): the features such as the number of hashtags, number of emoticons, number of uppercase and lowercase letters, number of positive and negative words, and sentiment score by (hutto and gilbert, 2014) were considered. each tweet is lowercased and stemmed using the porter stemmer followed by creating unigram, bigram, and trigram features each weighted by its tf-idf. intra-user tweets(f intra ): given a tweet id, the user's historical post is collected by twitter api. given a tweet t, we collected m tweets posted by this user i.e. z @xmath1, z 2 ,....z m }. these intra-user tweets are passed through bert to get f intra . the value of m is 200 for each user and used in the batches of 50, 100, and 200 during training. interuser tweets(f inter ): the similarity between users on twitter is calculated by the formula proposed in (almahmoud and alkhalifa, 2018) which uses seven features. these are following and followers, mention, retweet, favourite, common hashtag, common interests, and profile similarity. we selected three similar users and included their tweets to represent inter-user information. the semantically similar tweets to the target are used to add features to the target tweet. all the similar tweets were passed to the encoder to generate f inter . proposed approach as shown in figure 1 the trained weight parameters shown in red obtained from mtl is transferred to get the target tweet feature t. the other three features, namely f intra , f inter , and f tb were concatenated with t and passed on to fully connected layer with softmax activation function to obtain the final prediction. ## experiment setup the experiments were performed using a 5-fold cross-validation approach. the 4-fold training set is split into 15% validation and 85% training while the last fold is treated as the test set to evaluate the model. all the deep learning models were implemented using keras @xcite with tensorflow @xcite as the backend. the number of filters used in cnn is 100, and the kernel width ranges from 1 to 4. for the gru, the number of hidden nodes is set to 100. the concatenation of word2vec embedding @xcite and fasttext @xcite is used for cnn and gru in stl and mtl. categorical cross-entropy is used as a loss function, and adam (kingma and ba, 2014) optimizer is used for optimizing the network. the learning rate of 2e-5 is used for all the models. the batch size of 32 is used to train the shared encoder and an epoch of 3 is found to be optimal. the value for bias is randomly initialized to all zeros, the relu activation function is employed at the intermediate layer, and softmax is utilized at the last dense layer. to achieve stability in the results produced, every classifier is run for 5 times and the output values were aggregated. ## results and analysis the bert-based multi-task learning outperforms the single task learning by reducing false positives. the inclusion of intra-user and inter-user features helped in improving the overall performance for all the three datasets. the model significantly improves upon adding all the features. the high performance observed in d2 is attributed to the fact that the instances in solid @xcite were generated by an ensemble of four models. hence the model is overfitting with the correct information in the case of d2. ## conclusion and future work in this work, we propose a novel method for hate speech detection by experimenting with cnn, gru, bert, and albert in stl and mtl fashion. the augmentation of inter-user, intra-user, and other tweet-based features outperforms the sota approaches in terms of macro-f1 and weighted-f1. furthermore, intra-user and inter-user representation learning can be generalized to other text classification tasks, where either user history or a large collection of unlabeled data is available. in addition to detecting potential hate speech, this method can also help study the dark triad personality (narcissism, machiavellianism, and psychopathy).