Leveraging Dependency Grammar for Fine-Grained Offensive Language Detection using Graph Convolutional Networks (2022)

Abstract: The last few years have witnessed an exponential rise in the propagation of offensive text on social media. Identification of this text with high precision is crucial for the well-being of society. Most of the existing approaches tend to give high toxicity scores to innocuous statements (e.g., “I am a gay man”). These false positives result from over-generalization on the training data where specific terms in the statement may have been used in a pejorative sense (e.g., “gay”). Emphasis on such words alone can lead to discrimination against the classes these systems are designed to protect. In this paper, we address the problem of offensive language detection on Twitter, while also detecting the type and the target of the offense. We propose a novel approach called SyLSTM, which integrates syntactic features in the form of the dependency parse tree of a sentence and semantic features in the form of word embeddings into a deep learning architecture using a Graph Convolutional Network. Results show that the proposed approach significantly outperforms the state-of-the-art BERT model with orders of magnitude fewer number of parameters.

## introduction offensive language can be defined as instances of profanity in communication, or any instances that disparage a person or a group based on some characteristic such as race, color, ethnicity, gender, sexual orientation, nationality, religion, etc. @xcite . the ease of accessing social networking sites has resulted in an unprecedented rise of offensive content on social media. with massive amounts of data being generated each minute, it is imperative to develop scalable systems that can automatically filter offensive content. the first works in offensive language detection were primarily based on a lexical approach, utilizing surface-level features such as n-grams, bag-ofwords, etc., drawn from the similarity of the task to another nlp task, i.e., sentiment analysis (sa). these systems perform well in the context of foul language but prove ineffective in detecting hate speech. consequently, the main challenge lies in discriminating profanity and hate speech from each other @xcite . on the other hand, recent deep neural network based approaches for offensive language detection fall prey to inherent biases in a dataset, leading to the systems being discriminative against the very classes they aim to protect. @xcite presented the evidence of a systemic bias in classifiers, showing that such classifiers predicted tweets written in african-american english as abusive at substantially higher rates. syntactic features are essential for a model to detect latent offenses, i.e., untargeted offenses, or where the user might mask the offense using the medium of sarcasm @xcite . syntactic features prevent over-generalization on specific word classes, e.g., profanities, racial terms, etc., instead examining the possible arrangements of the precise lexical internal features which factor in differences between words of the same class. hence, syntactic features can overcome the systemic bias, which may have arisen from the pejorative use of specific word classes. a significant property of dependency parse trees is their ability to deal with morphologically rich languages with a relatively free word order @xcite . motivated by the nature of the modern twitter vocabulary, which also follows a relatively free word order, we present an integration of syntactic features in the form of dependency grammar in a deep learning framework. in this paper, we propose a novel architecture called syntax-based lstm (sylstm), which integrates latent features such as syntactic dependencies into a deep learning model. hence, improving the efficiency of identifying offenses and their tar-s.no. hateful tweet scenarios 1 uses sexist or racial slurs. 2 attacks a minority. 3 seeks to silence a minority. 4 criticizes a minority (without a well-founded argument). 5 promotes but does not directly use hate speech or violent crime. 6 criticizes a minority and uses a straw man argument. 7 blatantly misrepresents truth or seeks to distort views on a minority with unfounded claims. 8 shows support of problematic hashtags. e.g. "#banislam," "#whoriental," "#whitegenocide" 9 negatively stereotypes a minority. 10 defends xenophobia or sexism. 11 contains an offensive screen name we evaluate our model on two datasets; one treats the task of hate speech and offensive language detection separately @xcite . the other uses a hierarchical classification system that identifies the types and targets of the offensive tweets as a separate task @xcite . ## related work hate speech detection, as a topical research problem, has been around for over two decades. one of the first systems to emerge from this research was called smokey @xcite . it is a decision-treebased classifier that uses 47 syntactic and semantically essential features to classify inputs in one of the three classes (f lame, okay or maybe). smokey paved the way for further research in using classical machine learning techniques to exploit the inherent features of natural language over a plethora of tasks such as junk filtering @xcite , opinion mining @xcite etc. owing to the unprecedented rise of social networks such as facebook and twitter, most of the research on hate speech detection has migrated towards the social media domain. to formalize this new task, a set of essential linguistic features was proposed @xcite . initial research in this direction focused more on detecting profanity, pursuing hate speech detection implicitly @xcite . using these systems, trained for detecting profanities, to detect hate speech reveals that they fall prey to inherent biases in the datasets while also proving ineffective in classifying a plethora of instances of hate speech @xcite . research has also shown the importance of syntactic features in detecting offensive posts and identifying the targets of such instances @xcite . on social media, it was found that hate speech is primarily directed towards specific groups, targeting their ethnicity, race, gender, caste, etc. @xcite . elsherief et al. (2018) make use of linguistic features in deep learning models, which can be used to focus on these directed instances. the problem with this approach is two-fold. first, these linguistic features learn inherent biases within the datasets, thus discriminating against the classes they are designed to protect. second, the use of explicit linguistic features to detect hate speech leaves the model prone to the effects of domain shift. altogether, there is a need to develop more robust techniques for hate speech detection to address the above mentioned issues. while the use of syntactic features for the task has proven useful, there has been little effort towards incorporating non-euclidean syntactic linguistic structures such as dependency trees into the deep learning sphere. graph neural networks (gnns) provide a natural extension to deep learning methods in dealing with such graph structured data. a special class of gnns, known as graph convolutional networks (gcns), generalize convolutional neural networks (cnns) to non-euclidean data. the gcns were first introduced by @xcite , following which, @xcite presented a scalable, first order approximation of the gcns based on chebyshev polynomials. the gcns have been extremely successful in several domains such as social networks @xcite , natural language processing @xcite and natural sciences @xcite . @xcite were the first to show the effectiveness of gcns for nlp by presenting an analysis over semantic role labelling. their experiments paved the way for researchers to utilize gcns for feature extraction in nlp. since then, gcns have been used to generate embedding spaces for words @xcite , documents @xcite and both words and documents together @xcite . even though gcns have been used in nlp, their inability to handle multirelational graphs has prevented researchers from incorporating the dependency parse tree in the deep feature space. in this paper, we present a first approach towards transforming the dependency parse tree in a manner that allows the gcn to process it. the final model is a combination of a bilstm based semantic encoder, which extracts semantic features and addresses long-range dependencies, and a gcnbased syntactic encoder, which extracts features from the dependency parse tree of the sentence. results show that the proposed approach improves the performance of automated systems in detecting hate speech and offensive language instances, differentiating between the two, and identifying the targets for the same. ## methodology traditionally, grammar is organized along two main dimensions: morphology and syntax. while morphology helps linguists understand the structure of a word, the syntax looks at sentences and how each word performs in a sentence. the meaning of a sentence in any language depends on the syntax and order of the words. in this regard, a sentence that records the occurrence of relevant nouns and verbs (e.g., jews and kill) can prove helpful in learning the offensive posts and their targets @xcite . further, the syntactic structure i ⟨intensity⟩ ⟨userintent⟩ ⟨hatetarget⟩, e.g., "i f*cking hate white people," helps to learn more about offensive posts, their targets, and the intensity of the offense @xcite . our approach incorporates both semantic features and the dependency grammar of a tweet into the deep feature space. the following subsections present a detailed discussion on the proposed methodology. ## experimental setup this section describes the dataset and the experimental setup for the models reported in the paper. ## results we evaluate two instances of our model, (1) with a randomly initialized embedding matrix (referred to as sylstm) and ( 2 ) utilizing the pretrained glove twitter embeddings (referred to as sylstm*). a paired student's t-test using the weighted-f1 measure of the model's performance shows that our models significantly outperform each of the baselines across all the tasks (@xmath0.001). ## conclusion in this paper, we present a novel approach called the sylstm which demonstrates how gcns can incorporate syntactic information in the deep feature space, leading to state-of-the-art results for finegrained offensive language detection on twitter data. our analysis uncovers the semantic and syntactic encoders' complementarity while revealing that the system's performance is largely unaffected for mislabeled dependencies over congruent dependency trees. leveraging the dependency grammar of a tweet provides a practical approach to simulating how humans read such texts. furthermore, the performance results of the sylstm indicate the robustness of the architecture in generalizing over small datasets. the added simplicity of the overall architecture promotes applicability over other nlp tasks. the sylstm can be used as an efficient and scalable solution towards accommodating graph-structured linguistic features into a neural network model. replication package. the replication package for this study is available at https://github. com/dv-fenix/sylstm .