Graph Convolutional Networks with Multi-headed Attention for Code-Mixed Sentiment Analysis (2021)

Abstract: Code-mixing is a frequently observed phenomenon in multilingual communities where a speaker uses multiple languages in an utterance or sentence. Code-mixed texts are abundant, especially in social media, and pose a problem for NLP tools as they are typically trained on monolingual corpora. Recently, finding the sentiment from code-mixed text has been attempted by some researchers in SentiMix SemEval 2020 and Dravidian-CodeMix FIRE 2020 shared tasks. Mostly, the attempts include traditional methods, long short term memory, convolutional neural networks, and transformer models for code-mixed sentiment analysis (CMSA). However, no study has explored graph convolutional neural networks on CMSA. In this paper, we propose the graph convolutional networks (GCN) for sentiment analysis on code-mixed text. We have used the datasets from the Dravidian-CodeMix FIRE 2020. Our experimental results on multiple CMSA datasets demonstrate that the GCN with multi-headed attention model has shown an improvement in classification metrics.

## introduction through the advent of social media, people from around the world can connect and exchange information instantly. the evolution of social media texts from blogs and messaging websites has created many new opportunities for information access worldwide @xcite . multilingual communities often express their thoughts on social media by mixing different languages in the same utterance @xcite . this mixing or alteration of two or more languages is known as code-mixing or code-switching @xcite . the computational modeling of code-mixed text is challenging due to the linguistic complexity, nature of mixing, the presence of non-standard variations in spellings, grammar, and transliteration @xcite @xcite . because of such non-standard variations, cm poses several unseen difficulties in fundamental fields of natural language processing (nlp) tasks such as language identification, part-ofspeech tagging, shallow parsing, natural language understanding, sentiment analysis, etc. to encourage research on code-mixing and to solve the problems related to code-mixed text, the nlp community organizes several tasks and workshops such as task9: sentimix, semeval 2020 @xcite , and 4th workshop on computational approaches for linguistic code-switching @xcite , dravidian code-mix fire 2020 @xcite . all of these tasks focus on sentiment analysis on code-mixed text. some researchers used traditional methods, such as support vector machines (svm's) @xcite and logistic regression (lr), where the code-mixed text is represented with sparse linguistic features (e.g., bag-of-words and n-grams). others used deep learning models such as convolutional neural networks (cnn) @xcite , recurrent neural networks (rnn) @xcite , transformer models @xcite , etc. the above deep learning models can capture semantic and syntactic information well in local consecutive word sequences. such position-dependent models are not suitable for dravidian languages that follow free word order format. recently, an evolving research direction known as graph neural networks @xcite or graph representation learning has attracted wide attention. in graph-based approaches, the whole corpus is viewed as a graph @xcite . graph convolutional networks have been effective at tasks for knowledge representation and can preserve global structure information of a graph in graph embeddings. in this work, we propose the use of the graph neural network for code-mixed sentiment analysis. our method focuses on constructing a single large graph from an entire corpus. we modeled the graph with a graph convolutional network (gcn) @xcite with multiheaded attention. a graph neural network captures high-order neighborhood information well by using an adjacency matrix. this adjacency matrix helps the graph neural network to not dependent on consecutive word order format. thus, it is well suited for the dravidian language datasets that follow a free word order. the paper is organized as follows. section 2 provides related work on code-mixed sentiment analysis. section 3 describes the proposed work. section 4 presents the experimental setup and the performance of the model. section 5 concludes our work. ## related work in this section, we study the related work in the field of code-mixed sentiment analysis. ## proposed work in this section, we describe our proposed work. first, we present the pre-processing steps to filter the data, and then we describe graph-convolutional networks with multi-headed attention for codemixed sentiment analysis. the data we have used for cmsa consists of code-mixed tamil-english and malayalam-english youtube comments, and it has many irregularities. such as the roman script is used to write tamil and malayalam text with variations in the spelling, irregular grammatical structure, mixing of languages at the word level, informal abbreviations, slang, and use of emoticons in text. we can minimize some of these irregularities by using pre-processing methods. ## experiments in this section, we compare our method with several baselines. we used only the back transliterated dataset to run all the models. ## conclusion this paper describes the graph convolution networks with multi-headed attention for code mixed sentiment analysis on dravidian languages. initially, we transliterate the data and build a word document graph for the whole corpus. this approach considers global dependencies between words and documents in the corpus. a simple threelayer gcn with multi-headed attention on cmsa demonstrates encouraging results by outperforming numerous state-of-the-art methods. our approach obtained the best results on the malayalam-english cm dataset with a weighted-f1 score and an accuracy of 0.75 and 0.73. we will explore our model's performance on various graph networks with different filtering and smoothing techniques for future work.