HCMS at S em E val-2020 Task 9: A Neural Approach to Sentiment Analysis for Code-Mixed Texts (2020)

Abstract: Problems involving code-mixed language are often plagued by a lack of resources and an absence of materials to perform sophisticated transfer learning with. In this paper we describe our submission to the Sentimix Hindi-English task involving sentiment classification of code-mixed texts, and with an F1 score of 67.1%, we demonstrate that simple convolution and attention may well produce reasonable results.

## introduction sentiment analysis has gained importance in the current world where social media is crucial in gauging public opinion on products, political campaigns, latest trends and more. a large portion of the world's population is multilingual, and so is the content on social networks. code-mixing is a natural phenomenon among bilinguals where phrases and words from one language are employed in another. typically, the underlying grammar of the primary language that is being spoken is kept intact and phrases from another language are embedded into it. india in particular, has 23 officially recognized languages and a majorly bilingual population, requiring robust computational tools to effectively exploit the data it produces. deep neural networks have had great success in predicting sentiment encapsulated in text -a big part of this success has been their ability to utilize pretrained word embeddings. in code-mixed tweets however, hindi words are written in roman script instead of its native devanagari script, forcing one to use a transliterator in order to take advantage of pretrained aligned word embeddings, like fasttext @xcite . these words in roman script are unnormalized and may have multiple spellings; for example, bahut can be spelt in roman script as bahut, bohot, bohut etc., and such errors propagate from the transliterator to our downstream task, thereby reducing performance. in this work, we propose a deep convolution network with self-attention which shows promising results, without any pretraining. convolution neural networks (cnns) are known to capture local relations @xcite or local context, and work here as feature extractors acting as a substitute for handcrafted sentiment features. a self-attention layer is then applied over these features (presenting as vectors), which allow each individual feature to attend to all other features @xcite , providing global context. we call this the hierarchical context modeling system (henceforth referred to as hcms), due to the hierarchical nature of context extraction performed on both levels. ## background early sentiment analysis systems made use of corpus based statistics @xcite , linguistic tools like wordnet @xcite and lexicon based classifiers @xcite . more recently, a large portion of work has involved recurrent neural networks (rnns), long short term memory networks (lstms) and cnns at word and character levels @xcite . usage of sub-word level representations (based on character n-grams) @xcite , combining both cnns and lstms have also been demonstrated, where the cnn was used to generate sub-word features which were then augmented to word level features. @xcite also set precedent for normalizing code mixed text by dealing with the multiple transliterated word forms being generated as a result of the romanization of non-english scripts. attention mechanisms were first introduced for neural machine translation, and achieved state-of-the-art results at the time. the idea, which produces significant gains in our own model, has been applied to a variety of tasks such as image captioning, text classification, question answering @xcite @xcite , and even aspect level sentiment classification for more fine grained feature extraction @xcite . attention helps in obtaining better representations of text which in turn boosts downstream performance and thus, is a vital component of hcms where self-attention is used to boost the performance of the underlying architecture. approaching the problem as one of sequence classification, we present the hcms architecture as depicted in figure 1 . for every tweet (represented as a sequence of word embeddings), our model first performs a convolution and maxpooling operation to get local context between words that are adjacent to one another. in the next step, self-attention allows every context vector to attend to every other context vector in the sentence to get global context, summarizing the tweet. lastly, the model passes this summary through a dense layer to produce probabilities over the three classes, namely positive, negative and neutral. ## results and analysis we used the official validation and test sets provided. while our contest results stand at 67.1% f1, we have fine tuned our model since, and the latest scores for all experiments are shown in the svm @xcite trained on n-grams of sizes 1, 2 and 3 returned an f1 score of 65.97%. given its ability to handle out-of-vocab words, fasttext @xcite (also an n-gram based model) outperformed the svm model, achieving 66.26% f1. ablating the self-attention mechanism from the hcms model left a vanilla 1d cnn with max pooling. both, the lstm and the cnn performed similarly to the n-gram based fasttext model, but it is clear from the results that applying self-attention on top of features extracted by the respective models improved their performance significantly, providing credibility to our hypothesis of local and global features in the data. hcms can be seen outperforming the models lacking self-attention by a significant ~1%, giving a score of 68.44%. it was observed that cleaning the data improved performance. we also analysed the data and observed language distribution across the corpus. the numbers have been presented in figure 2 . as expected, the vocab is not equally split between the two languages. in fact, there is greater presence of hindi, and that too is romanized and not in its native devnagari script. in our experiments we tried to use a pretrained transliterator to de-romanize the hindi words, and then tried to employ pretrained word-embeddings, models and sentiment lexicon -due to the noisy devnagari-to-roman transliteration by the authors of the tweets however, our reverse transliteration failed and most words did not map into their respective embedding spaces. 4 the results of those experiments are presented in experiment f1 reverse transliteration + aligned fasttext 55.9 reverse transliteration + multilingual bert 52.3 reverse transliteration + sentiment annotation 52. 7 in our experiments, another noteworthy occurrence was that even though some models showed high validation scores, they were outperformed on the test set by models that produced lower validation score in comparison. this was a recurring theme in our a-b testing over validation and test data, and we feel that this can again be attributed to the noisy nature of code-mixed language. ## conclusion in this paper we have discussed a way to perform sentiment analysis on code-mixed hindi-english data, and we feel that the system is versatile enough to be applied to any mix of languages, especially when 3 links to library used for emoji replacement. 4 links to indictrans transliterator, aligned fasttext embeddings, multilingual-bert and sentiment lexicon. while it is encouraging to see that our model performs consistently, there are a couple of caveats that need to be addressed -even though the lack of pretraining, a low resource setting, and noise in data seem surmountable hurdles to the task, it is evident that the most promising avenues for improvement lie in exploring data cleaning/normalization and transfer learning. further analysis of the present methodology could also reveal the kind of transfer learning required -whether it be pretrained word embeddings, large scale language modeling or perhaps a better transliteration pipeline to name a few. to do so would require larger quantities of data and the application of the system to other linguistic tasks of settings similar to that of sentimix. all of these steps qualify within the scope of future work, and committing to them will reward us with more confidence in the hypotheses we propose in this paper, providing further proof of the system's robustness and effectiveness. we hope to undertake efforts for the same soon.