BAKSA at S em E val-2020 Task 9: Bolstering CNN with Self-Attention for Sentiment Analysis of Code Mixed Text (2020)

Abstract: Sentiment Analysis of code-mixed text has diversified applications in opinion mining ranging from tagging user reviews to identifying social or political sentiments of a sub-population. In this paper, we present an ensemble architecture of convolutional neural net (CNN) and self-attention based LSTM for sentiment analysis of code-mixed tweets. While the CNN component helps in the classification of positive and negative tweets, the self-attention based LSTM, helps in the classification of neutral tweets, because of its ability to identify correct sentiment among multiple sentiment bearing units. We achieved F1 scores of 0.707 (ranked 5th) and 0.725 (ranked 13th) on Hindi-English (Hinglish) and Spanish-English (Spanglish) datasets, respectively. The submissions for Hinglish and Spanglish tasks were made under the usernames ayushk and harsh_6 respectively.

## introduction the research problem of sentiment analysis of code-mixed social media text appeared as part of the semeval shared challenge 2020 @xcite . mixing languages while writing text, also called code-mixing, is a typical pattern observed in almost all forms of communication, including social media text. we only focus on two popular bilingual code-mixing styles namely hinglish and spanglish. sentiment analysis is a term broadly used to classify states of human affection and emotion. interpreting code-mixed languages is difficult not only because the sentences may not fit a particular language model, but also because mixed text on social-media usually contains tokens such as hashtags, and usernames. in this paper, we present an ensemble of cnn and self-attention based lstm, utilizing the xlm-r embeddings @xcite . while cnns have been used for sentiment analysis before @xcite , none of the previous works have used a self-attention based lstm along with it. we found that while the cnn component worked well for positive and negative tweets, the self-attention component worked better for neutral tweets, necessitating an ensemble of the two. the implementation of our system is made available via github 1 . ## related work performing standard nlp tasks on code-mixed data has presented significant challenges. @xcite attempted to find methods for pos tagging of code-mixed social media text. another work by @xcite used cnns to learn subword level embeddings and then utilized these embeddings in a bilstm network to learn subword level information from social media text. subword level representations are particularly important while dealing with noisy texts containing misspellings and punctuations. however, this work doesn't capture information about word-level semantics. more recent work by @xcite uses two parallel bilstms, which they call the collective and specific encoder and an additional feature network. this approach combines recurrent neural networks utilizing attention mechanisms, which helps in evaluating the overall sentiment using attention weights when presented with a mixture of local sentiments. the tweets have been originally provided in the latin script with their corresponding language tags. before feeding the tweets to any training stage, they are preprocessed using the following procedure (figure 1 ): ## data description we used the dataset provided by the organizers of task-9 of semeval 2020 @xcite for training both hinglish and spanglish models. the data has been annotated semi-automatically. the statistics of the dataset are shown in ## experiments and results we first trained a vanilla cnn model on the provided dataset using the xlm-r embeddings. the cnn model seemed to be confused on neutral data points but worked well on positive and negative tweets. figure 7: confusion matrix for ensemble on spanglish validation data foot_1 the self-attention model outperforms the previous model on neutral data points though it performs worse on the positive and negative samples. the good performance on neutrals can be attributed to the fact that neutral tweets may contain multiple sentiment bearing units which the model is capable of handling. combining the results of cnn with those of the self-attention model was the primary motivation for using an ensemble of the two. the ensemble outperforms all our previous models, achieving a recall of 0.705 with an f1-score of 0.707 on the hinglish test dataset and a recall of 0.696 with an f1-score of 0.725 on the spanglish test dataset (see ## analysis 6.1 visualization of the individual components to visualize the sentence embeddings learned by the model for the hinglish test dataset, we projected the sentence vectors obtained before the final fully connected layer onto a lower-dimensional subspace using the t-sne algorithm (van der maaten and hinton, 2008) for the two components (see figure 8 ). for cnn, the positive and negative tweets seem to form two distinct clusters, while the neutral tweets are scattered among them. in contrast, for the self-attention component, neutrals seem to form a distinct cluster, while the positive and negative classes are partially dispersed in a wide region. thus, the two components, in a way, complement each other for better predictions over all the three classes. ## conclusion for our system, we use an ensemble of cnn and self attention architectures with xlm-r multilingual embeddings. we analyze which models work better for different classes of tweets. our self-attention system helps in better classification of neutral tweets, which are difficult to classify due to multiple sentiment bearing units. creating an ensemble with cnn helps in better classification of all the three classes. we also visualize how our model performs on different classes of tweets using the t-sne algorithm. our results show an improvement over some of the previous works in this field.