T ue M ix at S em E val-2020 Task 9: Logistic Regression with Linguistic Feature Set (2020)

Abstract: Commonly occurring in settings such as social media platforms, code-mixed content makes the task of identifying sentiment notably more challenging and complex due to the lack of structure and noise present in the data. SemEval-2020 Task 9, SentiMix, was organized with the purpose of detecting the sentiment of a given code-mixed tweet comprising Hindi and English. We tackled this task by comparing the performance of a system, TueMix - a logistic regression algorithm trained with three feature components: TF-IDF n-grams, monolingual sentiment lexicons, and surface features - with a neural network approach. Our results showed that TueMix outperformed the neural network approach and yielded a weighted F1-score of 0.685.

## introduction the task of sentiment analysis with code-mixing is gaining more and more attention, given that on social media platforms, bilingual or multilingual individuals create content by mixing two or more different languages within a single sentence. code-mixing occurs when phrases, words, and morphemes of one language are inserted into a sentence of another language @xcite . people may code-mix to make themselves appear noteworthy or imply a certain social status @xcite , to show solidarity, or simply to express their feelings @xcite . a substantial number of individuals combine english and hindi on social media platforms, and as a result, the need to process code-mixed content has become essential if we want to build robust and competitive systems for sentiment detection. sentiment analysis of code-mixed text faces multiple challenges due to the lack of a formal grammar for the hybrid code-mixed generated language and the existing characteristics of multilingual content where there are atypical variations in spelling, ungrammatical structures, and transliteration @xcite . the international workshop on semantic evaluation, or semeval, addressed this challenge by proposing the shared task sentimix: sentiment analysis for code-mixed social media text @xcite . the training data provided contained 14000 hindi-english tweets annotated with the corresponding sentiment label (positive, neutral, or negative), and with additional language labels at the word-level (english (eng), hindi (hin), emoji (emt), or universal (0) for other tokens). the class distribution was fairly balanced with 33.1% positive, 37.6% neutral, and 29.29% negative tweets. our strategy consisted of a machine learning and a neural network-based approach. our main system, tuemix, used a logistic regression algorithm trained with three categories of features: tf-idf n-grams, monolingual sentiment lexicons, and a number of surface features. in our findings, tuemix achieved better results than the neural network systems, placing in the top third of 62 teams (listed as "guzimanis" in the official codalab rankings). the addition of the linguistic features beyond the tf-idf n-grams were also shown to enhance our model. in this paper, we detail our approach. ## related work a growing body of work has addressed some of the aforementioned challenges of sentiment analysis of code-mixed content. the sail code-mixed shared task @xcite focused on the detection of emotion in indian languages with code-mixed text. most of the participants attempted to solve the task by using machine learning algorithms such as support vector machines (svms) or naive bayes trained with n-grams or sentiment lexicon-based features. another approach used in the same competition employed the use of emoji dictionaries annotated with the corresponding sentiment. the participants who submitted solutions based on deep learning systems did not report better results than those who submitted solutions based on machine learning. another approach to sentiment analysis of code-mixed text involved the construction of a system composed of a monolingual and a multilingual model @xcite . high accuracy and f1-scores were reported when pursuing a hybrid approach, specifically a system that initially creates sub-word level representations for words employing a convolutional neural network (cnn), which are subsequently fed together with orthographic features and word embeddings into a dual encoder network @xcite . similarly high results were reported in an approach consisting of two bidirectional long short-term memory networks (lstms) combined with orthographic features and monolingually trained word embeddings @xcite . another attempt included svm, naive bayes, logistic regression, and neural networks trained with tf-idf vectors and two monolingual dictionaries, one for english and one for hindi @xcite . an additional analysis of 840 experiments revealed the best combination for emotion detection of hinglish text to be a combination of tf-idf feature vectors, feature selection based on gain ratio, and a radial basis function neural network @xcite . ## model in this section we present the architecture of our systems. in line with previous work on the effectiveness of using tf-idf vectors for sentiment analysis, we first created a sparse feature matrix of word and character n-grams using sklearn's tf-idf vectorizer @xcite . baseline measures were calculated for seven different sklearn classifiers, namely logistic regression, linear svm, random forest, multinomial naive bayes, gaussian naive bayes, k-nearest neighbors, and extreme gradient boosting. a preliminary analysis of the scores revealed logistic regression as the highest-performing model. as a point of comparison, we also developed neural network models in parallel using keras (chollet and others, 2015). the following additional steps were undertaken to improve the performance of our models. ## conclusion & future work in our approach to this task, we compared a neural network approach with a logistic regression classifier, tuemix, consisting of three categories of linguistic features. the linguistic features beyond tf-idf n-grams were shown to enhance the model's performance and resulted in a final f1-score of 0.685. tuemix was also much more effective at predicting the positive and negative classes than the neutral class. further work could seek to improve predictions for the neutral class and investigate which additional linguistic features impact sentiment analysis and in which direction. our error analysis also revealed some of the difficulties of collecting and analyzing code-mixed data. our results are in line with other tasks which have shown n-gram-based machine learning models to outperform neural networks (c ¸öltekin @xcite . however, because we devoted more attention to tuning and feature engineering tuemix, we cannot make an assertion that a machine learning approach is strictly superior to a neural network approach for this task. overall, due to the aforementioned challenges, much work remains to be done in the field of sentiment analysis of code-mixed social media text. several avenues of future research include combining rule-based systems with machine learning systems, adding differing weights to different features and parts of the text, and the generation of more multilingual or code-mixed lexical resources for sentiment analysis.