A lpha B rains@ D ravidian L ang T ech: Sentiment Analysis of Code-Mixed T amil and T ulu by Training Contextualized ELM o Word Representations (2023)

Abstract: Sentiment analysis in natural language processing (NLP), endeavors to computationally identify and extract subjective information from textual data. In code-mixed text, sentiment analysis presents a unique challenge due to the mixing of languages within a single textual context. For low-resourced languages such as Tamil and Tulu, predicting sentiment becomes a challenging task due to the presence of text comprising various scripts. In this research, we present the sentiment analysis of code-mixed Tamil and Tulu Youtube comments. We have developed a Bidirectional Long-Short Term Memory (BiLSTM) networks based models for both languages which further uses contextualized word embeddings at input layers of the models. For that purpose, ELMo embeddings have been trained on larger unannotated code-mixed text like corpora. Our models performed with macro average F1-scores of 0.2877 and 0.5133 on Tamil and Tulu code-mixed datasets respectively.

## introduction sentiment analysis, a subfield of natural language processing (nlp), pursuits to computationally identify and extract subjective data, such as opinions, emotions, and attitudes from textual data. it plays a crucial role in understanding human's evaluations expressed in numerous forms of communication, including social media, customer reviews, and online forums @xcite . the proliferation of social media platforms allows individuals to proportion their perspective public opinions in written form on the internet @xcite . the users having knowledge of multiple languages, often post their thoughts and reaction in multilingualism. it happens due to no restrictions or limitations on the usage of diverse languages or their syntactic rules @xcite . the practice of blending multiple languages at various levels, including sentences, words, or sub-words within the same text, is known as codemixing. there are several reasons for code-mixing such as bilingualism, social community, vocabulary, the speaker and their conversation partner, the context or situation, and social prestige @xcite . these are considered the primary factors influencing code-mixing on social media networks. code-mixing often occurs due to the unavailability of a particular word or phrase in a particular language, compelling individuals to incorporate words or phrases from their native language in order to enhance comprehension for the receiver @xcite . although sentiment analysis has gained significant attention in recent years, most of the research has primarily focused on monolingual text, predominantly in english. however, the emergence of code-mixed text brings forth distinctive challenges and opportunities for researchers. the developing presence of code-mixed text presents unique demanding situations and possibilities for sentiment analysis. there is a limited amount of research on sentiment analysis in low-resourced languages particularly for tamil and tulu. the datasets for this research, contains various types of languages including tamil, tulu, english, romanized tamil, and tulu, as well as mixed text and emoticons. the diverse range of languages in the text presents significant difficulties in achieving higher accuracy of sentiment prediction models. in this paper, we present the sentiment analysis of code-mixed tamil and tulu youtube comments as a shared task 1 . we propose bidirectional long-short term memory (bilstm) networks based models for both languages; tamil and tulu which further use contextualized word embeddings at input layers of the models. for that purpose, embeddings from language models (elmo) em-beddings have been trained on larger unannotated code-mixed like corpora. for both language, the transfer learning by using trained elmo models was quite helpful to achieve the improved sentiment prediction results. our models performed with the macro average f1-scores of 0.2877 and 0.5133 on tamil and tulu code-mixed datasets respectively. elmo embeddings have shown state of the art performances for low-resourced nlp tasks such as part of speech tagging @xcite , phrase chunking @xcite , constituency @xcite and dependency parsing @xcite . the next sections of the paper present literature review, corpora details, model architecture and results. ## literature review tamil, being one of the ancient languages with a vibrant literary heritage, is predominantly spoken in the indian state of tamil nadu and certain regions of sri lanka @xcite . there has been a surge in interest regarding sentiment analysis in tamil, given its extensive usage in diverse domains such as social media, news, and product reviews. numerous research studies have concentrated on constructing sentiment analysis models specifically for tamil, employing a range of methodologies, including rule-based techniques, machine learning algorithms, and deep learning architectures. for code-mixed tamil-english text's sentiment analysis, chakravarthi et al. (2020a) developed a corpus, tamilmixsentiment foot_1 , which is a corpus comprising tanglish (a mix of tamil and english) comments from youtube videos. the development of tamilmixsentiment followed guidelines based on the work from mohammad (2016) and without annotating language tags at the word-level. the inter-annotator agreement was found to be 0.6, indicating a moderate level of agreement among the annotators. they annotated 15,744 comments, making it the largest sentiment corpus for the underresourced language featuring code-mixing phenomena. they detailed the procedure of developing a code-mixed corpus and attributing polarities. further, they presented the outcomes of sentiment analysis trained on the corpus, serving as a benchmark. chakravarthi et al. (2020b) employed bilstm and recurrent neural networks (rnn) with subword representation to categorize text based on its polarity. additionally, for code-mixed tamil-english corpus, chakravarthi et al. (2022) introduced three sentiment assessment frameworks: bert (bidirectional encoder representations from transformers) and logistic regression classifier, distilbert, and rapid text-mod. for tamil code-mixed sentiment analysis, shanmugavadivel et al. (2022) analyzed machine learning frameworks. the research objective was to develop hybrid deep learning models that combine convolutional neural network (cnn) with lstm and cnn+bilstm. hybrid models performance was compared with state-of-the-art methods, including traditional machine learning techniques. among all their developed models, the proposed cnn+bilstm framework outperformed with an accuracy of 66%. tulu is an indian language belonging to the dravidian language family, spoken mainly in the region. it is gaining attention in sentiment analysis research. however, compared to other languages, tulu has not been extensively studied in this area. the limited focus on sentiment analysis in tulu can be attributed to the lack of annotated datasets and linguistic resources available for the language. for sentiment analysis of tulu-english codemixed text, @xcite developed a corpus comprising 5,536 youtube comments. the dataset construction focused on extracting comments written in the latin script of tulu and tulu-english code-mixed. the annotated tulu-english dataset was then utilized to implement various machine learning (ml) and deep learning models, and a transformer-based classifier using bert framework. keras embeddings and term-frequency-inverse document-frequencies (tf-idf) were used as attributes for deep learning and machine learning models respectively. the bilstm framework demonstrated the best performance with notable f 1 -scores across all the classes. hegde et al. (2022a) worked on corpus creation for code-mixed tulu text for sentiment analysis. they scraped 7,171 youtube comments and subsequently annotated them to predict emotions within the code-mixed tulu data, establishing a foundational benchmark. they utilized traditional ml algorithms employing tf-idf features derived from word bigrams and trigrams. in all sentiment classes, the multi-layer perceptron (mlp) and support vector machine (svm) classifiers performed comparably better and reported an f 1 -score of 0.60. ## code-mixed corpora in this section, we present details of corpora and datasets which have been used to train the sentiment analysis and transfer learning. the train, development and test sets were released by the organizers of the shared task. moreover, we used additional corpora to train elmo embeddings to perform transfer learning for both tamil and tulu code-mixed text. ## model we developed the sentiment analysis model by using bidirectional long-short term memory (bil-stm) networks. bilstm based neural models are quite capable to learn sequence labels which can also be used to predict sentence level tasks like sentiment analysis. the model has two lstm layers, first layer scans the word sequences in the forward direction while the other layer scans the word sequences in opposite i.e. backward direction. the lstm based models learn next and previous words to attain the contextual information within sentences. the input sequence of n words x 1 ,x 2 ,...,x n is given as input. the bilstm(x 1:n ,i) function has been shown in the equation 1 which demonstrates the concatenation of forward and backward layers. lstm f represents the forward layer whereas lstm r shows the backward layer. the function denotes to a vector i by conditioning the past antiquity x 1:i and the forthcoming sequence x i:n as well. bilst m (x1:n, i) = lst m f (x1:i).lst mr(xn:i) (1) input layer word vectors pre-trained elmo embeddings w 1 w n ... ... hidden layers train samples dense (softmax) blstm (200 units) word representations predicted labels contextual representations sent 1 label 1 sent 2 label 2 sent 3 label 3 . . . . sent n label n p_label 1 p_label 2 p_label 3 . . . . p_label n vector encodings probabilities w 2 ... the softmax non-linearity function is employed at the output layer which performs the multi-class classification returning sentiment label classes l i for each input sequence of n words x 1 ,x 2 ,...,x n as shown in equation 2 . the figure 1 shows our bilstm based neural model architecture which has been trained for sentiment analysis for the tamil and tulu code-mixed datasets. the dataset contains sentences followed by sentiment labels. the training sentences have been transformed to vectors to use them in the neural model at input layer. the word vectors have been concatenated with the contextualized vectors achieved from the emlmo embeddings. the concatenated vectors are further fed to hidden lstm layers. we experimented with two hidden lstm layers. the dropout layers have been added between lstm layers and before the output layer. the lstm layers are followed by a flatten layer to perform sentiment labeling. the softmax nonlinearity has been used at dense output layer to perform multi-class classification. finally, the sentiment classes are predicted on the basis of maximum likelihood. the predicted labels have been evaluated against gold labels for development and test sets. the sentiment analysis model has been developed by using keras library with the tensorflow back-end in python-3. the bidirectional lstm layers had 200 hidden units. the value for dropout layers was set to 0.2 (20%). root mean squared propagation (rmsprop(0.001)) optimization function has been used in the model. the categorical cross-entropy loss function was used in all experiments. the word vectors have been trained to have 128 dimensions, however, elmo embeddings have been trained with 256 projection dimensions. the lstm sentence length was set to the longest sentence with padding sequences in the datasets for both tamil and tulu. transfer leaning by training elmo embeddings were quite useful to achieve better results. the following section describes the details of elmo embeddings and its parameters. ## results the results for tamil code-mixed sentiment analysis have been improved by enhancing the size and the quality of the datasets for transfer learning. the macro average f 1 -score has been improved with a gain of 0.0727 points. the model performed better on development set with a macro average f 1 -score of 0.4625 as compared to the test set. the reason behind the significant difference is the size of evaluation sets for tamil. the development set contains 3,786 sentence whereas the test set contains only 649 sentences. on the other hand, the tulu evaluation sets have almost same number of samples. the tulu model performed with the macro average f 1 -scores of 0.5386 and 0.5133 on development and test set respectively. the tulu dataset has less variations as compared to tamil data as it mostly contains romanized tulu comments which resulted higher results. from the results, it is quite evident that transfer learning has the ability to produce competitive results for code-mixed corpora. however, the data preparation is an important task before the training process. there should also be a balance in the corpus having representation of various types of comments. in this way, the sentiment analysis models could perform better. the sentiment analysis of code-mixed text is important research topic which requires more research to analyze online text. ## conclusion this paper presents the sentiment analysis of codemixed tamil and tulu youtube comments. the code-mixed text contains text from different scripts, such as, tamil, tulu, english, romanized tamil and tulu, mixed text and emoticons. the variety of languages makes it quite challenging to predict sentiments with higher accuracy. we proposed a bidirectional long-short term memory networks based model for both languages which further uses contextualized word embeddings at the input layers of the model. for that purpose, elmo embeddings have been trained on larger unannotated code-mixed text corpora. the transfer learning by using trained elmo models for both language was quite helpful to achieve improved sentiment analysis results. our models performed with the macro average f 1 -scores of 0.2877 and 0.5133 on tamil and tulu code-mixed datasets respectively.