Robust Deep Learning Based Sentiment Classification of Code-Mixed Text (2019)

Abstract: India is one of unique countries in the world that has the legacy of diversity of languages. Most of these languages are influenced by English. This causes a large presence of code-mixed text in Social Media. Enormous presence of this code-mixed text provides an important research area for Natural Language Processing (NLP). This paper proposes a novel Attention based deep learning technique for Sentiment Classification on Code-Mixed Text (ACCMT) of Hindi-English. The proposed architecture uses fusion of character and word features. Non availability of suitable Word Embedding to represent these Code-Mixed texts is another important hurdle for this league of NLP tasks. This paper also proposes a novel technique for preparing Word Embedding of Code-Mixed text. This embedding is prepared with two separately trained word-embedding on Romanized Hindi and English respectively. This embedding is further used in the proposed deep learning based architecture for robust classification. The Proposed technique achieves 71.97% accuracy, which exceeds the baseline accuracy.

## introduction languages used in india belong to several language families. historical presence of british on indian soil has led to a very high influence of english language on many of these indian languages. people belonging in a multi-lingual society of india, gives rise of a large amount of text in various social media @xcite . inclusion of english is very common in these texts. essentially, an utterance in which a user makes use of grammar, lexicon or other linguistic units of more than one language is said to have undergone code-mixing @xcite . hindi is the widely spoken language of india and used in various media. the number of native hindi speakers is about 25% of the total indian population; however, including dialects of hindi termed as hindi languages, the total is around 44% of indians, mostly accounted from the states falling under the hindi belt foot_0 . this community contributes a large amount of text on social media. the form of hindi language used in social media is mixed with english and are available in roman scripts. according to the study @xcite most common reason for this kind of code mixing in a single text is 'ease of use'. the code-mixed hindi and english language poses various types of challenges @xcite , which makes the text classification task on code-mixed text, an exciting problem in nlp community. despite a wide research on classification of code mixed texts, there remains open opportunities with two major aspects; first technique of preparing word embedding on code-mixed texts and second utilization of character and word features together to improve the accuracy. this research targets these two open points for exploration. ## related work various research works have tried to tackle these challenges. recent work of @xcite utilizes character level lstms to learn sub word level information of social media text. then this information is used to classify the sentences using an annotated corpus. the work is very interesting and achieves good accuracy. however the work does not intend to capture the information related to word level semantics. this provides a further scope of research to study the impact of word ## dataset we have considered hi-en code-mixed dataset 2 , shared by @xcite as a baseline for this research. ## proposed method the proposed method consists of two major parts. first one is preparing a suitable word-embedding of code-mixed text and later one is a robust deep learning architecture for classification on codemixed text. ## experimental setup this research used keras on python for all required implementations. the baseline dataset is divided into 3 splits i.e. training, validation and testing. initially the dataset is randomly divided into 80-20 train-test split. further train is randomly divided into 90-10 train-validation unlike the baseline implementation which splits 80-20 as trainvalidation. the results are reported over the test split here. we have experimented with various possible values of hyper parameters and the best set of hyper parameters is shown in the fig 2 . as discussed earlier first part of the architecture is meant for character based input. here a single sentence is considered to be of sequence of 200 characters. characters beyond 200 are ignored for sentence having more than 200 characters. a sentence with less than 200 characters is zero padded. point need to mention is that we have considered space also as valid character input. for the second part of the network we have use word embedding of different dimensions for example 100, 200 and 300. however it achieved best accuracy with 300 dimensional word-embedding. while training the fasttext word-embedding, 'minn' & 'maxn' parameters were set to 2 and 10 respectively. for word based input, a sentence of length 40 words is considered. a sentence with lesser than 40 words is zero embedding padded whereas words beyond 40 are ignored if sentence is having more than 40 words. also we empirically found that having two stacked lstm layers similar to @xcite gave optimal performance. we have used default keras implementations of categorical cross entropy for loss functions in different experiments. available implementation of focal loss 10 is used during few experiments. the intention of apply focal loss (lin, 2017) is to check the robustness of the proposed accmt architecture with respect to different loss function. of late focal loss has migrated from object detection to various other tasks, for example speech emotion recognition tripathi (2019) etc. we wanted to experiment and capture the impact of focal loss on classification of code-mixed text. default keras implementation for adam optimizer is used for experiments. on the other hand learning rate of 0.0008 and a decay of 0.000012 is set for rms prop in various set of experiments. dropout at character-lstm part is set to 0.2 and word-lstm is set to 0.4, where as the dropout of dense layers are set to 0.4. we have used the available implementation of attention layer in our code for model architecture. the model is trained over 50 epochs and batch size of 64 with 10-fold cross-validation. during each fold, the best model is picked based on validation accuracy. the experiments are conducted in the anaconda environment on a machine with intel core i5 processor and nvidia processor for gpu acceleration, 16 gb of ram and a 1 tb of hdd with windows 10 operating system. the 50 10 https://github.com/mkocabas/focal-loss-keras epochs of training of accmt takes 25 minutes in average. ## results and analysis we have conducted all experiments in the computing environment mentioned in above section. in the same environment, the implementation of @xcite attained maximum accuracy of 66.29% across 5 different executions. whereas the best performance of accmt is 71.97% exceeds the baseline performance by 5.68% in the same computing environment. to understand the impact of attention on the classification of code-mixed text, we have also experimented without attention. we have removed three attention layers from the accmt and created a deep learning architecture which uses only fusion of character and word features. this architecture showed a maximum of 69.845% accuracy on the same dataset. this implies that attention has improved accuracy with 2.125%. we also compared against @xcite which gave an accuracy of 64.3%. ## conclusion this paper shows the architecture of attention based deep learning architecture (accmt) which does fusion of character and word feature to develop a robust classifier for code-mixed text. the proposed accmt architecture performs well on the hi-en code-mixed dataset and outperforms the baseline accuracy. a major contribution of this paper is the technique of training word embedding for code-mixed text. this technique is used for