“Did you really mean what you said?” : Sarcasm Detection in H indi- E nglish Code-Mixed Data using Bilingual Word Embeddings (2020)

Abstract: With the increased use of social media platforms by people across the world, many new interesting NLP problems have come into existence. One such being the detection of sarcasm in the social media texts. We present a corpus of tweets for training custom word embeddings and a Hinglish dataset labelled for sarcasm detection. We propose a deep learning based approach to address the issue of sarcasm detection in Hindi-English code mixed tweets using bilingual word embeddings derived from FastText and Word2Vec approaches. We experimented with various deep learning models, including CNNs, LSTMs, Bi-directional LSTMs (with and without attention). We were able to outperform all state-of-the-art performances with our deep learning models, with attention based Bi-directional LSTMs giving the best performance exhibiting an accuracy of 78.49%.

## introduction with the advent of social media, a large part of human interaction is carried out online. this leads to generation of huge amounts of textual data that can be used to draw meaningful inferences. social media websites like facebook, twitter, reddit etc are used by people across cultures to communicate with each other and voice their opinions. with the large amount of data available from social media, the study of various types of linguistic expressions like irony, humor, sarcasm, aggression, hate etc has become a keen research area. especially in the field of nlp, automatic detection of these expressions is being widely explored @xcite . automatic detection involves using computational methods to detect the presence of a particular emotion. although english is the language most commonly used on these websites, a majority of people are not native english speakers. these people therefore prefer to communicate in languages other than english @xcite . a study on the languages that are most commonly used for exchange of information on twitter showed that around 50% of the posts are written in languages other than english @xcite . this raises the opportunity of dealing with multi-lingual data generated by the social media sites. various statistics show that around 26% of the indian population is bilingual foot_0 . this gives rise to the phenomenon of code-switching and code-mixing @xcite . code mixing takes place when speakers use two or more languages below clause level in a single social context. multilinguals use such a mixture of languages, particularly on social media @xcite . there are multiple challenges of working with code-mixed data like large amount of new constructions that are a result of combining lexicons and syntax of two different languages, availability of very small amounts of annotated data and use of very different approaches when compared to mono-lingual data @xcite . in this paper, we wish to work on detecting one of the most popular linguistic constructs used across social medias, 'sarcasm'. the cambridgedictionary foot_1 defines sarcasm as 'the use of remarks that clearly mean the opposite of what they say'. example: "you have been working hard," he said with heavy sarcasm, as he looked at the empty page. starting with the earliest known work which focuses on sarcasm detection in speech @xcite , this domain has been widely explored in sentiment analysis. since sarcasm is a sentiment, detection of sarcasm is important in order to predict the sentiment of a sentence. being a challenging problem, automatic detection of sarcasm has been a popular area of research. although a lot of work has been carried out on sarcasm detection in english @xcite , the detection of sarcasm in code-mixed language like hinglish (hindi-english) is relatively unexplored. the current stateof-art performance is proposed via a random forest model on a dataset of 5000 hinglish tweets @xcite . the contributions of our work includes : ## experimental settings for the training sessions, we made a ten percent validation split and shuffled the training dataset, so that the model does not capture sequence trends, if any, in the training data, for a total of 20 epochs. the model checkpoints were saved at every epoch and those checkpoints which were saved before the model begins to overfit and the difference between the training and validation accuracies becomes significant, were used to calculate the accuracy numbers on the ten percent test dataset split. there are many important hyper-parameters in the training script of embeddings as well as the proposed models, which are tuned to produce the best training results on the validation data split. for training the word embeddings (both word2vec and fasttext), we used an embedding size of 300, window length of 10 and negative sampling polarity. in all the models, adam optimizer along with binary cross entropy loss function has been used. we used the same hyper parameter values for models 2.2 and 2.3, as in model 2.1, so as to study the impact of imposing bidirectional nature to the lstm layer, as well as exploring the effect of attention introduction. the parameters resulted in best outputs as confirmed later by trying out different values for the same. ## results the dataset, as presented in @xcite , being insufficient and skewed for our deep learning model architectures, we ran the state-of-theart models on our proposed dataset to carry out unbiased accuracy comparison of state-of-the-art techniques and neural network based models. the results for the same have been presented in to implement and analyze deep learning model architectures and different word representations for detection of sarcasm in hindi-english code-mixed data with a dataset large enough for deep learning models. all the proposed deep learning models performed better than the traditional state-of-the-art models, where the attention based bi-directional lstm network produced the best accuracy of 78.49%. in the lack of clean data and linguistic complexities associated with code-mixed data are the major challenges related to the task of sarcasm detection in hindi-english code mixed data. to allow the model to accommodate the noise in textual data, spelling errors, multiple contexts, and stemming words, even larger data is required along with cautiously labelled classes. ## conclusion social media, in recent years, has become a medium widely used by people for expression of thoughts and opinions, further leading to the realisation of tasks like emotion analysis and opinion mining. sarcastic content in these texts make it even more challenging to figure out the overall sentiment of the text, thus needing proper processing and analysis. in this paper, we presented a class-balanced hindi-english code mixed dataset for the problem of sarcasm detection, by scraping relevant tweets from twitter. we compared two representations, fasttext and word2vec, both based on different word representation learning mechanisms and trained on custom scraped data from scratch. we created two versions of embeddings, one trained with purely hinglish data, the other with a mixture of hinglish and english data, and compared the performance in each case. we analyzed the performance of different deep learning models, which take as input the generated word embeddings, to solve the problem of sarcasm detection. as future work, we plan to compare the vectors aligned with multilingual word embeddings after generation using muse with fasttext pre aligned word embeddings. we can also explore bert embeddings and evaluate their performance on the same task.