Sarcasm Identification and Detection in Conversion Context using BERT (2020)

Abstract: Sarcasm analysis in user conversion text is automatic detection of any irony, insult, hurting, painful, caustic, humour, vulgarity that degrades an individual. It is helpful in the field of sentimental analysis and cyberbullying. As an immense growth of social media, sarcasm analysis helps to avoid insult, hurts and humour to affect someone. In this paper, we present traditional machine learning approaches, deep learning approach (LSTM -RNN) and BERT (Bidirectional Encoder Representations from Transformers) for identifying sarcasm. We have used the approaches to build the model, to identify and categorize how much conversion context or response is needed for sarcasm detection and evaluated on the two social media forums that is twitter conversation dataset and reddit conversion dataset. We compare the performance based on the approaches and obtained the best F1 scores as 0.722, 0.679 for the twitter forums and reddit forums respectively.

## introduction social media have shown a rapid growth of user counts and have been object of scientific and sentiment analysis as in @xcite . sarcasm occurs frequently in user-generated content such as blogs, forums and micro posts, especially in english, and is inherently difficult to analyze, not only for a machine but even for a human. sarcasm analysis is useful for several applications such as sentimental analysis, opinion mining, hate speech identification, offensive and abusive language detection, advertising and cyber bullying. in figurative language 2020 task 2: shared task on sarcasm detection in social media forums. it focuses to identify the given conversion text is sarcastic or not and find how much context is helpful for sarcasm identification have modelled either the given instance may be isolated or combined. it focuses on two social media forums that are twitter conversion dataset and reddit conversion dataset @xcite . for both the datasets, the organizer provides the context and response that is the response is reply to the context and the context is a full dialogue conversation thread. the computational task is to detect and identify the sarcasm and to understand how much conversation context is needed or helpful for sarcasm detection. the challenges of this shared task include: a) small dataset is hard to train the complex models; b) the characteristics of the language on social media forums difficulties such as non-vocabulary words and ungrammatical context c) how much conversion text to detect sarcasm and the usage of slangs, rhetorical questions, capitalized words, numbers, abbreviations, pro-longed words, hashtags, url, repetitions of punctuations, contractions, continuous words without spaces. we address the problem in hash tags, continuation of words without spaces, url and to classify which context is helpful to find sarcasm. to address the problem, we pre-processed the text by using machine learning libraries like ntlk, gensim and classified by using different traditional machine learning techniques, deep learning technique and finally we obtained the ## data and methodology in our approach, we have used twitter and reddit dataset given by figurative language processing 2020 shared task on sarcasm detection. the dataset is given with columns namely, label, context and response where the response is the reply of context and the context is the full conversion dialogue and it is separated as c1, c2, c3 etc. c2 is the reply of the c1 context and c3 is the reply of c2 context respectively. both the datasets consists of the labels namely sarcasm and not_sarcasm. in the twitter dataset, the train data has 5000 conversion tweets in that 2500 sarcasm tweets and 2500 not sarcasm tweets and the test data has 1800 tweets. in the reddit dataset, the train data has 4400 conversion tweets in that 2200 sarcasm tweets and 2200 non sarcasm tweets and the test data have 1800 tweets. we have the pre-processed the text to removal of @user, url and the pro longed words like "ohhhhhh" and replace the words like f * * king as fucking, replace the question tags like didn't as did not, removal of hashtags and separate the words into the continuous space less sentence. tweet tokenizer is used to tokenize the word and to get the vocabulary words. we have employed the traditional machine learning techniques, recurrent neural network with lstm (rnn-lstm) and bert. in the figure 2: not sarcastic words we have performed doc2vec transformer and tfidf vectorizer for feature extraction and classified by using the logistic regression (lr), random forest classifier (rf), xgboost classifier (xgb), linear support vector machine (svc), gaussian naïve binomial (nb). by using tfidf vectorizer, we got the 28761 features for 5000 tweets. in twitter data, we have chosen the scores which are above 0.70 from the cross validation accuracies of the machine learning techniques. based on the cross validation scores, we have obtain the best accuracies score in svm, logistic regression and nb classifiers of the combined context text (cr) in tfidf vectorizer and the best accuracies score in logistic regression and gaussian nb models of the isolated response (r) text in tfidf vectorizer. in reddit data, we have chosen the scores which are above 0.55 from the cross validation accuracies of the machine learning techniques. based on the cross validation scores, we have obtain the best accuracies score in logistic regression and xgboost classifier of the combined text (cr) in tfidf vectorizer and the best accuracies score in logistic regression and gaussian nb models of the isolated response text (r) in tfidf vectorizer. in both the dataset, the result shows doc2vec transformer is not performed well because of non-grammatical sentences and tfidf vectorizer performs well when compared with the doc2vec transformer in dialogue conversion thread. in the rnn-lstm method, we have used the combined context text with response to perform the pre-process using nltk libraries, tokenize the word by using the word tokenizer and lemmatize the word after that to remove the stop words. finally, we have obtained the train data has 325382 words total, with a vocabulary size of 32756, max sentence length is 568 and the test data has 30782 words total, with a vocabulary size of 8824, max sentence length is 467. we used the word2vec embedding model for the embedding the words and obtain the 32668 unique tokens. we have evaluated using the rnn-lstm and trained the deep learning models with a batch size 128 and dropout 0.2 for 5 epochs to build the model. we got the accuracy is 0.4890 which is low when compared with the machine learning approach. in the bert model, google research team releases bert @xcite and achieve good performance on many nlp tasks. we have used the combined context text, isolated context, and isolated response to perform the model. we have used the bert uncased model for training the model, batch size is 32, learning rate is 2e-5, and number of train epochs is 3.0. warmup is a period of time where the learning rate is small and gradually increases usually helps training. warmup proportion is 0.1 and the model configuration is checkpoints is 300, summary steps is 100. we got the accuracy is 0.77 score. we have compared over all cross validation accuracies scores, bert performs good than the machine learning approaches and deep learning technique. ## results we have evaluated the test data of twitter and reddit dataset which is shared by figurative language processing 2020 shared task organizers. the performance is evaluated by using the metrics as precision, recall and f1 score. we have chosen the classifiers to predict the test data based on the performance of the cross validation of training data. we have performed to predict the test data by using various combinations of conversion context and response that are cr represents the combined context of sentences with response, c represents the combined full context of sentences without response, pcrw represents the processed combined context of meaningful words and response, pcw represents the combined full context of meaningful words without response, pc1rw represents the processed isolated first context of meaningful words and response, pc1w represents the isolated first context of meaningful words without response, r represents the response, pc1r represents the processed second context with response, pr represents the processed response. the results of the approaches are presented in the ## conclusion we have implemented traditional machine learning, deep learning approach and bert model for identifying the sarcasm from conversion dialogue thread and to detecting sarcasm from social media. the approaches are evaluated on figurative language 2020 dataset. the given utterance of combined text and isolated text are preprocessed and vectorized using word embeddings in deep learning models. we have employed rnn-lstm to build the model for both the datasets. the instances are vectorized using doc2vec and tfidf score for traditional machine learning models. the classifiers namely logistic regression (lr), random forest classifier (rf), xgboost classifier (xgb), linear support vector machine (svc), gaussian naïve binomial (nb) were employed to build the models for both the twitter and reddit datasets. bert uncased model with isolated response context gives better results for both the datasets respectively. the performance may be improved further by using larger datasets.