C -Net: Contextual Network for Sarcasm Detection (2020)

Abstract: Automatic Sarcasm Detection in conversations is a difficult and tricky task. Classifying an utterance as sarcastic or not in isolation can be futile since most of the time the sarcastic nature of a sentence heavily relies on its context. This paper presents our proposed model, C-Net, which takes contextual information of a sentence in a sequential manner to classify it as sarcastic or non-sarcastic. Our model showcases competitive performance in the Sarcasm Detection shared task organised on CodaLab and achieved 75.0% F1-score on the Twitter dataset and 66.3% F1-score on Reddit dataset.

## introduction sarcasm detection plays a crucial role in improving the effectiveness of chatbot systems. sentiment classification systems can fail in the absence of a robust sarcasm detection system. a sarcastic sentence can express a negative sentiment even with the presence of positive or neutral sentiment words in that sentence. hence, accurate detection of sarcasm can take an artificially intelligent agent closer to imitate human behaviour and enable it to better understand true intentions and emotions of humans @xcite . this paper represents work on the sarcasm detection shared task which is a part of the second workshop on figurative language processing, colocated with acl 2020. the shared task aims to investigate and understand how much conversation context is needed or helpful for sarcasm detection. two datasets, one of reddit and the other of twitter, were provided for developing and testing multiple sarcasm detection systems. in this paper, we present our study on the effectiveness of contextual information to decide if an utterance is sarcastic or not. for this, the baseline models were first created using traditional machine learning algorithms like logistic regression, svm etc. which were trained to classify utterances without considering their contextual information. sequence models like vanilla rnn and lstm were trained similarly. then different types of word embeddings (elmo and glove) and sentence embedding (deepmoji) to capture emotional states in the sentences were also experimented to detect incongruities within the text. the latest state-of-the-art transformer based models like bert, xlnet and roberta were also used for classifying sentences in isolation. our investigations for creating systems which can use the context information effectively in a sequential manner led to the creation of our proposed model, which showed decent performances in both the test datasets. ## related works the evolution of various trends in sarcasm detection research can be seen in @xcite . sarcasm detection was initially performed using rule-based approaches. @xcite presented rule-based classifiers that look for a positive verb and a negative situation phrase in a sentence. @xcite proposed using hashtag sentiment as an indicator for sarcasm and @xcite introduced pos sequences and semantic imbalance as features. statistical featurebased approaches were also used for this task e.g. @xcite introduced features related to ambiguity, unexpectedness, emotional scenario, etc. to better capture situational dependencies for the presence of sarcasm. machine learning algorithms were also used for sarcasm detection. majority of work in sarcasm detection earlier relied on svm ( @xcite ; @xcite ; @xcite 2007); @xcite ; @xcite ). the importance of context information was first presented in @xcite which described their annotation study where annotators repeatedly asked for context information to judge a text to be sarcastic or not. many times they changed their previously given labels to a text after being shown the context behind it. @xcite and @xcite tried to include the author context by analysing the author's past tweets and sentiments. to consider the conversational context, @xcite and @xcite used a sequence labeling approach. @xcite also tried to use the topical context of a text, since some topics are more likely to generate sarcasm as compared to other topics. recent works in this domain include deep learning methods such as @xcite included several types of long short-term memory (lstm) networks that can model both, the conversation context and the response. @xcite used cnns to incorporate various contextual information. @xcite used pre-trained roberta weights combined with an rcnn to capture contextual information to detect sarcasm. ## data the dataset used for this study was provided in the shared task on sarcasm detection, organized at codalab. it included two separate datasets, twitter and reddit, each of them having equal no. of sarcastic and non-sarcastic responses. for each response provided in the dataset, the conversation context consists of an ordered list of previous dialogues. ## methods we used three kinds of approaches to experiment in this task. first, methods that classified utterances in isolation were investigated. then, approaches that considered partial conversation context for classifying texts were experimented. finally, methods that can potentially utilise the complete conversation context information were looked into. ## reproducibility to reproduce the best results mentioned in this paper, c-net with ses should be used, utilising the two latest context sentences associated with each response. the pre-trained bert (base-uncased) model, provided in the transformers library by huggingface @xcite , was trained similarly as mentioned in the implementation section using the fastai library @xcite . while fine-tuning bert on response sentences, the learning rates used were ranging between 1e-5 to 1e-4. but while training on context sentences, the learning rates used were ranging between 1e-6 to 1e-5. the optimum parameter α for ses was found out from the training set with a grid search. in our experiments, it was found out that the value 0.395 for α best fits the twitter training data and 0.2 best fits the reddit training data. ## results from the results in the case of c-net for both the fusion methods (lr and ses) are better as compared to the results of the bert classifier trained only on response in the twitter data. however, this is not true for the reddit dataset. the bert and roberta model trained only on response sentences in the reddit dataset performed better as compared to the c-net approach. this is counterintuitive as per the theory that context information helps in sarcasm detection. however, it's possible that the reddit response-only dataset contains many flags for sarcasm, which are also present in the large dataset the models were pre-trained with. further pre-training of the models on the target reddit and twitter dataset may further improve the results. ## conclusion in this paper, we compared the performances of various approaches for the sarcasm detection task. we experimented with traditional machine learning based approaches, and the latest state-of-theart transformer architectures. the results obtained show that our proposed model, c-net, has the potential to effectively use the conversation context of an utterance to capture the sarcastic nature of a conversation. the variable number of context sentences for each response sentence makes it difficult to capture the long range dependency. hence, as future work, approaches that can effectively deal with variable context set size can be investigated.