Utilizing Weak Supervision to Create S 3 D : A Sarcasm Annotated Dataset (2022)

Abstract: Sarcasm is prevalent in all corners of social media, posing many challenges within Natural Language Processing (NLP), particularly for sentiment analysis. Sarcasm detection remains a largely unsolved problem in many NLP tasks due to its contradictory and typically derogatory nature as a figurative language construct. With recent strides in NLP, many pre-trained language models exist that have been trained on data from specific social media platforms, i.e., Twitter. In this paper, we evaluate the efficacy of multiple sarcasm detection datasets using machine and deep learning models. We create two new datasets - a manually annotated gold standard Sarcasm Annotated Dataset (SAD) and a Silver-Standard Sarcasm-annotated Dataset (S3D). Using a combination of existing sarcasm datasets with SAD, we train a sarcasm detection model over a social-media domain pre-trained language model, BERTweet, which yields an F1-score of 78.29%. Using an Ensemble model with an underlying majority technique, we further label S3D to produce a weakly supervised dataset containing over 100,000 tweets. We publicly release all the code, our manually annotated and weakly supervised datasets, and fine-tuned models for further research.

## introduction figurative language, such as the use of metaphors, irony and sarcasm, is ubiquitous in human communication, from ancient religious texts to social media micro texts. the detection of sarcasm in human communication is a challenging task where the goal is to identify sarcastic utterances from the data provided. there is no one definitive definition of sarcasm due to its nature as a language construct relying on factors such as domain and context, even regional differences @xcite , but a widely accepted definition is "a form of verbal irony that is intended to express contempt or ridicule" @xcite . sarcasm has a diminishing effect on sentiment analysis due to sarcastic text often having the op-posite implied meaning to a literal word-for-word meaning of the text @xcite . for example, "i just love it when my flight gets delayed for 4 hours", is clearly sarcastic, as using the word "love" to express feelings on something rather inconvenient would be unusual outside of a sarcastic context. such challenges demonstrate the importance of recognising sarcasm in social media @xcite , as recognising the potential for a given text utterance to be sarcastic can bridge the gap in human-machine communication. the nlp research community has investigated the detection of sarcasm using various machine/deep learning approaches @xcite @xcite . several datasets exist for the task of sarcasm detection using text @xcite @xcite as well as multimodal datasets @xcite , which support the extraction of features from video and speech. transformer @xcite based language models have shown to perform very well for classification tasks in various nlp sub-areas, and a number of bert @xcite based language models have been released which can help perform this nlp task. in this paper, we attempt to collate these efforts for the task of sarcasm detection. we restrict our focus to the detection of sarcasm on a social media platform, i.e., twitter. initially, we curated our dataset (sad) by crawling for tweets and labelling them with the help of two annotators. we extensively evaluate machine and deep learningbased approaches on various existing datasets and our dataset. we apply standard pre-processing and combine all the datasets to evaluate several classification approaches. using an ensemble of the best language models trained over the largest datasets, we further label 100k tweets to create silver-standard sarcasm-annotated dataset (s3d). the key contributions of our work are as follows: 1) a sarcasm-annotated dataset (sad) of social media microblogs, 2) performance evaluation of various existing language models for the binary classification task of sarcasm detection, 3) curation and weak-supervision-based labelling for a silver-standard sarcasm-annotated dataset (s3d), 4) release of code, data, and models created on github, and huggingface platforms, publicly, for the research community 1 . this paper is organised as follows. section 2 briefly describes previous approaches to sarcasm detection. section 3 describes our chosen datasets and their sources. section 4 explains the methodology behind the proposed experiments, summarising the approaches for our machine learning and deep learning experiments. section 5 discusses choices made for running our experiments, section 6 discusses the results of these experiments in detail, along with the approach used to obtain a new weakly supervised dataset. ## related work transformer-based approaches have increased in prevalence within nlp and also within sarcasm detection literature. this is most notably due to their ability to accurately pick up semantic and syntactic relationships within a piece of text. @xcite discuss various approaches to the task of sarcasm detection including rule-based and machine learning-based, and also discusses sarcasm from the linguistics perspective. shangipour ataei et al. (2020) dicusses several approaches to perform sarcasm detection. these include a bert @xcite model with no concatenated layers, bert encodings with a logisitic regression model, and other language models such as ian @xcite which are trained and evaluated on a twitter-based sarcasm dataset. in these experiments, the bert language model with no added layers performs the best on the dataset, achieving an f1-score of 73.4. some existing literature investigates methods for performing sarcasm detection in arabic @xcite , where a multitude of transformers are used, including mbert, xlm-roberta @xcite and language-specific models like mar-bert @xcite . the best model in this research achieves an f1-score of 58.4 in a low-resource scenario. in @xcite , an rcnn-roberta methodology was proposed, where a roberta transformer was utilized 1 https://github.com/surrey-nlp/s3d with bilstm to improve upon f1-scores from state-of-the-art neural network classifiers on the dataset released with the semeval 2018 shared task 3 @xcite . this paper also reports that the rcvv-roberta approach achieved an f1-score of 90.0 on the riloff dataset @xcite . @xcite demonstrate a variety of results on a twitter dataset, training a collection of architectures involving convolution neural network (cnn) and long-short term memory (lstm) to achieve an impressive f1-score of 92.1 with their best configuration. an ensemble approach was demonstrated in @xcite where a weighted average ensemble of a cnn, an lstm and a gated recurrent unit (gru) based architectures are trained with glove @xcite word embeddings in order to identify sarcasm, showing that the ensemble outperformed others by up to 8% on sarc @xcite , a reddit comments dataset. machine learning approaches have decreased in popularity due to the improvements shown by transformers-based architectures in recent developments. earlier approaches to sarcasm detection include @xcite and @xcite that used a naive bayes and decision tree model, respectively, in order to identify sarcasm where both achieve the best f1-scores over 70 on their chosen datasets. to curate sarcasm-annotated datasets, one can perform manual annotation, which involves a significant cost in terms of time and money. moreover, manual annotations for subjective linguistic constructs like sarcasm are questionable unless multiple annotators label the data, and an almost perfect inter-annotator agreement can be seen within the labelling. an example of this approach is the creation of the riloff dataset @xcite . on the other hand, sarcasm research has also utilised 'selfannotated tags' from social media forums, such as '#sarcasm' from tweets and '/s' in reddit comments. such data collection methods can be automated, and a large amount of data can easily be collected. however, the quality of such datasets in terms of label accuracy can be questioned. selfannotation was used in the creation of the ptacek dataset @xcite from english tweets, and the creation of the sarc dataset @xcite from reddit comments. however, we follow a hybrid approach as we collect sad using '#sarcasm' from twitter and then manually label it. a limitation of publicly available datasets based on tweet ids, e.g., @xcite is that the tweet data retrieval based on the ids can diminish over time. if a significant number of tweets are deleted, then it would not be possible to reproduce the results on the original dataset. in e.g., @xcite , the number of tweets, at the time of writing the paper, that can be retrieved related to the ids in the dataset is 710 compared to the original 3000 data instances. the contribution of our weak supervision-based approach is to help produce labelled data, the benefit of which could be to augment existing datasets that have diminished over time with automatically labelled data or also to create new silver standard datasets. ## datasets we test our proposed approach for sarcasm detection on a total of six datasets, summarised in ## methodology for our machine learning experiments we use dt @xcite and lr (cox, 1958) models. our approaches to vectorising text for feature extraction utilise word2vec @xcite and glove @xcite . word2vec is a model architecture for computing vector representations of words from text, as is glove, which has an additional focus on latent semantic analysis. for our deep learning based experiments, a total of five pre-trained language models were used: bert @xcite , roberta base & roberta large @xcite , twitter-roberta @xcite and @xcite . bert was introduced as a state-of-the-art transformer that improved results on multiple benchmarked nlp tasks. the language model was demonstrated as being able to be fine-tuned to create models for a wide range of tasks including question inference and next sentence prediction. roberta was built on bert through modifying key hyper-parameters and removing the nextsentence-prediction pre-training objective, on top of training with much larger batches and learning rates. the roberta large configuration follows the same architecture but contains more hidden units and twice the number of encoder layers. twitter-roberta was introduced as rob-rt by @xcite and is a roberta base model that was trained on a total of 60m tweets, consisting of 584 million individual tokens. bertweet has the same architecture of bert-base and is trained on an 80gb corpus of 850m english tweets. each of these models was fine-tuned for the purpose of sarcasm detection. the fine-tuning process comprises adding a dropout layer on top of the pre-trained model, followed by a fully connected layer which was then fed into a final layer using a softmax activation function for classification. ## experiment setup as discussed in section 4, the experiments have been split into the two categories of machine ## results and discussion comment word2vec+ dt label glove+ lr label ground truth 'didnt realize @user referees were so fluent in russian' 1 1 1 'well hello depression nice to see ya again didnt think youd stay away' much longer 0 1 1 'dont you just love the hip hop music and club music they played in the background of the @user movie i do' 0 0 1 the semeval dataset achieves its highest f1score of 61.62 using the word2vec+lr model. the riloff dataset has the weakest set of f1-scores across each approach, with it is best f1-score (35.29) still being lower than any f1-score for any other dataset. interestingly, the word2vec+lr model achieves a perfect precision score, whereas the associated scores for this model are the lowest for all experiments. from from there is more success with the unbalanced riloff dataset within the deep learning experiments as opposed to the machine learning experiments. the lowest f1-score using the riloff dataset in our sad dataset has high f1-scores across each model, 87.06 being the highest achieved by the roberta base language model. the bert language model achieves the weakest f1-score on the dataset (68.45), followed closely by the bertweet model (68.91). this was unexpected as the bertweet language model was pre-trained only on tweets. further unexpectedly, the roberta base model actually achieves the best overall f1-score on the sad dataset, despite the model not being pre-trained on any tweets at all. this performance may be attributed to the significantly larger dataset used for training the roberta model. ironically, despite being pre-trained solely on 850m tweets, the bertweet model achieves the highest f1-score of 78.39 on the sarc dataset, the only dataset that does not include any tweets. from comment bertweet label ground truth 'more fragmentation is exactly what we need in mobile payments' 1 1 'hockey wouldnt work in quebec city' 1 1 'this is new and interesting' 1 1 'i call them suckers' 1 0 'by doing the same thing i do every night and day nothing' 0 1 'huge moves were making gonna take this league by storm' 0 1 there are entries where the model incorrectly labels sarcastic text extracts. in the fourth row, an instance of a false positive can be seen, where our pre-trained model incorrectly determines a tweet is sarcastic when it was not labelled as such. the word "suckers" might indicate some humorous intent to the text, implying sarcasm may be used in the comment. the last two entries in although the model made the correct prediction in the rather specific domain of "quebec" and "hockey", it makes an incorrect prediction in this broader context. this is demonstrable of how figurative language and the understanding of such truly rely on contextual differences. these contextual differences impact human, and, particularly, machine understanding of sarcasm. again, this struggle of the models' prediction capabilities in a broader context is seen in the final entry, where the user has intended the text to be sarcastic, but it has not been labelled by our bertweet model as such. even with this small scope of examples where our model has made incorrect predictions, our fine-tuned bertweet model is still our highestscoring language model on our largest datasets, and thus we will use fine-tuned bertweet models for the purpose of labelling a weakly supervised dataset. ## conclusion and future work in this paper, we utilise several existing machineand deep learning-based approaches to perform the task of sarcasm detection over various datasets. from a social media platform, we curate and manually label a sarcasm dataset and benchmark its efficacy with these approaches. we also perform an exhaustive evaluation with the help of pre-trained language models, including some models specifically trained using social media data. using an ensemble model based on multiple fine-tuned bertweet models, we labelled an additional 100, 000 tweets and release this silver-standard sarcasm annotated corpus, called s3d. we also perform a fine-tuning experiment after concatenating s3d with the 'combined' dataset and achieve the best f1-score of 78.87 over the large datasets discussed in this paper. by contributing a weak supervision-based approach, we facilitate the automatic production of labelled data that can be used to augment existing datasets or create new silver standard datasets. we also release the code, the manually labelled dataset, and models created with our experiments publicly for further research. in future, we would like to perform a more finegrained annotation for sarcasm with sub-categories as defined in existing linguistic literature. we also aim to perform similar experiments for multimodal sarcasm detection in order to contribute further resources to the community.