FII UAIC at S em E val-2022 Task 6: i S arcasm E val - Intended Sarcasm Detection in E nglish and A rabic (2022)

Abstract: The “iSarcasmEval - Intended Sarcasm Detection in English and Arabic” task at the SemEval 2022 competition focuses on detectingand rating the distinction between intendedand perceived sarcasm in the context of textual sarcasm detection, as well as the level ofirony contained in these texts. In the contextof SemEval, we present a binary classificationmethod which classifies the text as sarcasticor non-sarcastic (task A, for English) based onfive classical machine learning approaches bytrying to train the models based on this datasetsolely (i.e., no other datasets have been used).This process indicates low performance compared to previously studied datasets, which in2dicates that the previous ones might be biased.

## introduction one of the most challenging tasks facing natural language processing (nlp) is the automatic figurative language detection (gifu daniela, samson mihai, 2021), (c. van hee, e. lefever, and v. hoste, 2018), such as humor, sarcasm or irony. in general, this way of expressing takes advantage of linguistic elements in order to project complex and explainable meanings. in this paper, we investigate the binary classification models for figurative language, as is sarcasm (dan @xcite . in general, it is omnipresent on the public space, being disruptive of computational systems that harness this data to perform tasks such as opinion mining and sentiment analysis in elections @xcite . the political actors themselves introduce a specific language based on sarcasm or irony, making their message analysis very challenging @xcite . in order to identify the figurative meaning of a specific message, it is required to encode each sentence separately. the research question guiding this paper is what are the most efficient sarcasm detection algorithms? we propose an approach based on three classical machine learning (ml) approaches by trying to train the models based on this dataset solely (i.e., no other datasets have been used). furthermore, we experimented with architectures ranging from naïve bayes (multinomial, complement and bernoulli variants), support vector machines (svms with linear, polynomial and rbf kernels) to logistic regression which we tried to train using only the dataset provided by the semeval-2022 task 6 competition (ibrahim abu farha, @xcite . the rest of the paper is organized as follows: section 2 briefly presents studies related to sarcasm detection, section 3 presents the dataset, the required pre-processing and plausible methods for it, section 4 resumes the results of the conducted experiments, with their interpretations, followed by section 5 with the conclusions. ## related work this topic is a widely researched subject in recent years, evidenced in this competition at several workshops (e.g., semeval-2017 task 4: using sarcasm detection for enhancing sentiment classification and quantification or semeval-2018 task 3: irony detection in english tweets). such a competition is challenging, especially since the problem of labeled data is time consuming and not cheap. moreover, the automatic sarcasm detection depends on the annotation process, which always introduce some biases to the data. for the binary task, as in this case, there are many computational models to solve it or to capture the (actual) sarcasm or subcategories of it (john s. @xcite (silviu oprea and walid magdy, 2020). thus, work on this topic was never followed by high results, as this problem is still debatable and text classification even for humans is very controversial and biased. it is a task that can be considered as sentiment analysis for which most of the authors used lstm (dzmitry @xcite , @xcite or cnn (yi tay, anh tuan luu, siu cheung hui, and jian su, 2018) and (tao shen, tianyi zhou, guodong long, jing jiang, shirui pan, and chengqi zhang, 2018). new approaches concentrate on using attention-based methods @xcite , in particular transformer architectures such as bert @xcite , roberta, spanbert (amardeep kumar, vivek anand, 2020) etc. these transformers are pretrained on unlabeled data to be later fine-tuned for a variety of tasks like single sentence classification, sentence pair classification, etc. to understand role of context for sarcasm detection. here, we used five machine learning approaches by trying to train the models based on this dataset solely (i.e., no other datasets have been used). ## dataset and methods this section is focused on two issues: twitter dataset in english and two types of methods for classification task, both binary and multi-class. for the first type, we mostly tried classical machine learning approaches that resulted in satisfactory outcomes when evaluated on the training set. on the second one, a rnn with a few gru layers was the architectural choice. at this moment, the results were modest, probably due to the lack of to much data (it is empirically known that neural networks require a high volume of data to work properly) ## results and interpretations we would like to make it clear that the results have been obtained on the training set here, since we didn't have a test set available at the time we were working on the project. moreover, the results should be viewed with caution since the models have been trained on a small sized data set that has not to many samples, hence not to many words. a big problem arises: what do we do when the future testing samples don't contain words we've seen before? to answer this, one could simply drop the unseen words and focus on those seen only. moving on, the main metrics we have used throughout the project are: recall, precision, f-score, balanced accuracy, accuracy and the roc graph. it's worth noticing that the metrics have been evaluated on all the numerical representations we have mentioned before. we will start with the naïve bayes algorithms since these were the ones that surprisingly produced the best results ( moving on to logistic regression, we can see that the results are really far away from nb. therefore, we can definitely say that the naïve bayes independence supposition here is strongly untrue. we can see that the recall on the positive labels is very small, but the precision is almost close to 100%. in other words, among all the texts that are positive in reality, only a small percentage of them are classified as positive, but those that are predicted positive, are done so with very high certainty. so, we could say that someone who is interested in getting a very precised sarcastic prediction should choose this model. also, an interesting observation that we can draw from all the analysed algorithm results is that a high recall on the positive class leads to a high precision on the negative class and alternatively a high precision on the positive class leads to a high recall on the negative class ( the support vector machines are the ones that behaved the most poorly among the tried models. this is somehow weird since most of the articles out there indicate svms as the best option for binary classification in nlp when having little data. as mentioned before, we tried different kernel variants, but nothing good came up, unfortunately. the polynomial kernel seems to have given the best results, but these are weak. ## conclusion for sarcasm classification task (binary or multiclass), a small dataset is a really challenging. the classical machine learning approaches we have presented so far cannot answer convenient to it. this lack of data is a big downside because it means that not many words are numerically encoded, hence having the same vocabulary on a testing set is absolutely mandatory. in practice, it is very rarely. we may conclude that the best model is the multinomial naïve bayes one, but again, it is not guaranteed that it has not overfitted the training data. in the future, some more attention could be payed to the neural network if data imputation is done and the dataset reaches a reasonable number of samples.