A mrita_ CEN at S em E val-2022 Task 6: A Machine Learning Approach for Detecting Intended Sarcasm using Oversampling (2022)

Abstract: This paper describes the submission of the team Amrita_CEN to the shared task on iSarcasm Eval: Intended Sarcasm Detection in English and Arabic at SemEval 2022. We employed machine learning algorithms towards sarcasm detection. Here, we used K-Nearest Neighbor (KNN), Support Vector Machine (SVM), Naïve Bayes, Logistic Regression, and Decision Tree along with the Random Forest ensemble method. Additionally, feature engineering techniques were applied to deal with the problems of class imbalance during training. Among the models considered, our study shows that the SVM, logistic regression and ensemble model Random Forest exhibited the best performance, which was submitted to the shared task.

## introduction sarcasm is an ironic form showing a disparity between the actual and intended meaning of the text affecting the decision-making process. these are reflected in our day-to-day communication with each other happening in social media forums. twitter exhibits rich sarcasm phenomena, thereby encouraging automatic sarcasm detection methods and removing such tweet data. due to the sociocultural aspects of sarcastic communication, the majority of the sarcasm detection work has been focused only on the english language @xcite , and only a limited amount of work was done in other languages such as arabic @xcite . identification of sarcastic comments from social media contexts is essential since the author and the receiver are at various places. therefore, exchanging conversations may sometimes lead to a negative meaning of the text that even the author has not meant to convey. moreover, the data stream for sarcasm does not exhibit any static structure like specific tags in the form of #sarcasm, and #irony @xcite . this event can lead to noisy labels due to several reasons, as outlined by @xcite . other works reported on the topic mainly depend on manual labelling, provided with manually annotated sarcasm labels. in @xcite the authors pointed out that manual labelling represents the author annotation in contrast with the intention of the authors. the sarcasm prediction on twitter that influences machine intelligence is a challenging task @xcite . it can be achieved with the help of the natural language processing (nlp) approach, and many recent works on automatic sarcasm detection have focused on twitter data as it primarily requires an understanding of the human expressions, language, and emotions expressed via textual or non-textual content @xcite . therefore, the goal of the semeval shared task is to facilitate the development of machine learning models that can detect sarcasm from tweets. the shared task consists of two subtasks: in this paper, we describe the machine learning models designed for solving the problems given in isarcasm shared tasks @xcite . the performance of the models was evaluated using the f1-score. the models submitted achieved the following scores: 0.4966 in english, 0.6127 in arabic and 0.0567 f1-score in subtasks a and b, respectively. ## literature review the majority of the published works developed for the text sarcasm detection used datasets that were annotated using a weak supervision method, where the texts were regarded as sarcastic only if they met preset criteria, including specific tags like sarcasm and irony @xcite @xcite . in @xcite , s.v opera and w magdy reported that labelling using a weak supervision method could lead to noisy labels. other works on this topic were based on manual labelling, where the human annotators are given the role of labelling the texts @xcite @xcite . the disadvantage of such a labelling procedure is that it represents the perception of the annotator, which may differ from the author's intention @xcite . in addition to the above-mentioned method, a significant majority of works on sarcasm detection were centered exclusively on the english language (oprea and magdy, 2019) (campbell and katz, 2012) @xcite @xcite @xcite . it is because of its sociocultural aspects on sarcastic communication @xcite , leading to the uncertainty that, the models trained on english could generalize to other languages. all the reported works on sarcasm detection in other languages such as arabic @xcite @xcite (farha and magdy, 2020) were relied on the afore-mentioned labelling techniques. ## dataset and task description the dataset comprises tweets in english and arabic. there are two subtasks in english and one in arabic. tweets in the english dataset were categorized into two: sarcastic and non-sarcastic. it contains 3,467 instances of tweets and ten columns containing the attributes (id, tweet, sarcastic, rephrase, sarcasm, irony, satire, understatement, overstatement, rhetorical question). the objective of task-1 is to determine whether a given text is sarcastic or not. task-2 is a multi-label classification that aims to classify a tweet into different ironic speech categories, such as sarcasm, irony, satire, understatement, overstatement, and rhetorical questions. the shared task-1 in arabic focused on categorizing a tweet into sarcastic or non-sarcastic, similar to task-1 in english. the arabic dataset contains 2,601 instances of tweets and five attributes (id, tweet, sarcastic, rephrase, dialect). ## system overview this section discusses the overview of the models submitted to the shared tasks. the flow of the model building is illustrated in figure 1 . ## experimental setup we implemented the models using python version 3. the training data is split into train and validation sets for confirming the best performing model. in the arabic sarcasm identification model using the svm classifier (subtask 1), we used a range of gamma values (0.1, 1, 10, 100) and c regularization parameter values (0.1, 1, 10, 100) and changed the kernel type to rbf, linear and polynomial to see how the accuracy and f1-score vary. in random forest classifier different, n_estimators value (10, 100, 1000) and the maximum features are given to sqrt, log2 to see the changes @xcite . the english tweet irony detection model (subtask 2) is a multi-class classification problem and implemented using a multioutput classifier set to multilabel. the model performance was analyzed using macro f1-score obtained using the sklearn metrics along with the accuracy, precision and recall @xcite of the trained model. ## conclusion this paper presents the submission of amrita_cen towards the semeval 2022 task 6 competition named " isarcasmeval -intended sarcasm detection in english and arabic ". a total of six machine learning algorithms were used, including five