reamtchka at S em E val-2022 Task 6: Investigating the effect of different loss functions for Sarcasm detection for unbalanced datasets (2022)

Abstract: This paper describes the system used in SemEval-2022 Task 6: Intended Sarcasm Detection in English and Arabic. Achieving 20th,3rd places with 34& 47 F1-Sarcastic score for task A, 16th place for task B with 0.0560 F1-macro score, and 10, 6th places for task C with72% and 80% accuracy on the leaderboard. A voting classifier between either multiple different BERT-based models or machine learningmodels is proposed, as our final model. Multiple key points has been extensively examined to overcome the problem of the unbalance ofthe dataset as: type of models, suitable architecture, augmentation, loss function, etc. In addition to that, we present an analysis of ourresults in this work, highlighting its strengths and shortcomings.

## introduction sarcasm detection in any language text is hard for many reasons. first sarcasm is much more than just written words. it is the tone, emphasis, experience, personal knowledge, and even facial expressions and body language that convey the meaning. when written into text such information is lost completely. another reason is cultural references which make it hard for non-natives to get. in addition, all that is needed to show sarcasm is to be clear and articulate and not be as sketchy and without formatting, punctuation, proper use of language as texting might be for some people in their usual use of such media. sarcasm is widespread on the social web and, by definition, may be extremely disruptive to machine learning/ deep learning models that use this data to perform tasks such as sentiment analysis, opinion mining, author profiling, and harassment identification. thus, sarcasm detection might be the first crucial step in these systems. several methods have been proposed to detect sarcasm in text as the recent shared-task for sarcasm detection in arabic language @xcite , where several teams used pretrained language models such as arabert and marbert @xcite @xcite @xcite . in some of the work proposed in english language sarcasm detection involved rule-based and statistical approaches using: (a) unigrams and pragmatic features (for example emoticons, etc.) (b) sentiment and polarity estimation (c) extraction of common patterns @xcite @xcite . a recent shared-task for english language @xcite , where several methods has been proposed were based on pretrained model (i.e. bert and roberta) @xcite @xcite . in @xcite , the goal is to determine if sarcasm is presented in text or not, based on the text that is manually labeled by the task organizers. the shared task consists of three subtasks: the main contributions of our work are as follows: ## system overview in this section, we discuss our prepossessing techniques, different hand-crafted features. we further explore different training techniques using bert @xcite , roberta @xcite , berttweet @xcite , deberta @xcite , xlnet @xcite , and canine @xcite models for english language, and arabert (antoun et al., 2020), marbert @xcite and qarib @xcite models for arabic language. moreover we investigate machine learning based models. ## experimental setup experiments were conducted via python and py-torch framework, running on google colab resources, which are nvidia tesla p100-pcie-16gb gpu, intel ® xeon ® cpu @ 2.20 ghz, and 12gb ram. we used 70%-10%-20% strategy for train-validation-test splits respectively for the training phase and 5-fold cross-validation during training. all of the presented models (submitted to the leaderboard) were trained on the provided dataset for the shared task only with no augmentation and the same splitting criteria. if augmentation is used in the dev-phase/test-phase, the following types of augmentation were used: 1) spelling augmentation and contextual word embedding augmentation for insertion and substitution using bert and roberta. 2) synonym augmentation and contextual word embedding for sentence augmentation. precision, recall, f-score (f1-sarcastic for subtask a, and macro average f1-score for subtask b), and accuracy (for subtask c) were used as evaluation metrics. all bert-based models were trained using an adamw optimizer with an initial learning rate of 0.001, weight decay rate of 0.0000001, and cosine annealing learning rate scheduler with minimum learning rate values of 0.0000001 and maximum temperature of 500. all the models were fine-tuned for 3-5 epochs. pre-processing wasn't conducted on the dataset. during inferences, an ensemble of the 5 models is used (due to 5-fold validation)and is referred to as 1 model during our discussion. ## results in this section, we discuss our main results during the development and evaluation phases. in addition to that failed cases analysis is discussed. ## conclusion in this paper, we have discussed our system submitted to the semeval-2022 task 6. our model ranked 20th out of 43 participating teams in subtask a english language, 3rd out of 32 participating teams in subtask a arabic language, 16th out of 22 participating teams in subtask b, 10th out of 16 teams in subtask c english language, and 6th out of 12 participating teams in subtask c arabic language. we proposed a voting classifier that leverages fine-tuned, per-trained models. we proposed the usage of the different loss functions in each task to accommodate dataset imbalance and improve model training. overall we showed the power of the loss function to improve model performance without the need to access any external dataset or to use any kind of augmentation. we also investigated the main common error, which disturbs model performance. in future efforts, we plan to further improve our model to better handle data-imbalance constraints and world knowledge needed to improve model performance.