stce at S em E val-2022 Task 6: Sarcasm Detection in E nglish Tweets (2022)

Abstract: This paper describes the systematic approach applied in “SemEval-2022 Task 6 (iSarcasmEval) : Intended Sarcasm Detection in English and Arabic”. In particular, we illustrate the proposed system in detail for SubTask-A about determining a given text as sarcastic or non-sarcastic in English. We start with the training data from the officially released data and then experiment with different combinations of public datasets to improve the model generalization. Additional experiments conducted on the task demonstrate our strategies are effective in completing the task. Different transformer-based language models, as well as some popular plug-and-play proirs, are mixed into our system to enhance the model’s robustness. Furthermore, statistical and lexical-based text features are mined to improve the accuracy of the sarcasm detection. Our final submission achieves an F1-score for the sarcastic class of 0.6052 on the official test set (the top 1 of the 43 teams in “SubTask-A-English” on the leaderboard).

## introduction sarcasm is a sophisticated communication technique to express emotions, attitudes, feelings, and evaluations. sarcastic and ironic texts typically do not contain words with negative polarity, hostile attitudes, or offensive in their literal sense, but rather express the contradiction or opposite of the literal meanings @xcite @xcite . sarcasm detection can be considered a particular sentiment analysis task, applied to detect texts that are intended to use some exaggeration, understatement, or rhetoric content to express criticism or praise for people or events. many researchers have conducted different deep learning methods @xcite @xcite , traditional machine learning method @xcite @xcite , and big data approaches @xcite @xcite to improve the accuracy of irony or sarcasm autodetection. semeval-2022 task 6 (isarcasmeval) is a sarcasm detection task @xcite . the standard training dataset includes 3468 english tweets and 3102 arabic tweets. the english training dataset provides 862 sarcastic tweets along with their non-sarcastic rephrases, while the arabic datasets provides 745 sarcastic samples. for the english dataset, each sarcastic tweet is also labeled as a fine-grained multi-class and multi-label ironic tag such as satire, understatement, overstatement, and rhetorical questions. subtask-a is a binary classification task to predict whether a given tweet is sarcastic or not. in this paper, we demonstrate the following contributions: 1) the discrepancy in prediction performance using different transformer-based language models; 2) the improvement of adding the public dataset and mining effective text features; 3) the enhancement obtained by incorporating various constrative learning loss functions; 4) model generalization is improved by incorporating the multi-sample dropout layer into the output of pretrained language models. on subtask-a, our system achieves an f1 score for the sarcasm category of 0.6052 and a macro f1 score of 0.7675. ## system overview the final submitted result is a contribution from various classification models using the voting mech- sarcastic tweet @pftompkins her family should definitely not seek mental health guidance. sarcastic tweet rephrase they should seek guidance. non-sarcastic example i wonder if it's too late for me to re-enroll in university and relive it all just one last time. the cross-entropy loss (l xent ) is used as the classification loss, and the additional xnet loss (l n t xent ) with a temperature of 0.2 is selected as the metric learning loss in our system. equation 1 shows the combination of two types of loss. the weight parameter (w) used to balance the combination of multiple losses is set as 0.1. additionally, three external datasets are added to the official semeval-2022 data in the proposed models for further training. text features are directly concatenated to the training texts in some proposed models as well. the featuring mining, text preprocessing, and the voting method are described below. the scheme of data preparation, training, and prediction processes is demonstrated in figure 1 . ## experimental setup four major improvements in f1 score are given by adding the public dataset, multi-sample dropout layer, text features, and tuning the parameters in the contrastive loss function. in this paper, we evaluate different modules and tricks that are adopted in our proposed model to show their effect on the semeval-2022 official dataset. the deberta-v3-large model outperforms about 5% the other two pretrained models we used in this task. many other tasks show the outperformance of the deberta model as well. de-berta modes proposed two novel tricks to improve the ability to solve many natural language tasks. compared with the bert and roberta models, the disentangled attention mechanism is applied to show each word in two vectors, which represent its content and relative position. the disentangled matrices are used to calculate the attention weights between the word content and position. an enhanced mask decoder is adopted to help with model pre-training by using the absolute position to predict the masked tokens. the deberta model also shows a big improvement in how well the model generalizes when the virtual adversarial training method is used. multi-sample dropout is a regularization technique which can accelerate training convergence and improve the model generalization compared to the network structure with a traditional dropout layer @xcite . four dropout layers were applied to the pooling layer output from the pretrained model. moreover, many effective adversarial training and vitual adversarial training methods can improve the model robustness and the regularization on classification tasks @xcite @xcite @xcite @xcite . in this task, we adopted projected gradient descent (pgd) and the fast gradient method (fgm) to implement the perturbation on sequence embedding in this task @xcite . additionally, the constrastive loss is considered in our training progress. it maximizes the amount of agreement between different augmented views of the same dataset through adding a contrastive loss in the latent space. @xcite @xcite . many tasks are competitively performed by adding contrastive loss functions, such as triple margin loss, npair loss, infonce loss, and supcon loss @xcite @xcite @xcite . in this task, we considered the ntxent and supcon losses as the additional contrastive loss. we added the selected contrastive loss to the cross-entropy loss to improve the classification accuracy. the effect of the contrastive temperature reflects the attention of difficult samples. the smaller temperature pays more attention to the separation of the sample from the most similar one to it @xcite . we tuned the contrastive temperature in both ntxent and supcon loss to train different classification models. ntxent with a contrastive temperature of 0.2 creates the best performance on the competition blind test set according to furthermore, the weighted voting on multiple models with different pretrained models and random seeds improves final performance on the competition's blind test set. figure 3 shows the minimum, mean, and maximum f1, precision and recall scores for predictions from 15 single models. the mixed voting method we applied for the final sub- dataset pretrained model adversarial training method text features multisample dropout contrastive loss/temp f1 @ sarcasm macro f1 pgd false false ntxent/0.5 0.4125 0.6402 pgd false true ntxent/0.5 0.4582 0.6794 fgm false true ntxent/0.5 0.4691 0.6788 fgm false true supcon/0.5 0.4788 0.6802 fgm false true supcon/0.2 0.4813 0.6833 fgm true true ntxent/0.2 0.4862 0.6957 deberta-v3-large fgm false true ntxent/0.2 0.5370 0.7132 xlmrobertalarge fgm false true ntxent/0.2 0.3871 0.6420 robertalarge fgm false true ntxent/0.2 0.5570 0.7374 deberta-v3-large fgm false true ntxent/0.2 0.5882 0.7445 xlmrobertalarge fgm false true ntxent/0.2 0.5615 0.7409 xlmrobertalarge fgm true true ntxent/0.2 0.6029 0.7676 semeval2022(en) robertalarge semeval2022 (en+ar) semeval2022(en) + semeval2018 + isarcasm semeval2022(en) + semeval2018 + isarcasm + multi-model semeval2022 (en+ar) + semeval2018 + isarcasm + multi-model ## conclusion according to the performance on the blind test set, the proposed model with the highest f1 score in the sarcastic category applied four layers of multisample dropout with a rate of 0.4 following the pooling layer outputting from the xlm-robertalarge model. the model is trained in the fast gradient method using the adamw optimizer at a learning rate of 1e-05. the combination of the crossentropy and the ntx contrastive loss is applied during the training process. additionally, incorporating text features and data from other sources can help improve the prediction's accuracy.