PALI - NLP at S em E val-2022 Task 6: i S arcasm E val- Fine-tuning the Pre-trained Model for Detecting Intended Sarcasm (2022)

Abstract: This paper describes the method we utilized in the SemEval-2022 Task 6 iSarcasmEval: Intended Sarcasm Detection In English and Arabic. Our system has achieved 1st in SubtaskB, which is to identify the categories of intended sarcasm. The proposed system integrates multiple BERT-based, RoBERTa-based and BERTweet-based models with finetuning. In this task, we contributed the following: 1) we reveal several large pre-trained models’ performance on tasks coping with the tweet-like text. 2) Our methods prove that we can still achieve excellent results in this particular task without a complex classifier adopting some proper training method. 3) we found there is a hierarchical relationship of sarcasm types in this task.

## introduction generally speaking, when we communicate through natural language, the literal meaning of the words is consistent with the meaning we want to express. sarcasm is a form of linguistic expression when this "congruence" is broken @xcite . due to the inherent metaphorical nature and subtle sentimental expression of this particular form of language expression. the detection task related to this kind of text, which is a negative expression of a positive emotion or the positive expression of negative emotion, is extremely difficult for machines @xcite . this sarcasm data also weakens the detection modules that are widespread in our society @xcite . previous work shows that sarcasm often comes with incongruity between expectation and reality @xcite . many works attempt to model this incongruity within the text @xcite . for multi-model data, some works use the features from different modalities @xcite , and some works shows that inter-modality incongruity is also an important feature for multi-modal sarcasm detection @xcite . the semeval-2022 task6 (abu farha et al., 2022) is designed to detect sarcasm and sarcasm types in twitter texts. in subtask b, if a tweet is not sarcastic, it should not be annotated with any sarcasm label; if it is sarcastic, we need to detect which sarcasm it is, and it could have different sarcasm type at the same time. the main metric of this task is the macro-f1 score of all sarcasm types. we design a simple and effective system for this task. the system is based on a large-scale pretrained model based on bi-directional transformers @xcite and fine-tuned to obtain the final output. first, we augmented the isarcasm dataset with additional datasets, and then we set an appropriate learning rate for each layer and set the model with an appropriate initialization state. next, we strengthen the model's generalization ability through adversarial training, multi-sample dropout and other approaches. finally, we use the [cls] token of the last layer of the encoder to perform fine-tuning on the final training dataset and ensemble them using the hierarchical way. in the last layer of the pre-trained encoder, we can acquire the representation of all tokens. in this task, we only select the representation of the first [cls] token. after that, a layer-norm operation and multi-sample dropout will be utilized on the representation from the encoder. finally, we use bec loss as our loss function. ## conclusion we employ the large pre-trained models and finetune them for sarcasm category discrimination. we compare the performance of different pre-trained models on subtask b of semeval-2022 task 6. the results show that the difference between the pre-training corpus and the downstream task corpus will significantly affect the performance of the model. we find that the pre-trained model using the default training settings performed poorly on this task, and good model initialization and training strategies can help improve this situation. we also find that there is a hierarchical relationship between the types of sarcasm which we believe is an important feature worth exploiting.