D eep B lue AI at S em E val-2021 Task 7: Detecting and Rating Humor and Offense with Stacking Diverse Language Model-Based Methods (2021)

Abstract: This paper describes the winning system for SemEval-2021 Task 7: Detecting and Rating Humor and Offense. Our strategy is stacking diverse pre-trained language models (PLMs) such as RoBERTa and ALBERT. We first perform fine-tuning on these two PLMs with various hyperparameters and different training strategies. Then a valid stacking mechanism is applied on top of the fine-tuned PLMs to get the final prediction. Experimental results on the dataset released by the organizer of the task show the validity of our method and we win first place and third place for subtask 2 and 1a.

## introduction humor and offense detection continue to be challenging ai problems since humor and offense involve in-depth world-knowledge, common sense, and the ability to perceive relationships across entities and objects at various levels of understanding @xcite . the recognition of humor and offense in the text has been receiving much attention @xcite . accordingly, semeval-2021 task 7, detecting and rating humor and offense, which aims to automatically recognize humor in english jokes was held @xcite . in this paper, we introduce our system for accomplishing the above task by leveraging pre-trained models (plms). there are two main steps for our system, i) fine-tuning two kinds of plms, including albert @xcite and roberta @xcite with various hyperparameters and training strategies, achieving diverse models; ii) applying a validity stacking mechanism on top of these plms to do the final predictions. our experimental results show that merging plms with different training strategies together can achieve great improvement which verifies the effectiveness of increasing model diversity. as a result, our system achieves the f1-score of 96.76% in subtask 1a and the rmse of 41.2% in subtask 2, which ranks third and first among all the participated teams respectively. ## experiments evaluation metrics as mentioned in the official evaluation procedure of semeval-2021 task 7, the main evaluation metrics for the binary classification tasks is f1-measure and the metric for the regression tasks is root mean squared error (rmse). parameter settings all models are implemented based on the open-source transformers library of hugging face @xcite , which provides thousands of pre-trained models that can be quickly downloaded and fine-tuned on specific tasks. to do better performance estimation, we gather the training set and validation set together as the new training set and then do 7-fold cross-validation on it. we set batch size as 16 and run 10 epochs for each fold. the learning rate is 1e-5. for robertalarge and albertbase, the k is set as 1024 and 128 respectively. ## conclusion in this paper, we propose a top-performing approach for the task of detecting and rating humor and offense. we fine-tune two kinds of pretrained language models including albert and roberta with different training strategies such as pseudo labeling and knowledge distillation. then, we stack them with a simple linear regression model. experimental results show the effectiveness of this ensemble method and we win first place and third place for subtask 2 and 1a. for future work, it would be interesting to test the performance of our best-performing system on other humor detection datasets to validate its portability and robustness.