abcbpc at S em E val-2021 Task 7: ERNIE -based Multi-task Model for Detecting and Rating Humor and Offense (2021)

Abstract: This paper describes our system participated in Task 7 of SemEval-2021: Detecting and Rating Humor and Offense. The task is designed to detect and score humor and offense which are influenced by subjective factors. In order to obtain semantic information from a large amount of unlabeled data, we applied unsupervised pre-trained language models. By conducting research and experiments, we found that the ERNIE 2.0 and DeBERTa pre-trained models achieved impressive performance in various subtasks. Therefore, we applied the above pre-trained models to fine-tune the downstream neural network. In the process of fine-tuning the model, we adopted multi-task training strategy and ensemble learning method. Based on the above strategy and method, we achieved RMSE of 0.4959 for subtask 1b, and finally won the first place.

## introduction humor, as a highly subjective phenomenon, can be affected by various factors. automatic humor recognition relies on annotated data to determine whether the text is humorous or not @xcite . however, such a binary classification does not capture the level of humor, so assessing the level of humor is of great significance @xcite . since humor can be influenced by many factors, such as age, and may offend others. based on such a situation, semeval-2021 task 7 focuses on linking humor and offense across different age groups @xcite . but there are still many challenges to this task. for example, the dataset for the task is small and the texts are short, which does not allow for adequate training. to address these issues, we utilized unsupervised pre-trained language models and fine-tuned these models for specific downstream subtasks. after conducting research and extensive comparative experiments, the results show that ernie 2.0 @xcite and deberta pre-trained models performed best on the subtasks. these large unsupervised language models were pre-trained on a large amount of unlabeled data to extract valuable lexical, syntactic, and semantic information from the corpus. vector representations of text computed by these models are applied to fine-tune the downstream neural networks for the subtasks. the multi-task training and ensemble learning method significantly improve our model's performance. the rest of the paper is organized as follows: section 2 provides a brief description of the related work, and section 3 describes our proposed approach in detail. in section 4, the experiments are described in detail and the results are presented. finally, we summarize the whole paper and discuss future research directions in section 5. ## related work in the early research of humor and offense detection and evaluation, traditional machine learning methods and n-gram language model were mostly used. recent research has shown that unsupervised language pre-trained models using large amounts of unlabeled data have achieved state-of-the-art results in a large number of natural language processing tasks. for example, bert @xcite is a model built based on transformer encoder, which is used for downstream tasks by pre-training on the masked language models task and the next sentence prediction task, and then for fine-tuning. inspired by this approach, many pre-training language models have been proposed. for example, albert @xcite adopts factorized embedding parameterization and cross-layer parameter sharing strategies, and adds the sentence order prediction task, so that the model can greatly reduce the number of parameters without lose accuracy compared to bert. roberta @xcite removes the next sentence prediction task and uses dynamic mask, which is optimized for bert. deberta @xcite improves the bert and roberta models by using two new techniques. the first one is using disentangled attention mechanism and secondly, enhanced mask decoder is used. mt-dnn @xcite combines multi-task learning and pre-trained models to improve the performance of various natural language processing tasks. ernie 1.0 @xcite employs an entitylevel and phrase-level mask, the extension of its training corpus and the use of multiple rounds of conversation to replace sentence pair classification further enhance the model's semantic representation capability. ernie 2.0 @xcite is an optimized version of ernie 1.0, which introduces a large number of pre-training tasks and continuously updates the pre-training model through multi-task learning to help the model learn lexical, syntactic and semantic representations efficiently. ernie 2.0 constructs three pre-training tasks, namely word-aware pre-training tasks, structureaware pre-training tasks and semantic-aware pretraining tasks. the performance of the model is improved by constructing pre-training tasks from multiple perspectives. the ernie 2.0 model outperformed bert and xlnet @xcite almost across the board on the english task and achieved the best results on 7 glue tasks; on the chinese task, the ernie 2.0 model outperformed bert across the board on all 9 chinese nlp tasks. ## conclusion in this paper, we proposed a multi-task training system based on ernie 2.0. we describe the architecture of the model and the training process in detail. besides, we experimentally demonstrate that the strategy performs better with multi-task training compared to single-task training. moreover, the ensemble learning method makes the model more robust. as a result, we have won the first place in a subtask for the competition of semeval-2021 task 7. in our future work, we will further explore pretrained language model and optimize the multi-task training.