CS - UM 6 P at S em E val-2021 Task 7: Deep Multi-Task Learning Model for Detecting and Rating Humor and Offense (2021)

Abstract: Humor detection has become a topic of interest for several research teams, especially those involved in socio-psychological studies, with the aim to detect the humor and the temper of a targeted population (e.g. a community, a city, a country, the employees of a given company). Most of the existing studies have formulated the humor detection problem as a binary classification task, whereas it revolves around learning the sense of humor by evaluating its different degrees. In this paper, we propose an end-to-end deep Multi-Task Learning (MTL) model to detect and rate humor and offense. It consists of a pre-trained transformer encoder and task-specific attention layers. The model is trained using MTL uncertainty loss weighting to adaptively combine all sub-tasks objective functions. Our MTL model tackles all sub-tasks of the SemEval-2021 Task-7 in one end-to-end deep learning system and shows very promising results.

## introduction humor is a human trait that defines the emotional and behavioral characteristics of an individual. it refers to the quality of being amusing, comic, sarcastic, etc. most dictionaries define humor also as a message, whose ingenuity or verbal skill, or incongruity, that has the power to make individual laughing. humor and offensive language detection tasks are increasingly becoming hot research topics in natural language processing (nlp) @xcite @xcite @xcite . existing research works have tackled humor detection as either a binary classification problem @xcite or a ranking task @xcite @xcite . similarly, most research works on offensive language detection have proposed methods and approaches to discriminate between offensive and not-offensive texts @xcite , whereas, other research works have classified offensive content into more fine-grained levels @xcite @xcite . fine-tuning pre-trained transformer-based language models on the target task data has shown state-of-the-art (sota) results in many nlp applications @xcite . for instance, several research works on humor and offensive language detection have achieved sota performances using pre-trained transformer-based language models @xcite @xcite . in this paper, we describe our system submitted to the semeval-2021 task-7 (sub-tasks 1 and 2) @xcite . we propose an end-toend deep multi-task learning (mtl) model based on roberta encoder @xcite and taskspecific attention layers. the attention mechanism is applied on top of the encoder's contextualized word embedding to extract task-specific features. the classification and regression modules are fed with their task-specific attention output and the shared pooled output of the encoder. in order to adaptively combine all tasks' losses, we employed the mtl uncertainty loss weighting method @xcite . we also investigate the base and the large variants of bert @xcite and roberta encoders for both single-task and mtl. the obtained results show that our mtl model outperforms its single-task counterparts on both task 1 and task2. the best performances are obtained using roberta-large encoder. our system is ranked 18th, 9th, 7th and 20th on sub-tasks 1a, 1b, 1c and 2a, respectively. the remainder of this paper is organized as fol-lows. section 2 describes the semeval-2021 task-7 and the provided data. section 3 presents our mtl system. section 4 summarizes the obtained results. section 5 concludes the paper. ## task description the semeval-2021 task 7 consists of two main tasks: the first task seeks recognizing and rating humor while the second task aims to rate offense @xcite . to this end, the organizers have provided 8000 sentences for the training, and 1000 sentences for the validation and test. all training and validation sentences are labeled for humor detection and offense rating, while only humorous sentences are labeled for humor and controversy rating. the dataset is labeled by 20 annotators. they have a balanced set of age groups from 18 to 70. ## system description we propose an end-to-end deep mtl model based on pre-trained transformer-based language model @xcite and taskspecific attention layers. first, we apply the encoder to the input text in order to obtain its contextual word embedding (cwe). the task-specific attention layers are applied on the cwe. the classifier (task 1a, task 1c) or the regressor (task 1b, task 2a) is fed with the concatenation of its taskspecific attention output and the encoder's pooled output. the model is then trained to minimize the binary cross-entropy loss and the rmse loss for the classification and regression tasks, respectively. finally, these losses are combined using uncertainty loss weighting for mtl. ## conclusion in this paper, we have presented our system for humor and offense detection and rating. our system consists of an end-to-end mtl model based on the state-of-art pre-trained transformer encoder and task-specific attention layers. the latter layers are applied on top of the contextualized word embedding to extract task-discriminative features. we have employed two classification and regression modules to tackle the four tasks. our mtl model is trained to minimize the four tasks losses, while weighting them adaptively using the mtl uncertainty loss weighting. we have also investigated the performance of our mtl model as well as its single-task counterparts using four pretrained transformer-based encoders. the best performances are obtained using our mtl model while employing roberta-large encoder. in future work, we would like to improve our model, by considering the relationship between the different tasks. besides, we want to use our model not only to detect humorous and offensive content, but also to perform other related tasks.