H umor H unter at S em E val-2021 Task 7: Humor and Offense Recognition with Disentangled Attention (2021)

Abstract: In this paper, we describe our system submitted to SemEval 2021 Task 7: HaHackathon: Detecting and Rating Humor and Offense. The task aims at predicting whether the given text is humorous, the average humor rating given by the annotators, and whether the humor rating is controversial. In addition, the task also involves predicting how offensive the text is. Our approach adopts the DeBERTa architecture with disentangled attention mechanism, where the attention scores between words are calculated based on their content vectors and relative position vectors. We also took advantage of the pre-trained language models and fine-tuned the DeBERTa model on all the four subtasks. We experimented with several BERT-like structures and found that the large DeBERTa model generally performs better. During the evaluation phase, our system achieved an F-score of 0.9480 on subtask 1a, an RMSE of 0.5510 on subtask 1b, an F-score of 0.4764 on subtask 1c, and an RMSE of 0.4230 on subtask 2a (rank 3 on the leaderboard).

## introduction humor, appreciated by people with almost any age or cultural background, is perhaps one of the most fascinating human behaviors. besides providing entertainment, humor can also be beneficial to mental health by serving as a moderator of life stress @xcite , and plays an important role in regulating human-human interaction. as @xcite have pointed out, people respond to computers in the same way as they do to real people, which indicates that modeling humor computationally could bring positive effects in human-computer interaction @xcite . despite being universal to human beings, the extent to which people find something humorous varies according to one's age, gender, or socio-economic status, making humor a highly subjective experience. this poses many challenges to the field of computational humor. abundant research has been done to enable computers to automatically decide whether humor is entailed in a given piece of text. early work @xcite uses manually engineered features to recognize humor in text, while more recent work @xcite adopts deep learning approaches and pre-trained language models. semeval 2021 task 7: hahackathon: detecting and rating humor and offense @xcite aims at detecting and rating humor as well as offense in short english text. there are four subtasks involved. subtask 1a is a binary classification task, predicting if the text would be considered humorous for an average user. subtask 1b is a regression task and predicts the humor rating of the text if it is considered humorours. subtask 1c is again a binary classification task and predicts whether the humor rating is controversial, whose ground-truth label is decided based on the variance of the annotators' ratings. this task also involves offense detection. subtask 2a predicts how offensive the text is for a general user. all the regression subtasks have scores ranging from 0 to 5. in this paper, we present our system submitted to semeval 2021 task 7. we followed the architecture of deberta @xcite , an improved version of bert @xcite and roberta @xcite by using two novel techniques: disentangled attention and decoding enhanced masking. we mainly relied on the disentangled attention mechanism, where the attention weights of the input words are calculated based on their content vectors and relative position vectors. for the four subtasks, we used the same base structure and the only difference is at the output layer, where the classification tasks have two out-put units and the regression tasks only have one. the pre-trained deberta model has two variants that differ in size. during the evaluation phase, the large version achieved an f-score of 0.9480 on subtask 1a, an rmse of 0.5510 on subtask 1b, an f-score of 0.4764 on subtask 1c, and an rmse of 0.4230 on subtask 2a (rank 3 on the leaderboard). in addition, we also experimented with the bert and roberta models as our baselines, and found them generally under-performed by deberta. our code has been made publicly available. foot_0 2 related work @xcite used several human-centric features such as alliteration and synonym to recognize humor in one-liners. @xcite approached the problem by calculating the semantic relatedness between the set-up and the punchline. @xcite proposed a generative language model and leveraged background text sources to identify humor in yelp reviews. @xcite proposed to model sentiment association between elementary discourse units and designed features based on discourse relations. @xcite calculated the uncertainty and surprisal of the set-up and the punchline according to the incongruity humor theory, which were found useful in humor recognition. recent work also developed neural network based models to recognize humor in text. @xcite and chen and soo (2018) adopted convolutional neural networks, while @xcite used a transformer architecture. ## dataset semeval 2021 task 7 provides three datasets: the training set (8,000), the validation set (1,000), and the final test set (1,000). ## system overview with the increasingly powerful neural networks such as the transformer @xcite , the performance on many downstream nlp tasks has been greatly improved by fine-tuning large pre-trained language models on smaller but taskspecific datasets. traditional transformer-based language models such as bert @xcite and roberta @xcite use absolute positional embeddings in the input layer, which are added up with the word embeddings and serve as the input to the following transformer layers. the self attention weights between the tokens are calculated solely based on their hidden representations. however, recent work @xcite has shown that relative position representations are more effective for nlp tasks. our system leverages the disentangled attention mechanism from the deberta model @xcite , where the attention weights between input tokens are calculated based on their content vectors as well as their relative positions. as shown in figure 2 , for each transformer layer, h i 's are the input representations from last layer, and h o i 's are the output representations after applying the self attention. instead of using absolute positional embeddings at the input layer, we create a relative positional embedding table, which is shared across all layers, to represent the relative position between token i and token j. more specifically, the index of the relative position between token i and j is defined as then, the attention weight a ij between token i and token j are calculated as follow: when aggregating the input representations h, we apply a scaling factor 1/ √ 3d to obtain the output representations h o : for subtask 1a and 1c, which are binary classification tasks, we use softmax output layer and cross entropy loss. for subtask 1b and 2a, which are regression tasks, we use mean square error as the loss function. otherwise, the base structure is the same, and we initialize the model with the pre-trained deberta weights. ## experimental setup we evaluated and compared our system with several baselines on the provided dataset, whose statistics are provided in section 3. in this section, we are going to elaborate the setup of our experiment. ## results the performance of our system and the baselines is shown in we infer this is because most of the humorous examples in the training set contains a question, usually followed by a short answer serving as the punchline. ## conclusion in this paper, we describe our system submitted to semeval 2021 task 7. we adopted the disentangled attention mechanism from the deberta model, and participated in all the four subtasks. during the evaluation phase, we got a rank of 3 on the leaderboard for subtask 2a. for future work, we would like to combine human-centric features with the current architecture using the disentangled attention mechanism, and develop a hybrid model. in addition, we plan to expand the provided dataset with extra jokes from various sources such as reddit forums, hoping to further improve the performance of our system.