T eam_ KGP at S em E val-2021 Task 7: A Deep Neural System to Detect Humor and Offense with Their Ratings in the Text Data (2021)

Abstract: This paper describes the system submitted to SemEval-2021 Task-7 for all four subtasks. Two subtasks focus on detecting humor and offense from the text (binary classification). On the other hand, the other two subtasks predict humor and offense ratings of the text (linear regression). In this paper, we present two different types of fine-tuning methods by using linear layers and bi-LSTM layers on top of the pre-trained BERT model. Results show that our system is able to outperform baseline models by a significant margin. We report F1 scores of 0.90 for the first subtask and 0.53 for the third subtask, while we report an RMSE of 0.57 and 0.58 for the second and fourth subtasks, respectively.

## introduction automatic humor and offense detection have considerable importance in the modern age, especially in chatbots and virtual assistants @xcite . detection of humor and offense in the text is a highly subjective phenomenon that varies with age, gender, race, socio-economic status, etc. therefore, it is one of the most challenging research fields in natural language processing @xcite . task-7 of semeval 2021 @xcite ) is concerned with humor detection (binary classification) primarily. the first subtask is to check if a text is humorous, another subtask follows up: how humorous is the text (rated from 0-5). we also have to predict whether the text is generally offensive (binary classification task). the last subtask predicts how generally offensive a text is (rated from 0-5), regardless of whether it is classed as humorous or offensive overall. the dataset consists entirely of english texts. the traditional methods deployed earlier for humor detection include support vector machine (svm) with rbf kernel, random forest classifier, and sgd with logical classifier, all of which provide modest results @xcite . recently more state-of-the-art transformers have provided better results @xcite . the system presented in this paper is finetuning one of the best and most popular state-of-theart models, bidirectional encoder representations from transformers (bert). we have used pretrained bert embeddings to represent the words and used the features of the last layer of the 12layers bert model to detect and rate both humor and offense. bert can model complex interactions between different levels of hierarchical information @xcite . this task is perhaps the first task that focuses on the humor and offense ratings and combines humor and offense detection. our model obtained an f1 score of 0.92 and 0.56 for the first and the third subtasks, respectively, on the test data. we obtained an rmse of 0.57 and 0.58 for the second and fourth subtasks, respectively, on the test data. the rest of the paper is organized as follows. section 2 discusses the related work. section 3 describes the dataset for the task. section 4 provides the experimental setup and evaluation metric. section 5 presents the systems implemented to address the task. section 6 shows the results and section 7 concludes the paper. ## related work we found the following works very relevant for the task to be addressed, i.e., humor and offense detection. lexical syntactic feature (lsf): this architecture @xcite was used to detect offensive language in social media. it bridges the gap between message-level and user-level offensive language detection. in particular, this paper incorporates a user's writing style, structure, and specific cyberbullying content as features to predict the user's potentiality to send out offensive content. netflix-style collaborative filtering: this method @xcite identifies a mean humor direction, analyses sense-of-humor word embeddings to predict individual differences in word humor. it proposes netflix-style collaborative filtering to predict humor ratings. bert-base and bert-large: bert models have been used previously for automatic humor detection and scoring @xcite . the pre-trained model is fine-tuned on the available data, producing appreciable results. our system lies in close proximity to the last work. we have fine-tuned a pre-trained bert model as well. the difference lies in the fine-tuning process. different approaches have been used to building the neural network, varying from linear layer approach to bi-lstm approach, for both the classification and regression tasks. ## dataset the dataset used in this paper is obtained from task 7 of semeval 2021. in this task, the organizers collected labels and ratings from a balanced set of age groups from 18 to 70. the annotators also represented a variety of genders, political stances, and income levels. the training set consists of 8000 texts, and the development and test datasets consist of 1000 texts each. the variation of the number of texts vs. word length of texts is shown in figure 1 . each text is represented by a unique id, and each subtask has a separate column for each text. the annotators were asked: the annotators could also give a rating of 0 to the second question if they do not get it due to the text's structure or content. the label of the first task is is humor and is based on the majority class given by 20 annotators. in case of a tie, the humor label was selected. the humor rating is based on the annotators' average rating, under the label humor rating. by generally offensive, the organizers mean that the text targets a person or group for merely belonging to a specific group and ask users if they think that a significant number of people would find this offensive. if the variance of a text was higher than the median variance of all texts, the humor of the text was labeled as controversial under the label humor controversy. it is a binary classification task. in the last question, we consider the ratings 1-5 and also consider a no rating to be 0. this score (average of all the ratings) was calculated regardless of whether the text is classed as humorous or offensive overall, under the label offense rating. figure 3 : the distribution of offense rating of the texts from the training set. ## experimental setup and evaluation metric preprocessing: a data pipeline preprocesses the raw data to remove irrelevant features. the stopwords and emojis (if any) are removed from the data, and the output of the pipeline is lower-case stemmed word sequences. the stop-words are selected from the nltk stop-words list @xcite . max sequence length: from figure 1 , we observe that there is no sequence having a length greater than 70 words. so, we restrict our max seq length parameter of the bert tokenizer to 70. we pad the sentences to the maximum sequence length. adam optimizer: for optimizing the model, we use adam optimizer. adam combines the best properties of the adagrad and rmsprop algorithms to provide an optimization algorithm that can handle sparse gradients on noisy problems and works well on deep neural networks @xcite . for subtasks-1 and 3, we used a learning rate of 2e-5, and for subtasks-2 and 4, a learning rate of 1e-5 was used. experimental tools: we used google colab to run the experiments, which provided us gpu to run our model. the external libraries used are publicly available transformers (version 3.0.0) from the pytorch huggingface api foot_0 and python-based scikit-learn package. evaluation metric the metric for the classification tasks (subtask 1 and 3) is f1 score. for the regression tasks (subtask 2 and 4), the metric is root mean square error (rmse). the task organizers suggested the following baseline models: bert has proven promising for many nlp tasks @xcite . our system implements finetuning strategies on pre-trained bert architecture. it is a bidirectional transformer pre-trained using a combination of masked language modeling objective and next sentence prediction on a large corpus comprising the toronto book corpus and wikipedia. ## results and analysis for each subtask, we trained our data on the entire training set of 8000 texts. we used the development set of 1000 texts as cross-validation data. the results tabulated in this section are reported on the gold test set of 1000 texts. each proposed model for all the subtasks was run for 10 epochs with a batch size of 32. hyperparameters are discussed in section 5. ## conclusion in this paper, we describe a system developed to address the semeavl task-7. the task has four subtasks, viz., detecting humor, detecting offense, predicting humor rating, and predicting offense rating. our system is able to perform all four subtasks with varying levels of performance for each task. our system deploys linear layers and bi-lstm layers independently to process the features produced by the bert model. results show that our system using bert with linear layers outperforms the baseline model by a significant margin for the first and the third subtasks. on the other hand, the bi-lstm layers-based system gives the best performance for the other two fine-grained rating prediction tasks.