S arcasm D et at S em E val-2021 Task 7: Detect Humor and Offensive based on Demographic Factors using R o BERT a Pre-trained Model (2021)

Abstract: This paper presents one of the top winning solution systems for task 7 at SemEval2021, HaHackathon: Detecting and Rating Humor and Offense. This competition is divided into two tasks, task1 with three sub-tasks 1a,1b, and 1c, and task2. The goal for task1 is to predict if the text would be considered humorous or not, and if it is yes, then predict how humorous it is and whether the humor rating would be perceived as controversial. The goal of the task2 is to predict how the text is considered offensive for users in general. Our solution has been developed using RoBERTa pre-trained model with ensemble techniques. The paper describes the submitted solution system’s architecture with the experiments and the hyperparameter tuning that led to this robust system. Our model ranked third and fourth places out of 50 teams in tasks 1c and 1a with F1-Score of 0.6270 and 0.9675, respectively. At the same time, the model ranked one of the top 10 models in task 1b and task 2 with an RMSE scores of 0.5446 and 0.4469, respectively.

## introduction in our daily life, the obstacles and difficulties in dealing with sarcasm, bullying, or even abuse of all kinds and ways are increasing day by day @xcite @xcite . technically, sarcasm and bullying are among the most complex and challenging topics that major companies and institutes seek to address. artificial intelligence and text processing techniques are the most potent current methods for detecting these problems within texts and images. sarcasm and abuse are associated with attacking a specific person or group of people either through an unintended joke or, in many cases, by directly affecting the target's psyche. irony and offensiveness are characterized by their vocabularies that are peppered with humor to conceal the opposite @xcite . task 7 at semeval-2021, "hahackathon: detecting and rating humor and offense", provides two main tasks: task-1 with three sub-tasks (1a,1b, 1c) and task-2. the goal of task-1 is to predict if the text would be considered humorous or not, and if it is yes, then expect how funny it is and whether the humor rating would be perceived as controversial. the goal of task 2 is to predict how the text is considered offensive for users in general. our solution, sarcasmdet, has been ranked among the top four teams in two sub-tasks. the proposed approach uses the provided dataset, which contains 10k of row text data. we have experimented with several pre-trained language models using the simple transformers library. it is worth mentioning that using the hard-voting ensemble technique has increased our score remarkably. the paper is constructed as follows: section 2 provides the related works. section 3 and 4 describe the shared task and the provided dataset, respectively. section 5 describes our system solution. section 6 shows our experiments. section 7 provides the results, and finally, the conclusion is in section 8. ## related works in recent years, social media's development and growth have motivated the nlp research community to detect humor and offensiveness. in 2018, semeval provided different shared tasks to detect emotions and irony in tweets @xcite . the top teams' proposed models mostly used lstm and word embeddings @xcite @xcite . in 2019, semeval also introduced a shared task to discover offensive language in social media. researchers in @xcite used the dataset of the offensive language identification dataset (olid) provided by @xcite . they ranked first in the task with an f1 (macro) score of 0.8286 by applying linear model, lstm, and bert pre-trained model. in 2020, one of the shared tasks presented in semeval was about how to change a chunk of text to make the text funnier. the authors @xcite applied a pre-trained bert model with different preprocessing for the presented dataset. this paper presents our solution to task 7 in semeval2021, to detect humor and offensive simultaneously and explains it in detail. ## tasks description all subtasks of the semeval2021 task 7 have different requirements. in this section, we have detailed the description for each task. ## dataset description the dataset provided by @xcite dataset is divided into three phases: training, development, and evaluation phase datasets. the dataset was collected by surveying us english native speakers of various ages between 18-70 and different genders, political situations, and income levels. the training set contains 8,000 rows of texts with four labels, every text in the data set have been classified based on four questions that were asked to the participants, and each question is related to a specific task. each of the development set and the test set contain 1000 texts. ## systems description in our solution, we have used the pre-trained language model, roberta @xcite , that uses a robustly optimized nlp method to improve the bidirectional encoder representations from transformers. we have also used the bert pretrained model @xcite . roberta is built based on bert's language masking strategy, which learns to predict knowingly hidden sections of text within unannotated language examples. we have chosen roberta pre-trained model because of the significant improvements in the performance by tuning the bert training procedure and the architecture based on bert-large. we have experimented with several deep learning models. in our best-performed solution system, we implemented ensemble technique @xcite on the bestscored models that include roberta-large, bertlarge trained on 24-layer, 1024 hidden, 16-heads, 355m parameters. we used roberta model from huggingface @xcite and simpletransformers pre-trained models. more details about each subtask are as follows. ## experiments we have experimented with several pre-trained nlp models to detect humor and offensive through the development and evaluation phases. the pre-trained models include bert(base/large) that is developed by google researchers. also, albert(base/large) @xcite , which is a lite version of bert to reduce parameters and increase the model speed by reducing memory consumption. another pre-trained model is xl-net(base/large) @xcite , which introduced the automatic regressive pre-training method and outperformed bert model in several tasks sentiment analysis, question answering and others. finally, roberta model, which outperformed most of the pre-trained models, if not all. we have implemented our experiments on google colab using cpu and gpu. using collab gpu increased the speed of the experiments by 100%. we used simple transformers library with various hyperparameters, learning @xmath0, manual @xmath1, train batch @xmath2. our best results accomplished on all tasks was using hard-voting ensemble technique on top of best-scored results by roberta-large and bert-large-cased. the use of hard-voting technique increased the performance, and the accuracy, remarkably. in the development phase, our model ranked first place in three task1a, task1c, task2, and second place in task1c. however, for the evaluation phase, we have ranked 4th place in task1a, and 3rd place in task1c 3rd. our solution system results are divided into two phases development and evaluation phase. we experienced several pre-trained language models (roberta, bert, albert, and xlnet) and implemented them using a simple transformers library in the development phase. in the evaluation phase(test phase), we improved our system solution's capabilities by using different hyperparameters with ensemble techniques. in the following sub-sections, we provided in detail all the results of the evaluation phase. ## conclusion this paper presents and describes our solution system for the semeval2021 task7: hahackathon detecting and rating humor and offense. we have applied several pre-trained language models, such as roberta, bert, albert, and xlnet, with hard-voting ensemble technique to detect humor and offense mechanism. our final solution was based on bert-large-cased model and robertalarge model, which showed remarkable improvements and a high overall outperformance. our solution system ranked 4th place in task1a with a 0.9675 f-score, 10th place in task1b with a 0.5446 rmse, 3rd place in task1c with a 0.6270 f-score, and 10th place in task2 with a 0.4469 rmse.