ES - JUST at S em E val-2021 Task 7: Detecting and Rating Humor and Offensive Text Using Deep Learning (2021)

Abstract: This research presents the work of the team’s ES-JUST at semEval-2021 task 7 for detecting and rating humor and offensive text using deep learning. The team evaluates several approaches ( i.e.Bert, Roberta, XLM-Roberta, and Bert embedding + Bi-LSTM ) that employ in four sub-tasks. The first sub-task deal with whether the text is humorous or not. The second sub-task is the degree of humor in the text if the first sub-task is humorous. The third sub-task represents the text is controversial or not if it is humorous. While in the last task is the degree of an offensive in the text. However, Roberta pre-trained model outperforms other approaches and score the highest in all sub-tasks. We rank on the leader board at the evaluation phase are 14, 15, 20, and 5 through 0.9564 F-score, 0.5709 RMSE, 0.4888 F-score, and 0.4467 RMSE results, respectively, for each of the first, second, third, and fourth sub-task, respectively.

## introduction dealing with natural languages has long been a challenge and an interesting topic for researchers @xcite . understanding and generating languages is part of natural language processing (nlp) @xcite . recently, the language model is able to deal with sequence-tosequence problems such as question and answer, translation, multiple choice. in addition, it is able to capture complex relationships, semantic meaning, word meaning disambiguation, and word aspectbased @xcite . humorous text is one of the important things we watched every day. it is commonly used to express an opinion on issues (societal, political, sports, and economic), whether in posts on social media platforms, or as advertising for a specific product @xcite . in addition, the humor in the text makes the text complex in terms of interpretation and understanding of the text. because of the manipulation of the meaning of words and the way the text is written to express the sense of humor in the words. on the other hand, understanding the humorous in the text varies according to the age or gender of the person, or even according to the culture, social status and mentality of the person @xcite . in this task, a dataset was collected in the english language that represents humor and joke in the text and words. we participated in this task to build an approach capable of distinguishing a text that is humorous or not. here we have explicitly used pre-trained models that deal with the concept of contextual text such as bert @xcite , roberta @xcite , and xlm-roberta @xcite . in addition, as a baseline we worked on training the dataset by the bert embedding layer and extracting weights to feed it into the bi-lstm and dense layers. in all sub-tasks we used as a baseline bil-stm @xcite layer with a bert embedding layer, as well as, pre-trained models such as bert @xcite , roberta @xcite , and xlm-roberta @xcite . in all sub-tasks, roberta model showed superiority compared to other approaches. we ordered according to the official results among 36 participating teams. in the first sub-task, we achieved 26th rank with an 0.9564 f-score result. on the second sub-task, ranked 26th with a score of 0.5709 rmse. a third sub-task placed 25th with 0.4888 fscore @xcite . the last sub-mission we took the 9th rank with a score of 0.4467 rmsr @xcite . the remainder of this paper is organized as follows: background in section 2. the properties of the dataset and the system in section 3. section 4 explains the experiment and analyzing results. the last section 5 shows conclusions and future work. ## background in @xcite developed a new humor corpus, which consist of 15,095 news headlines in english. they substituted the headlines with few words to be funny. also, @xcite used attention-based with bi-directional long short-term memory (attbilstm) to classify slang language into negative humor or positive humor. in (annamoradnejad, 2020) utilized a bert embedding layer with several parallel hidden layer to categorize 200k humorous sentences whether (positive or negative). while @xcite used two kinds of attention mechanisms (internal and external) to capture sense of humor in words. most of the previous works came to predict the humor polarity (positive, negative) or the humor rating (range values) in the text. however, this research addressing the humor and offensive score detection. ## experimental and results in the experimental phase, the dataset was divided into three parts (training, development, and testing). we used the training dataset to train the model, and the development dataset to fine-tune the model to capture the best hyper-parameters without occurring over-fitting or under-fitting the model. moreover, we used the test data set to check the performance of the model with an unseen dataset and to ensure the generalizability of the model. however, to perform the experiments we used collaborative google colab as a platform, which provides a number of gpus available for use with modest memory size foot_1 . in addition, in our experiments with pretrained models, we used the transformers library that is based on the pytorch language and allows you to fine-tune the models and train them on your own dataset foot_2 . in training the model, we did not use any pre-processing technique in the entered dataset. although, there are some symbols, upper and lower case letters, misspellings, and some abbreviations in the dataset. however, we did not treat these issues in dataset, where the dataset is trained as it is. in order, for the model to be more realistic and robust in dealing with the real dataset. as well as, the model might deal with those cases as features for each case in the dataset for the model learning phase. just in pre-processing phase, we needed to remove nan values in both sub-task (1b and 1c ). in order to test the performance of the approaches used in this task, where each sub-task has a metric that meets the type of output of each sub-task such as regression metric or classification metric. accuracy and f-score metrics were a measure of the ## conclusion in this paper, we presented several approaches that addressed four sub-tasks. we obtained high scores using a pre-trained roberta model for each subtask. in the first sub-task, predicting if the text is humorous or not, we gained a 0.9564 f-score. while in the second sub-task, finding a humorous text representation rate from 0 to 5, that was got a 0.5709 rmse. a third sub-task, verification of the text is controversial or not, obtained a 0.4888 fscore. the last sub-task is to find the offensive rate in the text for the range of 0 to 5, which achieved 0.4467 rmse. for future works, we are going to do more experiments and using ensemble technique to enhance the results. moreover, adding more dataset with the original to treat the biased label.