FII FUNNY at S em E val-2021 Task 7: H a H ackathon: Detecting and rating Humor and Offense (2021)

Abstract: The “HaHackathon: Detecting and Rating Humor and Offense” task at the SemEval 2021 competition focuses on detecting and rating the humor level in sentences, as well as the level of offensiveness contained in these texts with humoristic tones. In this paper, we present an approach based on recent Deep Learning techniques by both trying to train the models based on the dataset solely and by trying to fine-tune pre-trained models on the gigantic corpus.

## introduction the figurative language of social media is one of the most challenging topics facing natural language processing (nlp). in this study, we refer at humor that requires a multidisciplinary approach for its @xcite . imagine, a viral topic on social media as elections @xcite or a political crisis @xcite . social media users themselves introduce a specific language based on common practices (e.g., humor, irony), making their message analysis very challenging @xcite . the legitimate research questions of this paper intend to answer: is humor an insurmountable barrier for artificial intelligence (ai)? we propose an approach based on recent dl techniques for sentiment analysis (sa). furthermore, we experimented with multiple types of dl architectures ranging from convolutional neural networks (cnn) to recurrent neural networks (rnn) which we tried to train using only the dataset provided by the semeval-2021 task 7 competition, but we also used pretrained transformer architectures which we fine-tuned using the available data. the rest of the paper is organized as follows: section 2 describes the literature related to sentiment analysis and humor detection, section 3 presents the dataset and method of this study, section 4 summarizes the results of the conducted experiments, with discussions after experiments, followed by section 5 with the conclusions. ## related work this topic has attracted significant attention in recent years, evidenced by increasing number of workshops of the same competition (e.g., semeval-2017 task 6: hashtagwars: learning a sense of humor or semeval-2020 task 7: assessing humor in edited news headlines). such a competition is attractive, especially since the problem of labeled data is somewhat solved, considering the fact that the automatic humor recognition depends on these. for the binary task, as in this case, there are many computational models to solve it or to detect the humor intensity or humor dimension @xcite . thus, work on this topic was never followed by high results, as this problem is still almost subjective and text classification even for humans is very controversial and biased. never the less, the task can be approached like any sa task for which most of the authors used lstms (murthy, dr and allu, shanmukha and andhavarapu, bhargavi and bagadi, mounika, 2020) or cnns @xcite . new approaches concentrate on using attention based methods @xcite , in particular transformer architectures such as bert (devlin, jacob and chang, ming-wei and lee, kenton and toutanova, kristina, 2018), roberta (liu, yinhan and ott, myle and goyal, naman and du, jingfei and joshi, mandar and chen, danqi and levy, omer and lewis, mike and zettlemoyer, luke and stoyanov, veselin, 2019), albert (lan, zhenzhong and chen, mingda and goodman, sebastian and gimpel, kevin and sharma, piyush and soricut, radu, 2019), and videobert (sun, chen and myers, austin and vondrick, carl and murphy, kevin and schmid, cordelia, 2019). these transformers are pre-trained on unlabeled data to be later fine-tuned for a variety of tasks. for this task we used the bert architecture. ## dataset and methods this section contains details about the dataset built as part of semeval-2021 task 7 hahackathon: detecting and rating humor and offense, which was the basis for solving the subtasks of this competition. ## results and discussion below are presented the official results for all subtasks. for the classification tasks we also report the results in the post-evaluation phase and the ranking as of march 2021. in the official competition we accidentally performed a mistake for the humor detection task and reverted the labels that we submitted. that is why we obtained such a poor performance and this is the reason we also report these results as it better reflect the actual performance. we report accuracy (acc), f1-score (f1), and root mean square error (rmse). the official results are summarized in tables 2 and 3 and the results in the post-evaluation phase are presented in the results on our validation set for all the techniques are summarized in the graphics depicting the evolution of the metrics over the epochs (figure 4 ). the test label refers to our validation set and not the official test set from the competition. we can easily see that all the methods overfitted the training set. techniques such as dropout and regularization have been applied, but we observed that they only delayed the moment when overfitting occurred and did not increase the performance on the validation set. the best accuracy on the validation set of the techniques used on the first task is presented in as it can be observed from the table, the best method turned out to be to fine tune a pretrained bert model, therefore we used this technique on the rest of the tasks. despite of its success we can still observe that the accuracy on the validation set (95.4%) differs from the one we obtained in the competition (92.2%). for approaches to sa on a small dataset, the best way is to fine tune pretrained models, rather than trying to train a model from scratch with the available data. the humor detection task turned out to be a very challenging one, fact that can be best expressed by the results on the second classification task where we assume that the very diverse interpretations of the annotators on what constitutes a controversy humor made the task impossible to solve with any dl model. but for more approachable tasks like the first classification task or the regression tasks, probably the more consensus among annotators made the task more tractable to an artificial intelligence model. ## conclusion according with the legitimate question, we still consider that detecting someone's sense of humor is a difficult problem and identifying the level of offensiveness and irony is an even harder one. in this case, a figurative content could be consider irony, satire, joke, and sarcasm. as with humor, all these figures of speech depends on the listener or reader to be in on this context. this paper presents a system participating at semeval 2021 task 7 and tried to adapt existing deep learning techniques to the problem of humor detection. this approach indi-