Y oung S heldon at S em E val-2021 Task 7: Fine-tuning Is All You Need (2021)

Abstract: In this paper, we describe our system used for SemEval 2021 Task 7: HaHackathon: Detecting and Rating Humor and Offense. We used a simple fine-tuning approach using different Pre-trained Language Models (PLMs) to evaluate their performance for humor and offense detection. For regression tasks, we averaged the scores of different models leading to better performance than the original models. We participated in all SubTasks. Our best performing system was ranked 4 in SubTask 1-b, 8 in SubTask 1-c, 12 in SubTask 2, and performed well in SubTask 1-a. We further show comprehensive results using different pre-trained language models which will help as baselines for future work.

## introduction humor is an intelligent form of communication with the capability of providing amusement and provoking laughter @xcite . it helps in bridging the gap between various languages, cultures, and demographics. humor is a very subjective phenomenon. it can have different intensities, and people may find some jokes funnier than others. in certain situations, some jokes may be offensive to a certain group of people. all these characteristics of humor pose an interesting linguistic challenge to nlp systems. semeval 2021 task 7: hahackathon: detecting and rating humor and offense @xcite aims to draw attention to these challenges in humor detection. the task provides a dataset of humorous content annotated using people representing different age groups, gender, political stances, and income levels. the content of the provided dataset was in english. participating in all subtasks, we propose a finetuning based approach on pre-trained language models. pre-trained language models learn syntactic and semantic representations by training on large amounts of unsupervised data. recently there has been a lot of interest in plms. researchers have come up with different pre-training methods using auto encoding(ae) and auto-regressive(ar) language modeling techniques. often these pre-trained models contain millions of parameters and are computationally expensive. finetuning different models may lead to different results on downstream tasks. this makes the choice of plm an important factor. we present a comparative study of different plm models and their performance in all subtasks of semeval 2021 task 7: hahackathon. our proposed fine-tuning approach for each plm made use of a single layer of one neuron stacked on the plm features. we performed experiments using bert @xcite , elec-tra(clark et al., 2020), roberta @xcite , xlnet @xcite , mpnet @xcite , and albert @xcite . for regression tasks, we also used averaging technique, which we describe later in the paper. our model performed well in subtask 1-b and 1-c, achieving a rank of 4 and 8 respectively on the official leader board. for subtask 2, our proposed averaging technique outperformed individual fine-tuned models by a good margin and was ranked 12. our code is available online foot_0 for method replicability. ## background the task of automatic humor recognition refers to deciding whether a given sentence expresses humor. the problem of humor recognition is often formulated as a binary classification problem aiming to identify if the given text is humorous. @xcite performed a study to identify if a joke is humorous or not using transformers @xcite . they used the body of the joke, the punchline exclusively, and both parts together. combining both parts lead to better performance and it was found that a punchline carries more weight than the body of a joke for humor identification. (de oliveira and rodrigo, 2015) experimented with svm's, rnn's, and cnn's for identifying humor in yelp reviews using a bag of words and mean word vector representations. @xcite used cnn-based models for identifying humor content. (annamoradnejad, 2021) uses a neural network built on bert embeddings learning features for sentences and whole text separately and then combining them for prediction on 200k short texts for humor detection dataset on kaggle foot_1 . binary classification tasks help us to separate humorous content but are unable to quantify the degree of humor. semeval-2017 task 6: #hashtagwars @xcite aimed to study the relative humor content of funny tweets by either generating the correct pairwise comparisons of tweets (subtask a) or finding the correct ranking of the tweets (subtask b) based on their degree of humor content. semeval-2020 task 7: assessing humor in edited news headlines @xcite presented a study on editing news headlines to make them humorous. the task involved quantifying the humor of the edited headline on a scale of (0-3) as well as comparing the humor content of the original and edited headline. semeval-2020 task 8: memotion analysis-the visuo-lingual metaphor! @xcite provides details on humor classification as well as predicting its semantic scale on internet memes using both images and text. offenseval @xcite ) @xcite provides insights for identifying offensive content on social media. humor is an intelligent way of communication in our daily lives. it helps bridge the gap between people from various cultures, ages, gender, languages, and socioeconomic status making it a powerful tool to connect with the audience. humor is a highly subjective phenomenon. people from different demographics may have a different perception of humor, and some may even find it offensive. this makes identifying humor a tough task. semeval 2021 task 7: hahackathon: linking humor and offense across different age groups aims to study this subjective nature of humor, which has two sub-tasks which we describe as: subtask 1: given a labeled dataset d of texts, the task aims to learn a function that can: subtask 2: given a labeled dataset d of texts, the task aims to learn a regression function that can quantify how offensive a text is for general users within a range of (0-5). ## experimental setup we used ekphrasis @xcite library for pre-processing the text inputs. it normalized date, time, numbers to a standard format and also performed spelling correction. for tokenization, we used hugging face's @xcite implementation of fast tokenizers for each pre-trained model. we fixed the sequence length of samples to 150 tokens. models were developed on keras foot_2 @xcite using the transformers foot_3 @xcite library by hugging face. we used adam (kingma and ba, 2017) optimizer for fine-tuning. learning rate of 1e-4 was used for electra. for other models, we experimented with 1e-5, 2e-5, and 3e-5. we used binary cross-entropy loss for classification tasks and logcosh loss for regression tasks. batch size of 16 was used for all models. fine-tuning was performed on tpu's on google colab. we fine-tuned for four epochs on subtask 1 and 8 epochs on subtask 2. f1 score and rmse were used as an evaluation metric for classification and regression tasks. weights with the best performance on the development set were used for making predictions on the test set. ## results and analysis figure 1 and figure 2 show plots of confusion matrices for our best performing model fine-tuned on bert. for subtask 1-a, our model was efficient in separating the humorous and non-humorous content as false positives are low for each class. for subtask 1-c, our model performed well in identifying the controversial text but did not perform model precision recall f1 accuracy subtask 1-a 1-c 1-a 1-c 1-a 1-c 1-a 1-c bert-base .9254 .4630 .9691 .9426 .9467 .6210 .9330 .4780 electra-base .9269 .4747 .9073 .7419 .9170 .5790 .8990 .5105 roberta-base .9518 .4708 .8991 .8960 .9247 .6172 .9100 .4959 mpnet-base .9658 .4879 .9203 .7275 .9425 .5841 .9310 .5300 xlnet-base .9489 .4812 .9365 .6881 .9427 .5663 .9300 .5219 albert-large .9632 .4666 .8943 .8530 .9274 .6032 .9140 .4910 ## conclusion the paper describes our system used for competing in all subtasks of semeval 2021 task 7: ha-hackathon: detecting and rating humor and offense. we used a simple fine-tuning approach for analyzing the performance of various pre-trained language models for the task of humor detection. we performed well in all pre-trained language models with new and better models coming up. these models are large and computationally expensive. choosing a model becomes a difficult task as they may have different results on downstream tasks. we, therefore, performed experiments with the recent state-of-the-art models and provide a comparative analysis of their performance. in the future, we would like to work on the effect of pre-training plms with additional task-specific data and then fine-tuning to see their performance on downstream tasks.