R o M a at S em E val-2021 Task 7: A Transformer-based Approach for Detecting and Rating Humor and Offense (2021)

Abstract: In this paper we describe the systems used by the RoMa team in the shared task on Detecting and Rating Humor and Offense (HaHackathon) at SemEval 2021. Our systems rely on data representations learned through fine-tuned neural language models. Particularly, we explore two distinct architectures. The first one is based on a Siamese Neural Network (SNN) combined with a graph-based clustering method. The SNN model is used for learning a latent space where instances of humor and non-humor can be distinguished. The clustering method is applied to build prototypes of both classes which are used for training and classifying new messages. The second one combines neural language model representations with a linear regression model which makes the final ratings. Our systems achieved the best results for humor classification using model one, whereas for offensive and humor rating the second model obtained better performance. In the case of the controversial humor prediction, the most significant improvement was achieved by a fine-tuning of the neural language model. In general, the results achieved are encouraging and give us a starting point for further improvements.

## introduction detecting humor has become a popular research field at the same time that the bad phenomenon of offensiveness spreading exaggeratedly grows in social media. in this scenario it is very frequent to find out alarming volumes of heterogeneous data such as textual messages, images, advertisements, etc., that harm some age groups, ethnicity, sexual gender or other demographic characteristics ( betul @xcite . most of these harmful contents are often masquerade as innocent jokes or simply as a funny content. therefore, it is crucial to shed light on the commonalities and differences between both phenomena in order to properly addressing the challenge of computationally distinguishing humorous messages from aggressive or offensive ones. recognizing humorous and offensive utterances on written messages is a very difficult task for human beings and even more for computers @xcite . these difficulties increase when the textual messages are isolated from the context in which they are produced. additional knowledge from gestures, prosody features, visual content, situational environment and sociocultural rules play an important role in how humans properly understand the real meaning behind funny and hateful contents. all this makes humor recognition and offensiveness detection challenging tasks within natural language processing (nlp) and human-computer interaction (hci). on this line, the task 7, hahackathon: detecting and rating humor and offense at semeval-2021 aims at computationally recognizing humor and offensiveness in english tweets @xcite . to address the four subtasks launched in ha-hackathon we propose two distinct architectures which rely on neural language model based representation (deep-representation), particularly learned by transformer architectures. our first architecture combines the learned representation with a snn in order to learn in automatically way a metric for discriminating a pair of messages of the same class from a pair of messages of different classes. also, we considered applying a graph-based clustering method to each class independently for creating representative prototypes. these prototypes were used to build the training and testing pairs. our second architecture relied on the principle of fusing representations. for that, the deep-representation are mixed with linguistic information (linguistic-representation) and given as inputs to a linear regression model which is specialized in predicting humor and offensive scores. the paper is organized as follows: in section 2 we briefly introduce the description of the four subtasks. section 3 presents our proposed architectures and gives details about their modules. in section 4 are described the experiments and results. finally, we present our conclusions and provide interesting directions that we plan to explore in future work. the source code associated with this paper is online available on github: https: //github.com/mjason98/semeval21_humor. ## task descriptions we investigated the performance of our proposed architectures in the four subtasks introduced in ha-hackathon: i) given a tweet determining whether it is humorous or not (subtask 1a); ii) given a tweet predicting the humor rate in the range of 0 to 5, where 0 indicates that it is not a funny message and 5 indicates that the message is strong humorous (subtask 1b); iii) given a tweet determining whether it is considered as controversial (i.e., it is rated with highly variable values of humor from one annotator to another) (subtask 1c); the last subtask, iv) given a tweet predicting its offensiveness rating in a range of 0 to 5, where 0 indicates the tweet does not contain any kind of offensiveness and 5 indicates that the message is strong offensive (subtask 2). organizers provided a dataset for training and test labeled according to the objectives of each subtask. the whole dataset was manually annotated by several annotators in order to minimize the noise in the data and increase the agreement in the annotation procedure. the dataset is composed by 8000 tweets for training and 1000 tweets for testing purposes, respectively. the training set contains 3068 funny and 4932 non-funny tweets. this slight imbalance in the training set imposes an additional difficulty to the learning algorithm for accurately predicting the funny messages. the problem increases in the task of controversial humor prediction where only 2465 tweets are labeled as controversial and the remainder 5535 are noncontroversial. the most complex scenario regarding the data distribution is appreciated in the tasks of humor and offensiveness rating. at a first glance on the figure 1 can be inferred that the majority of the offensive scores are accumulated in the interval (0,1). as a consequence of that, the tweets which are not offensive at all or those with scores closer to zero are over-represented whereas the tweets with strong offensive content are under-represented in the dataset. therefore, from the learning perspective, it is more difficult to score tweets which strong offensive content. conversely to this scenario, the funniness scores are distributed more uniform. also, it is important to highlight that tweets with offensive scores greater than 0 in most cases also were scored as funny tweets. this relation reveals the usage of some humor devices as a way for masqueraded offensive messages. ## our proposals in this section we present the proposed models and provide details about their modules. our models have a modular structure. they are composed of both, an encoder module (encoder) and a prediction module (classifier), which are trained independently. particularly, we evaluate two distinct methods for the classification module. the first one is based on a siamese neural network and the second one relies on fusing representations and training a linear regression model. ## experiments and results in this section we describe the conducted experiments for evaluating the performance of our systems on hahackathon development dataset (devdataset). for that, we employed the metrics proposed by the task organizers, f1-score over the positive class and accuracy (acc) for classification subtasks, and the root mean squared error (rmse) for regression subtasks. ## conclusions and future work in this work we presented two models for addressing humor and offensiveness prediction in english tweets. both models employ the deeprepresentations learned by transformers methods for encoding the messages. the first model is based on a siamese neural network combined with a graph-based clustering method. the second model combines feature representations learned by three transformers language models with affective features captured by an bilstm-based model. these representations are used to train a linear regression model. the achieved results show that the siamese architecture outperformed the fine-tuned transformer models for humor detection task. the performance of this architecture relies on how the tweets are represented by the encoder and the strategy to find the positive and negative sets of prototypes. in the second model, the affective features play an important role to determine the offensiveness scores with any combination of features learned by the state-of-the-art language models, showing that they successfully captured underlying affective cues present in offensive and funny speech. we plan to investigate two interesting directions as future works. the first direction is an in-depth study of the harmfulness of humor on human stereotypes taking advantage of the over-lapping between offensiveness and humor in the hahackathon dataset. the second one is an exhaustive analysis of clustering methods for building prototypes and how they may influence the learning of the siamese neural network for humor prediction.