UMUT eam at S em E val-2021 Task 7: Detecting and Rating Humor and Offense with Linguistic Features and Word Embeddings (2021)

Abstract: In writing, humor is mainly based on figurative language in which words and expressions change their conventional meaning to refer to something without saying it directly. This flip in the meaning of the words prevents Natural Language Processing from revealing the real intention of a communication and, therefore, reduces the effectiveness of tasks such as Sentiment Analysis or Emotion Detection. In this manuscript we describe the participation of the UMUTeam in HaHackathon 2021, whose objective is to detect and rate humorous and controversial content. Our proposal is based on the combination of linguistic features with contextual and non-contextual word embeddings. We participate in all the proposed subtasks achieving our best result in the controversial humor subtask.

## introduction in this manuscript we describe our participation in the shared task hahackathon 2021 @xcite , proposed in the forum for information @xcite whose objective is the identification and evaluation of humorous and offensive texts written in english. humor allows you to present the reality by highlighting the comic and ridiculous side of life. humor is hard to identify, even for humans @xcite . on the one hand, there are many forms of humor: from mild forms, such as jokes or puns, that result in better and safer social environments, to biting forms, like sarcasm, which is used as a rhetorical device. humor can also have a constructive end, as happens in satire, where irony, double meaning, and sharp analogies are used to ridicule someone or something. on the other hand, the sharper the humor, the more effort it takes to understand it. when humor is misunderstood, or when humor itself has a transgressive purpose, it can lead to controversies and confrontations. furthermore, an added difficulty is that humor is highly subjective and context dependent, making it even more difficult to understand @xcite . nevertheless, the benefits of endowing to a machine basic notions about humor and figurative language understanding outweigh the challenges they pose, because they lead to natural language-based interfaces to feel more naturally, such as chat-bots and virtual assistants @xcite and more reliable results in tasks such as opinion mining. we participate in all the proposed subtasks of hahackathon'2021 with two systems that combines linguistic features (lf) extracted with a tool developed by our research group with (1) pretrained word embeddings (pwe) from fasttext @xcite and glove @xcite , and (2) contextual word embeddings from bert @xcite . our best result was achieved in task 1c (controversial humor), in which we trained a classification neural network that distinguishes between non-humor, humor, and offensive but outputs a binary prediction which indicates whether a text is controversial or not. the code is available at https: //github.com/smolky/hahackathon-2021. ## background the hahackathon 2021 challenge consists in two binary classification problems of humorous content and controversial tweets and two regression problems to identify how humorous and controversial the annotators considered the texts. we only used the dataset given by the organizers, two pretrained word embeddings models, and the contextual word embeddings from bert. the dataset provided consisted in 10k tweets written in english and posted in three subsets, namely training, development, and testing, with a ratio of 80-10-10. all tweets were annotated by us english-speaking annotators of different genders and age groups with the following questions: most of the literature found on the detection of humor highlights the importance of figurative language in which, contrary to the literal sense of language, words and expressions change their conventional meaning to refer to something without saying it directly (del pilar salas-zárate et al., 2020). the reader can find in that work a compendium of works that analyze sarcasm, irony, and satire in english. modern approaches rely on novel deeplearning transformers, such as the work described in @xcite , focused on sarcasm from a figurative language perspective, and in which the authors used bert to build an aspect-based sentiment analysis system to determine whether the response is sarcastic. other works, such as the one described in del pilar @xcite , explores the differences and similarities in how satire is perceived in countries that share the language, but not the cultural background. to do this, they compare the use of linguistic characteristics with a corpus of satirical news written in spanish from spain and another written in spanish from mexico. ## system overview our proposal is based on the usage of lf, pwe, and cwe to detect humor and controversial content. cwe have outperformed previously state of the art results regarding text classification tasks. however, we state that both pwe and cwe ignore relevant clues that are present in writings. for example, we observed in the dataset provided, the presence of capital letters (commonly used when shutting or to raise the voice), quoted sentences or dialogues that reproduce real or figurative conversations that are not captured with any of the above techniques. to obtain the lf, we use a subset of the features extracted with umutextstats @xcite . this tool is inspired in liwc (tausczik and pennebaker, 2010) but designed by our research group from scratch for the spanish language. as the hahackathon 2021 dataset only deals with english, we only extract statistical features discarding all features that we had that work with spanish lexicons. specifically, we select: the next step was to obtain the best deeplearning architecture for each subtask. we evaluate two main approaches. on the one hand, we combine the lf with pwe and different deep-learning architectures. on the other, we combine the lf with cwe from bert (bert-base-uncased) using huggingface foot_0 . after performing these two models, we sent our results to the platform to evaluate them with the development dataset, achieving our best result with bert for subtask 1a and rnns for subtasks 1b, 1c, and 2a. after performing these two models, we sent our results to the platform to evaluate them with the development dataset, achieving our best result with bert for subtask 1a and rnns for subtasks 1b, 1c, and 2a. ## experimental setup our experimental setup is depicted in figure 1 and, in a nutshell, we can described as follows. first, we perform a preprocessing stage that consist of: second, as the organizers of the shared task released the labels of the development dataset in the last stage of the competition, we begun the competition by dividing the 8k tweets of the training data into two folds in a ratio of 80-20. third, two main approaches were evaluated. on the one hand, we use talos foot_1 to evaluate different pretrained word embeddings models (fasttext, glove, and word2vec) and several neural networks architectures (mlp, cnn, lstm, bilstm, gru, and bigru). on the other, we evaluate contextual word embeddings from bert. these two processes are described in detail in the next paragraph. finally, the classification subtasks (1a and 1c), the results are evaluated with the accuracy and the f1-measure whereas the regression subtasks (1b and 2a) are evaluated with the root mean squared error (rmse). in case of bert, we proceed as follows: we fine tune bert with the training split of the ha-hackathon 2021 dataset for 2 epochs and a batch size of 64. then, we freeze the resulting model and concatenate it to the lf in a new model composed of two hidden layers of 32 and 16 neurons respectively and trained for 10 epochs. in case of pwe, we evaluate word embeddings from fasttext and glove but also we evaluate that the weights of the embeddings were learned from scratch in the embedding layer. next, the lf and the word embeddings are concatenated, and used as input to a mlp, in which we evaluated (1) the number of hidden layers (between 1 and 8), (2) the number of neurons @xcite @xcite @xcite , and (3) the shape of the network (funnel, rhombus, long funnel, brick, diamond, and triangle). we also evaluate the dropout rate to avoid overfitting (0, 0.2, 0.5, and 0.8), the activation function of the hidden layers (relu, sigmoid, tanh, selu, and elu), the learning rate, and the batch size (16, 32, and 64). each combination of parameters was evaluated during 1000 epochs with an early stopping mechanism. due to time constraints, we evaluated only 1000 combinations of these hyperparameters randomly selected. ## results first, the best hyper-parameter combinations (validation set) are shown in during the hyper parameter tuning stage, we evaluated the reliability of the pwe without the lf with our custom training and evaluating splits. the results in our development dataset were slightly better with the combination of both feature sets in three subtasks: 83.516% (lf+pwe) vs 83.379% (pwe) of accuracy in subtask 1a, 0.79820 vs 0.81958 of rmse in subtask 1b, and 61.125% (lf+pwe) vs 60.688% (pwe) of accuracy in subtask 1c. however, in subtask 2a, pwe performed better without lf: 0.68037 (pwe) vs 0.70100 (lf+pwe). ## conclusions while we are pleased with our participation since it has given us the opportunity to evaluate novel tech-