Tsia at S em E val-2021 Task 7: Detecting and Rating Humor and Offense (2021)

Abstract: This paper describes our contribution to SemEval-2021 Task 7: Detecting and Rating Humor and Of-fense.This task contains two sub-tasks, sub-task 1and sub-task 2. Among them, sub-task 1 containsthree sub-tasks, sub-task 1a ,sub-task 1b and sub-task 1c.Sub-task 1a is to predict if the text would beconsidered humorous. Sub-task 1c is described asfollows: if the text is classed as humorous, predictif the humor rating would be considered controver-sial, i.e. the variance of the rating between annota-tors is higher than the median.we combined threepre-trained model with CNN to complete these twoclassification sub-tasks. Sub-task 1b is to judge thedegree of humor. Sub-task 2 aims to predict how of-fensive a text would be with values between 0 and5.We use the idea of regression to deal with thesetwo sub-tasks. We analyze the performance of ourmethod and demonstrate the contribution of eachcomponent of our architecture. We have achievedgood results under the combination of multiple pre-training models and optimization methods.

## introduction humor is an intellectual activity that can cause certain emotions in human thinking. humor is not only very important but also very common in daily life. people's research on humor has involved many fields such as psychology, sociology, linguistics and so on.of course, it also has special value for the research of computing languages.because of its complexity and inherent subjectivity, the development of automatic humor recognition and assessment poses a great challenge in computational linguistics, and therefore is a popular subject in various shared task competitions. @xcite )however, we must also recognize the difficulties of humor research.first of all, although humans can easily judge whether a sentence is humorous in daily life, but due to humor is restricted by geography, environment, social background and other aspects, we usually not only pay attention to this sentence or whether the matter is humorous, we have to figure out how humorous or funny this context is? in other words, we pay more attention to its degree of humor which is not so easy for computers.and because humor is affected by the environment, different people have different understanding of humor. just like sometimes your humor is based on the suffering of others. things you find funny, but others don't necessarily find them funny. in other words, humor is controversial. so we have to determine the specific humor rate. this task is to take a median as the criterion for humor. it is also difficult to judge the humor of this dispute by computer language. more recently, some humorous sentences can also have derogatory and offensive elements. whether humor can cause offense is also one of the researches in this thesis. i believe that the study of humor not only helps to improve the computer's understanding of humor in certain aspects, but also purifies our network environment. the four sub-tasks of semeval-2021 task7 are designed to solve the above problems. for deep learning, the computer must not only judge whether a sentence is humorous.it is more important to understand this humorous sentence.in our paper, we designed two effective systems to solve the above four sub-tasks. for sub-task 1a and sub-task 1c, we take them as a binary classification task.we designed an efficient system using the idea of bert-cnn.this is not a new idea because people have tried in the past.we also use the other popular pretraining models,including the derived albert and roberta.for sub-task 1a, we need to judge whether a sentence is humorous.we predict a label l: where @xmath0, not is humor -0}.for sub-task 1c, we also need to judge whether a sentence is controversial and predict a label l: where @xmath1, not humor controversy -0}.for sub-task 1b, we combined regression ideas with the current popular pre-training model to complete these two subtasks.the two input sentences were split into two lists and fed into the regression model, which made a prediction about the funniness of each sentence.then we compared the results of the prediction to determine the funnier of the two sentences.we compare the humor between each sentence and finally return a humor value. @xcite finally, the humor rate and the level of controversy are mapped to the range from 0 to 5.we use mean square error to measure these two tasks. ## background the judgment of humor is the same as other text classification problems in natural language processing.the most important thing is to find suitable features to represent sentences.the task is to assign predefined categories to a given text sequence. many work has shown that pre-trained models on large corpora are beneficial for text classification and other nlp tasks, which can avoid training new models from scratch.since 2013,people have proposed some word embedding approaches such as word2vec @xcite and glove @xcite .however,because their word embeddings are all in the same space, they cannot express the role of polysemy.in other words,they are non-contextual embedding, they cannot capture the high-level concepts of sentences, such as semantics and context @xcite .later, someone proposed the elmo model to solve this problem.compared with word2vec and glove, elmo captures contextual information and not just individual information of words. in word2vec, the vector representations of words are completely consistent in different contexts, and elmo is optimized for this @xcite . more recently, pretrained language models have shown to be useful in learning common language representations by utilizing a large amount of unlabeled data: such as openai @xcite and bert @xcite . bert is based on a multi-layer bidirectional transformer @xcite and is trained on plain text for masked word prediction and next sentence prediction tasks.this paper tried other two new pre-training models of al-bert @xcite and roberta @xcite based on bert. and we fine-tuned down-stream tasks for a variety of pre-trained models. finally,we completed these four sub-tasks effectively. ## experimental setup the code this time is mainly based on transformers under hugging face. the neural network tool we use is pytorch.for sub-task 1a and sub-task 1c,we use the same method a(the same structure).we just read different id of training set.sub-task 1b and sub-task 2 used the other method b. ## results for sub-task 1a and sub-task 1c, we first tried bert-base as our word embedding model. on this basis, we added lstm @xcite to further extract the features of words. long-term short-term memory (lstm) network has the ability to maintain long-term memory. the ability of has proven to be particularly useful for learning sequences containing long-term patterns of unknown length.we also tried two other pre-training models (albert and roberta) as a comparative experiment. these two new language models have made some improvements on the basis of bert, but they have different effects on different data sets. roberta is suitable for complex neural network architecture, albert architecture is relatively streamlined. from the data in ## conclusion in this paper,we gave a description of the bert+cnn architecture and the popular roberta pre-training model architecture, and finally solved four sub-tasks. the best f1 for sub-task 1a is 0.9206 and the best f1 for sub-task 1c is 0.6744. the best root mean square errors of sub-task 1b and sub-task 2 are 0.6510 and 0.5588. our four sub-tasks all appeared on the leaderboard(note: the data from our computer is a little different from the official evaluation results.the results of the final four sub-tasks in the leadboard are shown in