SSN _ NLP at S em E val-2020 Task 7: Detecting Funniness Level Using Traditional Learning with Sentence Embeddings (2020)

Abstract: Assessing the funniness of edited news headlines task deals with estimating the humorness in the headlines edited with micro-edits. This task has two sub-tasks in which one has to calculate the mean predicted score of humor level and other deals with predicting the best funnier sentence among given two sentences. We have calculated the humorness level using microtc and predicted the funnier sentence using microtc, universal sentence encoder classifier, many other traditional classifiers that use the vectors formed with universal sentence encoder embeddings, sentence embeddings and majority algorithm within these approaches. Among these approaches, microtc with 6 folds, 24 processes and 3 folds, 36 processes achieve the least Root Mean Square Error for development and test set respectively for subtask 1. For subtask 2, Universal sentence encoder classifier achieves the highest accuracy for development set and Multi-Layer Perceptron applied on vectors vectorized using universal sentence encoder embeddings for the test set.

## introduction assessing the funniness of edited news headlines @xcite task of semeval-2020 deals with estimating the funniness of news headlines that have been modified by humans using a micro-edit to make them funny. the goal of the task is to make the machine predict whether the short edits on the given text makes them funny or not. thus, the system has to assess the intensity of humorness in the edited sentence. this system has various applications including humor generation where such a system can be used in a generate-and-test scheme to generate many potentially humorous texts and rank them in terms of funniness. this task has two sub tasks that include a regression task, in which the mean funniness of the edited headline has to be predicted and a classification task, in which the funnier of the two edited headlines has to be predicted. ## related work similar task called humor analysis based on human annotation (haha) has been part of iberlef 2018 @xcite and iberlef 2019 @xcite which deals with automatic detection and automatic rating of humor in spanish tweets. different methodologies have been reported in these tasks. convolutional rnn (giudice, 2019) have been used, in which input tweet is represented as a list with fixed length of sparse vectors and each vector of the list represents an individual character of the tweet and contains flags whose values are either 0 or 1 is fed to the neural network whose output was rounded to 0 or 1 to get a binary value and for the funniness score prediction subtask it was multiplied by five to get a value between 0 and 5. in a bert based approach @xcite a mutli-lingual model with modified hidden layers and output layers have been used. in bi-directional lstm approach @xcite , the manually-extracted features are passed through a neural network with sigmoid activation function. gaussian process @xcite have also been done in which vectors are obtained using pretrained spanish twitter embeddings and then trained and tested in the gaussian process regressor model. ingeotec @xcite have made use of evomsa, a multilingual sentiment classifier and microtc, a minimistic classifier tool that uses support vector machine. ## dataset description each subtask has a dataset with the instances described in sub task 1 dataset : @xmath0: 76 < @xmath1.2 inference : edited sentence : "us imposes metal tariffs on key holes" score : 0.2 sample instance : 10722-3702 < @xmath2: fed up @xmath3, some conservatives look to @xmath4.6 < @xmath5: fed up @xmath6, some conservatives look to @xmath7.4 < @xmath8 inference: edited sentence 1 : " i 'm done : fed up with pancakes , some conservatives look to texas" score of sentence 1 : 0.6 edited sentence 2 : " i 'm done : fed up with life, some conservatives look to texas" score of sentence 2 : 0.4 label : 1 the first subtask deals with predicting the humor level in the edited news headline. we have used microtc @xcite for predicting the funniness score with varying parameters such as number of folds and number of processes. microtc is a machine learning approach that classifies data based on svm linear kernel. it is a combinatorial function of four steps namely pre-processing, tokenizing, vector representation and classification using svm linear kernel. pre-processing includes removal of tags, punctuations, urls and converting them to lower case. tokenization is done by skip-gram and n-gram models and they are converted to vectors by tf-idf, tf methods and concatenating the maximum and minimum filters and then classifying the data. the edited sentences are given to the model in json file format. the format is as follows: sample input in training json file: {"text": "us imposes metal tariffd on key holes ", "decision function": 0.2, "klass": "1"} {"text": "for the first time in years shops have more guns than bullets ", "decision function": 0.0, "klass": "0"} ## conclusion assessing the funniness of edited headlines have been approached using different traditional approaches that include microtc, various classifiers namely mlp, svm, decision tree, knn, random forest, gaussian naive bayes and adaboost classifier that classifies the sentences vectorized using universal sentence encoder embeddings and sentence transformers, an ensemble method method of bert classifier and microtc, universal sentence encoder classifier and a majority algorithm among them. for subtask 1, microtc with 6 folds, 24 processes and microtc with 3 folds, 36 processes achieve the least root mean square error for development set and test set respectively. for subtask 2, universal sentence encoder classifier achieves the highest accuracy for development set and multi-layer perceptron applied on vectors vectorized using universal sentence encoder embeddings for the test set. the performance may be further improved by using other embeddings and by fine tuning the microtc with other parameters.