UMUT eam at S em E val-2022 Task 6: Evaluating Transformers for detecting Sarcasm in E nglish and A rabic (2022)

Abstract: In this manuscript we detail the participation of the UMUTeam in the iSarcasm shared task (SemEval-2022). This shared task is related to the identification of sarcasm in English and Arabic documents. Our team achieve in the first challenge, a binary classification task, a F1 score of the sarcastic class of 17.97 for English and 31.75 for Arabic. For the second challenge, a multi-label classification, our results are not recorded due to an unknown problem. Therefore, we report the results of each sarcastic mechanism with the validation split. For our proposal, several neural networks that combine language-independent linguistic features with pre-trained embeddings are trained. The embeddings are based on different schemes, such as word and sentence embeddings, and contextual and non-contextual embeddings. Besides, we evaluate different techniques for the integration of the feature sets, such as ensemble learning and knowledge integration. In general, our best results are achieved using the knowledge integration strategy.

## introduction sarcasm is a form of rhetorical device based on biting humour to disarm an opponent during a dialog @xcite . on the web and, specially in social networks and opinion forums, sarcasm is very popular because it is funny to read and helps to stimulate the viral phenomenon of social media content @xcite . as sarcasm usually relies on figurative language and wordplay, in which words diverts from their conventional meaning, sarcastic statements hinder the ability of automatic classification tasks to perform sentiment analysis, hate-speech detection, or author analysis among other tasks. from a natural language processing (nlp) perspective, sarcasm, among other forms of figurative speech, such as irony or satire, has been ex- * corresponding author plored in @xcite . in this work, the authors explored what stands out the most discriminant features for satire and irony detection. the authors identified a total of 25 feature sets. apart from lexicon-based features and wordn-grams features, such as unigrams or bigrams, the authors identified style features, sentiment and emotional features, pragmatic features, and punctuation features, to name a few. our proposal for solving sarcasm detection includes languageindependent feature sets extracted with a custom tool, umutextstats @xcite @xcite . in this work, we describe the participation of the umuteam at semeval 2022 task 6, concerning sarcasm identification in arabic and english @xcite . in this edition, three challenges are proposed. the first one is a binary classification problem to determine whether a document is sarcastic or not. this challenge is for english and arabic. the second challenge is available only in english, and consists in a multi-label classification task discerning among different types of ironic speech. finally, the last challenge consists of the identification of a sarcastic document between itself and a non-sarcastic rephrase, but with the same meaning. our team attemted to participate in the first and second challenge. we achieve a f1 score of the sarcastic class of 17.97% for english, and 31.75% for arabic in the first challenge. however, our results are not considered in the official leader board for the second challenge due to an unknown error. therefore, we report the results for the second challenge with the validation split of dataset. ## dataset the dataset proposed at isarcasm 2022 was annotated by the authors themselves. besides, each author was asked to rephrase their sarcastic texts without the usage of sarcasm. finally, some linguistic from the training set, we select a 20% of instances to build the validation set using stratified sampling, in order to keep the balance. ## system architecture figure 1 depicts the architecture of our proposal. basically, we build two systems: one for english and one for arabic, so we could apply different preprocessing techniques and apply language-specific pretrained embeddings. in a nutshell, this pipeline can be described as follows. first, is the preprocessing step module. for both languages, we ensure that the dataset does not contains hyperlinks, hashtags, quotations or emojis. plus, for the english dataset we expand acronyms. second, is the datasplitter, to divide the isarcasm dataset into training and validation. as there was a strong imbalance in the dataset, we keep this imbalance in both splits. third, is the feature extraction module, for extracting the language-independent linguistic features and the sentence embeddings. forth, is the training of several neural networks using hyperparameter selection. finally, is the feature integration module, in which we evaluate ensemble learning and knowledge integration in order to combine the results of each neural network. next, the feature extraction module is described. the first feature set is a subset of languageindependent linguistic features (lf) from umu-textstats. this feature sets includes part-of-speech (pos) features and stylometric features concerning several linguistic metrics such as type token ratio (ttr), punctuation symbols and corpus length. the second and third feature sets are, respectively, non-contextual word and sentence embeddings from fasttext @xcite . for this, we use the arabic and english pretrained models. the word embeddings allow to evaluate convolutional and recurrent neural network architectures, apart from multi-layer perceptrons that suitable for feature sets of fixed size. the forth feature set is contextual sentence embeddings. we use bert @xcite for english, and arabic bert @xcite for arabic. to extract these embeddings, we applied a similar method as described at @xcite . before extracting the sentence embeddings, we used ray-tune @xcite to fine tune bert and arabic bert. for this, we used tree of parzen estimators (tpe) to select the best hyperparameters from a total of 10 trials. the hyperparameters evaluated are: (1) the weight decay (between 0 and 0.3); (2) two training batch sizes: 8 and 16 (we were limited to the gpu); (3) four warm-up steps: 0, 250, 500, 1000; (4) the number of training epochs, between 1 and 5; and (5) a learning rate between 1e-5 and 5e-5. once the feature sets are extracted, the next step in the pipeline is the training of the neural networks. we train a neural network per feature set and a neural network combining all the feature sets, using a knowledge integration strategy. each training is performed with an hyperparameter optimisation stage. this evaluation includes 20 shallow neural networks, in which one or two hidden layers are stacked and that contains the same number of neurons. for the shallow neural networks we evaluate the following activation functions: (linear, relu, sigmoid, and tanh). we also evaluate 5 deep-learning networks, composed from 3 to 8 hidden layers, in which the number of neurons per layer are arranged different shapes (brick, triangle, diamond, rhombus, and funnel). the activation functions evaluated for the deep-learning networks are the sigmoid, tanh, selu and elu. we also adjust the learning rate to evaluate 10e-03 or 10e-04. as commented above, the word embeddings from fasttext allow us to evaluate 10 convolutional and 10 bidirectional recurrent neural networks. as the dataset was heavily imbalanced, we evaluate large batch sizes, so in all the experiments, we evaluate large batch sizes: 128, 256, and 512. we apply these large batch sizes for ensuring all batches contains sufficient number of instances of both classes. in addition, we apply regularisation by using a dropout mechanism ([false, .1, .2, .3]). besides, we evaluate two ensemble learning strategies to combine all the features. one method consisting of hard voting the labels predicted by each model (mode) and another method consisting of averaging the probabilities of the label of each model (mean). it is worth mentioning that one of the tasks of english language is a multi-label problem. to solve it, we trained several binary classification models, one per ironic speech label. ## results and validation during the development stage, we evaluate our models with a custom validation split. the results for the first task are depicted in first, in case of english, we can observe that the results for identifying sarcasm (f1-pos) are limited, reaching a f1-score of 45.82% with the knowledge integration strategy. we focus on this metric because the validation split is unbalanced, containing 173 sarcastic documents and 521 non-sarcastic documents. the results of the lf are limited in case of english. lf are based mostly on stylometric and pos features, which are not enough for the correct identification of sarcasm in english. moreover, the results concerning the sarcasm label achieved with the non-contextual word embeddings (we) are similar to the ones achieved with the lf (40.10% vs 40.08%), both slightly worse than non-contextual sentence embeddings (se, 42.27%). the results achieved with transformers (bf) are the best results achieved with the feature sets evaluated in isolation (f1-pos of 45.10%). when combining the results, we observe that we achieve slightly superior results by the knowledgte integration approach, improving to a f1-pos of 45.82%. however, the results are more limited with the the ensemble strategies, achieving the lower results of 38.06% (mode) and 37.96% (mean). the identification of the nonsarcasm label (f1-neg) is more similar (and even slightly superior) compared with the rest of the strategies. second, in case of arabic, we observe astonishing results for all feature sets. we reviewed the in fact, our validation split has 102 sarcasm documents whereas the rest (598) are non-sarcasm. the official test contains 1400 documents, 200 labelled as sarcasm whereas the rest were labelled as nonsarcasm. the results achieved with the rest of the feature sets se, we, and bf are in the same line, achieving the best result with a macro f1-score of 44.85% with the knowledge integration strategy. next, we report the results achieved for each trait separately in concerning the official results, we achieved very limited results for the binary classification challenge in english, achieving position 41 with a f1score of the sarcastic class of 17.97% (see the results achieved with the arabic dataset for the binary classification were better, with a f1score of the sarcastic label of 31.75%, reaching position 22 of a total of 32 participants. (see ## conclusions and further research lines in this working notes we have described the participation of the umuteam in the isarcasm shared task of semeval 2022. in this shared task, the participants were required to solve a binary and a multi-label classification task regarding sarcasm identification in english and arabic. we achieved a f1 score of the sarcastic class of 17.97% for english, and 31.75% for arabic in the first challenge. after sending the official results, we received the annotated test set. although we are happy with our participation in this shared task, as we have evaluated some of our methods, such as a subset of language-independent feature sets in arabic, we