I 2 C at S em E val-2022 Task 6: Intended Sarcasm in E nglish using Deep Learning Techniques (2022)

Abstract: Sarcasm is often expressed through several verbal and non-verbal cues, e.g., a change of tone, overemphasis in a word, a drawn-out syllable, or a straight looking face. Most of the recent work in sarcasm detection has been carried out on textual data. This paper describes how the problem proposed in Task 6: Intended Sarcasm Detection in English (Abu Arfa et al. 2022) has been solved. Specifically, we participated in Subtask B: a binary multi-label classification task, where it is necessary to determine whether a tweet belongs to an ironic speech category, if any. Several approaches (classic machine learning and deep learning algorithms) were developed. The final submission consisted of a BERT based model and a macro-F1 score of 0.0699 was obtained.

## introduction existing social media analysis systems are limited by their inability to accurately detect and interpret figurative language. sarcasm is often used by individuals to express opinions on complex matters and regarding specific targets @xcite . early computational models for verbal and irony and sarcasm detection have relied on shallow methods exploiting conditional token count regularities. but lexical clues alone are insufficient to discern sarcasm intent. appreciating the context of expression is critical for this; even for humans @xcite . indeed, the exact same sentence can be interpreted as literal or sarcastic, depending on the speaker. consider the sarcastic tweet in figure 1 (ignoring for the moment the attached #sarcasm hashtag). without knowing the author's political inclination, it would be difficult to conclude with certainty whether the tweet was intended as sarcastic or not. this task is about the binary classification of tweets in english based on the category of ironic speech. as a multilabel-classification, a tweet can belong to multiple categories or none. this research is important because social networking sites have changed our lives in recent years. for companies, this social reality has turned into an obligation to set up communication and marketing channels through social networks. besides, companies require feedback from consumers, through comments or messages that mark the acceptance or rejection of each of the proposals, products or services. for this competition 2 models/strategies were used: i2c at semeval-2022 task 6: intended sarcasm in english using deep learning techniques adrián moreno monterde laura vázquez ramos escuela técnica superior de ingeniería escuela técnica superior de ingeniería universidad de huelva (spain) universidad de huelva (spain) figure 1: an illustrative tweet. in this task we have analysed the sarcastic behaviour of people on social networks, in this case twitter, and the ways in which people express it. the dataset proposed by the organization was very unbalanced, so we had to apply several data balancing techniques in order to achieve good results. the rest of this paper is organized as follows: in section 2 we explain the dataset, the meaning of the labels and data distribution. therefore, we refer to other research that has helped us in our approach to this one. in section 3, several techniques and methods applied to our models to improve their performance and metrics are described. in section 4 we explain the libraries used and their usefulness. finally, in section 5, the scores obtained with our proposed approaches are presented. ## background as mentioned above, this paper is focused on subtask b: binary multilabel classification. the original dataset has 3467 tweets in english with a maximum size of 280 characters (tweet limit). as a matter of fact, only the first 867 tweets will be useful for our task because the rest of the tweets do not have the labels that we are going to work with. furthermore, the columns "unnamed:0", "rephrase" and "sarcastic" have been removed because they are useless for our task perfomance. for a better analysis and understanding of the multiple labels that we have in our dataset it is important to mention the ironic speech exposed in @xcite : but are not obviously critical towards an addressee. 3. satire: tweets that appear to support an addressee but contain underlying disagreement and mocking. 4. understatement: tweets that undermine the importance of the state of affairs they refer to 5. overstatement: tweets that describe the state of affairs in terms that are obviously exaggerated. 6. rhetorical question: tweets that include a question whose invited inference (implication) is obviously contradicting the state of affairs. in figure 2 , two samples of the dataset with the information used in our approaches can be seen. when it comes to the multiple labels (categories) of sarcasm, this challenge has been approached by different researchers. in @xcite , experiments with semi-supervised sarcasm identification on a twitter dataset (5.9 million tweets) were carried out using 50 twitter tags and 15 emojis as sentiment labels. they used a 5-fold cross validation on their classifier getting a f1-score of 0.55. in addition, in @xcite , they propose a semi supervised system for sarcasm recognition over 66,000 products reviews from amazon. they used the same strategy as in the previous mention and obtained and f-score of 0.83 on the product reviews dataset. in (khatri, p, pranav, y m, dr. anand kumar 2020), a model using machine learning techniques with bert and glove embeddings to detect sarcasm in tweets was proposed. ## system overview this section describes the two types of models that were submitted and the techniques and methods applied to each model to improve their performance and metrics. ## experimental setup to obtain the above models, some libraries were used: for all the experiments, tweets were preprocessed and then they were randomly split using a stratified method (80% training and 20% testing). that means that the proportion of values in the samples produced is the same as the proportion of values provided for the parameter to stratify. during the training phase of the competition, we have only focused on the macro-f1 score as the key metric of the subtask b. 5 results two submissions were sent using bayesian networks: the first one with the text processing v1.0 and the second one with the text processing v3.0. regarding the final submission using bertbase-uncased, different approaches were used. augmentation in minority class only. taking a look at according to the official metrics, as was mentioned before, we achieved a macro f1-score of 0.0699 and we were ranked 10 th among 22 teams that participated on this subtask. analyzing our systems, we can state that the main problem found in the subtask was the lack of data towards unbalanced data at the dataset, which is why we have been constantly applying data augmentation on the minority classes and even inserting data from an external database. applying these two techniques, a big improvement in the performance of our systems can be seen. furthermore, our research shows that any kind of preprocessing technique is mostly useless because any character, capital letter, overextended word or symbol, could be the determining factor in recognizing ironic speech. ## conclusion in this paper our approach to solve task 6 (isarcasmeval) -subtask b: given a text, determine which ironic speech category it belongs to, if any; in english, has been described. our best result was reached with a deep learning algorithm (bert) model, with which we achieved a macro f1-score of 0.0699. we obtained the 10th position in the ranking. for future works, an improved version of our bert model could be developed by training with a bigger dataset. it is also possible to look for new preprocessing techniques that enable the removal of information that is useless to the meaning of the tweet but still maintain the ironic speech patterns (if any).