¡ Q ué maravilla! Multimodal Sarcasm Detection in S panish: a Dataset and a Baseline (2021)

Abstract: We construct the first ever multimodal sarcasm dataset for Spanish. The audiovisual dataset consists of sarcasm annotated text that is aligned with video and audio. The dataset represents two varieties of Spanish, a Latin American variety and a Peninsular Spanish variety, which ensures a wider dialectal coverage for this global language. We present several models for sarcasm detection that will serve as baselines in the future research. Our results show that results with text only (89%) are worse than when combining text with audio (91.9%). Finally, the best results are obtained when combining all the modalities: text, audio and video (93.1%). Our dataset will be published on Zenodo with access granted by request.

## introduction figurative language is one of the most difficult forms of natural language to model computationally and there have been several studies in the past focusing on its subcategories such as metaphor interpretation @xcite , humor generation @xcite and analyzing idioms @xcite . sarcasm is one of the extreme forms of figurative language, where the meaning of an utterance has little to do with the surface meaning (see @xcite . understanding sarcasm is difficult even for us humans as it requires certain mental capacities such as a theory of mind (see @xcite and it is very dependent on the context and speaker who is being sarcastic. there are also very different view to sarcasm in the literature, for example, according to @xcite sarcasm requires an allusion to a failed expectation and pragmatic insincerity (see @xcite to be present in the same time. however, @xcite highlights that these two preconditions are not enough, as sarcasm needs an ironic context to take place. @xcite argues that, in the context of sarcasm, the meaning difference can either be the complete opposite of the semantic meaning of a sentence or somewhat different as seen in the lexical opposition of the words and the intended meaning. the fact that there are several different theoretical ways of understanding sarcasm, highlights the complexity of the phenomenon. in this paper, we present an audio aligned dataset for sarcasm detection in spanish. the dataset containing text and video timestamps has been released openly on zenodo foot_0 . an access to the dataset with the video clips foot_1 can be granted upon request for academic use only. in addition, we will present a baseline model for this dataset to conduct multimodal sarcasm detection in spanish. ## related work in this section, we will present some of the recent related work on sarcasm detection. there has been some work also on sarcasm generation @xcite and interpretation @xcite , but they are rather different as tasks and we will not discuss them in detail. @xcite show an approach for sarcasm detection in online reivews. they train a cnn (convolutional neural network) based model on separate feature embeddings for sarcasm, humor, sentiment and hate speech. similarly, @xcite also detect sarcasm in text. they combine an lstm (long short-term memory) model with bert. dubey et al. (2019) approach. @xcite present a multimodal sarcasm dataset in english. the dataset consists of annotated videos from tv sitcoms such as friends and the big bang theory, apart from being in english instead of spanish, the main difference is that our dataset consists of animated cartoons instead of tv shows played by real people. another big difference is in the data collection as they opted for querying sarcastic video clips, where as the data we work with represents full episodes. @xcite use this data and present a multimodal sarcasm detection framework based on a bi-gru model. many of the related work has been focusing on text only. research on multimodal approaches has been carried out only for english data, not unlike the textual approaches. ## dataset we base our work on the sarcasm annotated dataset from the ma thesis of the second author of this paper @xcite foot_2 . this dataset is based on two episodes of south park with voice-overs in latin-american spanish and two episodes of archer with voice-overs in spanish of spain. the dataset has the speaker, their utterance and sarcasm annotations for each utterance in all of the episodes. however, the released data has been released shuffled for copyright reasons and it contains text only. unlike the recent multimodal dataset for english @xcite , this data is expert annotated according to several different theories on sarcasm. annotation based on theories is important in order to establish a concrete meaning for sarcasm and to avoid the same mistakes as @xcite had. in their paper, they report that the most sarcastic character in the big bang theory is sheldon, however this cannot be true as one of the main characteristics of sheldon is that he does not understand sarcasm. therefore, their annotations ignore the fundamentally important characteristic of sarcasm, which is speaker intent, and rather they consider sarcasm purely based on subjective intuition. in order to produce a multimodal dataset out of the existing one, we locate the corresponding videos for the annotations and manually align them with the video. we use our own in-house tool jus-tannotate for this task foot_3 . this was a time consuming task, but as a result we ended up with a highquality dataset with audio, video and text aligned. while aligning the dataset, we found several errors in the original transcriptions that we fixed. we did not alter the sarcasm annotations. in addition to the alignment, we introduced scene annotations. an episode of a tv show consists of many different scenes, and sarcasm is typically highly contextual, we indicate in the data which utterances belong to the same scene to better capture the context of each utterance. figure 1 : archer uttering a sarcastic sentence that goes against the common sense figure 1 shows an example of a scene in the corpus. in this particular scene, archer asks sarcastically ¿dónde se compra la leche materna? (where does one buy breast milk?). this is an example of sarcasm in the corpus where sarcasm violates common sense. depending on the speaker, the utterance might be sarcastic or the speaker might lack knowledge on the topic. in figure 2 cartman comments on the neckpiece of stan by saying esas corbatas están de moda, tiene suerte de tenerla (those neckpieces are fashionable, you are lucky to have one). this is an example of a very different type of sarcasm that cannot be detected just by having common knowledge about the world. in order to understand the sarcastic intent, a system would need to have an access to the video as well to detect the unfashionable neckpiece and the disappointed facial expression of stan. ## method in this section, we present our method for detecting sarcasm in the multimodal dataset. we experiment with text only, text and audio and all modalities. all models are trained by using the same random train (80%) and test (20%) splits. for the neural model, 10% of the training split is used for validation. ## results in this section, we report the accuracy of predictions by the neural model and the three svm models that are based on 1) text only, 2) text and audio, and 3) text, audio and video. the results can be seen in ## conclusions we have presented the first multimodal dataset for detecting sarcasm in spanish. the dataset has been released on zenodo. our initial results serve as a baseline for any future work on sarcasm detection on this dataset. based on the results, it is clear that multimodality aids in detecting sarcasm as more contextual information is exposed to the model. despite the improvements when considering multiple modalities, sarcasm detection is a very difficult task to model as it demands a global understanding of the world and the specific context the sarcastic utterance is in, as discussed in our error analysis. even though the overall accuracy is high, it is clear the model makes errors that indicate that it has learned the data, but not the phenomenon.