Predict and Use: Harnessing Predicted Gaze to Improve Multimodal Sarcasm Detection (2023)

Abstract: Sarcasm is a complex linguistic construct with incongruity at its very core. Detecting sarcasm depends on the actual content spoken and tonality, facial expressions, the context of an utterance, and personal traits like language proficiency and cognitive capabilities. In this paper, we propose the utilization of synthetic gaze data to improve the task performance for multimodal sarcasm detection in a conversational setting. We enrich an existing multimodal conversational dataset, i.e., MUStARD++ with gaze features. With the help of human participants, we collect gaze features for 20% of data instances, and we investigate various methods for gaze feature prediction for the rest of the dataset. We perform extrinsic and intrinsic evaluations to assess the quality of the predicted gaze features. We observe a performance gain of up to 6.6% points by adding a new modality, i.e., collected gaze features. When both collected and predicted data are used, we observe a performance gain of 2.3% points on the complete dataset. Interestingly, with only predicted gaze features, too, we observe a gain in performance (1.9% points). We retain and use the feature prediction model, which maximally correlates with collected gaze features. Our model trained on combining collected and synthetic gaze data achieves SoTA performance on the MUStARD++ dataset. To the best of our knowledge, ours is the first predict-and-use model for sarcasm detection. We publicly release the code, gaze data, and our best models for further research.

## introduction sarcasm originates from the greek word sarkasmós adapted from sarkázein, which means a sneering or cutting remark. sarcasm depends on "bitter, caustic, and other ironic expressions that are usually directed against an individual." @xcite . it 1 https://www.cfilt.iitb.ac.in/emnlp23sarcgaze is a complex linguistic phenomenon that gets expressed with words that mean the opposite of what the speaker intends to say; e.g., i love being ignored expresses the bitterness of the speaker. the roots of sarcasm lie in incongruity @xcite , which makes computational sarcasm detection a challenging problem; and the nlp community has attempted to tackle this problem using innovative approaches. sarcasm detection in the text has largely been attempted by focusing on lexical indicators @xcite , sentiment incongruity @xcite , etc., in both rule-based and learning-based systems @xcite . however, sarcasm is also expressed through tonal changes and/or facial expressions. hence researchers have started investigating modalities other than text, viz., audio and video, to help detect sarcasm @xcite @xcite @xcite . @xcite observed that gaze features are helpful in detecting sarcasm within short sentences without context, which is our inspiration. in a conversational setting, sarcasm often results from an earlier utterance, which is the problem we focus on in this work. to the best of our knowledge, ours is the first attempt at multimodal detection of sarcasm using gaze behaviour in a conversational setting. ## related work existing studies demonstrate how cognitive features have been used to improve performance for various nlp tasks. user understandability of sarcasm can be evaluated with the help of gaze behaviour @xcite , where incongruity in the text induces gaze behaviour characterized by longer fixation durations, repeated regressions, and also scan path complexity @xcite . previously, sarcasm detection based on only textual input has shown minor improvements with the help of gaze-based features @xcite . gaze behaviour has also been used to identify a reader's native language @xcite , as well as to detect grammatical errors in compressed sentences @xcite . @xcite also show that gaze behaviour can be used to evaluate the output of machine translation systems better than automated metrics. similarly, gaze-based features have also been shown to help the task of cognate and false friends' detection @xcite . gaze behaviour has also been used to evaluate how a reader would rate the quality of a piece of text @xcite . similarly, @xcite also perform the task of essay grading in a zero-shot setting using only gaze-based features and show the efficacy of gaze-based features for performing nlp tasks @xcite . however, existing research does not discuss the correlation of multimodal features (like visual and audio) with gaze-based features, and does not investigate these features for multimodal sarcasm detection in a conversational setting. in the subsection below, we discuss the literature on multimodal studies in nlp. lack of data has been a common problem in cases of both sarcasm as well as cognitive nlp. numerous efforts have been made in building gaze feature predictors in order to reduce dependency on gold gaze data by producing high quality synthetic gaze data. study in takmaz (2022) utilizes "adapter" in a language model to match the results of a fully fine tuned language model for predicting eye tracking features with a highly efficient network in terms of the number of parameters. @xcite propose a bi-lstm-based network that, with the help of a few psycho-linguistic features, predicts eye tracking features. the paper states that the readability of a text reflected in the linguistic features is important to predict eye movement patterns @xcite . the creation of synthetic gaze data has also been performed in multilingual settings. in srivastava (2022), a model trained on a completely different set of languages predicts gaze data for a completely new language. ## dataset and gaze annotation mustard++ is a multimodal dataset that consists of textual utterances with context, audio, and video from a corresponding clip. this data has been acquired from publicly available sources for five television shows: friends, the big bang theory (seasons 1-8), the golden girls, burnistoun, and the silicon valley. each dialogue is presented as a combination of the main 'utterance' and the 'context' in which it was uttered. it contains a total of 1,202 instances, out of which 601 are sarcastic, and 601 are non-sarcastic. along with sarcasm annotation, the dataset also provides additional information like an emotion class, valence, arousal, and sarcasm type. we chose this dataset for our experiments and performed gaze annotation on 231 samples, where 129 are sarcastic, and 102 are nonsarcastic. to avoid any skew, the sarcastic instances are chosen to encompass all four types of sarcasm with a distribution similar to the one in the source data from mustard++. the selected instances include dialogues with short contexts (in the range of 2-5 speaker turns) as well as long contexts (6-13 speaker turns). ## our approach an architecture diagram for our setup is shown in figure 3 . we reproduce the multimodal sarcasm detection experiments as in @xcite as the baseline, with the addition of gaze features as the fourth modality in addition to text, speech and visual modalities. for textual features, we utilize the pre-trained bart language model @xcite and obtain embeddings for both textual utterance and the context from the dialogue. bart provides a feature vector representation x @xmath0. we encode the text using the bart large model with d @xmath1. for audio features, like in ray et al. ( 2022 ), we sampled the audio signal at 22.5khz as a preprocessing step. since the audio has background noise and canned laughter (we deal with sitcoms), we used the vocal-separation method foot_0 to process it. we extract three low-level features: mel frequency cepstral coefficients (mfcc), mel spectrogram (using librosa library (mcfee et al., 2022)), and prosodic features using opensmile foot_1 . we split the audio signal into equal segments of 1-second duration each to maintain consistent feature representation in all instances. since the audio signal length varies with utterances, this segmentation helps in keeping the vector size constant across the dataset. for each segment, we extract mfcc, mel spectrogram and prosodic features of size d m , d s , d p respectively. then we take the average across segments to get the final feature vector. here d @xmath2, d @xmath3, d @xmath4, so our audio feature vector is of size for visual features from the videos, we use a pool-5 layer from pre-trained resnet-152 @xcite image classification model. to improve the video representation and reduce noise, we extract the keyframes to be passed to resnet-152. the computer vision community widely uses key frame extraction, which is defined as the frames that form the most appropriate summary of a given video @xcite . we use an opensource tool, katna 4 , to perform key-frame extraction. for the final feature vectors, we average the vectors of each key frame of an instance (context and utterance) extracted from resnet-152. the size of the final video feature representation is for gaze-based features, we obtain a total of 31 features from the sr research experiment dataviewer software foot_3 , but do not use them all for our experiments. we employ the kbest feature selection method from the scikit-learn library @xcite to optimize the features. given the sarcasm label along with gaze-based features, the method resulted in the selection of a total of 25 correlated gaze-based features. due to space constraints, we provide this list in appendix c with a feature description in ## conclusion and future work this paper discussed the use of gaze-based features for the task of sarcasm detection in a multimodal and conversational setting. we propose the use of textual, audio, and video in combination with the gaze modality by showing a substantial improvement in performance with the addition of collected gaze-based features. we collect gaze data over a small number of samples and predict these features for a larger portion of the data, both of which we will release with the code and the best models from our experiments. with predicted gaze-based features, however, we observe a small improvement in the task performance in this case. to the best of our knowledge, our results indicate that adding collected gaze-based features certainly improves task performance in every feature combination, proving the efficacy of gaze-based features. our qualitative analysis also suggests that better audio and visual features should help improve task performance. in future, we would like to improve the quality of predicted gaze-based features further in a multi-task setting of sarcasm detection and gaze prediction.