All-in-One: A Deep Attentive Multi-task Learning Framework for Humour, Sarcasm, Offensive, Motivation, and Sentiment on Memes (2020)

Abstract: In this paper, we aim at learning the relationships and similarities of a variety of tasks, such as humour detection, sarcasm detection, offensive content detection, motivational content detection and sentiment analysis on a somewhat complicated form of information, i.e., memes. We propose a multi-task, multi-modal deep learning framework to solve multiple tasks simultaneously. For multi-tasking, we propose two attention-like mechanisms viz., Inter-task Relationship Module (iTRM) and Inter-class Relationship Module (iCRM). The main motivation of iTRM is to learn the relationship between the tasks to realize how they help each other. In contrast, iCRM develops relations between the different classes of tasks. Finally, representations from both the attentions are concatenated and shared across the five tasks (i.e., humour, sarcasm, offensive, motivational, and sentiment) for multi-tasking. We use the recently released dataset in the Memotion Analysis task @ SemEval 2020, which consists of memes annotated for the classes as mentioned above. Empirical results on Memotion dataset show the efficacy of our proposed approach over the existing state-of-the-art systems (Baseline and SemEval 2020 winner). The evaluation also indicates that the proposed multi-task framework yields better performance over the single-task learning.

## introduction the content and form of content shared on online social media platforms have changed rapidly over time. currently, one of the most popular forms of media shared on such platforms is 'memes'. according to its definition from oxford dictionary, a meme is a piece of data, often in the form of images, text or videos that carry cultural information through an imitable phenomenon with a mimicked theme, that is shared (sometimes with slight modification) rapidly by internet users. every meme can be associated with five affect values, namely humour (hu), sarcastic (sar), offensive (off), motivational (mo), and sentiment (sent). hence, in a broad sense, memes can be categorized into four intersecting sets viz. humorous memes, sarcastic memes, offensive memes, and motivational memes. humour refers to the quality of being amusing or comic. formally, humour is defined as the nature of experiences to induce laughter and provide amusement. humourous memes are the most popular and widely used on social media platforms. an example for humourous memes is shown in figure 1a . sarcasm is often used to convey thinly veiled disapproval humorously. a sarcastic meme is a meme where an incongruity exists between the intended meaning and the way it is expressed. these are generally used to express dissatisfaction or to veil insult through humour. as we can see in figure 1a , the person on the right is made fun of, without explicitly expressing it, which is a typical example of a sarcastic meme. offensive content include a lot of insulting, derogatory terms. it is contrary to the moral sense or good. as social media expands, offensive language has become a huge headache to maintain sanity on social media. as memes are growing to become more and more popular, detecting offensive memes on such platforms is becoming an important and challenging task. figure 1a , figure 1c and figure 1d are the instances of offensive memes. motivation is derived from the word 'motive' which means needs or desires within the individuals. it is the process of stimulating people to actions to achieve their goals. by its definition, motivational memes are those that benefit a certain group of people to achieve their plans or goals. motivation can be both either positive or negative. however, we usually consider motivation in a positive sense. figure 1b is an excellent example for the positive motivation. sentiment analysis refers to the process of computationally identifying and categorizing opinions expressed in a piece of communication, especially to determine whether the writer's attitude towards a particular topic, product, etc. is positive, negative, or neutral. this has been a very prominent and important task in natural language processing. sentiment analysis on memes refers to the task of systematically extracting its emotional tone in understanding the opinion expressed by the meme. figure 1b is an example for positive sentiment towards the government and figure 1c for negative sentiment towards ph.d. in electrical engineering. generally, specific labels of one task have a strong relation to the other labels of sarcasm, offensive, humour or motivational tasks. through proper representation, training, and evaluation, these relations can be modelled to help each other for better classification. for example, in figure 1b , just by seeing text, the meme can be either sarcastic or motivational, but the image in the meme confirms that this has an overall positive sentiment and hence motivational. similarly, in figure 1c , knowing that the meme is sarcastic and has a negative sentiment makes it highly probable to being offensive. as seen above, humorous, motivational, offensive, and sarcastic nature of the memes are closely related. thus, a multi-task learning framework would be extremely beneficial in such scenarios. in this paper, we exploit these relationships and similarities in the tasks of humour detection, sarcasm detection, offensive content detection, motivational content detection, and sentiment in a multi-task manner. the main contributions and/or attributes are as follows: (a). we propose a multi-task multimodal deep learning framework to leverage the util-ity of each task to help each other in a multi-task framework; (b). we propose two attention mechanisms viz. itrm and icrm to better understand the relationship between the tasks and between the classes of tasks, respectively; and (c). we present the state-of-the-art results for meme prediction in the multi-modal scenario. humour: early feature-based models attempt to solve humour include the models based on word overlap with jokes, presence of ambiguity, and word overlap with common idioms @xcite , human-centeredness, and negative polarity @xcite . some of the recent multi-modal approaches include utilizing information from the various modalities, such as acoustic, visual, and text, using deep learning models @xcite @xcite . @xcite employs a paragraph decomposition technique coupled with fine-tuning bert @xcite model for humour detection on three languages (chinese, spanish and russian). sarcasm: starting from the traditional approaches, such as rule-based methods @xcite , lexical features @xcite , and incongruity @xcite to all the way up to multi-modal deep learning techniques @xcite , sarcasm detection has been showing its presence. @xcite al. (2019 ) created a multi-modal conversational dataset, mustard from the famous tv shows, and provided baseline svm approaches for sarcasm detection. recently, chauhan et al. (2020) proposed a multi-task learning framework for multi-modal sarcasm, sentiment and emotion analysis to explore how sentiment and emotion helps sarcasm. the author used the mus-tard dataset and extended the mustard dataset with sentiment (implicit and explicit) and emotion (implicit and explicit) labels. offensive: razavi et al. (2010) used a threelevel classification model taking advantage of various features from statistical models and rulebased patterns and various dictionary-based features. chen et al. (2012) proposed a feature-based lexical syntactic feature (lsf) architecture to detect the offensive contents. gomez et al. (2020) created a multi-modal hate-speech dataset from twitter (mmhs150k) to introduce a deep-learningbased multi-modal textual kernels model (tkm) and compare it with various existing deep learning architectures on the proposed mmhs150k dataset. motivation: swieczkowska et al. ( 2020 ) proposes a novel chaining method of neural networks for identifying motivational texts where the output from one model is passed on to the second model. sentiment: an important task to leverage multimodality information effectively is to combine them using various strategies. @xcite employs a hierarchical feature fusion strategy, divide, conquer, and combine for affective computing. @xcite uses the inter-modal interaction module (iim) to combine information from a pair of modalities for multi-modal sentiment and emotion analysis. some of the other techniques include a contextual inter-modal attention based framework for multi-modal sentiment classification @xcite . multi-task: some of the early attempts to correlate the tasks like sarcasm, humour, and offensive statements include a features based classification using various syntactic and semantic features, such as frequency of words, the intensity of adverbs and adjectives, the gap between positive and negative terms, the structure of the sentence, synonyms and others @xcite . more recently, @xcite proposed a convolution-based model to extract the embedding by fine-tuning the same for the tasks of sentiment, sarcasm, humour, and hate-speech and then concatenating these representations to be used in a sentiment classifier. in our current work, we propose a multi-task multi-modal deep learning framework to simultaneously solve the tasks of sarcasm, humour, offensive, and motivational on memes. further, to the best of our knowledge, this is the very first attempt at solving the multi-modal affect analysis on memes in a multi-task deep learning framework. we demonstrate through a detailed empirical evaluation that a multi-task learning framework can improve the performance of individual tasks over a single task learning framework. ## proposed methodology we propose an attention-based deep learning model to solve the problem of multi-task affect analysis of memes. the inputs to the model are the meme itself and the manually corrected text extracted through ocr. the overall architecture is depicted in figure 2. the source code is available at http://www. iitp.ac.in/ ˜ai-nlp-ml/resources.html. ## dataset we perform experiments using the dataset released in the memotion analysis 1.0 @semeval 2020 task @xcite foot_0 . this dataset consists of 6992 samples. each sample consists of an image, corrected text extracted from the meme, and the five labels associated with the five tasks, viz., humour, sarcasm, offensive, motivational, and overall sentiment. the distribution of the classes associated with each of the five tasks with label is shown in a. humour classification: there are four classes associated with the humour task, namely not funny, funny, very funny, and hilarious, which are labelled as 0, 1, 2, and 3, respectively. ## experimental setup in accordance with the semeval 2020 @xcite , the project is organized into three sets of tasks foot_1 . please note that, in task a, as it is not a multitask scenario, icrm and itrm are not applicable. for all the other sets of tasks, the entire network is shown in figure 2 . we evaluate our proposed model on the multimodal memotion dataset. we perform grid search to find the optimal hyper-parameters (c.f. ## results and analysis we evaluate our proposed architecture with bimodal inputs (i.e., text and visual). we show the obtained results for task-a (i.e., sentiment analysis) in l a b e l s task-a p r m a -f1 m i -f1 sentiment 36.99 35.70 35.81 50.58 task-b has four different tasks, i.e., humour, sarcasm, offensive, and sentiment with binary-class labels (c.f. binary-class classification in section 5). the results are shown in task-c has also four different tasks, i.e., humour, sarcasm, offensive, and sentiment with multi-class labels (c.f. multi-class classification in section 5). the results are shown in in both the tasks b and c, we outline the comparison between the multi-task (mtl) and single-task (stl) learning frameworks in for the offensive task, we find that stl performs better than mtl. we hypothesize that this is due to the model getting confused between the offensive and sarcastic (or humorous) memes. from ## comparative analysis we compare the results obtained in our proposed model against the baseline model and semeval 2020 winner, which also made use of the same dataset. the comparative analysis is shown in it is evident from ## error analysis we perform error analysis (i.e. for task-c) on the predictions of our proposed model. we take some utterances (c.f. we also present the attention heatmaps for icrm and itrm of the multi-task learning framework in figure 4 and figure 5 , respectively. we take the fifth utterance from figure 3 : few examples for human error analysis corresponding to the main objective of icrm is to develop the relationship between the classes of tasks. figure 4 shows the established relationship between the tasks. we see the established relationship between the classes of tasks in figure 4 . for predicting the fifth utterance correctly in in this paper, we have successfully established the concept of obtaining effective relationships between inter-tasks and between inter-classes for multi-modal affect analysis. we have proposed a deep attentive multi-task learning framework which helps to obtain very effective inter-tasks and interclasses relationship. to capture the interdependence, we have proposed two attention-like mechanisms viz., inter-task relationship module (itrm) and inter-class relationship module (icrm). the main motivation of itrm is to learn the relationship between the tasks, i.e. which task helps the other tasks. in contrast, icrm develops the relations between the classes of tasks. we have evaluated our proposed approach on a recently published memotion dataset. experimental results suggest the efficacy of the proposed model over the existing state-of-the-art systems (baseline and semeval 2020 winner). the evaluation shows that the proposed multi-task framework yields better performance over single-task learning. the dataset used for the experiments is relatively small for training an effective deep learning model and is heavily biased. therefore, assembling a large, and more balance dataset with quality annotations is an important job. moreover, the memes are a complicated form of data which includes both text and image that repeat over numerous memes (meme templates). hence quality representation of memes for affect analysis is challenging future work.