Happy Dance, Slow Clap: Using Reaction GIFs to Predict Induced Affect on Twitter (2021)

Abstract: Datasets with induced emotion labels are scarce but of utmost importance for many NLP tasks. We present a new, automated method for collecting texts along with their induced reaction labels. The method exploits the online use of reaction GIFs, which capture complex affective states. We show how to augment the data with induced emotion and induced sentiment labels. We use our method to create and publish ReactionGIF, a first-of-its-kind affective dataset of 30K tweets. We provide baselines for three new tasks, including induced sentiment prediction and multilabel classification of induced emotions. Our method and dataset open new research opportunities in emotion detection and affective computing.

## introduction affective states such as emotions are an elemental part of the human condition. the automatic detection of these states is thus an important task in affective computing, with applications in diverse fields including psychology, political science, and marketing @xcite . training machine learning algorithms for such applications requires large yet task-specific emotion-labeled datasets @xcite . borrowing from music @xcite and film @xcite , one can distinguish between two reader perspectives when labeling emotions in text: perceived emotions, which are the emotions that the reader recognizes in the text, and induced emotions, which are the emotions aroused in the reader. however, with the exception of @xcite , this distinction is mostly missing from the nlp literature, which focuses on the distinction between author and reader perspectives @xcite . the collection of perceived emotions data is considerably simpler than induced emotions data, and presently most human-annotated emotion datasets are labeled with perceived emotions (e. g., @xcite @xcite . induced emotions data can be collected using physiological measurements or self-reporting, but both methods are complex, expensive, unreliable and cannot scale easily. still, having well-classified induced emotions data is of utmost importance to dialogue systems and other applications that aim to detect, predict, or elicit a particular emotional response in users. @xcite used distant supervision to detect induced emotions from facebook posts by looking at the six available emoji reactions. although this method is automatic, it is limited both in emotional range, since the set of reactions is small and rigid, and accuracy, because emojis are often misunderstood due to their visual ambiguity @xcite . to overcome these drawbacks, we propose a new method that innovatively exploits the use of reaction gifs in online conversations. reaction gifs are effective because they "display emotional responses to prior talk in text-mediated conversations" @xcite . we propose a fullyautomated method that captures in-the-wild texts, naturally supervised using fine-grained, induced reaction labels. we also augment our dataset with sentiment and emotion labels. we use our method to collect and publish the reactiongif dataset. foot_0 2 automatic supervision using gifs figure 1a shows a typical twitter thread. user a writes "i can't take this any more!". user b replies with a reaction gif depicting an embrace. our method automatically infers a hug reaction, signaling that a's text induced a feeling of love and caring. in the following, we formalize our method. ## reactiongif dataset we applied our proposed method to 30k englishlanguage (t,g) 2-turn pairs collected from twitter in april 2020. t are text-only root tweets (not containing links or media) and g are pure gif reactions. we label each tweet t with its reaction category @xmath0. see appendix a for samples. the resulting dataset, reactiongif, is publicly available. figure 3 shows the category distribution's long tail. the top seven categories (applause to eyeroll) label more than half of the samples (50.9%). each of the remaining 36 categories accounts for between 0.2% to 2.8% of the samples. label augmentation reaction categories convey a rich affective signal. we can thus augment the dataset with other affective labels. we add sentiment labels by using the positive and negative reaction category clusters, labeling each sample according to its cluster's sentiment ( §2.2). furthermore, we add emotion labels using a novel reactions-to-emotions mapping: we asked 3 annotators to map each reaction category onto a subset of the 27 emotions in demszky et al. (2020)see ## baselines as this is the first dataset of its kind, we aim to promote future research by offering baselines for predicting the reaction, sentiment, and emotion induced by tweets. we use the following four models in our experiments: we hold out 10% of the samples for evaluation. the code is publicly available along with the dataset for reproducibility. the experiment results are summarized in affective reaction prediction is a multiclass classification task where we predict the reaction category r for each tweet t. roberta achieves a weight-averaged f 1 -score of 23.9%. induced sentiment prediction is a binary classification task to predict the sentiment induced by tweet t by using the augmented labels. roberta has the best performance with accuracy 70.0% and f 1 -score of 69.8%. finally, induced emotion prediction uses our reaction-to-emotion transformation for predicting emotions. this is a 27-emotion multilabel classification task, reflecting our dataset's unique ability to capture complex emotional states. roberta is again the best model, with label ranking average precision (lrap) of 0.596. ## discussion reaction gifs are ubiquitous in online conversations due to their uniqueness as lightweight and silent moving pictures. they are also more effective and precise 2 when conveying affective states compared to text, emoticons, and emojis @xcite . consequently, the reaction category is a new type of label, not yet available in nlp emotion datasets: existing datasets use either the discrete emotions model @xcite or the dimensional model of emotion @xcite . the new labels possess important advantages, but also present interesting challenges. advantages the new reaction labels provide a rich, complex signal that can be mapped to other types of affective labels, including sentiment, emotions and possibly feelings and moods. in addition, because reaction gifs are ubiquitous in online conversations, we can automatically collect large amounts of inexpensive, naturally-occurring, high-quality affective labels. significantly, and in contrast with most other emotion datasets, the labels measure induced (as opposed to perceived) affective states; these labels are of prime importance yet the most difficult to obtain, with applications that include gif recommender systems, dialogue systems, and any other application that requires predicting or inducing users' emotional response. ## conclusion our new method is the first to exploit the use of reaction gifs for capturing in-the-wild induced affective data. we augment the data with induced sentiment and emotion labels using two novel mapping techniques: reaction category clustering and reactions-to-emotions transformation. we used our method to publish reactiongif, a first-of-its-kind dataset with multiple affective labels. the new method and dataset offer opportunities for advances in emotion detection. moreover, our method can be generalized to capture data from other social media and instant messaging platforms that use reaction gifs, as well as applied to other downstream tasks such as multimodal emotion detection and emotion recognition in dialogues, thus enabling new research directions in affective computing.