That’s so cute!: The CARE Dataset for Affective Response Detection (2022)

Abstract: Social media plays an increasing role in our communication with friends and family, and in our consumption of entertainment and information. Hence, to design effective ranking functions for posts on social media, it would be useful to predict the affective responses of a post (e.g., whether it is likely to elicit feelings of entertainment, inspiration, or anger). Similar to work on emotion detection (which focuses on the affect of the publisher of the post), the traditional approach to recognizing affective response would involve an expensive investment in human annotation of training data. We create and publicly release CARE DB, a dataset of 230k social media post annotations according to seven affective responses using the Common Affective Response Expression (CARE) method. The CARE method is a means of leveraging the signal that is present in comments that are posted in response to a post, providing high-precision evidence about the affective response to the post without human annotation. Unlike human annotation, the annotation process we describe here can be iterated upon to expand the coverage of the method, particularly for new affective responses. We present experiments that demonstrate that the CARE annotations compare favorably with crowdsourced annotations. Finally, we use CARE DB to train competitive BERT-based models for predicting affective response as well as emotion detection, demonstrating the utility of the dataset for related tasks.

## introduction social media and other online media platforms have become a common means of not only interacting and connecting with others, but also finding interesting, informing, and entertaining content. users of those platforms depend on the ranking systems of the recommendation systems to show them information they will be most interested in and to safeguard them against unfavorable experiences. towards this end, a key technical problem is to predict the affective response that a user may have when they see a post. some affective responses can be described by emotions (e.g., angry, joyful), and others may be described more as experiences (e.g., entertained, inspired). predicting affective response differs from emotion detection in that the latter focuses on the emotions expressed by the publisher of the post (referred to as the publisher affect in @xcite ) and not on the viewer of the content. while the publisher's emotion may be relevant to the affective response, it only provides a partial signal @xcite , and the two are not always equivalent (see figure 2 for an illustrative example). affective response for recommender systems has shown to be critical in several applications such as music, emotional health monitoring systems, product and travel recommendations @xcite @xcite @xcite . current approaches to predicting affective response require obtaining training data from human annotators who try to classify content into classes of a given taxonomy. however, obtaining enough training data can be expensive, and moreover, due to the subjective nature of the problem, achieving consensus among annotators can be challenging. some methods explore inferring responses from physiological data or facial expressions from users, but this is a highly invasive process and can be difficult to scale to multiple users. @xcite @xcite . this paper introduces the common affective response expression method (care for short), a means of obtaining labels for affective response in an unsupervised way from the comments written in response to online posts. care uses patterns and a keyword-affect mapping to identify expressions in comments that provide high-precision evidence about the affective response of the readers to the post. for example, the expression "what a hilarious story" may indicate that a post is humorous and "this is so cute" may indicate that a post is adorable. we seed the system with a small number of high-precision patterns and mappings. we then iteratively expand on the initial set by considering frequent patterns and keywords in unlabeled comments on posts labeled by the previous iteration. using care, we create the largest dataset to date for affective response, care db , which contains 230k posts annotated according to 7 affective responses. we validate the effectiveness of care by comparing the care annotations with crowdsourced annotations. our experiments show that there is a high degree of agreement between the annotators and the labels proposed by care (e.g., in 90% of the cases, at least two out of three annotators agree with all the care labels). furthermore, we show that the care patterns/lexicon have greater accuracy than applying sota emotion recognition techniques to the comments. using care db , we train care-bert foot_0 , a bert-based model that can predict affective response without relying on comments. care-bert provides strong baseline performance for the task of predicting affective response, on par with the sota models for emotion recognition. furthermore, we show that care-bert can be used for transfer learning to a different emotion-recognition task, achieving similar performance to @xcite , which relied on manually-labeled training data. in summary, our contributions are as follows: ## related work we first situate our work with respect to previous research on related tasks. ## the care method in this section, we provide a formal description of care for annotating the affective response of posts. before we proceed, we note two aspects of affective responses. first, there is no formal definition for what qualifies as an affective response. in practice, we use affective responses to understand the experience that the user has when seeing a piece of content, and these responses may be both emotional and cognitive. second, the response a user may have to a particular piece of content is clearly a very personal one. our goal here is to predict whether a piece of content is generally likely to elicit a particular affective response. in practice, if the recommendation system has models of user interests and behavior, these would need to be combined with the affect predictions for personalized predictions. ## evaluating care db in this section we apply our method to social media posts and validate these annotations using human evaluation (section 4.1). section 4.2 discusses class-wise error analysis, and in section 4.3, we explore the alternative possibility of creating care db using a sota publisher-affect classifier @xcite to label the comments instead of using the care patterns/lexicon. care db : our experiments use a dataset that is created from reddit posts and comments in the pushshift.io database that were created between 2011 and 2019. we create our dataset, care db , as follows. we used care patterns and the care lexicon to annotate 34 million comments from 24 million distinct posts. after filtering with a threshold of @xmath0, we obtained annotations for 400k posts (the total number of posts that have at least 5 comments was 150 million). the low recall is expected given the specificity of care patterns/lexicon. we also filtered out posts that have less than 10 characters, resulting in a total of 230k posts in care db . ## predicting affective response for posts without comments in this section we describe care-bert, a multilabel affective response classifier that is trained only on the post-level text and annotations in care db . such a model is important in order to ar goemotion label % agree amused amusement 79.8 approving admiration, approval 89.3 excited joy 81.3 angered anger, annoyance, disgust 93.3 saddened disappointment, sadness 90.9 scared fear, nervousness 84.9 note that the model is not given the comments text and is therefore not restricted to the care pattern/lexicon semantics. in section 5.2, we describe how care-bert can be further fine-tuned for related tasks like emotion detection. ## conclusion we described a method for extracting training data for models that predict the affective responses of a post on social media. care is an efficient, accurate, and scalable way of collecting unsupervised labels and can be extended to new classes. using care, we created care db , a large dataset which can be used for affective response detection and other related tasks, as demonstrated by the competitive performance of care-bert to similar bertbased models in emotion detection. we release the annotations and models in the hopes that this will unlock future research. in particular, there are two main cases in which care can be improved upon: (1) when there does not exist a set of common phrases that are indicative of an affect, and (2) when an indicator maps to multiple affects. in the latter case, there is still partial information that can be gleaned from the labels. in addition to developing methods for the above cases, future work also includes incorporating emojis, negations, and punctuation, extending to new classes, or possibly using embedding similarity rather than exact match for the care patterns. finally, we also plan to investigate the use of care for predicting the affective response to images as well as multi-modal content such as memes.