Demonstrating the Reliability of Self-Annotated Emotion Data (2021)

Abstract: Vent is a specialised iOS/Android social media platform with the stated goal to encourage people to post about their feelings and explicitly label them. In this paper, we study a snapshot of more than 100 million messages obtained from the developers of Vent, together with the labels assigned by the authors of the messages. We establish the quality of the self-annotated data by conducting a qualitative analysis, a vocabulary based analysis, and by training and testing an emotion classifier. We conclude that the self-annotated labels of our corpus are indeed indicative of the emotional contents expressed in the text and thus can support more detailed analyses of emotion expression on social media, such as emotion trajectories and factors influencing them.

## introduction social media platforms are being widely used by people to express their feelings. while some such platforms are generic in their purpose (e.g., twitter), others have specific goals, such as connecting with people with similar health issues (e.g., pa-tientslikeme foot_0 ). vent foot_1 belongs to the latter class of platforms: its stated goal is to encourage people to express and share their feelings. vent enables people to post messages expressing their own feelings and to react to posts from others. interestingly, vent requires people to label their posts with the emotion they feel at the time of posting. the platform thus provides us with an opportunity to study, at scale, how people express emotions, to what emotions they react, how emotions change over time, and what factors influence their trajectory. vent data is self-annotated for emotion, which is of particular interest to us. studies on emotions in social media often derive labels from texts, either with the help of annotators, or using sentiment analysis techniques (see, for example, reviews of annotated datasets by @xcite ; @xcite ). we note, however, that information that external observers (annotators or algorithms) can extract from a text may not be sufficient to reliably identify the affective state of the text's author at the time of posting. this could be because the texts are too short to provide enough context, are ambiguous, or require extra-textual context to interpret. even when richer context is available, external observers may not necessarily assign a definitive affective label to a text. for example, psychological construction theory @xcite states that emotion labels are a result of categorisation of the current state of the organism, in the current context; consequently, the same episode may be categorised differently by the person who experiences it and by an outside observer. given this, self-assigned affective labels may provide a more direct access to a person's emotional state than labels attributed after the fact. our ultimate goal is to study emotion trajectories (on social media) and the factors that affect them, potentially leading to the automatic identification of mental health issues. however, before we can employ data such as that provided in vent to study emotion sharing and changes in emotions, we must establish whether the self-annotated labels are reasonable indicators of emotional states. this is because, even with self-assigned labels, there are concerns that may arise: for example, the label choice may be a byproduct of poor user interface design. establishing that the labels are reasonable is thus our central aim in this paper. we conduct a multi-step analysis of the vent data, showing that we can use this kind of data to study how people express their feelings and how people react to them. the rest of the paper is structured as follows. we begin with a short summary of related research in section 2. this is followed by a description of the vent platform and the data we have from it in section 3. we describe the data selection steps in section 4. we then present the analysis steps we have taken to ascertain that the labels adequately reflect the affective states expressed in the texts in section 5. section 6 concludes this paper and outlines future research directions. ## related work there is a growing number of datasets annotated with affect information. many of these are annotated by experts or via crowdsourcing and fall out of the scope of our work. instead, we refer the reader to the surveys by @xcite ; @xcite . to the best of our knowledge, self-annotated affective datasets are rare; the reviews by @xcite ; mohammad (2020) mention only one such dataset. isear ("international survey on emotion antecedents and reactions") is a selflabelled affective dataset created by @xcite . it was collected by administering a questionnaire, in which people were asked to describe recent experiences of one of the seven emotions (anger, fear, joy, sadness, disgust, shame, guilt) and to answer questions about their physiological and psychological state during these emotion episodes. overall, roughly 3,000 people from 37 countries completed the questionnaire, providing 7,666 textual descriptions. in comparison, our dataset contains considerably more data. a more widely used approach to produce emotion annotation without using experts is to rely on distant supervision -for example, treating twitter hashtags like #happy or #sad as selfassigned emotion labels. examples of datasets constructed with distant supervision include those by @xcite ; @xcite ; @xcite ; @xcite ; @xcite ; @xcite ; @xcite . emotion classifiers using these datasets are reported to perform well: the best results thus far were produced by @xcite , who used a gated recurrent neural network (grnn) classifier on 1.6 million tweets labelled with emotions from plutchik's categorisation @xcite and reached an averaged f1-score foot_2 of 0.9568. @xcite used web-scraping techniques to collect 33 million messages from the vent platform, from around 1 million users with public profiles (meaning that anybody on the platform could see these posts). they presented a broad descriptive exploration of these data, along with an analysis of emotions in texts and user networks, but they did not investigate the quality of the annotations. in comparison, our dataset is directly provided via a 2019 data science partnership with vent. our data includes all posts (anonymised for this research). our goal here is to assess the alignment between affect in self-assigned affective labels and texts. ## vent and its dataset vent advertises itself as a platform to "express your feelings and connect with people who care". vent is thus specifically geared towards sharing one's emotions, unlike twitter or facebook, which support many other activities. this makes vent particularly interesting for investigating emotion expression on social media. users (venters) register anonymously, with only an email address. once registered, they can create short text messages (vents), read messages by other venters and react to them, using comments or interactions (short predefined reactions, for example, "hug", "lol", or an emoji). vent's creators have given us access to the data from the platform over a 5-year period, from the late 2013 until the end of 2019, as part of a collaborative project to study mental health. 4 overall, the raw dataset contains over 107 million vents, from close to 1.5 million users, including both public and private posts, along with additional types of information, namely comments, interactions, follower/followee links and the information on discussion groups. due to ethical and privacy concerns, the dataset is not publicly available. vent's labels 5 are arranged in a two-level hier-archy. at the top level, there are 85 emotion categories, which we can categorise into these 5 groups: affective states. this group contains the following 9 categories: affection, anger, creativity, fear, feelings, happiness, positivity, sadness, surprise. dates. there are 46 categories linked to dates and seasonal events, such as autumn, ramadan, paralympics, etc. ## data selection for our analyses, we restrict our data to the vents that correspond to the following six highlevel categories, which we call "core categories": affection, anger, fear, happiness, sadness, and surprise. these labels are always available to the users. importantly, out of all vent's categories, they are most easily interpretable in terms of affective states. many psychological accounts of human emotion repertoire include some or all of these categories (see, e.g., in addition, we exclude the following categories of users: ## assessing the alignment of the labels and the texts to ascertain the alignment between the text and the labels, we conduct the following analyses: we use these four methods to establish that the self-annotated labels do indeed reflect emotional state. the methods are complementary. the qualitative analysis attempts to capture idiosyncratic uses of labels, which may be hard to anticipate and thus hard to analyse automatically. the vocabularybased and emoji-based analyses establish whether individual emotion-loaded tokens in the texts are congruent with the labels. finally, the classification approach allows the exploration of the connection between entire texts and their labels, capitalising on context beyond individual tokens. these analyses are described below. ## conclusions in this paper, we have presented an analysis of the quality of self-annotated emotion data from the vent platform, which is specifically focused on emotion sharing. our results suggest that selfassigned labels in vent have a reasonable degree of connection to the affective states expressed in the texts. a qualitative analysis of the vents and their labels indicates that labels which are not meant to communicate affect are rare. a vocabularybased analysis based on emolex shows that vent labels align with affect polarity of the texts, and that words associated with a certain emolex emotion are most frequently encountered in vents in the corresponding vent category. the top 10 emojis in each category are consistent with the category label. finally, a bert classification model can predict correct labels most often, and the classification mistakes often preserve emotion valence. overall, we conclude that self-assigned labels produced in a non-controlled naturalistic setting can be used as a reasonably accurate representation of the author's affective state, and thus can support more complex analyses of emotions in social media. our analyses focused on the assumption that each text conveys one dominant emotion which may or may not be congruent with the assigned label. we adopted this approach as a first step, allowing us to explore simple models matching the structure of the data (one message -one label). this is an oversimplification, as suggested by examples such as (4) in one particular research direction we are currently exploring is tracking the changes in reported emotion over time, the factors influencing these changes, and the connection of these properties with mental health well-being.