Hashtags, Emotions, and Comments: A Large-Scale Dataset to Understand Fine-Grained Social Emotions to Online Topics (2020)

Abstract: This paper studies social emotions to online discussion topics. While most prior work focus on emotions from writers, we investigate readers’ responses and explore the public feelings to an online topic. A large-scale dataset is collected from Chinese microblog Sina Weibo with over 13 thousand trending topics, emotion votes in 24 fine-grained types from massive participants, and user comments to allow context understanding. In experiments, we examine baseline performance to predict a topic’s possible social emotions in a multilabel classification setting. The results show that a seq2seq model with user comment modeling performs the best, even surpassing human prediction. More analyses shed light on the effects of emotion types, topic description lengths, contexts from user comments, and the limited capacity of the existing models.

## introduction social media have become a popular outlet for people to voice opinions, share viewpoints, and exchange ideas. it provides us with rich resource to research public opinions of the trending topics and hear people's voice over important social events, such as the global covid-19 crisis. however, our ever-changing physical world leads to the rapid evolution of discussion topics in online world; it is far beyond humans' capability to catch them in real time. it consequently presents a pressing need for automatic sentiment @xcite and emotion @xcite analysis. nevertheless, most of the related work focus on the feelings from writers @xcite @xcite and the existing studies concerning reader emotions mostly tackle well-written texts, such as news reports @xcite @xcite . limited work has been done to characterize collective feelings from the public (henceforth social emotions) to an online topic described with fragmented and colloquial social media language. where some previous efforts gather viewpoints from limited readers through user replies @xcite or manual annotations @xcite , we focus on social emotions reflecting aggregated feelings from large amount of people. in light of these concerns, we present a novel task to infer social emotions to online topics. its goal is to predict the possible emotions from the majority of readers given a few words description of a trending social media topic. the task will benefit various applications, such as topic analysis @xcite and event detection @xcite . it would help people understand and foresee how the general public thinks of an event even before it becomes a trending topic on social media. more importantly, it is to induce a society's collective emotions to a discussion topic, which is helpful to gain machines' affective analysis ability for more appropriate response behaviors and will potentially advance human-computer interactions @xcite . to better illustrate our task, figure 1 displays a topic trended on a popular chinese microblog sina weibo (henceforth weibo). here hashtags (phrases between two "#"s) are considered as userannotated discussion topics following the common practice @xcite . also shown are the top three reader emotions voted by the online users. 2 as can be seen, the social media style writing -short and informal -makes it challenging to capture what feelings a topic is likely to evoke. the example concerns the story of two celebrities (lay and gai) in a variety show, where one needs to make sense of "整蛊" (an internet slang means "play tricks on") and access some background to get why most voters leaned on hilarious emotions. as the pilot study for online topics' social emotions, we also present the very first dataset for this task, which contains large-scale popular topics, their corresponding social emotions, and user comments for context understanding. we first gather over 13 thousand weibo topics (in hashtag forms) associated with fine-grained emotion types represented as 24 emoji labels and contributed by more than 3 thousand online users on average. compared with the related resource from news websites, our dataset prepared via social media crowd sourcing exhibits larger scale, more annotators, and finergrained emotion types. moreover, we collect abundant user comments for each topic, which enables context modeling to access public thoughts. it will later benefit future work to examine how social emotions are shaped in online discussions. extensive experiments are carried out on our dataset. we first discuss how the baselines and popular multi-label classification models work to predict the top three reader emotions. the results show that seq2seq-based models obtain the best overall results and user comments can further boost the performance via providing richer contexts. we also find that machines exhibit a superiority on our task compared with humans and point out the possible bias from individuals to sense public emotions. afterwards, model performances are quantified over varying emotions and hashtag length, where user comments consistently result in the performance gain. finally, we probe into our output to analyze how user comments help and the limitation of the existing models. as for the analysis of readers' emotions (a.k.a., affective analysis) @xcite , some of them collected emotion signals (or emojis) from user replies @xcite or manual annotation @xcite . responsive emotions are collected from limited readers and hence cannot reflect social emotions from the public. other studies adopt emotion votes on news websites to gather public feelings on news @xcite @xcite . however, news reports usually exhibit more formal style than online topics in social media language. none of the above work examine public emotions to online discussion topics, which is the gap filled in this study. ## data collection and analysis data collection. our dataset is built based on a weibo emotion vote, where it provides users to vote for an emoji from a total of 24 emojis in the form of a questionnaire to represent their feelings to a trending hashtag. here comes the steps to collect the data. first, we tracked the trending hashtags following the everyday weibo topic summary list foot_1 from apr to may 2020. for topics trended before that, we resort to a webpage listing the historical hot topics since nov 2019. 4 then, we searched and parsed their emotion vote webpage via querying the hashtag in http requests foot_4 with the selenium package. next, the crawled pages were parsed and analyzed using lxml package to gather the topics' emotion voting results. at last, hashtags with less than 100 voters were removed to filter out biased results. as weibo only keeps emotions gaining the top three votes, we will hence focus on the top three emotions in the following discussions. these emotions were selected by over 83% voters on average and can still reflect feelings from the majority. furthermore, to access the contexts of hashtags, we collected some user comments involved in a hashtag's discussion. concretely, we first visited the hashtag page 6 through http requests and obtained the popular weibo messages that carry the hashtag. then from the html codes, we extracted the ids of the top four messages. their comments were later gathered from the messages' comment pages 7 using an open source toolkit. in our dataset, most of the topics are hot events in real life. for example, a few topics collected in the late 2019 concern the social unrest in hong kong, while many topics trended in 2020 are about the covid-19 outbreak. we characterize the topics by keywords and find that 15% topics contain the word "新冠" (covid-19) while the number rises 6 https://s.weibo.com/weibo?@xmath0, where xxx refers to a hashtag name. to further analyze the 24 fine-grained emotions, figure 2 shows how the voter number distribute over varying emotions. this implies the diverse preferences of user voters to pick up varying emotion labels and the challenging label imbalance issue of our task. we also observe that positive and negative emotions are approximately equally distributed, both used more often than emotions in "others" group, probably due to their complicated and vague meanings. ## conclusion we have investigated social emotions to online discussion topics. a large-scale chinese weibo dataset is built containing trending hashtags, emotion votes, and user comments (for context modeling). in experiments, we have shown that the prediction of social emotions is challenging and the modeling of user comments may usefully bridge topic descriptions and public emotions.