TWEETSUM : Event oriented Social Summarization Dataset (2020)

Abstract: With social media becoming popular, a vast of short and noisy messages are produced by millions of users when a hot event happens. Developing social summarization systems becomes more and more critical for people to quickly grasp core and essential information. However, the publicly available and high-quality large scale social summarization dataset is rare. Constructing such corpus is not easy and very expensive since short texts have very complex social characteristics. In this paper, we construct TWEETSUM, a new event-oriented dataset for social summarization. The original data is collected from twitter and contains 12 real world hot events with a total of 44,034 tweets and 11,240 users. Each event has four expert summaries, and we also have the annotation quality evaluation. In addition, we collect additional social signals (i.e. user relations, hashtags and user profiles) and further establish user relation network for each event. Besides the detailed dataset description, we show the performance of several typical extractive summarization methods on TWEETSUM to establish baselines. For further researches, we will release this dataset to the public.

## introduction social media has become an important real-time information source, especially during emergencies, natural disasters and other hot events. according to a new pew research center survey, social media has surpassed traditional news platforms (such as tv and radio) as a news source for americans: about twothirds of american adults (68%) get news via social media. among all major social media sites, twitter is still the site americans most commonly use for news, with 71% of twitter's users get their news from twitter. however, it can often be daunting to catch up with the most recent contents due to high volume and velocity of tweets. hence, social summarization aiming to acquire the most representative and concise information from massive tweets when a hot event happens is particularly urgent. in recent years, many large-scale summarization datasets have been proposed such as new york times @xcite , gigaword @xcite , newsroom @xcite and cnn/dailymail @xcite . however, most of these datasets focus on formal document summarization. actually, social media text has many different characteristics from formal documents: 1) short. the length of a tweet is limited to 140 characters, which is much shorter than formal document. 2) informal. tweets usually contains informal expressions such as abbreviations, typos, special symbols and so on, which make tweets more difficult to deal with. 3) social signal. there are different kinds of social signals on social media such as hashtags, urls and emojis. 4) potential relations. tweets are generated by users and hence have potential connections through user relationship. because of these characteristics, traditional summarization methods often do not perform well on social media. though there exists some social media summarization datasets @xcite @xcite @xcite . however, these datasets only consider the text on social media and ignore the potential social signals on social network. in a social context, the interactions between friends are obviously different from that between strangers. this phenomenon demonstrates that social relationship can affect user behavior patterns and consequently affect the tweets content they post. this inspires us to consider integrating social relations relevant signals when analyzing social information. in this paper, we construct an event-oriented large-scale dataset with user relations for social summarization called tweetsum. it contains 12 real world hot events with a total of 44,034 tweets and 11,240 users. in summary, this paper provides the following contributions: (1) construct an event-oriented social media summarization dataset, tweetsum, which contains social signals. to our knowledge, it is the first summarization dataset that contains user relationships relevant social signals, such as hashtags and user profiles and so on; (2) create expert summaries for each social event and verified the existence of sociological theory in real data, including social consistency and contagion; (3) evaluate the performance of typical extractive summarization models on our tweetsum dataset to provide benchmarks and validate the effectiveness of the dataset. ## data properties and analysis figure 3 : visualization of one user social networks. user relations are the unique property of our dataset compared with other summarization datasets. we collect users and their corresponding relations in each event to construct social networks and further analyze the statistics of the generated network, which is shown in the second part of figure 4 shows the 1-gram and 2-gram overlap ratio under all 12 events. the average 1-gram and 2-gram overlap ratio between friends (26.92% and 4.01%) are consistently higher than that between strangers (25.40% and 3.45%), which demonstrates the impact of social relations on user behavior. we further conduct two sample t-test where null hypothesis h 0 means there is no difference between tweets posted by friends and those randomly selected tweets, while alternative hypothesis h 1 means the distance between tweets posted by friends is smaller than that of randomly selected tweets. we define the distance of two tweets as : d @xmath0, where t i is the tfidf representation of the i-th tweet. the p-value shown in user profiles include the following information: 1) user id. this is the unique identity of each user. 2) historical tweet records. tweets posted by users contain lots of user information, from which we can obtain abundant information, such as user interests and preference. 3) tweet timestamp records the time of the creation of tweets. 4) retweet count is used to reflect the popularity of each tweet. expert summary has been described in section 2.2. ## experiments 4.1 compared methods to verify the effectiveness of our tweetsum dataset, we choose some typical extractive summarization methods as baseline methods. (1) expert: denotes the average mutual assessment of expert summaries. (2) random: selects tweets randomly from each hot events set to form summaries. (3) lexrank: adopts pagerank-like algorithm to rank and select tweets @xcite . ( 4 ) lsa: exploits svd @xcite and selects the highest-ranked tweets from each singular vector. (5) mdss: uses two-level sparse representation model @xcite . ( 6 ) dsdr: uses data reconstruction method @xcite . ( 7 ) snsr: integrates the social relationship into a unified sparse coding optimization framework @xcite . ( 8 ) fine-tuning bert: we fine tune the pre-trained bert model @xcite and learn representations for tweets. summaries are choosen according to cosine similarity. ## conclusion and future work in this paper, we construct an event-oriented social media summarization dataset, called tweetsum. to better explore how social signals help social summarization, we filter some outliers, keeping social network dense to some extent, and conduct experiments to verify the influence of social signals on user generated content. we further analyze the characteristics of this dataset in detail and validate the influence of social relations on tweets content selection. both traditional summarization methods and neural network-based methods are tested on our dataset. in the future, the dataset can be further expanded to include more events as well as more various social signals. in addition, manually annotating data is an expensive and labor-consuming task, therefore we will further try to explore approaches to construct social summarization dataset automatically. more research space can be extended based on this dataset and we hope the tweetsum dataset can foster the development of social summarization.