NLPOP : a Dataset for Popularity Prediction of Promoted NLP Research on T witter (2022)

Abstract: Twitter has slowly but surely established itself as a forum for disseminating, analysing and promoting NLP research. The trend of researchers promoting work not yet peer-reviewed (preprints) by posting concise summaries presented itself as an opportunity to collect and combine multiple modalities of data. In scope of this paper, we (1) construct a dataset of Twitter threads in which researchers promote NLP preprints and (2) evaluate whether it is possible to predict the popularity of a thread based on the content of the Twitter thread, paper content and user metadata. We experimentally show that it is possible to predict popularity of threads promoting research based on their content, and that predictive performance depends on modelling textual input, indicating that the dataset could present value for related areas of NLP research such as citation recommendation and abstractive summarization.

## introduction the now not-so-recent neural revolution caused a widespread increase of interest in machine learning research. through improvements obtained across the field by applying deep neural networks, every application of machine learning became open for researchers to publish work pushing pre-neural boundaries, whether that work applied a neural architecture to a problem @xcite , unveiled the black box of deep neural networks @xcite or coming up with a new architecture altogether @xcite . the rapid progress paved way for more researchers to enter the field, which resulted in an ever increasing volume of research work published year by year. the large volume of work meant that it is difficult for a single person to keep up to date with relevant research. thus, a need emerged for a platform where work can be shared, filtered and discussed on a scale larger than research labs. twitter, a microblogging social network emerged as the chosen forum. the otherwise prohibitive 280 character limit on each post ("tweet") can in this context be viewed as a feature -it promotes succinctness and discourages lengthy academic prose. a portion of researchers accepted that promoting your academic work on twitter is something that you do -and if done well, it is believed that your research pedigree and citation count will increase. while this statement has not yet been put to test, the increase of posts promoting research work indicates that many believe it (figure 1 ). along with sharing a link to your paper, it is common to provide a concise summary outlining the main idea and contributions of your work in form of a post thread. in scope of this paper we aim to collect a dataset of twitter threads promoting research work and evaluate whether the popularity of a post can be determined from the content of the thread, paper, or user information. we would like to emphasize that we do not believe that scientific work being popular implies that the work itself is good, but rather aim to analyse whether it is possi-ble to determine factors which lead to higher visibility. researchers could then use findings from such analysis to hopefully reach a broader audience. ## related work predicting popularity of messages is a straightforward task from the perspective of machine learning and has been framed both as a regression @xcite and classification problem @xcite @xcite , while work on information cascades @xcite @xcite focuses on modeling the entire lifetime of a post as a point process. work in the area of computational linguistics mainly focuses on analysing the underlying causes of popularity: @xcite evaluate whether wording affects popularity of posts and find a number of patterns which popular posts adhere to, @xcite analyse how use of language gets people involved in online discussions, while approaches such as @xcite and @xcite aim to help users reach a larger audience. only recently has the effect of social media on collaboration between researchers been analysed @xcite although there have been indications that younger generations of scholars prefer using social media to foster collaboration @xcite -which also might relate to the newly discovered phenomenon of preference for citing recent work @xcite . ## dataset when constructing our dataset, we limit ourselves to posts promoting academic research in the field of nlp on twitter in english language. this choice was motivated by two reasons: (1) we believe that for a popularity prediction model to be successful, the domain should be narrow and (2) as all authors of this paper are involved in nlp research, we have a deep personal interest in whether it is possible to determine what constitutes a "good" post which promotes academic reseach on social media. we first selected a set of nlp researcher twitter users, which we then manually validated. we then fetched all posts of these users that contained a link which resolved on arxiv foot_0 , and then selected a subset of these posts which formed threads containing comments from the same root user. the latter step was done to avoid bot accounts which automatically share all preprints and as an attempt to ensure that threads contain a summary of the paper referred to. nevertheless, these simple rules are by no means exhaustive. it is likely that the dataset contains threads from users which do not summarize the paper, while it definitely contains summaries written by users that are not authors of the paper. while we considered manually validating each thread, we chose not to as doing so would make scaling the dataset in the future infeasible. for the sake of space, we omit the detailed description of dataset construction to appendix a. ## methodology we will first define the notion of popularity. while some other works @xcite have considered only the final number of reshares (retweets) as the popularity criterion, we also consider predicting the number of likes a post receives @xcite as another task. ## results when reporting results, we will mainly be looking to answer the following questions: (1) do more complex text encoders improve prediction performance?; to answer the first two questions, we perform an ablation study and report the results in analysing the effect of feature groups, we can see that the thread itself performs the best in isolation for both target variables, except for rnn models -indicating that a good summary influences the popularity the most. when analysing the thread features in combination with other feature groups for the like prediction case, the bio offers the most improvement, with paper the second most important group, indicating that paper content matters for popularity. for the retweet case, surprisingly, adding any feature group diminishes the performance of the bert model, emphasizing the fact that the content of the thread is the most discriminative feature for determining popularity. analysing the effect of the task formulation, in finally, we aim to understand whether the models are able to understand the class boundaries. to this end, we will take a look at the confusion matrices of the best performing models for the #likes ( ## conclusion we have introduced nlpop: a novel dataset for popularity prediction which combines twitter thread data, academic paper content and biographical user for future work, we aim to widen the pool of considered users by automating the manual validation process and plan on ensuring that the person promoting the work is an author of the paper -which could improve the quality of the summary. in scope of the paper we focused on presenting a proof-ofconcept study, aiming to determine whether it is feasible to predict popularity of twitter posts based on content, and whether such a dataset of significant size can be collected. we believe we have sufficiently demonstrated the quality of the dataset and the feasibility of the task to indicate its value for related nlp research areas. fan zhou, xovee xu, goce trajcevski, and kunpeng zhang. 2021. a survey of information cascade analysis: models, predictions, and recent advances. acm computing surveys (csur), 54(2):1-36.