T witter Topic Classification (2022)

Abstract: Social media platforms host discussions about a wide variety of topics that arise everyday. Making sense of all the content and organising it into categories is an arduous task. A common way to deal with this issue is relying on topic modeling, but topics discovered using this technique are difficult to interpret and can differ from corpus to corpus. In this paper, we present a new task based on tweet topic classification and release two associated datasets. Given a wide range of topics covering the most important discussion points in social media, we provide training and testing data from recent time periods that can be used to evaluate tweet classification models. Moreover, we perform a quantitative evaluation and analysis of current general- and domain-specific language models on the task, which provide more insights on the challenges and nature of the task.

## introduction social media platforms, e.g., twitter, snapchat, tiktok and instagram, provide an environment for content creation and information sharing among people. on social platforms, every individual can express their views about current events or anything that they care about, influencing and guiding discussions among their friends and followers. social media platforms are highly studied to understand behaviors among users, groups, organizations, or even societies @xcite , and in particular to understand opinion of people regarding a variety of topics such as politics @xcite , diversity and inclusion @xcite , tv shows @xcite , sports events @xcite , or finance @xcite . however, one of the biggest challenges in understanding this type of user generated content, is the noise and variety of these texts @xcite . consequently, identifying topics within social media platforms from their posts is not a trivial task. existing solutions can be divided into topic modeling and topic classification. for topic modeling, topics are detected in an unsupervised way with models such as latent dirichlet allocation (lda) @xcite and subsequent variations @xcite . similarly, solutions that use new bert contextualized embeddings (like bertopic @xcite ) have increased in popularity as they offer increased performance. however, these approaches assume that (i) all the topics of interest are represented in the documents included in the study, and (ii) the terms present in these documents are enough to characterize each topic. for these reasons, these methods are usually built as an ad-hoc analysis. another limitation of these models is interpretability, as it is hard to generalize and label each cluster topic. on the other hand, topic classification approaches the problem in a supervised manner and assigns multiple topics to each document based on a predefined set of categories. this approach overcomes the issues of interpretability and is not based on assumptions about the vocabulary distribution mentioned above. however, the downside of topic classification is that relies on curated datasets labeled by human annotators, and this can be expensive and time consuming to create. in this paper, we introduce tweettopic, a topic classification dataset on twitter data. to the best of our knowledge, this is the first large-scale topic classification dataset specifically tailored to social media, rather than standard text as news articles @xcite or scientific papers @xcite . the dataset consists of a total of 11,267 tweets collected through a time period from september 2019 to august 2021. each tweet is assigned one or more topics from a predefined set of categories curated by social platform experts. aiming to test the robustness of our dataset through time and across topics, we perform several classification experiments, both single-label and multi-label, while utilizing state-of-the-art language models. 3 2 related work social media. social media have become an important aspect of the daily life of millions of people, with 81% of adults in the u.s. stating to have used at least one social platform in 2021 @xcite and over 57% of people in eu interacting through social media in 2020 @xcite . in recent years, an increasing number of corporations seem to dedicate a more significant portion of their marketing funds to advertising on social platforms compared to other more traditional mediums @xcite . at the same time, social media has become a political battleground where politicians both debate between them and try to communicate with their voters, @xcite . finally, social platforms have been used extensively by their users as a means for almost instantaneous news updates both for day-to-day events @xcite , and human and natural disasters (e.g., the ukrainian war or the covid-19 pandemic) @xcite . therefore, a large volume of content is being generated in social media everyday. its polymorphism also means that performing any targeted analysis on the data can be a challenging and timeconsuming process @xcite . furthermore, even though there are various existing tools focused on analyzing social media data @xcite , there is no established way to efficiently identify and filter only relevant and valuable content @xcite . topic modeling. topic models are unsupervised methods to identify relevant topics given a text corpus. lda @xcite is one of the most popular algorithms for topic modeling. however, despite being successful in identifying topics in traditional media @xcite , lda often struggles when applied to short, unstructured, and constantly evolving texts, such as twitter data @xcite . it also typically underperforms when compared to other supervised methods @xcite . more recently, several variations of lda have been proposed to address these challenges with social media texts, such as combining author-topic modelling with lda @xcite , frameworks like twitter-lda @xcite where noisy words and author information are taken into account, and sklda (tajbakhsh and bagherzadeh, 2019), where semantic relations between words extracted from wordnet are taken into account. however, lda-based methods are often not ideal when we need to assign more than one topic to a document. even though there are approaches to acquire multiple labels for each topic, they are usually based on hierarchical @xcite or graph @xcite ) architectures which, depending on the use case, make assumptions about relations of the topics that may not be present in a given corpus (i.e. parent/children topics). furthermore, semi-supervised or supervised variations of lda, such as plda @xcite and slda @xcite , have been been used on twitter data @xcite . while such methods have potential for increased performance they usually require prior labelling or information about the documents and thus remove a major advantage they have compared to supervised approaches. finally, as a mainly unsupervised technique, evaluating the results of topic modeling can be a hard task. metrics such as purity, mutual information and pairwise f-measure are used to evaluate the quality of topics/clusters created by the models @xcite . on the other hand, qualitative analysis is usually difficult to perform due to the lack of interpretability of topics produced and the difficulty increases with the amount of topic. in contrast to traditional lda approaches, techniques such as bertopic @xcite and top2vec @xcite attempt to make use of existing knowledge from pretrained language models by extracting embedding representations of tweets and using them to perform topic clustering. both berttopic and top2vec tend to be easier to use than lda, without the need for extensive hyper-parameter tuning, and often result in increased performance @xcite . however, they do have disadvantages, namely: not performing well on small datasets (abuzayed and al-khalifa, 2021), generating a lot of outlier topics @xcite , and requiring existing knowledge. finally, these approaches suffer similar drawbacks to lda regarding evaluation and interpretability. ## tweet topic classification this section presents the pipeline to construct tweettopic, our topic classification dataset based on twitter data. this pipeline is divided into three steps: (i) tweet collection, (ii) data filtering, and (iii) topic annotation. these steps are explained in more detail in the next subsections. ## evaluation in this section, we present our experimental results. ## analysis in this section, we analyse two important aspects of the tweettopic dataset, mainly its temporal dimension (section 5.1) and the errors made by the systems (section 5.2). ## conclusions & future work in this paper we presented tweettopic, the first large-scale dataset for tweet topic classification. given the prominence of social media in recent times, this dataset can help build supervised models for clustering and organising the online content. the curated set of topics contains a diverse and broad set of categories that cover most topics present in social platform data. this dataset can further motivate research on the evolution of these initial topics on social platforms, i.e., the extension of the existing categorization to new topics or subtopics that will emerge and fade over time due to user engagement. moreover, tweettopic has been shown to be relatively resilient to temporal changes, and it offers easily interpretable results. based on these contributions, we believe that this dataset will be useful for a significant number of researchers and practitioners working on social media, including computational social science and data mining experts, given the relevance of the topic for extracting information and understanding online behavior. finally, while this first iteration of tweettopic focuses on english, our aim is to apply the same methodology to other languages, for which our guidelines and process to construct the dataset described in section 3 can serve as the main basis.