# H ow Y ou T ag T weets: Learning User Hashtagging Preferences via Personalized Topic Attention (2021)

Abstract: Millions of hashtags are created on social media every day to cross-refer messages concerning similar topics. To help people find the topics they want to discuss, this paper characterizes a user’s hashtagging preferences via predicting how likely they will post with a hashtag. It is hypothesized that one’s interests in a hashtag are related with what they said before (user history) and the existing posts present the hashtag (hashtag contexts). These factors are married in the deep semantic space built with a pre-trained BERT and a neural topic model via multitask learning. In this way, user interests learned from the past can be customized to match future hashtags, which is beyond the capability of existing methods assuming unchanged hashtag semantics. Furthermore, we propose a novel personalized topic attention to capture salient contents to personalize hashtag contexts. Experiments on a large-scale Twitter dataset show that our model significantly outperforms the state-of-the-art recommendation approach without exploiting latent topics.

## introduction virtual communications are playing an increasingly crucial role in our daily activities. in this profound revolution in interpersonal communications, social media has become the key channel to connect an individual with human society. people are now turning to the online world to exchange viewpoints, voice opinions, and engage in topics they are interested in. nonetheless, the deluge of the online contents streaming through social media every day has presented the concrete challenges for users to catch what they essentially need. figure 1 : the hashtag context snippet of #book (h) is shown on the top and the target user u 's hashtagging history snippet on the bottom. u later tweeted and tagged h. the topic words indicating u 's hashtagging preferences are in italic and blue, so is the tweet in h's context implying its potential to be engaged with u . to deal with this issue, many platforms encourage users to tag their messages with hashtags (e.g., "#covid19" and "#nlpeople") -topic labels referring to similar posts and allowing easier microblog search. however, users may only want to explore the hashtags (and the linked messages) they are interested in, whereas platforms tend to display popular hashtags for all the users. in light of this concern, how to curate personalized hashtags to draw better user engagements? this paper studies hashtag personalization which explores users' hashtagging preferences and predicts how likely a user will put a hashtag in their future posts (henceforth user-hashtag engagements). we hypothesize that one's hashtagging behavior is highly related to two factors: (1) the user's personal interests reflected by their past hashtagging history (user history) and (2) the contents appear with the hashtag in posts (hashtag contexts). to illustrate this intuition, figure 1 shows the sample tweets tagged with #book (h) and some history tweets of a user u . as can be seen, u was a big fan of mystery books, who later tweeted to share a book and tagged h for topic indication. such future engagements can be signalled by the contexts of h from words like "thriller" and "mys-tery" which suggest u 's potential interests to h. in previous research, most studies @xcite ignore the possible semantic change of hashtags and hence insurmountable to tackle a ubiquitous issue in real life -new hashtags might appear after the model is trained and even the old ones may exhibit different meanings in the future (and are hence likely to engage different user groups). another similar-by-name concept is "hashtag recommendation", which attempts to fit a hashtag to a post and is thus post-oriented @xcite . our task is oriented at users, whose past hashtagging behavior is explored to foresee their preferences to the future hashtags. if we consider a broader area of online content recommendation (e.g., tweets @xcite , conversations @xcite ), many studies rely on user-content interaction history with pairwise labels for supervised learning. they are hence unable to adapt the user interests trained from history data to personalize future contents, due to the semantic gap between past and future data. this is however unignorable when studying social media contents which commonly exhibit evolving topics. to leverage user hashtagging history (in the past) to hashtag contexts in the future, we first conduct unsupervised learning to encode hashtags, whose embeddings are later used for joint training to gain user interests from hashtagging history. for hashtag modeling, a pre-trained bert @xcite and a neural topic model (ntm) @xcite work together to couple the effects of local contexts on message level (handled by bert) and global contexts observed from word statistics (handled by ntm). furthermore, we propose a novel mechanism of personalized topic attention over hashtag contexts to capture the keypoints therein, where the useful features might be sparse and the tremendous noise is likely to hinder the model's capability to learn anything helpful. for instance, in figure 1 , only the first tweet in h's contexts explicitly hint the connection of h with u 's interests, while the others are relatively useless. to address such issues, we leverage latent topics and user embeddings to attend salient contextual messages exhibiting the positive match with the user hashtagging history. to the best of our knowledge, we are the first to personalize hashtags via joint training to adapt user interests gained in history to future hashtag semantics, where the joint effects of latent hash-tag topics and user hashtagging preferences are explored in a novel personalized topic attention. for the experiments, we gather a large-scale twitter dataset and use absolute time to separate the training and validation data (before the time) from the test (after the time). in this way, hashtags available for user preference modeling (in training) will exhibit a gap compared with the hashtags (in test). moreover, we focus on hashtags that do not appear in the target user's history, considering that users may prefer new and unseen hashtags. main experimental results show that our model significantly outperforms the state-of-the-art recommendation model. for example, we present 0.311 map compared with 0.277 obtained by adapting @xcite . we then examine the effects of varying user history and hashtag contexts, where we observe consistently better results on varying users, while hashtag modeling might be benefited from richer contexts. next, an analysis of our model indicates that all its components collaborate to effectively capture user hashtagging preferences and meaningful topics can be discovered to help attend salient contents for personalizing hashtags. at last, we compare the attended history tweets and the future tweets presenting successful engagements and shed light on our potential to not only answer "yes-or-no" to whether a user will be engaged with a hashtag but also how it happens. ## related work this paper is in line with prior studies for personalizing hashtags. they adopt shallow word statistics (e.g., latent topics @xcite and content factors @xcite ) or handcrafted features @xcite to characterize user behavior. different from them, we employ neural networks to explore users' hashtagging preferences in the deep semantic space, which enables natural inclusion of richer information and better language understanding ability. our work is also related to hashtag recommendation, which "recommends" a hashtag to a microblog post via hashtag ranking @xcite @xcite or generation @xcite unaware of the personalized information. others @xcite encode authors' writing styles for text consistency with the inserted hashtag. bayesian graphical models are adopted for user modeling, which requires massive figure 2: our model that couples the effects of hashtag context encoding (top) and user history encoding (bottom left) with a personalized topic attention (bottom right) to predict user-hashtag engagements. ## hashtag personalization framework this section presents how to predict a user u's future engagement with a hashtag h via leveraging u's hashtagging history and h's contextual messages. figure 2 shows our model architecture. input and output. before we start the story about how our model works, we formulate the task of hashtag personalization here. the inputs are a user u and a hashtag h, where h was not tagged by u in the history messages. to capture u's hashtagging preferences, we employ u's past chatting messages with other hashtags for user modeling. for the modeling of h, the messages sharing the hashtag h are adopted as the hashtag contexts. the output is how likely u will tag h in a future posth forms a user-hashtag engagement pair with u. ## experimental setup datasets and setup. a large twitter dataset was first gathered with the official streaming api in feb 2013, which contains 900m tweets. 3 we then filtered out tweets without hashtags and capped the user history at 50 tweets. next, hashtag texts are hidden from both history and contexts to avoid the trivial features learned by the models, and the tweets presenting hashtags in the middle were ignored to enable better semantic learning (following @xcite ). at last, we removed users who posted original hashtags only (never tagged by others) because these users cannot be taken for prediction. here we distinguish hashtag first-use from reuse because the latter is dominant (taking 82% of hashtags) yet relatively valueless (users tend to see new hashtags), while prior settings tend to mix them together @xcite . for training and evaluation, we rank the tweets by time and take the earliest 80% for training, the latest 10% for test, and the remaining 10% for validation. in this way, the model is trained with past data and tested for the future, which reflects a more realistic scenario compared to random split @xcite . based on our setup, users newly appearing in the test will not be taken away because they have no history to learn user embedding. number of tweets 33,881 number of users 2,571 number of hashtags 22,320 average tweet number per user 13 average hashtags number per user 12 average tweet number per hashtag 3 new hashtag rate (%) 55 ## experimental results in this section, we first discuss the main results in section 5.1 and quantify the effects of varying lengths of user history and hashtag contexts for measuring their engagement potential in section 5.2. then, section 5.3 further interprets our superiority with an ablation study and a case study. at last, we analyze our ability to predict users' future hashtagging behavior in section 5.4. ## conclusion this paper has studied the learning of user hashtagging preferences from history data to predict their future trajectory. neural topic model is jointly trained with the prediction of user-hashtag engagements in a novel personalized topic attention. experimental results demonstrate our effectiveness benefited from the ability to align user hashtagging interests gained from the history to customize their future behavior in the future.