Simple Temporal Adaptation to Changing Label Sets: Hashtag Prediction via Dense KNN (2023)

Abstract: User-generated social media data is constantly changing as new trends influence online discussion and personal information is deleted due to privacy concerns. However, traditional NLP models rely on fixed training datasets, which means they are unable to adapt to temporal change—both test distribution shift and deleted training data—without frequent, costly re-training. In this paper, we study temporal adaptation through the task of longitudinal hashtag prediction and propose a non-parametric dense retrieval technique, which does not require re-training, as a simple but effective solution. In experiments on a newly collected, publicly available, year-long Twitter dataset exhibiting temporal distribution shift, our method improves by 64% over the best static parametric baseline while avoiding costly gradient-based re-training. Our approach is also particularly well-suited to dynamically deleted user data in line with data privacy laws, with negligible computational cost/performance loss.

## introduction distribution shift, particularly in the target labels of classification tasks, presents a serious challenge for the deployment of nlp systems in real-world scenarios. the distribution of text data changes over time due to new language usage, events, or trends @xcite @xcite , which causes statically-trained models to become stale without further re-training on new data @xcite @xcite . temporal shift is most evident in social media data, which we study through multi-label tweet-hashtag prediction, where tweets are classified into rapidly changing user-generated trends. recent temporal analysis of named entity recognition models for social media @xcite first, a month's tweets are encoded into embeddings and added to an external, indexed datastore with their associated hashtags-a process that could be automated in a production setting. second, a test tweet is encoded and used to query the most recent datastore, which contains the latest month's training data for temporal adaptation. the top-k tweets from this datastore are then retrieved and re-ranked. preotiuc-pietro, 2020) identifies the advantages of continual exposure to late-breaking training data. given the ever-shifting nature of user-generated data, deployed models must be explicitly designed to (1) adapt to dynamic test distributions to prevent temporal performance degradation, and (2) abide by data privacy laws, such as gdpr, ccpa, etc. @xcite , that mandate prompt removal of user data. in this paper, our main contribution is an empirical analysis of a known, but understudied learning paradigm in the context of temporal adaptation: non-parametric classification via dense retrieval from a datastore. by using a k-nearest neighbor (knn) classifier in conjunction with a static neural text encoder and simple reranker (as depicted in fig 1 ), we demonstrate that continual and automatic updates of the training datastore facilitate quick adaptation and deletion. in contrast to con-temporary neural paradigms, which require gradient descent during fine-tuning, our approach requires no gradient updates-both adaptation and retroactive deletion use minimal compute. in order to analyze our proposed temporal adaptation method, we introduce a new supervised dataset ( §3) focused on hashtag prediction, a multilabel classification task in the social media domain that exhibits label shift over time. hashtag prediction, in which a system must predict the set of hashtags to be assigned to a given input tweet, is particularly amenable to our non-parametric method due to the intrinsic, user-generated categorization. in other words, the latest supervised training data can be automatically scraped and updated periodically on-demand. we release the dataset as tweet ids. in addition to comparing several text encoding strategies and re-ranking methods on the dataset ( §5), we compare against two state-of-the-art parametric baselines based on bart @xcite , a large pre-trained sequence-to-sequence model. these variants include a fine-tuned bart encoder-based classifier, along with the full bart seq2seq model fine-tuned on tweet-hashtag pairs. we find that our best non-parametric approach outperforms static parametric models with an average relative gain of 64% recall under test distribution shift-and, even outperforms conventional gradient-based temporal adaptation of parametric baselines with an average relative gain of 12% recall. together, our empirical analyses highlight non-parametric techniques as a practical and promising direction for handling distribution shift and user-deletion, and may facilitate future work on real-world, temporal nlp system deployment. ## dense knn hashtag prediction model language in user-generated social media data constantly changes, which makes the prediction of new trend categories, or hashtags, challenging for nlp models. fundamentally, hashtag prediction is a multi-label classification problem, in which a set of hashtags must be suggested for a single tweet. as new events influence online discussion, the classification labels change drastically: roughly half of hashtag types in our dataset are newly replaced after 4-5 months (see details in app. fig. 3 ). for this task, we consider a known, but understudied learning paradigm in the context of temporal adaptation: non-parametric classification via dense retrieval from a datastore. by combining a k-nearest neighbor (knn) classifier with a static neural text encoder and simple reranker, as depicted in fig. 1 , we enable simple but effective updating of an external datastore when train/test data changes, in contrast to full re-training of parametric models. our model first retrieves the top-k nearest hashtag labels from a datastore based on l2 distance in neural embedding space, based on this, our dense knn model has three main components: (1) a static neural tweet encoder, such as the fine-tuned bart model @xcite ) that we use, which is only trained once and does not require updates, (2) a datastore that enables fast nearest neighbor search, and (3) a reranker to boost recall performance. we explain components (2) and (3) in more detail next. ## longitudinal hashtag dataset since we require a dataset with fine-grained temporal annotations and aim to study longitudinal effects of temporal distribution shift, we collect a new large-scale benchmark dataset for temporal hashtag prediction on twitter. we scrape tweets from the entire 2021 calendar year organized by week published, keeping only tweet-hashtags in the top-10k most frequent hashtags that week. we observe large label distribution shift over time, as roughly 50% of labels are new after 4-5 months (for detail, see fig. 3 ). for temporal evaluation, we divide our dataset into alternating 3-week train/1-week test splits (nearly, but not exactly aligned with month boundaries), such that the test data is always from a future date. we chose to divide our dataset into weeks because we (a) needed a fine-grained enough division to investigate how model performance decays over time, yet (b) enough data points in each division for supervised training/fine-tuning of high-capacity neural baselines for comparison. of course, these two decisions are at odds with each other since very fine-grained divisions will reduce the amount of training data in each division. finally, (c) we desired a temporal setup that would be practical for real-world use (i.e. avoid excessively frequent updates). for comparison, we also consider the setting with no temporal shift, where each week contains its own train/test splits and all of the 3week test splits are aggregated for evaluation. to facilitate models like ours that require a frozen historical training corpus (e.g. to train our static tweet encoder), we reserve the first 3-week time split. ## experimental setup for the parametric neural classifier, we use bartlarge @xcite with a multi-label classification head. to explore if performance degradation over time is simply due to a fixed label set, we provide another parametric model, a neural seq2seq baseline based on bart with conditional generation head, which can generate hashtags unseen during training. the model is trained to maximize the conditional probability of the tokenized sequence of concatenated hashtags, given the tweet text as input. during test time, we force the model to generate 120 tokens, and then select the first 5 hashtags for calculating recall independent of their order. for our model's neural encoder for knn, we use the same bart architecture as a tweet text encoder. in this setup, we use the neural classifier fine-tuned once on the initial 3-week training set for hashtag classification, and then freeze and re-use it for encoding every subsequent train/test week in the year. that is, instead of continuously retraining our neural encoder for knn, we simply reuse it and swap out the encoded datastores during updating. we describe other encoder variants and their results in §5. ## related work hashtag recommendation. earlier works on hashtag recommendation compute tf-idf vectors from extracted keywords and apply classifiers on these vectors @xcite , while late works incorporate neural nets @xcite @xcite . @xcite collected a dataset of millions of tweets/hashtags across two months of 2012 and designed a personal hashtag recommendation model using user metadata. instead, we release a public dataset of tweets over an entire year, show how to continually update a non-parametric datastore to handle temporal shift. another line of research models semantic word shift as one form of temporal change @xcite @xcite . ## conclusion in this paper, we show that a non-parametric dense knn retriever model is particularly well-suited for the task of temporal adaptation as it can efficiently adapt to changing test distributions over time by easily updating and swapping out datastores. we find it improves by 64% over the best static parametric baselines while avoiding their costly gradient-based re-training. our model also performs well under selective deletion of training data, which is an important feature for removing harmful training examples, or obeying user deletion requests in line with data privacy laws, such as gdpr and ccpa. we hope the release of our dataset will encourage more work on temporal model adaptation in the future.