Mitigating Temporal-Drift: A Simple Approach to Keep NER Models Crisp (2021)

Abstract: Performance of neural models for named entity recognition degrades over time, becoming stale. This degradation is due to temporal drift, the change in our target variables’ statistical properties over time. This issue is especially problematic for social media data, where topics change rapidly. In order to mitigate the problem, data annotation and retraining of models is common. Despite its usefulness, this process is expensive and time-consuming, which motivates new research on efficient model updating. In this paper, we propose an intuitive approach to measure the potential trendiness of tweets and use this metric to select the most informative instances to use for training. We conduct experiments on three state-of-the-art models on the Temporal Twitter Dataset. Our approach shows larger increases in prediction accuracy with less training data than the alternatives, making it an attractive, practical solution.

## introduction prediction performances of live machine learning systems degrade over time due to changes in the statistical properties of the data used for training them. this degradation, also known as temporal drift, happens in different ml tasks, including named entity recognition (ner). due to the nature of the task, authors also call this language drift @xcite . temporal drift effects are amplified in social media. due to the ecosystem's very nature, topics reflect events and interests of a diverse user base and are continuously and rapidly evolving. to study the impact of language drift, we focus our analysis on the case of ner on twitter data. emerging and trending topics are an essential part of twitter. they change quite rapidly, reflecting diverse topics and world 1 we release the code at https://github.com/ ritual-uh/trending_ner . figure 1 : examples of tweets that include the emerging topic 'us', a horror movie released in 2019 events of interest. entities are a significant component of these changes, generating a diverse set of ne tokens. these ever-evolving topics pose a challenge as new entities frequently arise. the new entities are especially problematic as they might not exist in our previous vocabulary or can potentially transform the meaning of a previously observed term. figure 1 shows tweets that include the emerging topic 'us'. after the release of the film, the topic 'us' became trending and aroused wide discussion. to mitigate the impact of temporal drift, we investigate how to effectively and efficiently adapt an already trained ner model to sustain prediction performance over time. we propose an intuitive approach to measure the trendiness of tweets and use this metric to select the most informative instances for retraining. we show that labeling instances based on this approach can yield better downstream performance than randomly sampling tweets for annotation. note that topics such as semantic shift @xcite and active learning @xcite are related to the work we present here. in semantic shift, the core problem is how to trace temporal changes in lexical semantics, including linguistic drifts and cultural shifts. unlike this task, our goal is to leverage the emergence of trends to guide an already trained model. in active learning, researchers have focused on incremental annotation of instances by selecting the most informative ones. the goal is to achieve better results than random sampling. multiple approaches exist to measure the informativeness of data points, but all of them are domain agnostic @xcite . our proposed solution is more straightforward than using uncertainty in ensembles or adversarial networks. however, it effectively increases model performance, and, similar to active learning approaches, it is more efficient than random sampling. to summarize, we make the following contributions: ## emerging trend detection we want to exploit social media's inherent characteristics @xcite , with a focus on twitter, to update model parameters efficiently. we assume that named entities associated with posts that are likely to become trends will be more informative and result in larger performance gains. our emerging trend detection strategy is based on contrasting frequency of words in older data (training data) against frequency in newly collected data (recent data). more specifically, we formulate this task as detection of trending n-grams. we compute the trend scores for each n-gram, n, as follows: when computing trend scores, we filter out stop words as they are usually the most common words but contain the least information. after we compute trend scores for all n-grams in newly collected data, we assign trend scores to the instances by summing over the scores of each n-gram in that instance (tweet). we then use the score to rank instances for labeling and updating the ner model. our approach is flexible as it can be used in combination with any ner model architecture. ## experiments we empirically study the impact of retraining ner models on trending data in two different scenarios. in scenario 1, we retrain the model in an incremental manner with n instances from a newer batch of data in the following year at every iteration. in scenario 2, we retrain the model incrementally as well, but the pool of data we used to select instances includes all years available in the training partitions. in both cases, instances are selected based on their trend scores. we use the temporal twitter dataset from rijhwani and preotiuc-pietro (2020) for all experiments. this dataset is temporally distributed and balanced with a variety of topics. it has 12k tweets collected from 2014 to 2019, with 2k samples from each year. in our experiments, the training set comprises of splits from 2014 to 2018. the validation set and test set have a random sample of 500 (25%) and 1,500 (75%) tweets from 2019, respectively. ## related work previous work has studied trend detection in online social media platforms such as twitter and facebook @xcite @xcite . benhardus and kalita (2013) outlined the methodologies for using the data from online platforms and proposed criteria based on the frequency of words to identify trending topics in twitter. @xcite presented a system to detect bursty keywords that suddenly appear in tweets at an unusually high rate. recently, miot and drigout (2020) investigated the efficiency of deep neural networks to detect trends. however, these techniques are applied without taking named entities into consideration. towards emerging named entities, recent work has mainly focus on identification and classification of unusual and previously unseen named entities. @xcite investigated the effects of data drift and the evaluation of the ner models on temporally unseen data. @xcite studied the disambiguation of named entities with explicit consideration of temporal background. rijhwani and preotiuc-pietro (2020) reported improvements on performance for overlapping named entities under the impact of temporal drift. due to the limitation of resources and lack of annotated data from social media, these ner models tend to have lower performances on emerging named entities. ## conclusion in this work, we propose a simple approach to update model parameters and prevent degradation performance from temporal drifts. our approach is inspired by our observations of how twitter data follows trends in topics that can change very quickly. experimentally, we show that leveraging emerging trends can benefit the recognition of named entities and reduce performance degradation, especially in low-resource scenarios. our proposal is model agnostic, and can potentially be adapted to other nlp tasks that target social media and face the same problems of data evolution and temporal drift. (a) bilstm + crf (b) bert + crf (c) bertweet + crf figure 6: data can only be accessed year by year -each step represents a year from 2014 to 2018. at each step, we add instances from its respective year to the training set. for temporal, we randomly select instances from that given year. for trend, we rank instances based on their trending score. we experiment with 50 (appendix b), 100 and 200 (appendix b) instances per step to show the impact of training size.