Temporally-Informed Analysis of Named Entity Recognition (2020)

Abstract: Natural language processing models often have to make predictions on text data that evolves over time as a result of changes in language use or the information described in the text. However, evaluation results on existing data sets are seldom reported by taking the timestamp of the document into account. We analyze and propose methods that make better use of temporally-diverse training data, with a focus on the task of named entity recognition. To support these experiments, we introduce a novel data set of English tweets annotated with named entities. We empirically demonstrate the effect of temporal drift on performance, and how the temporal information of documents can be used to obtain better models compared to those that disregard temporal information. Our analysis gives insights into why this information is useful, in the hope of informing potential avenues of improvement for named entity recognition as well as other NLP tasks under similar experimental setups.

## introduction natural language processing models are now deployed on a large scale in many applications and used to drive automatic analyses or for making predictions. the usual setup is that these models are trained and evaluated on the data available at model building time, but are used to make inferences on data coming in at a future time, making models susceptible to data drift. the data distribution of the test set used to measure the model's performance after training may be different from the distribution of data from future time periods @xcite . this temporal drift in data often results in lower performance during inference. drift is especially prevalent in information extraction tasks, such as named entity recognition (ner), where the context and the target entities differ across time as a result of changes in language use or the events being discussed @xcite . despite its intuitive value, there has been little research on using the temporal information contained in text documents to inform modeling of a task @xcite , and no past research on modeling sequence labeling tasks in particular. since sequence labeling models are currently trained and evaluated by randomly splitting the available data, performance is measured in an artificially temporal drift-free scenario that is not realistic or similar to how models are used in practice @xcite . when splitting the available training data temporally and testing on the data from the most recent time period, we formulate the following hypotheses: a) models trained on data from a closer time to the test set obtain better results, assuming the same model and data size are used; b) models trained on the combined data from all time periods outperform models trained on subsets of the data, as more data usually leads to better models. in these cases, the commonly used setup of pooling all the data for training while disregarding temporal information may lead to sub-optimal performance. in this paper, we study the temporal aspects of text data, focusing on the information extraction task of named entity recognition in the twitter domain. we make the following contributions: a) a new data set for twitter named entity recognition consisting of 12,000 english tweets evenly distributed across six years; b) experimental results that demonstrate the performance drift of models trained on data from different time periods and tested on data from a future interval; c) extensive analysis of the data that highlights temporal drift in the context of named entities and illustrates future modeling opportunities; d) simple extensions to state-of-the-art ner models that leverage temporal information associated with the training data, which results in an improvement in f1 score over standard pooling methods. ## related work language change is a popular topic of research in linguistics @xcite . in natural language processing, using data from online platforms such as twitter or discussion fora, language change and adoption have been studied at the community level @xcite @xcite and at the individual level @xcite . in some cases, the senses of the same word are known to shift over time @xcite , and modeling such changes in word semantics has been explored using diachronic word embeddings @xcite @xcite . temporal information has been used to create topic models of better quality, usually by adding smoothing properties @xcite . for text classification, the temporal periodicity of twitter hashtags was modeled in preot ¸iuc-pietro and cohn (2013) and used as a prior for text classification models for predicting hashtags on future data, which resulted in performance improvements. most similar to our experimental setup, @xcite study the impact of temporal data splits in text classification, finding that performance worsens on data from future periods, and use standard domain adaptation techniques to incorporate time information and improve results. @xcite introduce a method for training neural networks on data from multiple time intervals while enforcing temporal smoothness between representations. temporal information has also been used to improve named entity disambiguation on a data set of historical documents @xcite . finally, @xcite present a model that uses diachronic word embeddings combined with a method inspired by domain adaptation to improve document classification. a related, but distinct, task built on the assumption of language change with time is automatic prediction of the date on which a document is written @xcite @xcite . named entity recognition (ner) is the task of identifying entities such as organizations, persons, and locations in natural language text. ner is a well-studied nlp task over the past 20 years @xcite ) and is a key information extraction task as its used in various downstream applications such as named entity linking @xcite , relation extraction @xcite and question answering @xcite . on social media text, such as tweets, the performance lags far behind that of standard news corpora @xcite , with data drift as one of the suggested causes @xcite . @xcite show that ner models decay substantially on entity mentions from a different distribution than those seen in training. ner systems struggle to generalize over diverse genres with limited training data @xcite . domain adaptation for ner @xcite @xcite ) is related to our task of improving performance over temporal drift, as the data from a future time period can be considered as a target domain with an unknown distribution. however, the relationship between domains is implied from temporal similarity, and temporal information is very fine-grained in contrast to the standard single source to single target domain adaptation setup. ## temporal twitter data set in this paper, we focus on the task of named entity recognition on english tweets as a case study for our hypotheses and analysis regarding model drift with time. twitter data represents an ideal testbed for our analysis as it contains readily accessible timestamp information for each tweet. further, users on social media post about current events, which are likely to include entities that change over time. social media also reflects changes in language use more timely than other sources of data (e.g., newswire), resulting in the potentially rapid evolution of the contexts and ways in which named entities are discussed in natural language. this drift in twitter data has previously been demonstrated qualitatively in the context of named entity year 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 broad twitter corpus 5 127 2,414 275 6,022 -----current data set ----2,000 2,000 2,000 2,000 2,000 2,000 therefore, we create a new collection of tweets annotated with named entities that attempts to alleviate the lack of temporal diversity in existing twitter data sets as well as provide us with a suitable experimental setup to study our research questions about temporal entity drift and ner model performance. in this section, we present the details of our data set, including the collection and annotation methodology, as well as an analysis of the named entity mentions in the corpus. the data set can be downloaded at https://github.com/ shrutirij/temporal-twitter-corpus . ## data drift to determine the utility of temporal information, we first attempt to evaluate whether temporal drift in the data affects the performance of ner models. to this end, we conduct experiments to answer the following research questions: ## modeling temporal information supported by the analysis that temporal drift in the training data can impact the performance of ner systems, in this section, we experiment with techniques to account for temporal information while training the ner model. we look at leveraging temporality in two broad ways: a) by altering the architecture of the base model; b) by modifying how the training data set is constructed. these methods are intended to be an initial exploration of using temporal information, with a focus on techniques that do not require significant modification to the base model. we present these in the hope that they will inspire future research on models robust to temporal drift. the specific methods are discussed below, followed by experimental results. ## conclusions this paper studies and models text data drift in the information extraction task of named entity recognition. we introduce a new data set of 12,000 english tweets stratified by time, which allows us to study the effects of drift and evaluate named entity recognition models in a realistic scenario of performing inference on temporally unseen data. by analyzing the data, we quantify the temporal drift in named entity type and mention usage and identify that, as expected, the data distribution is more similar when drawn from closer time intervals. we then use current state-of-the-art approaches for named entity recognition and demonstrated that, through modeling of temporal information, performance can be improved when testing on future data. we expect our data, results, and error analysis to inform the design of similar experimental setups for other nlp tasks beyond ner, such as part-of-speech tagging or relation extraction. vikas yadav and steven bethard. 2018. a survey on recent advances in named entity recognition from deep learning models. in proceedings of the 27th international conference on computational linguistics, pages 2145-2158, santa fe, new mexico, usa. association for computational linguistics. justine zhang, robert filbin, christine morrison, jaclyn weiser, and cristian danescu-niculescu-mizil. 2019. finding your voice: the linguistic development of mental health counselors. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 936-947, florence, italy. association for computational linguistics.