Named Entity Recognition in T witter: A Dataset and Analysis on Short-Term Temporal Shifts (2022)

Abstract: Recent progress in language model pre-training has led to important improvements in Named Entity Recognition (NER). Nonetheless, this progress has been mainly tested in well-formatted documents such as news, Wikipedia, or scientific articles. In social media the landscape is different, in which it adds another layer of complexity due to its noisy and dynamic nature. In this paper, we focus on NER in Twitter, one of the largest social media platforms, and construct a new NER dataset, TweetNER7, which contains seven entity types annotated over 11,382 tweets from September 2019 to August 2021. The dataset was constructed by carefully distributing the tweets over time and taking representative trends as a basis. Along with the dataset, we provide a set of language model baselines and perform an analysis on the language model performance on the task, especially analyzing the impact of different time periods. In particular, we focus on three important temporal aspects in our analysis: short-term degradation of NER models over time, strategies to fine-tune a language model over different periods, and self-labeling as an alternative to lack of recently-labeled data. TweetNER7 is released publicly ( https://huggingface.co/datasets/tner/tweetner7 ) along with the models fine-tuned on it (NER models have been integrated into TweetNLP and can be found at https://github.com/asahi417/tner/tree/master/examples/tweetner7_paper ).

## introduction named entity recognition (ner) is a longstanding nlp task that consists of identifying an entity in a sentence or document, and classifying it into an entity-type from a fixed typeset. one of the most common and successful types of ner system is achieved by fine-tuning pre-trained language models (lms) on a human-annotated ner dataset 1 https://huggingface.co/datasets/tner/ tweetner7 2 ner models have been integrated into tweetnlp @xcite and can be found at https://github.com/asahi417/tner/tree/master/ examples/tweetner7_paper with token-wise classification @xcite @xcite @xcite . remarkably, lm finetuning based ner models @xcite already achieve over 90% f1 score in standard ner datasets such as conll2003 @xcite and ontonotes5 @xcite . however, ner is far from being solved, specialized domains such as financial news @xcite , biochemical @xcite , or biomedical @xcite still pose additional challenges @xcite . lower performance in these domains may be attributed to various factors such as the usage specific terminologies within those domains, which lms have not seen while pre-training @xcite . among recent studies, social media has been acknowledged as one of the most challenging domains for ner @xcite . social media texts are generally more noisy and less formal than conventional written languages in addition to its vocabulary specificity. in social media, there is another particular feature that needs to be addressed, which is the presence of (quick) temporal shifts in the text semantics (rijhwani and preotiuc-pietro, 2020), where the meaning of words is constantly changing or evolving over time. this is a general issue with language models @xcite , but it is especially relevant given the dynamic landscape and immediacy present in social media @xcite . there have been a few specific approaches to deal with the temporal shifts in social media. for instance, @xcite addressed this issue by pre-training language models on a large tweet collection from different time period, highlighting the importance of having an up-to-date language model. agarwal and nenkova (2022) studied the temporal-shift in various nlp tasks including ner and analyzed methods to overcome the temporal-shift with strategies such as self-labeling. in this paper, we propose a new ner dataset for twitter (tweetner7 henceforth). tweetner7 contains tweets from diverse topics that are distributed uniformly from september 2019 to august 2021. it contains 11,382 annotated tweets in total, spanning seven entity types (person, location, corporation, creative work, group, product, and event). to the best of our knowledge, tweet-ner7 is the largest twitter ner datasets with a high coverage of entity types ttc @xcite ) contains about same amount of annotation yet with three entity types, while wnut17 @xcite has six entity types yet suffer from very small annotations. the tweets for tweetner7 were collected by querying tweets with weekly trending keywords so that the tweet collection covers various topics within the period, and we further removed near-duplicated tweets and irrelevant tweets without any specific topics in order to improve the quality of tweets. we provide baseline results with language model finetuning that showcases the difficulty of tweetner7, especially when dealing with time shifts. finally, we provide a temporal analysis with different strategies including self-labeling, which does not prove highly beneficial in our context, and provide insights in the model inner working and potential biases. ## related work there is a large variety of ner datasets in the literature. conll2003 @xcite and ontonotes5 @xcite are widely used common ner datasets in the literature, where the texts are collected from public news, blogs, and dialogues. wikiann @xcite and multinerd @xcite are both multilingual ner datasets where the training set is constructed by distantsupervision on wikipedia and babelnet. as far as domain-specific ner datasets are concerned, fin @xcite is a ner dataset of financial news, while bionlp2004 @xcite and biocreative @xcite are both constructed from scientific documents of the biochemical and biomedical domains. however, none of these datasets address the same challenges posed by the social media domain. in the social media domain, the pioneering broad twitter corpus (btc) ner dataset @xcite included users with different demographics with the aim to investigate spatial and temporal shift of semantics in ner. more recently, the test set of wnut2017 @xcite contained unseen entities in the training set from broader social media including twitter, reddit, youtube, and stackexchange. the recent twee-bankner dataset @xcite annotated tweebank @xcite with entity labels to investigate the interaction between syntax and ner. the most similar dataset to ours is the temporal twitter corpus (ttc) ner dataset. (rijhwani and preotiuc-pietro, 2020), which was also aimed at analysing the temporal effects of ner in social media. for this dataset, 2,000 tweets every year from 2014 to 2019 were annotated. in general, however, these social media datasets suffer from limited data, non-uniform distribution over time, or limited entity types (see subsection 3.3 for more details). in this paper, we contribute with a new ner dataset (tweetner7) based on recent data until 2021, which is specifically designed to analyze temporal shifts in social media. ## tweetner7: dataset construction, statistics and baselines in this section, we present our time-aware ner dataset from publicly available tweets with seven general entity types, which we refer as tweetner7. in the following subsections, we describe the data collection (subsection 3.1) and annotation (subsection 3.2) processes. we also share relevant statistics (subsection 3.3) and baseline results (subsection 3.4) of our dataset. ## temporal analysis to better understand the effect of the temporal-shift, we conduct three additional comparative experiments: (i) temporal vs. random splits, (ii) joint vs. continuous fine-tuning, and (iii) self-labeling as a solution to deal with temporal shifts. ## conclusion in this paper, we have constructed tweetner7, a new ner dataset for twitter, in which we annotated 11,382 tweets with seven entity types. the collected tweets are distributed uniformly over time from september 2019 to august 2021, which facilitates temporal analysis in ner for social media. the dataset is diverse topic-wise, as we leveraged weekly trending topics to query tweets and nearduplicated and irrelevant tweets were dropped. to establish baselines on tweetner7, we fine-tuned standard lms including a few twitter-specific lms. moreover, we performed a few targeted temporalrelated analyses in order to better understand the short-term temporal effect. finally, we show that self-labeling is not enough to mitigate the temporalshift and had no noticeable improvement over the baseline vanilla fine-tuning, which further highlights the challenging nature of the dataset.