The challenges of temporal alignment on T witter during crises (2022)

Abstract: Language use changes over time, and this impacts the effectiveness of NLP systems. This phenomenon is even more prevalent in social media data during crisis events where meaning and frequency of word usage may change over the course of days. Contextual language models fail to adapt temporally, emphasizing the need for temporal adaptation in models which need to be deployed over an extended period of time. While existing approaches consider data spanning large periods of time (from years to decades), shorter time spans are critical for crisis data. We quantify temporal degradation for this scenario and propose methods to cope with performance loss by leveraging techniques from domain adaptation. To the best of our knowledge, this is the first effort to explore effects of rapid language change driven by adversarial adaptations, particularly during natural and human-induced disasters. Through extensive experimentation on diverse crisis datasets, we analyze under what conditions our approaches outperform strong baselines while highlighting the current limitations of temporal adaptation methods in scenarios where access to unlabeled data is scarce.

## introduction patterns of language use change constantly over time, often in predictable and analyzable ways @xcite @xcite . as language changes, the performance of nlp systems can be negatively impacted @xcite . in most scenarios, training corpora are derived from a snapshot of data at some moment of time in the past, which puts the reliability of model performance on future data into question. yet, there lacks a concrete reasoning or evidence that temporal adaptation elevates 1 we publish the code for our experiments at https://github.com/ukplab/ emnlp2022-temporal-adaptation . model performance. despite the popularity of large language models and their usefulness in many nlp domains @xcite , the representation of temporal knowledge in those models so far remains an open challenge. the increased interest in temporal adaptation (i.e. scenarios in which the training and test datasets are drawn from different periods of time) has led to the curation of a number of datasets such as nyt annotated corpus @xcite and amazon reviews @xcite that have been the focus of most of the recent work in this area. however, these benchmark datasets are curated in such a way that they can only capture temporal change of language over long periods of time (from years to decades), giving access to a large amount of data. in the contrary, on social media, language changes can happen rapidly @xcite . word usage and topics can even change over the span of a single day @xcite , especially during very dynamic scenarios like crisis or disastrous events @xcite . we denote these phenomena induced by linguistic and semantic changes over time as temporal drift. accounting for temporal drift is critical in crisis situations in which information patterns can vary greatly between the phases of emergency management for crisis. for this purpose, we study short text classification in crisis situations. given the time-critical nature of crisis scenarios, gathering annotations is too time-consuming and transfer learning is challenging due to the innate differences among the type of events (hurricane vs. earthquake) and the respective information needs. thus, we offer a study investigating the impact of temporal drift on crisis datasets spanning shorter time periods (days/weeks), as well as datasets with relatively few samples (ranging from ∼1k to 22k). in summary, we make the following contributions: ## related work analyzing semantic change of text over time has been of great interest since the pioneering work by @xcite and others @xcite @xcite . however, its influence on downstream task performance has only recently gained attention. most importantly, the advent of contextualized word embeddings and large pretrained language models has led researchers to re-evaluate the role of temporality in language modeling @xcite @xcite and text classification @xcite @xcite . the performance degradation due to temporal factors has been confirmed in several studies and across multiple domains. @xcite analyzed the temporal performance degradation of age and gender classification models based on user's social media posts. based on features derived from latent dirichlet allocation and word embeddings, they find that models perform best if test and training data come from the same time span. @xcite investigated temporal effects on hate speech detection in italian social media over the period of five months. their results suggest that models trained on data temporally closer to the test data perform better with transformer based models. @xcite studied semantic shifts in social media and proposed a dataset annotated with words that have undergone a semantic shift over the past two years. @xcite focus on twitter as text domain and contribute pretrained language models which have been further trained on time-specific data from twitter. @xcite propose to use sequential subspace alignment (ssa) to adapt contextualized word embeddings for language change over time. their results suggest that ssa applied on past data is able to outperform baselines which have access to data from all time-steps. röttger and pierrehumbert (2021) compared time-agnostic domain adaptation with temporal domain adaptation which considers the temporal order of the data. they found that, while temporal adaptation clearly outperforms domain adaptation in language modeling, this does not necessarily translate onto downstream classification performance due to updated tokens not being relevant for the task. @xcite found the temporal model performance deterioration to be less significant when using language representations which have been pretrained on temporally closer data. finally, @xcite have made the effort of conducting a large-scale study of temporal misalignment, the generalized scenario where training and evaluation data are drawn from different periods of time. across multiple nlp classification tasks and domains they identify performance degradation with varying degrees but with social media and news being the most affected domains. we contribute to the existing line of work by quantifying the temporal effects on downstream task performance over short time periods (days and weeks) during crisis events. in such a scenario and in contrast to previous work, we do not assume access to large corpora of unlabeled data for temporal adaptation via continuous pretraining. our proposed approaches temporally adapt pretrained contextualized embeddings to learn time-aware embeddings and we evaluate their effects on downstream classification tasks. therefore, in this section we adapt and evaluate methods which are specifically designed for combining temporal adaptation and fine-tuning. their training procedures are adapted to incorporate temporal information about the data along with the textual input. we describe each approach in the following: ## results and analysis in this section, we attempt to answer the following questions: q1. to what degree is temporal performance degradation present in short-term twitter data during crisis events? (section 5.1) q2. does temporal adaptation improve model performance? (section 5.2) q3. does the domain of the data play a role in temporal drift? (section 5.3) q4. how do the proposed models perform when trained continually? (section 5.4) ## discussion adapting temporally by training on timestamp patterns as text prepended to the input (bert+tm) underperforms in all experiments. we argue that the added information affects all tokens equally via the self-attention mechanism although only some tokens will experience a semantic shift relevant for text classification in the crisis scenario. similarly, the lmsoc and dcwe adaptation approaches cannot outperform the baseline without any temporal adaptation. the additional parameters for computing the temporal offset are not well-tuned for predicting temporal distributions which have not been observed during training. figure 3 shows that tda correctly learns to put maximum attention weight on the word katrina (i.e. reference to a previous hurricane) in the temporal context of hurricane. we provide representative examples of tweets in appendix a that all other models but tda fail to classify correctly. forcing the model to learn time-invariant embeddings during training using an adversarial signal leads to tda performing better over all other approaches. although, taph does not fall far behind, it approximates temporal information to create time-static bins. the discrete approximation of temporal information is the main reason behind its performance drop. ## conclusion the usage of natural language inevitably changes over time which influences performance of text classification models applied on data from different temporal distributions. we show that this effect is also prevalent for rapid temporal drift using social media during crisis events as an example. with the rise of pretrained contextualized embeddings, a dominant approach is to continue language modeling on data temporally closer to the target distribution. however, during crisis events such data is not available and annotated data is often scarce. we investigate approaches which work without any additional data besides the input text and its temporal metadata. our results show that under ideal conditions, i.e. high data quality and sufficient annotated instances, they outperform strong baselines. however, most crucially, our work highlights a critical gap of temporal adaptation for rapid temporal drift, namely if unlabeled data for alignment is missing and annotated data is scarce. our work opens the door for future research on methods which do not rely on pretraining in unlabeled target domain data. in this sense, crisis data provides an interesting use case for evaluation. we release all our code and models, fostering future work in this area.