T ime LM s: Diachronic Language Models from T witter (2022)

Abstract: Despite its importance, the time variable has been largely neglected in the NLP and language model literature. In this paper, we present TimeLMs, a set of language models specialized on diachronic Twitter data. We show that a continual learning strategy contributes to enhancing Twitter-based language models’ capacity to deal with future and out-of-distribution tweets, while making them competitive with standardized and more monolithic benchmarks. We also perform a number of qualitative analyses showing how they cope with trends and peaks in activity involving specific named entities or concept drift. TimeLMs is available at github.com/cardiffnlp/timelms.

## introduction neural language models (lms) @xcite @xcite are today a key enabler in nlp. they have contributed to a general uplift in downstream performance across many applications, even sometimes rivaling human judgement @xcite , while also bringing about a new paradigm of knowledge acquisition through pre-training. however, currently, both from model development and evaluation standpoints, this paradigm is essentially static, which affects both the ability to generalize to future data and the reliability of experimental results, since it is not uncommon that evaluation benchmarks overlap with pre-training corpora @xcite . as an example, neither the original versions of bert and roberta are up to date with the current coronavirus pandemic. this is clearly troublesome, as most of the communication in recent years has been affected by it, yet these models would barely know what we are referring to when we talk about covid-19 or lockdown, to name just a few examples. the lack of diachronic specialization is especially concerning in contexts such authors marked with an asterisk (*) contributed equally. in this paper, we address this issue by sharing with the community a series of time-specific lms specialized to twitter data (timelms). our initiative goes beyond the initial release, analysis and experimental results reported in this paper, as models will periodically continue to be trained, improved and released. ## related work there exists a significant body of work on dealing with the time variable in nlp. for instance, by specializing language representations derived from word embedding models or neural networks @xcite @xcite @xcite . concerning the particular case of lms, exposing them to new data and updating their parameters accordingly, also known as continual learning, is a promising direction, with an established tradition in machine learning @xcite @xcite . other works, however, have proposed to enhance bert-based topic models with the time variable @xcite . with regards to in-domain specialization, there are numerous approaches that perform domain adaptation by pre-training a generic lm on specialized corpora. a well-known case is the biomedical domain, e.g., biobert @xcite , scibert @xcite or pubmedbert @xcite . in addition to these approaches to specialize language models, there have been similar temporal adaptation analyses to the one presented in our paper @xcite . in particular, these works showed that training language models in recent data can be beneficial, an improvement that was found to be marginal in @xcite in a different setting. in terms of continual lifelong learning, which is tangential to our main goal, @xcite provide a detailed survey on the main techniques proposed in the nlp literature. more relevant to this paper, on the other hand, are lms specialized to social media data, specifically twitter, with bertweet (nguyen et al., 2020), tweeteval @xcite and xlm-t @xcite being, to the best of our knowledge, the most prominent examples. however, the above efforts barely address the diachronic nature of language. crucially, they do not address the problem of specializing lms to social media and putting the time variable at the core of the framework. moreover, it is desirable that such timeaware models are released alongside usable software and a reliable infrastructure. our timelms initiative, detailed in section 3, aims to address the above challenges. ## timelms: diachronic language models from twitter in this section, we present our approach to train language models for different time periods. ## evaluation in this section, we aim at evaluating the effectiveness of time-specific language models (see section 3) on time-specific tasks. in other words, our goal is to test the possible degradation of older models over time and, accordingly, test if this can be mitigated by continuous training. evaluation tasks. we evaluated the released language models in two tasks: (1) tweeteval (barbieri et al., 2020), which consists of seven downstream tweet classification tasks; and (2) pseudoperplexity on corpora sampled from different time periods. while the first evaluation is merely aimed at validating the training procedure of the base language model, the second evaluation is the core contribution of this paper in terms of evaluation, where different models can be tested in different time periods. ## python interface in this section we present an integrated python interface that we release along with the data and language models presented in this paper. as mentioned in section 3.3, all language models will be available from the hugging face hub and our code is designed to be used with this platform. our interface, based on the transformers package @xcite , is focused on providing easy single-line access to language models trained for specific periods and related use cases. the choice of language models to be used with our interface is determined using one of four modes of operation: (1) 'latest': using our most recently trained twitter model; (2) 'corresponding': using the model that was trained only until each tweet's date (i.e., its specific quarter); (3) custom: providing the preferred date or quarter (e.g., '2021-q3'); and (4) 'quarterly': using all available models trained over time in quarterly intervals. having specified the preferred language models, there are three main functionalities within the code, namely: (1) computing pseudo-perplexity scores, (2) evaluating language models in our released or customized test sets, and (3) obtaining masked predictions. users can measure the extent to which the chofor a more extensive evaluation of language models using pseudo-perplexity, we provide a random subset of our test data across 2020 and 2021. 3 to evaluate other models from the transformers package, we provide the 'eval_model' method (tlms.eval_model()) to compute pseudoperplexity on any given set of tweets or texts (e.g., the subset we provide) using other language models supported by the transformers package. both scoring methods not only provide the pseudo-perplexity scores specific to each model (depending on specified model name, or timelms specified mode), but also the pll scores assigned to each tweet by the different models. finally, predictions for masked tokens of any given tweet or text may be easily obtained as demonstrated in code 2. ## analysis to complement the evaluation in the previous section, we perform a more detailed analysis in three important aspects: (1) a quantitative analysis on the degradation suffered by language models over time; (2) the relation between time and size (section 6.2); and (3) a qualitative analysis where we show the influence of time in language models for specific examples (section 6.3). ## conclusion in this paper we presented timelms, language models trained on twitter over different time periods. the initiative also includes the future training of language models every three months, thus providing free-to-use and up-to-date language models for nlp practitioners. these language models are released together with a simple python interface which facilitates loading and working with these models, including time-aware evaluation. in our evaluation in this paper, we have shown how timeaware training is relevant, not only from the theoretical point of view, but also the practical one, as the results demonstrate a clear degradation in performance when models are used for future data, which is one of the most common settings in practice. as future work, we are planning to explicitly integrate the time span variable in the language models, i.e., introducing string prefixes, along the lines of @xcite and @xcite .