Language Models Lack Temporal Generalization and Bigger is Not Better (2025)

Abstract: This paper presents elaborate testing of various LLMs on their generalization capacities. We finetune six encoder models that have been pretrained with very different data (varying in size, language, and period) on a challenging event detection task in Early Modern Dutch archival texts. Each model is finetuned with 5 seeds on 15 different data splits, resulting in 450 finetuned models. We also pre-train a domain-specific Language Model on the target domain and fine-tune and evaluate it in the same way to provide an upper bound. Our experimental setup allows us to look at underresearched aspects of generalizability, namely i) shifts at multiple places in a modeling pipeline, ii) temporal and crosslingual shifts and iii) generalization over different initializations. The results show that none of the models reaches domain-specific model performance, demonstrating their incapacity to generalize. mBERT reaches highest F1 performance, and is relatively stable over different seeds and datasplits, contrary to XLM-R. We find that contemporary Dutch models do not generalize well to Early Modern Dutch as they underperform compared to crosslingual as well as historical models. We conclude that encoder LLMs lack temporal generalization capacities and that bigger models are not better, since even a model pre-trained with five hundred GPUs on 2.5 terabytes of training data (XLM-R) underperforms considerably compared to our domain-specific model, pre-trained on one GPU and 6 GB of data. All our code, data, and the domain-specific model are openly available.

## introduction generalizability is a vital aspect of machine learning. a model learns patterns from its training data that it should be able to apply to data it has not seen 1 see our repo for code and data and our huggingface page for the model. before and might differ in some aspects. since the rise of deep learning, black box models and 'big data' in natural language processing, knowing what generalizibility exactly is and how to study it has become problematic @xcite . one aspect of generalizability is generalization over domains and genres; a model trained to detect events in newspaper articles should also be able to do so in tweets. a more specific type of domain shift is temporal shift: where the training and test data come from different periods. temporal shifts are complex as they involve different styles and even different entities and events. these days, vast amounts of contemporary data are available, but systems trained on these data do not always work well on historical data @xcite , not only because of orthographical and syntactic differences, but also because of semantic shift @xcite . event detection is a well-studied task and ex-tremely useful for many different areas of society, such as news digestion @xcite , clinical decision making @xcite and historical research @xcite . it is a complex task and performance on this task has been shown to be prone to suffer from situations where the train and test data come from different domains @xcite . @xcite draw out a taxonomy of various aspects of generalizability in their genbench initiative and highlight understudied regions. they point out that the vast majority of studies focus on shifts between training (finetuning) and testing, not considering shifts between pre-training and training. furthermore, a comparatively small percentage of studies investigate shifts in multiple stages of the modeling pipeline. hupkes et al. (2022) also point out that only a limited number of tasks investigate temporal generalizability, none including event detection. the globalise project 2 is building software that allows historians to search through the archives of the dutch east india company (verenigde oostindische compagnie; voc): a corpus of over 5 million handwritten pages from the 17th and 18th centuries containing valuable information on the history of early colonialism. the corpus has been transcribed and partially annotated for event extraction. the data provide an excellent use-case to study temporal generalizibilty. in this study, we finetune several encoder large language models (llms) on this newly defined event detection task in early modern dutch archival texts. we finetune models trained on contemporary dutch (bertje @xcite and robbert @xcite ), a model trained on dutch from 1500 to 1950 (gysbert @xcite ), a different version of that model for which the voc corpus was added to its pre-training data (gysbert-v2), and multilingual models @xcite . in doing so, our study provides a unique opportunity to investigate shifts in pretrain-train scenarios as well as finetune-test scenarios, investigating temporal generalizability as well as crosslingual generalizability. to the best of our knowledge, we are the first to study temporal generalization with respect to event detection. additionally, we address the issue of generalization of models over different initializations, some-2 https://globalise.huygens.knaw.nl thing genbench does not explicitly take into account. stochasticity is a known problem in nlp @xcite . if a pre-trained model finetuned ten times produces one good model but nine mediocre ones, what does that say about the level of generalizability the pretrained model carries? our study considers this by comparing consistency between models finetuned with different seeds. ## related work earlier literature in nlp shows great effort at creating datasets for event detection in english @xcite @xcite @xcite @xcite . since then many systems were built to tackle this task, from feature engineering techniques @xcite to deep learning @xcite . to improve scores by just a few decimals (68.0 f1 for a finetuned roberta vs. 68.5 for a more complex approach @xcite ), recent event detection systems involve heavy engineering, for example including graph structures (nguyen and grishman, 2018) or ensembling various types of modules into a pipeline @xcite . most of these methods have been tweaked to work well on the most popular benchmarks -ace @xcite and maven @xcite ) -but they remain unable to generalize to other datasets @xcite . machine learning systems have been shown to heavily underperform when tested out of domain in many fields @xcite . in recent years, the field of nlp has also started to raise concerns about perfecting systems on benchmarks, showing how models that reach excellent performance on certain train/test splits fail on simple challenge examples and commit errors in real-world scenarios @xcite , indicating they may rely on stereotypes and memorization @xcite . this suggests that generalization by nlp models is often overestimated @xcite . now, scholars experiment with prompting generative llms in a zero-shot fashion for event detection, but the results vary wildly and are unreliable @xcite . to better understand what generalization means and to make progress, we should investigate what models can or cannot learn when it comes to event detection. domain-specific pre-training of encoder llms has shown to improve downstream performance in various domains @xcite @xcite . it has also been shown that the crosslingual capacities of multilingual llms can work well for early modern dutch @xcite . however, no work has been done yet that thoroughly compares the performance of contemporary, historical and multilingual models on historical text to test their generalizability. ## avenues for future work many of the findings in this paper need further investigation to be explained. as mentioned, we find that the gysbert models are not representative of the voc domain. however, the performance may also be lower because they were based on the uncased version of bert or because they performed quality filtering of the pretraining data, discarding very noisy data. we consciously did not perform any data filtering for globertise in order to represent the noise of the domain. since there is very little documentation available for the gysbert models, it is hard to study these hypotheses further. it remains unclear why mbert outperforms xlm-r. it might be the case that the next sentence prediction (nsp) objective during pre-training, which teaches the model a form of topic modelling @xcite , proves helpful in a topicdependent event detection task like ours. similarly, it might be worthwhile to investigate what makes bert models less stochastic than roberta models. again, it might be the inclusion of nsp, but it could also be due to the different input formatting (that refrains from using two sentences) or roberta's larger batch size. further research could also look into the differences in performance depending on the datasplit in finetuning/testing. since none of the metadata show correlation with the scores per split, the subject matter of the tested document might be the deciding factor in the performance of the model. this deserves more attention. ## conclusion we have shown that encoder llms lack the ability to generalize to different domains and different periods in history. none of these models, including models that have dutch in their pre-training data, comes close to a domain-specific model pre-trained on only 6gb of data on an event detection task in early modern dutch. we do not find a clear correlation between stability over seeds and datasplits and overall performance, but we find that roberta models tend to be more stochastic than bert models. multilingual models are more capable of temporal generalization than single-language models. of the two multilingual models evaluated, the smaller mbert outperformed the larger xlm-r. contemporary single-language models are shown to be least capable of generalization compared to single-language models that included data from before the 19th century. however, even historical models that were not adapted to the specific domain of the archives of the voc company underperformed. our research re-iterates that building language technology that takes the specifics of a domain into account will outperform general models, even if they are much larger.