Efficient Continue Training of Temporal Language Model with Structural Information (2023)

Abstract: Current language models are mainly trained on snap-shots of data gathered at a particular time, which decreases their capability to generalize over time and model language change. To model the time variable, existing works have explored temporal language models (e.g., TempoBERT) by directly incorporating the timestamp into the training process. While effective to some extent, these methods are limited by the superficial temporal information brought by timestamps, which fails to learn the inherent changes of linguistic components. In this paper, we empirically confirm that the performance of pre-trained language models (PLMs) is closely affiliated with syntactically changed tokens. Based on this observation, we propose a simple yet effective method named (SG-TLM), which could learn the inherent language changes by capturing an intrinsic relationship between the time prefix and the tokens with salient syntactic change. Experiments on two datasets and three tasks demonstrate that our model outperforms existing temporal language models in both memorization and generalization capabilities. Extensive results further confirm the effectiveness of our approach across different model frameworks, including both encoder-only and decoder-only models (e.g., LLaMA). Our code is available at https://github.com/zhaochen0110/TempoLM .

## introduction while pre-trained language models (plms) have achieved remarkable success on most of nlp tasks @xcite , they neglect the time variable due to their training on snap-shots of data collected at a particular point of time @xcite . however, our language is constantly evolving and changing with new words being created @xcite ; amba hombaiah figure 1 : representative examples of syntactic role changes across time. we track the two most frequent syntactic roles of the word "oval" and "justify" from 2015 to 2018, where the syntactic role distribution of the word "justify" is stable across years, while the word "oval" varies dramatically at different times. @xcite and existing words changing in meanings and usage @xcite @xcite . this "static" training paradigm prevents plms from generalizing over time and modeling language change @xcite @xcite . to make the model training more "dynamic", existing studies have explored temporal language models (tlms), which model temporality by incorporating timestamp directly into representation when pre-training lms, i.e., @xcite , tempot5 @xcite . their methods involve prepending a special time token to each sequence in the training data. through stacking multiple attention layers, each token at different positions can capture the temporal information brought by time. with different time prefixes, plms can adaptively compute the corresponding temporal representations. moreover, tlms have better generalization foot_0 capability over time as standard lms are unaware of which data is "new" and which is "old" due to the absence of timestamps during the training process @xcite . though the methods mentioned above can cap-ture temporal information to a certain degree, they can merely incorporate "superficial" temporal information provided by the time prefix with the masked language model (mlm) objective. thus, it is natural to leverage more temporal-specific information captured by textual tokens, e.g., utilizing a small set of lexicons with salient lexical semantic change @xcite @xcite in the very recent lmlm method @xcite . in this paper, we launch a thorough study to explore the effects of different methods for lexicon selection based on statistical patterns or linguistic attributes and find that the distributional change of syntactic roles @xcite is the most effective strategy in temporal-specific lexicon selection. as shown in figure 1 , the discrete syntactic role distribution of the word "oval" dramatically changes over time, while the word "justify" is stable across years. based on the above observations, we propose a syntax-guided temporal language model (sg-tlm), which consists of two masking strategies: a syntax-guided masking (sgm) and a temporal-aware masking (tam) strategy. experimental results demonstrate that our proposed method significantly improves performance over other tlms methods on two datasets and three tasks. extensive results further confirm the efficiency of our method than the state-of-the-art lexicon selection solution based on semantic change, remarkable transferability across various model frameworks, and its positive impact on adaption to future data. ## related work temporal language model several works have explored the temporal effects in language models @xcite @xcite @xcite @xcite @xcite . recently, existing works have investigated the temporal language model to model temporality information and generalize over time. @xcite and @xcite directly prefix the time token to text sequences and fine-tune on time-stratified data. @xcite and rosin and radinsky (2022) modify the structure of the language model to create time-specific contextualized word representations. @xcite recently proposed a semantic-based, lexical masking strategy to enhance plms' temporal generalization. we extend this work with a detailed study using syntactic role changes to harness temporal-specific information efficiently. diachronic language change diachronic language change can be mainly divided into semantic change @xcite , morphological change @xcite @xcite , and syntactic change @xcite @xcite . previous work focused on discovering the words that have undergone diachronic change under the supervised settings @xcite @xcite @xcite . recently, several works have demonstrated that contextualized word representations have dynamic representation capabilities @xcite @xcite @xcite , which are adopted with unsupervised methods to represent, cluster, and differentiate words across different time periods @xcite . our method utilizes a syntax-based method to detect the salient change words within the text sequence, making the process more interpretable @xcite @xcite . to our best knowledge, this is the first work that incorporates syntactic knowledge into the training of temporal language models. ## conclusion in this paper, we enhance the temporal language model from the syntactic perspective and discover that predicting the syntax-guided lexicons is more challenging for static plms compared to other methods. building upon these findings, we propose a syntax-guided temporal language model (sg-tlm) that incorporates time information into tokens with significant syntactic changes. our sg-tlm achieves the sota performance, reduces computational costs during lexicon construction, and demonstrates excellent transferability to new data and frameworks compared to other baselines.