Improving Temporal Generalization of Pre-trained Language Models with Lexical Semantic Change (2022)

Abstract: Recent research has revealed that neural language models at scale suffer from poor temporal generalization capability, i.e., language model pre-trained on static data from past years performs worse over time on emerging data. Existing methods mainly perform continual training to mitigate such a misalignment. While effective to some extent but is far from being addressed on both the language modeling and downstream tasks. In this paper, we empirically observe that temporal generalization is closely affiliated with lexical semantic change, which is one of the essential phenomena of natural languages. Based on this observation, we propose a simple yet effective lexical-level masking strategy to post-train a converged language model. Experiments on two pre-trained language models, two different classification tasks, and four benchmark datasets demonstrate the effectiveness of our proposed method over existing temporal adaptation methods, i.e., continual training with new data. Our code is available at https://github.com/zhaochen0110/LMLM.

## introduction neural language models (lms) are one of the frontier research fields of deep learning. with the explosion of model parameters and data scale, these language models demonstrate superior generalization capability, which can enhance many downstream tasks even under the few-shot and zeroshot settings @xcite @xcite . although these models have achieved remarkable success, they are trapped by the time-agnostic setting in which the model is trained and tested on data with significant time overlap. however, real-world applications usually adopt language models pre-trained on past data (e.g., @xcite figure 1 : examples of lexical semantic change across time, where the semantic of word church is stable as it often refers to a building or a local congregation for christian religious activities. however, the semantic of word consistent varies dramatically at different times. and roberta @xcite ) to enhance the downstream task-specific models for future data, resulting in a temporal misalignment @xcite . recent works have empirically demonstrated that such a misalignment hurts the performance of both the upstream language models and downstream task-specific methods @xcite . to better understand and solve the temporal misalignment problem, a series of studies have been launched on pre-trained language models (plms) and downstream tasks. the analysis on plms @xcite revealed that plms (even with larger model sizes) encounter a serious temporal generalization problem, and the misalignment degree increases with time. they also found that continually pre-training plms with upto-the-minute data does mitigate the temporal misalignment problem but suffers from catastrophic forgetting and massive computational cost since further pre-training the converged plms is as difficult as pre-training from scratch. the study on downstream tasks further indicates that temporal adaptation (i.e., continually pre-training with unlabelled data that is mostly overlapped in time), while effective, has no apparent advantages over domain adaptation @xcite (i.e., continually pre-training with domain-specific un-labelled data) and fine-tuning on task-specific data from the target time @xcite . to analyze the reason behind the limited performance of temporal adaptation, we launch a study from the lexical level, which also matches the token-level masking operation in advanced plms. unlike existing research that launches analysis on part-of-speech (pos), topic words, and newly emerging words, we mainly explore the correlations between language model performance and tokens/words with salient lexical semantic change, which is also an extensively studied concept in computational linguistics @xcite @xcite to investigate how the semantics of words change over time. experimental results demonstrate that tokens/words with salient lexical semantic change do contribute much more than the rest of tokens/words to the temporal misalignment problem, manifested as their significantly higher perplexity (ppl.) over randomly sampled tokens from the target time. however, the widelyadopted masked language model (mlm) objective in state-of-the-art plms uniformly deals with each token/word, letting the salient lexical-level semantic change information over time being overwhelmed by other tokens/words, which can also explain why temporal adaptation has no obvious advantage compared with domain adaptation. based on the above findings, we propose a lexical-based masked language model (lmlm) objective to capture the lexical semantic change between different temporal splits. experimental results demonstrate that our proposed method yields significant performance improvement over domain adaptation methods on two different plms and four benchmark datasets. extensive studies also show that lmlm is effective when utilizing different lexical semantic change metrics. in a nutshell, our contributions are shown below: ## linking temporal misalignment with lexical semantic change recent work on temporal adaptation @xcite has found that post-tuning the converged plms with unlabeled time-specific data by reusing the mlm objective can make the plms perceive related event-driven changes in language usage. such adaptation can achieve decent performance because the widely-adopted mlm objective can capture the overall changes in the data distribution by randomly masking a specific ratio of the whole sequence. however, such a training objective makes the lexical-level temporal information ignored or overwhelmed by the time-agnostic tokens/words, resulting in little to no performance superiority over domain adaptation methods. based on the above background, it is natural to explore the role of lexical-level temporal information in temporal adaptation, i.e., whether these tokens/words with salient lexical-semantic changes foot_0 over time impair the temporal adaptation performance. as a result, we launch a thorough study from the perspective of lexical semantic change to figure out the reason behind the limited performance of temporal adaptation in the specific domain. to the best of our knowledge, this is the first study that explores the correlation between the lexical-semantic change and the temporal adaptation of plms. we will firstly illustrate our methods to find those semantic changed words in section 2.1 and introduce the discovery experiment as well as analyze the results in section 2.2. ## lmlm objective masked language model (mlm) objective is a widely-adopted unsupervised task proposed by @xcite , which randomly masks proportional tokens and predicts them. since the degradation of the plm in a specific domain over time is mainly attributed to the words with salient semantic change, we should make the plm more aware of them. thus, we propose our lexicalbased masked language model (lmlm) objective. contrary to the traditional random masking strategy (mlm), lmlm preferentially masks the words with salient semantic change over time. formally, given the text set d @xmath0, d t 2 , • • • , d t n } at time t, we select the candidate words w t with the aforementioned detection method and rank them according to their jsd value. then, we select k (@xmath1, 200, • • • , 1000}) words which have relative high scores as the masking candidates w t mask . given masking ratio α, lmlm firstly selects the words in the w t mask to mask. if there are not enough candidates to meet the total number of masking tokens, lmlm masks the other words in the text randomly. the whole process is corresponding to the step ④ in figure 2 . assuming it masks m tokens in total and the sentence after masking is d t ′ i . the optimization objective of lmlm can be formulated as: we conduct experiments on the classification task by employing the pre-trained bert model implemented with the hugging-face transformers package foot_6 in all experiments. further details about model training and parameters can be found in appendix c. we will introduce the datasets and the time-stratified settings in section 4.1, the baselines in section 4.2, and show the results in section 4.3. ## study in this section, we conduct extensive studies to help better understand our method. it is worth noting that all the experiments in this section are conducted on the bert model with the arxiv dataset unless there is a clear explanation. ## conclusion & future work in this paper, we investigate the temporal misalignment of the plms from the lexical level and observe that the words with salient lexical seman-tic change contribute significantly to the temporal problems. we propose a lexical-based masked language model (lmlm) objective based on the above observation. experiments on two plms with the sequence classification task on three datasets under the specific domain setting and one name entity recognition task under the open-domain setting confirm that our proposed method performs better than the previous temporal adaptation methods and the state-of-the-art domain adaptation methods. in the future, we will keep discovering such temporal misalignment problems in the text generation tasks, e.g., machine translation, and improve our method by reducing the extra offline computational cost on procedures like semantic change detection.