Semantic Shift Stability: Efficient Way to Detect Performance Degradation of Word Embeddings and Pre-trained Language Models (2022)

Abstract: Word embeddings and pre-trained language models have become essential technical elements in natural language processing. While the general practice is to use or fine-tune publicly available models, there are significant advantages in creating or pre-training unique models that match the domain. The performance of the models degrades as language changes or evolves continuously, but the high cost of model building inhibits regular re-training, especially for the language models. This study proposes an efficient way to detect time-series performance degradation of word embeddings and pre-trained language models by calculating the degree of semantic shift. Monitoring performance through the proposed method supports decision-making as to whether a model should be re-trained. The experiments demonstrated that the proposed method can identify time-series performance degradation in two datasets, Japanese and English. The source code is available at https://github.com/Nikkei/semantic-shift-stability .

## introduction the use of word embeddings and pre-trained language models has become common practice in natural language processing. word embeddings like word2vec @xcite are used in many applications, and pre-trained language models starting with bert @xcite are updating state-of-the-art performance on a daily basis. researchers and developers use or fine-tune such kinds of models to their own tasks. while the general practice is to start from publicly available models, there are also significant advantages in creating or pre-training unique models that match the domain. in regard to pre-trained * these authors contributed equally. figure 1 : methodology for observing time-series performance degradation by semantic shift stability. it is difficult from a cost perspective to create a pre-train language model each time and compare the performance. instead, by monitoring the degree of semantic shift of the corpora from period to period, we can estimate timeseries performance degradation. language models, for example, scibert @xcite , biobert @xcite , and fin-bert (araci, 2019) are proposed. these models have performed better than other bert models on downstream domain-specific tasks. a similar approach is traditionally used in word embeddings. there are numerous studies and applications of obtaining word embeddings in their own corpora. in creating domain-specific language models, we have to be careful of time-series changes in the characteristics of the corpus. language changes continuously, especially when there are some socially important events. the semantic shift @xcite of existing words and the appearance of new words are occurring regularly. some have reported that such time-series changes cause degradation of performance @xcite @xcite . henceforth, we refer to this phenomenon as time-series performance degradation. one of the solutions to tackle time-series performance degradation is re-training, but the high cost of model building is a bottleneck especially with language models. it is reported that large-scale pretraining requires large amounts of computation. for example, gpt-3 with 175b parameter consumed several thousand petaflop/s-days of compute during pre-training @xcite , and palm with 540b parameter was trained on 6144 tpu v4 chips @xcite . this trend is accelerated by empirical scaling laws for language model performance @xcite , where the loss scales as a power-law with model size, dataset size, and the amount of compute used for training. this study designs a methodology for observing time-series performance degradation of word embeddings and pre-trained language models using semantic shift in a corpus. the degree of semantic shift is computed by comparing two word2vec models created from corpora of different timespan. monitoring performance leads to the decision whether the model should be re-trained (figure 1 ). the methodology has the advantage of avoiding large-scale training to measure performance. the required input is two word2vec models, which can be created much more efficiently than pre-training of language models. for word embeddings, it is also a benefit if we can infer the downstream task performance without experiments. our contributions are as follows. our experiments report that a large time-series performance degradation occurs in the years when semantic shift stability is smaller (section 5). ## related work this section describes the related work from three perspectives and highlights our study. ## semantic shift stability in this section, we define a metric named semantic shift stability based on the degree of semantic shift of two corpora. we propose to use it for detecting time-series performance degradation of word embeddings and pre-trained language models. semantic shift stability is a metric calculated for whole word embeddings. we compute the stability of the semantic shift (stab(w)) on each word w and use the average of all words in the common vocabulary of two word embeddings as the overall score. the procedure to calculate stab(w) and semantic shift stability from two corpora is described in figure 2 . there are four steps followed in the method proposed by @xcite : 1. create word embeddings, 2. set anchor words, 3. intro-duce the rotation matrix, and 4. calculate stab(w). our new point in this study is that we define a metric that averages stab(w), to quantify the semantic shift of two corpora. ## preliminary experiments: time-series performance degradation in this section, we create models that vary by timeseries and analyze them to reveal the performance degradation. the purpose of this preliminary experiments was to quantify the performance degradation that should be detected in the next section. the rest of this section describes the dataset, model creation, and their time-series performance degradation. we used roberta @xcite for pretrained language models and word2vec for word embeddings. roberta is a optimized version of bert, and word2vec is a well-known word embeddings. ## experiments in this section, we calculated semantic shift stability and analyzed the relationship to the time-series performance degradation shown in section 4. ## conclusion and future work this study designs a methodology for observing time-series performance degradation of word embeddings and pre-trained language models by semantic shift stability. it is a metric that can be calculated more efficiently than pre-training language models, which requires large computational cost. monitoring performance via semantic shift stability supports decision-making as to whether a model should be re-trained. we created word embeddings and pre-trained language models that vary by time-series. in particular, we pre-trained and analyze 12 roberta models on a corpus of japanese financial news at different time-span. we quantified the time-series performance degradation in experiments on two corpora, japanese and english. the experiments confirmed that a large time-series performance degradation occurred in the years when semantic shift stability was smaller. our effort is one of the first attempts to propose an efficient way to detect time-series performance degradation, designed as a decision-making support application without large re-training. in future work, we plan to conduct further experiments with more diverse corpora and models. in the present study, the relationship between semantic shift stability and time-series performance degradation was discussed qualitatively based on the calculated quantitative information. additional research should lead us to explore ways to formulate this discussion in a more persuasive manner. our word2vec models, a word2vec model was created with the nikkei corpus from march 23, 2010 to october 31, 2019. for comparison, we used wikientvec, shiroyagi and chive. wikientvec and shiroyagi were trained in japanese wikipedia, and chive was trained in japanese web corpus. each model was evaluated using the japanese word similarity dataset (jwsd) @xcite , the japanese word similarity and relatedness dataset (jwsan) (inohara and utsumi, 2021), and the nikkei corpus. jwsd is a dataset that assigns similarity values from 0 to 10 to words, and has four parts of speech: adjectives (jwsd-adv), verbs (jwsd-verb), nouns (jwsd-noun), and adverbs (jwsd-adj). jwsan is a dataset of similarity and relatedness of nouns, verbs, and adjectives, with similarity and relatedness assigned values from 1 to 7, respectively. there are two datasets: one with all 2145 word pairs (jwsan-2145) and the other with 1400 word pairs (jwsan-1400) carefully selected for distributed representation. spearman's rank correlation coefficient foot_2 was used as the evaluation metric. in the task of nikkei, using the nikkei corpus, genres were predicted from the keywords contained in the articles. keywords are manually assigned by the editors, mainly nouns extracted from the article texts. the average of the word embeddings of each keyword was used as input. the genres were the same as described in section 4.1. accuracy was used as the evaluation metric. the nikkei corpus from january 1, 2020 to november 30, 2021 was used for validation. in particular, the nikkei task showed the highest accuracy among the four models, suggesting that the created word2vec model was useful for the analysis of the nikkei corpus.