Contextualized embeddings for semantic change detection: Lessons learned (2022)

Abstract: We present a qualitative analysis of the (potentially erroneous) outputs of contextualized embedding-based methods for detecting diachronic semantic change. First, we introduce an ensemble method outperforming previously described contextualized approaches. This method is used as a basis for an in-depth analysis of the degrees of semantic change predicted for English words across 5 decades. Our findings show that contextualized methods can often predict high change scores for words which are not undergoing any real diachronic semantic shift in the lexicographic sense of the term (or at least the status of these shifts is questionable). Such challenging cases are discussed in detail with examples, and their linguistic categorization is proposed. Our conclusion is that pre-trained contextualized language models are prone to confound changes in lexicographic senses and changes in contextual variance, which naturally stem from their distributional nature, but is different from the types of issues observed in methods based on static embeddings. Additionally, they often merge together syntactic and semantic aspects of lexical entities. We propose a range of possible future solutions to these issues.

## introduction lexical semantic change detection (lscd) is a relatively recent sub-field within natural language processing. however, comprehensive surveys of data-driven modeling of diachronic semantic change are already available @xcite @xcite . dedicated workshops on computational approaches to historical language change took place at the acl conferences @xcite @xcite and the results of the semeval-2020 task 1 on unsupervised lexical semantic change detection were announced in march 2020 @xcite . shared tasks for other languages followed soon @xcite . the majority of the semeval-2020 shared task participants employed methods based on word embeddings of various types. about half of them tried to make use of contextualized ('token-based') architectures like elmo @xcite or bert @xcite . although the winning systems still used non-contextualized ('static' or 'type-based') embeddings like word2vec @xcite , the difference in scores was not dramatic and we are most likely going to see more work in this direction. we agree with @xcite that as the contextualizing technologies mature, there will be a be er understanding of how to properly use them for semantic change related tasks. indeed, at the rushi eval shared task on lscd for russian @xcite , the leader-board was already dominated by contextualized models. the current paper aims to contribute to this improved understanding by qualitatively analyzing the output of contextualized embedding-based approaches to the diachronic semantic change detection task. hence, our work falls into the second category of ground truth semantic change evaluation, as defined by @xcite : what is evaluated is the ranked output of the methods under investigation. we here focus on subtask 2 of semeval-2020 task 1: to rank a list of words by the degree of their semantic change between two historical corpora belonging to different time bins. the submissions were evaluated by their spearman rank correlation against human annotations. this task was offered for four languages, each with their own word list and corpora: english, german, latin and swedish. one of the submissions in this subtask was delivered by the uio-uva team @xcite . it used pre-trained elmo models and achieved the average score of 0.37 at the evaluation phase (the second best contextual-ized embedding-based system in this phase), and 0.62 at the post-evaluation phase (the best result overall in this phase). we chose their methods for closer inspection, because the implementations were publicly available, and the methods themselves are quite typical for the semantic change detection field (see below). the contributions of this paper are twofold: we propose a categorization of such problematic cases, relating them to inherent properties of pretrained contextualized architectures in particular and distributional approaches in general. ## contextualized methods for detecting semantic change two methods for estimating semantic change were proposed in @xcite : prt and apd (further detailed below). the methods are architectureagnostic and can be used with any model able to produce contextualized token representations for a given sequence of word tokens. overall, these methods can be considered typical representatives of using contextualized word embeddings for the task of semantic change detection: they boil down to directly comparing token embeddings of the target word in two periods; see @xcite for a similar technique. another possible approach (which we hope to analyze in the future) is clustering token embeddings into groups loosely corresponding to word senses and then comparing their time-specific distributions @xcite @xcite . the common part of both the prt and apd methods is as follows. given two time periods 1 and 2 , two corresponding corpora 1 and 2 , and a set of target words, a language model (regardless of what it has been pre-trained on) is used to obtain contextualized token embeddings foot_0 of each occurrence of the target words in 1 and 2 . each target word is then represented by two 'usage matrices' u 1 and u 2 consisting of all token embeddings produced for . a change score is computed from these matrices, indicating the degree of semantic change undergone by a word between 1 and 2 . the target words are ranked by this value. the methods differ in how exactly change scores are computed: however, in a realistic se ing it is obviously problematic to assume knowledge of the statistical properties of the target words beforehand. so, how should one choose between the prt and apd methods? we found that simply averaging the prt and apd estimates yields very robust predictions. in note that in addition to the 4 shared task test sets, we also report results on the gems semantic change test set for english @xcite . for individual test sets, the performance of prt/apd usually lies in between prt and apd, but when averaged over all five test sets, it ranks higher than any individual method, and this effect holds for both elmo and bert, with the best result yielded by elmo. when compared to the shared task leader-board @xcite , the prt/apd + elmo combination outperforms all contextualized embedding-based systems in subtask 2, supporting the same observation in @xcite . thus, the apd and prt methods are complimentary, although their predictions are strongly correlated (see the bo om of ## data and models used for our in-depth analysis of the results, we use textual data from the corpus of historical american english or coha @xcite (it is certainly desirable to reproduce this analysis for other languages, which we leave for future work). in particular, we deal with 5 coha sub-corpora corresponding to five decades: @xcite note that this setup is slightly different from the semeval-2020 task 1 in that we have a sequence of five time bins. with this, we aim to trace the lasting evolution of word meaning, not limited to changes between two time periods. the employed time span means we deal with relatively shortterm meaning changes. we chose elmo as a contextualizer based on its better performance ( the list of words to analyze is a concatenation of all words from the semeval-2020 task 1 english test set, all words from the gems test set, and 1000 randomly sampled words occurring in all five coha sub-corpora with frequency in each sub-corpus higher than 100. af-ter excluding numerals, function words and the words with a total frequency of less than 1000 occurrences across all decades (to discard unstable representations of rare words), the resulting word list contains 690 entries. for each of them, we used our elmo model to calculate their prt/apd scores in the four consecutive pairs of the coha decades @xcite , thus producing a score matrix @xmath0. below we examine the actual scores in this matrix, and how they are related to processes in the recent history of english. ## well-behaved examples for many words, the scores do signal real changes, like a new emergent sense. let us consider the word 'cell' as an example. the dataset from @xcite , based on the oxford english dictionary definitions, mentions it as having acquired a new sense of 'mobile phone' a er 2000. recall that prt/apd produces as an output a measure of how strong the semantic change of a target word was between two time bins; this measure characterizes a pair of decades in our case. 'cell' received a change coefficient of 0.673 for the 1960-1970 pair (arguably corresponding to the start of its widespread usage in the biological sense). a er that, the estimated degrees of change were smaller, with 0.669 for 1970-1980s and 0.672 for 1980-1990s. however, 1990-2000s had a change coefficient of 0.695 (the highest for this word across all decades), most likely reflecting the new 'mobile phone' sense. as a side note, it might look like the prt/apd values show very li le variation: in fact the average standard deviation of m values across four time period pairs is 0.04, with the average prt/apd value being about 0.70. this means that the change coefficients for 'cell' are actually lower than the mean value in our dataset (z-score of 0.695 is -0.17). see more on this in the next section 5. unlike the static word embedding approaches, using contextualized models allows one to visually explore the individual occurrences of a given word in different senses. for this purpose, we use principal component analysis (pca) to reduce the contextualized token embeddings of 'cell' in our diachronic sub-corpora to their 2-dimensional projections. figure 1 shows these projections for the decades from the 1970s through the 2000s. even at a glance, it is possible to see that in the 2000s, some radical changes in the groupings of the 'cell' token embeddings occurred. the three previous decades are all characterized by a rather vague separation of this word's usages into two clusters (at the le and at the right part of the vector space). in the 2000s, we observe the appearance of a new cluster: now there are two strong clusters to the le and a third one to the right. but what senses do these clusters correspond to? fortunately, since each point on the plot represents a particular 'cell' occurrence from a particular decade's sub-corpus, we can retrieve their corpus contexts and manually inspect them. of course, we did not inspect all occurrences: both due to their amount (thousands) and due to the absence of clear-cut cluster boundaries. instead, we randomly sampled about 20 occurrences from the core area of each apparent cluster and examined them. 2 we observe that in the 1970s, 1980s and 1990s, the right-hand cluster mostly contains sentences with 'cell' in the sense of 'prison cell', see example 1: the le cluster (stably increasing its relative size over time) mostly contains sentences with 'cell' in the biological sense, with examples given in 2. 3. 'if those cells are found to be cancerous a er being sent to a lab, that's a definite diagnosis. ' a er exploring the points in the 2000s plot in the same way, one observes that the two clusters on the le correspond to the old senses of 'cell' (biological still at the bo om and prison at the top). but the new large cluster on the right almost exclusively consists of sentences mentioning 'cell' in the sense of 'mobile phone' (see examples in 3 and figure 2 displaying these clusters with labels). (3) 1. 'but how well do the service providers fulfill that objective, and what about the other health and safety risks -exposure to radio waves and potentially fatal driver distraction -that the growing use of cell phones raise?' 2. '…he walked past, nearly dislodging the cell phone she had balanced between her chin and her le shoulder. ' 3. 'you still have the same cell number. ' one can also visualize token embeddings for 'cell' across all five time bins in one plot, as shown in figure 3. here, pca dimensionality reduction is performed for all occurrences of this word (about 7500 total), and thus we can see how usages from different decades (shown in different colors) are grouped in relation to each other. the top right cluster is inhabited almost exclusively with the occurrences from the 2000s and to a figure 3: pca projection of contextualized token embeddings of 'cell' in all 5 coha decades. colors correspond to time periods. less extent the 1990s. not surprisingly, it contains sentences where 'cell' is used in the 'mobile phone' sense. at the same time, in other parts of the plot, occurrences from all decades are distributed more or less uniformly, supporting our previous observation that in the 1960s, 1970s and 1980s, this word did not experience significant semantic changes. in the case of 'cell', the groupings of contextualized representations and the detected changes are undoubtedly connected to a new sense emerging (thus, a diachronic semantic shi ). the relations between different senses of 'cell' fall into the category of homonymy, where word senses are not directly related to each other (at least, synchronically). however, one can trace the cases of polysemy as well, where senses are synchronically related to each other. as an example, let us look at the adjective 'virtual'. it experienced its strongest change of 0.769 in the 1980-1990 pair (its z-score is 1.9 in the full m). before 1990s, 'virtual' was used mostly in two closely related senses: 'being such in essence or effect though not formally recognized or admitted' (major one) and 'related to a hypothetical particle whose existence is inferred from indirect evidence' (minor). 3 however, the 1990s saw the emergence of a large number of 'virtual' usages in the sense of 'simulated on a computer or com-figure 4 : pca projection of contextualized token embeddings of 'virtual' in all 5 coha decades. puter networ ', especially in the expression 'virtual reality' (almost one third of all usages). this sense is related to the previous ones, thus manifesting a case of polysemy. the emergence of a new related sense in the 1990s is captured by contextualized embedding based methods, producing a higher change score for this time bin in comparison to the previous 1980s decade. we can also observe a much weaker change score of 0.740 in the 1990-2000 pair. the manual inspection of the occurrences shows that in the 2000s, 'virtual' was still used a lot in this new third sense (interestingly, the 'virtual reality' expression itself almost came out of usage, constituting now only 6% of all 'virtual' occurrences). on the plot of 'virtual' token embeddings across five coha decades (figure 4 ), the 'simulated on a computer or computer networ ' usages occupy the le part of the plot, with the 'virtual reality' phrases concentrated in the le top corner (as confirmed by manual inspection). the le part contains almost exclusively the occurrences from the 1990s and from the 2000s, while the le top corner is dominated by the 1990s. so far so good: the contextualized embeddingbased methods not only demonstrate high performance on the evaluation sets, they also produce interpretable predictions corresponding to well-known diachronic semantic shi s. but let us also look at a darker side of the m score matrix. ## problematic examples the picture is not as clear if one gets beyond handpicked well-behaved examples. as mentioned above, the change coefficient of 'virtual' when comparing the 1990s to the 1980s was 0.769. but the absolute values (and even z-scores) here are not very informative. there is no well-defined threshold: it is not the case that the change coefficients higher than, say, 0.7 always correspond to some breaking points in the word evolution. there are much stronger bursts which do not yield to such an explanation. ## limitations our analysis in section 5 was based on the top 10 most changed words according to each change detection method. we acknowledge that more insights can be obtained by analyzing more top ranking words (this is also true for static embeddings). another important limitation of this work is our focus on false positives: that is, words which are assigned a high semantic change score when this arguably should not be the case. the study of false negatives (words known to have changed but assigned low scores by the models) is a topic of its own. it is related to possible analysis of the prt, apd and prt/apd predictions on the 'stable' versus 'changed' words from the semeval-2020 test set @xcite . we hope to deal with these aspects in the future. the plots in sections 4 and 5 show token representations of our target word. a potentially more powerful visualization approach could include showing also some 'anchor' or 'seed' words serving to be er disambiguate senses of different tokens (or time-dependent representations for static word embeddings). note, however, that choosing such anchor words is a separate task in itself, see, for example, @xcite . in addition, the plots could arguably be made more visually enticing and insightful by using different markers and sub-sampling of data points (to make the plots look cleaner). this was out of scope for this work. we have qualitatively analyzed the outputs of contextualized embedding-based methods for detecting diachronic semantic change. first, we improved the results of prior work by proposing an ensemble of two methods from @xcite , which proved to be a robust solution across the board, outperforming prior contextualized methods on the semeval-2020 task 1 test sets @xcite and on the gems test set @xcite . our 'prt/apd' method is more suitable for a realistic case of not knowing the gold score distribution beforehand. using prt/apd together with elmo, we produced semantic change coefficients for 690 english words across five decades of the 20 and 21 century using the coha corpus @xcite , and systematically examined these predictions. although many cases of strong detected change do correspond to well-known semantic shi s, we also found multiple less clear-cut cases. these are the words for which a high change score is produced by the model, but it is not related to any 'proper' diachronic semantic shi (not causing a new entry in a dictionary). we discuss such cases in detail with examples, and propose their linguistic categorization. note that these issues do not depend on a particular training algorithm (or an ensemble of algorithms). there is no reason for them to not appear also when using bert or any other token-based embedding architecture; see @xcite and @xcite who show that bert generates representations which form structures tightly coupled with syntax and even sentiment. to properly test it empirically could be an interesting future work, but we have already shown that semantic change detection approaches based on static word embeddings (as opposed to contextualized tokenbased architectures) yield different sorts of problematic predictions. it is not immediately clear whether improving the quality and representativeness of diachronic corpora can help alleviating this issue (producing more historical data is o en not feasible if not impossible). still, it would be interesting to refine our results using larger or cleaner historical corpora: for example, clean coha @xcite . we also plan to analyze the semantic change modeling results for other languages besides english, as well as using different neural network layers to infer semantic change predictions. the data (change scores for all target words) and code (including visualization tools) used in this work is available at https://github.com/ltgoslo/lscd_ lessons .