A Tale of Two Laws of Semantic Change: Predicting Synonym Changes with Distributional Semantic Models (2023)

Abstract: Lexical Semantic Change is the study of how the meaning of words evolves through time. Another related question is whether and how lexical relations over pairs of words, such as synonymy, change over time. There are currently two competing, apparently opposite hypotheses in the historical linguistic literature regarding how synonymous words evolve: the Law of Differentiation (LD) argues that synonyms tend to take on different meanings over time, whereas the Law of Parallel Change (LPC) claims that synonyms tend to undergo the same semantic change and therefore remain synonyms. So far, there has been little research using distributional models to assess to what extent these laws apply on historical corpora. In this work, we take a first step toward detecting whether LD or LPC operates for given word pairs. After recasting the problem into a more tractable task, we combine two linguistic resources to propose the first complete evaluation framework on this problem and provide empirical evidence in favor of a dominance of LD. We then propose various computational approaches to the problem using Distributional Semantic Models and grounded in recent literature on Lexical Semantic Change detection. Our best approaches achieve a balanced accuracy above 0.6 on our dataset. We discuss challenges still faced by these approaches, such as polysemy or the potential confusion between synonymy and hypernymy.

## introduction recent years have seen a surge to model lexical semantic change (lsc) with computational approaches based on distributional semantic models (dsms) @xcite . while most research in this area has concentrated on developing approaches for automatically detecting lsc for individual words, as in the dedicated semeval 2020 shared task @xcite , there has also been some work on validating or even proposing laws of semantic changes through new dsm-based approaches @xcite @xcite . ultimately, this line of work is very promising as it can provide direct contributions to the field of historical linguistics. in this paper, we consider two laws of semantic change that are very prominent in historical linguistics, but that have to date given rise to very little computational modeling studies. specifically, the law of differentiation (ld), originally due to bréal (1897, chapter 2), posits that synonymous words tend to take on different meanings over time; or one of them will simply disappear. 1 the same idea is also discussed in more recent work, such as @xcite . as an example, the verbs spread and broadcast used to be synonyms (especially in farming), but now the latter is only used in the sense of transmit, by means of radio, television or internet. the verbs plead and beseech are synonyms, but beseech is no longer used nowadays compared to plead. by contrast, the law of parallel change (lpc), foot_1 inspired from the work of @xcite , claims that two synonyms tend to undergo the same semantic change and therefore remain synonyms. as an illustration, @xcite describes the change of swiftly and its synonyms from the sense of rapidly to the stronger sense of immediately. @xcite also observes a parallel change affecting animal terms which acquire a metaphorical sense. these two laws are interesting under several aspects. firstly, these laws go beyond the problem of detecting semantic change in individual words, as they concern the question of whether a lexical relationship between words, in this case synonymy, is preserved or not through time. secondly, these laws make very strong, seemingly opposite, predictions on how synonyms evolve: either their meanings diverge (under ld) or they remain close (under lpc). it is likely that both of these laws might be at work, but they possibly apply to different word classes, correspond to different linguistic or extra-linguistic factors, or operate at different time scales. a largescale study, fueled by computational methods over large quantities of texts, would be amenable to statistical analyses addressing these questions. in this work, we focus on predicting the persistence (or disappearance) of synonymy through time, as a first step toward more complete analyses. prima facie, dsms appear to provide a natural resource for constructing a computational approach for assessing the importance of these laws, as they inherently -through the distributional hypothesiscapture a notion of semantic proximity, which can be used as a proxy for synonymy. following this idea, @xcite propose the first dsm-based method for predicting how synonymous word pairs of english evolve over time @xcite . this research decisively concludes that there is "evidence against the law of differentiation and in favor of the law of parallel change" for adjectives, nouns and verbs alike (i.e., the three considered pos). however, this pioneering work suffers from some limitations that cast some doubts on this conclusion. first off, the predictions made by their approach are not checked against a ground truth, thus lacks a proper evaluation. second, the approach is strongly biased against ld, as only pairs in which both words have changed are considered, excluding pairs in which differentiation may occur (e.g. in spread/broadcast, only the latter word changed in meaning). this paper addresses these shortcomings by introducing a more rigorous evaluation framework for testing these two laws and evaluating computational approaches. we build a dataset of english synonyms that was obtained by combining lexical resources for two time stamps (1890 and 1990) that records, for a given list of synonym pairs at time 1890, whether these pairs are still synonymous or not in 1990. the analysis of this dataset reveals that, contra @xcite and though using the same initial synonym set, synonymous words show a strong tendency to differentiate in meaning over time. with some variation across pos, we found that between 55 and 80% of synonyms in 1890 are no longer synonyms in 1990. moreover, we propose several new computa-tional approaches foot_2 , grounded in more recent dsms, for automatically predicting whether synonymous words diverge or remain close in meaning over time, which we recast as a binary classification problem. inspired by @xcite , our first approach is unsupervised and tracks pairwise synchronic distances over time, computed over sgnsbased vector representations. our second approach is supervised and integrates additional variables into a logistic regression model. this latter model achieves a balanced accuracy above 0.6 over the proposed dataset. ## related work data-driven methods to detect lsc have gained popularity in the recent years @xcite , using increasingly powerful and expressive word representations, ranging from the simple cooccurrence word vectors @xcite to static word embeddings @xcite and transformer-based contextualized word representations @xcite . this line of research lead to the development of shared tasks @xcite @xcite . most often, these tasks concern the evolution of individual words, in effect focusing on absolute semantic change (of words individually). in this paper, we take a different stand, considering the problem of relative change in meaning among pairs of words, specifically focusing on synonym pairs. previous work on word pairs are rare in the current lsc research landscape. a first exception is (turney and mohammad, 2019), who also study the evolution of synonyms. they propose a dataset to track how usage frequency of words evolve over time within a sets of synonyms, as well as a new task: namely, to predict whether the dominant (most frequent) word of a synonyms set will change or not. this task is actually complementary to the one we address in this work. while turney and mohammad (2019) assume the stability of most synonym pairs between 1800 and 2000, and rather investigate the dynamic inside sets of synonymous words across time, we question this alleged stability and attempt to track whether these words remain synonymous at all in this time period. another distinctive motivation of our work is in the empirical, large-scale evaluation of two proposed laws of semantic change, originating from historical linguistics. previous work investigating laws of semantic change with dsms include @xcite and @xcite , who measured semantic change of words between 1800 and 2000 and attempted to draw statistical laws of semantic change from their observations. later, @xcite contrasted these observations and showed that even if these effects may be real, it may be to a lesser extent. the closest work to the current research is the study of @xcite , as they already focus on the two laws of differentiation (ld) and parallel change (lpc). their main motivation was to automatically measure, using dsms, which of the two laws was predominant between 1890 and 1999. to study which of the two laws actually operates, they focus on word pairs that (i) are synonyms in the 1890s and (ii) where both words changed significantly in meaning between 1890 and the 1990s. first, they represent words as probability distributions of direct contexts, using normalized cooccurrence count vectors. then, they measure the (synchronic) semantic proximity of words by computing the jensen-shannon divergence between the corresponding distributions. semantic change in a word is quantified by comparing its semantic space neighborhoods in the 1890s and in the 1990s. finally, for every selected synonymous pair, they pick a control word pair that has a smaller divergence in the 1890s than the associated synonyms. at a later time in the 1990s, if the divergence for the synonyms is larger than that for the control pair, they decide these synonyms have undergone ld, otherwise they predict lpc. ultimately, they found that most pairs (around 60%) have undergone lpc, which would be the dominant law. the pioneering work of @xcite faces a number of shortcomings. first, their restriction to synonymous pairs in which both words changed mechanically excludes certain cases of ld (i.e., where one one word has changed), thus introducing an artificial bias against ld. moreover, they often select near-synonyms as controls (e.g. instructive and interesting) because they constrain control pairs to be closer in divergence in the 1890s than the associated synonym pairs. furthermore, and more importantly, @xcite did not compare their predictions to any ground-truth and there is no evaluation of the reliability of their method. finally, their choice of word representations is not among the state-of-the-art for static methods. in this paper, we consider all synonymous pairs, thus avoiding the bias against ld. we propose different approaches that we compare to @xcite 's control pairs, and we provide results obtained with more recent distributional semantic models. most importantly, we propose a complete evaluation framework to benchmark the different methods, something missing in this prior work. ## problem statement our overarching goal is to develop new computational approaches that are able to automatically predict which pairs of synonymous words underwent ld or lpc. these predictions could be used as a first step towards providing a more refined and statistically meaningful analysis of the two laws. an important milestone towards developing such an approach is to compare it to some ground truth. otherwise, there is no way to assess whether statistics obtained for ld or lpc are indeed reliable, a problem faced by @xcite . unfortunately, there is no existing large-scale resource that records instances of ld/lpc, beyond a handful of examples found in research papers and textbooks in historical linguistics. what exists however are historical lists of synonyms, which we can compare to obtain some form of ground truth. this forces us to consider a slightly different methodological framework, focusing on a more constrained prediction task, namely to detect pairs of synonyms at time t 1 that have remained synonymous or that are no longer synonymous at time t 2(> t 1). ## evaluation dataset this section presents a dataset designed to track the evolution of english synonymous word pairs between two time stamps t 1 and t 2, with t 2 > t 1. specifically, the two time periods considered are the 1890's decade (t 1) and the 1990's decade (t 2). for extracting synonymous pairs in the 1890's (noted s (t 1) ), we use fernald's english synonyms and antonyms (fernald, 1896) as @xcite did. pairs were selected based on a set of specific target words (see appendix a.7). as shown in the construction of this dataset relies on two crucial hypotheses, which seem reasonable to make. first, both lexical resources rely on the same definition of synonymy. second, s (t 2) meets some exhaustivity criterion, in the sense that (u, v) ∈ s (t 1) not appearing in s (t 2) should indicate that u and v are no longer synonymous at t 2, and not be due to a lack of coverage of the resource (i.e., a false negative). wordnet is assumed to be exhaustive enough, as we checked that every word involved in at least one synonymous pair has its own entry in wordnet's database. we exploit the structure of the wordnet database to analyze the different cases of "diff ". word-net includes lexical relations of hyper-/hypo-nymy (e.g., seat/bench) as well as holo-/mero-nymy (e.g., bike/wheel) and antonymy (e.g., small/large) defined over synsets 4 . note that the hyper-/hyponymy relation does not exist in wordnet among adjectives. among nouns and verbs, we observe that around 30% of pairs that were synonyms at t 1 are in an hyper-/hypo-nymy relation at t 2 and two third of them are direct hypernyms in wordnet (their synsets are direct parent/child) indicating the preservation of a very close semantic link. for a further depiction of the dataset in terms of distance in wordnet's graph, see figure 3 in appendix a.4. one cannot entirely exclude that s (t 1) includes some hyper-/hypo-nyms as synonyms. however, even if we extend the notion of synonymy at t 2 to include these cases, we would have only around 45% of all pairs still considered synonyms among 4 as we did for synonyms, we assume that two words w1 and w2 are instances of one of these relations r if r holds for one of their corresponding synset pair. nouns and verbs. this indicates that "diff" largely remains the most common phenomenon with an estimated proportion between 55% and 80%. this finding contradicts the experimental results reported by @xcite with their computational approach (only 40% of differentiation). in lack of additional indication that some of these hyper-/hypo-nym cases at t 2 are indeed synonyms, or that they may also have been hyper-/hypo-nym at t 1, we decided to still consider them as instances of "diff". another argument for this decision is precisely that there are well-known reported cases of lexical semantic changes in which the meaning of a particular word in effect "widens" to denote a larger subset (i.e., becomes an hypernym): this is the case of dog in english that used to denote a specific breed of dogs @xcite . ## approaches this section presents two classes of computational approaches, unsupervised and supervised, for predicting whether pairs of synonyms at t 1 remain synonyms ("syn") or cease to be so ("diff") at a later time t 2. common to all of these approaches is that they are based on two time-aware dsms, one for each time stamp. ## conclusion in this work, we considered two contradicting laws about the semantic change of synonyms. we discussed the necessary adaptations of the problem statement for this particular type of lsc and elaborated a framework to evaluate models for this new classification problem. the use of linguistic resources from two different time periods allowed us to improve model analysis with respect to prior work on the matter. then we proposed unsupervised and supervised approaches relying on measures of semantic change extracted or inspired by existing literature on lsc, and also leveraged the usefulness of explicit word usage frequency information. we compared these approaches in our evaluation framework, finding that distances in vector spaces from different time periods should not be considered equally. we also observed that explicit frequency information actually help distributional methods to capture the change of synonymy. finally we discussed challenges that dsm approaches still face and opened a discussion about the interplay between hypernymy and synonymy.